From b1c7c473ece7d3e0ca203b65169a31f66dcb590c Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Wed, 4 Sep 2024 15:16:19 +0200 Subject: [PATCH] Update week36.ipynb --- doc/pub/week36/ipynb/week36.ipynb | 1483 +++++++++++------------------ 1 file changed, 582 insertions(+), 901 deletions(-) diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb index d2f12bf9e..0e6e87934 100644 --- a/doc/pub/week36/ipynb/week36.ipynb +++ b/doc/pub/week36/ipynb/week36.ipynb @@ -3,9 +3,7 @@ { "cell_type": "markdown", "id": "fa423ef9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "86472638", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Week 36: Linear Regression and Statistical interpretations\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", @@ -28,9 +24,7 @@ { "cell_type": "markdown", "id": "9c3168de", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plans for week 36\n", "\n", @@ -62,9 +56,7 @@ { "cell_type": "markdown", "id": "8cfccc82", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Material for lecture Monday September 2" ] @@ -72,9 +64,7 @@ { "cell_type": "markdown", "id": "95ab8c41", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Important technicalities: More on Rescaling data\n", "\n", @@ -131,10 +121,7 @@ "cell_type": "code", "execution_count": 1, "id": "4b92772e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", @@ -158,9 +145,7 @@ { "cell_type": "markdown", "id": "d8fe3be4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Let us try to understand what this may imply mathematically when we\n", "subtract the mean values, also known as *zero centering*. For\n", @@ -172,9 +157,7 @@ { "cell_type": "markdown", "id": "3ce0bc21", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n", @@ -184,9 +167,7 @@ { "cell_type": "markdown", "id": "f26c8e8f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Recall also that we use the squared value. This expression can lead to an\n", "increased penalty for higher differences between predicted and\n", @@ -201,9 +182,7 @@ { "cell_type": "markdown", "id": "2447a11d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial \\beta_j} = 0,\n", @@ -213,9 +192,7 @@ { "cell_type": "markdown", "id": "4c8e28c1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "for all $j$. For $\\beta_0$ we have" ] @@ -223,9 +200,7 @@ { "cell_type": "markdown", "id": "dff48586", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n", @@ -235,9 +210,7 @@ { "cell_type": "markdown", "id": "3c924625", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Multiplying away the constant $2/n$, we obtain" ] @@ -245,9 +218,7 @@ { "cell_type": "markdown", "id": "951e0c99", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n", @@ -257,9 +228,7 @@ { "cell_type": "markdown", "id": "a33b68df", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Let us specialize first to the case where we have only two parameters $\\beta_0$ and $\\beta_1$.\n", "Our result for $\\beta_0$ simplifies then to" @@ -268,9 +237,7 @@ { "cell_type": "markdown", "id": "3ec7c9d4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n", @@ -280,9 +247,7 @@ { "cell_type": "markdown", "id": "4f448ffb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We obtain then" ] @@ -290,9 +255,7 @@ { "cell_type": "markdown", "id": "f9e6cdf3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n", @@ -302,9 +265,7 @@ { "cell_type": "markdown", "id": "3e22df42", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "If we define" ] @@ -312,9 +273,7 @@ { "cell_type": "markdown", "id": "6d121b0a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mu_{\\boldsymbol{x}_1}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1},\n", @@ -324,9 +283,7 @@ { "cell_type": "markdown", "id": "8e1e3ace", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and the mean value of the outputs as" ] @@ -334,9 +291,7 @@ { "cell_type": "markdown", "id": "05da5eda", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mu_y=\\frac{1}{n}\\sum_{i=0}^{n-1}y_i,\n", @@ -346,9 +301,7 @@ { "cell_type": "markdown", "id": "d13a3da7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we have" ] @@ -356,9 +309,7 @@ { "cell_type": "markdown", "id": "ee2e3df8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0 = \\mu_y - \\beta_1\\mu_{\\boldsymbol{x}_1}.\n", @@ -368,9 +319,7 @@ { "cell_type": "markdown", "id": "747e45be", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In the general case with more parameters than $\\beta_0$ and $\\beta_1$, we have" ] @@ -378,9 +327,7 @@ { "cell_type": "markdown", "id": "f3949436", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n", @@ -390,9 +337,7 @@ { "cell_type": "markdown", "id": "46fdff7a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can rewrite the latter equation as" ] @@ -400,9 +345,7 @@ { "cell_type": "markdown", "id": "1892080c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\beta_j,\n", @@ -412,9 +355,7 @@ { "cell_type": "markdown", "id": "ebd6eadf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have defined" ] @@ -422,9 +363,7 @@ { "cell_type": "markdown", "id": "43d90e59", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mu_{\\boldsymbol{x}_j}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{ij},\n", @@ -434,9 +373,7 @@ { "cell_type": "markdown", "id": "5f4e5751", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "the mean value for all elements of the column vector $\\boldsymbol{x}_j$.\n", "\n", @@ -446,9 +383,7 @@ { "cell_type": "markdown", "id": "c356279a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n", @@ -458,9 +393,7 @@ { "cell_type": "markdown", "id": "f0000876", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "If we minimize with respect to $\\boldsymbol{\\beta}$ we have then" ] @@ -468,9 +401,7 @@ { "cell_type": "markdown", "id": "49f614ae", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n", @@ -480,9 +411,7 @@ { "cell_type": "markdown", "id": "7f339bb7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n", "and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n", @@ -493,9 +422,7 @@ { "cell_type": "markdown", "id": "a443b3a1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n", @@ -505,9 +432,7 @@ { "cell_type": "markdown", "id": "f1c98dff", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "What does this mean? And why do we insist on all this? Let us look at some examples.\n", "\n", @@ -517,13 +442,42 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 1, "id": "1308492f", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "True beta: [2, 0.5, 3.7]\n", + "Fitted beta: [2.08376632 0.19569961 3.97898392]\n", + "Sklearn fitted beta: [2.08376632 0.19569961 3.97898392]\n", + "MSE with intercept column\n", + "0.00411363461744314\n", + "MSE with intercept column from SKL\n", + "0.004113634617443147\n", + "Manual intercept: 2.083766322923899\n", + "Fitted beta (without intercept): [0.19569961 3.97898392]\n", + "Sklearn intercept: 2.0837663229239043\n", + "Sklearn fitted beta (without intercept): [0.19569961 3.97898392]\n", + "MSE with Manual intercept\n", + "0.00411363461744314\n", + "MSE with Sklearn intercept\n", + "0.004113634617443131\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhYAAAGdCAYAAABO2DpVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABswElEQVR4nO3dd3gUVffA8e9m0yskpAGBhE4IBBBDkypNqgjSlKqgCEp5Ebs0pUkTFRSQ8gMBBSlKCaB0kF4khBoSaiCEQHrZMr8/8rIvSxJIILubcj7Ps4/u7Mzds4dN5uTeO3dUiqIoCCGEEELkAytLByCEEEKIokMKCyGEEELkGykshBBCCJFvpLAQQgghRL6RwkIIIYQQ+UYKCyGEEELkGykshBBCCJFvpLAQQgghRL6xNvcb6vV6bt26hYuLCyqVytxvL4QQQohnoCgKiYmJlC5dGiurnPslzF5Y3Lp1Cz8/P3O/rRBCCCHywfXr1ylbtmyOr5u9sHBxcQEyA3N1dc23djUaDdu3b6dNmzbY2NjkW7vCmOTZfCTX5iF5Ng/Js3mYMs8JCQn4+fkZzuM5MXth8XD4w9XVNd8LC0dHR1xdXeVLa0KSZ/ORXJuH5Nk8JM/mYY48P20ag0zeFEIIIUS+kcJCCCGEEPlGCgshhBBC5Buzz7HIDZ1Oh0ajydMxGo0Ga2tr0tLS0Ol0JopMSJ7NpyjlWq1WY21tLZeYC1EMFLjCIikpiRs3bqAoSp6OUxQFHx8frl+/Lr+8TEjybD5FLdeOjo74+vpia2tr6VCEECZUoAoLnU7HjRs3cHR0xNPTM0+/TPV6PUlJSTg7Oz9x4Q7xfCTP5lNUcq0oChkZGdy9e5fIyEgqV65cqD+PEOLJClRhodFoUBQFT09PHBwc8nSsXq8nIyMDe3t7+aVlQpJn8ylKuXZwcMDGxoarV68aPpMQomgqkL+tikK3rxDCWGEvjoQQuSM/6UIIIYTIN1JYCCGEECLfSGEhhBBCiHwjhUU+GDBgACqVCpVKhY2NDd7e3rRu3ZrFixej1+tz3c7SpUspUaKE6QIVQgghTKxIFhY6vcI/EffYeOom/0TcQ6fP25oYz6Jdu3ZER0cTFRXF1q1badGiBSNGjKBjx45otVqTv78QQojiTadX2Dl/JFYXNnLoUoxZzn3ZKXKFRWjYbV6atpPeCw8xYvUpei88xEvTdhIaFm3S97Wzs8PHx4cyZcpQt25dPv30UzZu3MjWrVtZunQpALNmzaJmzZo4OTnh5+fHe++9R1JSEgC7d+9m4MCBxMfHG3o/xo8fD8CKFSuoV68eLi4u+Pj40KdPH2JiYkz6eYQQQhQeoWHRfD56AJzYQofk31n4y3KznPuyU6QKi78v3GPYypNEx6cZbb8dn8bQFSfMnuCWLVsSHBzMunXrgMzL7ebOnUtYWBjLli1j586djB07FoBGjRoxZ84cXF1diY6OJjo6mjFjxgCQkZHBpEmTOH36NBs2bCAyMpIBAwaY9bMIIYQomELDopm8dA2d9x2h3CF7Vt6oymGCLHbuK1ALZD0PnV5h+l9XyK7jRwFUwIQ/w2kd6IPaynzrZFSrVo1///0XgJEjRxq2BwQEMGnSJIYOHcq8efOwtbXFzc0NlUqFj4+PURuDBg0y/H+FChWYO3cuISEhhlUZhRBCFE86vcK4Df/yydkFlEiGO+5g2/JdCLfcua/I9FgcjYrjTmJGjq8rQHR8Gkci48wXFJnLGT9c8GvXrl20bt2aMmXK4OLiQr9+/bh37x7JyclPbOPkyZN06dKF8uXL4+LiQvPmzQG4du2aqcMXQghRgB2JjKPX5SlUjdShUcOykLbYOXsYXrfEua/IFBYxiem53C/t6Tvlo3PnzhEQEMDVq1dp3749QUFB/P777xw/fpwffvgB4Il3ck1OTqZNmzY4OzuzYsUKjh49yvr164HMIRIhhBDF17kdi2h59A4Au+p5cdCpVbb7mfPcV2SGQrxc7HK5n/nuUbBz507OnDnDqFGjOHbsGFqtlpkzZxqWNv7tt9+M9re1tc1ye+zz588TGxvL1KlT8fPzA+DYsWPm+QBCCCEKrLiYmwQsX4qtDi76WzHb5z/kdCY057mvyPRYvOjvjreLLTmNIKkAXzd7QgLcTfL+6enp3L59m5s3b3LixAkmT55Mly5d6NixI/369aNixYpotVq+++47rly5wvLly/nxxx+N2vD39ycpKYm///6b2NhYUlJSKFeuHLa2tobj/vjjDyZNmmSSzyCEEKLw2DVrIKXuQ7wjzK4xBFTqLPuY+tyXnSJTWKitVIxtVQEgS3Hx8Pm4ToEmm7wSGhqKr68v/v7+tGvXjl27djF37lw2btyIWq2mdu3azJo1i2nTphEUFMQvv/zClClTjNpo1KgR7777Lj179sTT05Pp06fj6enJ0qVLWbNmDYGBgUydOpUZM2aY5DMIIYQoHE5uX0E3+8Oo2j0g/PWXuWpTySLnvuyoFEUx6woaCQkJuLm5ER8fj6urq9FraWlpREZGEhAQkOfbKuv1ehISEjh4LYVJm88ZXXLq62bPuE6BtAvyzZfPUJw9zLOrq6vcrdLEilqun+fn25Q0Gg1btmyhffv22NjYWDqcIkvynH/u3IjAblFTSpDEIZ83aPDuPELDopnwZzhxSalMD9Ex9ogad2eHfD33Pen8/agiM8fioXZBPrQN8uVIZBwxiWl4uWR2AZmzWhNCCCFMIT01hf0jX6WGv5a7bpWoO3AWAO2CfGkd6MOhyzHEnjvE4v4v0qCSl0XOfUWusIDMYZGGFT2evqMQQghRiKwb8Qq1w7QkRLhTYum32Nr9r/dPbaUiJMCdLeew6B/Uhb9/VQghhCgGti+dRM19mbdziGwXSJXglywcUfaksBBCCCEKuFtR53CYvxK1Auer2tB90q+WDilHUlgIIYQQBZii13NoZC9KxcM9VwiZsxq1dcGdySCFhRBCCFGArfm8J9XPZ6BTQfK7vSkTEGjpkJ5ICgshhBCigLpxOQynU2cACHvJk7aDvrRwRE8nhYUQQghRAGWkp5G6egBt60VzroU9XeeGWjqkXMlzYXHz5k3efPNNPDw8cHR0pHbt2hw/ftwUsRV6zZs3N7pVuhBCCJFbx5f+h8raSyRbOdHo0zXYOThaOqRcyVNhcf/+fRo3boyNjQ1bt24lPDycmTNnUqJECROFVzgMGDAAlUqV5TF9+nSj+3r4+/szZ84cywUqhBCiUPhz7mju7dhOik7FlYZT8PGrZOmQci1P00qnTZuGn58fS5YsMWzz9/fP75gKpXbt2hnlBcDT0xO1OutNYYQQQoicRJw5iOeyrbglW/OHc1V6te1v6ZDyJE+FxR9//EHbtm15/fXX2bNnD2XKlOG9995j8ODBOR6Tnp5Oenq64XlCQgKQuW68RqMx2lej0aAoCnq9Hr1en5fQeHjLk4fHm5OiKNja2uLl5WW0vWXLlgQHBzN79mxatmzJ1atXGTVqFKNGjQLIcov0wsCSeS5uilqu9Xo9iqKg0WgKVMH98PfQ47+PRP6SPOeOTqvl7NghVE6G2+7Q4qtVecqZKfOc2zbzVFhcuXKF+fPnM3r0aD799FOOHDnCBx98gJ2dHf369cv2mClTpjBhwoQs27dv346jo/F4kbW1NT4+PiQlJZGRkYGiKKRp8vYLNfXegzztnxN7GytUqtwth6rRaNBqtYai6SGtVktGRgYJCQksWbKEl156iQEDBhhy9fj+hUliYqKlQyg2ikquMzIySE1NZe/evWi1WkuHk8WOHTssHUKxIHl+srjNs2kQqUOjhsgubUg4fgY4k+d2TJHnlJSUXO2Xp8JCr9dTr149Jk+eDECdOnU4e/Ys8+fPz7Gw+OSTTxg9erTheUJCAn5+frRp0ybbu5tev34dZ2dn7O3tScnQUmeaZb6EYeNb42ibu/TY2Niwbds2ypYta9jWrl07rK2tsbW1xdXVFVdXV2xsbChVqhSVK1c2VdgmpygKiYmJuLi45LrwEs+mqOU6LS0NBwcHmjZtWuDubrpjxw5at24td900Icnz0x3YMA//A3cAOPuyH31Hz8hzG6bMc27/GM5TYeHr60tgoPHCHNWrV+f333/P8Rg7Ozvs7OyybLexscnyoXU6HSqVCisrK8PDUvLy/iqVihYtWjB//nzDNicnJ3r37m34PI/uW5hvgf2wS76wf47CoKjl2soqsxcwu5/9gqCgxlXUSJ6zF3fnOsqcH7HVwaUANT1mbXmu1TVNkefctpenqBs3bsyFCxeMtl28eJHy5cvnpZlcc7BREz6xba721ev1JCYk4uLqki+/hB1s8jYG7OTkRKVKhWfWrhBCiIJj/7yhlE6GeCeoMX1BgV6y+2nyFPmoUaNo1KgRkydPpkePHhw5coQFCxawYMECkwSnUqlyPRyh1+vR2qpxtLUusH/d2draFsoJm0IIIUznxLbldFbt49YrNpzyH0KDmo0sHdJzydMZ+MUXX2T9+vWsWrWKoKAgJk2axJw5c3jjjTdMFV+R4u/vz969e7l58yaxsbGWDkcIIYSF3bkRQYV/PgbgWvketH9nsoUjen557mvp2LEjHTt2NEUsRd7EiRN55513qFixIunp6YbLCYUQQhQ/6akp/DO0M17lrPD0rUzdgbMsHVK+KLyDOAXI0qVLs92+e/duo+cNGjTg9OnTpg9ICCFEgbduxCvUvqQn9aoz6Qu/wtau4Fwt9TwK5mQEIYQQogjbvnQSNffFAHC5QyBB9dtZOKL8I4WFEEIIYUY3I8NxmL8StQLnq9rQbdKvlg4pX0lhIYQQQpiJTqvl8MhelIqHe64QMmd1ob60NDtSWAghhBBm8vsXvah+QYNOBcnv9qZMQODTDypkpLAQQgghzODG5TDsI/4FIOwlT9oO+tLCEZmGFBZCCCGEiWWkp5G6egCdA+8Q0d6JrnNDLR2SyUhhIYQQQpjYsSWjqKy9RDxO1Bu1GjsHx6cfVEhJYSGEEEKY0J9zR5Owbid3MqyJaDgNH7+ifV+pojUVVQghhChAIs4cxHPZVtySrdjtGUDPtn0tHZLJSY+FEEIIYQI6rZazY4fglgy3PaD1rPWWDskspLAwE5VKxYYNG3J83d/fnzlz5pgtnmfxxRdf8M477xieK4rCkCFDcHd3R6VScerUKZo3b87IkSOf+T2WLl1KiRIlnj9Y8UQDBgzg1VdfzZe2YmJi8PT05ObNm/nSnhBFxW//6UjlSB0ZalCPGoa7t5+lQzILKSzyQUxMDO+88w7lypXDzs4OHx8f2rZtyz///GPp0PLNnTt3+Pbbb/nkk08M20JDQ1m6dCmbNm0iOjqaoKAg1q1bx6RJkwz75LVg6tmzJxcvXsxTbM9bzJjb7t27UalUPHjw4Lna8ff3R6VSGT3Kli2bP0HmgZeXF3379mXcuHFmf28hCqq9a78n8K+rAIS3Kk/T7sMtHJH5yByLfNCtWzc0Gg3Lli2jQoUK3Llzh7///pu4uDhLh2YkIyMDW1vbZzr2559/pmHDhvj7+5OQkABAREQEvr6+NGrUyLCfu7v7c8Xo4OCAg4PDc7XxrJ4nP5YyceJEBg8ebHiuVqstEsfAgQMJCQnhm2++oWTJkhaJQYiCIu7OdXSzf8BWB5cC1PSYucnSIZlVwe6xUBTISM79Q5OSt/2f9MjlLc0fPHjA/v37mTZtGi1atKB8+fKEhITwySef0KFDhxyPmzhxIt7e3pw6dSrb1+Pj4xkyZAheXl64urrSsmVLozujRkRE0KVLF7y9vXF2dubFF1/kr7/+MmrD39+fr776igEDBuDm5sbgwYMNQw3btm2jevXqODs7065dO6Kjo5/4OVevXk3nzp0NzwcOHMj777/PtWvXUKlU+Pv7A8a9B82bN+fq1auMGjXK8Bf10zw+FDJ+/Hhq167N8uXL8ff3x83NjV69epGYmAhkdunv2bOHb7/91vAeUVFRAISHh9O+fXucnZ3x9vamb9++xMbGGtpu3rw5w4cPZ/To0ZQqVYrWrVsDcPbsWTp06ICrqysuLi40adKEiIgIw3FLliyhevXq2NvbU61aNebNm2d4LSoqCpVKxerVq2nUqBH29vbUqFHDcKfbqKgoWrRoAUDJkiVRqVQMGDDgqXnJiYuLCz4+PoaHp6cnOp2Ot956i4CAABwcHKhatSrffvvtE9tZu3YtNWvWxMHBAQ8PD1q1akVycnKuPjNAzZo18fHxYf364jGGLMSTHFn0AVZ6hQdOUGP6giK3ZPfTFOxPq0mByaVztasVUCI/3/vTW2Dr9NTdnJ2dcXZ2ZsOGDTRo0AA7O7sn7q8oCiNHjmTDhg3s37+fypUrZ7tPhw4dcHd3Z8uWLbi5ufHTTz/x8ssvc/HiRdzd3UlKSqJ9+/Z89dVX2Nvbs2zZMjp16sSFCxcoV66coa1vvvmGL774gs8//xyA/fv3k5KSwowZM1i+fDlWVla8+eabjBkzhl9++SXbmO/fv09YWBj16tUzbJszZw6VKlViwYIFHD16NNu/lNetW0dwcDBDhgwx+qs6ryIiItiwYQObNm3i/v379OjRg6lTp/L111/z7bffcvHiRYKCgpg4cSIAnp6eREdH06xZMwYPHsysWbNITU3lo48+okePHuzcudPQ9rJlyxg6dCgHDhxAURRu3rxJ06ZNad68OTt37sTV1ZUDBw6g1WoBWLhwIePGjeP777+nTp06nDx5ksGDB+Pk5ET//v0N7X744YfMmTOHwMBAZs2aRefOnYmMjMTPz4/ff/+dbt26ceHCBVxdXQ09NJMnT2by5MlPzMXWrVtp0qTJE/fR6/WULVuW3377jVKlSnHw4EGGDBmCr68vPXr0yLJ/dHQ0vXv3Zvr06XTt2pXExET27duH8t/iOrefOSQkhH379jFo0KAnxidEUXZi23LaaXaS8LKaw5VH07Bmo6cfVMQU7MKiELC2tmbp0qUMHjyYH3/8kbp169KsWTN69epFrVq1jPbVarX069ePY8eOceDAgRzHw3ft2sWZM2eIiYkxFCozZsxgw4YNrF27liFDhhAcHExwcLDhmK+++or169fzxx9/MHz4/8byWrZsyZgxYwzP9+/fj0aj4ccff6RixYoADB8+3HBSzs7Vq1dRFIXSpf9X5Lm5ueHi4oJarcbHxyfb49zd3VGr1Ya/qp+VXq9n6dKluLi4ANC3b1/+/vtvvv76a9zc3LC1tcXR0dHoPebPn0/dunWNTtSLFy/Gz8+PixcvUqVKFQAqVarE9OnTDft8+umnuLm5sXr1amxsbAAM+wJMmjSJmTNn8tprrwEQEBBAeHg4P/30k9FJdvjw4XTr1s0QS2hoKD///DNjx441DBd5eXkZ9c68++67Rid+vV5PUlISzs7OWFlldi6WKVPGKDcfffSRoWiEzOLkgw8+YMKECYZtAQEBHDx4kN9++y3HwkKr1fLaa69Rvnx5ILMHIq+fuUyZMpw8eTJL+0IUF9HXLlHhn48BOOvXm9b9P3/KEUVTwS4sbBwzew5yQa/Xk5CYiKuLi+GX8HO/dy5169aNDh06sG/fPv755x9CQ0OZPn06ixYtMurmHjVqFHZ2dhw6dIhSpUrl2N7x48dJSkrCw8PDaHtqaqqhSz45OZkJEyawadMmbt26hVarJTU1lWvXrhkd82gvw0OOjo6GogLA19eXmJiYHONJTU0FwN7ePuckmJC/v7+hqICnxwuZOdy1axfOzs5ZXouIiDAUC4/n59SpUzRp0sRQVDzq7t27XL9+nbfeesuoB0ar1eLm5ma0b8OGDQ3/b21tTb169Th37twTY3Z3dzeao6LX60lISMDV1TXH7/SHH35o9B17+L368ccfWbRoEVevXiU1NZWMjAxq166dbRvBwcG8/PLL1KxZk7Zt29KmTRu6d+9OyZIl8/SZHRwcSElJeeJnFKKoSk9N4fjgV7HxtaV8xYq8MGCmpUOymIJdWKhUuRqOAECvBxtd5v75UVjkkb29Pa1bt6Z169Z8+eWXvP3224wbN87ol37r1q1ZtWoV27Zt44033sixLb1ej6+vr2Fc/lEP/8L98MMP2bZtGzNmzKBSpUo4ODjQvXt3MjIyjPZ3csqav8dPmiqVytDtnZ2HJ6v79+9nKXbMIbt49Xr9E4/R6/V06tSJadOmZXnN19fX8P+P5+dJE0cfvufChQupX7++0Wu5mTT5tDkmzzIUUqpUKSpVMl7F77fffmPUqFHMnDmThg0b4uLiwjfffMPhw4ezbVOtVrNjxw4OHjzI9u3b+e677/jss884fPgwjo6ZBXZuPnNcXByenp5PjF+IomrdiHbUvqon7ZY9KQM+w9bOMn+IFQQFu7AoxAIDA7OsW9G5c2c6depEnz59UKvV9OrVK9tj69aty+3bt7G2tjZMinzcvn37GDBgAF27dgUgKSnJMGkxv1WsWBFXV1fCw8OznMSextbWFp1OZ5K4nvQedevW5ffff8ff3x/rPEycqlWrFsuWLUOj0WQpaLy9vSlTpgxXrlx5YmEIcOjQIZo2bQpk/nV//PhxwxDVwytPHo/5WYZCsrNv3z4aNWrEe++9Z9j26OTT7KhUKho3bkzjxo358ssvKV++POvXr2f06NG5/sxhYWE0b978qfEJUdRsWzyRmvvuAnCpYyA9mnezcESWVbCvCikE7t27R8uWLVmxYgX//vsvkZGRrFmzhunTp9OlS5cs+3ft2pXly5czcOBA1q5dm22brVq1omHDhrz66qts27aNqKgoDh48yOeff86xY8eAzLkB69at49SpU5w+fZo+ffo89a/4Z2VlZUWrVq3Yv39/no/19/dn79693Lx50+iKjPzk7+/P4cOHiYqKIjY2Fr1ez7Bhw4iLi6N3794cOXKEK1eusH37dgYNGvTEQmf48OEkJCTQq1cvjh07xqVLl1i+fDkXLlwAMq9SmTJlimHS6JkzZ1iyZAmzZs0yaueHH35g/fr1nD9/nmHDhnH//n3DpMby5cujUqnYtGkTd+/eJSkpCcgcCqlUqZLRo0KFCkbPc3MpbqVKlTh27Bjbtm3j4sWLfPHFFxw9ejTH/Q8fPszkyZM5duwY165dY926ddy9e5fq1avn+jOnpKRw/Phx2rRp89T4hChKbkaG4/TjKtQKnKtqQ7dJv1o6JIuTwuI5OTs7U79+fWbPnk3Tpk0JCgriiy++YPDgwXz//ffZHtO9e3eWLVtG3759WbduXZbXVSoVW7ZsoWnTpgwaNIgqVarQq1cvoqKi8Pb2BmD27NmULFmSRo0a0alTJ9q2bUvdunVN9jmHDBnC6tWr81y8TJw4kaioKCpWrGiybvIxY8agVqsJDAzE09OTa9euUbp0aQ4cOIBOp6Nt27YEBQUxYsQI3NzcnjgHx8PDg507d5KUlESzZs144YUXWLhwoaH34u2332bRokUsXbqUmjVr0qxZM5YuXUpAQIBRO1OnTmXatGkEBwezb98+Nm7caBhSKlOmDBMmTODjjz/G29vbaLJtfnj33Xd57bXX6NmzJ/Xr1+fevXtGvRePc3V1Ze/evbRv354qVarw+eefM3PmTF555ZVcf+aNGzdSrly5p16xIkRRotNqOTyyFx4JcM8V6s9ZXewuLc2OSnnS4LoJJCQk4ObmRnx8PK6urkavpaWlERkZSUBAQJ4nCuZmopt4doqi0KBBAz744APDGg+S56yioqIICAjg5MmTOU6WzK3C9J0OCQlh5MiR9OnTJ8d9nufn25Q0Gg1btmyhffv22U7aFfmjKOb5t0+6UXN9ODoV3PqoD20GfGHpkEya5yedvx9VsH9biQJDpVKxYMECw3oOQjwUExND9+7d6d27t6VDEcJsrl8+g8udU2it4EwTzwJRVBQUUliIXAsODqZv3+e75e8rr7xiWFTs8cfTrogQBZOXlxdjx47N1cqqQhQFGelppK0eyCvlYonp4spr34ZaOqQCRQaDhFktWrTIsC7G4573PiMFgb+//xMv3RVCFH5HFo/kJe0lHuBM0LCV2Dnkft2j4kAKC2FWublcUgghCqo/vx2Fy6+7OdfInrQ206hTtuLTDypmZChECCGEyIWIMwfx/L9QvONUnLlRljpt3rR0SAWSFBZCCCHEU+i0Ws6OHYJbMtz2gFbfbrB0SAWWDIUIIYQQj9HpFY5ExhGTmIaXiz1R375B7UgdGWqw/s9w3L39LB1igSWFhRBCCPGI0LBoJvwZTnR8GgBN0v5mzN9XATjXujy9XhtmyfAKPBkKMbHmzZszcuTIZzp2586dVKtWzWRLdZvb7t27UalUPHjwIMd9VCpVlnusFAebNm2iTp06RebfWojCKjQsmqErThiKClddHIOObMVWBxf8rXAdsNDCERZ8UljkgwEDBqBSqbI8Ll++zLp165g0aZJhX39/f+bMmZOrdseOHctnn31W4FddzE/R0dGGpaRzY+nSpYY7vhYW2X0HOnbsiEqlYuXKlZYJSgiBTq8w4c9wHr1gfJTVr6Q76XngBLOC3mHS1ovo9HJJ+ZMUnzOWibVr147o6GijR0BAAO7u7ri4uOS5vYMHD3Lp0iVef/11E0RbcPn4+GBnZ2f299XpdBbvLRg4cCDfffedRWMQojg7Ehln6KkAaGt1hIEO+3i54R0WN2/HNeuKRMencSQyzoJRFnxSWOQTOzs7fHx8jB5qtdpoKKR58+ZcvXqVUaNGGXo1crJ69WratGljdE+F8ePHU7t2bRYvXky5cuVwdnZm6NCh6HQ6pk+fjo+PD15eXnz99ddGbc2aNYuaNWvi5OSEn58f7733nuGOmvC/v/q3bdtG9erVcXZ2NhRKD2U3pPPqq68yYMAAw/MVK1ZQr149XFxc8PHxoU+fPsTExOQpj48OhURFRaFSqVi3bh0tWrTA0dGR4OBg/vnnHyBzaGXgwIHEx8cb8jl+/HgAMjIyGDt2LGXKlMHJyYn69euze/fuLJ9506ZNBAYGYmdnx9WrV0lPT2fs2LH4+flhZ2dH5cqV+fnnnw3HhYeH0759e5ydnfH29qZv375Gd21t3rw5w4cPZ/jw4ZQoUQIPDw8+//xzw6JZT/oOdO7c2XAnViGE+cUk/q+oqKiPYJpN5rDHz/qO7LBple1+IqtCUVikaFJyfKTr0nO9b5o2LVf7msq6desoW7YsEydONPRq5GTv3r3Uq1cvy/aIiAi2bt1KaGgoq1atYvHixXTo0IEbN26wZ88epk2bxueff86hQ4cMx1hZWTF37lzCwsJYtmwZO3fuZOzYsUbtpqSkMGPGDJYvX87evXu5du0aY8aMydPny8jIYNKkSZw+fZoNGzYQGRlpVHg8q88++4wxY8Zw6tQpqlSpQu/evdFqtTRq1Ig5c+bg6upqyOfDmAcOHMiBAwdYvXo1//77L6+//jrt2rXj0qVLRp95ypQpLFq0iLNnz+Ll5UW/fv1YvXo1c+fO5dy5c/z44484OzsDmcM0zZo1o3bt2hw7dozQ0FDu3LlDjx49jOJdtmwZ1tbWHD58mLlz5zJ79mwWLVoEPPk7UL58eby8vNi3b99z50wIkXdeLpl/yJXQxfLpwfnsP+PEUU0AM7WvZ7ufyF6huCqk/sr6Ob7WpEwT5rWaZ3jeYk0L0nTZV5P1vOuxpN0Sw/N2v7fjfvr9LPud6X8mzzFu2rTJcAKCzHtirFmzxmgfd3d31Gq14S/6J4mKiqJ06dJZtuv1ehYvXoyLiwuBgYG0aNGCCxcusGXLFqysrKhatSrTpk1j9+7dNGjQAMCopyEgIIBJkyYxdOhQ5s37X940Gg0//vgjFStmriI3fPhwJk6cmKccDBo0yPD/FSpUYO7cuYSEhJCUlGSUm7waM2YMHTp0AGDChAnUqFGDy5cvU61aNdzc3FCpVEb5jIiIYNWqVdy4ccOQwzFjxhAaGsqSJUsM9yTRaDTMmzeP4OBgAC5evMhvv/3Gjh07aNWqleFzPDR//nzq1q1rdE+TxYsX4+fnx8WLF6lSpQoAfn5+zJ49G5VKRdWqVTlz5gyzZ89m8ODBT/0OlClThqioqGfOlRDi2YUEuOPtbM3YQzMoHQvxKdZ8GtAHjU3mqVIF+LjZExJQ+G8/YEqForAoDFq0aMH8+fMNz52cnJ6rvdTU1GxvLe3v7280Z8Pb2xu1Wm00wdPb29toCGLXrl1MnjyZ8PBwEhIS0Gq1pKWlkZycbIjT0dHRUFQA+Pr65nkY4+TJk4wfP55Tp04RFxdnmLNw7do1AgMD89TWo2rVqmUUF2TeUbNatWrZ7n/ixAkURTGc6B9KT0/Hw8PD8NzW1tao7VOnTqFWq2nWrFm27R4/fpxdu3ZlWyRFREQY3q9BgwZGQxwNGzZk5syZ6HQ61Gr1Ez+rg4MDKSmm6zUTQuRMbaViWNTXBF7WorWC/2vcmEs2NYDMogJgXKdA1FZyw70nKRSFxeE+h3N8TW1l/It61+u7cryKwkplvD20W/7dkc7JyYlKlSrlW3ulSpXi/v2svSk2NjZGz1UqVbbbHp7Ur169Svv27Xn33XeZNGkS7u7u7N+/n7feeguNRvPEdh+9mZaVlVWWm2s9enxycjJt2rShTZs2rFixAk9PT65du0bbtm3JyMjI46fP+TM/PGE/aaKlXq9HrVZz/PjxLCfyR4sCBwcHowLAwcHhiXHo9Xo6derEtGnTsrz2sOB5XnFxcXh6euZLW0KIvNkwYxgv7L8LwF+NyrLFqavhNR83e8Z1CqRdUP78rBdlhaKwcLTJ/Z3jHG0cc315Zl7azS+2trbodLqn7lenTh3Cw8Of+/2OHTuGVqtl5syZhrz89ttveW7H09PTaD6ATqcjLCyMFi1aAHD+/HliY2OZOnUqfn5+hvc2tezyWadOHXQ6HTExMTRp0iTXbdWsWRO9Xs+ePXsMQyGPqlu3Lr///jv+/v5YW+f8o/Po/JaHzytXrmwocnL6DqSlpREREUGdOnVyHbMQIn8c3rYCv+U7sQLCajnwwYLtNHhk5c2QAHfpqcilQjF5syjx9/dn79693Lx50+hqgse1bduW/fv3P/f7VaxYEa1Wy3fffceVK1dYvnw5P/74Y57badmyJZs3b2bz5s1cvHiRYcOGGS10Va5cOWxtbQ3v88cffxit32Eq/v7+JCUl8ffffxMbG0tKSgpVqlThjTfeoF+/fqxbt47IyEiOHj3KtGnT2LJlyxPb6t+/P4MGDTJMPt29e7ehEBs2bBhxcXH07t3bcPXG9u3bGTRokFGhcP36dUaPHs2FCxdYtWoV3333HSNGjDB6n+y+A4cOHcLOzo6GDRuaIFNCiJzE3b1F8qSvcUyHq6VVvLLwL9RWKhpW9KBL7TI0rOghRUUeSGFhZhMnTiQqKoqKFSs+scv7zTffJDw8nAsXLjzX+9WuXZtZs2Yxbdo0goKC+OWXX5gyZUqe2xk0aBD9+/dnwIABdOzYkYCAAENvBWT2aCxdupQ1a9YQGBjI1KlTmTFjxnPFnhuNGjXi3XffpWfPnnh6ejJ9+nQAlixZQr9+/fjPf/5D1apV6dy5M4cPHzb0puRk/vz5dO/enffee49q1aoxePBgkpOTAShdujQHDhxAp9PRtm1bgoKCGDFiBG5ubka9ZP369SM1NZWQkBCGDRvG+++/z5AhQwyv5/QdWLVqFW+88QaOjubvSROiuNLrdFxd8haOdRKJLgUBM+fh7CaTM5+HSnl84NzEEhIScHNzIz4+HldXV6PX0tLSiIyMJCAgINuJi0+i1+tJSEjA1dW1yKxUOXbsWOLj4/npp58sHYpBUcxzfmrevDm1a9fO9eqqD929e5dq1apx7NgxAgICgKKX6+f5+TYljUbDli1baN++fZa5RiL/FNQ8/7P4QxpeW0C6YsOVjr9S/cWXLR3SczFlnp90/n5U4f9tVYR99tlnlC9fPldzMkThFhkZybx58wxFhRDC9DbMHEbpC0sBOF17XKEvKgqKQjF5s7hyc3Pj008/tXQYwgxCQkIICQmxdBhCFBsH/1hIuWU7uWtVijPdq9Gx6/uWDqnIkMJCiHz06LLhQoiC6XpEGJopsyiZAZFlrWgxYqmlQypSZChECCFEsZGemsKpYT3xug9xrlB19kKcXNwsHVaRIoWFEEKIYmP90JepFKUn3RoyRg+mYs1Glg6pyJHCQgghRLGwdmI/gg89AOBi12Ba9Bpt2YCKKCkshBBCFHmX//0H571HAPi3nis9Jq22cERFlxQWQgghirT4e3dwXN+fZo2iCWtkT+efdlg6pCJNCgshhBBFVkZ6GtcW9KK0coc4tTdtvgnFwSnnxZ3E85PLTYUQQhRZ699qhrM+lvIV7cjotRw3D29Lh1TkSY+FmahUKjZs2JDj6/7+/nleBtrcvvjiC9555x3Dc0VRGDJkCO7u7qhUKk6dOkXz5s0ZOXLkM7/H0qVLKVGixPMHK0xi06ZN1KlT54m3rReioFjzeU9qHUugwglbdpV8nYAa9S0dUrEghUU+iImJ4Z133qFcuXLY2dnh4+ND27Zt+eeffywdWr65c+cO3377LZ988olhW2hoKEuXLmXTpk1ER0cTFBTEunXrjO5qmteCqWfPnly8eDFPsT1vMWNuu3fvRqVSGd0d9lmOf9Jj6dKl+RrzQx07dkSlUrFy5UqTtC9Eftm5aiZVNvwLwOkGJegy+gcLR1R8yFBIPujWrRsajYZly5ZRoUIF7ty5w99//01cXJylQzOSkZGBra3tMx37888/07BhQ/z9/UlISAAgIiICX19fGjX633Xg7u7Pd1dABwcHHBwcnquNZ/U8+TGnRo0aER0dbXg+YsQIEhISWLJkiWGbm9v/FvzR6XSoVKp8u5HZwIED+e6773jzzTfzpT0h8tul0wewm7UIWy1cDlDTdf7flg6pWCkUPRb6lJScH+npud83LS1X++bFgwcP2L9/P9OmTaNFixaUL1+ekJAQPvnkEzp06JDjcRMnTsTb25tTp05l+3p8fDxDhgzBy8sLV1dXWrZsyenTpw2vR0RE0KVLF7y9vXF2dubFF1/kr7/+MmrD39+fr776igEDBuDm5sbgwYMNQw3btm2jevXqODs7065dO6MTVXZWr15N586dDc8HDhzI+++/z7Vr11CpVPj7+wPGvQfNmzfn6tWrjBo1yvCX9NM8PhQyfvx4ateuzfLly/H398fNzY1evXqRmJgIwIABA9izZw/ffvut4T2ioqIACA8Pp3379jg7O+Pt7U3fvn2JjY01tN28eXOGDx/O6NGjKVWqFK1btwbg7NmzdOjQAVdXV1xcXGjSpAkRERGG45YsWUL16tWxt7enWrVqzJs3z/BaVFQUKpWK1atX06hRI+zt7alRo4Zhqe+oqCjD7eZLliyJSqViwIABT83Lo2xtbfHx8TE8HBwcDD1lPj4+hIaG4uvry6ZNmwgMDMTOzo6rV69m27Pz6quvGr1/RkYGY8eOpUyZMjg5OVG/fv0sy5R37tyZI0eOcOXKlTzFLYQ5JCfGc+k/g3FPhJiSUPv71dg5OFo6rGKlUPRYXKj7Qo6vOTVrSrlHbit+uUlTlNTUbPd1fPFFyi//v//t+3IrdPfvZ9mv+vlzuY7N2dkZZ2dnNmzYQIMGDbCzs3vi/oqiMHLkSDZs2MD+/fupXLlytvt06NABd3d3tmzZgpubGz/99BMvv/wyFy9exN3dnaSkJNq3b89XX32Fvb09y5Yto1OnTly4cIFy5coZ2vrmm2/44osv+PzzzwHYv38/KSkpzJgxg+XLl2NlZcWbb77JmDFj+OWXX7KN+f79+4SFhVGvXj3Dtjlz5lCpUiUWLFjA0aNHUavVWY5bt24dwcHBDBkyhMGDB+cqn9mJiIhgw4YNbNq0ifv379OjRw+mTp3K119/zbfffsvFixcJCgpi4sSJAHh6ehIdHU2zZs0YPHgws2bNIjU1lY8++ogePXqwc+dOQ9vLli1j6NChHDhwAEVRuHnzJk2bNqV58+bs3LkTV1dXDhw4gFarBWDhwoWMGzeO77//njp16nDy5EkGDx6Mk5MT/fv3N7T74YcfMmfOHAIDA5k1axadO3cmMjISPz8/fv/9d7p168aFCxdwdXU19NBMnjyZyZMnPzEXW7dupUmTJk/NWUpKClOmTGHRokV4eHjg5eWVq1wPHDiQqKgoVq9eTenSpVm/fj3t2rXjzJkzhu9q+fLl8fLyYt++fVSoUCFX7QphLpuHvEzNGwqptmD72Rj8KgZZOqRip1AUFgWZtbU1S5cuZfDgwfz444/UrVuXZs2a0atXL2rVqmW0r1arpV+/fhw7dowDBw5QtmzZbNvctWsXZ86cISYmxlCozJgxgw0bNrB27VqGDBlCcHAwwcHBhmO++uor1q9fzx9//MHw4cMN21u2bMmYMWMMz/fv349Go+HHH3+kYsWKAAwfPtxwUs7O1atXURSF0qVLG7a5ubnh4uKCWq3Gx8cn2+Pc3d1Rq9W4uLjkuE9u6PV6li5diouLCwB9+/bl77//5uuvv8bNzQ1bW1scHR2N3mP+/PnUrVvX6ES9ePFi/Pz8uHjxIlWqVAGgUqVKTJ8+3bDPp59+ipubG6tXr8bGxgbAsC/ApEmTmDlzJq+99hoAAQEBhIeH89NPPxkVFsOHD6dbt26GWEJDQ/n5558ZO3asYbjIy8vLqHfm3XffpUePHkafOykpCWdnZ8MwRpkyZXKVM41Gw7x584y+I08TERHBqlWruHHjhuHfesyYMYSGhrJkyRKjXJYpU8bQMyREQXFk/Vy8XGNIt3Hiap8mdO34lqVDKpYKRWFR9cTxnF987C/lSvv25jyW/Nj2Sn//lf1+edStWzc6dOjAvn37+OeffwgNDWX69OksWrTIqJt51KhR2NnZcejQIUqVKpVje8ePHycpKQkPDw+j7ampqYYu+eTkZCZMmMCmTZu4desWWq2W1NRUrl27ZnTMo70MDzk6OhqKCgBfX19iYmJyjCf1vz1A9vb2OSfBhPz9/Q1FBTw9XsjM4a5du3B2ds7yWkREhKFYeDw/p06dokmTJoai4lF3797l+vXrvPXWW0Y9MFqt1mhOA0DDhg0N/29tbU29evU4d+7JPWHu7u5Gc1T0ej0JCQm4urrmeX6Era1tlsL2aU6cOIGiKEaFFEB6enqW76KDgwMpeRw2FMKULhzbSe1TE7D11bJjSHu6vv+jpUMqtgpFYWHlmPvxMStHx1z/Es5Lu09jb29P69atad26NV9++SVvv/0248aNMyosWrduzapVq9i2bRtvvPFGjm3p9Xp8fX2zvQX3w79wP/zwQ7Zt28aMGTOoVKkSDg4OdO/enYyMDKP9nZycsrTx+ElTpVKhKEqO8Twsgu7fv5/lBGMO2cX7tMsd9Xo9nTp1Ytq0aVle8/X1Nfz/4/l50sTRh++5cOFC6tc3vmwtu6Ggxz1tjkl+DoU4ODhkeT8rK6ss/84ajcbw/3q9HrVazfHjx7N8nscLtLi4ODw9PZ8ahxDmcOHUHvS/v42tnZaTjo15+T25AsSSCkVhURgFBgZmWbeic+fOdOrUiT59+qBWq+nVq1e2x9atW5fbt29jbW1tmBT5uH379jFgwAC6du0KQFJSksm6pitWrIirqyvh4eFUqlQpT8fa2tqi0+lMEteT3qNu3br8/vvv+Pv7Y22d+695rVq1WLZsGRqNJktB4+3tTZkyZbhy5coTC0OAQ4cO0bRpUyCzR+P48eOGIaqHV548HnN+DoVk5+Hck4d0Oh1hYWGGyaR16tRBp9MRExPzxOIlLS2NiIgI6tSp88yxCJFfkuLjiBw9FNcEO/a0KMsLo3/BKheFvjCdQnFVSEF27949WrZsyYoVK/j333+JjIxkzZo1TJ8+nS5dumTZv2vXrixfvpyBAweydu3abNts1aoVDRs25NVXX2Xbtm1ERUVx8OBBPv/8c44dOwZkzg1Yt24dp06d4vTp0/Tp08dkixZZWVnRqlUr9u/fn+dj/f392bt3Lzdv3jS6IiM/+fv7c/jwYaKiooiNjUWv1zNs2DDi4uLo3bu34QqG7du3M2jQoCcWOsOHDychIYFevXpx7NgxLl26xPLly7lw4QKQeZXKlClTDJNGz5w5w5IlS5g1a5ZROz/88APr16/n/PnzDBs2jPv37zNo0CAgc/KjSqVi06ZN3L17l6SkJCBzKKRSpUpGjwoVKhg9f55LcVu2bMnmzZvZvHkz58+f57333jNaS6NKlSq88cYb9OvXj3Xr1hEZGcnRo0eZNm0aW7ZsMex36NAh7OzsjIZ7hLCUrYNbUf6Wgp0GHFp/grNrSUuHVOxJYfGcnJ2dqV+/PrNnz6Zp06YEBQXxxRdfMHjwYL7//vtsj+nevTvLli2jb9++rFu3LsvrKpWKLVu20LRpUwYNGkSVKlXo1asXUVFReHtnLkc7e/ZsSpYsSaNGjejUqRNt27albt26JvucQ4YMYfXq1XkuXiZOnEhUVBQVK1Y0Wdf5mDFjUKvVBAYG4unpybVr1yhdujQHDhxAp9PRtm1bgoKCGDFiBG5ubk8cKvPw8GDnzp0kJSXRrFkzXnjhBRYuXGjovXj77bdZtGgRS5cupWbNmjRr1oylS5cSEBBg1M7UqVOZNm0awcHB7Nu3j40bNxqGlMqUKcOECRP4+OOP8fb2Nppsa0qDBg2if//+9OvXj2bNmhEQEGDorXhoyZIl9OvXj//85z9UrVqVzp07c/jwYfz8/Az7rFq1ijfeeAPHfBxKFOJZ/PqfjgT9m4oeuNGvJSFtZG2VgkClPGlw3QQSEhJwc3MjPj4eV1fjG8GkpaURGRlJQEBAnicKPs9EN/F0iqLQoEEDPvjgA8MaD5LnrKKioggICODkyZPUrl37udoqiN/pu3fvUq1aNY4dO5almHqa5/n5NiWNRsOWLVto3759tpN2Rf7I7zxvWzSO0rN+w1oPp5p70/vH3c8fZBFgyu/zk87fjyoYv61EgadSqViwYIFhPQdRPEVGRjJv3rw8FxVC5Kezh7fjNi+zqDhf2YYe3+fPFX4if0hhIXItODiYvn37Plcbr7zyimFRsccfT7siQlheSEgIPXv2tHQYohhLS03m4uRRuKVAdClosmAj6jxM0BamJ/8awqwWLVpkWBfjcc97n5GCwN/f/4mX7gohnp2i13Pmx0G8XPMWu/GizAdfUcpXes8KGikshFk9z+WSQoji7fCvU2kQH4pOrSLgP9Op2TTrlXfC8gpkYSF/8QlR9MjPtXgem+d9hPrIGvTl4WiV0TSQoqLAytMci/HjxxvuIPnw8Tz3gHjcw9X+Hl89UghR+D1cAlyuvBB5dWrfRjwX/kH5Q3asiQmkfu/PLR2SeII891jUqFHD6PbcuVnKONfBWFvj6OjI3bt3sbGxydMldnq9noyMDNLS0grMpXlFkeTZfIpKrhVFISUlhZiYGEqUKJGvvzNE0Rd35zqxn31MmVS44QVtJq5FVYh/HoqDPBcW1tbW+dpL8SiVSoWvry+RkZFcvXo1T8cqikJqamq290gQ+UfybD5FLdclSpQw2e8OUTTptFp2v9OR6jGQ4AjeU6ZT0lPmaRV0eS4sLl26ROnSpbGzs6N+/fpMnjyZChUq5Lh/eno66enphucJCQlA5iIej94A6SGVSoW/vz8ajSZPY7JarZaDBw/SqFGjPN0bQuSN5Nl8ikquVSoV1tbWqNXqArkOysPfQ9n9PhL551nyvHZkO2qfz0Cngti3u9AmpJ38Oz2FKb/PuW0zTytvbt26lZSUFKpUqcKdO3f46quvOH/+PGfPns3xrpfjx49nwoQJWbavXLlSlgQWQgiRrbtHfqXhupNYKXComTfu7UdZOqRiLyUlhT59+jx15c3nWtI7OTmZihUrMnbsWEaPHp3tPtn1WPj5+REbG/vEwPJKo9GwY8cOWrduLZPDTEjybD6Sa/OQPJtHXvJ8K+oc4dNew++ADecD7ei44h9ZBCuXTPl9TkhIoFSpUk8tLJ7rX8rJyYmaNWty6dKlHPexs7PDzs4uy3YbGxuT/BCbql1hTPJsPpJr85A8m8fT8pyc+AD9qr68UvYu2zsE0OqTrdg/x119iytTfJ9z295zTa1NT0/n3Llz+Pr6Pk8zQgghBDqtltPf9yZAH0UsJag9aj0lPOT8UtjkqbAYM2YMe/bsITIyksOHD9O9e3cSEhLo37+/qeITQghRTPz23sto/ojkQrIDse0X4VVGlusujPI0FHLjxg169+5NbGwsnp6eNGjQgEOHDlG+fHlTxSeEEKIY2DBjKLX2xmCFilPUp2dIa0uHJJ5RngqL1atXmyoOIYQQxdThrcvwW74bKyAs2IGe32y0dEjiOcjyZUIIISwm+up5UiZNxTEdrpZW8cqCv55+kCjQpLAQQghhEZqMdI681x2fOLjvDAEz5+Hs5m7psMRzksJCCCGERfz+QRuqROjIUEPKiP5UrdPc0iGJfCCFhRBCCLM7uX0FLUuGcb2MwoXOgbTq+7GlQxL5RJYyE0IIYVZXzx2nyoH/4GSnxblrY9oM/9nSIYl8JD0WQgghzObmlTDOfvcmTqo0ztoG8+KQeZYOSeQzKSyEEEKYlE6vcCQyDq0mg39HvEnATthwxRfft1dhY5v1lg+icJOhECGEECYTGhbNhD/DiUtK5Yuo6QRG6Um3hgcN3sTdq4ylwxMmID0WQgghTCI0LJqhK04QHZ/GG7ELeOF4EgB/NqzIhGt1CA2LtnCEwhSksBBCCJHvdHqFCX+GowCtUzbR+WDmXbAPBTvxs8dQACb8GY5Or1gwSmEKUlgIIYTId0ci44iOT+PF9IMM3rcbOw1cLm/FNxU+BUABouPTOBIZZ9E4Rf6TwkIIIUS+i0lMw5s4+iauxSUVrntDypuj0VnZZtlPFC1SWAghhMh3riSzzHYancveJqIFTKs/FHvXUln283Kxt0B0wpTkqhAhhBD56v7dm7hu6Es1q+vcVkoytcRH3LUqBegM+6gAHzd7QgLk3iBFjRQWQggh8k1qcgL7+rfFOVFLxEsuvMdH3FQ8seV/kzRV//3vuE6BqK1U2TckCi0ZChFCCJEvdFotm/s1pfIVHR73VVyqOIhRb7yKj5vxcIePmz3z36xLuyBfC0UqTEl6LIQQQuSLNW81IfhsOnoVRLzxEq+9PRGA1oE+HLocQ+y5Qyzu/yINKnlJT0URJj0WQgghntvq99sQfPgBAGEdq/DaJwsNr6mtVIa5FCEB7lJUFHFSWAghhHgua77oTfCO6wCcauZFz282WjgiYUlSWAghhHhmJ3asxHf7KQDO1HGixw9/WzYgYXEyx0IIIcQzuXhiN9X2jySuhZ6D18vRdckB1NZyWinupMdCCCFEnkWEHcLzjzdxVKVz360mry4+KLdAF4AUFkIIIfIo/OgObgweyNkYFZesKxMwbB22drKCpsgkhYUQQohcu3bpNLdHfYDXfdCccsZ1wK84u5a0dFiiAJHCQgghRK7cv3uT8Hd74xsLD5ygxJTJeJetaOmwRAEjhYUQQoinSk1OYG//tpS/qZBiB5pPhlGnWVdLhyUKICkshBBCPJFOq2Vz/2ZUuaIjwxpihr1G0+7DLR2WKKCksBBCCPFEaz/uQI2wtP8u1d2YV4Z8bemQRAEmFxwLIYTI0aGVX9Hd6RBba3mSXL4GPT9ZZOmQRAEnPRZCCCGydWzTAhpc/Aa1Ckq17yFLdYtckR4LIYQQWfz57UjYuYWMWnDSpzv1+31l6ZBEISGFhRBCCCN/LZ9K2UXbsNeoWV+qCq+PX4DKSjq4Re7IN0UIIYTBoa3LcJ21DHsNXClnRYdZ27BSqy0dlihEpLAQQggBwLmjf6EdNxWXVLjhreKFRRtwdnO3dFiikJHCQgghBNcjwoge9T4eCRBTEgK+X4hPucqWDksUQlJYCCFEMZeSlMC/Q3sYlup2nfo1lWo2tnRYopCSwkIIIYoxrSaDC/N64lv9AXGuD5fqfs3SYYlCTK4KEUKIYkrR6zkxbwAhKQdJL2HDhRmTqdVU7v8hno/0WAghRDH169DmlLzxNzpFRXjjOVJUiHwhhYUQQhRDqz9oS/Ceu8TvcGdvhQ+o0+ZNS4ckiggpLIQQophZM643wduvAXApxJMW/SdaOCJRlEhhIYQQxcgfc0ZSbc0pAM7UdqLHvJ2WDUgUOVJYCCFEMfHX8qn4/bwNaz2cr2pD1/87gNpa5vCL/CWFhRBCFANHtq8wWqr75aV/YWNrZ+mwRBEkpaoQQhRxd29FUXLPeK54qInXqHlh0QZcS3pZOixRRElhIYQQRVj8/ViSFnWhss1dVI1Lo+qxRJbqFiYlQyFCCFFE3b97k52ftSZAH0UsJXAcsJ6KNepbOixRxElhIYQQRVBqcgJ7B7Sl2s40/rzkRXy3XykdUM3SYYliQAoLIYQoYnRaLZv7N6NKhI4MNVi91J2KNRtYOixRTEhhIYQQRcyawU2pEZaGXgWX+zSkw9Aplg5JFCMyeVMIIQoRnV7hSGQcMYlpeLnYExLgjtpKZXh91Yh21P7nPgBhHSrT87PFlgpVFFNSWAghRCERGhbNhD/DiY5PM2zzdbNnXKdA2gX5smZcH2ptuwrA6Sae9Jrxh6VCFcWYFBZCCFEIhIZFM3TFCZTHtt+OT2PoihN888I9XK8fAuw4U9uR1+fLUt3CMqSwEEKIAk6nV5jwZ3iWogJAAWqqrvBK2CScyqWz0bM2XcZvlaW6hcXIN08IIQq4I5FxRsMfj6qbcYSZDotxUqVzyjqYdhO2YmfvaOYIhfgfKSyEEKKAi0nMvqiorAln9P7fiLJ14kbjEsS8+hO1pagQFiaXmwohRAHn5WKfZZu39iYfH16MRwKotSq+0L5NaS+5/4ewPCkshBCigAsJcMfXzZ6HF5W66B8w7uQcSsdCvBPMaNSb9FJBhAS4WzROIUAKCyGEKPDUVirGdQoEwEafysSwKQTcVEixhXkvteOM7QuM6xRotJ6FEJYicyyEEKIQaBfky5yu/ugm9qTalcylupc1acAln47M/+86FkIUBFJYCCFEIZCWkoT71qGoYnVo1HDstfr0HjCD2Y+tvCmEpUlhIYQQBVxy4gOivutME05zqbkT58v04q0PZlk6LCGyJYWFEEIUYDeuhHF6bl86lLhCsmKP5tWf6dTwFUuHJUSOZPKmEEIUUFfCj3BuwOuU35bGrlgPbnb5lUApKkQBJ4WFEEIUQBdO7ubq4P6UjYFkB1A1HUGVus0tHZYQTyVDIUIIUcD8e3Az9/8zBp/78MAZGD+G5h3fsnRYQuSKFBZCCFGAHP97DamffIlXAtxzBfsp46j3ci9LhyVErklhIYQQBcTJvRvJ+OhLPJIgpiSUnDGdWo07WTosIfJE5lgIIUQBcCXsMH5/D+deOS23PcDn+/lSVIhC6bkKiylTpqBSqRg5cmQ+hSOEEMXPxRN78FjblVKqBCrXdqLCkpVUfaG5pcMS4pk881DI0aNHWbBgAbVq1crPeIQQoljZuuAztJvX4h+czAXbavgM24xbyVKWDkuIZ/ZMPRZJSUm88cYbLFy4kJIlS+Z3TEIIUSz8OXc0vnPXUemCFRuv+lPmg1ApKkSh90w9FsOGDaNDhw60atWKr7766on7pqenk56ebniekJAAgEajQaPRPMvbZ+thW/nZpshK8mw+kmvzsFSe/5z5HhWX78dWB5f9rWg+9Q/sHJyL7L+3fJ/Nw5R5zm2bKkVRlLw0vHr1ar7++muOHj2Kvb09zZs3p3bt2syZMyfb/cePH8+ECROybF+5ciWOjo55eWshhCgS7u5ZRP3Qy1jr4UJFNdq+H2Pj4GLpsIR4opSUFPr06UN8fDyurq457penwuL69evUq1eP7du3ExwcDPDUwiK7Hgs/Pz9iY2OfGFheaTQaduzYQevWrbGxscm3doUxybP5SK7Nw9x5XvdlL4I2hGOlwLlqNry8ZCeOzm4mf19Lk++zeZgyzwkJCZQqVeqphUWehkKOHz9OTEwML7zwgmGbTqdj7969fP/996Snp6NWq42OsbOzw87OLktbNjY2JvlymapdYUzybD6Sa/MwR553Lv6Mypszi4qzQfZ0Wn4AO4fi1XMr32fzMEWec9tengqLl19+mTNnzhhtGzhwINWqVeOjjz7KUlQIIYTIdOiXCbS89j1Hmzvx770yvLZkPza2Wf/oEqKwy1Nh4eLiQlBQkNE2JycnPDw8smwXQggBOq2WnT++T+vYlQBoa3Sjx+BvUVnJ+oSiaJIlvYUQwkR0Wi1r3mpCwJkHXG5lx93qA2nQf4oUFaJIe+7CYvfu3fkQhhBCFC2ajHTW929M8MlkAI5l1KXXwGkWjkoI05OyWQgh8ll6agob+zSg5n+LitPtK9BrTqiFoxLCPKSwEEKIfJSSFM/m3g2pEZaGXgVnXqtBr1mbLR2WEGYjhYUQQuSThPsxbO/1EtXPZ6C1gnO969Fj8lpLhyWEWUlhIYQQ+SAtJYnzP/TAMVmDRg2X+zel+5fLLR2WEGYnV4UIIcRzSk58QNR3nQnRnuZOQ3uO+Q2g61CZqCmKJykshBDiOVyPCOPY9Dfo6htFkuLAvc5L6dCgnaXDEsJiZChECCGe0ZXwI1wY+DrV9mSw+Zont7qsJlCKClHMSWEhhBDP4MLJ3Vwd3J8yMZDgCM6vjKRK3eaWDksIi5OhECGEyKN/D27m/n/G4HMfHjiDMv5DmnUcZOmwhCgQpMdCCFFs6fQKRyLjADgSGYdOrzz1mON//0bCyDF43Yd7rmAzbRyNpKgQwkB6LIQQxVJoWDQT/gwnLimV6SEwaNlR3J0dGNcpkHZBvtkec+7oTtI/GodHEsSUhJIzZ1CrUQczRy5EwSY9FkKIYic0LJqhK04QHZ9mtP12fBpDV5wgNCw6yzERZw7hvbk/iUFp3PYAn3nzpagQIhtSWAghihWdXmHCn+FkN+jxcNuEP8ONhkXOH9tJqd9fw50EKlYsSbWVG6hap7kZohWi8JHCQghRrByJjMvSU/EoBYiOTzPMvdi64DOiPn4PRZvKeevqeA7fjm/5qmaKVojCR+ZYCCGKlZjEnIuKx/f789uR+C3chp1WxV+Xy/LKwlCcXEqYNkAhCjkpLIQQxYqXi32u9nuw4QvqrPkHGx1c9rfi5e+kqBAiN2QoRAhRrIQEuOPrZo8qh9dVwJD4hdT9LbOouFDJmmardlHSs4w5wxSi0JLCQghRrKitVIzrFAiQpbhQAUNiv6PL7gtY6+FcNVva/noA15JeZo9TiMJKCgshRLHTLsiX+W/WxcfNeFhkoM0Wmp2+ipUCZ4Ps6bDqHxycXC0UpRCFk8yxEEIUS+2CfGkd6MOhyzHEnjvE7HIHaR+9gsst7DgeE8BrC/ZiY2tn6TCFKHSkx0IIUWyprVS84OdKyvFltI/+AYC7lXrSY/E/UlQI8Yykx0IIUWzptFo2vNOCwOPxHH3ZGW2dN2nQfwoqK/mbS4hnJT89Qohi6f7dm2zs/gLBR+Kx1cFlpQYNB06TokKI5yQ/QUKIYufsoVCOdW9N9fMZ6FTwT8vSdJ+12dJhCVEkyFCIEKJY2b5kIq7fr6JsMiQ6wJ23OuLh95KlwxKiyJAeCyFEsbH5+5H4zFiFWzJElwKbWV/R7p3Jlg5LiCJFeiyEEEWeVpPBsYXDeOXub2yp6I1Wb0ejnzbgVaYiGo3G0uEJUaRIYSGEKNKuXTjB7d9G0ED3L6jAtcMrNBw4XS4nFcJEpLAQQhRZ+zcuQD9lNknuehLr2XGp0XSathtg6bCEKNKksBBCFEnrpw6h/Mp9OGSAld6KC82+p97LPSwdlhBFnhQWQogiRafV8tt7Lam19y5WQGRZK6rNXUKFwBBLhyZEsSCFhRCiyIiNjmTfO12ofTFzQuaZ2o50WLgTJxc3C0cmRPEhl5sKIYqE29cucWhgB6pd1KC1gtMdK9Fj9XEpKoQwMykshBCFXvihUGwXt6R84H3iXODGyNfoNeNPS4clRLEkQyFCiEItdP4YXr69GBuVjvslK+D6y1LKVwm2dFhCFFtSWAghCqXkxHi2vN2SymdTuNzOhhSfJlR/9/9wdJahDyEsSYZChBCFTsSZg+zp2pCg0ynYaOF0Ri3qjl4vRYUQBYAUFkKIQmXP2u+IfustAm4opNrCxYFN6TV3h9zuXIgCQoZChBCFxu+T+lPxtyPYaeBuCVB/OoqunYdYOiwhxCOksBBCFHh6nY5147tRY80FAK6Us6Lmd8spV7WuhSMTQjxOCgshRIGWnPiAiz++wauq/ewo70NiKVc6LfgbBydXS4cmhMiGFBZCiALr5J4NOO78nDqq62SorHF7ewivvP4fS4clhHgCKSyEEAXSph8+xHvhJi5V0FKyZgniOv5MoxdbWTosIcRTSGEhhChQdFota8Z0JGjbVdQKPLhnQ3KvlVSr/qKlQxNC5IIUFkKIAiMpPo7Qt14mOCwNgHPVbGmxcAslPctYODIhRG7Jhd9CiALhwqm97O/6EjXC0tADp1v60mXtcSkqhChkpLAQQljchRO7uTn8HcrfUkixg8uDW9Jr3k7U1tKpKkRhI4WFEMKijm1eSLmNr+NcN5HbHqCZ+jFd/vODpcMSQjwj+XNACGERmox0ts8eRIfUTaAC+9LBVPhsGaW8ZehDiMJMCgshhNlFXz3P0Xe7UzpaR1RbO24HdOfFt+fK0IcQRYAMhQghzOrI9hWc792VypE6bLRw3PkVGrw7T4oKIYoI+UkWQpjNH3NGUHrJdnzS4b4zpIwcQLc3P7J0WEKIfCSFhRDC5HRaLWtGvkLNv29gpcA1XxXlZ3xPoxdaWjo0IUQ+k8JCCGFSaanJbBzThuC/4wA4W8OONou241rSy8KRCSFMQeZYCCFMJvbWVa7ObMGr7mFcL61wurUfXX89JkWFEEWY9FgIIUxiz5q5VD4zm6pWccSrnfD+4mvatOhm6bCEECYmhYUQIt/99tnrVN0QxslaVmRU98O6z2qCKwVZOiwhhBlIYSGEyDfpqSlsGNyCWscSANDFO+A27C9KevhYODIhhLnIHAshRL6IOneM7V1fNBQVpxuW5JUNx6WoEKKYkR4LIcRzWzdlMKXX7KdSCqRbw+XuL9Br/ApLhyWEsAApLIQQzyw+7i4nF7yF/6pLOGTAbQ+I7PMG1dsNRadXUFupLB2iEMLMpLAQQjyT07vW4LtnLM2J4+/6JTmd5MlMv5EkXS8BCw/h62bPuE6BtAvytXSoQggzksJCCJEnt6LOcXBMH8qVjsHLPYmrlOaHUkM44VHFaL/b8WkMXXGC+W/WleJCiGJEJm8KIXLtz7mjiXz9NWqEpaH5x5X9Ht3oZzODE0qVLPsq//3vhD/D0emVLK8LIYomKSyEEE8VczOCNT3rUmneVtwTIdYNEga/hrr9N1xNzPk4BYiOT+NIZJzZYhVCWJYMhQghnmjzvI9xW7qRoMyrSDlT25GWs9dSyjeAjadu5qqNmMQ0E0YohChIpLAQQmQrNTmRHTPfpPLKiwDcc4UH/TrQY/gMwz5eLva5aiu3+wkhCj8pLIQQWZw/sgOnrR/QWbnF5srepNo703jGSnzLVzPaLyTAHV83e27Hp5HdLAoV4ONmT0iAu1niFkJYnsyxEEIYPLgXzeoBDSm5vhd+yi1icKf0iAl0X3MiS1EBoLZSMa5TIJBZRDzq4fNxnQJlPQshihEpLIQQAPz1f5M51aklwYcecOzfUhwt8Qp2I45Sp1XvJx7XLsiX+W/WxcfNeLjDx81eLjUVohjK01DI/PnzmT9/PlFRUQDUqFGDL7/8kldeecUUsQkhzCDhfgxbRnWh5uEHWCkQ7wQZTZvz4sifct1GuyBfWgf6cCQyjpjENLxcMoc/pKdCiOInT4VF2bJlmTp1KpUqVQJg2bJldOnShZMnT1KjRg2TBCiEMJ1dq2ehfL+Q4NjM5+eq2VJ3ys80qF4vz22prVQ0rOiRzxEKIQqbPBUWnTp1Mnr+9ddfM3/+fA4dOiSFhRCFiCYjnQ3jX6fahktY6yHBEW50DaHbF8ssHZoQopB75qtCdDoda9asITk5mYYNG+a4X3p6Ounp6YbnCQmZF8NrNBo0Gs2zvn0WD9vKzzZFVpJn8zFVrqPCj6L6YxhtraI44+zDXS9rqn41j841GhTLf1f5TpuH5Nk8TJnn3LapUhQlT2vtnjlzhoYNG5KWloazszMrV66kffv2Oe4/fvx4JkyYkGX7ypUrcXR0zMtbCyGegyYjlaQ9C+jlfgp7Kx0PFGf+dOqKc8WWWFmrLR2eEKKAS0lJoU+fPsTHx+Pq6prjfnkuLDIyMrh27RoPHjzg999/Z9GiRezZs4fAwMBs98+ux8LPz4/Y2NgnBpZXGo2GHTt20Lp1a2xsbPKtXWFM8mw++ZnrQ5uXkjp7Nn53FG6+lIJH5er49pmHh49fPkVbeMl32jwkz+ZhyjwnJCRQqlSppxYWeR4KsbW1NUzerFevHkePHuXbb7/lp5+yn0FuZ2eHnZ1dlu02NjYm+XKZql1hTPJsPs+T6/TUFNb/pxPV99yilA5S7CDOtwEvf7gKlZVcbf4o+U6bh+TZPEyR59y299wrbyqKYtQjIYQoGI5sX0Hc1MkE38rslLzsb0WFCbPoUb+thSMTQhRleSosPv30U1555RX8/PxITExk9erV7N69m9DQUFPFJ4TII71Ox5rPXqPqpouU10KqLVxqV5Xuk9eitpZV/IUQppWn3zJ37tyhb9++REdH4+bmRq1atQgNDaV169amik8IkQc3r5zjwerBVM+IQK0rwZVyVpT5YjI9m3SxdGhCiGIiT4XFzz//bKo4hBDPQavJYOsPI3n5/lrKqNJJcbFj+6DGdBw5D2sbW0uHJ4QoRqRfVIgCSKdXOBIZB8CRyDgaVPLKcXnsfw/8yfUJH1Pupp5bHSC9RE1K9FrIqxWqmzNkIYQApLAQosAJDYtmwp/hxCWlMj0EBi07iruzA+M6BRrd0Eun1bL20+5UDr1AhQxIt4bj1o15/aO1WKllXQohhGXI9WZCFCChYdEMXXGC6Pg0o+2349MYuuIEoWHRAJw9vJ0tHYOp9ccFHDLgmq+K9Bmf0HPyeikqhBAWJT0WQhQQOr3ChD/DyW7FOgVQAeM3hpGwegQV/jxDpXTIUMO5ZqXpOvNP7BxkJVshhOVJYSFEAXEkMi5LT8WjSvGAiWk/o7sWhVO6Hde9VTj/ZxS9Og82Y5RCCPFkUlgIUUDEJGZfVCh6HV30OxnvsJaSqiSSy6v50/1Funy9Bgen/FsWXwgh8oMUFkIUEF4u9lm2Jd29wpQzP+NxX4tjyyTOqspzrdlMerWStWOEEAWTFBZCFBAhAe74utlzOz4NBeh172cqbT2PawroVCrm3a/PWt/R7G3ZytKhCiFEjqSwEKKAUFupGNcpkFULp9Ht/A4qX9UDEO0By19ozW7HtszvEpzjehZCCFEQSGEhRAERee44KZ8P4uPzGQBoreB0HWe+8RuNo5sX8x9bx0IIIQoiKSyEsLDY29eIWDuOunc3cj7NE7DiXFUbdL2G4+LozbzqDZ648qYQQhQkUlgIYSE3I8PZN/FtGvhEUt8uBVRg84Intwe8xmu9RqPRaNiyZQshAe5SVAghCg0pLIQws/t3b7JtXD8qHrxFcBqEBTmTXrsc2hZf0qpxB0uHJ4QQz0UKCyHMJDU5gT/G98Vv10WCkzK33fYATb3mVBn7EyorWWFfCFH4SWEhhIkpej3rpryFx+ZD1Mq8YSlxrnDj5UBe/XK5LMUthChSpLAQwoTC9m3Ebs8k7M48wDvOhkQHuPKSH+3GL6Oxh1zhIYQoeqSwEMIE/v5lOvqwdbS2OweAQ0UH9jtXotmXiwgpX83C0QkhhOlIYSFEPjq0dRnR82ZS7ZKGq3560hqqOeX9GpW7j6eXd1lLhyeEECYnhYUQ+eDs4e2cm/ER1cLSqKaAHkh2tie693YaVK9n6fCEEMJspLAQ4jlcu3CCf74aSrWTCdTQZm67VEFNybeH0u21YZYNTgghLEAKCyGeQUpSPKd/n0b8/l+pddQWgKtlVNC7O53fnmjh6IQQwnKksBAiD5IT49n7f1/y4t0NNOQBGb4QWrEM+uZN6TTqe9TW8iMlhCje5LegELmgyUhnw9cD8d52EhcFXNs84Ja1N7fqjabDl29LQSGEEP8lvw2FeAKdVsvmuaNw3PAXQTGZ2xIcYbt7H9q8N5vSdvaWDVAIIQoYKSyEyMH2pZPQrlhF5RsKAKm2cLG+Jy3HL6F+mYoWjk4IIQomKSyEeMzVc8c5v2I05X7P7KLIUMP52i68+Nn39AoMsXB0QghRsElhIcR/RZ47Tuy2b6h7P5TyNgpb/H1IcXKgyuhJ9GzcydLhCSFEoSCFhSj2roQf4ejXwwgIT8K/XQxqO4WTTi9RdcY4KgZJD4UQQuSFFBai2Lpz4zK7xg+kypFYamVkbtt7pyw135tNnXotLRucEEIUUlJYiGIn4X4Mm8f3pcK+awSnZG676QUpr7Wh6/CZcumoEEI8B/kNKooNnVbLkY3fkz7zJ2rHZW67WwJi2tWly6eLsbG1s2h8QghRFEhhIYo8nVbLmd1rcDs4hYb6q2wq64ldug3Xmlek4/hfcHJxs3SIQghRZEhhIYqsiDMHOfLTl/icvIlX/QcEuKaQgCOOL3ei6nejaejtZ+kQhRCiyJHCQhQpmox0ts4bi+7vXVSK0FBbn7k9MtyVhK6vEdj9S1p6eFs2SCGEKMKksBBFwtWLp/ln+nDK/htL5YT/bb/hrSKuXgVeGj6dMgGBlgtQCCGKCSksRKGVlppC+J7fsD61gsCUY5Q+64NHgopkO7hS3Qmvrm/SuudIS4cphBDFihQWotA5uHkJ11f9iE9EAi+2isZRrYAV3HvBkTslKtNqxCzqeZYBQKdXOBIZR0xiGl4u9oQEuKO2Uln4EwghRNElhYUoFGKjI/l77hjcjpyj/E2Fkv/dvu+uJ651X8Gv5RC6VgoyOiY0LJoJf4YTHZ9m2ObrZs+4ToG0C/I1Y/RCCFF8SGEhCixFr+fQlqXc+L/vqXA+1bA6pk4FVwLU6Jo2ot37M3Bwcs1ybGhYNENXnEB5bPvt+DSGrjjB/DfrSnEhhBAmIIWFKHDu343mwl9L8Lz8Gz6JtyjxrweQuZjVrWBvar39GZ1fbJ3j8Tq9woQ/w7MUFQAKoAIm/BlO60AfGRYRQoh8JoWFKBA0GelsW/g5GTu2Y6tPo2Pd2wCkO9iwoYETTi+2oO3gr3K1OuaRyDij4Y/HKUB0fBpHIuNoWNEjvz6CEEIIpLAQFnbu6F+cXvQVpU/foeKDzG1aKytO1SpHWo3eVG/zFj3zuO5ETGLORcWz7CeEECL3pLAQZqfVZLB13liUbX9RIVJH8H/HLFJtIaKaA+5detG85+hnvhmYl4t9vu4nhBAi96SwEGZz7eK/3Ny9iEq3/sDuqgq/K5kn9qtlVMS/WI2XR8ykrm/Ac79PSIA7vm723I5Py3aehQrwccu89FQIIUT+ksJCmNT9u7f469tROB0Kw75CEi1LPwCgUmk3jtZzwa/Xu7TrOChf31NtpWJcp0CGrjiBCoyKi4dTNcd1CpSJm0IIYQJSWAiT2L3mO2J+X0aFc8kEpWduu6534N8KldHWfpOgFr2oYGe6oYh2Qb7Mf7NulnUsfGQdCyGEMCkpLES+SYyPY8uk/rgfjaDsHYWHUy7jXOF6rVIEDhxLrcadzBZPuyBfWgf6yMqbQghhRlJYiOei6PVcOPoXif8spsb9nbj9W5Kyd6zQWsHlijZYtWxOu3en0tjB0SLxqa1UckmpEEKYkRQW4plEhB3k1KKJeJ2+RfUmd6lmpwEVqKpbc6qKLyHvTKRrzUaWDlMIIYSZSWEhcu3s4W2E/fEztqfD0VzREazP3H78Rgm8XngR10aDaFPvZVRWVpYNVAghhMVIYSFydPv6Za6f2E7s6S04bYnA8wHUeuT16z4q7terRNP3p+NbvpqlwhRCCFGASGEhDE7t28jFLf+H1cUIrJ3TeTXgFj5Aik7F5URf9Cq45aUipowjZbr2pc3rIywdshBCiAJGCotiSqfV8u++jVzc9gvWF6/gfSMdjwSo+d/Xoz0VtP5WXLGpxD3fEO4NcyK4XX8eaEvhfO4QNtUboNMrcoWFEEIII1JYFBM6rZbzx/4i+eZ5VNcOUObBCVK32FIr8X/7aK3glreK++XccKj9ImlvTaCKa0kg8zbkPX4NJy4pkukhMGjZUdydHWRNCCGEEEaksCiidFotx/5eTdSO37C5fBWfGxnYaOGFLrdQqwAVHC/rTUK8NQ/KueFYO4QGr39AzTIVs7QVGhbN0BUnUAA79f+2345PY+iKE8x/s64UF0IIIQApLIoMRa/n2qV/ObJqKtYnw/G5oaFEsvFkS60VHNBUwabCSzhXaU6L0U1x/m+PRE50eoUJf4Zne88Nhcwlsif8GU7rQB8ZFhFCCFE0CgudXuFIZBwARyLjaFDJq8Cf5B7G/KwrQuq0Wo5sXcrVXesp66lQLf0s5XlA2MVSVLhgC4BGDTd9VMSXL4lz3YY0en0ENb398hTnkcg4oyWxH6cA0fFpHImMk4WohBBCFP7CIjQsmgl/hhOXlFpoxv4fxvzoCdv3Kfew0Go0HNqymBu7NmAXcYPSN7WUSIESwINWiZQqlUi6YoN1WXdOO9jh8kJjGr3+AbU8yzxXrDGJORcVz7KfEEKIoq1QFxaFcez/0Zgf9XjMep2OqHPHiDnzN7ePbab0zjg8UuHRPoF0a7jpa0WqewjhbQdToXZT2jo45Wu8Xi65u1FYbvcTQghRtBXawqIwjv0/KWaVkkGTtD1cnvkNG+7ew750Cu1Kx1IBiFDbkZHqQbpNZiGR4O9ByReb0aj7+9Qu6WXSmEMC3PF1s+d2fFr2cZN5x9CQAHeTxiGEEKJwKLSFRWEc+8+MORVXUvDmLsEJBymXfB2/mDj8onU4PfJxrqAmxdeOyw5BJPmHkFjNgcbd3qe2m3lP4GorFeM6BTJ0xQkeL88ePh/XKbDAFG9CCCEsq9AWFgVx7D8jPY1LJ3dz68IJEm9cIj0mGqv7D7BJTMU+SUt6CT3hgfdwVKWjVyBsT2lsdP87PtUWbvhacd+vFKWbv4JNj1HUsrUzW/w5aRfky/w36xrmsjzk85R5IUIIIYqfQltYmHvs//a1S1w5s5+4K2GkRF9Ffy8Wq/hEcIDaFa0ooY2lpP4+2t99KavLvo2bGgVHVToACThxrSxordTc8HTjdIla7LZ/GY2VA6sGNygwvSwPtQvypXWgD4cuxxB77hCL+79YKK6+EUIIYV6FtrDIr7H/jLRUroQf4ua5o8Rfu0hGzC1UVloqlC2JfdodnDPuErNRj2sKlCTz8aibPgpVykdnPrGCB64K1loVSc4qUp3UZLjYoS/hitrDE2f/qnSNrsS5ZGfSsIM6WWP2LcDzFdRWKkIC3NlyjjxfHiuEEKJ4KLSFRW7G/j9q6s6Jv1YTE3EabXI83l6lIDEa25Q73N4egdsDBddksNZDaTIfADe9FV50OWxo76ZN5iuptpDgDMnOVqQ526BxdUDl483JRp/j7FmOkr7+NPrUBxtb2xzjVv33qhAVGBVEMl9BCCFEUVBoCwv439j/+I1h9E5dSdqKk3wTr8E5WY9rMjhtyNzPmcxioUGLaMOx+xJ9cU/MPIHrgQQnSHJWkeJsRVopJ/4JeA1rtzI4eJTFvmk6JSvVoXq5yvkW8+PrWMh8BSGEEEVBoS4s4H9j//cnvsW5646UemD81366DcQ7Q5KrNcdcW6Fx8kHlWpr7JW6TWsKb0tVeoGLNxjg4uZo95udZeVMIIYQoiAp9YQGZwyIXy/Umou5hbjmXwN63HB4ValA+qDFVyldDbV3wPqbaSlXgJmgKIYQQz6vgnXGf0Yt9v+Kuxxbat2+PjY2NpcMRQgghiiWrvOw8ZcoUXnzxRVxcXPDy8uLVV1/lwoULpopNCCGEEIVMngqLPXv2MGzYMA4dOsSOHTvQarW0adOG5ORkU8UnhBBCiEIkT0MhoaGhRs+XLFmCl5cXx48fp2nTpvkamBBCCCEKn+eaYxEfHw+Au3vOCzqlp6eTnp5ueJ6QkACARqNBo9E8z9sbedhWfrYpspI8m4/k2jwkz+YheTYPU+Y5t22qFEXJbuHKp1IUhS5dunD//n327duX437jx49nwoQJWbavXLkSR0fHZ3lrIYQQQphZSkoKffr0IT4+HlfXnJdoeObCYtiwYWzevJn9+/dTtmzZHPfLrsfCz8+P2NjYJwaWVxqNhh07dtC6dWu5KsSEJM/mI7k2D8mzeUiezcOUeU5ISKBUqVJPLSyeaSjk/fff548//mDv3r1PLCoA7OzssLPLeodOGxsbk3y5TNWuMCZ5Nh/JtXlIns1D8mwepshzbtvLU2GhKArvv/8+69evZ/fu3QQEBDxTcEIIIYQomvJUWAwbNoyVK1eyceNGXFxcuH37NgBubm44ODiYJEAhhBBCFB55Wsdi/vz5xMfH07x5c3x9fQ2PX3/91VTxCSGEEKIQyfNQiBBCCCFETvLUYyGEEEII8SRSWAghhBAi35j97qYPh1MersCZXzQaDSkpKSQkJMilTCYkeTYfybV5SJ7NQ/JsHqbM88Pz9tOmRZi9sEhMTATAz8/P3G8thBBCiOeUmJiIm5tbjq8/88qbz0qv13Pr1i1cXFxQqVT51u7DFT2vX7+eryt6CmOSZ/ORXJuH5Nk8JM/mYco8K4pCYmIipUuXxsoq55kUZu+xsLKyeupqnc/D1dVVvrRmIHk2H8m1eUiezUPybB6myvOTeioeksmbQgghhMg3UlgIIYQQIt8UmcLCzs6OcePGZXvDM5F/JM/mI7k2D8mzeUiezaMg5NnskzeFEEIIUXQVmR4LIYQQQlieFBZCCCGEyDdSWAghhBAi30hhIYQQQoh8U6gKi3nz5hEQEIC9vT0vvPAC+/bte+L+e/bs4YUXXsDe3p4KFSrw448/minSwi0veV63bh2tW7fG09MTV1dXGjZsyLZt28wYbeGV1+/zQwcOHMDa2pratWubNsAiJK+5Tk9P57PPPqN8+fLY2dlRsWJFFi9ebKZoC6+85vmXX34hODgYR0dHfH19GThwIPfu3TNTtIXT3r176dSpE6VLl0alUrFhw4anHmP2c6FSSKxevVqxsbFRFi5cqISHhysjRoxQnJyclKtXr2a7/5UrVxRHR0dlxIgRSnh4uLJw4ULFxsZGWbt2rZkjL1zymucRI0Yo06ZNU44cOaJcvHhR+eSTTxQbGxvlxIkTZo68cMlrnh968OCBUqFCBaVNmzZKcHCweYIt5J4l1507d1bq16+v7NixQ4mMjFQOHz6sHDhwwIxRFz55zfO+ffsUKysr5dtvv1WuXLmi7Nu3T6lRo4by6quvmjnywmXLli3KZ599pvz+++8KoKxfv/6J+1viXFhoCouQkBDl3XffNdpWrVo15eOPP852/7FjxyrVqlUz2vbOO+8oDRo0MFmMRUFe85ydwMBAZcKECfkdWpHyrHnu2bOn8vnnnyvjxo2TwiKX8prrrVu3Km5ubsq9e/fMEV6Rkdc8f/PNN0qFChWMts2dO1cpW7asyWIsanJTWFjiXFgohkIyMjI4fvw4bdq0Mdrepk0bDh48mO0x//zzT5b927Zty7Fjx9BoNCaLtTB7ljw/Tq/Xk5iYiLu7uylCLBKeNc9LliwhIiKCcePGmTrEIuNZcv3HH39Qr149pk+fTpkyZahSpQpjxowhNTXVHCEXSs+S50aNGnHjxg22bNmCoijcuXOHtWvX0qFDB3OEXGxY4lxo9puQPYvY2Fh0Oh3e3t5G2729vbl9+3a2x9y+fTvb/bVaLbGxsfj6+pos3sLqWfL8uJkzZ5KcnEyPHj1MEWKR8Cx5vnTpEh9//DH79u3D2rpQ/NgWCM+S6ytXrrB//37s7e1Zv349sbGxvPfee8TFxck8ixw8S54bNWrEL7/8Qs+ePUlLS0Or1dK5c2e+++47c4RcbFjiXFgoeiweevw264qiPPHW69ntn912YSyveX5o1apVjB8/nl9//RUvLy9ThVdk5DbPOp2OPn36MGHCBKpUqWKu8IqUvHyn9Xo9KpWKX375hZCQENq3b8+sWbNYunSp9Fo8RV7yHB4ezgcffMCXX37J8ePHCQ0NJTIyknfffdccoRYr5j4XFoo/fUqVKoVarc5S+cbExGSpxB7y8fHJdn9ra2s8PDxMFmth9ix5fujXX3/lrbfeYs2aNbRq1cqUYRZ6ec1zYmIix44d4+TJkwwfPhzIPPkpioK1tTXbt2+nZcuWZom9sHmW77Svry9lypQxuj109erVURSFGzduULlyZZPGXBg9S56nTJlC48aN+fDDDwGoVasWTk5ONGnShK+++kp6lfOJJc6FhaLHwtbWlhdeeIEdO3YYbd+xYweNGjXK9piGDRtm2X/79u3Uq1cPGxsbk8VamD1LniGzp2LAgAGsXLlSxkdzIa95dnV15cyZM5w6dcrwePfdd6latSqnTp2ifv365gq90HmW73Tjxo25desWSUlJhm0XL17EysqKsmXLmjTewupZ8pySkoKVlfEpSK1WA//7i1o8P4ucC002LTSfPbyU6eeff1bCw8OVkSNHKk5OTkpUVJSiKIry8ccfK3379jXs//ASm1GjRinh4eHKzz//LJeb5kJe87xy5UrF2tpa+eGHH5To6GjD48GDB5b6CIVCXvP8OLkqJPfymuvExESlbNmySvfu3ZWzZ88qe/bsUSpXrqy8/fbblvoIhUJe87xkyRLF2tpamTdvnhIREaHs379fqVevnhISEmKpj1AoJCYmKidPnlROnjypAMqsWbOUkydPGi7rLQjnwkJTWCiKovzwww9K+fLlFVtbW6Vu3brKnj17DK/1799fadasmdH+u3fvVurUqaPY2toq/v7+yvz5880cceGUlzw3a9ZMAbI8+vfvb/7AC5m8fp8fJYVF3uQ11+fOnVNatWqlODg4KGXLllVGjx6tpKSkmDnqwieveZ47d64SGBioODg4KL6+vsobb7yh3Lhxw8xRFy67du164u/cgnAulNumCyGEECLfFIo5FkIIIYQoHKSwEEIIIUS+kcJCCCGEEPlGCgshhBBC5BspLIQQQgiRb6SwEEIIIUS+kcJCCCGEEPlGCgshhBBC5BspLIQQQgiRb6SwEEIIIUS+kcJCCCGEEPlGCgshhBBC5Jv/B0LFhVYonimoAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ "%matplotlib inline\n", "\n", @@ -618,9 +572,7 @@ { "cell_type": "markdown", "id": "02debd8b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The intercept is the value of our output/target variable\n", "when all our features are zero and our function crosses the $y$-axis (for a one-dimensional case). \n", @@ -639,9 +591,7 @@ { "cell_type": "markdown", "id": "b0fe4b39", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\beta_j^2,\n", @@ -651,9 +601,7 @@ { "cell_type": "markdown", "id": "87f17073", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "but when we take out the intercept, this equation becomes" ] @@ -661,9 +609,7 @@ { "cell_type": "markdown", "id": "91eb895d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\beta_j^2.\n", @@ -673,9 +619,7 @@ { "cell_type": "markdown", "id": "c3816f1e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "For Lasso regression we have" ] @@ -683,9 +627,7 @@ { "cell_type": "markdown", "id": "f0a71210", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\beta_j\\vert.\n", @@ -695,9 +637,7 @@ { "cell_type": "markdown", "id": "841a6167", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It means that, when scaling the design matrix and the outputs/targets,\n", "by subtracting the mean values, we have an optimization problem which\n", @@ -713,11 +653,113 @@ "cell_type": "code", "execution_count": 3, "id": "e848f1d3", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Beta values for own Ridge implementation\n", + "[ 1.03032441e+00 6.28336218e-02 -6.24175744e-01 5.21169159e-02\n", + " 2.80847477e-01 2.12552073e-01 8.13220608e-02 -1.69634577e-02\n", + " -6.50846111e-02 -7.38962192e-02 -5.94226022e-02 -3.50227564e-02\n", + " -9.80609616e-03 1.08299273e-02 2.41882037e-02 2.93492130e-02\n", + " 2.64742912e-02 1.63249532e-02 -5.01831050e-05 -2.15098090e-02]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[ 1.03032441e+00 6.28336218e-02 -6.24175744e-01 5.21169159e-02\n", + " 2.80847477e-01 2.12552073e-01 8.13220608e-02 -1.69634577e-02\n", + " -6.50846112e-02 -7.38962192e-02 -5.94226022e-02 -3.50227564e-02\n", + " -9.80609615e-03 1.08299273e-02 2.41882037e-02 2.93492130e-02\n", + " 2.64742912e-02 1.63249532e-02 -5.01831190e-05 -2.15098090e-02]\n", + "MSE values for own Ridge implementation\n", + "4.3632959111950474e-07\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "4.363295916366933e-07\n", + "Beta values for own Ridge implementation\n", + "[ 1.03630548 -0.01963611 -0.37900111 -0.07062318 0.12182967 0.16343471\n", + " 0.13003291 0.07490892 0.02365049 -0.01449782 -0.03814292 -0.04909093\n", + " -0.05009826 -0.04389027 -0.03279636 -0.01866537 -0.00289724 0.01348565\n", + " 0.02976145 0.04543942]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[ 1.03630548 -0.01963611 -0.37900111 -0.07062318 0.12182967 0.16343471\n", + " 0.13003291 0.07490892 0.02365049 -0.01449782 -0.03814292 -0.04909093\n", + " -0.05009826 -0.04389027 -0.03279636 -0.01866537 -0.00289724 0.01348565\n", + " 0.02976145 0.04543942]\n", + "MSE values for own Ridge implementation\n", + "5.194042826649355e-06\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "5.194042826815211e-06\n", + "Beta values for own Ridge implementation\n", + "[ 1.04220758 -0.10931453 -0.17641709 -0.06020587 0.02208512 0.05789007\n", + " 0.06491736 0.05785343 0.04537385 0.03196357 0.01969145 0.00934499\n", + " 0.00107405 -0.00526348 -0.00992331 -0.01318643 -0.01531845 -0.01655318\n", + " -0.01708852 -0.01708781]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[ 1.04220758 -0.10931453 -0.17641709 -0.06020587 0.02208512 0.05789007\n", + " 0.06491736 0.05785343 0.04537385 0.03196357 0.01969145 0.00934499\n", + " 0.00107405 -0.00526348 -0.00992331 -0.01318643 -0.01531845 -0.01655318\n", + " -0.01708852 -0.01708781]\n", + "MSE values for own Ridge implementation\n", + "2.0940821989652176e-05\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "2.0940821989627646e-05\n", + "Beta values for own Ridge implementation\n", + "[ 1.01219292 -0.06043581 -0.10391807 -0.05651951 -0.01898855 0.00312361\n", + " 0.01463049 0.01975848 0.02123176 0.02068067 0.01905883 0.01691985\n", + " 0.01458337 0.01223198 0.00996754 0.00784393 0.00588657 0.00410387\n", + " 0.00249435 0.00105081]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[ 1.01219292 -0.06043581 -0.10391807 -0.05651951 -0.01898855 0.00312361\n", + " 0.01463049 0.01975848 0.02123176 0.02068067 0.01905883 0.01691985\n", + " 0.01458337 0.01223198 0.00996754 0.00784393 0.00588657 0.00410387\n", + " 0.00249435 0.00105081]\n", + "MSE values for own Ridge implementation\n", + "0.00031535148309577417\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "0.0003153514830958095\n", + "Beta values for own Ridge implementation\n", + "[ 8.38916861e-01 1.31276579e-01 8.97497404e-03 -1.72271878e-02\n", + " -2.11744554e-02 -1.91492986e-02 -1.57201944e-02 -1.23002365e-02\n", + " -9.30466214e-03 -6.81048318e-03 -4.78184120e-03 -3.15130074e-03\n", + " -1.84923989e-03 -8.13661243e-04 7.46984697e-06 6.56636616e-04\n", + " 1.16805821e-03 1.56912044e-03 1.88168312e-03 2.12318726e-03]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[ 8.38916861e-01 1.31276579e-01 8.97497404e-03 -1.72271878e-02\n", + " -2.11744554e-02 -1.91492986e-02 -1.57201944e-02 -1.23002365e-02\n", + " -9.30466214e-03 -6.81048318e-03 -4.78184120e-03 -3.15130074e-03\n", + " -1.84923989e-03 -8.13661243e-04 7.46984697e-06 6.56636616e-04\n", + " 1.16805821e-03 1.56912044e-03 1.88168312e-03 2.12318726e-03]\n", + "MSE values for own Ridge implementation\n", + "0.01507238889517717\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "0.01507238889517706\n", + "Beta values for own Ridge implementation\n", + "[0.37396662 0.14174745 0.0764924 0.04892055 0.03447512 0.02586427\n", + " 0.02024962 0.01633913 0.01347916 0.0113104 0.0096208 0.00827728\n", + " 0.00719176 0.00630331 0.00556826 0.0049544 0.00443743 0.0039987\n", + " 0.0036237 0.003301 ]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[0.37396662 0.14174745 0.0764924 0.04892055 0.03447512 0.02586427\n", + " 0.02024962 0.01633913 0.01347916 0.0113104 0.0096208 0.00827728\n", + " 0.00719176 0.00630331 0.00556826 0.0049544 0.00443743 0.0039987\n", + " 0.0036237 0.003301 ]\n", + "MSE values for own Ridge implementation\n", + "0.2640931530791004\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "0.26409315307910025\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAYAAABB4NqyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABThUlEQVR4nO3deVxU9eL/8deArCK4oKAJuGSuaYqoaG4Vy2hdMyvvrSxLLW/lkr82rq1+b9ert2umZdomeculrpZmDIqVOymaaLllpmEKmRu4scic3x/qXEcWAcEDzPv5eMxD58xnzrxnonh35vM5x2IYhoGIiIiIC3EzO4CIiIjItaYCJCIiIi5HBUhERERcjgqQiIiIuBwVIBEREXE5KkAiIiLiclSARERExOXUMDtAZWS32zl06BC1atXCYrGYHUdERERKwDAMTp48SaNGjXBzK/4YjwpQIQ4dOkRISIjZMURERKQMDhw4QOPGjYsdowJUiFq1agHnP0B/f3+T04iIiEhJZGVlERIS4vg9XhwVoEJc/NrL399fBUhERKSKKcn0FU2CFhEREZejAiQiIiIuRwVIREREXI7mAF2F/Px88vLyzI4hUmE8PDxwd3c3O4aISLlTASoDwzDIyMjgxIkTZkcRqXC1a9cmODhY58QSkWpFBagMLpafBg0a4Ovrq18MUi0ZhsGZM2c4fPgwAA0bNjQ5kYhI+VEBKqX8/HxH+alXr57ZcUQqlI+PDwCHDx+mQYMG+jpMRKoNTYIupYtzfnx9fU1OInJtXPxZ13w3EalOVIDKSF97iavQz7qIVEcqQCIiIuJyVIBERETE5agAiRRh//79WCwWUlNTixyzcuVKLBaLTokgIlLFqAC5kKFDh2KxWBg5cmSBxx5//HEsFgtDhw51bDt8+DCPPfYYoaGheHl5ERwcTExMDMnJyY4xTZo0wWKxFLj985//vBZvqcz69OnjyOrp6Unz5s2Ji4sjJyfHMSYkJIT09HTatWtnYtKiP+OLtz59+pR533369GHs2LHlllVE5ErO5WazPmEW53KzTc2hZfAuJiQkhPnz5/PGG284ljhnZ2czb948QkNDncYOGjSIvLw8PvroI5o1a8bvv//O119/zbFjx5zGTZgwgREjRjhtq1WrVsW+kXIwYsQIJkyYQG5uLikpKTz88MMATJw4EQB3d3eCg4PNjAhASkoK+fn5AKxfv55Bgwaxe/du/P39AfD09DQznohIqXy3/EN6bn6C678exU//ysbiZs6xGB0BKg+GAadPm3MzjFJF7dSpE6GhoSxatMixbdGiRYSEhNCxY0fHthMnTrB27VomTZpE3759CQsLo0uXLsTFxdG/f3+nfdaqVYvg4GCnW82aNYvMcPz4cR588EHq1KmDr68vVquVPXv2XPgoDerXr8/ChQsd42+66SYaNGjguJ+cnIyHhwenTp0Czq9Sev/99xk4cCC+vr60aNGCJUuWXPGz8PX1JTg4mNDQUAYNGkRUVBTLly93PF7YV2AJCQnccMMN+Pj40LdvX/bv319gv++99x4hISH4+voycOBApkyZQu3atZ3GfPnll4SHh+Pt7U2zZs149dVXOXfuXKE569ev7/hc69atC0CDBg0c23bt2kWvXr3w8fEhJCSE0aNHc/r0acfzZ8yYQYsWLfD29iYoKIi7774bOH9EcNWqVbz55puOo0mFvR8RkfJk2/gJAJ1paFr5ARWg8nHmDPj5mXM7c6bUcR9++GFmz57tuP/hhx/yyCOPOI3x8/PDz8+PL774wulrofIwdOhQNm3axJIlS0hOTsYwDPr160deXh4Wi4VevXqxcuVK4HxZ2rFjB3l5eezYsQM4P+8mPDwcPz8/xz5fffVV7r33XrZt20a/fv24//77CxypKs7WrVtZt24dHh4eRY45cOAAd911F/369SM1NZXhw4fz/PPPO41Zt24dI0eOZMyYMaSmphIVFcVrr73mNGbZsmU88MADjB49mh07djBr1izi4+MLjCuJH374gZiYGO666y62bdvGggULWLt2LU8++SQAmzZtYvTo0UyYMIHdu3eTmJhIr169AHjzzTeJjIxkxIgRpKenk56eTkhISKkziIiUhu3kFgCszWLMDWJIAZmZmQZgZGZmFnjs7Nmzxo4dO4yzZ8/+b+OpU4Zx/ljMtb+dOlXi9/XQQw8ZAwYMMP744w/Dy8vL2Ldvn7F//37D29vb+OOPP4wBAwYYDz30kGP8f//7X6NOnTqGt7e30b17dyMuLs7YunWr0z7DwsIMT09Po2bNmk63b7/9ttAMP/30kwEY69atc2w7cuSI4ePjY3z66aeGYRjGtGnTjHbt2hmGYRhffPGF0blzZ+Ouu+4y3n77bcMwDCM6Otp47rnnHM8HjBdeeOGSfxynDIvFYthstiI/i969exseHh5GzZo1DU9PTwMw3NzcjP/+97+OMfv27TMAY8uWLYZhGEZcXJzRunVrw263O8Y899xzBmAcP37cMAzDGDx4sNG/f3+n17r//vuNgIAAx/2ePXsa//jHP5zG/Oc//zEaNmxYZN6Lvv32W6fXGzJkiPHoo486jVmzZo3h5uZmnD171li4cKHh7+9vZGVlFfk5jBkzptjXLPRnXkSkDNL3bjV4BYNXMDJ++aHc91/c7+/LaQ5QefD1hQtfx5jy2qUUGBhI//79+eijjzAMg/79+xMYGFhg3KBBg+jfvz9r1qwhOTmZxMREJk+ezPvvv+80WfqZZ55xug9w3XXXFfraO3fupEaNGnTt2tWxrV69erRs2ZKdO3cC5yfmjhkzhiNHjrBq1Sr69OlDaGgoq1at4tFHH2X9+vUFJu62b9/e8feaNWtSq1YtxzWsinL//fczfvx4srKymDRpEv7+/gwaNKjI8Tt37qRbt25OJwaMjIx0GrN7924GDhzotK1Lly4sXbrUcX/z5s2kpKQ4HfHJz88nOzubM2fOlOos45s3b+bnn3/mk08+cWwzDAO73c6+ffuIiooiLCyMZs2aERsbS2xsrOOrQhGRa22ZbToA4Sd8CWpq7gITFaDyYLFAMXNeKqNHHnnE8TXJ22+/XeQ4b29voqKiiIqK4qWXXmL48OG8/PLLToUnMDCQ66+/vkSvaxQxZ8kwDEexaNeuHfXq1WPVqlWsWrWKCRMmEBISwmuvvUZKSgpnz57l5ptvdnr+5V9dWSwW7HZ7sVkCAgIcuT/++GPatm3LBx98wLBhw0qVvaj3UdTz7HY7r776KnfddVeB53t7e1/xNS7f12OPPcbo0aMLPBYaGoqnpyfff/89K1euZPny5bz00ku88sorpKSkFJiXJCJS0Wx7l0EAWP07mR1Fc4BcVWxsLLm5ueTm5hITU/LvYdu0aeM0wba02rRpw7lz59iwYYNj29GjR/npp59o3bo1gGMe0OLFi/nxxx/p2bMnN954I3l5ecycOZNOnTqV+yozDw8P/va3v/HCCy9wpoh5VW3atOG7775z2nb5/VatWrFx40anbZs2bXK636lTJ3bv3s31119f4OZWygmBnTp1Yvv27YXu6+LqsBo1anDbbbcxefJktm3bxv79+/nmm2+A8yvILq4wExGpSOdys1nu9RsA1q4PmJxGBchlubu7s3PnTnbu3FnoFb6PHj3KLbfcwscff8y2bdvYt28fn332GZMnT2bAgAFOY0+ePElGRobTLSsrq9DXbdGiBQMGDGDEiBGsXbuWrVu38sADD3Ddddc57bdPnz7MnTuX9u3b4+/v7yhFn3zyyVWd96Y49913HxaLhRkzZhT6+MiRI9m7dy/jxo1j9+7dzJ07l/j4eKcxo0aNIiEhgSlTprBnzx5mzZqFzWZzOir00ksvMWfOHF555RW2b9/Ozp07WbBgAS+88EKpMz/33HMkJyfzxBNPkJqayp49e1iyZAmjRo0CYOnSpUybNo3U1FR+/fVX5syZg91up2XLlsD5cwxt2LCB/fv3c+TIkSseNRMRKauNKz7iuLdBnWwLXW57yOw4KkCuzN/f33Eumcv5+fnRtWtX3njjDXr16kW7du148cUXGTFiBG+99ZbT2JdeeomGDRs63Z599tkiX3f27NmEh4dz++23ExkZiWEYJCQkOH2N1bdvX/Lz853KTu/evcnPz6d3795X98aL4OnpyZNPPsnkyZMdS+wvFRoaysKFC/nyyy/p0KEDM2fO5B//+IfTmB49ejBz5kymTJlChw4dSExM5KmnnnL6aismJoalS5eSlJREREQE3bp1Y8qUKYSFhZU6c/v27Vm1ahV79uyhZ8+edOzYkRdffJGGDRsCULt2bRYtWsQtt9xC69atmTlzJvPmzaNt27YAPP3007i7u9OmTRvq169PWlpaqTOIiJSEbcPHAETlNKaGZ+m+7q8IFqMkExtcTFZWFgEBAWRmZhYoCNnZ2ezbt4+mTZuWer6GuKYRI0awa9cu1qxZY3aUMtHPvIiUh85P1WRz7TPMrjeMoU++XyGvUdzv78tpErRIOXv99deJioqiZs2a2Gw2PvrooyK/VhMRcQW/7/uRzbXPz6+M7Vdw0YYZVIBEytnGjRuZPHkyJ0+epFmzZkybNo3hw4ebHUtExDTLbeenTnQ84UNws/ZXGH1tqACJlLNPP/3U7AgiIpWKbW8i+IO1VscrD75GNAlaREREKkx+Xi7LPM4vsLB2ud/kNP+jAiQiIiIVJuXrORzzMQjIhm7Rj1z5CdeI6QVoxowZjtUl4eHhxa6UWbRoEVFRUdSvXx9/f38iIyNZtmyZ05j4+HjHla0vvWVnZ1f0WxEREZHL2L77DwBROddViuXvF5lagBYsWMDYsWMZP348W7ZsoWfPnlit1iLPRbJ69WqioqJISEhg8+bN9O3blzvuuIMtW7Y4jfP393dc3friTct3RURErj1b5mYArE2iTE7izNRJ0FOmTGHYsGGOFTJTp05l2bJlvPPOO0ycOLHA+KlTpzrd/8c//sHixYv58ssv6djxfxOrLBYLwcHBFZpdREREivdH2k42BZy/fFKsdZTJaZyZdgQoNzeXzZs3Ex0d7bQ9Ojqa9evXl2gfdrudkydPUrduXaftp06dIiwsjMaNG3P77bcXOEJ0uZycHLKyspxuYq79+/djsVhITU0tcszKlSuxWCycOHHimuUSEZGSW54wHcMCHU5406iF+RdAvZRpBejIkSPk5+cTFBTktD0oKIiMjIwS7ePf//43p0+f5t5773Vsa9WqFfHx8SxZsoR58+bh7e1Njx492LNnT5H7mThxIgEBAY5bSEhI2d5UJTd06FAsFgsjR44s8Njjjz+OxWJxusr74cOHeeyxxwgNDcXLy4vg4GBiYmJITk52jGnSpEmhc67++c9/FpmjT58+jnGenp40b96cuLg4cnJyHGNCQkJIT0+nXbt25fPmy6io93fxdjXXJevTpw9jx44tt6wiIpWNbY8NAKvfTeYGKYTp5wG69CKRAIZhFNhWmHnz5vHKK6+wePFiGjRo4NjerVs3unXr5rjfo0cPOnXqxPTp05k2bVqh+4qLi2PcuHGO+1lZWdW2BIWEhDB//nzeeOMNfHx8gPOXOpg3bx6hoaFOYwcNGkReXh4fffQRzZo14/fff+frr7/m2LFjTuMmTJjAiBEjnLZd6WrtI0aMYMKECeTm5pKSksLDDz8M4Pjq093dvVJ8jZmSkuK4Wvr69esZNGgQu3fvdpxi/eIV10VExJk9/xzLPH4FwBpxn8lpCjLtCFBgYCDu7u4FjvYcPny4wFGhyy1YsIBhw4bx6aefcttttxU71s3NjYiIiGKPAHl5eTkuDFrcBUKrg06dOhEaGsqiRYsc2xYtWkRISIjTPKoTJ06wdu1aJk2aRN++fQkLC6NLly7ExcXRv39/p33WqlWL4OBgp1vNmjWLzeHr60twcDChoaEMGjSIqKgoli9f7ni8sK/AEhISuOGGG/Dx8aFv377s37+/wH7fe+89QkJC8PX1ZeDAgUyZMoXatWs7jfnyyy8JDw/H29ubZs2a8eqrr3Lu3LlCc9avX9/xni5+1dqgQQPHtl27dtGrVy98fHwICQlh9OjRnD592vH8GTNm0KJFC7y9vQkKCuLuu+8Gzh+NW7VqFW+++abjaFJh70dEpKra9PV/OOJj4J8DkdHDzI5TgGkFyNPTk/DwcJKSkpy2JyUl0b179yKfN2/ePIYOHcrcuXML/CIujGEYpKamOq6OXREMw+B07mlTbmW5lu3DDz/M7NmzHfc//PBDHnnE+dwMfn5++Pn58cUXXzh9NVURtm7dyrp165yuBn+5AwcOcNddd9GvXz9SU1MZPnw4zz//vNOYdevWMXLkSMaMGUNqaipRUVG89tprTmOWLVvGAw88wOjRo9mxYwezZs0iPj6+wLiS+OGHH4iJieGuu+5i27ZtLFiwgLVr1/Lkk08CsGnTJkaPHs2ECRPYvXs3iYmJ9OrVC4A333yTyMhIRowY4VipWF2POoqIa7IlzwEgKvs6PLx9TU5TkKlfgY0bN44hQ4bQuXNnIiMjeffdd0lLS3PMUYmLi+PgwYPMmXP+Q5w3bx4PPvggb775Jt26dXMcPfLx8SEgIACAV199lW7dutGiRQuysrKYNm0aqampvP322xX2Ps7kncFvol+F7b84p+JOUdOz+KMtlxsyZAhxcXGOoyzr1q1j/vz5rFy50jGmRo0axMfHM2LECGbOnEmnTp3o3bs3f/7zn2nf3vk6Ls899xwvvPCC07alS5cWOz9mxowZvP/+++Tl5ZGbm4ubm1ux/4zeeecdmjVrxhtvvIHFYqFly5b88MMPTJo0yTFm+vTpWK1Wnn76aQBuuOEG1q9fz9KlSx1jXnvtNZ5//nkeeughAJo1a8b//d//8eyzz/Lyyy9f8bO71L/+9S/uu+8+xzyeFi1aMG3aNHr37s0777xDWloaNWvW5Pbbb6dWrVqEhYU5jrIFBATg6enpOBImIlLd2E6kQG2IDbvV7CiFMrUADR48mKNHjzJhwgTHhNeEhATCwsIASE9Pdzon0KxZszh37hxPPPEETzzxhGP7Qw89RHx8PHD+q5tHH32UjIwMAgIC6NixI6tXr6ZLly7X9L1VZoGBgfTv35+PPvoIwzDo378/gYGBBcYNGjSI/v37s2bNGpKTk0lMTGTy5Mm8//77TpOln3nmGaf7ANddd12xGe6//37Gjx9PVlYWkyZNwt/fn0GDBhU5fufOnXTr1s1pflhkZKTTmN27dzNw4ECnbV26dHEqQJs3byYlJcXpiE9+fj7Z2dmcOXMGX9+S/1/K5s2b+fnnn/nkk08c2wzDwG63s2/fPqKioggLC6NZs2bExsYSGxvLwIEDS/UaIiJV0ZEDu9l4cfl77JMmpymc6ZOgH3/8cR5//PFCH7tYai669AhFUd544w3eeOONckhWcr4evpyKO3VNX/PS1y6LRx55xPFVTXFHXry9vYmKiiIqKoqXXnqJ4cOH8/LLLzsVnsDAQK6//vpSvX5AQIDjOR9//DFt27blgw8+YNiwwr8nLslXfYVNoL/8eXa7nVdffZW77rqrwPNLe7JMu93OY489xujRows8FhoaiqenJ99//z0rV65k+fLlvPTSS7zyyiukpKQUmJckIlKdLP9qGoYFbsz0pnHLCLPjFMr0AlQdWCyWUn8NZbbY2Fhyc3MBiImJKfHz2rRpwxdffFGuWTw8PPjb3/5GXFwcf/nLXwo9QlLY63733XdO91u1asXGjRudtm3atMnpfqdOndi9e3epC1thOnXqxPbt24vdV40aNbjtttu47bbbePnll6lduzbffPMNd911F56eno4VZiIi1Uniz4lQC6y+7a882CSmXwtMzOHu7s7OnTvZuXMn7u7uBR4/evQot9xyCx9//DHbtm1j3759fPbZZ0yePJkBAwY4jT158iQZGRlOt9KeTPK+++7DYrEwY8aMQh8fOXIke/fuZdy4cezevZu5c+cWOEI4atQoEhISmDJlCnv27GHWrFnYbDano0IvvfQSc+bM4ZVXXmH79u3s3LmTBQsWFJjDVBLPPfccycnJPPHEE6SmprJnzx6WLFnCqFHnz3a6dOlSxxy0X3/9lTlz5mC322nZsiVw/hxDGzZsYP/+/Rw5cgS73V7qDCIilY09/xyJ7vsAsHb+i8lpiqYC5MKKW/Lv5+dH165deeONN+jVqxft2rXjxRdfZMSIEbz11ltOY1966SUaNmzodHv22WdLlcXT05Mnn3ySyZMnc+pUwa8TQ0NDWbhwIV9++SUdOnRg5syZ/OMf/3Aa06NHD2bOnMmUKVPo0KEDiYmJPPXUU05fbcXExLB06VKSkpKIiIigW7duTJkyxTHvrDTat2/PqlWr2LNnDz179qRjx468+OKLjhWHtWvXZtGiRdxyyy20bt2amTNnMm/ePNq2bQvA008/jbu7O23atKF+/fpFXgNPRKQq+f7bufzha1ArB3rEPmp2nCJZjLKso67msrKyCAgIIDMzs0BByM7OZt++fY4r2EvlNmLECHbt2sWaNWvMjlJl6WdeRErj/ybcykvGNww80ZBFbxy6pq9d3O/vy2kOkFQrr7/+OlFRUdSsWRObzcZHH31U5NdqIiJS/mzHNkIdsIYVf6Jis6kASbWyceNGJk+ezMmTJ2nWrBnTpk1j+PDhZscSEXEJxw7tZUPA+WkMsTFPXGG0uVSApFr59NNPzY4gIuKyli99E7sbtM30IqR1V7PjFEuToEVERKRcJP6UAIDV50aTk1yZClAZae64uAr9rItISTgtfw//s8lprkwFqJQuXrDzzJkzJicRuTYu/qwXd7FaEZHUVQv43deOXy7cbH3M7DhXpDlApeTu7k7t2rU5fPgwAL6+vgUuvyBSHRiGwZkzZzh8+DC1a9cu9ISZIiIX2dbFA3DrmWA8fcy5QHhpqACVwcWrd18sQSLVWe3atXXFehG5ItvRDeeXv4feYnaUElEBKgOLxULDhg1p0KABeXl5ZscRqTAeHh468iMiV3Q8fR/JAScBiI0u/ALnlY0K0FVwd3fXLwcREXF5SV9Nw+4GrTM9CWvbw+w4JaJJ0CIiInJVbLuWAmD1rvzL3y9SARIREZEyM+x2Et1+AcDa6V6T05ScCpCIiIiU2dY1n5FR007NXOhpHWl2nBJTARIREZEys635EIBbzgThVbP4K7BXJipAIiIiUma2I98BYG3c1+QkpaMCJCIiImVy4vdfWR+QBUBs1F9NTlM6KkAiIiJSJiuWTiPfDVpmedK0fS+z45SKCpCIiIiUiW3XlwBYPduanKT0VIBERESk1Ay7nUTLXgCsHe8xOU3pqQCJiIhIqf2wbhGHatrxzYVe/arW/B9QARIREZEysK3+AIC+Zxrg7Vfb3DBloAIkIiIipWb7IxkA63W9TU5SNipAIiIiUipZf/zGOv9MAKxRVePq75dTARIREZFSWbF0Gufc4YYsD5p16GN2nDJRARIREZFSse1YAkCsZxuTk5SdCpCIiIiUmGG3Y2MPANYOd5ucpuxUgERERKTEticv5qCfHe886N2vas7/ARUgERERKQXbqvcB6Hu6Pj7+dU1OU3YqQCIiIlJitsPrAbA2qlrX/rqcCpCIiIiUyMmjh1jrfwIA621V7+zPl1IBEhERkRL5eul08tzh+iwPru94q9lxrooKkIiIiJSIbfsXAFg9WpsbpByoAImIiMgVGXY7NuP88vfY9neZnObqqQCJiIjIFe347ksO+OXjdQ769H/C7DhXTQVIRERErijxwvL3Pqfq4RsQaHKaq6cCJCIiIldk+30dANaGVXv5+0UqQCIiIlKsU8cyWFPrOADWWx4zOU35UAESERGRYn3z1XRya0CzkzVo0SnK7DjlQgVIREREimX78QsArDVaY3GrHtWherwLERERqRCG3Y4tfzcAse3uNDdMOVIBEhERkSLtSkng11r5eJ6Dvv2fNDtOuVEBEhERkSIlfnt++Xvvk3WpWaeByWnKjwqQiIiIFMmWsQYAa3BPk5OULxUgERERKdTp44dZVesYANZbHjU5TflSARIREZFCffvVW+TWgCYna9Cyc6zZccqVCpCIiIgUyvbD5wBY3VtWm+XvF1WvdyMiIiLl4vzy910AWNsNNDlN+VMBEhERkQJ+2ryMfbXOXVj+XvWv/n45FSAREREpwPbNuwD0PFkHv7rBJqcpf6YXoBkzZtC0aVO8vb0JDw9nzZo1RY5dtGgRUVFR1K9fH39/fyIjI1m2bFmBcQsXLqRNmzZ4eXnRpk0bPv/884p8CyIiItVO4sXl70E9TE5SMUwtQAsWLGDs2LGMHz+eLVu20LNnT6xWK2lpaYWOX716NVFRUSQkJLB582b69u3LHXfcwZYtWxxjkpOTGTx4MEOGDGHr1q0MGTKEe++9lw0bNlyrtyUiIlKlnck8wkq/owBY+44wOU3FsBiGYZj14l27dqVTp0688847jm2tW7fmzjvvZOLEiSXaR9u2bRk8eDAvvfQSAIMHDyYrKwubzeYYExsbS506dZg3b16h+8jJySEnJ8dxPysri5CQEDIzM/H39y/LWxMREamyEua+Sv89rxB6yp39k3KrzAqwrKwsAgICSvT727R3lJuby+bNm4mOjnbaHh0dzfr160u0D7vdzsmTJ6lbt65jW3JycoF9xsTEFLvPiRMnEhAQ4LiFhISU4p2IiIhUL7ZtCwGwWm6oMuWntEx7V0eOHCE/P5+goCCn7UFBQWRkZJRoH//+9785ffo09957r2NbRkZGqfcZFxdHZmam43bgwIFSvBMREZHqxZa3EwBrNbr6++VqmB3AYrE43TcMo8C2wsybN49XXnmFxYsX06CB88XZSrtPLy8vvLy8SpFaRESketqzOYm9/ufwyIdbqtHV3y9nWgEKDAzE3d29wJGZw4cPFziCc7kFCxYwbNgwPvvsM2677Tanx4KDg8u0TxEREQHb1zMBuDmrNrXqNTI5TcUx7SswT09PwsPDSUpKctqelJRE9+7di3zevHnzGDp0KHPnzqV///4FHo+MjCywz+XLlxe7TxERETkvMf3C8vcG1fv3pqlfgY0bN44hQ4bQuXNnIiMjeffdd0lLS2PkyJHA+bk5Bw8eZM6cOcD58vPggw/y5ptv0q1bN8eRHh8fHwICAgAYM2YMvXr1YtKkSQwYMIDFixezYsUK1q5da86bFBERqSLOZh3j25p/AGDtPdzkNBXL1KndgwcPZurUqUyYMIGbbrqJ1atXk5CQQFhYGADp6elO5wSaNWsW586d44knnqBhw4aO25gxYxxjunfvzvz585k9ezbt27cnPj6eBQsW0LVr12v+/kRERKqSVQkzyPaAxqfcaRs5wOw4FcrU8wBVVqU5j4CIiEh1MSbuJqZ5b2XE6Va8O3mn2XFKrUqcB0hEREQqF1vuDgCsbf5kcpKKpwIkIiIi7E39hj3+edTIh1v7jzI7ToVTARIRERFsSecvS9UjKwD/+o1NTlPxVIBEREQE26FVAFjrR5qc5NpQARIREXFx2adO/G/5e69hJqe5NlSAREREXNzqhHc46wHXnXLjxh53mR3nmlABEhERcXG2LZ8BEMv11fbq75dzjXcpIiIiRbLlbgfA2rr6L3+/SAVIRETEhe3btprd/rnUyIfbbq/+y98vUgESERFxYbakGQB0zwogoEGoyWmuHRUgERERF2b77VsAYgNd65qZKkAiIiIuKud0Ft/4HgbA2vMRk9NcWypAIiIiLmqNbSZnPKHhaTc69LzH7DjXlAqQiIiIi7J9/ykAsUZzl1n+fpFrvVsRERFxsGX/AIC11e0mJ7n2VIBERERc0K/b17EzIBd3O0TdPsbsONecCpCIiIgLsi17G4DITH9qB4WZnObaUwESERFxQbYD55e/W+u51vL3i1SAREREXEzO6Sy+9s0AILbHQyanMYcKkIiIiItZt+w9TntC0Bk3buo92Ow4plABEhERcTG2zfMBiM1vipt7DZPTmEMFSERExMXYzl5Y/t6yv8lJzKMCJCIi4kIO7NzA9oAc3OwQ1X+02XFMowIkIiLiQmzLpgPQLbMWdRs1NzmNeVSAREREXIjt128AsNbtYnISc6kAiYiIuIjcs6dY4ZsOgLXHUHPDmEwFSERExEWsX/Y+pzyhwRkLHfv82ew4plIBEhERcRG2TfMAiHHh5e8XqQCJiIi4CNuZbQBYW/QzOYn5VIBERERcwG+7U/ghIBs3O0S78PL3i1SAREREXEBi4lsAdMnyo17jFianMZ8KkIiIiAuw/boCAGvtCJOTVA4qQCIiItVcXvYZVngfAsDa3TWv/n45FSAREZFqbv2y98jygsCzFsJvud/sOJWCCpCIiEg1l7jp/NXfY/LCXH75+0UqQCIiItWc7VQqANYWVnODVCIqQCIiItXYoT3fs7V2NhYDYvqPMTtOpaECJCIiUo0l2s5f/T0isyaBIS1NTlN5qACJiIhUY7b9SQBYAzqbnKRyUQESERGpps7lZpPkdRAAa+QQk9NULipAIiIi1VTysvfJ9IZ6Zy10vkUF6FIqQCIiItWUbeNcAKLzQnH38DQ5TeWiAiQiIlJNJV5c/t481twglZAKkIiISDWU8cs2ttQ+C0BMv1Emp6l8VIBERESqocSEaQB0PuFLgyZtTU5T+agAiYiIVEO2X5YDYPUPNzlJ5aQCJCIiUs2cy81muddvAFi7PWBymspJBUhERKSa2ZA0mxPeBnXPWuhy21Cz41RKKkAiIiLVjG3DJwBE54Vo+XsRVIBERESqmcSTWwCIbRZtcpLKSwVIRESkGvl9349srn0GgNh+uvp7UVSAREREqpFlF5a/dzrhQ1DTdianqbxML0AzZsygadOmeHt7Ex4ezpo1a4ocm56ezn333UfLli1xc3Nj7NixBcbEx8djsVgK3LKzsyvwXYiIiFQOtl+WAWCt1cnkJJWbqQVowYIFjB07lvHjx7NlyxZ69uyJ1WolLS2t0PE5OTnUr1+f8ePH06FDhyL36+/vT3p6utPN29u7ot6GiIhIpZCfl8tyjwMAWLveb3Kays3UAjRlyhSGDRvG8OHDad26NVOnTiUkJIR33nmn0PFNmjThzTff5MEHHyQgIKDI/VosFoKDg51uIiIi1d3GFfEc8zGonW2ha9TDZsep1EwrQLm5uWzevJnoaOcZ6tHR0axfv/6q9n3q1CnCwsJo3Lgxt99+O1u2bCl2fE5ODllZWU43ERGRqsb23ccAROc0poanvvkojmkF6MiRI+Tn5xMUFOS0PSgoiIyMjDLvt1WrVsTHx7NkyRLmzZuHt7c3PXr0YM+ePUU+Z+LEiQQEBDhuISEhZX59ERERs9iyNgNg1fL3KzJ9ErTFYnG6bxhGgW2l0a1bNx544AE6dOhAz549+fTTT7nhhhuYPn16kc+Ji4sjMzPTcTtw4ECZX19ERMQMh/dvZ9OF5e8x1idNTlP51TDrhQMDA3F3dy9wtOfw4cMFjgpdDTc3NyIiIoo9AuTl5YWXl1e5vaaIiMi1ttz2FgA3nfChYfObzA1TBZh2BMjT05Pw8HCSkpKcticlJdG9e/dyex3DMEhNTaVhw4bltk8REZHKxvazDQCr303mBqkiTDsCBDBu3DiGDBlC586diYyM5N133yUtLY2RI0cC57+aOnjwIHPmzHE8JzU1FTg/0fmPP/4gNTUVT09P2rRpA8Crr75Kt27daNGiBVlZWUybNo3U1FTefvvta/7+REREroX8vFyWeZw/hYy1y30mp6kaTC1AgwcP5ujRo0yYMIH09HTatWtHQkICYWFhwPkTH15+TqCOHTs6/r5582bmzp1LWFgY+/fvB+DEiRM8+uijZGRkEBAQQMeOHVm9ejVdunS5Zu9LRETkWtr0zX846mMQkA2RMcPNjlMlWAzDMMwOUdlkZWUREBBAZmYm/v7+ZscREREp1iuv9OFVyyruzryOz6b8ZnYc05Tm97fpq8BERETk6tgyNwFgbRJlcpKqQwVIRESkCjtyYDcpAacBiIl9wuQ0VYcKkIiISBW2/KtpGBZof8Kb627obHacKkMFSEREpAqz7bmw/L1m0RcJl4JUgERERKooe/45ltXYD4A14i/mhqliVIBERESqqM3ffMIfvgb+OdA9ZoTZcaqUUhWgyZMnc/bsWcf91atXk5OT47h/8uRJHn/88fJLJyIiIkWyrf8IgNuyG+Hh7WtymqqlVAUoLi6OkydPOu7ffvvtHDx40HH/zJkzzJo1q/zSiYiISJFsJ1IAsIbdZnKSqqdUBejycybqHIoiIiLmOPrbHjYEnAIgNlZXfy8tzQESERGpgpISpmNYoF2mF41bRpgdp8pRARIREamCbD99BYDVp73JSaqmUl8M9f3338fPzw+Ac+fOER8fT2BgIIDT/CARERGpGPb8cyS67wPA2vnPJqepmkp1MdQmTZpgsViuOG7fvn1XFcpsuhiqiIhUZpu//pjOa4fglwtHXziJp4+f2ZEqhdL8/i7VEaD9+/dfTS4REREpB7Z18QDcdqahyk8ZaQ6QiIhIFWM7thEAa9gtJiepukpVgDZs2IDNZnPaNmfOHJo2bUqDBg149NFHnU6MKCIiIuXr2KG9fBdwfs6tNWaUyWmqrlIVoFdeeYVt27Y57v/www8MGzaM2267jeeff54vv/ySiRMnlntIEREROW9FwlvY3aBtphchrbuaHafKKlUBSk1N5dZbb3Xcnz9/Pl27duW9995j3LhxTJs2jU8//bTcQ4qIiMh5tl1LAYj1aWdykqqtVAXo+PHjBAUFOe6vWrWK2NhYx/2IiAgOHDhQfulERETEwZ5/jkS3XwCwdhpscpqqrVQFKCgoyLHEPTc3l++//57IyEjH4ydPnsTDw6N8E4qIiAgAW9d8RkZNOzVz4ebYx8yOU6WVqgDFxsby/PPPs2bNGuLi4vD19aVnz56Ox7dt20bz5s3LPaSIiIiAbc1sAG49E4xXTZ2n7mqU6jxAf//737nrrrvo3bs3fn5+xMfH4+np6Xj8ww8/JDo6utxDioiICNiOboA6YA3pa3aUKq9UBah+/fqsWbOGzMxM/Pz8cHd3d3r8s88+o1atWuUaUERERODE77+SHJAFgDXmCZPTVH2lKkCPPPJIicZ9+OGHZQojIiIihUta+ib5btA605Owtj3MjlPllaoAxcfHExYWRseOHSnFJcRERETkKiXuWgp+EOut5e/loVQFaOTIkcyfP59ffvmFRx55hAceeIC6detWVDYREREBDLudRMteAKwd7zE5TfVQqlVgM2bMID09neeee44vv/ySkJAQ7r33XpYtW6YjQiIiIhVk29r/cqimHd9c6NXvcbPjVAulvhiql5cXf/nLX0hKSmLHjh20bduWxx9/nLCwME6dOlURGUVERFyabfX5ubW3nGmg5e/l5KquBm+xWLBYLBiGgd1uL69MIiIicgnbke8AsDbW8vfyUuoClJOTw7x584iKiqJly5b88MMPvPXWW6SlpeHn51cRGUVERFxW5uE01gVkAmCN0tdf5aVUk6Aff/xx5s+fT2hoKA8//DDz58+nXr16FZVNRETE5a1YOp18N2iZ5UnT9r3MjlNtlKoAzZw5k9DQUJo2bcqqVatYtWpVoeMWLVpULuFERERcXeLOL8EPrJ5tzY5SrZSqAD344INYLJaKyiIiIiKXMOx2bOwBIPamQSanqV5KfSJEERERuTZ+XP85B/3s+ORB7/66/EV5uqpVYCIiIlJxbKs+AKDv6fp4+9U2N0w1owIkIiJSSdn+SAbA2qi3yUmqHxUgERGRSijrj99Y638CAGvUX80NUw2pAImIiFRCX381nXPu0CLLg+Y33WJ2nGpHBUhERKQSsu1YAoDVs43JSaonFSAREZFKxrDbSTTOL3+3drjb5DTVkwqQiIhIJbPjuy854JePdx701tXfK4QKkIiISCVjW/UeAH1OB+LjX9fkNNWTCpCIiEglY/t9PQDWhrr2V0VRARIREalETh49xBr/4wBYbx1pcprqSwVIRESkEvnmq7fIc4fmWTVoER5ldpxqSwVIRESkErH9+AUAVo/W5gap5lSAREREKgnDbsdm/ASAtb2u/l6RVIBEREQqiV0pCaT55eN1Dvro6u8VSgVIRESkkrB9e375e++T9fANCDQ5TfWmAiQiIlJJ2DLWAmBt2NPkJNWfCpCIiEglcOpYBqtrHQPAesujJqep/lSAREREKoFvv3qb3BrQ9GQNbgiPMTtOtacCJCIiUgnYfvwcAKt7Kyxu+vVc0Uz/hGfMmEHTpk3x9vYmPDycNWvWFDk2PT2d++67j5YtW+Lm5sbYsWMLHbdw4ULatGmDl5cXbdq04fPPP6+g9CIiIlfPsNux5e8GwHrjQJPTuAZTC9CCBQsYO3Ys48ePZ8uWLfTs2ROr1UpaWlqh43Nycqhfvz7jx4+nQ4cOhY5JTk5m8ODBDBkyhK1btzJkyBDuvfdeNmzYUJFvRUREpMx+2ryM/bXO4XkO+vZ/0uw4LsFiGIZh1ot37dqVTp068c477zi2tW7dmjvvvJOJEycW+9w+ffpw0003MXXqVKftgwcPJisrC5vN5tgWGxtLnTp1mDdvXolyZWVlERAQQGZmJv7+/iV/QyIiImUwddJAnsr+gtuO1yVp6lGz41RZpfn9bdoRoNzcXDZv3kx0dLTT9ujoaNavX1/m/SYnJxfYZ0xMTLH7zMnJISsry+kmIiJyrdjSz0//sAbfbHIS12FaATpy5Aj5+fkEBQU5bQ8KCiIjI6PM+83IyCj1PidOnEhAQIDjFhISUubXFxERKY0zmUdYVev8UR9r3xEmp3Edpk+CtlgsTvcNwyiwraL3GRcXR2ZmpuN24MCBq3p9ERGRkvp26Vvk1ICwk+60iuhndhyXUcOsFw4MDMTd3b3AkZnDhw8XOIJTGsHBwaXep5eXF15eXmV+TRERkbKy/bAIfMDq3lLL368h0z5pT09PwsPDSUpKctqelJRE9+7dy7zfyMjIAvtcvnz5Ve1TRESkIhh2O7ZzOwGwtrvT3DAuxrQjQADjxo1jyJAhdO7cmcjISN59913S0tIYOXIkcP6rqYMHDzJnzhzHc1JTUwE4deoUf/zxB6mpqXh6etKmTRsAxowZQ69evZg0aRIDBgxg8eLFrFixgrVr117z9yciIlKcPd8n8cuF5e+39B9ldhyXYmoBGjx4MEePHmXChAmkp6fTrl07EhISCAsLA86f+PDycwJ17NjR8ffNmzczd+5cwsLC2L9/PwDdu3dn/vz5vPDCC7z44os0b96cBQsW0LVr12v2vkREREoi8Zt3Aeh5sg5+dYNNTuNaTD0PUGWl8wCJiMi1YH2qPom1j/Avz/48HbfU7DhVXpU4D5CIiIgrO5t1jJU1jwBg7a3l79eaCpCIiIgJVn71FtkeEHLKnTbd7jA7jstRARIRETGBbetCAKyWFlr+bgJ94iIiIiaw5V1Y/t72TnODuCgVIBERkWvs5y1f87N/Hh75cOvtWv5uBhUgERGRayxxxUwAbs6qTa16jUxO45pUgERERK4x26FVAFgb6CoFZlEBEhERuYayT53g25p/ABDb6xGT07guFSAREZFraNVXb3PWA6475Ua77gPNjuOyVIBERESuIVvqfwGwouXvZtInLyIicg3ZcrcDYG3zJ5OTuDYVIBERkWvkl60r+ck/jxr5cNvto82O49JUgERERK4RW9IMAHpkBeBfv7HJaVybCpCIiMg1knjwwvL3+pEmJxEVIBERkWsg+9QJvvE9DEBsz4dNTiMqQCIiItfAGttMznhCo9NutL/5brPjuDwVIBERkWvAtuUzAGKN5lr+Xgnon4CIiMg1YMv+EQBrqztMTiKgAiQiIlLh9v+4ll0Bubjbtfy9slABEhERqWC2ZW8D0D3Tn9pBYSanEVABEhERqXCJv60EwBrYzdwg4qACJCIiUoFyTmfxtW8GANaeuvp7ZaECJCIiUoHWJs7itCcEn3ajQ897zI4jF6gAiYiIVCDb9wsAiLU30/L3SkT/JERERCqQ7ezF5e+3m5xELqUCJCIiUkHSdiSzIyAHNztE9dfy98pEBUhERKSC2Ja9BUBkZi3qNGxqchq5lAqQiIhIBbGlfQOAtV5Xk5PI5VSAREREKkDu2VP/W/7eY6i5YaQAFSAREZEKsC7xXU55QtAZN27qPdjsOHIZFSAREZEKYNs0H4CY/Ca4udcwOY1cTgVIRESkAtjObgPAekN/k5NIYVSAREREytmBnRv48cLy9+jbx5gdRwqhAiQiIlLOEi9c/b1rph91GzU3OY0URgVIRESknNl+XQGAtW4Xk5NIUVSAREREylFe9hlW+KQDYO3+kMlppCgqQCIiIuVo/bL3OOkFDc5Y6NT3PrPjSBFUgERERMqRLWUeADH5TbX8vRJTARIRESlHttNbAYi9PtbkJFIcFSAREZFycvCnTWyrnY3FgGhd/b1SUwESEREpJ4mJ56/+3iWzJoEhLU1OI8VRARIRESkntv0Xlr/XjjA5iVyJCpCIiEg5yMs+Q5L3QQCskQ+anEauRAVIRESkHHyX9CFZXhB41kLnW4eYHUeuQAVIRESkHNg2zgUgJi9My9+rABUgERGRcmA7lQpo+XtVoQIkIiJyldL3ppJa+ywWA2K0/L1KUAESERG5SokJ0wDonFmT+qGtTU4jJaECJCIicpVs+5IAsAaEm5xESkoFSERE5Cqcy80myevC8vduWv1VVagAiYiIXIXvln/ICW+DemctRNyq8/9UFSpAIiIiVyHxwvL36LxQ3D08TU4jJaUCJCIichVsJ78HwNpcy9+rEtML0IwZM2jatCne3t6Eh4ezZs2aYsevWrWK8PBwvL29adasGTNnznR6PD4+HovFUuCWnZ1dkW9DRERcUMYv2/i+9lkAoq1PmpxGSsPUArRgwQLGjh3L+PHj2bJlCz179sRqtZKWllbo+H379tGvXz969uzJli1b+Nvf/sbo0aNZuHCh0zh/f3/S09Odbt7e3tfiLYmIiAtZZpsOQPgJX4KatjM5jZSGqefqnjJlCsOGDWP48OEATJ06lWXLlvHOO+8wceLEAuNnzpxJaGgoU6dOBaB169Zs2rSJ119/nUGDBjnGWSwWgoODS5wjJyeHnJwcx/2srKwyviMREXEltr3LIACs/p3MjiKlZNoRoNzcXDZv3kx0dLTT9ujoaNavX1/oc5KTkwuMj4mJYdOmTeTl5Tm2nTp1irCwMBo3bsztt9/Oli1bis0yceJEAgICHLeQkJAyvisREXEV53KzWe71GwDWrg+YnEZKy7QCdOTIEfLz8wkKCnLaHhQUREZGRqHPycjIKHT8uXPnOHLkCACtWrUiPj6eJUuWMG/ePLy9venRowd79uwpMktcXByZmZmO24EDB67y3YmISHW3ccVHHPc2qJNtoWvUw2bHkVIy/XK1FovF6b5hGAW2XWn8pdu7detGt27dHI/36NGDTp06MX36dKZNm1boPr28vPDy8ipTfhERcU2JGz4BN4jOaazl71WQaUeAAgMDcXd3L3C05/DhwwWO8lwUHBxc6PgaNWpQr169Qp/j5uZGREREsUeARERESsuWtRkAa/MYk5NIWZhWgDw9PQkPDycpKclpe1JSEt27dy/0OZGRkQXGL1++nM6dO+Ph4VHocwzDIDU1lYYNG5ZPcBERcXmH929nU+0zAMRYR5mcRsrC1GXw48aN4/333+fDDz9k586dPPXUU6SlpTFy5Ejg/NycBx/832nFR44cya+//sq4cePYuXMnH374IR988AFPP/20Y8yrr77KsmXL+OWXX0hNTWXYsGGkpqY69ikiInK1liWcX/7e8YQPwc3am5xGysLUOUCDBw/m6NGjTJgwgfT0dNq1a0dCQgJhYWEApKenO50TqGnTpiQkJPDUU0/x9ttv06hRI6ZNm+a0BP7EiRM8+uijZGRkEBAQQMeOHVm9ejVdunS55u9PRESqJ9veRPAHa62OZkeRMrIYF2cRi0NWVhYBAQFkZmbi7+9vdhwREalE8vNyafCiN8d8DNaEv83Ntz9udiS5oDS/v02/FIaIiEhVkvL1HI75GNTOttAt+hGz40gZqQCJiIiUgu27/wAQlXMdNTx1maWqSgVIRESkFBIzLyx/bxplchK5GipAIiIiJfRH2k5SAk4DENtvtMlp5GqoAImIiJTQ8oTpGBbocMKbhs1vMjuOXAUVIBERkRKy7bEBYPW7ydwgctVUgERERErAnn+OZR6/AmCNuM/kNHK1VIBERERKYNPX/+GIj4F/DkRGDzM7jlwlFSAREZESsCXPASAq+zo8vH1NTiNXSwVIRESkBBJPbALA2uQ2k5NIeVABEhERuYKjv+1hQ8ApAGJjnzQ5jZQHFSAREZErWP7VNAwLtD/hzXU3dDY7jpQDFSAREZErsO1JACC2ZnuTk0h5UQESEREphj3/HInu+wCwdv6LyWmkvKgAiYiIFOP7b+fyh69BrRzoEfuo2XGknKgAiYiIFMO2/iMAbjvbUMvfqxEVIBERkWLYjm0EwBqm5e/ViQqQiIhIEY4d2utY/m61jjI5jZQnFSAREZEiJH01DbsbtMv0onHLCLPjSDlSARIRESmCbfdXAFh9tPy9ulEBEhERKcSly99jwwebnEbKmwqQiIhIIVJXLeB3Xzt+uXCz9TGz40g5UwESEREphG1dPAC3ngnG08fP3DBS7lSARERECmE7ugEAa+gtJieRiqACJCIicpnj6ftIDjgJgDVGV3+vjlSARERELrMiYTp2N2iT6UVom0iz40gFUAESERG5jG3XUgCsPu1MTiIVRQVIRETkEobdTqJlLwDWTlr+Xl2pAImIiFxi65rPSK9pp2Yu3Byr5e/VlQqQiIjIJWxrPgTgljNBeNX0NzmNVBQVIBERkUvYjnwHgLVxX5OTSEVSARIREbngxO+/sj4gCwBrzBMmp5GKpAIkIiJywddfvUW+G7TK9KRJu5vNjiMVSAVIRETkAtvOJQBYvbX8vbpTARIREeHC8nd+BsDa8R6T00hFUwESEREBkhPf46CfHd9c6GkdaXYcqWA1zA4gIiJyrR1P38emtQtI2fUNKcd+ZKPH7xyqaQeg75kGePvVNjegVDgVIBERqdbOZB5hy9rPSPlxOSmHt5DCIfb45/1vQO3zf7jZ4cYsb+Jue8WMmHKNqQCJiEi1kZd9hu0bviRly1dsPLiRlHO/8qN/NvkXJ3xccl7D5lk1iKAREfVvoku7GDr2uJuadRqYkluuPRUgERGpkgy7nZ+3fM3GlM9J+TWZlLN7+d7vJNkeFwb4/W9s8Gk3uuQ1IKJOWyJa9qVzj3up17iFKbmlclABEhGRKuHgT5tISV7Ixp9XkXJyN5t8jnPC2zj/oPeFGxCQDZ3P1iGiVku6NO9FROTdXNciHIub1v3I/6gAiYhIpXPs0F42rf2UlN0XJykfJv3CJGVqAHXO/9XrHHQ85UcX7+ZEhEYS0flPtOgUhZu7fr1J8fQTIiIipro4SXnjD8tI+SOVFA7xcxGTlNtleRNRI5Qu10UQcVN/2nUbgIe3rym5pWpTARIRkWsmL/sMP363mJTUr0g5uImN535lezGTlLtwHRH1byKiXbQmKUu5UgESEZEKYc8/d2GS8hekpJ2fpLzF71Shk5QbnnYjIq8BXeq2I6LlLXS++V7qNmpuSm5xDSpAIiJy1Qy7nYN7NpOS/F9S9q5h48ldbPI5TuaFicmXT1KOOFuXiFotiWjeky7d7+G6GzqbFV1clAqQiIiU2rFDe0lZM5+U3d+Scnw7Gz0Ok1HIJGXvvPOTlCN8mtMlrDsRnQdwfcdbNUlZTKefQBERKdbp44fZsu6/bPzxf5OU9/qf+9+A2uf/cHdMUg4j4rrOdOl4B2273qFJylIpqQCJiIhDXvYZflj/OSlbE0g5tImN+Wlsr5WNvZBJytdneRBBI7o06Hh+kvLN9+AbEGhKbpHSUgESEXFR9vxz7Pk+iY0pi89PUs75hdSaRU9S7pLXgIi67ejS6lY63zyYOg2bmpJbpDyoAImIuADDbue33SmkfLeQjXtXk3LqJzb5HifL68IAnws3oHa2hc5n69DFvxURzXsRETlIk5Sl2lEBEhGpho7+toeUtQtI+enCJGXPP/jd98IkZQ+cJil3OlXrf5OUI+7k+o636rIRUu2ZXoBmzJjBv/71L9LT02nbti1Tp06lZ8+eRY5ftWoV48aNY/v27TRq1Ihnn32WkSNHOo1ZuHAhL774Inv37qV58+a89tprDBw4sKLfioiIKU4fP8z3az8lZXsSGw9vIcUtnV9qFT1JuYtHEyIaRRDRsb8mKYvLMrUALViwgLFjxzJjxgx69OjBrFmzsFqt7Nixg9DQ0ALj9+3bR79+/RgxYgQff/wx69at4/HHH6d+/foMGjQIgOTkZAYPHsz//d//MXDgQD7//HPuvfde1q5dS9euXa/1WxQRKVe5Z0/xw/ovzk9STj8/SXlHrZz/TVIO+N/YFhcnKQd1IqJdDDf1GKRJyiIXWAzDMMx68a5du9KpUyfeeecdx7bWrVtz5513MnHixALjn3vuOZYsWcLOnTsd20aOHMnWrVtJTk4GYPDgwWRlZWGz2RxjYmNjqVOnDvPmzStRrqysLAICAsjMzMTf3//KTyihnNNZpO/bVm77k+IZdnv57MeobPspn39ly/PzOX8zsNvPYdgNp21Of9oL2XbJ3+32/EIeL9mfdvsl++Li9kvHXPg7BbfbjfzixzjuX77vS557cYzT2Mv35zzOfvn+i/kzM/ckm3J+IdXvNDmF/K9ro9NudMkLIqJuOyJa3aJJyuKSSvP727QjQLm5uWzevJnnn3/eaXt0dDTr168v9DnJyclER0c7bYuJieGDDz4gLy8PDw8PkpOTeeqppwqMmTp1apFZcnJyyMnJcdzPysoq5bspmS1rPiVyw4gK2beIVBOWy/686LJJyhFn6xDh34ou1/cmIvJuGrXodA1DilR9phWgI0eOkJ+fT1BQkNP2oKAgMjIyCn1ORkZGoePPnTvHkSNHaNiwYZFjitonwMSJE3n11VfL+E5Kzs3NHZ+8K4+rjCymHSe8Opf/DjF9P+X0OVa+PBYsgJtxPpvl4p9YLvn7xe2WQu+7Xf6Y03OvvN3twr2C4yyXjS34p9ul2yw49lRw/GXbLJZLnu+8zek5lqLvu1ncLnltS+HPsVjwdveiQ+POmqQsUk5MnwRtsTj/p9wwjALbrjT+8u2l3WdcXBzjxo1z3M/KyiIkJOTK4UupS/TDnIl+uNz3KyIiIqVjWgEKDAzE3d29wJGZw4cPFziCc1FwcHCh42vUqEG9evWKHVPUPgG8vLzw8vIq8nERERGpXkw7hurp6Ul4eDhJSUlO25OSkujevXuhz4mMjCwwfvny5XTu3BkPD49ixxS1TxEREXE9pn4FNm7cOIYMGULnzp2JjIzk3XffJS0tzXFen7i4OA4ePMicOXOA8yu+3nrrLcaNG8eIESNITk7mgw8+cFrdNWbMGHr16sWkSZMYMGAAixcvZsWKFaxdu9aU9ygiIiKVj6kFaPDgwRw9epQJEyaQnp5Ou3btSEhIICwsDID09HTS0tIc45s2bUpCQgJPPfUUb7/9No0aNWLatGmOcwABdO/enfnz5/PCCy/w4osv0rx5cxYsWKBzAImIiIiDqecBqqwq6jxAIiIiUnFK8/tb6yhFRETE5agAiYiIiMtRARIRERGXowIkIiIiLkcFSERERFyOCpCIiIi4HBUgERERcTkqQCIiIuJyVIBERETE5Zh6KYzK6uLJsbOyskxOIiIiIiV18fd2SS5yoQJUiJMnTwIQEhJichIREREprZMnTxIQEFDsGF0LrBB2u51Dhw5Rq1YtLBZLue47KyuLkJAQDhw4oOuMXYE+q5LTZ1Vy+qxKTp9V6ejzKrmK+qwMw+DkyZM0atQIN7fiZ/noCFAh3NzcaNy4cYW+hr+/v/4FKSF9ViWnz6rk9FmVnD6r0tHnVXIV8Vld6cjPRZoELSIiIi5HBUhERERcjgrQNebl5cXLL7+Ml5eX2VEqPX1WJafPquT0WZWcPqvS0edVcpXhs9IkaBEREXE5OgIkIiIiLkcFSERERFyOCpCIiIi4HBUgERERcTkqQJVATk4ON910ExaLhdTUVLPjVEp/+tOfCA0Nxdvbm4YNGzJkyBAOHTpkdqxKZ//+/QwbNoymTZvi4+ND8+bNefnll8nNzTU7WqX12muv0b17d3x9faldu7bZcSqVGTNm0LRpU7y9vQkPD2fNmjVmR6qUVq9ezR133EGjRo2wWCx88cUXZkeqlCZOnEhERAS1atWiQYMG3Hnnnezevdu0PCpAlcCzzz5Lo0aNzI5RqfXt25dPP/2U3bt3s3DhQvbu3cvdd99tdqxKZ9euXdjtdmbNmsX27dt54403mDlzJn/729/MjlZp5ebmcs899/DXv/7V7CiVyoIFCxg7dizjx49ny5Yt9OzZE6vVSlpamtnRKp3Tp0/ToUMH3nrrLbOjVGqrVq3iiSee4LvvviMpKYlz584RHR3N6dOnzQlkiKkSEhKMVq1aGdu3bzcAY8uWLWZHqhIWL15sWCwWIzc31+wold7kyZONpk2bmh2j0ps9e7YREBBgdoxKo0uXLsbIkSOdtrVq1cp4/vnnTUpUNQDG559/bnaMKuHw4cMGYKxatcqU19cRIBP9/vvvjBgxgv/85z/4+vqaHafKOHbsGJ988gndu3fHw8PD7DiVXmZmJnXr1jU7hlQhubm5bN68mejoaKft0dHRrF+/3qRUUt1kZmYCmPbfJxUgkxiGwdChQxk5ciSdO3c2O06V8Nxzz1GzZk3q1atHWloaixcvNjtSpbd3716mT5/OyJEjzY4iVciRI0fIz88nKCjIaXtQUBAZGRkmpZLqxDAMxo0bx80330y7du1MyaACVM5eeeUVLBZLsbdNmzYxffp0srKyiIuLMzuyaUr6WV30zDPPsGXLFpYvX467uzsPPvgghoucyLy0nxXAoUOHiI2N5Z577mH48OEmJTdHWT4vKchisTjdNwyjwDaRsnjyySfZtm0b8+bNMy2DLoVRzo4cOcKRI0eKHdOkSRP+/Oc/8+WXXzr9xyQ/Px93d3fuv/9+Pvroo4qOarqSflbe3t4Ftv/222+EhISwfv16IiMjKypipVHaz+rQoUP07duXrl27Eh8fj5uba/2/Tll+tuLj4xk7diwnTpyo4HSVX25uLr6+vnz22WcMHDjQsX3MmDGkpqayatUqE9NVbhaLhc8//5w777zT7CiV1qhRo/jiiy9YvXo1TZs2NS1HDdNeuZoKDAwkMDDwiuOmTZvG3//+d8f9Q4cOERMTw4IFC+jatWtFRqw0SvpZFeZib8/JySnPSJVWaT6rgwcP0rdvX8LDw5k9e7bLlR+4up8tAU9PT8LDw0lKSnIqQElJSQwYMMDEZFKVGYbBqFGj+Pzzz1m5cqWp5QdUgEwTGhrqdN/Pzw+A5s2b07hxYzMiVVobN25k48aN3HzzzdSpU4dffvmFl156iebNm7vE0Z/SOHToEH369CE0NJTXX3+dP/74w/FYcHCwickqr7S0NI4dO0ZaWhr5+fmOc3Fdf/31jn8vXdG4ceMYMmQInTt3JjIyknfffZe0tDTNJyvEqVOn+Pnnnx339+3bR2pqKnXr1i3w33pX9sQTTzB37lwWL15MrVq1HPPJAgIC8PHxufaBTFl7JgXs27dPy+CLsG3bNqNv375G3bp1DS8vL6NJkybGyJEjjd9++83saJXO7NmzDaDQmxTuoYceKvTz+vbbb82OZrq3337bCAsLMzw9PY1OnTqZtly5svv2228L/Rl66KGHzI5WqRT136bZs2ebkkdzgERERMTluN7kABEREXF5KkAiIiLiclSARERExOWoAImIiIjLUQESERERl6MCJCIiIi5HBUhERERcjgqQiIiIuBwVIBEpsT59+jB27FizYxTq6NGjNGjQgP379wOwcuVKLBZLhV/ctKyvEx8fT+3atUv1nIiICBYtWlSq54hI4VSARMQ06enp3HfffbRs2RI3N7ciy9XChQtp06YNXl5etGnThs8//7zAmIkTJ3LHHXfQpEmTig1tohdffJHnn38eu91udhSRKk8FSERMk5OTQ/369Rk/fjwdOnQodExycjKDBw9myJAhbN26lSFDhnDvvfeyYcMGx5izZ8/ywQcfMHz48GsV3RT9+/cnMzOTZcuWmR1FpMpTARKRMjl+/DgPPvggderUwdfXF6vVyp49e5zGvPfee4SEhODr68vAgQOZMmWK09c+TZo04c033+TBBx8kICCg0NeZOnUqUVFRxMXF0apVK+Li4rj11luZOnWqY4zNZqNGjRpERkYWmffo0aP85S9/oXHjxvj6+nLjjTcyb948pzF9+vRh1KhRjB07ljp16hAUFMS7777L6dOnefjhh6lVqxbNmzfHZrMV2P+6devo0KED3t7edO3alR9++MHp8fj4eEJDQx2fxdGjR50e37t3LwMGDCAoKAg/Pz8iIiJYsWKF0xh3d3f69etXILeIlJ4KkIiUydChQ9m0aRNLliwhOTkZwzDo168feXl5wPlCMHLkSMaMGUNqaipRUVG89tprpX6d5ORkoqOjnbbFxMSwfv16x/3Vq1fTuXPnYveTnZ1NeHg4S5cu5ccff+TRRx9lyJAhTkeSAD766CMCAwPZuHEjo0aN4q9//Sv33HMP3bt35/vvvycmJoYhQ4Zw5swZp+c988wzvP7666SkpNCgQQP+9Kc/OT6LDRs28Mgjj/D444+TmppK3759+fvf/+70/FOnTtGvXz9WrFjBli1biImJ4Y477iAtLc1pXJcuXVizZk3JPjwRKZop16AXkSqpd+/expgxY4yffvrJAIx169Y5Hjty5Ijh4+NjfPrpp4ZhGMbgwYON/v37Oz3//vvvNwICAord9+U8PDyMTz75xGnbJ598Ynh6ejruDxgwwHjkkUecxnz77bcGYBw/frzI99OvXz/j//2//+eU4eabb3bcP3funFGzZk1jyJAhjm3p6ekGYCQnJzu9zvz58x1jjh49avj4+BgLFiwwDMMw/vKXvxixsbFOrz148OAiP4uL2rRpY0yfPt1p2+LFiw03NzcjPz+/2OeKSPF0BEhESm3nzp3UqFGDrl27OrbVq1ePli1bsnPnTgB2795Nly5dnJ53+f2SslgsTvcNw3DadvbsWby9vYvdR35+Pq+99hrt27enXr16+Pn5sXz58gJHWNq3b+/4u7u7O/Xq1ePGG290bAsKCgLg8OHDTs+79Ou3unXrOn0WO3fuLPD13OX3T58+zbPPPkubNm2oXbs2fn5+7Nq1q0A+Hx8f7HY7OTk5xb5fESleDbMDiEjVYxhGkdsvFpPLS0pxzytOcHAwGRkZTtsOHz7sKCIAgYGBHD9+vNj9/Pvf/+aNN95g6tSp3HjjjdSsWZOxY8eSm5vrNM7Dw8PpvsVicdp28T2VZCXWpZ/FlTzzzDMsW7aM119/neuvvx4fHx/uvvvuAvmOHTuGr68vPj4+V9yniBRNR4BEpNTatGnDuXPnnObPHD16lJ9++onWrVsD0KpVKzZu3Oj0vE2bNpX6tSIjI0lKSnLatnz5crp37+6437FjR3bs2FHsftasWcOAAQN44IEH6NChA82aNSswaftqfPfdd46/Hz9+nJ9++olWrVoB5z+vSx+/fPzFfEOHDmXgwIHceOONBAcHO85pdKkff/yRTp06lVtuEVelAiQipdaiRQsGDBjAiBEjWLt2LVu3buWBBx7guuuuY8CAAQCMGjWKhIQEpkyZwp49e5g1axY2m63AUaHU1FRSU1M5deoUf/zxB6mpqU5lZsyYMSxfvpxJkyaxa9cuJk2axIoVK5zOGRQTE8P27duLPQp0/fXXk5SUxPr169m5cyePPfZYgSNLV2PChAl8/fXX/PjjjwwdOpTAwEDuvPNOAEaPHk1iYiKTJ0/mp59+4q233iIxMbFAvkWLFpGamsrWrVu57777Cj3KtGbNmgKTwkWk9FSARKRMZs+eTXh4OLfffjuRkZEYhkFCQoLj66IePXowc+ZMpkyZQocOHUhMTOSpp54qMFenY8eOdOzYkc2bNzN37lw6duxIv379HI93796d+fPnM3v2bNq3b098fDwLFixwmn9044030rlzZz799NMi87744ot06tSJmJgY+vTpQ3BwsKOglId//vOfjBkzhvDwcNLT01myZAmenp4AdOvWjffff5/p06dz0003sXz5cl544QWn57/xxhvUqVOH7t27c8cddxATE1PgSM/BgwdZv349Dz/8cLnlFnFVFqMsX8qLiJTBiBEj2LVrV4Us405ISODpp5/mxx9/xM2tev6/3TPPPENmZibvvvuu2VFEqjxNghaRCvP6668TFRVFzZo1sdlsfPTRR8yYMaNCXqtfv37s2bOHgwcPEhISUiGvYbYGDRrw9NNPmx1DpFrQESARqTD33nsvK1eu5OTJkzRr1oxRo0YxcuRIs2OJiKgAiYiIiOupnl+Ui4iIiBRDBUhERERcjgqQiIiIuBwVIBEREXE5KkAiIiLiclSARERExOWoAImIiIjLUQESERERl/P/AZaP27Xvhj1uAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ "import numpy as np\n", "import pandas as pd\n", @@ -789,9 +831,7 @@ { "cell_type": "markdown", "id": "64072a5f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The results here agree when we force **Scikit-Learn**'s Ridge function to include the first column in our design matrix.\n", "We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.\n", @@ -803,11 +843,135 @@ "cell_type": "code", "execution_count": 4, "id": "167020f5", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Beta values for own Ridge implementation\n", + "[ 3.43579948e-02 -5.43330971e-01 -3.10141413e-03 2.47116868e-01\n", + " 2.18613217e-01 1.02054837e-01 -4.25617658e-04 -5.90475506e-02\n", + " -7.68534263e-02 -6.68929213e-02 -4.24906604e-02 -1.40927184e-02\n", + " 1.11482289e-02 2.88529063e-02 3.67047975e-02 3.38135733e-02\n", + " 2.02198703e-02 -3.46383926e-03 -3.63025821e-02]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[ 3.43579948e-02 -5.43330971e-01 -3.10141413e-03 2.47116868e-01\n", + " 2.18613217e-01 1.02054837e-01 -4.25617655e-04 -5.90475506e-02\n", + " -7.68534263e-02 -6.68929213e-02 -4.24906604e-02 -1.40927184e-02\n", + " 1.11482289e-02 2.88529063e-02 3.67047975e-02 3.38135733e-02\n", + " 2.02198702e-02 -3.46383925e-03 -3.63025821e-02]\n", + "Intercept from own implementation:\n", + "1.0330308045187757\n", + "Intercept from Scikit-Learn Ridge implementation\n", + "1.0330308045183194\n", + "MSE values for own Ridge implementation\n", + "3.139255958997547e-06\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "3.1392559585020426e-06\n", + "Beta values for own Ridge implementation\n", + "[-0.05807125 -0.29822833 -0.08551306 0.08156108 0.13679863 0.12333649\n", + " 0.08251519 0.03815288 0.00111756 -0.02498832 -0.04010697 -0.04566964\n", + " -0.04355837 -0.03562355 -0.02348765 -0.00848904 0.00831018 0.0260906\n", + " 0.04423486]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[-0.05807125 -0.29822833 -0.08551306 0.08156108 0.13679863 0.12333649\n", + " 0.08251519 0.03815288 0.00111756 -0.02498832 -0.04010697 -0.04566964\n", + " -0.04355837 -0.03562355 -0.02348765 -0.00848904 0.00831018 0.0260906\n", + " 0.04423486]\n", + "Intercept from own implementation:\n", + "1.0411487294305088\n", + "Intercept from Scikit-Learn Ridge implementation\n", + "1.0411487294305226\n", + "MSE values for own Ridge implementation\n", + "1.9601304850035702e-05\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "1.9601304850073734e-05\n", + "Beta values for own Ridge implementation\n", + "[-0.1416398 -0.14021063 -0.05383795 0.01367553 0.04784395 0.05796251\n", + " 0.05447415 0.044613 0.03267527 0.02098261 0.01066519 0.00217499\n", + " -0.00440346 -0.00917248 -0.01231917 -0.01405935 -0.0146081 -0.01416528\n", + " -0.01290947]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[-0.1416398 -0.14021063 -0.05383795 0.01367553 0.04784395 0.05796251\n", + " 0.05447415 0.044613 0.03267527 0.02098261 0.01066519 0.00217499\n", + " -0.00440346 -0.00917248 -0.01231917 -0.01405935 -0.0146081 -0.01416528\n", + " -0.01290947]\n", + "Intercept from own implementation:\n", + "1.049556996627824\n", + "Intercept from Scikit-Learn Ridge implementation\n", + "1.0495569966278269\n", + "MSE values for own Ridge implementation\n", + "5.4959161509357395e-05\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "5.4959161509366685e-05\n", + "Beta values for own Ridge implementation\n", + "[-0.13535942 -0.08593216 -0.03568439 -0.0036367 0.01397146 0.02229529\n", + " 0.02503753 0.0245528 0.02228115 0.01908936 0.01549377 0.01179792\n", + " 0.00817631 0.00472512 0.00149311 -0.00149956 -0.00424967 -0.00676387\n", + " -0.00905423]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[-0.13535942 -0.08593216 -0.03568439 -0.0036367 0.01397146 0.02229529\n", + " 0.02503753 0.0245528 0.02228115 0.01908936 0.01549377 0.01179792\n", + " 0.00817631 0.00472512 0.00149311 -0.00149956 -0.00424967 -0.00676387\n", + " -0.00905423]\n", + "Intercept from own implementation:\n", + "1.039967668952797\n", + "Intercept from Scikit-Learn Ridge implementation\n", + "1.0399676689527975\n", + "MSE values for own Ridge implementation\n", + "7.571105947979344e-05\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "7.571105947979412e-05\n", + "Beta values for own Ridge implementation\n", + "[-0.05100875 -0.04063602 -0.02723445 -0.01713366 -0.0100706 -0.00517114\n", + " -0.00174276 0.00068734 0.00243186 0.00369758 0.00462287 0.0053018\n", + " 0.00579953 0.006162 0.00642221 0.00660427 0.00672607 0.0068011\n", + " 0.00683964]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[-0.05100875 -0.04063602 -0.02723445 -0.01713366 -0.0100706 -0.00517114\n", + " -0.00174276 0.00068734 0.00243186 0.00369758 0.00462287 0.0053018\n", + " 0.00579953 0.006162 0.00642221 0.00660427 0.00672607 0.0068011\n", + " 0.00683964]\n", + "Intercept from own implementation:\n", + "0.999955585168597\n", + "Intercept from Scikit-Learn Ridge implementation\n", + "0.999955585168597\n", + "MSE values for own Ridge implementation\n", + "0.0007698473260556343\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "0.0007698473260556325\n", + "Beta values for own Ridge implementation\n", + "[-0.00834567 -0.00803064 -0.00673407 -0.00554552 -0.00458878 -0.0038335\n", + " -0.00323332 -0.00274989 -0.0023548 -0.00202756 -0.00175331 -0.00152117\n", + " -0.001323 -0.0011526 -0.00100519 -0.00087697 -0.00076495 -0.00066668\n", + " -0.00058016]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[-0.00834567 -0.00803064 -0.00673407 -0.00554552 -0.00458878 -0.0038335\n", + " -0.00323332 -0.00274989 -0.0023548 -0.00202756 -0.00175331 -0.00152117\n", + " -0.001323 -0.0011526 -0.00100519 -0.00087697 -0.00076495 -0.00066668\n", + " -0.00058016]\n", + "Intercept from own implementation:\n", + "0.9637117593816477\n", + "Intercept from Scikit-Learn Ridge implementation\n", + "0.9637117593816477\n", + "MSE values for own Ridge implementation\n", + "0.0023813163025848865\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "0.002381316302584885\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlIAAAG2CAYAAAC04mh6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABjN0lEQVR4nO3deVhU5fsG8HsYZmEHRRYTEM1UNE3QEMqtElwyt5RKSTNJslS0TNFcc89sc8sylzRX3Co1sQQX0NSv4EZqimEJIS4zrMMy7+8Pfk6NLDIIHJb7c11zxZx5zjn3HMl5fM+Z98iEEAJEREREZDIzqQMQERER1VRspIiIiIjKiY0UERERUTmxkSIiIiIqJzZSREREROXERoqIiIionNhIEREREZUTGykiIiKicmIjRURERFRObKSIiIiIyknyRmr58uXw9PSEWq2Gj48Pjhw5Ump9dHQ0fHx8oFar0aRJE6xcubJITUREBLy8vKBSqeDl5YWdO3cavT5//nx06NABNjY2cHJyQr9+/XDp0iWjmuHDh0Mmkxk9Onbs+OhvmIiIiGoNSRupLVu2ICwsDFOnTsWZM2fQqVMn9OzZE0lJScXWJyYmolevXujUqRPOnDmDKVOmYOzYsYiIiDDUxMbGIigoCMHBwYiPj0dwcDAGDx6MEydOGGqio6Pxzjvv4Pjx44iMjER+fj4CAgKQmZlptL8ePXogOTnZ8Ni7d2/lHAgiIiKqkWRS3rTY19cX3t7eWLFihWFZy5Yt0a9fP8yfP79I/aRJk7Bnzx4kJCQYloWGhiI+Ph6xsbEAgKCgIGi1Wuzbt89Q06NHDzg4OGDTpk3F5rh16xacnJwQHR2Nzp07Aygckbp37x527dpVEW+ViIiIaiFzqXacm5uL06dPY/LkyUbLAwICEBMTU+w6sbGxCAgIMFoWGBiI1atXIy8vDwqFArGxsRg/fnyRms8++6zELBqNBgBQr149o+VRUVFwcnKCvb09unTpgrlz58LJyanE7eh0Ouh0OsNzvV6PO3fuoH79+pDJZCWuR0RERNWHEALp6elo2LAhzMxKP3knWSOVlpaGgoICODs7Gy13dnZGSkpKseukpKQUW5+fn4+0tDS4urqWWFPSNoUQmDBhAp599lm0bt3asLxnz54YNGgQPDw8kJiYiGnTpuG5557D6dOnoVKpit3W/PnzMWvWrIe+dyIiIqr+bty4gUaNGpVaI1kjdd+DIzVCiFJHb4qrf3C5Kdt89913cfbsWRw9etRoeVBQkOHn1q1bo3379vDw8MBPP/2EAQMGFLut8PBwTJgwwfBco9HA3d0dN27cgK2tbYnviYiIiKoPrVYLNzc32NjYPLRWskbK0dERcrm8yEhRampqkRGl+1xcXIqtNzc3R/369UutKW6bY8aMwZ49e3D48OGHdpyurq7w8PDAlStXSqxRqVTFjlbZ2tqykSIiIqphynJZjmTf2lMqlfDx8UFkZKTR8sjISPj7+xe7jp+fX5H6AwcOoH379lAoFKXW/HebQgi8++672LFjB3799Vd4eno+NO/t27dx48YNuLq6lun9ERERUR0gJLR582ahUCjE6tWrxcWLF0VYWJiwsrIS169fF0IIMXnyZBEcHGyov3btmrC0tBTjx48XFy9eFKtXrxYKhUJs377dUHPs2DEhl8vFggULREJCgliwYIEwNzcXx48fN9S8/fbbws7OTkRFRYnk5GTDIysrSwghRHp6unjvvfdETEyMSExMFIcOHRJ+fn7iscceE1qttszvT6PRCABCo9E86qEiIiKiKmLK57ekjZQQQixbtkx4eHgIpVIpvL29RXR0tOG1YcOGiS5duhjVR0VFiXbt2gmlUikaN24sVqxYUWSb27ZtE82bNxcKhUK0aNFCREREGL0OoNjHmjVrhBBCZGVliYCAANGgQQOhUCiEu7u7GDZsmEhKSjLpvbGRIiIiqnlM+fyWdB6p2k6r1cLOzg4ajabUa6QKCgqQl5dXhcmIqpZSqXzoV4iJiKqLsn5+A9XgW3t1mRACKSkpuHfvntRRiCqVmZkZPD09oVQqpY5CRFSh2EhJ6H4T5eTkBEtLS07aSbWSXq/HzZs3kZycDHd3d/6eE1GtwkZKIgUFBYYm6v7UDUS1VYMGDXDz5k3k5+cbvmFLRFQb8KIFidy/JsrS0lLiJESV7/4pvYKCAomTEBFVLDZSEuNpDqoL+HtORLUVGykiIiKicmIjRVSJrl+/DplMhri4uBJroqKiIJPJ+O1NIqIaiI0UmWT48OGQyWQIDQ0t8tro0aMhk8kwfPhww7LU1FSMGjUK7u7uUKlUcHFxQWBgIGJjYw01jRs3hkwmK/JYsGBBVbylcuvatashq1KpRNOmTREeHg6dTmeocXNzQ3JyMlq3bi1h0pKP8f1H165dy73trl27IiwsrMKyEhHVJPzWHpnMzc0NmzdvxqeffgoLCwsAQE5ODjZt2gR3d3ej2oEDByIvLw/r1q1DkyZN8M8//+CXX37BnTt3jOpmz56NkJAQo2Vlueu21EJCQjB79mzk5ubi5MmTeOONNwAA8+fPBwDI5XK4uLhIGREAcPLkScOF3jExMRg4cCAuXbpkmGiO8zsREZUPR6TIZN7e3nB3d8eOHTsMy3bs2AE3Nze0a9fOsOzevXs4evQoFi5ciG7dusHDwwNPP/00wsPD0bt3b6Nt2tjYwMXFxehhZWVVYoa7d+/i9ddfh4ODAywtLdGzZ09cuXIFQOFEpw0aNEBERISh/qmnnoKTk5PheWxsLBQKBTIyMgAUXgz9zTffoH///rC0tESzZs2wZ8+ehx4LS0tLuLi4wN3dHQMHDkT37t1x4MABw+vFndrbu3cvnnjiCVhYWKBbt264fv16ke1+/fXXcHNzg6WlJfr3748lS5bA3t7eqOaHH36Aj48P1Go1mjRpglmzZiE/P7/YnA0aNDAc13r16gEAnJycDMt+//13dO7cGRYWFnBzc8PYsWORmZlpWH/58uVo1qwZ1Go1nJ2d8fLLLwMoHKGMjo7G559/bhjdKu79EBHVVmykqqHMzJIfOTllr83Ofnhteb3xxhtYs2aN4fm3336LESNGGNVYW1vD2toau3btMjrdVRGGDx+OU6dOYc+ePYiNjYUQAr169UJeXh5kMhk6d+6MqKgoAIVN18WLF5GXl4eLFy8CKLwuycfHB9bW1oZtzpo1C4MHD8bZs2fRq1cvDBkypMjIWWni4+Nx7NixUudJunHjBgYMGIBevXohLi4OI0eOxOTJk41qjh07htDQUIwbNw5xcXHo3r075s6da1Tz888/Y+jQoRg7diwuXryIr776CmvXri1SVxbnzp1DYGAgBgwYgLNnz2LLli04evQo3n33XQDAqVOnMHbsWMyePRuXLl3C/v370blzZwDA559/Dj8/P4SEhCA5ORnJyclwc3MzOQMRkSnyC/RoM/ldfBzxC/IL9NKGqdzb/tVtpd30MDs7W1y8eFFkZ2cXeQ0o+dGrl3GtpWXJtQ/c71k4OhatMdWwYcNE3759xa1bt4RKpRKJiYni+vXrQq1Wi1u3bom+ffuKYcOGGeq3b98uHBwchFqtFv7+/iI8PFzEx8cbbfP+TautrKyMHocOHSo2w+XLlwUAcezYMcOytLQ0YWFhIbZu3SqEEOKLL74QrVu3FkIIsWvXLtG+fXsxYMAAsWzZMiGEEAEBAWLSpEmG9QGIDz/80PA8IyNDyGQysW/fvhKPRZcuXYRCoRBWVlZCqVQKAMLMzExs377dUJOYmCgAiDNnzgghhAgPDxctW7YUer3eUDNp0iQBQNy9e1cIIURQUJDo3bu30b6GDBki7OzsDM87deok5s2bZ1Tz3XffCVdX1xLz3nfo0CGj/QUHB4u33nrLqObIkSPCzMxMZGdni4iICGFrayu0Wm2Jx2HcuHGl7rO033ciIlPN33pAYCYEJtuJW/cyK3z7pty0mCNSVC6Ojo7o3bs31q1bhzVr1qB3795wdHQsUjdw4EDcvHkTe/bsQWBgIKKiouDt7Y21a9ca1U2cOBFxcXFGD19f32L3nZCQAHNzc6PX69evj+bNmyMhIQFA4QXQFy5cQFpaGqKjo9G1a1d07doV0dHRyM/PR0xMDLp06WK03TZt2hh+trKygo2NDVJTU0s9DkOGDEFcXBxiY2MxePBgjBgxAgMHDiyxPiEhAR07djSaV8nPz8+o5tKlS3j66aeNlj34/PTp05g9e7Zh1M/a2towKpSVlVVq5gedPn0aa9euNdpWYGAg9Ho9EhMT0b17d3h4eKBJkyYIDg7Gxo0bTd4HEVFFWhb7NQCgtX4IHO2kndiaF5tXQ/9/2U6x5HLj56V9zps90CZX9KUrI0aMMJz+WbZsWYl1arUa3bt3R/fu3TF9+nSMHDkSM2bMMPp2n6OjIx5//PEy7VcIUeLy+w1K69atUb9+fURHRyM6OhqzZ8+Gm5sb5s6di5MnTyI7OxvPPvus0foPnpKTyWTQ60sfMrazszPk3rBhA1q1aoXVq1fjzTffNCl7Se+jpPX0ej1mzZqFAQMGFFlfrVY/dB8PbmvUqFEYO3Zskdfc3d2hVCrxv//9D1FRUThw4ACmT5+OmTNn4uTJk0Wu2yIiqmwXrqfiL+tdAIBpvd+SNgzYSFVLpVxjXWW1ZdGjRw/k5uYCAAIDA8u8npeXF3bt2lXu/Xp5eSE/Px8nTpyAv78/AOD27du4fPkyWrZsCQCG66R2796N8+fPo1OnTrCxsUFeXh5WrlwJb2/vCv9WoEKhwJQpUxAeHo5XX3212Nv/FPfejx8/bvS8RYsW+O2334yWnTp1yui5t7c3Ll26VObmszTe3t64cOFCqdsyNzfHCy+8gBdeeAEzZsyAvb09fv31VwwYMABKpZK3fiGiKvPB9+sAeR6s7nXA4M5tpY7Di82p/ORyORISEpCQkAD5g0NlKGxunnvuOWzYsAFnz55FYmIitm3bhkWLFqFv375Gtenp6UhJSTF6aLXaYvfbrFkz9O3bFyEhITh69Cji4+MxdOhQPPbYY0bb7dq1K77//nu0adMGtra2huZq48aNjzRvUmlee+01yGQyLF++vNjXQ0NDcfXqVUyYMAGXLl3C999/X+Q055gxY7B3714sWbIEV65cwVdffYV9+/YZjVJNnz4d69evx8yZM3HhwgUkJCRgy5Yt+PDDD03OPGnSJMTGxuKdd95BXFwcrly5gj179mDMmDEAgB9//BFffPEF4uLi8Oeff2L9+vXQ6/Vo3rw5gMI5qk6cOIHr168jLS3toaN4RETlpdcLRN7+BgAwwCPkIdVVg40UPRJbW1vDXEQPsra2hq+vLz799FN07twZrVu3xrRp0xASEoKlS5ca1U6fPh2urq5Gjw8++KDE/a5ZswY+Pj548cUX4efnByEE9u7da3R6rlu3bigoKDBqmrp06YKCgoIi10dVFKVSiXfffReLFi0yTK3wX+7u7oiIiMAPP/yAtm3bYuXKlZg3b55RzTPPPIOVK1diyZIlaNu2Lfbv34/x48cbnbILDAzEjz/+iMjISHTo0AEdO3bEkiVL4OHhYXLmNm3aIDo6GleuXEGnTp3Qrl07TJs2Da6urgAAe3t77NixA8899xxatmyJlStXYtOmTWjVqhUA4P3334dcLoeXlxcaNGiApKQkkzMQEZXFlz8cRp7tZSDXGguCX5E6DgBAJspy0QaVi1arhZ2dHTQaTZFmIycnB4mJifD09DT5mhaqe0JCQvD777/jyJEjUkcpF/6+E1FFmL1pH+aenIAm5p2QsGhVpe2ntM/vB/EaKaJqaPHixejevTusrKywb98+rFu3rsTThUREdcX0V3viw6AeSNNUn28Os5EiqoZ+++03LFq0COnp6WjSpAm++OILjBw5UupYRESSMzOTwcmhgr899QjYSBFVQ1u3bpU6AhFRtaHXC0xauwPhL/dCPVsLqeMY4cXmREREVK19vT8Wi2+8jAYfNUNWTp7UcYywkSIiIqJqbfGvhTOZNxHdYaku+X6mUmAjRURERNXWn//cwx/qLQCAic9LP5P5g9hIERERUbU1acP3gCIbKk0rjAzsKHWcIthIERERUbWk1wvs/qtwvqheLiEwM5M9ZI2qx0aKiIiIqqXvfjmFHPt4IF+Fj4cGSx2nWGykiEwQFRUFmUyGe/fulVizdu1a2NvbV1kmIqLaavupwwCAxlkvo2nDehKnKR4bKTLJ8OHDIZPJEBoaWuS10aNHQyaTYfjw4YZlqampGDVqFNzd3aFSqeDi4oLAwEDExsYaaho3bgyZTFbksWDBghJzXLt2Da+++ioaNmwItVqNRo0aoW/fvrh8+bKhRiaTYdeuXWV+b//NYWFhgRYtWuDjjz/Gf++i5O/vj+TkZNjZ2ZV5u5WhuOP138d//wxM1bhxY3z22WcVlpWIqLx+CH8PB1+6glWvzZA6Sok4ISeZzM3NDZs3b8ann34KC4vCidFycnKwadMmuLu7G9UOHDgQeXl5WLduHZo0aYJ//vkHv/zyC+7cuWNUN3v2bISEGN/J28bGptj95+bmonv37mjRogV27NgBV1dX/PXXX9i7dy80Gs0jvbf7OXJycnDw4EG8/fbbsLW1xahRowAU3pTYxcXlkfZREZKTkw0/b9myBdOnT8elS5cMy+7/uRAR1XTPt3tc6gil4ogUmczb2xvu7u7YsWOHYdmOHTvg5uaGdu3aGZbdu3cPR48excKFC9GtWzd4eHjg6aefRnh4OHr37m20TRsbG7i4uBg9rKyKvwXAxYsXce3aNSxfvhwdO3aEh4cHnnnmGcydOxcdOnR4pPd2P0fjxo0xcuRItGnTBgcOHDC8XtypvbVr18Ld3R2Wlpbo378/bt++XWS7c+bMgZOTE2xsbDBy5EhMnjwZTz31lFHNmjVr0LJlS6jVarRo0aLUe+v99zjZ2dlBJpMZLTt8+DB8fHygVqvRpEkTzJo1C/n5+Yb1Z86caRglbNiwIcaOHQsA6Nq1K/7880+MHz/eMLpFRCSFSzfSpI5QJmykqqHM3MwSHzn5OWWuzc7Lfmhteb3xxhtYs2aN4fm3336LESNGGNVYW1vD2toau3btgk6nK/e+HtSgQQOYmZlh+/btKCgoqLDt/pcQAlFRUUhISIBCUfLkbydOnMCIESMwevRoxMXFoVu3bpgzZ45RzcaNGzF37lwsXLgQp0+fhru7O1asWGFU8/XXX2Pq1KmYO3cuEhISMG/ePEybNg3r1q0zOfvPP/+MoUOHYuzYsbh48SK++uorrF27FnPnzgUAbN++HZ9++im++uorXLlyBbt27cKTTz4JoLAhbtSoEWbPno3k5GSjkS8ioqqyJToOLb52hcd7r0KvFw9fQUqCKo1GoxEAhEajKfJadna2uHjxosjOzi7yGmaixEevjb2Mai3nWpZY22VNF6Nax0WORWpMNWzYMNG3b19x69YtoVKpRGJiorh+/bpQq9Xi1q1bom/fvmLYsGGG+u3btwsHBwehVquFv7+/CA8PF/Hx8Ubb9PDwEEqlUlhZWRk9Dh06VGKOpUuXCktLS2FjYyO6desmZs+eLa5evWp8HAGxc+fOMr+3/+ZQKBQCgFCr1eLYsWOGmkOHDgkA4u7du0IIIV599VXRo0cPo+0EBQUJOzs7w3NfX1/xzjvvGNU888wzom3btobnbm5u4vvvvzeq+eijj4Sfn99Dc69Zs8Zof506dRLz5s0zqvnuu++Eq6urEEKITz75RDzxxBMiNze32O15eHiITz/99KH7NUVpv+9ERA9q/cFogZkQbuODJNl/aZ/fD+KIFJWLo6MjevfujXXr1mHNmjXo3bs3HB0di9QNHDgQN2/exJ49exAYGIioqCh4e3tj7dq1RnUTJ05EXFyc0cPX17fE/b/zzjtISUnBhg0b4Ofnh23btqFVq1aIjIx8pPd1P0d0dDS6deuGqVOnwt/fv8T6hIQE+Pn5GS178PmlS5fw9NNPGy377/Nbt27hxo0bePPNNw2jeNbW1pgzZw6uXr1q8ns4ffo0Zs+ebbStkJAQJCcnIysrC4MGDUJ2djaaNGmCkJAQ7Ny50+i0HxGRlFLvZuK8fAMAYMwzIQ+plh4vNq+GMsIzSnxNbiY3ep76fmqJtWYy4z75+rjrj5TrQSNGjMC7774LAFi2bFmJdWq1Gt27d0f37t0xffp0jBw5EjNmzDD6ZpmjoyMef9y0CwptbGzw0ksv4aWXXsKcOXMQGBiIOXPmoHv37uV6P//N8fjjjyMiIgKPP/44OnbsiBdeeKHYeiHKNuT84LVG/11Pr9cDKDy992DzKJcb/3mXhV6vx6xZszBgwIAir6nVari5ueHSpUuIjIzEwYMHMXr0aHz88ceIjo4u9TQmEVFVmLJxG6DSwjy9Ccb36yZ1nIdiI1UNWSmLv8i6KmvLokePHsjNzQUABAYGlnk9Ly8vk6YlKAuZTIYWLVogJiamwrbp4OCAMWPG4P3338eZM2eKvfDay8sLx48fN1r24PPmzZvjt99+Q3Dwv5PJnTp1yvCzs7MzHnvsMVy7dg1Dhgx55Nze3t64dOlSqY2phYWFoQl955130KJFC5w7dw7e3t5QKpWVdu0ZEdHDbPljFeAAPG8fAnN59T9xxkaKyk0ulyMhIcHw84Nu376NQYMGYcSIEWjTpg1sbGxw6tQpLFq0CH379jWqTU9PR0pKitEyS0tL2NraFtluXFwcZsyYgeDgYHh5eUGpVCI6OhrffvstJk2aZFSbmJiIuLg4o2WPP/44rK2ty/Qe33nnHSxcuBARERF4+eWXi7w+duxY+Pv7Y9GiRejXrx8OHDiA/fv3G9WMGTMGISEhaN++Pfz9/bFlyxacPXsWTZo0MdTMnDkTY8eOha2tLXr27AmdTodTp07h7t27mDBhQpmy3jd9+nS8+OKLcHNzw6BBg2BmZoazZ8/i3LlzmDNnDtauXYuCggL4+vrC0tIS3333HSwsLODh4QGgcB6pw4cP45VXXoFKpSr2lC0RUWXYHXMBGQ6xQIE5FgUPlzpO2VT6FVt1WHkvNq/O7l9sXpL/Xmyek5MjJk+eLLy9vYWdnZ2wtLQUzZs3Fx9++KHIysoyrOPh4SEAFHmMGjWq2H3cunVLjB07VrRu3VpYW1sLGxsb8eSTT4rFixeLgoICQ11x2wRQ4kXsJV1kHRISIlq1aiUKCgqKXGwuhBCrV68WjRo1EhYWFqJPnz5i8eLFRhd/CyHE7NmzhaOjo7C2thYjRowQY8eOFR07djSq2bhxo3jqqaeEUqkUDg4OonPnzmLHjh0lHuv7HrzYXAgh9u/fL/z9/YWFhYWwtbUVTz/9tFi1apUQQoidO3cKX19fYWtrK6ysrETHjh3FwYMHDevGxsaKNm3aCJVKJSrqr4ia+vtORFWr3eQwgZkQrmH9Jc1hysXmMiHKeJEHmUyr1cLOzg4ajabIyEpOTg4SExPh6ekJtVotUUKSSvfu3eHi4oLvvvtO6ihVgr/vRFQWicl3MXHDBjzXsh1Gv/isZDlK+/x+EE/tEVWyrKwsrFy5EoGBgZDL5di0aRMOHjz4yN8wJCKqbTxdHbB94hipY5iEjRRRJZPJZNi7dy/mzJkDnU6H5s2bIyIiosRvAhIRUc3BRoqokllYWODgwYNSxyAiqrb2nbyEQd8PwytNQ/HNu8OljmOS6v+9QiIiIqrVpu/6Bpn2J/Dj1R0PL65m2EhJjNf6U13A33MiKok2U4fTBWsBACHe1X8m8wexkZLI/Rmks7KyJE5CVPnuT9xanpnaiah2m7FpN4RFGswyG2JqUE+p45iM10hJRC6Xw97eHqmphbd4sbS0LHbmbKKaTq/X49atW7C0tIS5Of/KISJj684XzmT+jMWbUCtr3t8RNS9xLeLi4gIAhmaKqLYyMzODu7s7/7FAREZ+jbuKuw6/AEKG+YPflDpOubCRkpBMJoOrqyucnJyQl5cndRyiSqNUKmFmxisJiMjYhxGrAXOgviYAz7TykDpOubCRqgbkcjmvHSEiojqnh1cnXP7tf3iz7Sipo5QbbxFTiUyZYp6IiIiqB1M+vznWTkRERFRObKSIiIioSsVeTMJzs2bj5KW/pI7yyNhIERERUZWavHU1DmEGAlcMlzrKI2MjRURERFUmJzcfRzNXAwCGeo2UOM2jYyNFREREVWbe1v3QW/8NWXZ9zBnSX+o4j4yNFBEREVWZVadXAQC8zYbB1kolcZpHx0aKiIiIqsSpy3/jH9ufAACz+9W8GxQXh40UERERVYlJm78FzPSwvdsJvZ5uIXWcCsGZzYmIiKhK5BbkAvmWeK35W1JHqTCc2bwScWZzIiIiY3/d0sLWUlWtr4+qUTObL1++HJ6enlCr1fDx8cGRI0dKrY+OjoaPjw/UajWaNGmClStXFqmJiIiAl5cXVCoVvLy8sHPnTqPX58+fjw4dOsDGxgZOTk7o168fLl26ZFQjhMDMmTPRsGFDWFhYoGvXrrhw4cKjv2EiIqI6rFED22rdRJlK0kZqy5YtCAsLw9SpU3HmzBl06tQJPXv2RFJSUrH1iYmJ6NWrFzp16oQzZ85gypQpGDt2LCIiIgw1sbGxCAoKQnBwMOLj4xEcHIzBgwfjxIkThpro6Gi88847OH78OCIjI5Gfn4+AgABkZmYaahYtWoQlS5Zg6dKlOHnyJFxcXNC9e3ekp6dX3gEhIiKqhS7dSMOGX05LHaNSSHpqz9fXF97e3lixYoVhWcuWLdGvXz/Mnz+/SP2kSZOwZ88eJCQkGJaFhoYiPj4esbGxAICgoCBotVrs27fPUNOjRw84ODhg06ZNxea4desWnJycEB0djc6dO0MIgYYNGyIsLAyTJk0CAOh0Ojg7O2PhwoUYNapsd6nmqT0iIiKg59yF2J8/Gc0y3sDlj7+VOs5D1YhTe7m5uTh9+jQCAgKMlgcEBCAmJqbYdWJjY4vUBwYG4tSpU8jLyyu1pqRtAoBGowEA1KtXD0DhyFdKSorRdlQqFbp06VLqdnQ6HbRardGDiIioLssv0OPgna8BAM+4PStxmoonWSOVlpaGgoICODs7Gy13dnZGSkpKseukpKQUW5+fn4+0tLRSa0raphACEyZMwLPPPovWrVsbtnF/vbJuByi89srOzs7wcHNzK7GWiIioLvh8dxTyba8COhssDA6SOk6Fk/xic5lMZvRcCFFk2cPqH1xuyjbfffddnD17ttjTfqZmCw8Ph0ajMTxu3LhRYi0REVFd8PnRwpnMvQqGwMnBSuI0FU+yeaQcHR0hl8uLjPCkpqYWGQm6z8XFpdh6c3Nz1K9fv9Sa4rY5ZswY7NmzB4cPH0ajRo2M9gMUjky5urqWKRtQePpPpao930QgIiJ6FAlJt3DDqvCb8x/2rD1zR/2XZCNSSqUSPj4+iIyMNFoeGRkJf3//Ytfx8/MrUn/gwAG0b98eCoWi1Jr/blMIgXfffRc7duzAr7/+Ck9PT6N6T09PuLi4GG0nNzcX0dHRJWYjIiIiYx9sXA+Y58Lyng9e7dpO6jiVQtKZzSdMmIDg4GC0b98efn5+WLVqFZKSkhAaGgqg8FTZ33//jfXr1wMo/Ibe0qVLMWHCBISEhCA2NharV682Oi03btw4dO7cGQsXLkTfvn2xe/duHDx4EEePHjXUvPPOO/j++++xe/du2NjYGEaw7OzsYGFhAZlMhrCwMMybNw/NmjVDs2bNMG/ePFhaWuK1116rwiNERERUcx37Zy/gAPT3qJ2jUQAAIbFly5YJDw8PoVQqhbe3t4iOjja8NmzYMNGlSxej+qioKNGuXTuhVCpF48aNxYoVK4psc9u2baJ58+ZCoVCIFi1aiIiICKPXART7WLNmjaFGr9eLGTNmCBcXF6FSqUTnzp3FuXPnTHpvGo1GABAajcak9YiIiGqDbF2e+HD9HvF3mlbqKCYx5fObt4ipRJxHioiIqOapEfNIERERUe10LyMHObn5UseoEmykiIiIqEK9uXwVrD70wNDPVkkdpdKxkSIiIqIKo9cL/JTyNfRWN6HLy5U6TqVjI0VEREQV5tsDJ6CzOw/kqbH49aFSx6l0bKSIiIiowiz6pfB0XpOcwfBwtpc2TBVgI0VEREQVIilVgyvKLQCA97vV4rmj/oONFBEREVWISd99DyizoNS0xKiedeNOIGykiIiIqELs/utrAEAv57dgZiaTOE3VYCNFREREFWLVi1/DKysUi4YESx2lynBm80rEmc2JiIhqHs5sTkRERFQF2EgRERHRI3l7xQY8MXEENv76P6mjVDk2UkRERPRIvrv8Ja5Yr8HG2INSR6lybKSIiIio3LYejkem/W9AgQIfDxkudZwqx0aKiIiIym3O3sIpDx7L6ItWjZ0kTlP12EgRERFRuaRpsnBOtgEA8K5f3ZjJ/EFspIiIiKhcwjdsA9QamKd74v0Bz0sdRxJspIiIiKhctlwpPK3XzW4kzOV1s6Wom++aiIiIHkl+gR4d6gXCXNsUi157Q+o4kuHM5pWIM5sTEVFtp9eLWndfPc5sTkRERFWitjVRpmIjRURERCZZsvMQJn4bgaycPKmjSI6NFBEREZnko+jZWHzjZfRfvFjqKJJjI0VERERl9vOpy7jnEAXozTB74FCp40iOjRQRERGV2bSd3wAAGmh7wLelm8RppMdGioiIiMokIzsXp/LXAgBGtqubM5k/iI0UERERlcn073dDWN6CWaYrpr/SW+o41QIbKSIiIiqTdecKZzL3U4+AWmkucZrqgY0UERERPZQ2U4dcZABChnkvvyl1nGqD7SQRERE9lK2VCumfxSAq/ho6t/GUOk61wREpIiIiKrOubZtIHaFaYSNFREREpfo17ioSk+9KHaNaYiNFREREpXpl3Vg0WdYQY1dtljpKtcNrpIiIiKhEsReTcMtuHyAT6OPjI3WcaocjUkRERFSi8K3fAjIB+7vd0N2nmdRxqh02UkRERFSs3LwCHM1cDQAY2jJE4jTVExspIiIiKta8rftRYP0XZNn18NGQ/lLHqZbYSBEREVGxvjpVOJN5O7NhsLdWS5ymemIjRUREREX8+c89pFj9DACY1Zen9UrCRoqIiIiK8HC2R/yIRIx0XIsXfVtKHafakgkhhNQhaiutVgs7OztoNBrY2tpKHYeIiIjKwJTPb45IERERkZGsnDypI9QYbKSIiIjISPMpQ1Av7HmsP3hK6ijVHmc2JyIiIoPzif/gL+udgDwfliql1HGqPY5IERERkcEH368D5PmwuueLlzu1kTpOtcdGioiIiAAAer3AwTuFc0e97MkpD8qCjRQREREBAD7bHYU82z8AnQ0WDA2SOk6NwEaKiIiIAABfHC0cjWqZ/xpc6llLnKZmYCNFREREuHQjDX9aRgAApgTytF5ZsZEiIiIiODtYY5TrN2ieMRJDn/eROk6NwZnNKxFnNiciIqp5OLM5ERERURVgI0VERFTHdZ4xHX3mf4KrN+9IHaXG4czmREREdVhi8l0cKfgYyM1B37PPoGnDjlJHqlE4IkVERFSHfbBhI6DIgUrTGiMCfKWOU+OwkSIiIqqj9HqBH5NXAQBedH0LZmYyiRPVPGykiIiI6qg1kb8hx+4ckKfGx0OHSh2nRmIjRUREVEctOlg4k3mTnEHwdHWQOE3NxEaKiIioDvrrlhaXlZsAAOO7cCbz8pK8kVq+fDk8PT2hVqvh4+ODI0eOlFofHR0NHx8fqNVqNGnSBCtXrixSExERAS8vL6hUKnh5eWHnzp1Grx8+fBh9+vRBw4YNIZPJsGvXriLbGD58OGQymdGjY0d+k4GIiGqH5DtauGX3gfpeW4zu/azUcWosSRupLVu2ICwsDFOnTsWZM2fQqVMn9OzZE0lJScXWJyYmolevXujUqRPOnDmDKVOmYOzYsYiIiDDUxMbGIigoCMHBwYiPj0dwcDAGDx6MEydOGGoyMzPRtm1bLF26tNR8PXr0QHJysuGxd+/einnjREREEuvQvBGSlmzG3YWneJH5I5D0FjG+vr7w9vbGihUrDMtatmyJfv36Yf78+UXqJ02ahD179iAhIcGwLDQ0FPHx8YiNjQUABAUFQavVYt++fYaaHj16wMHBAZs2bSqyTZlMhp07d6Jfv35Gy4cPH4579+4VO1pVVrxFDBERUc1TI24Rk5ubi9OnTyMgIMBoeUBAAGJiYopdJzY2tkh9YGAgTp06hby8vFJrStpmaaKiouDk5IQnnngCISEhSE1NLbVep9NBq9UaPYiIiKqb91Zvx48nEh5eSA8lWSOVlpaGgoICODs7Gy13dnZGSkpKseukpKQUW5+fn4+0tLRSa0raZkl69uyJjRs34tdff8Unn3yCkydP4rnnnoNOpytxnfnz58POzs7wcHNzM2mfRERElS3lTgaWXB2BPvu9sC7ypNRxajzJbxEjkxmflxVCFFn2sPoHl5u6zeIEBQUZfm7dujXat28PDw8P/PTTTxgwYECx64SHh2PChAmG51qtls0UERFVK+EbtgKqdCi0j2PIcz5Sx6nxJGukHB0dIZfLi4wUpaamFhlRus/FxaXYenNzc9SvX7/UmpK2WVaurq7w8PDAlStXSqxRqVRQqVSPtB8iIqLKtC1xFWAPPF9vJMzlkn95v8aT7AgqlUr4+PggMjLSaHlkZCT8/f2LXcfPz69I/YEDB9C+fXsoFIpSa0raZlndvn0bN27cgKur6yNth4iISCoRR88h0/4EUGCOj18bLnWcWkHSU3sTJkxAcHAw2rdvDz8/P6xatQpJSUkIDQ0FUHiq7O+//8b69esBFH5Db+nSpZgwYQJCQkIQGxuL1atXG30bb9y4cejcuTMWLlyIvn37Yvfu3Th48CCOHj1qqMnIyMAff/xheJ6YmIi4uDjUq1cP7u7uyMjIwMyZMzFw4EC4urri+vXrmDJlChwdHdG/f/8qOjpEREQVa/aPXwMWwGMZfdHa89HO1ND/ExJbtmyZ8PDwEEqlUnh7e4vo6GjDa8OGDRNdunQxqo+KihLt2rUTSqVSNG7cWKxYsaLINrdt2yaaN28uFAqFaNGihYiIiDB6/dChQwJAkcewYcOEEEJkZWWJgIAA0aBBA6FQKIS7u7sYNmyYSEpKMum9aTQaAUBoNBqT1iMiIqpotzVZQjbZXmAmxJzN+6WOU62Z8vkt6TxStR3nkSIioupiwy+n8XpkAMzybZCz8BqvjyqFKZ/fkn9rj4iIiCrf0Od98KLv3zh8nk1UReKRJCIiqiPsrdV4qaOX1DFqFTZSREREtdyvcVeRX6CXOkatxEaKiIioFtNm6vDCZl9YfPAEouKvSR2n1mEjRUREVIt9uHEnhMVtCDMdOrZ0lzpOrcNGioiIqBb77uIqAMAzViOgVvI7ZhWNjRQREVEt9cuZP3DP4RAgZJg3aITUcWolNlJERES11NSIbwAADTQ98EwrD4nT1E5spIiIiGqhjOxc/Ja3BgDwRtsQidPUXmykiIiIaqFFEQcgLFNhlumCGa+8KHWcWotXnREREdVCM1/rjQY/RiPx1j+wVCukjlNrsZEiIiKqhczMZBjzUmepY9R6PLVHRERUy+j1QuoIdQYbKSIiolokJzcflu+3QdvJY5GYfFfqOLUeGykiIqJa5KMte6GzO49zYjMa2FtJHafWYyNFRERUi3zzv68BAD7mw2BtoZQ4Te3HRoqIiKiWOHnpL6Ta7gUAzOnPuaOqAhspIiKiWmLS5m8BMz3s7nZBYPsnpI5TJ7CRIiIiqgVy8wpwOKPwljCvteBoVFVhI0VERFQLLNh+AAXWNyDLccC8oQOljlNncEJOIiKiWuD5Nl746cIHsLawhr21Wuo4dYZMCMFZuyqJVquFnZ0dNBoNbG1tpY5DREREZWDK5zdP7RERERGVExspIiKiGiy/QI9Wk97GnM37kZtXIHWcOoeNFBERUQ22ZOevuGi5EtPiX4E2Syd1nDqHjRQREVENtjSmcCbz1vohcLSzlDhN3WNSI7Vo0SJkZ2cbnh8+fBg63b/db3p6OkaPHl1x6YiIiKhECUm3cMN6JwBgWu+3JE5TN5nUSIWHhyM9Pd3w/MUXX8Tff/9teJ6VlYWvvvqq4tIRERFRiSZuXAfI82B1rwMGd24rdZw6yaRG6sGZEjhzAhERkTT0eoEDaYWn9QZ4cCZzqfAaKSIiohroyx8OI8/2MpBrjQXBr0gdp87izOZEREQ1UJZOB5WmFTzN/dGwvo3Uceoskxupb775BtbW1gCA/Px8rF27Fo6OjgBgdP0UERERVZ7wwQGY9PI53EnPfngxVRqTbhHTuHFjyGSyh9YlJiY+UqjagreIISIiqnlM+fw2aUTq+vXrj5KLiIiIHpFeL/Det9sQPrA3nByspI5T5/FicyIiohrk6/2x+OzvILjOfxxZOXlSx6nzTGqkTpw4gX379hktW79+PTw9PeHk5IS33nrLaIJOIiIiqliLfy2c8qCJvgcs1QqJ05BJjdTMmTNx9uxZw/Nz587hzTffxAsvvIDJkyfjhx9+wPz58ys8JBEREQF//nMPf6i3AAAmPs+ZzKsDkxqpuLg4PP/884bnmzdvhq+vL77++mtMmDABX3zxBbZu3VrhIYmIiAiYtOF7QJENlaYVRgZ2lDoOwcRG6u7du3B2djY8j46ORo8ePQzPO3TogBs3blRcOiIiIgJQeJH57r9WAQB6uYTAzOzh36KnymdSI+Xs7GyY2iA3Nxf/+9//4OfnZ3g9PT0dCgXP1xIREVW07345hRz7eCBfhY+HBksdh/6fSY1Ujx49MHnyZBw5cgTh4eGwtLREp06dDK+fPXsWTZs2rfCQREREdd2u/x0DhAyNs15G04b1pI5D/8+keaTmzJmDAQMGoEuXLrC2tsbatWuhVCoNr3/77bcICAio8JBERER13c5JYTh8ti/yCgqkjkL/YdLM5vdpNBpYW1tDLpcbLb9z5w5sbGx4eu//cWZzIiKimqfSZjYfMWJEmeq+/fZbUzZLREREpbhwPRWtGjtJHYOKYVIjtXbtWnh4eKBdu3Yox0AWERERmWhLdBxe+dUHj6UPQNLirfy2XjVjUiMVGhqKzZs349q1axgxYgSGDh2KevV4wRsREVFlmbP3a8BSDzOYsYmqhkz61t7y5cuRnJyMSZMm4YcffoCbmxsGDx6Mn3/+mSNUREREFSz1bibOyzcAAMY8w5nMqyOTb1qsUqnw6quvIjIyEhcvXkSrVq0wevRoeHh4ICMjozIyEhER1UlTNm4DVFqYpzfB+H7dpI5DxTC5kfovmUwGmUwGIQT0en1FZSIiIiIAW/8ovEHx8/YhMJc/0kc2VRKT/1R0Oh02bdqE7t27o3nz5jh37hyWLl2KpKQkWFtbV0ZGIiKiOmd3zAWkO8QABeZY9NpwqeNQCUy62Hz06NHYvHkz3N3d8cYbb2Dz5s2oX79+ZWUjIiKqs2b/uBpQAa7pfdCmiYvUcagEJk3IaWZmBnd3d7Rr1w4yWcnfHNixY0eFhKvpOCEnERGVV1KqBpO++x6dW7TB272fkTpOnVJpE3K+/vrrpTZQREREVDHcneyw6b23pY5BD2HyhJxEREREVIhfASAiIqpG9p28BKvx7THs82+kjkJlwEaKiIioGpm+6xtk2Z/G/uu7pY5CZcBGioiIqJrQZupwumAtAGCkd4i0YahM2EgRERFVEzM27YawSINZZkNMC+oldRwqAzZSRERE1cT684UzmftbjIBaadL3wUgibKSIiIiqgV/jruKOw0FAyDB/0JtSx6EykryRWr58OTw9PaFWq+Hj44MjR46UWh8dHQ0fHx+o1Wo0adIEK1euLFITEREBLy8vqFQqeHl5YefOnUavHz58GH369EHDhg0hk8mwa9euItsQQmDmzJlo2LAhLCws0LVrV1y4cOGR3isREVFJPoxYDQCorwnAs60bSxuGykzSRmrLli0ICwvD1KlTcebMGXTq1Ak9e/ZEUlJSsfWJiYno1asXOnXqhDNnzmDKlCkYO3YsIiIiDDWxsbEICgpCcHAw4uPjERwcjMGDB+PEiROGmszMTLRt2xZLly4tMduiRYuwZMkSLF26FCdPnoSLiwu6d++O9PT0ijsARERE/69f2+fgdO9FvNl2lNRRyAQm3SKmovn6+sLb2xsrVqwwLGvZsiX69euH+fPnF6mfNGkS9uzZg4SEBMOy0NBQxMfHIzY2FgAQFBQErVaLffv2GWp69OgBBwcHbNq0qcg2ZTIZdu7ciX79+hmWCSHQsGFDhIWFYdKkSQAKb9bs7OyMhQsXYtSosv2S8xYxRERENY8pn9+SjUjl5ubi9OnTCAgIMFoeEBCAmJiYYteJjY0tUh8YGIhTp04hLy+v1JqStlmcxMREpKSkGG1HpVKhS5cupW5Hp9NBq9UaPYiIiKj2kqyRSktLQ0FBAZydnY2WOzs7IyUlpdh1UlJSiq3Pz89HWlpaqTUlbbOk/dxfz5TtzJ8/H3Z2doaHm5tbmfdJRER107ELf6LzjOk4duFPqaNQOUh+sfmDN0EWQpR6Y+Ti6h9cbuo2KypbeHg4NBqN4XHjxg2T90lERHVL+NbVOGL2Efp8zW/q1USSTVLh6OgIuVxeZIQnNTW1yEjQfS4uLsXWm5ubo379+qXWlLTNkvYDFI5Mubq6lnk7KpUKKpWqzPshIqK6LSc3H8eyvgWsgeBWnMm8JpJsREqpVMLHxweRkZFGyyMjI+Hv71/sOn5+fkXqDxw4gPbt20OhUJRaU9I2i+Pp6QkXFxej7eTm5iI6Otqk7RAREZVm3tb90Fv/DVl2fXz0Wj+p41A5SDpt6oQJExAcHIz27dvDz88Pq1atQlJSEkJDQwEUnir7+++/sX79egCF39BbunQpJkyYgJCQEMTGxmL16tVG38YbN24cOnfujIULF6Jv377YvXs3Dh48iKNHjxpqMjIy8McffxieJyYmIi4uDvXq1YO7uztkMhnCwsIwb948NGvWDM2aNcO8efNgaWmJ1157rYqODhER1XarTq8C7AFvs2GwteIZjRpJSGzZsmXCw8NDKJVK4e3tLaKjow2vDRs2THTp0sWoPioqSrRr104olUrRuHFjsWLFiiLb3LZtm2jevLlQKBSiRYsWIiIiwuj1Q4cOCQBFHsOGDTPU6PV6MWPGDOHi4iJUKpXo3LmzOHfunEnvTaPRCABCo9GYtB4REdV+Jy/9JTDdTGAmxA/HL0odh/7DlM9vSeeRqu04jxQREZXkhdlz8IuYBtu7naD57LDUceg/asQ8UkRERHVZgSgAcq3x6hO8yLwm44hUJeKIFBERlSblTgYsVQpeH1XNmPL5LenF5kRERHWZSz1rqSPQI+KpPSIioiqUkHQLq38+Ab2eJ4RqAzZSREREVWj8+tUYebwjmn0wXOooVAHYSBEREVWR/AI9frn7DQCgi0dXacNQhWAjRUREVEU+3x2FfNurgM4WC4YOljoOVQA2UkRERFXk86OrAABeBa/BycFK4jRUEdhIERERVYFLN9Jww2onAGBKIOeOqi3YSBEREVWBDzauB8xzYXnPB0Oe85Y6DlUQNlJERERV4GjKfgBAP3eORtUmnJCTiIioCiR/vA8Lth/AiBeekToKVSA2UkRERFVAqZBj+qs9pY5BFYyn9oiIiCrRvYwcZOXkSR2DKgkbKSIioko0csXXsJnhhlc+WS51FKoEbKSIiIgqiV4v8FPy19Bb/gO94L31aiM2UkRERJXk2wMnkGN3DsizwMfBQ6SOQ5WAjRQREVElWfTL1wCAJjmD4OFsL20YqhRspIiIiCrBX7e0uKLcDAB4v9tbEqehysJGioiIqBJ88N33gDILSk1LjOrpL3UcqiRspIiIiCrBrhuFNyju6RQCMzOZxGmosrCRIiIiqgTrB6zFk9nvYuGQYKmjUCWSCcHvY1YWrVYLOzs7aDQa2NraSh2HiIiIysCUz2+OSBERERGVExspIiKiCvT2ig1o+v7rWBd5UuooVAXYSBEREVWQqPhrWJX4Aa7ZfIctvx2SOg5VATZSREREFeDstRR0Xx8AvVUyVJrWWPVWqNSRqAqwkSIiInpESaka+H7ZA/m2V2Ge7omYt39Gowb8klFdwEaKiIjoEdzRZqPNvJeQYx8Psyxn7B9yAN7NGkodi6oIGykiIqJH8Oyc8dA4HAZ0tvi+13483+5xqSNRFWIjRURE9AhWD58Ci3tP4YuOPyCoy1NSx6EqZi51ACIioprMz8sd2sWnYS7n2ERdxD91IiIiE/WZ/wk+WLPD8JxNVN3FP3kiIiITDPv8G/yY+z4+vj4IEUfPSR2HJMZGioiIqIw+WLMD6++MAgD4FkzEwGeflDgRSY3XSBEREZXBkp2H8PG1VwFzPZ7IeBMxC+dLHYmqATZSRERED7Hhl9N479RLgDIXrvf6I37hSpiZyaSORdUAT+0RERGV4uSlv/D6gZ6AMgP2d7vh4kffQ63kOAQVYiNFRERUinaPN8STsldgcc8b56bugr21WupIVI2wpSYiIiqFudwMZ+Z9jtR7mXCpZy11HKpmOCJFRET0gNS7meg2axYysnMBAGZmMjZRVCw2UkRERP+RkZ0Lr9kDEYWZaPXhMKnjUDXHRoqIiOj/5Rfo0erDYbht/zOQa4lp3cdJHYmqOTZSREREAPR6Ae+pY5FkuxkoUGBOmx0Y2aOj1LGommMjRUREBOD5j2bjnMUyQMgwptF6TA0KlDoS1QBspIiIqM575ZPliMJMAECQzVJ88dYr0gaiGoONFBER1Xkdm3oBOht0wyxsfm+01HGoBuE8UkREVOeF9esK3yfOw7eFm9RRqIbhiBQREdVJ3+w/jp3Hzhue+3m58/55ZDKOSBERUZ2zO+YC3oruBQDYqo/Cy53aSJyIaiqOSBERUZ1y9Px1DNgVAKG+C6uc5ujcuqnUkagGYyNFRER1xoXrqXhuTXforW5CpWmFuIk/wcnBSupYVIOxkSIiojrhr1tadPi8B/Js/4A83QNHQ39G04b1pI5FNRwbKSIiqvXuZeTgybl9kW1/BrKsBtj3aiTaP/GY1LGoFmAjRUREtV6WLq/wB50Nvgvcj+4+zaQNRLUGv7VHRES1XsP6Nkicsw8/n/4dQV2ekjoO1SIckSIiolrr893Rhp/trdVsoqjCsZEiIqJaqfe8jxEW1xVPT/1A6ihUi7GRIiKiWmfEl2uwN6+wgXK0dJQ4DdVmbKSIiKhWCV+3C2vSRgIAOuROxF6OSFElkryRWr58OTw9PaFWq+Hj44MjR46UWh8dHQ0fHx+o1Wo0adIEK1euLFITEREBLy8vqFQqeHl5YefOnSbvd/jw4ZDJZEaPjh07PtqbJSKiSvXZrigs+OMVwEyPZhlv4PhHC6WORLWcpI3Uli1bEBYWhqlTp+LMmTPo1KkTevbsiaSkpGLrExMT0atXL3Tq1AlnzpzBlClTMHbsWERERBhqYmNjERQUhODgYMTHxyM4OBiDBw/GiRMnTN5vjx49kJycbHjs3bu3cg4EERE9sk1RZzD+t5cAcx1c7vXF2bmreBNiqnQyIYSQaue+vr7w9vbGihUrDMtatmyJfv36Yf78+UXqJ02ahD179iAhIcGwLDQ0FPHx8YiNjQUABAUFQavVYt++fYaaHj16wMHBAZs2bSrzfocPH4579+5h165d5X5/Wq0WdnZ20Gg0sLW1Lfd2iIjo4YZ9/g3W3wuB3d0uuDZ7H+rZWkgdiWooUz6/JRuRys3NxenTpxEQEGC0PCAgADExMcWuExsbW6Q+MDAQp06dQl5eXqk197dpyn6joqLg5OSEJ554AiEhIUhNTS31Pel0Omi1WqMHERFVjXXjRuLDJntwdspuNlFUZSRrpNLS0lBQUABnZ2ej5c7OzkhJSSl2nZSUlGLr8/PzkZaWVmrN/W2Wdb89e/bExo0b8euvv+KTTz7ByZMn8dxzz0Gn05X4nubPnw87OzvDw83N7SFHgYiIHkVi8l1cvXnH8Pyj4D5wd7KTMBHVNZLPbC6TGZ+/FkIUWfaw+geXl2WbD6sJCgoy/Ny6dWu0b98eHh4e+OmnnzBgwIBis4WHh2PChAmG51qtls0UEVElSdNkoe3CF5FrpkXM2z/Du1lDqSNRHSRZI+Xo6Ai5XF5k9Ck1NbXIaNF9Li4uxdabm5ujfv36pdbc32Z59gsArq6u8PDwwJUrV0qsUalUUKlUJb5OREQVIysnD14zX0a6QwxkOfb4M/UOGymShGSn9pRKJXx8fBAZGWm0PDIyEv7+/sWu4+fnV6T+wIEDaN++PRQKRak197dZnv0CwO3bt3Hjxg24urqW7Q0SEVGlyC/Qw2vqcNyy3wfkWWBFp5/Q/5nWUseiukpIaPPmzUKhUIjVq1eLixcvirCwMGFlZSWuX78uhBBi8uTJIjg42FB/7do1YWlpKcaPHy8uXrwoVq9eLRQKhdi+fbuh5tixY0Iul4sFCxaIhIQEsWDBAmFubi6OHz9e5v2mp6eL9957T8TExIjExERx6NAh4efnJx577DGh1WrL/P40Go0AIDQazaMeKiIiEkIUFOhFm0ljBGZCYJq5mPX9XqkjUS1kyue3pI2UEEIsW7ZMeHh4CKVSKby9vUV0dLThtWHDhokuXboY1UdFRYl27doJpVIpGjduLFasWFFkm9u2bRPNmzcXCoVCtGjRQkRERJi036ysLBEQECAaNGggFAqFcHd3F8OGDRNJSUkmvTc2UkREFeu5WbMLm6iZEG+v2CB1HKqlTPn8lnQeqdqO80gREVWcv25p0XjRkyiwTsIAi88R8cFYqSNRLVUj5pEiIiIyRaMGtogdeQyDrJayiaJqgyNSlYgjUkREjy4pVcO5oahKcUSKiIhqhdU/n0DjTz0x/putUkchKhYbKSIiqpb2HL+IkEO9INR3seHCGuj1PIFC1Q8bKSIiqnaOXfgT/SMCICzuwOqeL85N2wYzs5LvekEkFTZSRERUrSQk3UK3bwOgt/4bSk1LnHnvJ7jUs5Y6FlGx2EgREVG1cfN2Otp/2hN5tpchz3DH0VEH0KxRfaljEZWIjRQREVUbb3/9NbLsT0OW7YgfBh9Ah+aNpI5EVCo2UkREVG1ETAyDf8EUrH1hH3p2aC51HKKHMpc6ABER1W16vUB+gR5KhRzmcjMcmz1X6khEZcYRKSIiktSzM6bCc9Jg3MvIkToKkcnYSBERkWT6LliCWPP5uGm3Ax/vPCB1HCKTsZEiIiJJvLVsPfbo3gMABMjnYW7wSxInIjIdGykiIqpy0777AV+njgAA+OgmYN+UyRInIiofXmxORERV6ss9hzHn0mBAUYAm6a/j+MKPOWs51VgckSIioipzR5uNsKNBgCIHzvf64Nycb2Au50cR1Vz87SUioipTz9YCX3beBpd7L+HirC2wVCukjkT0SGRCCN5Ou5JotVrY2dlBo9HA1tZW6jhERERUBqZ8fnNEioiIKtWf/9yDY1ggth6OlzoKUYVjI0VERJUmTZOFNvP74LbDAQTvegW5eQVSRyKqUGykiIioUmTl5KHVzMHQOhwFcuywsf8WKBVyqWMRVSg2UkREVOHyC/R48sM3kWr/E5CnxrJnfsTLndpIHYuowrGRIiKiCqXXCzz94Xu4ZvMdoJdjRsvtGP3is1LHIqoUbKSIiKhCvbl0Dc6oPwMAjHJeg5lDeksbiKgSsZEiIqIKNXfIIDjcfR791J9i5ehgqeMQVSreIoaIiCpUw/o2SPn4Z15YTnUCR6SIiOiRLdp+ED3mLIBeXzjHM5soqivYSBER0SNZF3kSk870w88F4Qhd8Z3UcYiqFBspIiIqt72//Y43fukJKDPhcPd5LB4eJHUkoirFRoqIiMrlRMINvLStO4TFbVjd64Dz03bC1koldSyiKsVGioiITHbpRho6fx2AAuu/oNS0wOkJe9Gwvo3UsYiqHBspIiIySVZOHnyW9Eau3e+QZ7jhcMgBNHdzlDoWkSTYSBERkUks1Qr0dR8BWZYT9gw6AN+WblJHIpKMTAghpA5RW2m1WtjZ2UGj0cDW1lbqOEREFeqvW1o0asC/26j2MeXzmyNSRET0UHq9QO95H+PSjTTDMjZRRGykiIioDDrPnIa9eR+g7addkZGdK3UcomqDjRQREZVqwKLPcUw+FwAw0G0MrC2UEiciqj7YSBERUYneXrEBO7PDAAAvmM3BxvGjpA1EVM2wkSIiomLN3PgTVqYMBwA8lTMOP0+dIm0gomrIXOoARERU/Xy1NwazEgYBigJ4aofi5KIlMDOTSR2LqNphI0VEREU0a+gM8xwX1Mv0wvn538JczhMYRMVhI0VEREU891RTnLaNQcP6trBUK6SOQ1Rt8Z8YREQEADif+A8WbIs0PG/TxAWOdpYSJiKq/jgiRURE+POfe+jwRSBybM/jdvoWfDxioNSRiGoEjkgREdVxd7TZaDP/JeTYx8MsxxE9vZ+SOhJRjcFGioioDsvJzUerGa9A63AE0Nni+1778dxTTaWORVRjsJEiIqqj8gv0aD1lJFLs9wB5anzR8QcEdXlK6lhENQobKSKiOkivF+g4/QNctVkH6OWY0nwLxrzUWepYRDUOGykiojpILwQycjUAgJFOqzE3+CWJExHVTPzWHhFRHWQuN8PFhauw/KfX8W6fTlLHIaqxOCJFRFSHrIs8iZzcfACAmZmMTRTRI+KIFBFRDabXC2izdNBk5sDD2d6wPCr+Gi79nYKMnBykZ2cjU5eD67eTsV37Hibv7YGEj7bA3lotXXCiWoKNFBFRBYu7mozkO1pos7KRnp2DjJwcaP+/mdHr9UaTXb69YgPOp1xCTn4OdAXZ0BXkIFefg1yRDb0oQPKnuwy1rSeNxqWCfdCb5UDIcyDMswFzneH17PA8qJWFf60PWz8VSbabi4YzB2SQG+qI6NHw/yQiqlVS72bibkY2tFk5uJeRDW12DjKyCxsZlUKBt3r6GWrHrtqMG3dTkJ2Xg5z8HGTnFzYyuoIcWCtscXLux4baJyaOwN/5Z1Egy0GBLAd6eTbE/zc0ZrkOKPj4L0Nt5y8HId3hWPEBc63xMf5tpLZf3og0+/2AsvhyvV7AzEwGALiXdwv5dtdLfO/aTJ2hQXJUueKmtinkQg25UMNcWMAcajSxegqRH33ERoqogvD/JCKqdHq9QE5uvtHNbz/bFYV/NPdwO0OLu1laZORmIie/sKFxs2uEre+/Y6ht+v7r0OT/g3zkIF+W/W8zY5YNu7wWuP3Zv/eHe2xua+TbXC82h1LTAm/1TDA8X/X7XOjszhc+kQFQ/P8DgDyjEYB/G6nk/ARk2Z8u4f0Z39TXQmaPjBx7yAosYKZXw0yvhlxYwFyooYS1Ue3zjfrg4q3HoZKrYWFuAbW5GhYKNSwVFrBQqqEXAmYobKRWBs1Gyt33YGOhhq2lBWwt1bC3soCdlRr1bCyMmqPT85YAWFJsXiKqOGykiKhEer3AvYwc3LytRfIdLZLvanBLq0Vauha3M7Rwr++ED1/pYahtOjEYWXoNdEILnUyDfLkWBeZaCKUG9bXPIe2znw3bHv9bX0ClLbpTc+D83x0B/NtIXZf9Cr3D38VmzNTYGz030///dT/5SqBADVmBGmYFFpALNeyEp1HtkxY9cEv7JJRmaqjMLKCSq6EyL2xo6jk7GNXO67YQdzIyYK1Ww1pd2MjYWKhha6GGvbWFUe0/n/5Y2mE1svm90WWufdG3ZZlriahqsJEiqoXuN0B/pWmQfEcLawsVnmnlAQDIysnD61+uxL1sDbQ6LTLytMgq0CJbr0WO0KKFtR9OzFkIoHDm6/qfWpa4H8c/ehgaKTMzGa6r9gCq9GJrdTLjpsk+ywf52dlQCVuozWxhYWYNpVwNlZkaTRo2Map9p9li5ObnwVKlhpVSDRsLC0ND42xva1T7z+wzsFQpoFTIH3qc/nvq7mE4WSURFYeNFFE1lZWTh21H4pBqGAHS4G6WFpocLbQ6Ldq7tcXy0CEAgJQ7GfD86Jn/jABpAXm+YVtumiAkLSm88NhcboaIrLGFLyhR5Nqca/esDD8rFXIg1xpQZAI6W8jzbWFeYAuF3hYqmS1a2LY3WneI02KYm8lRz8oW9a1t4WxnByd7W7g62OIxRzuj2ruf/VrmY/HFW6+UuZbfRCOiqsRGiqgC6PUCaZos3LxTeAos9Z4W/2g0SEvXorWbO15/obDhSEy+i16fTkZmvgY5onAEKM+s8KFXaNFavIazC5YCAJLvpGN4zNPF71AJ/HPlFQCFjZS9tRo59meL1gkZoLOFwuzfbkmpkKNJejAUMhWsFbawUdnCTmULe0tb1Leyg1dHN6NNJL+XDEc7S5jLHz7t3Iawt8pwtIiIag/JG6nly5fj448/RnJyMlq1aoXPPvsMnTqVPEFcdHQ0JkyYgAsXLqBhw4b44IMPEBoaalQTERGBadOm4erVq2jatCnmzp2L/v37m7RfIQRmzZqFVatW4e7du/D19cWyZcvQqlWrij0AVGnyC/TQ64XhFE9Gdi5OXf4LGTk6ZObokKXLRaZOh+zcXGTpdPBp6mm4BiUpVYP312+ALl+HdF0GtLlaZOZpkZlf2Py84PYSvp9Q+Ht39Px1dNrWFDDTF5vD6/dQQyOlFwK/W60qMbNWe8fw82OOtpBnuMG8wA7K/x8BspDZwcrcFlYKW/g94WOoVSvNsbBVJOrb2MDZ3hYuDrZo5GhXYgN0dfH6Mh9Hl3rWDy8iIqqjJG2ktmzZgrCwMCxfvhzPPPMMvvrqK/Ts2RMXL16Eu7t7kfrExET06tULISEh2LBhA44dO4bRo0ejQYMGGDiw8OvEsbGxCAoKwkcffYT+/ftj586dGDx4MI4ePQpfX98y73fRokVYsmQJ1q5diyeeeAJz5sxB9+7dcenSJdjY2FTdQarm9HqBLF0etFk6ZGbnIj27sEFJz9bBy93ZMEFgQtItRMScRlauDjl5ucjO1SE7TwddXi5y8nV4pWNXw13nfznzB97b+iXy9LnI0+uQq9chX+QiX+iQL3R4/cmRWPLmIADA1sPxGLp7EPSyXAgzHYRZLoRcB8h1gDwf3TALv86YDgD4+fQlvPxLmxLfS4cLE/Gi7yIAwPWUO9iW+e6/Lz5wCuzsP40NPzvb2/zbROnNIMu1g1le4ekvhd4WHg3+vcD5MUdbPCebDTu1LewtbFHfyhb1bWzRwNYWrg52aOrqaKhVK82R/3FSmf8sPnj5hTLXEhFRxZAJIYRUO/f19YW3tzdWrFhhWNayZUv069cP8+fPL1I/adIk7NmzBwkJ/359OTQ0FPHx8YiNjQUABAUFQavVYt++fYaaHj16wMHBAZs2bSrTfoUQaNiwIcLCwjBp0iQAgE6ng7OzMxYuXIhRo0aV6f1ptVrY2dlBo9HA1tb24SuYIClVg0s3UpHxn5GVrP+MrrzWpSNaujcAAOyOuYD1R3+BriAXunwddPk65BbkIregsEmZ+WIoXu5U2GB8suNXLDjyMQpELvKhg16WiwLooDcr/Hly28/xUXAfAMDEbyOw+MbLJWYc7rAaa8aOAADM3rQPMy73KrF2oOUX2D5xDABg6Q9HMOZ/JV/YGyCfh58/DAcAbPz1fxh6xKfEWv+CKTg2ey4AIPL0FQTseAoyvQqyAhVkQgkzvQpmQgkzoUKgSzB2TgoDAFz56za6LRkFc5kKFnJr2CjsYKuyhZ3aFg6WtujUvBWGde8AoHDk6/z1f9Cwni0c7SwNc/4QEVHNZMrnt2QjUrm5uTh9+jQmT55stDwgIAAxMTHFrhMbG4uAgACjZYGBgVi9ejXy8vKgUCgQGxuL8ePHF6n57LPPyrzfxMREpKSkGO1LpVKhS5cuiImJKbGR0ul00On+nWVYqy3mq90VZOy332K3bkKJr1uqfkZL98L8Eb/FYEf2uH9fNPv/x/9PfXPyandDI3UtNaVwcsAS3Eq/Z/hZqVAULdDLgXwVZHol5Gb/nlJ6rF49WNxrBzlUkAsl5FDBXKaCuUwJhUyFNk88bqj1aeoB/5NToJKroJQroTZXQWmuhIVCBQuFCoFt2xlqn3/qCSzLOgILpRJWahUsVUpYq1WwUithbaGCs/2/p6W6+zSD8Mks8b39V7NG9fHXku1lqjWXm+Gppq5lqiUiotpFskYqLS0NBQUFcHZ2Nlru7OyMlJSUYtdJSUkptj4/Px9paWlwdXUtseb+Nsuy3/v/La7mzz//LPE9zZ8/H7NmzSrx9YpkrbYCtLaFIyv6whGV+6Mrcqhgb/XvN6+e8ngcUUeCoDBTQSFTQilXQWFW+F+VXIVnWjQ31Ab5+yH3yFqozZWwUKqgVihhqVLBSqWChVKJTq3+bXgm9g/Aq7dSYGOpgo2FCtYWyhK/cv5moC/eDPxfmd6bn5e7YRTpYVzqWWP0i8+WqZaIiKiiSX6xuUxmfBpECFFk2cPqH1xelm1WVM1/hYeHY8KEf0eJtFot3NzcSqx/FBvC3sIGlO0bUhP6d8OE/t3KVNu5jSc6t/F8eCEKvynGr5oTEVFdJlkj5ejoCLlcXmT0KTU1tchI0H0uLi7F1pubm6N+/fql1tzfZln26+LiAqBwZMrV1bXYmuKoVCqoVKoSXyciIqLa5eETw1QSpVIJHx8fREZGGi2PjIyEv79/sev4+fkVqT9w4ADat28Pxf9fr1NSzf1tlmW/np6ecHFxMarJzc1FdHR0idmIiIioDhIS2rx5s1AoFGL16tXi4sWLIiwsTFhZWYnr168LIYSYPHmyCA4ONtRfu3ZNWFpaivHjx4uLFy+K1atXC4VCIbZv326oOXbsmJDL5WLBggUiISFBLFiwQJibm4vjx4+Xeb9CCLFgwQJhZ2cnduzYIc6dOydeffVV4erqKrRabZnfn0ajEQCERqN5lMNEREREVciUz29JGykhhFi2bJnw8PAQSqVSeHt7i+joaMNrw4YNE126dDGqj4qKEu3atRNKpVI0btxYrFixosg2t23bJpo3by4UCoVo0aKFiIiIMGm/Qgih1+vFjBkzhIuLi1CpVKJz587i3LlzJr03NlJEREQ1jymf35LOI1XbVeY8UkRERFQ5TPn8luwaKSIiIqKajo0UERERUTmxkSIiIiIqJzZSREREROXERoqIiIionNhIEREREZUTGykiIiKicmIjRURERFRObKSIiIiIyslc6gC12f1J47VarcRJiIiIqKzuf26X5eYvbKQqUXp6OgDAzc1N4iRERERkqvT0dNjZ2ZVaw3vtVSK9Xo+bN2/CxsYGMpmsQret1Wrh5uaGGzdu8D5+D8FjVXY8VmXHY1V2PFZlx2NVdpV5rIQQSE9PR8OGDWFmVvpVUByRqkRmZmZo1KhRpe7D1taW/7OVEY9V2fFYlR2PVdnxWJUdj1XZVdaxethI1H282JyIiIionNhIEREREZUTG6kaSqVSYcaMGVCpVFJHqfZ4rMqOx6rseKzKjseq7Hisyq66HCtebE5ERERUThyRIiIiIionNlJERERE5cRGioiIiKic2EgRERERlRMbqVpEp9PhqaeegkwmQ1xcnNRxqqWXXnoJ7u7uUKvVcHV1RXBwMG7evCl1rGrn+vXrePPNN+Hp6QkLCws0bdoUM2bMQG5urtTRqqW5c+fC398flpaWsLe3lzpOtbN8+XJ4enpCrVbDx8cHR44ckTpStXP48GH06dMHDRs2hEwmw65du6SOVG3Nnz8fHTp0gI2NDZycnNCvXz9cunRJsjxspGqRDz74AA0bNpQ6RrXWrVs3bN26FZcuXUJERASuXr2Kl19+WepY1c7vv/8OvV6Pr776ChcuXMCnn36KlStXYsqUKVJHq5Zyc3MxaNAgvP3221JHqXa2bNmCsLAwTJ06FWfOnEGnTp3Qs2dPJCUlSR2tWsnMzETbtm2xdOlSqaNUe9HR0XjnnXdw/PhxREZGIj8/HwEBAcjMzJQmkKBaYe/evaJFixbiwoULAoA4c+aM1JFqhN27dwuZTCZyc3OljlLtLVq0SHh6ekodo1pbs2aNsLOzkzpGtfL000+L0NBQo2UtWrQQkydPlihR9QdA7Ny5U+oYNUZqaqoAIKKjoyXZP0ekaoF//vkHISEh+O6772BpaSl1nBrjzp072LhxI/z9/aFQKKSOU+1pNBrUq1dP6hhUg+Tm5uL06dMICAgwWh4QEICYmBiJUlFto9FoAECyv5/YSNVwQggMHz4coaGhaN++vdRxaoRJkybBysoK9evXR1JSEnbv3i11pGrv6tWr+PLLLxEaGip1FKpB0tLSUFBQAGdnZ6Plzs7OSElJkSgV1SZCCEyYMAHPPvssWrduLUkGNlLV1MyZMyGTyUp9nDp1Cl9++SW0Wi3Cw8OljiyZsh6r+yZOnIgzZ87gwIEDkMvleP311yHqyAT/ph4rALh58yZ69OiBQYMGYeTIkRIlr3rlOVZUPJlMZvRcCFFkGVF5vPvuuzh79iw2bdokWQbeIqaaSktLQ1paWqk1jRs3xiuvvIIffvjB6C+lgoICyOVyDBkyBOvWravsqJIr67FSq9VFlv/1119wc3NDTEwM/Pz8KititWHqsbp58ya6desGX19frF27FmZmdeffXuX5vVq7di3CwsJw7969Sk5XM+Tm5sLS0hLbtm1D//79DcvHjRuHuLg4REdHS5iu+pLJZNi5cyf69esndZRqbcyYMdi1axcOHz4MT09PyXKYS7ZnKpWjoyMcHR0fWvfFF19gzpw5huc3b95EYGAgtmzZAl9f38qMWG2U9VgV5/6/I3Q6XUVGqrZMOVZ///03unXrBh8fH6xZs6ZONVHAo/1eUSGlUgkfHx9ERkYaNVKRkZHo27evhMmoJhNCYMyYMdi5cyeioqIkbaIANlI1nru7u9Fza2trAEDTpk3RqFEjKSJVW7/99ht+++03PPvss3BwcMC1a9cwffp0NG3atE6MRpni5s2b6Nq1K9zd3bF48WLcunXL8JqLi4uEyaqnpKQk3LlzB0lJSSgoKDDM4/b4448b/p+sqyZMmIDg4GC0b98efn5+WLVqFZKSkni93QMyMjLwxx9/GJ4nJiYiLi4O9erVK/L3fF33zjvv4Pvvv8fu3bthY2NjuN7Ozs4OFhYWVR9Iku8KUqVJTEzk9AclOHv2rOjWrZuoV6+eUKlUonHjxiI0NFT89ddfUkerdtasWSMAFPugooYNG1bssTp06JDU0aqFZcuWCQ8PD6FUKoW3t7dkX1Ovzg4dOlTs79CwYcOkjlbtlPR305o1ayTJw2ukiIiIiMqpbl30QERERFSB2EgRERERlRMbKSIiIqJyYiNFREREVE5spIiIiIjKiY0UERERUTmxkSIiIiIqJzZSRFTlunbtirCwMKljFOv27dtwcnLC9evXAQBRUVGQyWSVfv+88u5n7dq1sLe3N2mdDh06YMeOHSatQ0TFYyNFRDVecnIyXnvtNTRv3hxmZmYlNmkRERHw8vKCSqWCl5cXdu7cWaRm/vz56NOnDxo3bly5oSU0bdo0TJ48GXq9XuooRDUeGykiqvF0Oh0aNGiAqVOnom3btsXWxMbGIigoCMHBwYiPj0dwcDAGDx6MEydOGGqys7OxevVqjBw5sqqiS6J3797QaDT4+eefpY5CVOOxkSIiSd29exevv/46HBwcYGlpiZ49e+LKlStGNV9//TXc3NxgaWmJ/v37Y8mSJUansxo3bozPP/8cr7/+Ouzs7Irdz2effYbu3bsjPDwcLVq0QHh4OJ5//nl89tlnhpp9+/bB3Ny81JtY3759G6+++ioaNWoES0tLPPnkk9i0aZNRTdeuXTFmzBiEhYXBwcEBzs7OWLVqFTIzM/HGG2/AxsYGTZs2xb59+4ps/9ixY2jbti3UajV8fX1x7tw5o9fXrl0Ld3d3w7G4ffu20etXr15F37594ezsDGtra3To0AEHDx40qpHL5ejVq1eR3ERkOjZSRCSp4cOH49SpU9izZw9iY2MhhECvXr2Ql5cHoLCxCA0Nxbhx4xAXF4fu3btj7ty5Ju8nNjYWAQEBRssCAwMRExNjeH748GG0b9++1O3k5OTAx8cHP/74I86fP4+33noLwcHBRiNbALBu3To4Ojrit99+w5gxY/D2229j0KBB8Pf3x//+9z8EBgYiODgYWVlZRutNnDgRixcvxsmTJ+Hk5ISXXnrJcCxOnDiBESNGYPTo0YiLi0O3bt0wZ84co/UzMjLQq1cvHDx4EGfOnEFgYCD69OmDpKQko7qnn34aR44cKdvBI6KSSXKrZCKq07p06SLGjRsnLl++LACIY8eOGV5LS0sTFhYWYuvWrUIIIYKCgkTv3r2N1h8yZIiws7MrddsPUigUYuPGjUbLNm7cKJRKpeF53759xYgRI4xqDh06JACIu3fvlvh+evXqJd577z2jDM8++6zheX5+vrCyshLBwcGGZcnJyQKAiI2NNdrP5s2bDTW3b98WFhYWYsuWLUIIIV599VXRo0cPo30HBQWVeCzu8/LyEl9++aXRst27dwszMzNRUFBQ6rpEVDqOSBGRZBISEmBubg5fX1/Dsvr166N58+ZISEgAAFy6dAlPP/200XoPPi8rmUxm9FwIYbQsOzsbarW61G0UFBRg7ty5aNOmDerXrw9ra2scOHCgyIhPmzZtDD/L5XLUr18fTz75pGGZs7MzACA1NdVovf+eVqxXr57RsUhISChy2vHB55mZmfjggw/g5eUFe3t7WFtb4/fffy+Sz8LCAnq9HjqdrtT3S0SlM5c6ABHVXUKIEpffb3AebHZKW680Li4uSElJMVqWmppqaGgAwNHREXfv3i11O5988gk+/fRTfPbZZ3jyySdhZWWFsLAw5ObmGtUpFAqj5zKZzGjZ/fdUlm/O/fdYPMzEiRPx888/Y/HixXj88cdhYWGBl19+uUi+O3fuwNLSEhYWFg/dJhGVjCNSRCQZLy8v5OfnG11fdPv2bVy+fBktW7YEALRo0QK//fab0XqnTp0yeV9+fn6IjIw0WnbgwAH4+/sbnrdr1w4XL14sdTtHjhxB3759MXToULRt2xZNmjQpcnH8ozh+/Ljh57t37+Ly5cto0aIFgMLj9d/XH6y/n2/48OHo378/nnzySbi4uBjmxPqv8+fPw9vbu8JyE9VVbKSISDLNmjVD3759ERISgqNHjyI+Ph5Dhw7FY489hr59+wIAxowZg71792LJkiW4cuUKvvrqK+zbt6/IKFVcXBzi4uKQkZGBW7duIS4uzqgpGjduHA4cOICFCxfi999/x8KFC3Hw4EGjOacCAwNx4cKFUkelHn/8cURGRiImJgYJCQkYNWpUkZGuRzF79mz88ssvOH/+PIYPHw5HR0f069cPADB27Fjs378fixYtwuXLl7F06VLs37+/SL4dO3YgLi4O8fHxeO2114od9Tpy5EiRi++JyHRspIhIUmvWrIGPjw9efPFF+Pn5QQiBvXv3Gk6DPfPMM1i5ciWWLFmCtm3bYv/+/Rg/fnyRa5natWuHdu3a4fTp0/j+++/Rrl079OrVy/C6v78/Nm/ejDVr1qBNmzZYu3YttmzZYnR91pNPPon27dtj69atJeadNm0avL29ERgYiK5du8LFxcXQ6FSEBQsWYNy4cfDx8UFycjL27NkDpVIJAOjYsSO++eYbfPnll3jqqadw4MABfPjhh0brf/rpp3BwcIC/vz/69OmDwMDAIiNPf//9N2JiYvDGG29UWG6iukomynOxARGRhEJCQvD7779Xytf39+7di/fffx/nz5+HmVnt/LfmxIkTodFosGrVKqmjENV4vNiciKq9xYsXo3v37rCyssK+ffuwbt06LF++vFL21atXL1y5cgV///033NzcKmUfUnNycsL7778vdQyiWoEjUkRU7Q0ePBhRUVFIT09HkyZNMGbMGISGhkodi4iIjRQRERFRedXOCwCIiIiIqgAbKSIiIqJyYiNFREREVE5spIiIiIjKiY0UERERUTmxkSIiIiIqJzZSREREROXERoqIiIionNhIEREREZXT/wEstM74qGAwcwAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ "import numpy as np\n", "import pandas as pd\n", @@ -892,9 +1056,7 @@ { "cell_type": "markdown", "id": "45077718", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see here, when compared to the code which includes explicitely the\n", "intercept column, that our MSE value is actually smaller. This is\n", @@ -907,9 +1069,7 @@ { "cell_type": "markdown", "id": "29c3d802", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Test Function for what happens with OLS, Ridge and Lasso\n", "\n", @@ -933,10 +1093,7 @@ "cell_type": "code", "execution_count": 5, "id": "5012c9d3", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -1006,9 +1163,7 @@ { "cell_type": "markdown", "id": "5f803cfb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "How can we understand this?" ] @@ -1016,9 +1171,7 @@ { "cell_type": "markdown", "id": "10d28c6d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Linking the regression analysis with a statistical interpretation\n", "\n", @@ -1045,9 +1198,7 @@ { "cell_type": "markdown", "id": "5fb5c55c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*} \n", @@ -1061,9 +1212,7 @@ { "cell_type": "markdown", "id": "1b1535c1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The randomness of $\\varepsilon_i$ implies that\n", "$\\mathbf{y}_i$ is also a random variable. In particular,\n", @@ -1080,9 +1229,7 @@ { "cell_type": "markdown", "id": "0da8bbbf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Assumptions made\n", "\n", @@ -1094,9 +1241,7 @@ { "cell_type": "markdown", "id": "e6ef7183", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n", @@ -1106,9 +1251,7 @@ { "cell_type": "markdown", "id": "adb80a26", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We approximate this function with our model from the solution of the linear regression equations, that is our\n", "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we want to minimize $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, our MSE, with" @@ -1117,9 +1260,7 @@ { "cell_type": "markdown", "id": "37077e81", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n", @@ -1129,9 +1270,7 @@ { "cell_type": "markdown", "id": "d18cf8fc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Expectation value and variance\n", "\n", @@ -1141,9 +1280,7 @@ { "cell_type": "markdown", "id": "456d9e60", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*} \n", @@ -1157,9 +1294,7 @@ { "cell_type": "markdown", "id": "78631f59", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "while\n", "its variance is" @@ -1168,9 +1303,7 @@ { "cell_type": "markdown", "id": "d43327dc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n", @@ -1191,9 +1324,7 @@ { "cell_type": "markdown", "id": "6917fcd4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD)." @@ -1202,9 +1333,7 @@ { "cell_type": "markdown", "id": "f4bb4a1b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Expectation value and variance for $\\boldsymbol{\\beta}$\n", "\n", @@ -1214,9 +1343,7 @@ { "cell_type": "markdown", "id": "c59e47ba", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E}(\\boldsymbol{\\hat{\\beta}}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n", @@ -1226,9 +1353,7 @@ { "cell_type": "markdown", "id": "b9f3fb2f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This means that the estimator of the regression parameters is unbiased.\n", "\n", @@ -1240,9 +1365,7 @@ { "cell_type": "markdown", "id": "0f5850a2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{eqnarray*}\n", @@ -1271,9 +1394,7 @@ { "cell_type": "markdown", "id": "1a1b57d1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have used that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n", "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n", @@ -1293,9 +1414,7 @@ { "cell_type": "markdown", "id": "be04bdbe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n", @@ -1305,9 +1424,7 @@ { "cell_type": "markdown", "id": "7fae0994", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see clearly that \n", "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n", @@ -1318,9 +1435,7 @@ { "cell_type": "markdown", "id": "ef37e705", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n", @@ -1330,9 +1445,7 @@ { "cell_type": "markdown", "id": "d5e80880", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n", "\n", @@ -1342,9 +1455,7 @@ { "cell_type": "markdown", "id": "9c33c200", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n", @@ -1354,9 +1465,7 @@ { "cell_type": "markdown", "id": "4310428b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The difference is non-negative definite since each component of the\n", "matrix product is non-negative definite. \n", @@ -1366,9 +1475,7 @@ { "cell_type": "markdown", "id": "47925ea6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Deriving OLS from a probability distribution\n", "\n", @@ -1389,9 +1496,7 @@ { "cell_type": "markdown", "id": "d72bd653", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -1401,9 +1506,7 @@ { "cell_type": "markdown", "id": "9509fe5b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Independent and Identically Distributed (iid)\n", "\n", @@ -1414,9 +1517,7 @@ { "cell_type": "markdown", "id": "055aaf91", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(y_i, \\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n", @@ -1426,9 +1527,7 @@ { "cell_type": "markdown", "id": "d69b50bc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\beta}$.\n", "\n", @@ -1438,9 +1537,7 @@ { "cell_type": "markdown", "id": "2faeec2e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{y},\\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta}).\n", @@ -1450,9 +1547,7 @@ { "cell_type": "markdown", "id": "54aeb6ca", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We will write this in a more compact form reserving $\\boldsymbol{D}$ for the domain of events, including the ouputs (targets) and the inputs. That is\n", "in case we have a simple one-dimensional input and output case" @@ -1461,9 +1556,7 @@ { "cell_type": "markdown", "id": "319be1e0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})].\n", @@ -1473,9 +1566,7 @@ { "cell_type": "markdown", "id": "9feac33c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In the more general case the various inputs should be replaced by the possible features represented by the input data set $\\boldsymbol{X}$. \n", "We can now rewrite the above probability as" @@ -1484,9 +1575,7 @@ { "cell_type": "markdown", "id": "7ae166c5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -1496,9 +1585,7 @@ { "cell_type": "markdown", "id": "cca200fe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It is a conditional probability (see below) and reads as the likelihood of a domain of events $\\boldsymbol{D}$ given a set of parameters $\\boldsymbol{\\beta}$." ] @@ -1506,9 +1593,7 @@ { "cell_type": "markdown", "id": "f19d0e98", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Maximum Likelihood Estimation (MLE)\n", "\n", @@ -1537,9 +1622,7 @@ { "cell_type": "markdown", "id": "316c0946", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A new Cost Function\n", "\n", @@ -1549,9 +1632,7 @@ { "cell_type": "markdown", "id": "ee61aa2f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=-\\log{\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})}=-\\sum_{i=0}^{n-1}\\log{p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})},\n", @@ -1561,9 +1642,7 @@ { "cell_type": "markdown", "id": "9112bcbe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which becomes" ] @@ -1571,9 +1650,7 @@ { "cell_type": "markdown", "id": "cece0c61", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}.\n", @@ -1583,9 +1660,7 @@ { "cell_type": "markdown", "id": "841d1673", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the derivative of the *new* cost function with respect to the parameters $\\beta$ we recognize our familiar OLS equation, namely" ] @@ -1593,9 +1668,7 @@ { "cell_type": "markdown", "id": "0242ac33", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right) =0,\n", @@ -1605,9 +1678,7 @@ { "cell_type": "markdown", "id": "07ed7750", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to the well-known OLS equation for the optimal paramters $\\beta$" ] @@ -1615,9 +1686,7 @@ { "cell_type": "markdown", "id": "7323439c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n", @@ -1627,9 +1696,7 @@ { "cell_type": "markdown", "id": "7a76a127", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics." ] @@ -1637,9 +1704,7 @@ { "cell_type": "markdown", "id": "c5d03cbf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More basic Statistics and Bayes' theorem\n", "\n", @@ -1657,9 +1722,7 @@ { "cell_type": "markdown", "id": "5aa6e83b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X \\cup Y)= p(X)+p(Y)-p(X \\cap Y).\n", @@ -1669,9 +1732,7 @@ { "cell_type": "markdown", "id": "cc714f28", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**The product rule (aka joint probability) is given by.**" ] @@ -1679,9 +1740,7 @@ { "cell_type": "markdown", "id": "4cf8fd97", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X \\cup Y)= p(X,Y)= p(X\\vert Y)p(Y)=p(Y\\vert X)p(X),\n", @@ -1691,9 +1750,7 @@ { "cell_type": "markdown", "id": "104305f0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we read $p(X\\vert Y)$ as the likelihood of obtaining $X$ given $Y$.\n", "\n", @@ -1703,9 +1760,7 @@ { "cell_type": "markdown", "id": "a77d6cf5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Marginal Probability\n", "\n", @@ -1715,9 +1770,7 @@ { "cell_type": "markdown", "id": "e1987035", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X)=\\sum_{i=0}^{n-1}p(X,Y=y_i)=\\sum_{i=0}^{n-1}p(X\\vert Y=y_i)p(Y=y_i)=\\sum_{i=0}^{n-1}p(X\\vert y_i)p(y_i).\n", @@ -1727,9 +1780,7 @@ { "cell_type": "markdown", "id": "61e1c7a7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conditional Probability\n", "\n", @@ -1739,9 +1790,7 @@ { "cell_type": "markdown", "id": "286ed532", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{p(Y)}=\\frac{p(X,Y)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)}.\n", @@ -1751,9 +1800,7 @@ { "cell_type": "markdown", "id": "2e26c494", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Bayes' Theorem\n", "\n", @@ -1763,9 +1810,7 @@ { "cell_type": "markdown", "id": "2e59091a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{p(Y)},\n", @@ -1775,9 +1820,7 @@ { "cell_type": "markdown", "id": "bc4fbdf1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we can rewrite as" ] @@ -1785,9 +1828,7 @@ { "cell_type": "markdown", "id": "d576f97e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)}=\\frac{p(Y\\vert X)p(X)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)},\n", @@ -1797,9 +1838,7 @@ { "cell_type": "markdown", "id": "99a81edf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is Bayes' theorem. It allows us to evaluate the uncertainty in in $X$ after we have observed $Y$. We can easily interchange $X$ with $Y$." ] @@ -1807,9 +1846,7 @@ { "cell_type": "markdown", "id": "8e5cf0af", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Interpretations of Bayes' Theorem\n", "\n", @@ -1826,9 +1863,7 @@ { "cell_type": "markdown", "id": "4941c8cf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Example of Usage of Bayes' theorem\n", "\n", @@ -1847,9 +1882,7 @@ { "cell_type": "markdown", "id": "43d36c51", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X=1\\vert Y=1) =0.8.\n", @@ -1859,9 +1892,7 @@ { "cell_type": "markdown", "id": "2aaa3bac", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of $80\\%$ for having cancer.\n", "It is however not correct, as the following Bayesian analysis shows." @@ -1870,9 +1901,7 @@ { "cell_type": "markdown", "id": "06f3f234", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Doing it correctly\n", "\n", @@ -1883,9 +1912,7 @@ { "cell_type": "markdown", "id": "2bf06ce9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(Y=1) =0.004.\n", @@ -1895,9 +1922,7 @@ { "cell_type": "markdown", "id": "68fe5cdc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have" ] @@ -1905,9 +1930,7 @@ { "cell_type": "markdown", "id": "0062a5f5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X=1\\vert Y=0) =0.1.\n", @@ -1917,9 +1940,7 @@ { "cell_type": "markdown", "id": "37530707", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute" ] @@ -1927,9 +1948,7 @@ { "cell_type": "markdown", "id": "0f7370b2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(Y=1\\vert X=1)=\\frac{p(X=1\\vert Y=1)p(Y=1)}{p(X=1\\vert Y=1)p(Y=1)+p(X=1\\vert Y=0)p(Y=0)}=\\frac{0.8\\times 0.004}{0.8\\times 0.004+0.1\\times 0.996}=0.031.\n", @@ -1939,9 +1958,7 @@ { "cell_type": "markdown", "id": "942c3e95", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "That is, in case of a positive test, there is only a $3\\%$ chance of having breast cancer!" ] @@ -1949,9 +1966,7 @@ { "cell_type": "markdown", "id": "f260b118", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Bayes' Theorem and Ridge and Lasso Regression\n", "\n", @@ -1963,9 +1978,7 @@ { "cell_type": "markdown", "id": "0ef62f7a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})],\n", @@ -1975,9 +1988,7 @@ { "cell_type": "markdown", "id": "1f155fd8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "is given by" ] @@ -1985,9 +1996,7 @@ { "cell_type": "markdown", "id": "867b2648", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -1997,9 +2006,7 @@ { "cell_type": "markdown", "id": "099aa023", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\\boldsymbol{\\beta}$ given a domain of events $\\boldsymbol{D}$? That is, how can we define the posterior probability" ] @@ -2007,9 +2014,7 @@ { "cell_type": "markdown", "id": "e15e2f85", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D}).\n", @@ -2019,9 +2024,7 @@ { "cell_type": "markdown", "id": "072f354d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Bayes' theorem comes to our rescue here since (omitting the normalization constant)" ] @@ -2029,9 +2032,7 @@ { "cell_type": "markdown", "id": "4356389d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})\\propto p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})p(\\boldsymbol{\\beta}).\n", @@ -2041,9 +2042,7 @@ { "cell_type": "markdown", "id": "d9488d50", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$!" ] @@ -2051,9 +2050,7 @@ { "cell_type": "markdown", "id": "fb4db12d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Ridge and Bayes\n", "\n", @@ -2067,9 +2064,7 @@ { "cell_type": "markdown", "id": "50da26f2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", @@ -2079,9 +2074,7 @@ { "cell_type": "markdown", "id": "2979fd74", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" ] @@ -2089,9 +2082,7 @@ { "cell_type": "markdown", "id": "664132a2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta\\vert\\boldsymbol{D})}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", @@ -2101,9 +2092,7 @@ { "cell_type": "markdown", "id": "e380fd1b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can now optimize this quantity with respect to $\\boldsymbol{\\beta}$. As we\n", "did for OLS, this is most conveniently done by taking the negative\n", @@ -2114,9 +2103,7 @@ { "cell_type": "markdown", "id": "c0f7231e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", @@ -2126,9 +2113,7 @@ { "cell_type": "markdown", "id": "9a330528", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and replacing $1/2\\tau^2$ with $\\lambda$ we have" ] @@ -2136,9 +2121,7 @@ { "cell_type": "markdown", "id": "3f3eecaf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", @@ -2148,9 +2131,7 @@ { "cell_type": "markdown", "id": "aed40b2c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is our Ridge cost function! Nice, isn't it?" ] @@ -2158,9 +2139,7 @@ { "cell_type": "markdown", "id": "ed0e221c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Lasso and Bayes\n", "\n", @@ -2170,9 +2149,7 @@ { "cell_type": "markdown", "id": "16accfb2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", @@ -2182,9 +2159,7 @@ { "cell_type": "markdown", "id": "45c691f0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" ] @@ -2192,9 +2167,7 @@ { "cell_type": "markdown", "id": "5a4d8a4d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", @@ -2204,9 +2177,7 @@ { "cell_type": "markdown", "id": "62586eaf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the negative\n", "logarithm of the posterior probability and leaving out the\n", @@ -2216,9 +2187,7 @@ { "cell_type": "markdown", "id": "fff1d30c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -2228,9 +2197,7 @@ { "cell_type": "markdown", "id": "4ed47eeb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and replacing $1/\\tau$ with $\\lambda$ we have" ] @@ -2238,9 +2205,7 @@ { "cell_type": "markdown", "id": "ac4d6ee9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -2250,9 +2215,7 @@ { "cell_type": "markdown", "id": "68874d9e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is our Lasso cost function!" ] @@ -2260,9 +2223,7 @@ { "cell_type": "markdown", "id": "da916a28", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Material for the active learning sessions Tuesday and Wednesday\n", "\n", @@ -2272,9 +2233,7 @@ { "cell_type": "markdown", "id": "24fa49ce", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Linear Regression and the SVD\n", "\n", @@ -2284,9 +2243,7 @@ { "cell_type": "markdown", "id": "f12ae6c1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -2296,9 +2253,7 @@ { "cell_type": "markdown", "id": "2a363e7d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Since the matrices here have dimension $p\\times p$, with $p$ corresponding to the singular values, we defined last week the matrix" ] @@ -2306,9 +2261,7 @@ { "cell_type": "markdown", "id": "2200cba7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma} = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\end{bmatrix},\n", @@ -2318,9 +2271,7 @@ { "cell_type": "markdown", "id": "8176f3bd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where the tilde-matrix $\\tilde{\\boldsymbol{\\Sigma}}$ is a matrix of dimension $p\\times p$ containing only the singular values $\\sigma_i$, that is" ] @@ -2328,9 +2279,7 @@ { "cell_type": "markdown", "id": "0b4b646e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\tilde{\\boldsymbol{\\Sigma}}=\\begin{bmatrix} \\sigma_0 & 0 & 0 & \\dots & 0 & 0 \\\\\n", @@ -2345,9 +2294,7 @@ { "cell_type": "markdown", "id": "3de20aa6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "meaning we can write" ] @@ -2355,9 +2302,7 @@ { "cell_type": "markdown", "id": "b1ad52ff", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2\\boldsymbol{V}^T.\n", @@ -2367,9 +2312,7 @@ { "cell_type": "markdown", "id": "1cde554b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Multiplying from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get" ] @@ -2377,9 +2320,7 @@ { "cell_type": "markdown", "id": "faee6b74", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2.\n", @@ -2389,9 +2330,7 @@ { "cell_type": "markdown", "id": "c40c4ebe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## What does it mean?\n", "\n", @@ -2403,9 +2342,7 @@ { "cell_type": "markdown", "id": "bb1be7c1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n", @@ -2415,9 +2352,7 @@ { "cell_type": "markdown", "id": "be866232", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In other words, each non-zero singular value of $\\boldsymbol{X}$ is a positive\n", "square root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. It means also that\n", @@ -2437,9 +2372,7 @@ { "cell_type": "markdown", "id": "2752dc1c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{X}]=\\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X},\n", @@ -2449,9 +2382,7 @@ { "cell_type": "markdown", "id": "29756910", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "meaning that every squared non-singular value of $\\boldsymbol{X}$ divided by $n$ (\n", "the number of samples) are the eigenvalues of the covariance\n", @@ -2464,9 +2395,7 @@ { "cell_type": "markdown", "id": "03a070a2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## And finally $\\boldsymbol{X}\\boldsymbol{X}^T$\n", "\n", @@ -2476,9 +2405,7 @@ { "cell_type": "markdown", "id": "69630b59", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{U}^T.\n", @@ -2488,9 +2415,7 @@ { "cell_type": "markdown", "id": "18fc105a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Since the matrices here have dimension $n\\times n$, we have" ] @@ -2498,9 +2423,7 @@ { "cell_type": "markdown", "id": "130b2e17", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\boldsymbol{0}\\\\ \\end{bmatrix}=\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix},\n", @@ -2510,9 +2433,7 @@ { "cell_type": "markdown", "id": "9c96d6b4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "leading to" ] @@ -2520,9 +2441,7 @@ { "cell_type": "markdown", "id": "ace4da9c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\boldsymbol{U}^T.\n", @@ -2532,9 +2451,7 @@ { "cell_type": "markdown", "id": "38aa035d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Multiplying with $\\boldsymbol{U}$ from the right gives us the eigenvalue problem" ] @@ -2542,9 +2459,7 @@ { "cell_type": "markdown", "id": "1d21feff", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U}=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}.\n", @@ -2554,9 +2469,7 @@ { "cell_type": "markdown", "id": "0c7f206f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by\n", "the non-zero singular values plus now a series of zeros. The column\n", @@ -2571,9 +2484,7 @@ { "cell_type": "markdown", "id": "c381860e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code for SVD and Inversion of Matrices\n", "\n", @@ -2585,10 +2496,7 @@ "cell_type": "code", "execution_count": 6, "id": "25ace52e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -2599,9 +2507,7 @@ { "cell_type": "markdown", "id": "144a7709", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD." ] @@ -2610,10 +2516,7 @@ "cell_type": "code", "execution_count": 7, "id": "6353462f", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -2650,9 +2553,7 @@ { "cell_type": "markdown", "id": "3c6d5e99", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Inverse of Rectangular Matrix\n", "\n", @@ -2670,9 +2571,7 @@ { "cell_type": "markdown", "id": "7efcc48d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}_{\\mathrm{PI}}= \\boldsymbol{V}\\boldsymbol{D}_{\\mathrm{PI}}\\boldsymbol{U}^T,\n", @@ -2682,9 +2581,7 @@ { "cell_type": "markdown", "id": "d632ac16", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{D}_{\\mathrm{PI}}$ can be calculated by creating a diagonal matrix from $\\boldsymbol{\\Sigma}$ where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD." ] @@ -2693,10 +2590,7 @@ "cell_type": "code", "execution_count": 8, "id": "7558fe77", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -2727,9 +2621,7 @@ { "cell_type": "markdown", "id": "76fe51fe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "As you can see from this example, our own decomposition based on the SVD agrees with the pseudoinverse algorithm provided by **Numpy**." ] @@ -2737,9 +2629,7 @@ { "cell_type": "markdown", "id": "a07ef75d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Ridge and LASSO Regression\n", "\n", @@ -2750,9 +2640,7 @@ { "cell_type": "markdown", "id": "c46a77e6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -2762,9 +2650,7 @@ { "cell_type": "markdown", "id": "42a601ca", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or we can state it as" ] @@ -2772,9 +2658,7 @@ { "cell_type": "markdown", "id": "5a0276f1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -2785,9 +2669,7 @@ { "cell_type": "markdown", "id": "370dc4d5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have used the definition of a norm-2 vector, that is" ] @@ -2795,9 +2677,7 @@ { "cell_type": "markdown", "id": "34d0e5ca", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", @@ -2807,9 +2687,7 @@ { "cell_type": "markdown", "id": "0ab6c5d5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## From OLS to Ridge and Lasso\n", "\n", @@ -2822,9 +2700,7 @@ { "cell_type": "markdown", "id": "5391e856", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -2835,9 +2711,7 @@ { "cell_type": "markdown", "id": "7e9bfe02", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to the Ridge regression minimization problem where we\n", "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n", @@ -2849,9 +2723,7 @@ { "cell_type": "markdown", "id": "001098d5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -2861,9 +2733,7 @@ { "cell_type": "markdown", "id": "ce74d9dd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we have a new optimization equation" ] @@ -2871,9 +2741,7 @@ { "cell_type": "markdown", "id": "ca079697", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -2884,9 +2752,7 @@ { "cell_type": "markdown", "id": "f30e41c5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", "\n", @@ -2896,9 +2762,7 @@ { "cell_type": "markdown", "id": "7a0c725e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", @@ -2908,9 +2772,7 @@ { "cell_type": "markdown", "id": "a547eec2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Deriving the Ridge Regression Equations\n", "\n", @@ -2920,9 +2782,7 @@ { "cell_type": "markdown", "id": "2fa3b745", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n", @@ -2932,9 +2792,7 @@ { "cell_type": "markdown", "id": "38e0ff51", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and \n", "taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", @@ -2946,9 +2804,7 @@ { "cell_type": "markdown", "id": "c45e7f2d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -2958,9 +2814,7 @@ { "cell_type": "markdown", "id": "50fa8f58", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" ] @@ -2968,9 +2822,7 @@ { "cell_type": "markdown", "id": "d85be6dc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n", @@ -2980,9 +2832,7 @@ { "cell_type": "markdown", "id": "86cf0944", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $t$ a finite positive number." ] @@ -2990,9 +2840,7 @@ { "cell_type": "markdown", "id": "310119e6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Note on Scikit-Learn\n", "\n", @@ -3002,9 +2850,7 @@ { "cell_type": "markdown", "id": "2e0368ba", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -3014,9 +2860,7 @@ { "cell_type": "markdown", "id": "ad85d4a3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In our codes where we compare our own codes with **Scikit-Learn**, we do thus not include the $1/n$ factor in the cost function." ] @@ -3024,9 +2868,7 @@ { "cell_type": "markdown", "id": "87e390e2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Comparison with OLS\n", "When we compare this with the ordinary least squares result we have" @@ -3035,9 +2877,7 @@ { "cell_type": "markdown", "id": "066ceac7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -3047,9 +2887,7 @@ { "cell_type": "markdown", "id": "6199a3c1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", "\n", @@ -3063,9 +2901,7 @@ { "cell_type": "markdown", "id": "d15bddad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## SVD analysis\n", "\n", @@ -3076,9 +2912,7 @@ { "cell_type": "markdown", "id": "f0344746", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n", @@ -3088,9 +2922,7 @@ { "cell_type": "markdown", "id": "4c6e3314", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "For Ridge regression this becomes" ] @@ -3098,9 +2930,7 @@ { "cell_type": "markdown", "id": "6d6661bb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", @@ -3110,9 +2940,7 @@ { "cell_type": "markdown", "id": "f926966e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$ from the SVD of the matrix $\\boldsymbol{X}$." ] @@ -3120,9 +2948,7 @@ { "cell_type": "markdown", "id": "ce1238e5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Interpreting the Ridge results\n", "\n", @@ -3132,9 +2958,7 @@ { "cell_type": "markdown", "id": "b504e4f8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda} \\leq 1.\n", @@ -3144,9 +2968,7 @@ { "cell_type": "markdown", "id": "918c1b4e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Ridge regression finds the coordinates of $\\boldsymbol{y}$ with respect to the\n", "orthonormal basis $\\boldsymbol{U}$, it then shrinks the coordinates by\n", @@ -3160,9 +2982,7 @@ { "cell_type": "markdown", "id": "a98c5ccc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More interpretations\n", "\n", @@ -3172,9 +2992,7 @@ { "cell_type": "markdown", "id": "31cd8d63", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=(\\boldsymbol{X}^T\\boldsymbol{X})^{-1} =\\boldsymbol{I}.\n", @@ -3184,9 +3002,7 @@ { "cell_type": "markdown", "id": "25a9c1a3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In this case the standard OLS results in" ] @@ -3194,9 +3010,7 @@ { "cell_type": "markdown", "id": "8d4ce13a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{n-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n", @@ -3206,9 +3020,7 @@ { "cell_type": "markdown", "id": "e886fb1a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -3216,9 +3028,7 @@ { "cell_type": "markdown", "id": "8137aa75", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n", @@ -3228,9 +3038,7 @@ { "cell_type": "markdown", "id": "1ce00fcd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\\lambda$, and\n", "the Ridge estimator converges to zero when the hyperparameter goes to\n", @@ -3245,9 +3053,7 @@ { "cell_type": "markdown", "id": "ab1d80ea", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Deriving the Lasso Regression Equations\n", "\n", @@ -3257,9 +3063,7 @@ { "cell_type": "markdown", "id": "bdc69772", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -3269,9 +3073,7 @@ { "cell_type": "markdown", "id": "e18347d6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicity)" ] @@ -3279,9 +3081,7 @@ { "cell_type": "markdown", "id": "784f8670", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{d \\vert \\beta\\vert}{d \\beta}=\\mathrm{sgn}(\\beta)=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\-1 & \\beta < 0, \\end{array}\\right.\n", @@ -3291,9 +3091,7 @@ { "cell_type": "markdown", "id": "e4afe07e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we have that the derivative of the cost function is" ] @@ -3301,9 +3099,7 @@ { "cell_type": "markdown", "id": "80a0806b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-\\frac{2}{n}\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n", @@ -3313,9 +3109,7 @@ { "cell_type": "markdown", "id": "309a5909", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and reordering we have" ] @@ -3323,9 +3117,7 @@ { "cell_type": "markdown", "id": "6e2dc877", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\lambda sgn(\\boldsymbol{\\beta})=\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -3335,9 +3127,7 @@ { "cell_type": "markdown", "id": "8eedf54d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. We have absorbed the factor $2/n$ in a redefinition of the parameter $\\lambda$. We will solve this type of problems using libraries like **scikit-learn**." ] @@ -3345,9 +3135,7 @@ { "cell_type": "markdown", "id": "5c8f4ed5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression\n", "\n", @@ -3360,9 +3148,7 @@ { "cell_type": "markdown", "id": "87dcc8ab", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2,\n", @@ -3372,9 +3158,7 @@ { "cell_type": "markdown", "id": "0f33bcd4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and minimizing we have that" ] @@ -3382,9 +3166,7 @@ { "cell_type": "markdown", "id": "1c4af3e3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\beta}_i^{\\mathrm{OLS}} = y_i.\n", @@ -3394,9 +3176,7 @@ { "cell_type": "markdown", "id": "4a8dfde5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Ridge Regression\n", "\n", @@ -3406,9 +3186,7 @@ { "cell_type": "markdown", "id": "249d6888", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,\n", @@ -3418,9 +3196,7 @@ { "cell_type": "markdown", "id": "7d404154", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and minimizing we have that" ] @@ -3428,9 +3204,7 @@ { "cell_type": "markdown", "id": "6ba36335", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\beta}_i^{\\mathrm{Ridge}} = \\frac{y_i}{1+\\lambda}.\n", @@ -3440,9 +3214,7 @@ { "cell_type": "markdown", "id": "1faaacaa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Lasso Regression\n", "\n", @@ -3452,9 +3224,7 @@ { "cell_type": "markdown", "id": "f9e0ba9d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\beta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\beta_i^2},\n", @@ -3464,9 +3234,7 @@ { "cell_type": "markdown", "id": "a35d4c80", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and minimizing we have that" ] @@ -3474,9 +3242,7 @@ { "cell_type": "markdown", "id": "1f16424c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "-2\\sum_{i=0}^{p-1}(y_i-\\beta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\beta_i)}{\\vert\\beta_i\\vert}=0,\n", @@ -3486,9 +3252,7 @@ { "cell_type": "markdown", "id": "98648b73", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to" ] @@ -3496,9 +3260,7 @@ { "cell_type": "markdown", "id": "3b6f867b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n", @@ -3510,9 +3272,7 @@ { "cell_type": "markdown", "id": "2091f138", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Plotting these results shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\beta_i$ as function of $\\lambda$." ] @@ -3520,9 +3280,7 @@ { "cell_type": "markdown", "id": "0b807cc0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Yet another Example\n", "\n", @@ -3532,9 +3290,7 @@ { "cell_type": "markdown", "id": "ad4a7f3b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y}=\\begin{bmatrix}4 \\\\ 2 \\\\3\\end{bmatrix},\n", @@ -3544,9 +3300,7 @@ { "cell_type": "markdown", "id": "7bba656d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and our inputs as a $3\\times 2$ design matrix" ] @@ -3554,9 +3308,7 @@ { "cell_type": "markdown", "id": "390d10c2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}2 & 0\\\\ 0 & 1 \\\\ 0 & 0\\end{bmatrix},\n", @@ -3566,9 +3318,7 @@ { "cell_type": "markdown", "id": "46c76961", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "meaning that we have two features and two unknown parameters $\\beta_0$ and $\\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression." ] @@ -3576,9 +3326,7 @@ { "cell_type": "markdown", "id": "ee1d500c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The OLS case\n", "\n", @@ -3588,9 +3336,7 @@ { "cell_type": "markdown", "id": "2bab8609", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -3600,9 +3346,7 @@ { "cell_type": "markdown", "id": "de07f3b4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Inserting the above values we obtain that" ] @@ -3610,9 +3354,7 @@ { "cell_type": "markdown", "id": "d2242d19", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n", @@ -3622,9 +3364,7 @@ { "cell_type": "markdown", "id": "93d84418", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The code which implements this simpler case is presented after the discussion of Ridge and Lasso." ] @@ -3632,9 +3372,7 @@ { "cell_type": "markdown", "id": "35f4f55a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Ridge case\n", "\n", @@ -3644,9 +3382,7 @@ { "cell_type": "markdown", "id": "8ca42409", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -3656,9 +3392,7 @@ { "cell_type": "markdown", "id": "005fea6c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Inserting the above values we obtain that" ] @@ -3666,9 +3400,7 @@ { "cell_type": "markdown", "id": "29d39ca4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n", @@ -3678,9 +3410,7 @@ { "cell_type": "markdown", "id": "38685dd7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2$ via the parameter $\\lambda$.\n", "Let us for simplicity assume that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\beta$ and $\\lambda$.\n", @@ -3691,9 +3421,7 @@ { "cell_type": "markdown", "id": "a9f98459", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Writing the Cost Function\n", "\n", @@ -3703,9 +3431,7 @@ { "cell_type": "markdown", "id": "3dd626ff", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{\\beta}=\\begin{bmatrix} 2\\beta_0 \\\\ \\beta_1 \\\\0 \\end{bmatrix},\n", @@ -3715,9 +3441,7 @@ { "cell_type": "markdown", "id": "120f7f98", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\beta_0^2+\\beta_1^2),\n", @@ -3727,9 +3451,7 @@ { "cell_type": "markdown", "id": "85ff7e31", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and taking the derivative with respect to $\\beta_0$ we get" ] @@ -3737,9 +3459,7 @@ { "cell_type": "markdown", "id": "233c83b0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0=\\frac{8}{4+\\lambda},\n", @@ -3749,9 +3469,7 @@ { "cell_type": "markdown", "id": "2de78886", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and for $\\beta_1$ we obtain" ] @@ -3759,9 +3477,7 @@ { "cell_type": "markdown", "id": "6eb69fcf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_1=\\frac{2}{1+\\lambda},\n", @@ -3771,9 +3487,7 @@ { "cell_type": "markdown", "id": "4446e11a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Using the constraint for $\\beta_0^2+\\beta_1^2=1$ we can constrain $\\lambda$ by solving" ] @@ -3781,9 +3495,7 @@ { "cell_type": "markdown", "id": "21808e0c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\frac{8}{4+\\lambda}\\right)^2+\\left(\\frac{2}{1+\\lambda}\\right)^2=1,\n", @@ -3793,9 +3505,7 @@ { "cell_type": "markdown", "id": "139b0cb7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which gives $\\lambda=4.571$ and $\\beta_0=0.933$ and $\\beta_1=0.359$." ] @@ -3803,9 +3513,7 @@ { "cell_type": "markdown", "id": "5c1d1cbd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Lasso case\n", "\n", @@ -3816,9 +3524,7 @@ { "cell_type": "markdown", "id": "2e7a4cc8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\vert\\beta_0\\vert+\\vert\\beta_1\\vert),\n", @@ -3828,9 +3534,7 @@ { "cell_type": "markdown", "id": "65c7221f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-4(4-2\\beta_0)+\\lambda\\mathrm{sgn}(\\beta_0)=0,\n", @@ -3840,9 +3544,7 @@ { "cell_type": "markdown", "id": "b1bb46c0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -3850,9 +3552,7 @@ { "cell_type": "markdown", "id": "5b7c39d2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}(\\beta_1)=0.\n", @@ -3862,9 +3562,7 @@ { "cell_type": "markdown", "id": "a224ebff", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We have now four cases to solve besides the trivial cases $\\beta_0$ and/or $\\beta_1$ are zero, namely\n", "1. $\\beta_0 > 0$ and $\\beta_1 > 0$,\n", @@ -3879,9 +3577,7 @@ { "cell_type": "markdown", "id": "b1b191c0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The first Case\n", "\n", @@ -3891,9 +3587,7 @@ { "cell_type": "markdown", "id": "e314ddd2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "-4(4-2\\beta_0)+\\lambda=0,\n", @@ -3903,9 +3597,7 @@ { "cell_type": "markdown", "id": "2f23eb08", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -3913,9 +3605,7 @@ { "cell_type": "markdown", "id": "cf235558", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "-2(2-\\beta_1)+\\lambda=0.\n", @@ -3925,9 +3615,7 @@ { "cell_type": "markdown", "id": "de9dbcdd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which yields" ] @@ -3935,9 +3623,7 @@ { "cell_type": "markdown", "id": "a3703fa0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0=\\frac{16+\\lambda}{8},\n", @@ -3947,9 +3633,7 @@ { "cell_type": "markdown", "id": "5706ae9d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -3957,9 +3641,7 @@ { "cell_type": "markdown", "id": "29896cf0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_1=\\frac{4+\\lambda}{2}.\n", @@ -3969,9 +3651,7 @@ { "cell_type": "markdown", "id": "25dda05a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Using the constraint on $\\beta_0$ and $\\beta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you." ] @@ -3979,9 +3659,7 @@ { "cell_type": "markdown", "id": "f778ea31", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple code for solving the above problem\n", "\n", @@ -3994,10 +3672,7 @@ "cell_type": "code", "execution_count": 9, "id": "fc3f8367", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import os\n", @@ -4054,9 +3729,7 @@ { "cell_type": "markdown", "id": "10dfce4e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see here that we reach a plateau. What is actually happening?" ] @@ -4064,9 +3737,7 @@ { "cell_type": "markdown", "id": "d9e60da2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## With Lasso Regression" ] @@ -4075,10 +3746,7 @@ "cell_type": "code", "execution_count": 10, "id": "e33316d6", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import os\n", @@ -4142,9 +3810,7 @@ { "cell_type": "markdown", "id": "3d1319b0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Another Example, now with a polynomial fit" ] @@ -4153,10 +3819,7 @@ "cell_type": "code", "execution_count": 11, "id": "74957a76", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import os\n", @@ -4240,7 +3903,25 @@ ] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.18" + } + }, "nbformat": 4, "nbformat_minor": 5 }