diff --git a/doc/LectureNotes/_build/.doctrees/environment.pickle b/doc/LectureNotes/_build/.doctrees/environment.pickle
index b851d2297..12f8a63dd 100644
Binary files a/doc/LectureNotes/_build/.doctrees/environment.pickle and b/doc/LectureNotes/_build/.doctrees/environment.pickle differ
diff --git a/doc/LectureNotes/_build/.doctrees/week37.doctree b/doc/LectureNotes/_build/.doctrees/week37.doctree
index c4c91427d..1b2e31d89 100644
Binary files a/doc/LectureNotes/_build/.doctrees/week37.doctree and b/doc/LectureNotes/_build/.doctrees/week37.doctree differ
diff --git a/doc/LectureNotes/_build/html/_images/week37_121_0.png b/doc/LectureNotes/_build/html/_images/week37_121_0.png
new file mode 100644
index 000000000..843d99fe9
Binary files /dev/null and b/doc/LectureNotes/_build/html/_images/week37_121_0.png differ
diff --git a/doc/LectureNotes/_build/html/_images/week37_137_1.png b/doc/LectureNotes/_build/html/_images/week37_137_1.png
new file mode 100644
index 000000000..a1e369a41
Binary files /dev/null and b/doc/LectureNotes/_build/html/_images/week37_137_1.png differ
diff --git a/doc/LectureNotes/_build/html/_images/week37_139_2.png b/doc/LectureNotes/_build/html/_images/week37_139_2.png
new file mode 100644
index 000000000..9ebeeb751
Binary files /dev/null and b/doc/LectureNotes/_build/html/_images/week37_139_2.png differ
diff --git a/doc/LectureNotes/_build/html/_images/week37_142_1.png b/doc/LectureNotes/_build/html/_images/week37_142_1.png
new file mode 100644
index 000000000..a8501ad62
Binary files /dev/null and b/doc/LectureNotes/_build/html/_images/week37_142_1.png differ
diff --git a/doc/LectureNotes/_build/html/_images/week37_146_0.png b/doc/LectureNotes/_build/html/_images/week37_146_0.png
new file mode 100644
index 000000000..fc9e0e560
Binary files /dev/null and b/doc/LectureNotes/_build/html/_images/week37_146_0.png differ
diff --git a/doc/LectureNotes/_build/html/_images/week37_148_9.png b/doc/LectureNotes/_build/html/_images/week37_148_9.png
new file mode 100644
index 000000000..9fa6833f5
Binary files /dev/null and b/doc/LectureNotes/_build/html/_images/week37_148_9.png differ
diff --git a/doc/LectureNotes/_build/html/_images/week37_151_1.png b/doc/LectureNotes/_build/html/_images/week37_151_1.png
new file mode 100644
index 000000000..03ff699cc
Binary files /dev/null and b/doc/LectureNotes/_build/html/_images/week37_151_1.png differ
diff --git a/doc/LectureNotes/_build/html/_sources/week37.ipynb b/doc/LectureNotes/_build/html/_sources/week37.ipynb
index 94e3e70ef..c24beccbd 100644
--- a/doc/LectureNotes/_build/html/_sources/week37.ipynb
+++ b/doc/LectureNotes/_build/html/_sources/week37.ipynb
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
- "id": "672fea2a",
+ "id": "7c0ddab1",
"metadata": {
"editable": true
},
@@ -14,13 +14,13 @@
},
{
"cell_type": "markdown",
- "id": "479a6913",
+ "id": "6c033bf4",
"metadata": {
"editable": true
},
"source": [
"# Week 37: Statistical interpretations and Resampling Methods\n",
- "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n",
+ "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo, Norway\n",
"\n",
"Date: **September 9, 2024**\n",
"\n",
@@ -29,419 +29,69 @@
},
{
"cell_type": "markdown",
- "id": "10b8f456",
+ "id": "e4cd5fa1",
"metadata": {
"editable": true
},
"source": [
- "## Plans for week 37\n",
+ "## Plans for week 37, lecture Monday\n",
"\n",
"**Material for the lecture on Monday September 9.**\n",
"\n",
"\n",
"\n",
- " * Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff\n",
+ " * Statistical interpretation of Ridge and Lasso regression, see also slides from last week\n",
"\n",
- " * Statistical interpretation of Ridge and Lasso regression\n",
+ " * Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff (this may partly be discussed during the exercise sessions as well.\n",
"\n",
" * Readings and Videos:\n",
"\n",
- " * Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap). \n",
+ " * Raschka et al, pages 175-192\n",
+ "\n",
+ " * Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap). See .\n",
"\n",
" * [Video on cross validation](https://www.youtube.com/watch?v=fSytzGwwBVw)\n",
"\n",
" * [Video on Bootstrapping](https://www.youtube.com/watch?v=Xz0x-8-cgaQ)\n",
"\n",
- " * [Video on bias-variance tradeoff](https://www.youtube.com/watch?v=EuBBz3bI-aA)\n",
+ " * [Video on bias-variance tradeoff](https://www.youtube.com/watch?v=EuBBz3bI-aA)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bf80a276",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Plans for week 37, lab sessions\n",
"\n",
- "**Material for the active learning sessions on Tuesday and Wednesday.**\n",
+ "**Material for the lab sessions on Tuesday and Wednesday.**\n",
"\n",
" * Calculations of expectation values\n",
"\n",
+ " * Discussion of resampling techniques\n",
+ "\n",
" * Exercise set for week 37\n",
"\n",
" * Work on project 1\n",
"\n",
- " * See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session. This note is added at the end of these slides.\n",
- "\n",
" * For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended."
]
},
{
"cell_type": "markdown",
- "id": "f7e43b1b",
+ "id": "0e1f0a90",
"metadata": {
"editable": true
},
"source": [
- "## Material from last week and relevant for the weekly exercises"
+ "## Material for lecture Monday September 9"
]
},
{
"cell_type": "markdown",
- "id": "af6a447a",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Linking the regression analysis with a statistical interpretation\n",
- "\n",
- "We will now couple the discussions of ordinary least squares, Ridge\n",
- "and Lasso regression with a statistical interpretation, that is we\n",
- "move from a linear algebra analysis to a statistical analysis. In\n",
- "particular, we will focus on what the regularization terms can result\n",
- "in. We will amongst other things show that the regularization\n",
- "parameter can reduce considerably the variance of the parameters\n",
- "$\\beta$.\n",
- "\n",
- "The\n",
- "advantage of doing linear regression is that we actually end up with\n",
- "analytical expressions for several statistical quantities. \n",
- "Standard least squares and Ridge regression allow us to\n",
- "derive quantities like the variance and other expectation values in a\n",
- "rather straightforward way.\n",
- "\n",
- "It is assumed that $\\varepsilon_i\n",
- "\\sim \\mathcal{N}(0, \\sigma^2)$ and the $\\varepsilon_{i}$ are\n",
- "independent, i.e.:"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b6b7ea38",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\begin{align*} \n",
- "\\mbox{Cov}(\\varepsilon_{i_1},\n",
- "\\varepsilon_{i_2}) & = \\left\\{ \\begin{array}{lcc} \\sigma^2 & \\mbox{if}\n",
- "& i_1 = i_2, \\\\ 0 & \\mbox{if} & i_1 \\not= i_2. \\end{array} \\right.\n",
- "\\end{align*}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e53d8d41",
- "metadata": {
- "editable": true
- },
- "source": [
- "The randomness of $\\varepsilon_i$ implies that\n",
- "$\\mathbf{y}_i$ is also a random variable. In particular,\n",
- "$\\mathbf{y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n",
- "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\beta}$ is a\n",
- "non-random scalar. To specify the parameters of the distribution of\n",
- "$\\mathbf{y}_i$ we need to calculate its first two moments. \n",
- "\n",
- "Recall that $\\boldsymbol{X}$ is a matrix of dimensionality $n\\times p$. The\n",
- "notation above $\\mathbf{X}_{i,\\ast}$ means that we are looking at the\n",
- "row number $i$ and perform a sum over all values $p$."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "401644e7",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Assumptions made\n",
- "\n",
- "The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)\n",
- "that there exists a function $f(\\boldsymbol{x})$ and a normal distributed error $\\boldsymbol{\\varepsilon}\\sim \\mathcal{N}(0, \\sigma^2)$\n",
- "which describe our data"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0603387c",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "eb2b6edf",
- "metadata": {
- "editable": true
- },
- "source": [
- "We approximate this function with our model from the solution of the linear regression equations, that is our\n",
- "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we want to minimize $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, our MSE, with"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f1a08194",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ad40fa0b",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Expectation value and variance\n",
- "\n",
- "We can calculate the expectation value of $\\boldsymbol{y}$ for a given element $i$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d3a4907d",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\begin{align*} \n",
- "\\mathbb{E}(y_i) & =\n",
- "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}) + \\mathbb{E}(\\varepsilon_i)\n",
- "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n",
- "\\end{align*}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d7908c9c",
- "metadata": {
- "editable": true
- },
- "source": [
- "while\n",
- "its variance is"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2528f74e",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n",
- "- \\mathbb{E}(y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( y_i^2 ) -\n",
- "[\\mathbb{E}(y_i)]^2 \\\\ & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n",
- "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \\\\ &\n",
- "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2 \\varepsilon_i\n",
- "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
- "\\ast} \\, \\beta)^2 \\\\ & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2\n",
- "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} +\n",
- "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \n",
- "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n",
- "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2. \n",
- "\\end{align*}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "525f3a59",
- "metadata": {
- "editable": true
- },
- "source": [
- "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
- "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD)."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ead1d419",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Expectation value and variance for $\\boldsymbol{\\beta}$\n",
- "\n",
- "With the OLS expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}}$ we can evaluate the expectation value"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e471f7ab",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mathbb{E}(\\boldsymbol{\\hat{\\beta}}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7886e97f",
- "metadata": {
- "editable": true
- },
- "source": [
- "This means that the estimator of the regression parameters is unbiased.\n",
- "\n",
- "We can also calculate the variance\n",
- "\n",
- "The variance of the optimal value $\\boldsymbol{\\hat{\\beta}}$ is"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "49c32ca5",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\begin{eqnarray*}\n",
- "\\mbox{Var}(\\boldsymbol{\\hat{\\beta}}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n",
- "\\\\\n",
- "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}]^{T} \\}\n",
- "\\\\\n",
- "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "% \\\\\n",
- "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} \\, \\mathbf{y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "% \\\\\n",
- "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{y} \\, \\mathbf{y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "\\\\\n",
- "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "% \\\\\n",
- "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^T \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
- "% \\\\\n",
- "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\boldsymbol{\\beta}^T\n",
- "\\\\\n",
- "& = & \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "\\, \\, \\, = \\, \\, \\, \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1},\n",
- "\\end{eqnarray*}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9bd27980",
- "metadata": {
- "editable": true
- },
- "source": [
- "where we have used that $\\mathbb{E} (\\mathbf{y} \\mathbf{y}^{T}) =\n",
- "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n",
- "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n",
- "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n",
- "variance of the estimate of the $j$-th regression coefficient:\n",
- "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $. This may be used to\n",
- "construct a confidence interval for the estimates.\n",
- "\n",
- "In a similar way, we can obtain analytical expressions for say the\n",
- "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n",
- "when we employ Ridge regression, allowing us again to define a confidence interval. \n",
- "\n",
- "It is rather straightforward to show that"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "184f9c3b",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d0f46be1",
- "metadata": {
- "editable": true
- },
- "source": [
- "We see clearly that \n",
- "$\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}$ for any $\\lambda > 0$.\n",
- "\n",
- "We can also compute the variance as"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "03e3ac5f",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "13ec2524",
- "metadata": {
- "editable": true
- },
- "source": [
- "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n",
- "\n",
- "With this, we can compute the difference"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "58c3f236",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}]-\\mbox{Var}(\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "bfcfc2a0",
- "metadata": {
- "editable": true
- },
- "source": [
- "The difference is non-negative definite since each component of the\n",
- "matrix product is non-negative definite. \n",
- "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. \n",
- "\n",
- "For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f3a9f084",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Material for lecture Thursday September 14"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a4338936",
+ "id": "1b290d64",
"metadata": {
"editable": true
},
@@ -464,7 +114,7 @@
},
{
"cell_type": "markdown",
- "id": "d7566301",
+ "id": "ceed3a82",
"metadata": {
"editable": true
},
@@ -476,7 +126,7 @@
},
{
"cell_type": "markdown",
- "id": "c8af7fae",
+ "id": "1d7e8f98",
"metadata": {
"editable": true
},
@@ -489,7 +139,7 @@
},
{
"cell_type": "markdown",
- "id": "959e33e8",
+ "id": "58722d7d",
"metadata": {
"editable": true
},
@@ -501,7 +151,7 @@
},
{
"cell_type": "markdown",
- "id": "c847b78b",
+ "id": "45120edc",
"metadata": {
"editable": true
},
@@ -513,7 +163,7 @@
},
{
"cell_type": "markdown",
- "id": "75c188ba",
+ "id": "4f3b680a",
"metadata": {
"editable": true
},
@@ -525,7 +175,7 @@
},
{
"cell_type": "markdown",
- "id": "60b89ba3",
+ "id": "08489797",
"metadata": {
"editable": true
},
@@ -536,7 +186,7 @@
},
{
"cell_type": "markdown",
- "id": "fd5d5a54",
+ "id": "a1c81f06",
"metadata": {
"editable": true
},
@@ -548,7 +198,7 @@
},
{
"cell_type": "markdown",
- "id": "56f25582",
+ "id": "4e092776",
"metadata": {
"editable": true
},
@@ -559,7 +209,7 @@
},
{
"cell_type": "markdown",
- "id": "9974b303",
+ "id": "79e5358a",
"metadata": {
"editable": true
},
@@ -571,7 +221,7 @@
},
{
"cell_type": "markdown",
- "id": "4112e053",
+ "id": "e5a4f865",
"metadata": {
"editable": true
},
@@ -581,7 +231,7 @@
},
{
"cell_type": "markdown",
- "id": "7be4a14c",
+ "id": "10b81ba7",
"metadata": {
"editable": true
},
@@ -612,7 +262,7 @@
},
{
"cell_type": "markdown",
- "id": "35078f48",
+ "id": "3349cb27",
"metadata": {
"editable": true
},
@@ -624,7 +274,7 @@
},
{
"cell_type": "markdown",
- "id": "cb25ead1",
+ "id": "0fd1502e",
"metadata": {
"editable": true
},
@@ -636,7 +286,7 @@
},
{
"cell_type": "markdown",
- "id": "5cc80036",
+ "id": "43cba0e8",
"metadata": {
"editable": true
},
@@ -646,7 +296,7 @@
},
{
"cell_type": "markdown",
- "id": "8ebc6099",
+ "id": "d6f4e7dc",
"metadata": {
"editable": true
},
@@ -658,7 +308,7 @@
},
{
"cell_type": "markdown",
- "id": "3160dff4",
+ "id": "50c2d673",
"metadata": {
"editable": true
},
@@ -668,7 +318,7 @@
},
{
"cell_type": "markdown",
- "id": "8a3aaa69",
+ "id": "30315a10",
"metadata": {
"editable": true
},
@@ -680,7 +330,7 @@
},
{
"cell_type": "markdown",
- "id": "7af6708c",
+ "id": "fd8cdf51",
"metadata": {
"editable": true
},
@@ -690,7 +340,7 @@
},
{
"cell_type": "markdown",
- "id": "c82e095e",
+ "id": "7323296b",
"metadata": {
"editable": true
},
@@ -702,7 +352,7 @@
},
{
"cell_type": "markdown",
- "id": "9431eca4",
+ "id": "e21334ab",
"metadata": {
"editable": true
},
@@ -712,7 +362,7 @@
},
{
"cell_type": "markdown",
- "id": "1db6c222",
+ "id": "82974124",
"metadata": {
"editable": true
},
@@ -732,7 +382,7 @@
},
{
"cell_type": "markdown",
- "id": "98c457ce",
+ "id": "e053e43c",
"metadata": {
"editable": true
},
@@ -744,7 +394,7 @@
},
{
"cell_type": "markdown",
- "id": "b6a3893f",
+ "id": "94098e13",
"metadata": {
"editable": true
},
@@ -754,7 +404,7 @@
},
{
"cell_type": "markdown",
- "id": "17f0e52e",
+ "id": "27625b1f",
"metadata": {
"editable": true
},
@@ -766,7 +416,7 @@
},
{
"cell_type": "markdown",
- "id": "c2bfaea6",
+ "id": "8f3cc8b4",
"metadata": {
"editable": true
},
@@ -778,7 +428,7 @@
},
{
"cell_type": "markdown",
- "id": "377eefdf",
+ "id": "d7d5464d",
"metadata": {
"editable": true
},
@@ -790,7 +440,7 @@
},
{
"cell_type": "markdown",
- "id": "6da6253e",
+ "id": "c292afee",
"metadata": {
"editable": true
},
@@ -802,7 +452,7 @@
},
{
"cell_type": "markdown",
- "id": "e5906b3e",
+ "id": "cdbbd444",
"metadata": {
"editable": true
},
@@ -814,7 +464,7 @@
},
{
"cell_type": "markdown",
- "id": "792d292c",
+ "id": "100a8014",
"metadata": {
"editable": true
},
@@ -826,7 +476,7 @@
},
{
"cell_type": "markdown",
- "id": "c3bc0674",
+ "id": "d2e40c4d",
"metadata": {
"editable": true
},
@@ -838,7 +488,7 @@
},
{
"cell_type": "markdown",
- "id": "0faadf3f",
+ "id": "1e48456e",
"metadata": {
"editable": true
},
@@ -850,7 +500,7 @@
},
{
"cell_type": "markdown",
- "id": "a2d478a7",
+ "id": "38ced5b6",
"metadata": {
"editable": true
},
@@ -860,7 +510,7 @@
},
{
"cell_type": "markdown",
- "id": "d51ac6ce",
+ "id": "1163c07b",
"metadata": {
"editable": true
},
@@ -872,7 +522,7 @@
},
{
"cell_type": "markdown",
- "id": "91d489d9",
+ "id": "becc087f",
"metadata": {
"editable": true
},
@@ -882,7 +532,7 @@
},
{
"cell_type": "markdown",
- "id": "ebe26b44",
+ "id": "5ca2b09d",
"metadata": {
"editable": true
},
@@ -901,7 +551,7 @@
},
{
"cell_type": "markdown",
- "id": "a67660e2",
+ "id": "6572c77d",
"metadata": {
"editable": true
},
@@ -922,7 +572,7 @@
},
{
"cell_type": "markdown",
- "id": "40b7eac0",
+ "id": "2a3f65e5",
"metadata": {
"editable": true
},
@@ -934,7 +584,7 @@
},
{
"cell_type": "markdown",
- "id": "16a3b6ea",
+ "id": "2d264f77",
"metadata": {
"editable": true
},
@@ -945,7 +595,7 @@
},
{
"cell_type": "markdown",
- "id": "f41ccc6d",
+ "id": "06b8a13c",
"metadata": {
"editable": true
},
@@ -958,7 +608,7 @@
},
{
"cell_type": "markdown",
- "id": "ae81305b",
+ "id": "6d958ed3",
"metadata": {
"editable": true
},
@@ -970,7 +620,7 @@
},
{
"cell_type": "markdown",
- "id": "f4829b81",
+ "id": "6330f3f2",
"metadata": {
"editable": true
},
@@ -980,7 +630,7 @@
},
{
"cell_type": "markdown",
- "id": "812273af",
+ "id": "ed6a9f67",
"metadata": {
"editable": true
},
@@ -992,7 +642,7 @@
},
{
"cell_type": "markdown",
- "id": "31ecba0e",
+ "id": "de42ab8a",
"metadata": {
"editable": true
},
@@ -1002,7 +652,7 @@
},
{
"cell_type": "markdown",
- "id": "ca6682ec",
+ "id": "4f36b126",
"metadata": {
"editable": true
},
@@ -1014,7 +664,7 @@
},
{
"cell_type": "markdown",
- "id": "662eaffb",
+ "id": "0a7eef00",
"metadata": {
"editable": true
},
@@ -1024,7 +674,7 @@
},
{
"cell_type": "markdown",
- "id": "08d5551d",
+ "id": "29940f94",
"metadata": {
"editable": true
},
@@ -1038,7 +688,7 @@
},
{
"cell_type": "markdown",
- "id": "03f3de9c",
+ "id": "f2204679",
"metadata": {
"editable": true
},
@@ -1050,7 +700,7 @@
},
{
"cell_type": "markdown",
- "id": "c57ace49",
+ "id": "d3f4597f",
"metadata": {
"editable": true
},
@@ -1060,7 +710,7 @@
},
{
"cell_type": "markdown",
- "id": "cb014b2b",
+ "id": "c9e9ca01",
"metadata": {
"editable": true
},
@@ -1072,7 +722,7 @@
},
{
"cell_type": "markdown",
- "id": "1054d47e",
+ "id": "1c3062b6",
"metadata": {
"editable": true
},
@@ -1082,7 +732,7 @@
},
{
"cell_type": "markdown",
- "id": "1485d44c",
+ "id": "b455f707",
"metadata": {
"editable": true
},
@@ -1094,7 +744,7 @@
},
{
"cell_type": "markdown",
- "id": "f457f909",
+ "id": "5ac8fdd3",
"metadata": {
"editable": true
},
@@ -1104,7 +754,7 @@
},
{
"cell_type": "markdown",
- "id": "59447b2a",
+ "id": "827927db",
"metadata": {
"editable": true
},
@@ -1116,17 +766,17 @@
},
{
"cell_type": "markdown",
- "id": "4fe26100",
+ "id": "0c740a13",
"metadata": {
"editable": true
},
"source": [
- "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$!"
+ "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta})$!"
]
},
{
"cell_type": "markdown",
- "id": "d73c2d9c",
+ "id": "a855fc3c",
"metadata": {
"editable": true
},
@@ -1142,7 +792,7 @@
},
{
"cell_type": "markdown",
- "id": "8cdf2c97",
+ "id": "9dda8b9e",
"metadata": {
"editable": true
},
@@ -1154,7 +804,7 @@
},
{
"cell_type": "markdown",
- "id": "7272ef9f",
+ "id": "d160205c",
"metadata": {
"editable": true
},
@@ -1164,7 +814,7 @@
},
{
"cell_type": "markdown",
- "id": "0dc5b2b7",
+ "id": "1b7501a6",
"metadata": {
"editable": true
},
@@ -1176,7 +826,7 @@
},
{
"cell_type": "markdown",
- "id": "b6c8c684",
+ "id": "d75a8c65",
"metadata": {
"editable": true
},
@@ -1189,7 +839,7 @@
},
{
"cell_type": "markdown",
- "id": "f681ed50",
+ "id": "c9f718df",
"metadata": {
"editable": true
},
@@ -1201,7 +851,7 @@
},
{
"cell_type": "markdown",
- "id": "5c3bc898",
+ "id": "b07b6425",
"metadata": {
"editable": true
},
@@ -1211,7 +861,7 @@
},
{
"cell_type": "markdown",
- "id": "a3cea046",
+ "id": "215a8730",
"metadata": {
"editable": true
},
@@ -1223,7 +873,7 @@
},
{
"cell_type": "markdown",
- "id": "68276410",
+ "id": "7013d0f0",
"metadata": {
"editable": true
},
@@ -1233,7 +883,7 @@
},
{
"cell_type": "markdown",
- "id": "dfd55611",
+ "id": "d56b4ce0",
"metadata": {
"editable": true
},
@@ -1245,7 +895,7 @@
},
{
"cell_type": "markdown",
- "id": "83656dd8",
+ "id": "f9c6b14d",
"metadata": {
"editable": true
},
@@ -1257,7 +907,7 @@
},
{
"cell_type": "markdown",
- "id": "b62e29fd",
+ "id": "0bde16cb",
"metadata": {
"editable": true
},
@@ -1267,7 +917,7 @@
},
{
"cell_type": "markdown",
- "id": "1de8855c",
+ "id": "0a2869fc",
"metadata": {
"editable": true
},
@@ -1279,7 +929,7 @@
},
{
"cell_type": "markdown",
- "id": "212152f1",
+ "id": "75b83b9b",
"metadata": {
"editable": true
},
@@ -1291,7 +941,7 @@
},
{
"cell_type": "markdown",
- "id": "afb042c4",
+ "id": "1e76d916",
"metadata": {
"editable": true
},
@@ -1303,7 +953,7 @@
},
{
"cell_type": "markdown",
- "id": "ab1fd3c5",
+ "id": "09dfe26d",
"metadata": {
"editable": true
},
@@ -1313,7 +963,7 @@
},
{
"cell_type": "markdown",
- "id": "43979108",
+ "id": "4037962f",
"metadata": {
"editable": true
},
@@ -1325,7 +975,7 @@
},
{
"cell_type": "markdown",
- "id": "96622b66",
+ "id": "39a85ba0",
"metadata": {
"editable": true
},
@@ -1335,7 +985,7 @@
},
{
"cell_type": "markdown",
- "id": "e17a5b6d",
+ "id": "b263a1fe",
"metadata": {
"editable": true
},
@@ -1354,7 +1004,7 @@
},
{
"cell_type": "markdown",
- "id": "24e35401",
+ "id": "41730165",
"metadata": {
"editable": true
},
@@ -1382,7 +1032,7 @@
},
{
"cell_type": "markdown",
- "id": "a37f1d06",
+ "id": "4076cee3",
"metadata": {
"editable": true
},
@@ -1408,7 +1058,7 @@
},
{
"cell_type": "markdown",
- "id": "58f54a23",
+ "id": "11139cb3",
"metadata": {
"editable": true
},
@@ -1425,7 +1075,7 @@
},
{
"cell_type": "markdown",
- "id": "6ed7c809",
+ "id": "b2bead9b",
"metadata": {
"editable": true
},
@@ -1445,7 +1095,7 @@
},
{
"cell_type": "markdown",
- "id": "bed9bb2f",
+ "id": "8df8364e",
"metadata": {
"editable": true
},
@@ -1474,7 +1124,7 @@
},
{
"cell_type": "markdown",
- "id": "2cafa0a3",
+ "id": "6b5273f1",
"metadata": {
"editable": true
},
@@ -1499,7 +1149,7 @@
},
{
"cell_type": "markdown",
- "id": "fefbb980",
+ "id": "a260b371",
"metadata": {
"editable": true
},
@@ -1519,7 +1169,7 @@
},
{
"cell_type": "markdown",
- "id": "4560153d",
+ "id": "2ff1e63c",
"metadata": {
"editable": true
},
@@ -1531,7 +1181,7 @@
},
{
"cell_type": "markdown",
- "id": "ffcbdd5a",
+ "id": "ab4b2e8c",
"metadata": {
"editable": true
},
@@ -1541,7 +1191,7 @@
},
{
"cell_type": "markdown",
- "id": "d055ea55",
+ "id": "85b6fb62",
"metadata": {
"editable": true
},
@@ -1556,7 +1206,7 @@
},
{
"cell_type": "markdown",
- "id": "004f0161",
+ "id": "42449344",
"metadata": {
"editable": true
},
@@ -1569,7 +1219,7 @@
},
{
"cell_type": "markdown",
- "id": "039f4ad5",
+ "id": "35e4f511",
"metadata": {
"editable": true
},
@@ -1582,7 +1232,7 @@
},
{
"cell_type": "markdown",
- "id": "413459f1",
+ "id": "c787234f",
"metadata": {
"editable": true
},
@@ -1594,7 +1244,7 @@
},
{
"cell_type": "markdown",
- "id": "bef022c8",
+ "id": "183711dc",
"metadata": {
"editable": true
},
@@ -1607,7 +1257,7 @@
},
{
"cell_type": "markdown",
- "id": "3d53693d",
+ "id": "2351078d",
"metadata": {
"editable": true
},
@@ -1618,7 +1268,7 @@
},
{
"cell_type": "markdown",
- "id": "e682c32a",
+ "id": "408ba3b5",
"metadata": {
"editable": true
},
@@ -1632,7 +1282,7 @@
},
{
"cell_type": "markdown",
- "id": "f44053ea",
+ "id": "3ea79849",
"metadata": {
"editable": true
},
@@ -1642,7 +1292,7 @@
},
{
"cell_type": "markdown",
- "id": "f10de930",
+ "id": "69439aad",
"metadata": {
"editable": true
},
@@ -1656,7 +1306,7 @@
},
{
"cell_type": "markdown",
- "id": "1f91402a",
+ "id": "e6c29fd8",
"metadata": {
"editable": true
},
@@ -1669,7 +1319,7 @@
},
{
"cell_type": "markdown",
- "id": "2d7a8e36",
+ "id": "4bb52549",
"metadata": {
"editable": true
},
@@ -1682,7 +1332,7 @@
},
{
"cell_type": "markdown",
- "id": "4c756ca2",
+ "id": "3430d548",
"metadata": {
"editable": true
},
@@ -1692,7 +1342,7 @@
},
{
"cell_type": "markdown",
- "id": "908f782e",
+ "id": "9ba636b3",
"metadata": {
"editable": true
},
@@ -1705,7 +1355,7 @@
},
{
"cell_type": "markdown",
- "id": "f12152da",
+ "id": "cb19e98f",
"metadata": {
"editable": true
},
@@ -1715,7 +1365,7 @@
},
{
"cell_type": "markdown",
- "id": "85dd290d",
+ "id": "3cc840d7",
"metadata": {
"editable": true
},
@@ -1728,7 +1378,7 @@
},
{
"cell_type": "markdown",
- "id": "57631d10",
+ "id": "a5062213",
"metadata": {
"editable": true
},
@@ -1740,7 +1390,7 @@
},
{
"cell_type": "markdown",
- "id": "2d4d5ccb",
+ "id": "8d555244",
"metadata": {
"editable": true
},
@@ -1759,7 +1409,7 @@
},
{
"cell_type": "markdown",
- "id": "a29c8551",
+ "id": "c4356604",
"metadata": {
"editable": true
},
@@ -1772,7 +1422,7 @@
},
{
"cell_type": "markdown",
- "id": "6509ba36",
+ "id": "09dc08a1",
"metadata": {
"editable": true
},
@@ -1784,7 +1434,7 @@
},
{
"cell_type": "markdown",
- "id": "733307b1",
+ "id": "dab5056a",
"metadata": {
"editable": true
},
@@ -1797,7 +1447,7 @@
},
{
"cell_type": "markdown",
- "id": "a69b7331",
+ "id": "8a1f8583",
"metadata": {
"editable": true
},
@@ -1817,7 +1467,7 @@
},
{
"cell_type": "markdown",
- "id": "ced659da",
+ "id": "1be08ebb",
"metadata": {
"editable": true
},
@@ -1831,16 +1481,16 @@
"With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ we found \n",
"$\\mathbb{E}(\\boldsymbol{\\beta}) = \\boldsymbol{\\beta}$, which means that the estimator of the regression parameters is unbiased.\n",
"\n",
- "We found also that the variance of the estimate of the $j$-th regression coefficient is\n",
+ "In the exercises this week we show that the variance of the estimate of the $j$-th regression coefficient is\n",
"$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $.\n",
"\n",
- "This quantity will be used to\n",
+ "This quantity can be used to\n",
"construct a confidence interval for the estimates."
]
},
{
"cell_type": "markdown",
- "id": "2c0d0a71",
+ "id": "2a368353",
"metadata": {
"editable": true
},
@@ -1855,7 +1505,7 @@
},
{
"cell_type": "markdown",
- "id": "d3a4d693",
+ "id": "6fee4a1a",
"metadata": {
"editable": true
},
@@ -1867,7 +1517,7 @@
},
{
"cell_type": "markdown",
- "id": "3ed3d852",
+ "id": "1a97c9ef",
"metadata": {
"editable": true
},
@@ -1887,7 +1537,7 @@
},
{
"cell_type": "markdown",
- "id": "e826e70b",
+ "id": "0a70afb2",
"metadata": {
"editable": true
},
@@ -1907,7 +1557,7 @@
},
{
"cell_type": "markdown",
- "id": "10b5d4d5",
+ "id": "dc32eaea",
"metadata": {
"editable": true
},
@@ -1931,7 +1581,7 @@
},
{
"cell_type": "markdown",
- "id": "1645ce3c",
+ "id": "016d0ab3",
"metadata": {
"editable": true
},
@@ -1952,7 +1602,7 @@
},
{
"cell_type": "markdown",
- "id": "4986483a",
+ "id": "3f371f0d",
"metadata": {
"editable": true
},
@@ -1982,7 +1632,7 @@
},
{
"cell_type": "markdown",
- "id": "fe853f3b",
+ "id": "abc56fe7",
"metadata": {
"editable": true
},
@@ -2006,7 +1656,7 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "81ec18f8",
+ "id": "a3d2885b",
"metadata": {
"collapsed": false,
"editable": true
@@ -2045,7 +1695,7 @@
},
{
"cell_type": "markdown",
- "id": "f0b5b588",
+ "id": "5db67125",
"metadata": {
"editable": true
},
@@ -2055,7 +1705,7 @@
},
{
"cell_type": "markdown",
- "id": "262a95e8",
+ "id": "6f1b5417",
"metadata": {
"editable": true
},
@@ -2066,7 +1716,7 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "d235b516",
+ "id": "9d98136a",
"metadata": {
"collapsed": false,
"editable": true
@@ -2086,7 +1736,7 @@
},
{
"cell_type": "markdown",
- "id": "c1620bb7",
+ "id": "c87eac4e",
"metadata": {
"editable": true
},
@@ -2104,7 +1754,7 @@
},
{
"cell_type": "markdown",
- "id": "fd3bd4b5",
+ "id": "2f0f4313",
"metadata": {
"editable": true
},
@@ -2116,7 +1766,7 @@
},
{
"cell_type": "markdown",
- "id": "4a129d86",
+ "id": "7895ee8b",
"metadata": {
"editable": true
},
@@ -2133,7 +1783,7 @@
},
{
"cell_type": "markdown",
- "id": "ff57b405",
+ "id": "be454b28",
"metadata": {
"editable": true
},
@@ -2145,7 +1795,7 @@
},
{
"cell_type": "markdown",
- "id": "5a13919e",
+ "id": "7475f8f9",
"metadata": {
"editable": true
},
@@ -2155,7 +1805,7 @@
},
{
"cell_type": "markdown",
- "id": "45d4a7b6",
+ "id": "fd472029",
"metadata": {
"editable": true
},
@@ -2167,7 +1817,7 @@
},
{
"cell_type": "markdown",
- "id": "c837442b",
+ "id": "b43c8f67",
"metadata": {
"editable": true
},
@@ -2184,7 +1834,7 @@
},
{
"cell_type": "markdown",
- "id": "4b3c7b8e",
+ "id": "c47cfb6d",
"metadata": {
"editable": true
},
@@ -2196,7 +1846,7 @@
},
{
"cell_type": "markdown",
- "id": "7e6f977d",
+ "id": "593188cd",
"metadata": {
"editable": true
},
@@ -2206,7 +1856,7 @@
},
{
"cell_type": "markdown",
- "id": "adeab0fd",
+ "id": "76de4d81",
"metadata": {
"editable": true
},
@@ -2218,7 +1868,7 @@
},
{
"cell_type": "markdown",
- "id": "63b18db5",
+ "id": "afc38c97",
"metadata": {
"editable": true
},
@@ -2228,7 +1878,7 @@
},
{
"cell_type": "markdown",
- "id": "9c7eb00d",
+ "id": "b0f2971c",
"metadata": {
"editable": true
},
@@ -2240,7 +1890,7 @@
},
{
"cell_type": "markdown",
- "id": "c3f4530b",
+ "id": "5a65e495",
"metadata": {
"editable": true
},
@@ -2250,7 +1900,7 @@
},
{
"cell_type": "markdown",
- "id": "90792236",
+ "id": "31419298",
"metadata": {
"editable": true
},
@@ -2266,7 +1916,7 @@
},
{
"cell_type": "markdown",
- "id": "d73c7cd4",
+ "id": "217ddd53",
"metadata": {
"editable": true
},
@@ -2277,7 +1927,7 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "58124207",
+ "id": "ece05002",
"metadata": {
"collapsed": false,
"editable": true
@@ -2342,7 +1992,7 @@
},
{
"cell_type": "markdown",
- "id": "4b4e7bf5",
+ "id": "de6dc4e2",
"metadata": {
"editable": true
},
@@ -2353,7 +2003,7 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "47b138a9",
+ "id": "da218205",
"metadata": {
"collapsed": false,
"editable": true
@@ -2410,7 +2060,7 @@
},
{
"cell_type": "markdown",
- "id": "37b66761",
+ "id": "5ea5df06",
"metadata": {
"editable": true
},
@@ -2448,7 +2098,7 @@
},
{
"cell_type": "markdown",
- "id": "5b5c601e",
+ "id": "40f41d74",
"metadata": {
"editable": true
},
@@ -2459,7 +2109,7 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "9bc32416",
+ "id": "03bb50ae",
"metadata": {
"collapsed": false,
"editable": true
@@ -2541,7 +2191,7 @@
},
{
"cell_type": "markdown",
- "id": "eb934dbe",
+ "id": "cd11e9fd",
"metadata": {
"editable": true
},
@@ -2566,7 +2216,7 @@
},
{
"cell_type": "markdown",
- "id": "c65f802b",
+ "id": "6cf68ff2",
"metadata": {
"editable": true
},
@@ -2594,7 +2244,7 @@
},
{
"cell_type": "markdown",
- "id": "9accabd4",
+ "id": "c445bfb2",
"metadata": {
"editable": true
},
@@ -2607,7 +2257,7 @@
{
"cell_type": "code",
"execution_count": 6,
- "id": "d5ed3f5c",
+ "id": "8e26035a",
"metadata": {
"collapsed": false,
"editable": true
@@ -2707,7 +2357,7 @@
},
{
"cell_type": "markdown",
- "id": "db1976f4",
+ "id": "86a6b012",
"metadata": {
"editable": true
},
@@ -2718,7 +2368,7 @@
{
"cell_type": "code",
"execution_count": 7,
- "id": "3c35540c",
+ "id": "4a0e1afd",
"metadata": {
"collapsed": false,
"editable": true
@@ -2807,7 +2457,7 @@
},
{
"cell_type": "markdown",
- "id": "e60fabdf",
+ "id": "bb1e0f15",
"metadata": {
"editable": true
},
@@ -2817,7 +2467,7 @@
},
{
"cell_type": "markdown",
- "id": "9b6750d6",
+ "id": "83c87d33",
"metadata": {
"editable": true
},
@@ -2830,7 +2480,7 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "5e6b91e1",
+ "id": "74eaac59",
"metadata": {
"collapsed": false,
"editable": true
@@ -2908,622 +2558,364 @@
},
{
"cell_type": "markdown",
- "id": "e83d7b10",
+ "id": "aa703354",
"metadata": {
"editable": true
},
"source": [
- "## Notes on scaling with examples\n",
- "\n",
- "The programs here use both ordinrary least squares (OLS) and Ridge\n",
- "regression with one value only for the hyperparameter $\\lambda$. The\n",
- "first example has no scaling and includes the intercept as well and we\n",
- "are trying to fit a second-order polynomial. The second code takes out\n",
- "the intercept and subtracts the mean values of each column of the\n",
- "design matrix and the mean value of the outputs.\n",
- "\n",
- "The third and final code uses **Scikit-Learn** as library in order to\n",
- "calculate the optimal parameters for OLS and Ridge regression. Note\n",
- "that it is highly recommended to not include the intercept in Ridge\n",
- "and Lasso regression, in order to avoid penalizing the optimization by\n",
- "the intercept. The second and third codes do thus not include the\n",
- "intercept. In the second code we do the scaling ourselves while the\n",
- "last code uses the standard scaler option included in **Scikit-Learn**, known as centering (where\n",
- "we subtract the mean values)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "id": "73807393",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "import matplotlib.pyplot as plt\n",
- "import numpy as np\n",
- "from sklearn.linear_model import LinearRegression\n",
- "from sklearn.preprocessing import PolynomialFeatures\n",
- "from sklearn.model_selection import train_test_split\n",
- "from sklearn.preprocessing import StandardScaler\n",
- "\n",
- "def MSE(y_data,y_model):\n",
- " n = np.size(y_model)\n",
- " return np.sum((y_data-y_model)**2)/n\n",
- "\n",
- "def OLS_fit_beta(X, y):\n",
- " return np.linalg.pinv(X.T @ X) @ X.T @ y\n",
- "\n",
- "def Ridge_fit_beta(X, y,L,d):\n",
- " I = np.eye(d,d)\n",
- " return np.linalg.pinv(X.T @ X + L*I) @ X.T @ y\n",
- "\n",
- "# Same random numbers for each test.\n",
- "np.random.seed(2018)\n",
- "n = 100\n",
- "d = 3\n",
- "# hyperparameter lambda\n",
- "Lambda = 0.01\n",
- "\n",
- "# Make data set, simple second-order polynomial\n",
- "x = np.linspace(-3, 3, n)\n",
- "y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)\n",
- "\n",
- "# The design matrix X includes the intercept and no scaling is made\n",
- "X = np.zeros((len(x), d))\n",
- "for p in range(d): \n",
- " X[:, p] = x ** (p) \n",
- "\n",
- "\n",
- "#Split data, no scaling is used and we include the intercept\n",
- "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
- "\n",
- "\n",
- "#Calculate beta, own code\n",
- "beta_OLS = OLS_fit_beta(X_train, y_train)\n",
- "beta_Ridge = Ridge_fit_beta(X_train, y_train,Lambda,d)\n",
- "print(beta_OLS)\n",
- "print(beta_Ridge)\n",
- "#predict value\n",
- "ytilde_test_OLS = X_test @ beta_OLS\n",
- "ytilde_test_Ridge = X_test @ beta_Ridge\n",
- "\n",
- "#Calculate MSE\n",
- "print(\" \")\n",
- "print(\"test MSE of OLS:\")\n",
- "print(MSE(y_test,ytilde_test_OLS))\n",
- "print(\" \")\n",
- "print(\"test MSE of Ridge\")\n",
- "print(MSE(y_test,ytilde_test_Ridge))\n",
- "\n",
- "plt.scatter(x,y,label='Data')\n",
- "plt.plot(x, X @ beta_OLS,'*', label=\"OLS_Fit\")\n",
- "plt.plot(x, X @ beta_Ridge, label=\"Ridge_Fit\")\n",
- "plt.grid()\n",
- "plt.legend()\n",
- "plt.show()"
+ "## Material for the lab sessions"
]
},
{
"cell_type": "markdown",
- "id": "b09d20a2",
+ "id": "f8bffca6",
"metadata": {
"editable": true
},
"source": [
- "In this example we do not include the intercept and we scale the data by subtracting the mean values. This follows the discussion in the [lecture material](https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#more-on-rescaling-data).\n",
- "see also the weekly slides [for week 36](https://compphysics.github.io/MachineLearning/doc/pub/week36/html/._week36-bs029.html).\n",
- "It is recommended whrn we use Ridge and Lasso regression to not include the intercept in the optimization process.\n",
+ "## Linking the regression analysis with a statistical interpretation\n",
"\n",
- "Before we discuss the code, we repeat some of the basic math from the slides of week 36.\n",
+ "We will now couple the discussions of ordinary least squares, Ridge\n",
+ "and Lasso regression with a statistical interpretation, that is we\n",
+ "move from a linear algebra analysis to a statistical analysis. In\n",
+ "particular, we will focus on what the regularization terms can result\n",
+ "in. We will amongst other things show that the regularization\n",
+ "parameter can reduce considerably the variance of the parameters\n",
+ "$\\beta$.\n",
"\n",
- "Let us try to understand what this may imply mathematically when we\n",
- "subtract the mean values, also known as *zero centering* or simply *centering*. For\n",
- "simplicity, we will focus on ordinary regression, as done in the above example.\n",
+ "The\n",
+ "advantage of doing linear regression is that we actually end up with\n",
+ "analytical expressions for several statistical quantities. \n",
+ "Standard least squares and Ridge regression allow us to\n",
+ "derive quantities like the variance and other expectation values in a\n",
+ "rather straightforward way.\n",
"\n",
- "The cost/loss function for regression is"
+ "It is assumed that $\\varepsilon_i\n",
+ "\\sim \\mathcal{N}(0, \\sigma^2)$ and the $\\varepsilon_{i}$ are\n",
+ "independent, i.e.:"
]
},
{
"cell_type": "markdown",
- "id": "ef1cec97",
+ "id": "109c07d6",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n",
+ "\\begin{align*} \n",
+ "\\mbox{Cov}(\\varepsilon_{i_1},\n",
+ "\\varepsilon_{i_2}) & = \\left\\{ \\begin{array}{lcc} \\sigma^2 & \\mbox{if}\n",
+ "& i_1 = i_2, \\\\ 0 & \\mbox{if} & i_1 \\not= i_2. \\end{array} \\right.\n",
+ "\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "e43c422b",
+ "id": "1ea48f22",
"metadata": {
"editable": true
},
"source": [
- "Recall also that we use the squared value. This expression can lead to an\n",
- "increased penalty for higher differences between predicted and\n",
- "output/target values.\n",
+ "The randomness of $\\varepsilon_i$ implies that\n",
+ "$\\mathbf{y}_i$ is also a random variable. In particular,\n",
+ "$\\mathbf{y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n",
+ "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\beta}$ is a\n",
+ "non-random scalar. To specify the parameters of the distribution of\n",
+ "$\\mathbf{y}_i$ we need to calculate its first two moments. \n",
"\n",
- "What we have done is to single out the $\\beta_0$ term in the\n",
- "definition of the mean squared error (MSE). The design matrix $X$\n",
- "does in this case not contain any intercept column. When we take the\n",
- "derivative with respect to $\\beta_0$, we want the derivative to obey"
+ "Recall that $\\boldsymbol{X}$ is a matrix of dimensionality $n\\times p$. The\n",
+ "notation above $\\mathbf{X}_{i,\\ast}$ means that we are looking at the\n",
+ "row number $i$ and perform a sum over all values $p$."
]
},
{
"cell_type": "markdown",
- "id": "852577bc",
+ "id": "212f90ec",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Assumptions made\n",
+ "\n",
+ "The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)\n",
+ "that there exists a function $f(\\boldsymbol{x})$ and a normal distributed error $\\boldsymbol{\\varepsilon}\\sim \\mathcal{N}(0, \\sigma^2)$\n",
+ "which describe our data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a019e7cb",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C}{\\partial \\beta_j} = 0,\n",
+ "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "4edd229a",
+ "id": "2ca46c43",
"metadata": {
"editable": true
},
"source": [
- "for all $j$. For $\\beta_0$ we have"
+ "We approximate this function with our model from the solution of the linear regression equations, that is our\n",
+ "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we want to minimize $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, our MSE, with"
]
},
{
"cell_type": "markdown",
- "id": "bb93d608",
+ "id": "c61ed9c8",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n",
+ "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "5ef162e3",
+ "id": "96d84043",
"metadata": {
"editable": true
},
"source": [
- "Multiplying away the constant $2/n$, we obtain"
+ "## Expectation value and variance\n",
+ "\n",
+ "We can calculate the expectation value of $\\boldsymbol{y}$ for a given element $i$"
]
},
{
"cell_type": "markdown",
- "id": "b7c5467d",
+ "id": "e53b66df",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n",
+ "\\begin{align*} \n",
+ "\\mathbb{E}(y_i) & =\n",
+ "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}) + \\mathbb{E}(\\varepsilon_i)\n",
+ "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n",
+ "\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "493c75d6",
+ "id": "71d542a3",
"metadata": {
"editable": true
},
"source": [
- "Let us specialize first to the case where we have only two parameters $\\beta_0$ and $\\beta_1$.\n",
- "Our result for $\\beta_0$ simplifies then to"
+ "while\n",
+ "its variance is"
]
},
{
"cell_type": "markdown",
- "id": "862f2222",
+ "id": "01b18637",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n",
+ "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n",
+ "- \\mathbb{E}(y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( y_i^2 ) -\n",
+ "[\\mathbb{E}(y_i)]^2 \\\\ & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n",
+ "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \\\\ &\n",
+ "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2 \\varepsilon_i\n",
+ "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
+ "\\ast} \\, \\beta)^2 \\\\ & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2\n",
+ "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} +\n",
+ "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \n",
+ "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n",
+ "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2. \n",
+ "\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "e0d0a4e0",
+ "id": "e35bdcc1",
"metadata": {
"editable": true
},
"source": [
- "We obtain then"
+ "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
+ "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD)."
]
},
{
"cell_type": "markdown",
- "id": "73d327de",
+ "id": "15b7ea3c",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Expectation value and variance for $\\boldsymbol{\\beta}$\n",
+ "\n",
+ "With the OLS expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}}$ we can evaluate the expectation value"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a4c5f625",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n",
+ "\\mathbb{E}(\\boldsymbol{\\hat{\\beta}}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "8be27cbd",
+ "id": "450007e4",
"metadata": {
"editable": true
},
"source": [
- "If we define"
+ "This means that the estimator of the regression parameters is unbiased.\n",
+ "\n",
+ "We can also calculate the variance\n",
+ "\n",
+ "The variance of the optimal value $\\boldsymbol{\\hat{\\beta}}$ is"
]
},
{
"cell_type": "markdown",
- "id": "d49a9bac",
+ "id": "3bcf81c1",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\mu_{\\boldsymbol{x}_1}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1},\n",
+ "\\begin{eqnarray*}\n",
+ "\\mbox{Var}(\\boldsymbol{\\hat{\\beta}}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n",
+ "\\\\\n",
+ "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}]^{T} \\}\n",
+ "\\\\\n",
+ "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "% \\\\\n",
+ "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} \\, \\mathbf{y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "% \\\\\n",
+ "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{y} \\, \\mathbf{y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "\\\\\n",
+ "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "% \\\\\n",
+ "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^T \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
+ "% \\\\\n",
+ "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\boldsymbol{\\beta}^T\n",
+ "\\\\\n",
+ "& = & \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "\\, \\, \\, = \\, \\, \\, \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1},\n",
+ "\\end{eqnarray*}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "dd964823",
+ "id": "f69a019e",
"metadata": {
"editable": true
},
"source": [
- "and the mean value of the outputs as"
+ "where we have used that $\\mathbb{E} (\\mathbf{y} \\mathbf{y}^{T}) =\n",
+ "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n",
+ "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n",
+ "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n",
+ "variance of the estimate of the $j$-th regression coefficient:\n",
+ "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $. This may be used to\n",
+ "construct a confidence interval for the estimates.\n",
+ "\n",
+ "In a similar way, we can obtain analytical expressions for say the\n",
+ "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n",
+ "when we employ Ridge regression, allowing us again to define a confidence interval. \n",
+ "\n",
+ "It is rather straightforward to show that"
]
},
{
"cell_type": "markdown",
- "id": "09a87e67",
+ "id": "95b37da0",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\mu_y=\\frac{1}{n}\\sum_{i=0}^{n-1}y_i,\n",
+ "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "9b8797a0",
+ "id": "241c2335",
"metadata": {
"editable": true
},
"source": [
- "we have"
+ "We see clearly that \n",
+ "$\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}$ for any $\\lambda > 0$.\n",
+ "\n",
+ "We can also compute the variance as"
]
},
{
"cell_type": "markdown",
- "id": "30aeafdb",
+ "id": "cf794384",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\beta_0 = \\mu_y - \\beta_1\\mu_{\\boldsymbol{x}_1}.\n",
+ "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "dc867cbb",
+ "id": "4cd5260a",
"metadata": {
"editable": true
},
"source": [
- "In the general case with more parameters than $\\beta_0$ and $\\beta_1$, we have"
+ "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n",
+ "\n",
+ "With this, we can compute the difference"
]
},
{
"cell_type": "markdown",
- "id": "f154321e",
+ "id": "7361e846",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n",
+ "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}]-\\mbox{Var}(\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "6015f267",
+ "id": "6a07837b",
"metadata": {
"editable": true
},
"source": [
- "We can rewrite the latter equation as"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f181e88f",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\beta_j,\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a6e6c0e7",
- "metadata": {
- "editable": true
- },
- "source": [
- "where we have defined"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "708652ea",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mu_{\\boldsymbol{x}_j}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{ij},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b360788c",
- "metadata": {
- "editable": true
- },
- "source": [
- "the mean value for all elements of the column vector $\\boldsymbol{x}_j$.\n",
+ "The difference is non-negative definite since each component of the\n",
+ "matrix product is non-negative definite. \n",
+ "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. \n",
"\n",
- "Replacing $y_i$ with $y_i - y_i - \\overline{\\boldsymbol{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8f473602",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "70f518a7",
- "metadata": {
- "editable": true
- },
- "source": [
- "If we minimize with respect to $\\boldsymbol{\\beta}$ we have then"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "674d02d9",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6715261c",
- "metadata": {
- "editable": true
- },
- "source": [
- "where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n",
- "and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n",
- "\n",
- "For Ridge regression we need to add $\\lambda \\boldsymbol{\\beta}^T\\boldsymbol{\\beta}$ to the cost function and get then"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "17600ee3",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a8c6f068",
- "metadata": {
- "editable": true
- },
- "source": [
- "Now we try to implement this."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "1d89f80d",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "\n",
- "np.random.seed(2018)\n",
- "n = 100\n",
- "# we do not include the intercept\n",
- "d = 2\n",
- "Lambda = 0.01\n",
- "\n",
- "# Make data set.\n",
- "x = np.linspace(-3, 3, n)\n",
- "y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)\n",
- "\n",
- "#Design matrix X does not include the intercept. \n",
- "X = np.zeros((len(x), d))\n",
- "for p in range(d): \n",
- " X[:, p] = x ** (p+1)\n",
- "\n",
- "\n",
- "#Split data in train and test\n",
- "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
- "\n",
- "# Scale data by subtracting mean value,own implementation\n",
- "#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable\n",
- "X_train_mean = np.mean(X_train,axis=0)\n",
- "#Center by removing mean from each feature\n",
- "X_train_scaled = X_train - X_train_mean\n",
- "X_test_scaled = X_test - X_train_mean\n",
- "#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered, note)\n",
- "y_scaler = np.mean(y_train)\n",
- "y_train_scaled = y_train - y_scaler\n",
- "\n",
- "\n",
- "#Calculate beta\n",
- "beta_OLS = OLS_fit_beta(X_train_scaled, y_train_scaled)\n",
- "beta_Ridge = Ridge_fit_beta(X_train_scaled, y_train_scaled,Lambda,d)\n",
- "print(beta_OLS)\n",
- "print(beta_Ridge)\n",
- "# calculate intercepts and print them\n",
- "interceptOLS = y_scaler - X_train_mean @ beta_OLS\n",
- "interceptRidge = y_scaler - X_train_mean @ beta_Ridge\n",
- "print(interceptOLS)\n",
- "print(interceptRidge)\n",
- "\n",
- "#predict value with intercept\n",
- "ytilde_test_OLS = X_test_scaled @ beta_OLS+y_scaler\n",
- "ytilde_test_Ridge = X_test_scaled @ beta_Ridge+y_scaler\n",
- "\n",
- "\n",
- "#Calculate MSE\n",
- "\n",
- "print(\" \")\n",
- "print(\"test MSE of OLS:\")\n",
- "print(MSE(y_test,ytilde_test_OLS))\n",
- "print(\" \")\n",
- "print(\"test MSE of Ridge\")\n",
- "print(MSE(y_test,ytilde_test_Ridge))\n",
- "\n",
- "plt.scatter(x,y,label='Data')\n",
- "plt.plot(x, X @ beta_OLS+interceptOLS,'*', label=\"OLS_Fit\")\n",
- "plt.plot(x, X @ beta_Ridge+interceptRidge, label=\"Ridge_Fit\")\n",
- "plt.grid()\n",
- "plt.legend()\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "522668c3",
- "metadata": {
- "editable": true
- },
- "source": [
- "Finally, instead of using our own function we repeat the same example\n",
- "using the **standardscaler** functionality of the library\n",
- "**Scikit-Learn**. Here we limit ourselves to Ridge regression only."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "id": "9954bcc1",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "from sklearn import linear_model\n",
- "np.random.seed(2018)\n",
- "n = 10\n",
- "d = 2\n",
- "Lambda = 0.01\n",
- "\n",
- "# Make data set.\n",
- "x = np.linspace(-3, 3, n)\n",
- "y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)\n",
- "\n",
- "# Design matrix X does not include the intercept. \n",
- "X = np.zeros((n, d))\n",
- "for p in range(d): \n",
- " X[:, p] = x ** (p+1)\n",
- "\n",
- "#Split data in train and test\n",
- "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
- "# Scale data by subtracting mean value of the input using scikit-learn\n",
- "scaler = StandardScaler(with_std=False)\n",
- "scaler.fit(X_train)\n",
- "X_train_mean = np.mean(X_train,axis=0)\n",
- "X_train_scaled = scaler.transform(X_train)\n",
- "X_test_scaled = scaler.transform(X_test)\n",
- "# We scale also the output, here by our own code\n",
- "y_scaler = np.mean(y_train)\n",
- "y_train_scaled = y_train - y_scaler\n",
- "y_test_scaled = y_test- y_scaler\n",
- "\n",
- "#Calculate beta\n",
- "OLS = LinearRegression()\n",
- "betaOLS=OLS.fit(X_train_scaled,y_train_scaled)\n",
- "ypredictOLS = OLS.predict(X_test_scaled)\n",
- "linear_model.Ridge(Lambda)\n",
- "RegRidge.fit(X_train_scaled,y_train_scaled)\n",
- "ypredictRidge = RegRidge.predict(X_test_scaled)\n",
- "betaOLS = OLS.coef_\n",
- "betaRidge = RegRidge.coef_\n",
- "print(betaOLS)\n",
- "print(betaRidge)\n",
- "interceptOLS = np.mean(y_train) - X_train_mean @ betaOLS\n",
- "interceptRidge = y_scaler - X_train_mean @ betaRidge\n",
- "print(interceptOLS)\n",
- "print(interceptRidge)\n",
- "#predict value \n",
- "ytilde_test_Ridge = X_test_scaled @ betaRidge+y_scaler\n",
- "ytilde_test_OLS = X_test_scaled @ betaOLS+y_scaler\n",
- "\n",
- "#Calculate MSE\n",
- "print(\" \")\n",
- "print(\"test MSE of OLS\")\n",
- "print(MSE(y_test,ytilde_test_OLS))\n",
- "print(\" \")\n",
- "print(\"test MSE of Ridge\")\n",
- "print(MSE(y_test,ytilde_test_Ridge))\n",
- "plt.scatter(x,y,label='Data')\n",
- "plt.plot(x, X @ RegRidge.coef_ + RegRidge.intercept_ , label=\"Ridge_Fit\")\n",
- "plt.grid()\n",
- "plt.legend()\n",
- "plt.show()"
+ "For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended."
]
}
],
diff --git a/doc/LectureNotes/_build/html/searchindex.js b/doc/LectureNotes/_build/html/searchindex.js
index 5f95f4674..63fe57975 100644
--- a/doc/LectureNotes/_build/html/searchindex.js
+++ b/doc/LectureNotes/_build/html/searchindex.js
@@ -1 +1 @@
-Search.setIndex({docnames:["chapter1","chapter10","chapter11","chapter12","chapter13","chapter2","chapter3","chapter4","chapter5","chapter6","chapter7","chapter8","chapter9","chapteroptimization","clustering","exercisesweek34","exercisesweek35","exercisesweek36","exercisesweek37","intro","linalg","project1","schedule","statistics","teachers","textbooks","week34","week35","week36","week37"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":4,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":3,"sphinx.domains.rst":2,"sphinx.domains.std":2,"sphinx.ext.intersphinx":1,sphinx:56},filenames:["chapter1.ipynb","chapter10.ipynb","chapter11.ipynb","chapter12.ipynb","chapter13.ipynb","chapter2.ipynb","chapter3.ipynb","chapter4.ipynb","chapter5.ipynb","chapter6.ipynb","chapter7.ipynb","chapter8.ipynb","chapter9.ipynb","chapteroptimization.ipynb","clustering.ipynb","exercisesweek34.ipynb","exercisesweek35.ipynb","exercisesweek36.ipynb","exercisesweek37.ipynb","intro.md","linalg.ipynb","project1.ipynb","schedule.md","statistics.ipynb","teachers.md","textbooks.md","week34.ipynb","week35.ipynb","week36.ipynb","week37.ipynb"],objects:{},objnames:{},objtypes:{},terms:{"0":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,24,26,27,28,29],"00":[0,1,5,6,9,11,26,27,28],"000":[1,3,9],"0000":9,"00000":[9,11],"000000":[5,11,27],"00000000e":[5,27],"0001":[1,17],"00019961646575285301":9,"00019998":5,"00024087":5,"00029012":5,"00031535148309577417":[6,28],"0003153514830958095":[6,28],"00034944":5,"000417932":2,"00042089":5,"000464088":2,"00050694":5,"00058016":[6,28],"00060704":6,"00061058":5,"00062592":6,"00063362":29,"00063866":29,"00066668":[6,28],"00068091":29,"00068734":[6,28],"00069123":6,"00073541":5,"00075590":29,"00076495":[6,28],"00076908":6,"0007698473260556325":[6,28],"0007698473260556343":[6,28],"00079125":6,"00079904":29,"00079968":5,"00083355":29,"00084707":6,"00085892":6,"00087697":[6,28],"00088573":5,"00089193":29,"00092678":6,"00092898":29,"00096314":5,"001":[1,2,8,13,17,28],"00100519":[6,28],"0010479245693919886":[6,29],"00105081":[6,28],"00106677":5,"00107405":[6,28],"00111756":[6,28],"0011526":[6,28],"00115999":5,"00118584":6,"00119688":29,"00128479":5,"001323":[6,28],"00137814":6,"00139705":5,"00140846":29,"00149311":[6,28],"00149956":[6,28],"00152117":[6,28],"00154733":5,"00154853":29,"00156374":6,"00168251":5,"00174276":[6,28],"00175331":[6,28],"00186347":5,"001880":5,"00200":8,"00202624":5,"00202756":[6,28],"00217499":[6,28],"00219195":29,"00224413":5,"00228740":6,"0023548":[6,28],"002381316302584885":[6,28],"0023813163025848865":[6,28],"00242954":29,"00242999":6,"00243186":[6,28],"0024401":5,"00249435":[6,28],"00266858":2,"00270244":5,"00274989":[6,28],"00289724":[6,28],"00293838":5,"00310113":2,"00312361":[6,28],"00315593":6,"0032153180657605116":[6,29],"00323332":[6,28],"0032542":5,"00325450":29,"003301":[6,28],"0033955154592040936":[6,29],"00353823":5,"0036237":[6,28],"0036367":[6,28],"00369758":[6,28],"0038335":[6,28],"003909404072811226":[6,29],"00391839":5,"0039987":[6,28],"004":[5,28,29],"004091940707753939":[6,29],"00410387":[6,28],"00410478":6,"004113634617443131":[6,27,28],"00411363461744314":[6,27,28],"004113634617443147":[6,27,28],"00415598634586135":[6,29],"00420072":29,"00424909":2,"00424967":[6,28],"00426027":5,"00433417":11,"00440346":[6,28],"00443743":[6,28],"00445655":11,"00445922":26,"004579219539673834":[6,29],"00458878":[6,28],"004610275230656212":[6,29],"00462287":[6,28],"00465099":29,"00471782":5,"00472199":6,"00472512":[6,28],"0049544":[6,28],"00497837":28,"004999999999999991":0,"00512927":5,"00517114":[6,28],"00526348":[6,28],"0053018":[6,28],"00554552":[6,28],"00556826":[6,28],"0056799":5,"00579953":[6,28],"00588657":[6,28],"00607783":6,"006162":[6,28],"00617499":5,"00630331":[6,28],"00642221":[6,28],"0064472":28,"00660427":[6,28],"00672607":[6,28],"00673407":[6,28],"00676387":[6,28],"0068011":[6,28],"00683748":5,"00683964":[6,28],"007024126888938232":[6,29],"00719176":[6,28],"00727646693":[0,26],"007315":27,"0074331":5,"00759119":6,"00779281":23,"00781918":29,"00784393":[6,28],"00803064":[6,28],"00805074":29,"00813803":6,"00817631":[6,28],"00823002":5,"00827728":[6,28],"00831018":[6,28],"00834567":[6,28],"00848904":[6,28],"0086649156":[0,26],"008675369724975977":5,"008831941890485846":27,"00890669":20,"00894639":5,"00902152e":27,"00905423":[6,28],"009163470508352218":5,"009164545680330616":[6,29],"00917248":[6,28],"00934327e":27,"00934499":[6,28],"009522910538005715":27,"0096208":[6,28],"00987022":26,"00990475":5,"00992331":[6,28],"00996754":[6,28],"00it":6,"01":[0,1,2,5,6,9,11,13,17,25,26,27,28,29],"010018312644137363":[6,29],"0100706":[6,28],"010331721306655127":[6,29],"010516485576645508":[6,29],"01066519":[6,28],"01076611":5,"0110":23,"011076219011379514":[6,29],"011225":2,"011265488125148788":27,"0113104":[6,28],"01173838":0,"01179792":[6,28],"01191824":5,"012073649446193166":[6,29],"01219292":[6,28],"01222822":27,"01223198":[6,28],"01231917":[6,28],"01265755":27,"012658":27,"01267007":28,"01290947":[6,28],"01295356":5,"013121574015585152":[6,29],"013121574015585155":[6,29],"01312254":26,"01318643":[6,28],"01347916":[6,28],"01348565":[6,28],"01367553":[6,28],"01384313":26,"01397146":[6,28],"01405935":[6,28],"01416528":[6,28],"01433809":5,"014436800088904942":[6,29],"01449782":[6,28],"01458337":[6,28],"014586":27,"01458611":27,"0146081":[6,28],"01463049":[6,28],"014891664497538563":26,"01496065":26,"01507238889517706":[6,28],"01507238889517717":[6,28],"01531845":[6,28],"01542292":29,"01549377":[6,28],"01549939":29,"01550546":29,"01558197":5,"015910":11,"01591047":11,"01594452":29,"01628578269596628":[6,29],"01633913":[6,28],"01640891":6,"01655318":[6,28],"016587414993043573":[6,29],"01691985":[6,28],"0169643":5,"01708781":[6,28],"01708852":[6,28],"01713366":[6,28],"01724499":5,"017355848195593347":[6,29],"01735584819559336":[6,29],"017665":5,"01783414e":27,"01813007e":27,"018232":27,"01831050e":[6,28],"01831190e":[6,28],"01846688":26,"01866537":[6,28],"01873344":11,"01873869":5,"01898855":[6,28],"01905883":[6,28],"01908936":[6,28],"01963611":[6,28],"01969145":[6,28],"01975416527185249":[6,29],"01975848":[6,28],"01it":6,"02":[0,4,6,7,12,26,27,28],"02024962":[6,28],"02054837e":[6,28],"02068067":[6,28],"02073509":5,"02077431":28,"02098261":[6,28],"02123176":[6,28],"021544346900318832":28,"021592704588025022":[6,29],"021592704588025025":[6,29],"02180258":26,"02198702e":[6,28],"02198703e":[6,28],"02208512":[6,28],"02228115":[6,28],"02229529":[6,28],"02252765":5,"022999498260366312":[6,29],"02348765":[6,28],"02365049":[6,28],"024217":11,"02421742":11,"02424794":29,"0245528":[6,28],"02492265":5,"02498832":[6,28],"02503753":[6,28],"02511518":6,"02522069":6,"02546675":29,"025709":[11,27],"02586427":[6,28],"0260906":[6,28],"02625193":8,"02629596":20,"02660572763718093":[6,29],"026605727637180936":[6,29],"02693858":28,"02707227":5,"02723445":[6,28],"027609773491022525":[6,29],"02760977349102253":[6,29],"027610":5,"02761005":5,"02857":4,"02895918":28,"029483":5,"029733":27,"02976145":[6,28],"02994311":5,"02f":[6,21],"02it":6,"03":[1,6,27,28],"03032441e":[6,28],"03077640549":4,"03099776":5,"031":[5,28,29],"031696741936652305":23,"03196357":[6,28],"03251863":5,"03256632e":1,"03267527":[6,28],"03279636":[6,28],"0330308045183194":[6,28],"0330308045187757":[6,28],"03308408":5,"0334336":26,"033657685071527665":[6,29],"03405654":11,"034057":11,"03447512":[6,28],"034557":27,"03562355":[6,28],"03568439":[6,28],"0359565":5,"03630548":[6,28],"03707133":11,"037299816041296285":9,"037813671417389005":[6,29],"03781367141738902":[6,29],"03814292":[6,28],"03815288":[6,28],"038300":[11,27],"03846169":28,"039039":5,"039967668952797":[6,28],"0399676689527975":[6,28],"04":[1,6,11,27,28],"040102":5,"04010697":[6,28],"0402":29,"04063602":[6,28],"041":9,"0411487294305088":[6,28],"0411487294305226":[6,28],"04193203":27,"04220758":[6,28],"04242885e":27,"04295757":29,"043":9,"04315108":5,"04321941e":27,"04346721":5,"04355837":[6,28],"04362":9,"04373024":26,"0437499":2,"04389027":[6,28],"04423486":[6,28],"04426744e":27,"044334":27,"044613":[6,28],"04537385":[6,28],"04543942":[6,28],"045548":11,"04554803":11,"04566964":[6,28],"0458":9,"04648335":5,"04664255":6,"04683565":5,"046842629321028326":11,"04784395":[6,28],"0481872773043029":[6,29],"04876305":26,"04892055":[6,28],"04909093":[6,28],"04912436":6,"04926746":29,"0494262":26,"04944931918420588":23,"049556996627824":[6,28],"0495569966278269":[6,28],"05":[1,4,6,21,27,28],"05009826":[6,28],"05061841":23,"05073207e":27,"05087888":11,"050879":11,"051006":11,"05100602":11,"05100875":[6,28],"05108705":26,"051418":5,"051649":[11,27],"0517473":5,"05210975":20,"052279218012056844":[6,29],"05227921801205686":[6,29],"052305":27,"05263":9,"05302":9,"053417":27,"05364854":8,"05383795":[6,28],"0538391":26,"053849":5,"05447415":[6,28],"054617":27,"054655":27,"05505310046363":2,"05519999":28,"05533":9,"055676":27,"055734":27,"055910":27,"055987":27,"05614483":5,"05623":9,"05648":9,"05651951":[6,28],"05667":9,"056683":27,"056767":5,"057076":5,"05716368155342608":[6,29],"057300":27,"057393":27,"057415":5,"057418":27,"057457":27,"057722":27,"057786":5,"057835":27,"05785343":[6,28],"05789007":[6,28],"05796251":[6,28],"05807125":[6,28],"058190":5,"058552":27,"058645":27,"05883":9,"05884":9,"058854":27,"058952":27,"058996":27,"059031":27,"059226":5,"059406":5,"059597":5,"05965043":20,"059711":5,"059797":5,"059807":27,"059951":27,"05999":9,"059993":27,"06":[6,27,28],"060003":5,"060043":5,"060083":27,"06020587":[6,28],"060254":27,"060340":5,"060422":5,"06043581":[6,28],"060612":5,"060716":27,"060756":27,"060764":5,"060864":5,"060872":27,"061076":5,"061146":5,"061163":27,"061359":27,"061367":5,"061443":27,"061452":27,"061496":5,"061614":27,"061822":5,"061882":5,"06200174":5,"062016":27,"06208238634231949":[6,29],"062242":5,"062292565":4,"062337":27,"062599":27,"062624":27,"062634":5,"062874":27,"063019":27,"063061":27,"063080":27,"063260":27,"063511":5,"063653":27,"063684":5,"063832":5,"063894":27,"063965":5,"064061":5,"064074":27,"064087":5,"064307":5,"06442194e":27,"06444":9,"064501":27,"064521":5,"064530":5,"06453579006728324":[6,29],"064606":27,"064609":27,"064699":27,"064713":5,"064793":27,"064841":5,"064889":5,"06491736":[6,28],"064933":5,"065024":5,"065077":27,"065101":5,"065115":5,"065186":5,"065273":5,"065299":5,"065304":5,"065359":5,"065361":5,"065378":27,"065390":27,"065410":27,"06547790180152355":[6,29],"06547790180152359":[6,29],"065588":5,"065633":5,"065860":5,"065930":5,"066066":27,"066204":5,"066349":5,"06637":9,"066380":5,"066395":5,"066402":5,"066406":5,"06644775":20,"066458":5,"066474":27,"066500":27,"066612":27,"066647":27,"0666807":2,"066787":27,"066837":[11,27],"066854":27,"06696769":20,"066999":27,"067097":5,"067149":5,"06719214":20,"06724062":5,"067284":5,"067392":5,"067429":5,"067482":5,"067560":5,"067637":27,"067707":27,"067729":5,"068022":5,"068264":27,"068301":5,"068307":27,"068441":27,"06844519414009445":[6,29],"06844519414009446":[6,29],"068605":11,"068629":27,"068727":27,"068906":27,"068963":5,"069028":27,"06926365":27,"069275":5,"0694169556368514":20,"069452":27,"069521":5,"069584":27,"069704":5,"069717":5,"07":[6,27,28],"070006":11,"070043":27,"070054":5,"070146":27,"07016":9,"07017":9,"070202":11,"070213":27,"070342":5,"07039":9,"070457":27,"070496":5,"070582":27,"07062318":[6,28],"070737":27,"070868":5,"070889":[11,27],"070986":27,"07115":9,"0713":[0,26],"071315":5,"07145103":11,"071462":11,"07160048164233102":[6,29],"07160048164233104":[6,29],"07172103":26,"071729":11,"072000":27,"072111":27,"072132":27,"072476":27,"072489":5,"072621":27,"072633":5,"072759":5,"07285":3,"072860":5,"072976":11,"073318":11,"073481":5,"073581":5,"073598":[11,27],"073627":5,"073634":5,"073644":27,"073934":5,"074067":[11,27],"074084":27,"074208":5,"074210":27,"07421084":5,"074323":27,"074407":5,"074548":5,"074552":11,"07456491":5,"074706":5,"07490892":[6,28],"074973":11,"075398":11,"075698":5,"075816":27,"075889":27,"075972":5,"076150":27,"076383":5,"07641937":29,"07643382":20,"0764924":[6,28],"076620":5,"076697":11,"076716":5,"07678":9,"07692307692307693":9,"076938":[11,27],"076964":5,"077082":11,"077168":27,"07745069":23,"077460":27,"077522":11,"07777777777777778":1,"078110":27,"07820":9,"078336":27,"078540":27,"07862755":23,"07864":9,"078704":5,"07871":9,"078785":11,"078824":11,"078974":27,"078986":27,"079381":11,"079412":5,"07944154":[20,26],"079578":5,"079642":5,"0796891867672603":[6,29],"0798590438380667":11,"079873":5,"079946":27,"08":[6,9,23,27],"080016":5,"080195":11,"080425":5,"08043851":5,"080616":27,"080630":11,"080764":27,"081276":27,"081308":11,"08131003":6,"081374":11,"081422":11,"08156108":[6,28],"081597":11,"08165104":27,"081655":27,"081798":11,"08251519":[6,28],"082592":11,"082619":11,"082653":27,"082661":11,"08277442":23,"0827744231113545":23,"082978":11,"08299273e":[6,28],"083096":11,"08318298e":1,"08323973e":27,"083309":11,"08333333333333333":[1,9],"08336233266":4,"083527":27,"083540":11,"08376632":[6,27,28],"083766322923899":[6,27,28],"0837663229239043":[6,27,28],"083851":11,"083906":11,"083943":11,"084022":11,"084076":27,"084157":11,"08426840630693411":[6,29],"08426840630693413":[6,29],"084362":11,"08455":9,"084574":11,"084650":11,"08474":9,"084809":27,"085152":11,"085237":11,"085259":11,"085265":11,"085346":11,"085383":11,"085431":11,"08551306":[6,28],"08557805981945521":5,"085693":11,"08576932":6,"08593216":[6,28],"086059":11,"08611111111111111":1,"086230":11,"086263":11,"086538":11,"086550":11,"086611":11,"086615":11,"086673":11,"086782":11,"086875":11,"086890":11,"08690":9,"087176":11,"08750220e":27,"08758":9,"087822":11,"087938":11,"087951":11,"088061":11,"088130":11,"088160":11,"0881981":5,"088222":11,"08823":[],"088361":11,"088402":11,"088422":11,"08860792":20,"08871404":5,"08881497884574564":27,"08888888888888889":1,"088893":11,"088908":11,"088986":11,"08902":9,"08939489":20,"0895387":27,"089539":27,"089629":11,"089654":11,"089682":11,"089702":11,"089739":11,"089892":11,"08996":9,"09":[1,6,27],"090000":11,"090025":11,"09004029":26,"090290":11,"09047191e":27,"090623":11,"090696":11,"090751":11,"091246":11,"091331":11,"091353":11,"091561":11,"091627":11,"09163947":6,"09166666666666666":1,"0917":9,"091700":11,"09221309":23,"092347":11,"09251":9,"092544":11,"092613":11,"092672":11,"09298511":23,"093078":11,"093132":11,"093429":11,"0940821989627646e":[6,28],"0940821989652176e":[6,28],"094426":11,"09444444444444444":1,"094577":11,"094659":11,"094671":11,"094970":11,"09609807":5,"09650726":26,"096803":11,"096837":11,"096928":11,"09715081560068299":26,"09744":9,"09780":9,"09791":9,"09861229":[20,26],"0988662":20,"09900923":26,"09903804":8,"09917303":20,"09919198949275869":[6,29],"099287":11,"09951287404314545":1,"09953815":20,"0n":[0,26],"0x0006":[3,4,14],"1":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,22,23,24,25,28,29],"10":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,20,21,22,23,24,26,27,28,29],"100":[0,1,2,3,4,5,6,8,9,10,11,13,14,15,16,17,20,23,24,26,27,28,29],"1000":[0,1,2,4,5,8,11,13,14,19,23,26,27],"10000":[2,5,6,10,11,13,23,28,29],"100000":8,"10001":10,"1001":[9,23],"1002":23,"1003":23,"10030":9,"10041812":28,"1005":23,"1007711427354898":[6,29],"1009":23,"1011":23,"1013":23,"1013904243":23,"10141413e":[6,28],"1015":23,"1015am":26,"1023":23,"10230":9,"1024":3,"1026":23,"1027":23,"103":1,"1030":23,"10340":9,"1037":23,"10378326e":1,"1038":23,"10391807":[6,28],"10398646080125032":[6,29],"10398646080125035":[6,29],"1040":23,"10405456":11,"10430":9,"104411":27,"1047":23,"105":28,"10520":9,"10555555555555556":1,"10572":29,"10589577":5,"106095":[11,27],"10694828":20,"1079":29,"10790125813226321":27,"108":[6,9],"10913":6,"10931453":[6,28],"10960":9,"10983159":11,"10th":9,"10x":[0,26],"11":[0,2,5,6,7,8,9,10,11,12,13,15,16,20,21,22,23,24,25,26,27,28,29],"1100":23,"1101":23,"11022072":11,"110221":11,"11022302e":5,"111":[1,7,12],"11100":9,"11202337":29,"112383":[11,27],"1124":9,"11368985":26,"11382896":28,"11388888888888889":1,"11390":9,"11400097":20,"114550":29,"11462415":5,"11482289e":[6,28],"11500354":20,"11507992e":1,"11547777218872497":[6,29],"115822":6,"11590":9,"11660":9,"11666666666666667":1,"11694":21,"117":8,"11744554e":[6,28],"11780":9,"118":[0,2,7],"11830757":28,"118318":27,"1184":29,"11840":9,"11890":9,"119":[0,2,7],"11938949":23,"11952661":23,"11967653":26,"119936":2,"12":[0,1,2,3,4,5,6,7,8,9,11,12,20,23,25,26,27,29],"120":[0,2,3,7],"1203":9,"1203284":8,"12044974":27,"120450":27,"121":[8,9,10],"12122786e":27,"1215pm":[24,26],"12182967":[6,28],"122":[2,8,9,10],"12222222222222222":1,"1225397":23,"123":2,"12318726e":[6,28],"12330033":27,"12333649":[6,28],"123711":6,"12380":9,"124":[0,26],"12400":9,"1242":8,"12552073e":[6,28],"12591227":27,"126":9,"1261":9,"12618549":5,"1265":9,"127":4,"12700133":20,"1271":6,"12777777777777777":1,"12790":9,"128":[3,4,13],"12871842":27,"129012":6,"12952339":28,"1297":9,"1298":9,"129963":29,"12pm":[24,26],"13":[0,2,5,6,7,9,11,12,18,20,21,23,26,27,29],"130":9,"13003291":[6,28],"130224":11,"13022421":11,"13055555555555556":1,"130694":27,"13069442":27,"131":9,"13118774":26,"13156239":20,"13179053e":27,"132":9,"13220608e":[6,28],"1325":6,"1326":9,"1326197715":26,"13280":9,"133":7,"13318759":20,"1332":29,"134":28,"135":9,"1350":29,"13535942":[6,28],"13588335":27,"136":[0,7],"13614217":20,"13633112":26,"13641587e":27,"13646574":5,"13658936":23,"13661243e":[6,28],"13679863":[6,28],"137":[0,7],"13740":9,"137652":[11,27],"138":[0,7],"1385":6,"13858317":26,"13865173":5,"13867518":26,"138775":[11,27],"1388888888888889":1,"13890":9,"139":[0,7],"1392559585020426e":[6,28],"139255958997547e":[6,28],"13929376":20,"139431112903922":29,"1394311129039245":29,"1395084586525954":29,"1395235273363669":29,"14":[0,2,4,5,6,7,8,9,10,11,12,20,21,23,25,27],"140":[0,2,7,9],"14021063":[6,28],"14092044":20,"141":[0,2,7],"14100":9,"1416398":[6,28],"14174745":[6,28],"1418":9,"142":[0,7,9],"14250":9,"14277718e":27,"143":[0,2,7,9],"1437":1,"144":[0,7],"14400":9,"1440501043841336":1,"14440":9,"1446729567":4,"145":2,"14568914":28,"146":2,"14629156":29,"14710":9,"14716":6,"14722222222222223":1,"147400":[11,27],"147420":[11,27],"1479":9,"148":[3,4,14],"14812206":6,"148282":6,"14932255":23,"149425":6,"14966323":20,"14984157":23,"14g":[6,29],"15":[0,2,4,6,7,8,9,12,13,15,16,17,20,23,26,29],"150":[3,4,8,9,14],"15005476":5,"150707":6,"15098090e":[6,28],"151":[3,4,14],"1512":29,"15130074e":[6,28],"152":9,"15200":9,"1527777777777778":1,"153":[3,4,14],"153036":27,"15313054":29,"15330788e":27,"15384615384615385":9,"154":[3,4,6,14],"15431202":20,"155":[3,4,9,14],"156":[3,4,14],"156956":5,"157":[3,4,14],"1575":9,"15792278643800434":5,"158":9,"15807529":26,"1587":9,"1590":9,"15990":9,"15am":26,"15g":[6,29],"16":[1,2,3,4,5,6,8,9,10,23,26,27,28,29],"1603":3,"16111111111111112":1,"16211139":5,"16220":9,"162246":5,"16231451":4,"1625":9,"1628":9,"16304863":27,"163049":27,"1630775253":1,"16342407":5,"16343471":[6,28],"16384":3,"16500":9,"16526258":20,"166":9,"16632351e":27,"16638104":20,"167787":5,"168":9,"16805821e":[6,28],"16855533":26,"17":[1,2,5,6,7,8,9,13,23,27,29],"1709":9,"171":28,"171525":27,"1715252":27,"17174962e":1,"17222222222222222":1,"1726":9,"17300":9,"1731":9,"17446471":6,"17483760":29,"17485935":20,"1752":9,"175300":27,"17561484":20,"17576105":26,"17641709":[6,28],"17748443e":27,"17755779e":27,"17777777777777778":1,"17801022":5,"17861098":6,"17917768":5,"17949575":5,"17953942":11,"1797":[1,3],"17979265":5,"179793":5,"18":[2,6,7,8,9,10,13,23,29],"1801709":28,"18029127":5,"1803":[],"1807":4,"18075088":23,"1809":9,"181":9,"1812":9,"1821":9,"182925111119767":20,"18307821":28,"18333333333333332":1,"18383522":27,"18388":6,"184":9,"1840":29,"184519":27,"185":28,"1860":9,"18611111111111112":1,"18613217e":[6,28],"18660":9,"18670072e":27,"18673098":11,"189496":27,"189622":27,"19":[2,6,9,15,23,26,29],"19166666666666668":1,"19207979":5,"19220":9,"1940":0,"194042826649355e":[6,28],"194042826815211e":[6,28],"1943":12,"1944":6,"1949983":26,"1956":9,"19569961":[6,27,28],"19688765":26,"1970":[20,26],"1973":9,"197370":[11,27],"19740":9,"1979":[6,29],"19800":9,"1999":[],"1_1":12,"1_2":12,"1_3":12,"1cm":[0,8,10,23,26],"1d":[1,2,3],"1e":[1,2,4,13,14],"1e10":14,"1e4":6,"1f":1,"1k":20,"1n":[0,26],"1x":[0,26],"2":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,19,20,21,22,23,25,29],"20":[0,1,2,6,7,8,9,13,23,24,26,28,29],"200":[0,2,3,4,8,9,10],"2000":[0,27],"20015436":6,"2004":13,"2006":25,"2008":26,"2010":1,"2011":1,"2014":4,"2015":1,"2016":[0,26],"2018":[0,6,27,29],"2019":[8,9],"2021":[6,14,27,28],"2022":26,"20222345":23,"2023":21,"2024":[15,16,17,18,26,27,28,29],"2027":9,"20277777777777778":1,"20488808e":27,"20500":9,"2060":9,"2069":9,"207545":[11,27],"20833333333333334":1,"20867052175034223":[6,29],"20916295":27,"20980":9,"21":[0,1,2,5,6,7,9,12,13,20,26,27,29],"210340":[11,27],"21058097":5,"21093775":6,"21117128e":27,"21130":9,"2116753732":4,"21169159e":[6,28],"213103":[11,27],"213743":[11,27],"21378153":26,"21517262":6,"21596432":6,"216290":[11,27],"216683":[11,27],"21733307e":27,"217962":11,"21796226":11,"21888121":20,"22":[0,1,2,5,6,9,12,13,20,23,26,27,29],"22019267":20,"22044605e":[5,27],"221":8,"2211108":20,"221180":27,"2216":9,"2218":9,"221805":2,"221921":5,"2222736":5,"222274":5,"222400":27,"22398902":26,"22467274":29,"225":4,"22668089":26,"226818":5,"22681817":5,"22690428":5,"22842468702219465":[6,29],"2284246870221947":[6,29],"22847924":5,"23":[1,2,6,9,12,15,20,23,26,29],"23002365e":[6,28],"23076923076923078":9,"23117504":26,"23167717":5,"23177156e":27,"232435":27,"23333333333333334":1,"2338675":27,"233868":27,"234":6,"23505507e":27,"2351":6,"23528337":29,"2364":9,"239":29,"2397":9,"23971032":27,"24":[0,1,2,6,9,20,26,29],"2401":21,"24140":9,"24155894":20,"24159785":27,"2416":9,"24175744e":[6,28],"2419":9,"2430":9,"24390":9,"24444444444444444":1,"24493666":29,"24502113":20,"2456969087678473":23,"24569691":23,"246":2,"24604925":20,"24829908":5,"24906604e":[6,28],"24975055":20,"25":[2,3,4,5,6,8,9,11,21,23,27,29],"250":[2,4,7,9],"25000":0,"250084":5,"2500845":5,"25185716":28,"251879":27,"252436":27,"25325161e":27,"25343155":29,"255":3,"255001":27,"25529356":26,"256":[2,4],"25617655e":[6,28],"25617658e":[6,28],"25695398":29,"2572":9,"2575":9,"259153":[11,27],"2597":9,"26":[2,6,9,16,29],"26093249":20,"26124692":6,"26292364":29,"26297455":27,"26301436":5,"26409315307910025":[6,28],"2640931530791004":[6,28],"2650":9,"265109911":4,"2654":9,"2662882":26,"26710969":5,"26773827":20,"26780278":5,"26792572":20,"268":9,"26814548":6,"26995402":27,"27":[0,1,2,6,29],"270350130579073":11,"27065633":28,"27486761":5,"274868":5,"2750":9,"27518808":26,"276263":[11,27],"27700":9,"27760":9,"27924636":5,"27n_":23,"28":[1,2,3,4,6,9,23,29],"280573":5,"280647":[11,27],"28205578e":27,"282075":5,"28207517":5,"28238649":20,"282727":[11,27],"2830637392":4,"28336218e":[6,28],"28390":9,"28391978":27,"28443039":29,"28475098":8,"2861":23,"2871":9,"2873":9,"2882":23,"2886":23,"2890":[0,26],"2892":23,"28986131":23,"29":[2,6,9,29],"2915":23,"29167186":5,"2931":26,"29313228":20,"29487549e":27,"296697":11,"29669702":11,"2968":26,"2980":26,"29822833":[6,28],"2990":26,"29920468":20,"299748":27,"2_":12,"2_1":12,"2_2":12,"2_3":12,"2_i":12,"2_m":[6,23,29],"2_t":13,"2_x":23,"2b":23,"2cm":8,"2d":[1,3,11,12,19,26],"2e":[6,29],"2f":[0,7,9,10,11,12,26,27],"2g":2,"2g_i":2,"2k":3,"2m":[6,29],"2n":[0,2,3,26,27],"2nd":9,"2p":23,"2pm":24,"2pt":4,"2u":[],"2x":[0,3,8,13,26],"2x_ix_jy_iy_j":8,"2x_j":8,"2y_i":10,"2y_j":8,"3":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,18,19,20,21,22,23,24,28,29],"30":[0,1,2,4,6,7,9,10,13,16,24,27,29],"30000":[0,26],"30010":9,"30119421":8,"30132427":20,"302":29,"30258509":[20,26],"30283188":28,"303":6,"30442964":27,"30466214e":[6,28],"30481195":20,"3072":3,"30764357e":27,"3076923076923077":9,"30787294":6,"30820040103372":11,"30873019":26,"30940":9,"3099":29,"31":[2,12,20,23],"310579":27,"3105791":27,"312":28,"3123314713548606":[6,29],"31276579e":[6,28],"313":28,"31318084":5,"313851":5,"31385129":5,"31457796":5,"31471398":26,"315":[6,28],"3155":[0,5,6,28,29],"31566127":26,"316":28,"31718909":11,"317367":11,"31896852":8,"31901056":23,"3199881":26,"32":[3,4,6,8,12,13,20,23,29],"3200":1,"3207":29,"32149601703519115":[6,29],"3214960170351912":[6,29],"3215":9,"32186558":26,"32238906":11,"322768":5,"3227683":5,"32351459e":27,"32399798":20,"324":2,"3245":2,"32450054":27,"3250":1,"3262":6,"326238":27,"32740721e":27,"32791346e":27,"33":[0,2,9,12,20,24,29],"33047051":5,"330471":5,"33066907e":[5,27],"33166055e":5,"331939":27,"333":7,"33435469":20,"33540382":20,"34":[2,9,20,29],"340071371496255":27,"3403":9,"340782":[11,27],"34114547":5,"34175429":23,"342680":27,"3436":[0,26],"34367970":6,"34369306":26,"3437":[0,26],"34412923":27,"34565973":11,"345660":11,"34569596":5,"34693879":26,"34782766":23,"3480159":20,"35":[0,2,6,17,21,23,24,26,29],"35140":9,"351636":[11,27],"35182854":5,"35389684e":27,"3544313922":6,"35444257":27,"35470445e":[5,27],"35514368":26,"35533773":6,"35577273":26,"356399":27,"357508":27,"35768161":26,"3581341341":4,"359":[5,28],"36":[0,2,5,6,21,23,29],"360":1,"36097931e":27,"3613":9,"361556":27,"3621311":5,"3632959111950474e":[6,28],"363295916366933e":[6,28],"36417623":26,"3655222":5,"367":[2,9],"36895644":29,"369139":[11,27],"36985812":26,"37":[9,21],"37040343":23,"370782966":4,"37239927e":27,"373465215701515":9,"37396662":[6,28],"37409975e":27,"37416502e":27,"37504943":20,"377372":27,"37737221":27,"3777801602":6,"37900111":[6,28],"38":[9,21,23],"380":9,"38135733e":[6,28],"38157698":26,"3817475779":[6,29],"3823545":20,"38267862":5,"382679":5,"38461538461538464":9,"386":9,"38629436":[20,26],"38827994":6,"38916861e":[6,28],"3892":29,"38962192e":[6,28],"39":[0,9,24,26],"39197698":27,"391977":27,"39214397":27,"392144":27,"394":9,"394285":11,"39428522":11,"3952007":20,"39579407":5,"39706027":23,"39706038":5,"397700":[11,27],"39789527":[20,26],"3983":6,"39931051e":27,"399836":27,"39990572e":27,"3d":[2,3,4,6,13,21,29],"3f":[1,3,9],"3n":20,"3x":[2,8],"3x_i":2,"3y":8,"3yk470mj5p931p9dtkk0y6jw0000gn":[1,6,26,29],"4":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,20,21,23,26,28,29],"40":[1,6,9,21,24,26,29],"400":4,"4000":[9,26],"40030144":20,"40103689":11,"401037":11,"401842":[11,27],"40332158":23,"40420245":20,"405":9,"4050":[25,26],"405890":[11,27],"40620066":27,"406201":27,"407":9,"40720059":23,"408":9,"4087":9,"4087793":5,"40892859":26,"40927184e":[6,28],"40contain":[],"41":[2,9,20],"4107":9,"41237041":26,"41253101":23,"41383693":26,"41433969":5,"4150":[],"41511965e":1,"4155":[2,15],"4162706317":6,"4166666666666667":9,"41691626":20,"41761415e":27,"41790782":26,"41847076":23,"418506":[11,27],"41858649":20,"41882037e":[6,28],"41988809":20,"42":[1,4,8,9,10,20],"4230769230769231":9,"42379872":23,"42441033":5,"42450":9,"426":[6,9],"42742185":26,"42765083e":27,"429":29,"43":[0,1,2,7,9,20],"43054282":5,"4310":26,"43246775e":27,"43295279":20,"433":29,"43330971e":[6,28],"43442461":20,"435163":27,"43579948e":[6,28],"43639284e":27,"436462435":4,"43766686":11,"438060758":6,"438136":27,"439230":6,"44":[0,1,2,20],"44049151":26,"44089210e":[5,27],"441":[3,4,14],"442":[3,4,14],"442600":[11,27],"443":[3,4,14],"443217":27,"44395301":28,"444":[3,4,9,14],"4442873":23,"445":[3,4,14],"446033":29,"44624525e":27,"44757627":23,"44970586e":1,"45":[0,2,9,24,26],"450":9,"450257":[11,27],"4504":9,"45312744":26,"45422069":20,"455173":29,"4557763":11,"455947":27,"456":9,"457":2,"458027":27,"458078":[11,27],"45915671e":27,"45937581":26,"45960079":5,"46":[2,9,24,26],"460":[9,11],"4601":9,"46124943e":27,"46153846153846156":9,"462":7,"46260797e":27,"46383925e":[6,28],"46383926e":[6,28],"46415888336127775":28,"46608732e":27,"46659511":26,"46713267":28,"46736215":29,"46749552e":27,"4676059":27,"467606":27,"46922796":5,"469228":5,"469596964296967":5,"46984697e":[6,28],"47":[2,9,24,26],"47042744":5,"470478":11,"4704785":11,"47051432":29,"470714":27,"47075725":6,"47116868e":[6,28],"47125748":5,"47132891":5,"47176716":29,"47176783":29,"47179152":29,"47187014":23,"47313680":29,"47610036":6,"47647836":26,"4768597":20,"47734350e":27,"47780495":5,"477805":5,"47815203":11,"47826973":23,"479465113":4,"48":[9,29],"481979":6,"48257387":[24,26],"48329035":23,"48447474":20,"48476997":11,"48494803":28,"48593774":20,"48655388":20,"48688666":11,"486887":11,"48780047":23,"48898864":20,"48994188":5,"49":[5,6,9,11,21,27],"49152":3,"4940954":[0,26],"49510364":20,"4959161509357395e":[6,28],"4959161509366685e":[6,28],"49622615e":27,"49724059":23,"49775827":20,"4990":23,"49905638":26,"4992":23,"4997":23,"4c4c7f":[9,10],"4d":3,"4f":6,"4pm":[24,26],"4y":8,"4y_i":10,"5":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,20,21,23,26,27,28,29],"50":[1,2,3,4,6,7,8,9,10,13,26,27,29],"500":[1,3,4,6,9,10,13,29],"50000455":5,"50000553":5,"50000718":5,"50000855":5,"50000969":5,"50001063":5,"50001142":5,"50001207":5,"50001261":5,"50001306":5,"50001343":5,"50001374":5,"500014":5,"50001414":5,"50001422":5,"50001439":5,"50001454":5,"50001466":5,"50001476":5,"50001485":5,"50001492":5,"50001498":5,"50001502":5,"50001506":5,"5000151":5,"50001512":5,"50001515":5,"50001517":5,"50001518":5,"50001519":5,"50001521":5,"50001522":5,"50001523":5,"50001524":5,"50001525":5,"50066971":23,"5018":23,"50227564e":[6,28],"50321091":5,"50427787":29,"50470576":26,"50530832":29,"506":0,"50641483":23,"506553":27,"50655336":27,"507d50":[9,10],"50846111e":[6,28],"50846112e":[6,28],"50j":13,"50x10":1,"51":10,"510":1,"51084243":20,"51097872":5,"51178631e":27,"511888":5,"51198558e":27,"51374050":29,"51383260e":27,"514219":27,"51542953":27,"51566561":26,"51569907":20,"51631859":26,"51681181e":27,"5177783846":4,"519327774467847":20,"52018478":20,"5222222222222223":1,"52362157e":27,"52511377":20,"52554699e":27,"52656770e":27,"526744":[11,27],"52780600":29,"528698":5,"52869849":5,"52870818":11,"52874252":5,"53":[2,9],"5303329":11,"5305555555555556":1,"531280":27,"53541027":20,"53603254":20,"53607935":20,"53703498":6,"53738247":29,"5378811":11,"5384615384615384":9,"539261":[11,27],"54":[2,6,9,23],"540":9,"54039921":5,"54041041e":5,"541605":27,"54213195":26,"543939":27,"54393936":27,"54428380e":27,"544439":27,"54637219":29,"55":[1,2,9],"55119373":20,"55217048":26,"5545673":26,"5555555555555556":1,"55640817":26,"5567":6,"55757089e":27,"557795":[11,27],"55796518":20,"55854694":11,"55960462":26,"56":[1,2,9],"56033697":5,"56198284":5,"56239321":23,"564":9,"56413916":23,"564374":[11,27],"56466087e":27,"565":9,"56536":[0,26],"56589683":29,"566":9,"56636616e":[6,28],"567":9,"568":[9,29],"56823411":20,"569":[1,9],"56912044e":[6,28],"56939714":5,"56963365":26,"56965674":29,"57":[0,2,8,9,24,26],"570":9,"571":[5,28],"571105947979344e":[6,28],"571105947979412e":[6,28],"57169969":26,"57201944e":[6,28],"574465":[11,27],"576":29,"57652181":26,"5769230769230769":9,"58":[2,9,10,24,26],"58102999e":27,"58310989":26,"58314173e":27,"5833333333333334":9,"583595":27,"58384652":26,"58492636e":27,"58734026":11,"58827236":6,"58886887":20,"5888888888888889":1,"59":9,"591317992":4,"59161025":20,"59354079":6,"5944444444444444":1,"596190":11,"5961905":11,"5972962":26,"59831212":20,"5cm":23,"5f":8,"5x":8,"5y":8,"6":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,20,23,24,26,27,28,29],"60":[1,3,6,9],"60000":4,"60004008e":27,"60010693e":27,"6019067271":4,"60236121":20,"60293962":5,"60314816":26,"60337375":23,"60342978":29,"60393882":20,"60420593":5,"60482174":20,"60518617":20,"60543038":27,"60656897e":27,"60673226":11,"606760":5,"60929685":29,"60943791":[20,26],"60958647":26,"6111111111111112":1,"61124231":20,"612939":27,"613579":27,"614808":[11,27],"6153846153846154":9,"61542957":20,"61562855e":27,"61602252":23,"61775176":29,"61810248":26,"618982":27,"62":2,"6200":26,"6201":26,"6202":26,"6203":26,"6204":26,"62201698e":27,"62373464":11,"625":7,"62528192":26,"62554572":23,"626635268":[6,29],"62725226":28,"62765236":20,"62788541e":27,"628":6,"62894215":5,"629019":11,"62901905":11,"62958747":26,"63":[0,1,2,6,9,29],"63023737":26,"63025821e":[6,28],"63037846":6,"63162342":27,"63249532e":[6,28],"63376422e":27,"63498144":5,"63588621":26,"63619296":20,"6377013":26,"63811548":20,"638618":20,"64":[1,2,3,4,13,20,26],"64012627":5,"640x480":[6,13,21],"64286758":20,"64316192":29,"64403731":20,"64605756":26,"646283":[11,27],"6473418":20,"64742912e":[6,28],"647473":[11,27],"649382":[11,27],"64961094":26,"64x50":1,"65":[1,2,8,9],"651":2,"6515332649941654":5,"65198643":23,"652":2,"652089":11,"65208925":11,"653":2,"6536392":27,"65367685e":27,"654":2,"65442354":27,"654424":27,"655":2,"65571174e":27,"65599456":27,"65743689":27,"658":2,"65833132":27,"65885453":5,"65891389":27,"658914":27,"659":2,"65904730e":27,"66":2,"660":2,"66064822":27,"66080313":27,"66204648":6,"66294408":27,"6638":9,"6652177":27,"66560":9,"66617168469673":23,"667":9,"668172":27,"66865889":26,"67":[2,9],"67047975e":[6,28],"672721":27,"67314874e":5,"674":29,"67496658":23,"67509467":5,"67801834":27,"67851352e":27,"68":2,"68078357e":27,"68192193":5,"68286725":27,"68534263e":[6,28],"68542204":5,"685942":11,"68594205":11,"6869":9,"68772578":26,"6887363571":4,"68929213e":[6,28],"689519":[11,27],"69":[2,9,23],"690":9,"690617":27,"69069n_":23,"69087404e":27,"691":1,"6923076923076923":9,"69313903":23,"69475072":5,"694751":5,"69484813e":5,"69504801":6,"69634577e":[6,28],"69695259":5,"6999536":11,"6n_":23,"6pm":26,"7":[0,1,2,3,4,5,6,7,8,9,11,12,13,14,15,20,22,23,25,26,27,28,29],"70":[1,6,7,9],"700258285068488":23,"70025829":23,"70037324e":27,"701370":5,"70162272":23,"70435575e":27,"70653767":4,"70710678":[5,27],"70832814":5,"7086067479626619":27,"71":1,"7106484566285455":23,"71142161":27,"711422":27,"7119":9,"712018":[11,27],"71384474e":27,"71538320e":27,"71737253":27,"71761101":27,"718165":5,"72048339":20,"72174172":11,"72219105e":27,"72271878e":[6,28],"7236674":5,"724":3,"72532661":26,"72545392e":27,"72664028e":27,"72793290e":27,"72859758":5,"72981762":8,"73":[6,9,29],"731000":27,"73184876":20,"73185968e":27,"73223692e":27,"733096":27,"73435999":26,"7345483":20,"73537890e":27,"73600557":23,"73972932":26,"74":[6,29],"740":9,"74063343e":27,"74070824e":27,"74081822":8,"74222022e":27,"7423220054006":23,"74232201":23,"74303070e":27,"74384949":27,"74474996":26,"7458376":26,"74607851":27,"74647981e":27,"74720185":20,"74829661":27,"74830119e":27,"74840039":26,"74840212":5,"74947597":11,"749476":11,"749765":27,"75":[5,6,8,9,11,21,27,29],"7500735612987705":11,"750445":27,"75050135":27,"75079864":26,"751699":[11,27],"75170092":5,"75199355":20,"75269037":27,"75309603":20,"75437273":11,"754373":11,"75576555":27,"756352":27,"7577":29,"75770568":27,"76":[2,9,24,26],"76160572":0,"76187240e":27,"76190591":28,"76212914":26,"76428845":26,"76471048":20,"76478383":23,"765":7,"76681867":11,"76690383":29,"76887029e":27,"7692307692307693":9,"76936315":5,"7694444444444445":1,"77":[9,24,26],"77152076":5,"7718":9,"77519276":20,"77714169":8,"7782028952":4,"77921992":20,"77966116":23,"7796611648499043":23,"78":2,"78009660e":27,"78058232e":27,"7809358":23,"78156479e":5,"78184120e":[6,28],"78546777":26,"78584667":23,"78890834":23,"78941903":5,"79":2,"79036971":5,"790370":5,"79111643":5,"79189954":26,"793167":27,"794282":[11,27],"7953443":28,"79633065":29,"79689487":26,"79704553":20,"79778008":6,"797e":6,"79909592":29,"7993408651198877":29,"79934087":29,"7995707762668065":29,"7d7d58":[9,10],"8":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,18,20,21,23,26,27,28,29],"80":[0,1,2,5,6,8,9,27,28,29],"800":[4,7],"80006807":26,"80291466":26,"80354994":6,"80460179":29,"80469739":5,"8055555555555556":1,"80609615e":[6,28],"80609616e":[6,28],"80816668":5,"808167":5,"80847477e":[6,28],"80919738":20,"81":1,"81048318e":[6,28],"81160425":5,"81194394":26,"81203214":23,"81333808":6,"815am":[24,26],"81633628":11,"816454":27,"816847":27,"81700981":26,"82":2,"82168579e":27,"82198978":5,"82566059e":27,"8265786":5,"82924408":5,"83":2,"8305555555555556":1,"831048":5,"8310483":5,"83146596":29,"83223115":23,"83460016":20,"83512277":5,"83870189":6,"83954195":23,"84":2,"842":[0,7],"842436":27,"843":[0,7],"84355903e":1,"84388592":26,"844":[0,7],"84443254e":1,"8461538461538461":9,"84716164":23,"84780262":6,"84880366":11,"8492168":26,"84923989e":[6,28],"84963695":5,"849637":5,"84994524":5,"85":[1,2,9],"85013763":26,"850164":5,"85263220":6,"85278920e":5,"85533937":20,"85601992":5,"858":9,"8583333333333333":1,"86":[0,2],"86062977":6,"86117291":5,"86134827":5,"86145244":11,"86260426":20,"86273371":28,"8638888888888889":1,"865":[0,7],"86528841":26,"86557766e":27,"866":[0,7],"86630":9,"8666666666666667":1,"867":[0,7],"86810":9,"87":9,"8702784034":4,"87068171":28,"87072815e":27,"87078":11,"8722222222222222":1,"87381451":5,"875":1,"87599676":20,"87627342":29,"877":6,"87923449":26,"88":[2,26],"88046261":5,"8805555555555555":1,"88168312e":[6,28],"88182591":27,"88253432e":27,"88323026":27,"88336879":5,"88347937":20,"88354703":20,"8840785":20,"88422263":20,"884669":27,"88529063e":[6,28],"88613493":27,"8888888888888888":1,"89":[2,9],"89126914e":27,"89261577":11,"892616":11,"89288636":11,"89410423":5,"89442642":20,"8944444444444445":1,"89685175":23,"89757312":23,"89794988":26,"89800468":26,"89906633":26,"89954169e":27,"89971681e":27,"8f":[6,29],"8g":[6,29],"8n":20,"8x8":1,"9":[0,1,2,3,4,5,6,7,8,9,11,12,13,14,18,20,21,22,23,24,26,27,28,29],"90":[1,2,6,9],"90075537":5,"9011":6,"90220243":5,"90266948":5,"9027777777777778":1,"9040":9,"90475506e":[6,28],"9054":29,"9055555555555556":1,"9059370566788297":23,"90593706":23,"906747":5,"9090666":20,"90914803":26,"909446359461397":11,"91":[24,26],"910":9,"9106590586410548":23,"9111111111111111":1,"91128596":5,"91358186":20,"91402008e":27,"9149137":20,"91492986e":[6,28],"91561163":20,"91660026":5,"9166666666666666":9,"91760278":5,"91812702":5,"918992":27,"91924234":23,"9197998358799464":23,"92":[6,9,24,26],"9230769230769231":9,"92397145":26,"924251":11,"9242511":11,"924e":6,"925":1,"92507116e":1,"92578916":5,"926564304542535":26,"92674248":20,"92740215":23,"92756256":28,"9277777777777778":1,"92878175":23,"9305555555555556":1,"931":[0,26],"93155188":5,"93158979":5,"93176482":26,"933":[5,28],"93356853e":27,"93492130e":[6,28],"93601008e":27,"937":23,"937082":27,"93742122":26,"93792515":26,"93799826":5,"938":23,"9387":9,"939":[0,23,26],"9392":6,"94":[9,23],"94226022e":[6,28],"94273542":27,"94284104":5,"94320205":5,"9444444444444444":1,"94556653e":27,"94591015":[20,26],"94639099":11,"946957":5,"9472222222222222":1,"94781064":28,"9482527":5,"94897736":26,"9492":6,"95":[1,9,11,29],"95008046":6,"95154558":20,"95231424":5,"952387":27,"95274315":20,"9527777777777777":1,"95284275":5,"953065564":1,"95351665":5,"954":23,"95511792":27,"955118":27,"9555555555555556":1,"955820c21e8b":4,"956563":[11,27],"95684892":5,"95782166e":27,"958228616652075":5,"95980855e":27,"96":[6,9,11,29],"960":23,"9601304850035702e":[6,28],"9601304850073734e":[6,28],"96024953":5,"9603":6,"96084663":5,"961":23,"962":23,"9637117593816477":[6,28],"9640435":5,"9649652536":4,"965548":27,"96631321":27,"96688672":5,"9674916":5,"967809":[11,27],"97005689":5,"97056672":20,"97065359":29,"97108525e":27,"9722222222222222":1,"97243128":5,"972591":5,"97300836":5,"97497404e":[6,28],"975":1,"97507735":5,"97630233":29,"97705827":5,"97758848":5,"9777777777777777":1,"9780387310732":25,"9780387848570":25,"9781098134174":26,"9781492032632":25,"9781801819312":26,"978553":5,"97898392":[6,27,28],"97906022e":27,"97926491":5,"97979891":6,"97992238e":27,"98":[0,1,9],"980":9,"9805555555555555":1,"98091621":5,"981321":27,"98139097":5,"98275501":5,"98279090e":27,"9828":29,"983310":27,"98368659":26,"9841":29,"98413059":5,"98454786":5,"985":23,"98566191":5,"986":23,"9861111111111112":1,"98625331e":27,"98650215":26,"98650823":26,"986699":5,"98680716":5,"98716878":5,"98808176":5,"98822371":6,"9888888888888889":1,"989":23,"9890348":5,"9893447":5,"98947894":27,"9898ff":[9,10],"99":[6,9,11,13,29],"990":9,"99009525":5,"99021575e":27,"99088801":5,"991":23,"99115119":5,"99176998":5,"992":23,"99217800e":27,"9923539830272697":27,"99242921":5,"99265097":5,"993":23,"99316252":5,"99371056":5,"99371466":23,"99389612":5,"9943201":5,"9947756":5,"99492986":5,"99509854":26,"9951502749739212":27,"99528218":5,"99539415":5,"99566069":5,"99578809":5,"996":[5,28,29],"99608161":5,"9963961":5,"9964486445275116":27,"99650061":5,"9967458":5,"99700706":5,"99709215":5,"99729756":5,"99751458":5,"99758326":5,"99762142":28,"99775587":5,"99793613":5,"99799099":5,"998100":5,"99810021":5,"99813653":5,"99828624":5,"9983295":5,"99845267":5,"99854557":27,"998577":5,"99861053":5,"99862019":27,"99869482":27,"99871521":5,"99876945":27,"99881845":5,"99883628":27,"99884384":5,"99884409":27,"99884986":28,"99891093":27,"99891873":27,"99893323":5,"99898558":27,"999":[9,23],"99901896":5,"99903755":5,"99906023":27,"9990927157696581":28,"99911427":5,"99912709":27,"99913489":27,"99918546":5,"99919837":5,"99920175":27,"99926459":5,"99927642":27,"9993237":5,"99933188":5,"9993511":27,"99938942":5,"99941797":27,"9994385":5,"99944272":5,"99949266":27,"99949306":5,"99953381":5,"99953475":5,"99956735":27,"99957911":5,"99961294":5,"99965056":5,"99967865":5,"99970894":27,"99970988":5,"9997332":5,"99975913":5,"99977849":5,"99978365":27,"99980002":5,"9998161":5,"99984732":5,"99987324":5,"99989476":5,"99991263":5,"99992746":5,"99993978":5,"99995":5,"999955585168597":[6,28],"9x":[6,21],"9y":[6,21],"\u00f8yvind":[6,27,28],"abstract":1,"boolean":4,"break":[0,4,6,11,14,26],"byte":[20,26],"case":[0,1,2,3,4,5,6,7,11,12,13,14,16,19,20,21,26,29],"catch":0,"class":[0,1,3,4,6,7,8,9,11,12,13,23,26,29],"default":[0,1,2,4,6,7,8,16,20,21,26,27,28],"do":[0,2,3,4,5,6,8,9,10,11,12,13,14,15,16,20,21,26,27],"ekstr\u00f8m":4,"export":9,"f\u00f8470":[24,26],"final":[0,1,2,3,4,5,6,7,8,9,10,11,13,14,16,18,21,22,23,24,26,29],"float":[0,3,4,5,9,11,13,14,20,26,27],"function":[2,3,4,5,9,14,15,16,17,18,19,20],"import":[0,1,2,3,4,6,7,8,9,10,11,12,13,14,15,16,17,21,23,29],"int":[0,1,2,3,4,5,6,11,13,14,20,23,27,29],"long":[0,1,3,4,12,13,26],"m\u00f8svatn":[6,21],"new":[0,1,2,3,5,6,7,8,9,10,11,13,14,17,20,26,27],"public":[0,19,26],"return":[0,1,2,3,4,5,6,7,8,9,11,13,14,20,21,23,26,27,28,29],"sch\u00f8yen":[6,27,28],"short":[4,5,21,28,29],"super":[5,27,28],"switch":0,"throw":[3,6,23,29],"true":[0,1,2,3,4,5,6,7,8,9,10,12,13,14,15,16,20,21,23,26,27,28,29],"try":[0,1,2,4,5,6,7,8,9,10,11,13,14,15,16,19,20,21,23,26,27,28,29],"var":[1,5,6,10,11,18,21,23,26,27,28,29],"while":[0,1,3,4,5,6,7,8,9,11,12,13,23,26,27,28,29],A:[2,3,5,6,7,10,11,12,13,16,18,19,20,21,22,23,24,25,27],AND:2,And:[0,3,4,5,6,9,13,19,21,23],As:[0,1,2,3,4,5,6,8,10,12,13,16,20,21,23,26,27,28,29],At:[0,4,6,13,21,26],BE:[0,26],Be:[2,19,26],Being:13,But:[0,1,2,3,5,6,9,10,23,27,28,29],By:[0,3,5,6,12,13,17,20,26,27,28,29],For:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,25,26,28,29],IF:[6,28,29],IN:25,If:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,15,16,19,20,21,23,26,27,28,29],In:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,19,20,21,23,25,26,27,28,29],Is:11,Ising:[5,12,27],It:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,19,20,21,23,26,27,28,29],Its:[1,2,4,11],NO:[9,11],No:[0,6,7,9,26,28],Not:[0,1,5,6,27,28,29],OR:23,Of:23,On:[0,3,9,15,21,22,23,24,25,26],One:[0,1,3,4,5,6,7,8,11,12,13,17,21,23,27,29],Or:[0,1,6,21,26],Such:[0,6,12,16,23,29],That:[0,5,7,10,11,12,14,21,23,26,28,29],The:[4,10,13,14,15,16,18,20,21,22,23,24,25],Then:[0,1,6,8,9,10,11,12,13,14,20,21,26,29],There:[0,3,4,5,6,8,9,11,12,14,20,21,22,23,24,26,27,28],These:[0,2,3,4,5,8,9,10,11,12,13,14,15,16,20,21,23,24,26,27,28],To:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,20,23,28,29],With:[0,5,6,8,9,10,11,12,14,18,20,21,23,26,27,29],_0:[5,8,10,11,13,27],_1:[2,5,6,8,10,11,12,13,14,20,27,28,29],_2:[2,5,8,11,12,13,20,27],_3:20,_4:20,_9:13,_:[0,1,2,4,5,6,7,8,9,10,11,12,13,17,18,20,21,26,27,28,29],__class__:10,__doc__:[6,29],__enter__:[0,7],__future__:[8,9],__getattr__:28,__getattribute__:26,__getitem__:[0,7],__init__:[0,1,2,3,4,7,14,28],__main__:2,__name__:[2,10,28],_accessor:26,_auto10:[6,12],_auto11:6,_auto12:6,_auto1:[2,3,4,5,6,7,12,13,20,23,27],_auto2:[2,3,4,5,6,12,13,20,23],_auto3:[3,4,5,6,12,13,20],_auto4:[4,6,12,13,20],_auto5:[4,6,12,13,20],_auto6:[4,6,12,20],_auto7:[4,6,12,20],_auto8:[6,12],_auto9:[6,12],_base:8,_build:[0,17,19,21,25,26],_c:1,_can_hold_identifiers_and_holds_nam:26,_check_optimize_result:[9,11],_class:8,_compon:11,_coordinate_desc:6,_depth:9,_fraction:9,_generatorcontextmanag:[0,7],_i:[0,1,2,5,6,7,8,11,12,13,15,16,17,21,26,27,28,29],_info_axi:26,_j:[0,1,2,3,5,6,8,13,17,21,27,28,29],_k:13,_l:12,_lambda:[6,26],_leaf:9,_ll:[3,4,14],_logist:[9,11],_m:10,_make_vjp:2,_multilayer_perceptron:1,_n:[2,5,8,11,13,27],_node:[2,9],_o:[3,4,14],_open_file_or_url:[0,7],_p:[5,8,27],_plugin_dir:[3,4,14],_ratio:11,_rc_params_in_fil:[0,7],_s:[3,4,14],_sampl:9,_split:[6,9,21],_t:13,_test:[6,21],_tf_getinputpropertieslist:[3,4,14],_trace:2,_valu:2,_varianc:11,_weight:9,a0:3,a0faa0:[9,10],a1:[0,26],a2:[0,26],a3:[0,26],a4:[0,26],a_0:[0,26],a_1a:[0,26],a_1x:[],a_2a:[0,26],a_2x:[],a_3:[0,26],a_3a:[0,26],a_4:[0,26],a_4a:[0,26],a_:[0,1,16,20,26,27],a_h:1,a_i:[0,1,2,12,26],a_j:[1,12],a_k:[0,1,12],aaron:25,ab:[0,2,5,13,14,21,26,27,28],ab_channel:19,abandon:1,abbrevi:22,abid:23,abil:[0,10],abl:[0,1,4,5,6,7,10,12,13,16,21,27],about:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,19,20,21,24,28,29],abov:[0,1,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,25,26,27,29],abovement:[6,26,29],abscissa:13,absolut:[0,2,5,6,13,26,27,28,29],absorb:[27,28],acceler:13,accept:[0,3,6,9,21,26,27],access:[0,3,11,23,26],accid:[4,6,29],accompani:[0,26,27],accomplish:[8,9,13],accord:[0,1,2,5,6,9,12,13,14,15,16,23,26,28,29],accordingli:11,account:[0,3,5,13,15,23,26,28,29],accumul:[12,13,23],accur:[0,3,4,6,10,13,29],accuraci:[0,1,3,4,5,6,7,9,10,11,12,26,27],accuracy_scor:[0,1,10,26],accuracy_score_numpi:1,achiev:[0,1,5,6,8,12,20,26,28,29],aco:23,acquaint:19,acquir:[1,19,26],acr:0,across:[1,3,6,9,19,26,29],act:[1,3,20],action:23,activ:[0,2,3,4,9,22,24,26,29],actual:[0,1,4,5,6,8,11,16,20,23,26,27,28,29],ad:[1,3,4,5,8,13,15,16,20,28,29],ada_clf:10,adaboostclassifi:10,adadelta:13,adagrad:[],adam:[1,3,4,24,26],adapt:[4,6,13,25,29],add:[0,1,2,3,4,5,6,8,10,11,12,15,16,17,21,23,26,27,28,29],add_outgrad:2,add_subplot:[1,7,12,14],addendum:5,addit:[0,2,3,5,6,7,8,9,10,12,13,15,17,19,20,21,23,24,25,26,27,28,29],addition:[12,13],address:[1,9,11,13,26],adjac:[3,12],adjoint:[5,27,28],adjust:[0,5,12,13],admir:[0,26],advanc:[4,6,12,25,26,29],advantag:[1,3,5,6,10,13,20,28,29],adversari:26,afecionado:26,affect:3,affin:[0,3,8,11,27],afford:3,aficionado:26,aforement:14,african:0,after:[0,1,2,4,5,6,9,11,12,13,15,19,20,21,23,26,27,28,29],afterward:[0,26],ag:[0,7,22,26,27],ag_0:2,again:[0,1,4,5,6,7,8,10,11,12,13,15,16,21,23,26,27,28,29],against:[1,4,7,10],agegroup:7,agegroupmean:7,aggreg:[9,10],agorithm:10,agre:[5,6,23,28,29],agreement:13,ahead:9,ai:[0,25],aid:11,aim:[0,1,4,6,7,11,14,15,16,19,20,21,27,29],ainv:5,airplan:3,aka:[5,28,29],al:[0,2,4,15,16,17,21,25,26,27,28,29],alarm:[5,7,28,29],aldo:27,algebra:[0,3,5,13,19,27,28,29],algorithm:[0,1,2,4,5,6,7,8,13,14,19,20,21,23,25,26,28,29],align:[0,2,5,6,7,8,13,21,23,26,27,28,29],all:[0,1,2,3,4,5,6,7,9,10,11,12,13,14,15,19,20,21,22,23,24,25,26,27,28,29],allevi:[1,13],alloc:[3,20],allow:[0,1,2,3,5,6,8,10,13,15,16,19,20,21,26,27,28,29],almost:[0,1,6,8,11,13,23,29],alon:[2,9],along:[2,3,4,5,6,9,10,11,19,20,26,27,28,29],alpha:[0,1,2,3,4,6,7,8,9,10,13,14,23,26,27,29],alpha_0:3,alpha_1:3,alpha_2:3,alpha_:10,alpha_i:[3,13],alpha_k:13,alpha_m:10,alpha_n:3,alpha_opt:13,alreadi:[2,3,4,5,6,10,12,19,20,23,26,27,28,29],also:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,26,27,28,29],alter:1,altern:[0,1,4,5,6,8,9,11,13,20,21,26,27,28,29],although:[0,1,5,6,8,10,13,16,26,28,29],alwai:[0,3,5,6,12,13,16,21,23,26,27,28,29],am:4,ame2016:[0,26],american:0,among:[0,3,5,9,10,12,20,26,27,28],amongst:[5,28,29],amount:[0,1,3,4,6,8,10,14,19,29],an:[1,2,3,5,6,7,8,9,11,12,13,14,15,16,17,19,20,21,23,24,25,27,28,29],an_:23,anaconda:[0,1,15,19,21,26],analog:13,analys:[6,28,29],analysi:[1,3,4,7,14,15,16,17,20,25],analyt:[2,3,5,6,7,12,13,15,19,21,26,27,28,29],analyz:[0,1,3,4,5,6,16,17,21,23,27,28],andrew:1,angl:[0,3,9,27],anharmon:3,ani:[0,1,2,3,4,5,6,7,8,9,10,12,14,18,23,26,27,28,29],anim:[4,12],ann:12,annot:[0,1,3,7,8,26],announc:26,anoth:[0,1,3,4,5,6,7,8,10,11,12,13,20,21,23,26,27],anp:2,ans_vspac:2,ansatz:[0,26],answer:[0,1,3,5,6,20,21,24,26,28,29],antialias:[2,6,21],anticip:4,anymor:[1,8],anyon:[4,8],anyth:[1,23],anytim:[24,26],apach:1,apart:[11,13],api:[1,19,26],appar:2,appear:[0,1,3,13,16,20,23],append:[1,3,4,8,9,13,26],appendix:21,appli:[0,1,2,3,4,6,7,8,9,10,11,12,13,15,16,21,23,25,26,27,28,29],applic:[0,1,3,4,5,6,7,9,12,13,20,21,23,25,26,27,29],apply_gradi:4,approach:[1,2,4,5,6,9,10,11,12,13,17,19,23,25,27],appropri:[2,6,9,12,13,19,23,29],approv:26,approx:[0,2,3,6,10,11,13,23,26,29],approxim:[0,1,2,3,4,5,6,7,10,11,13,18,21,23,26,27,28,29],apt:[0,15,19,21,26],aq:23,ar:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,19,20,21,22,23,24,25,26,27,28,29],aragorn:26,arang:[1,3,4,6,7,9,10,12,13,21,26],arbitrari:[1,4,6,8,12,13,23,29],arbitrarili:[0,1,11,26],arc:[6,21],architectur:[3,4,12],archiv:[],area:[0,3,6,9,21,25,26],arg:2,argmax:[1,11],argmin:[4,10,14],argnum:2,argnum_0:2,argnum_1:2,argsort:11,argu:[1,13],argument:[0,2,3,5,6,11,12,13,21,26,27,28,29],argval:2,aris:[0,6,12,13,23,26,29],arithmet:[0,13,20,26],arm:[6,28],armadillo:20,around:[0,1,4,5,6,11,23,26,28,29],arr:2,arrai:[0,1,2,3,4,5,6,7,8,9,11,12,13,14,19,21,23,27,28,29],arrang:[3,26],arraybox:13,arriv:[0,6,9,11,20,23,26,29],arrow:12,arrowprop:8,art:[0,1,15,19,26],articl:[0,3,4,6,10,18,21,26,27,28,29],artifici:[0,2,7,12,25,26],artificialneuron:12,arug:13,arxiv:[3,4,21],asarrai:[0,2,6,9,27,28],asid:27,ask:[5,6,11,12,21,28,29],aspect:[0,6,19,21,26,27,28],assembl:[0,3,26],assert:4,assess:[0,6,21,26,27,28,29],assici:4,assign:[0,7,8,9,12,13,14,22,24,25,26],associ:[0,6,9,12,14,23,26,29],assum:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,21,23,26,27,28,29],assumpt:[0,3,5,6,9,11,18,21,23,26,27],ast:[0,5,6,18,21,26,28,29],astronomi:[26,28,29],astyp:[4,9,10],asymmetri:[0,26],asymptot:[4,6,29],atom:[0,26],attempt:[0,4,6,7,8,10,26,28],attend:22,attent:[0,20,26],attr:28,attract:[0,10,26],attribut:[0,9,26,28],attributeerror:[26,28],audi:[0,26],audio:[3,4],august:[15,16,26],aurelien:[0,15,22,25,26],austfjel:[6,21],author:[0,1,10,23],authour:26,auto:[8,9,10,23],auto_exampl:[21,28],autocor:23,autocorrelation_tim:23,autocorrelform:23,autocovari:23,autoencod:[4,19,26],autoencond:19,autograd:[19,26],autom:[0,19,25,26],automac:20,automag:26,automat:[0,1,2,3,4,11,16,19,20,26],automobil:3,autonom:4,avail:[0,1,4,6,7,10,11,15,19,20,21,22,24,25,26,29],averag:[0,1,3,6,9,10,13,14,23,24,26,27,28,29],avoid:[0,4,5,6,9,11,13,20,26,27,29],awai:[2,3,6,27,28,29],awar:[2,10],award:[24,26],ax:[0,1,2,3,4,6,7,8,9,10,11,12,13,14,20,21,26,29],axes3d:[2,6,13,21],axes_grid1:6,axhlin:8,axi:[0,1,2,3,4,6,7,8,9,10,11,12,13,14,20,21,23,26,27,28,29],axiom:[5,28,29],axlabel:[0,26],axvlin:[4,8],axvspan:4,b1:8,b2:8,b3:8,b:[0,1,3,4,5,6,8,9,10,12,13,14,15,16,23,24,26,27,28,29],b_0:0,b_1:[0,2,12,13],b_2:[0,13],b_5:13,b_:[0,1,20],b_group:9,b_i:[0,1,2,12,26],b_ia_:[0,26],b_ia_i:0,b_index:9,b_j:[1,12],b_k:[0,1,12,13],b_m:12,b_score:9,b_valu:9,babcock:26,bachelor:[22,24],back:[0,3,4,5,6,8,9,10,15,20,23,26,27,28],backbon:20,backend:[0,1,4,7],background:[25,26,27],backpropag:1,backtrack:9,backup:20,backward:[1,2,4,12,20],backward_pass:2,bad:[6,27,28],badli:23,bag:[9,19,26],bag_clf:10,baggin:26,baggingboot:10,baggingclassifi:10,baggingtre:10,balanc:[6,29],band:20,bandwidth:20,bar:[0,6,11,15,16,21,26],barber:25,bare:[4,10],base:[0,1,3,4,5,7,8,9,10,14,19,23,24,25,26,27,28],basi:[5,7,8,10,11,12,13,20,27,28],basic:[2,6,8,12,13,14,19,21,23,26],batch:[3,4,11,12,13],batch_shap:4,batch_siz:[1,3,4],batchnorm:4,bay:7,bayesian:[5,19,25,26,28,29],beam:[26,29],becattini:24,becaus:[0,1,2,3,4,5,6,8,9,12,13,14,26,28,29],becom:[0,1,2,5,6,7,9,12,13,23,26,27,28,29],been:[0,1,2,3,4,5,6,11,12,13,19,20,21,26,27,28,29],befor:[0,1,2,3,4,5,6,7,8,12,13,14,20,21,23,26,27,28,29],beforehand:[0,23,26],begin:[0,1,2,3,4,5,6,7,8,9,11,12,13,14,20,21,23,24,26,27,28,29],behav:[1,6,13,29],behavior:[0,1,13,26],behaviour:12,behind:[0,1,6,8,13,26],being:[0,1,2,3,4,5,7,8,10,11,12,13,17,23,26,27,28,29],believ:[9,20],belong:[7,8,9,13,14],below:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,20,21,23,26,27,28,29],benchmark:10,benefici:[1,13],benefit:[0,1,4,11,13,15,19,26],bengio:[1,22,25,26,27],benign:[1,7,9],besid:[4,5,28],bessel:[5,27,28,29],best:[0,1,2,3,4,5,6,7,8,9,10,12,13,21,24,26,27,29],beta:[0,1,3,5,6,7,10,11,13,16,17,18,21,26,27],beta_0:[0,1,3,5,6,7,13,26,27,28,29],beta_0x_:[0,26,27],beta_1:[0,1,3,5,6,7,10,13,26,27,28,29],beta_1x_0:[0,26],beta_1x_1:[0,7,26],beta_1x_2:[0,26],beta_1x_:[0,26,27],beta_1x_i:[7,13,27],beta_2:[0,3,13,26,27],beta_2x_0:[0,26],beta_2x_1:[0,26],beta_2x_2:[0,7,26],beta_2x_:[0,26,27],beta_2x_i:27,beta_3:3,beta_3x_i:27,beta_4x_i:27,beta_:[0,3,6,7,13,26,27,28,29],beta_i:[0,3,5,26,27,28],beta_j:[0,5,6,13,18,21,26,27,28,29],beta_k:13,beta_linreg:13,beta_m:10,beta_mg_m:10,beta_n:3,beta_ol:29,beta_p:7,beta_px_p:7,beta_ridg:29,betaol:29,betaridg:29,betavalu:5,better:[0,1,2,3,4,6,9,10,11,12,13,26,27,28,29],between:[0,1,2,3,4,5,6,7,8,9,11,12,13,14,17,21,23,26,27,28,29],beyond:[0,1,5,6,8,13,21,26,27],bf:[13,14,20,23],bg:26,bgd:13,bia:[0,1,2,3,5,8,9,10,12,13,26,27,28],bias:[1,2,3,5,6,9,12,28],bibliographi:21,big:[0,1,2,5,6,14,18,28,29],bigger:[1,6,27,28],bigr:12,bike:9,bilbo:26,billion:[3,12,19],bin:[0,7,23],binari:[0,3,5,7,9,10,12,26,28,29],binarycrossentropi:4,bind:[0,27],binomi:[19,23,26],binsboot:[6,29],bioinformat:0,biolog:[1,12],bios1100:[19,26],bird:[0,3],birth:26,bishop:[22,25,26],bit:[1,4,20,23,26],bitwis:23,bivari:2,bk:[0,13],bla:[20,26],black:[8,9,14],blob:[16,21],block:[6,10,19,20,23,26,29],blog:26,blogpost:4,blue:[0,3],bmatrix:[0,1,3,5,7,8,11,13,20,26,27,28],bmi:1,bodi:[0,1,4,12,26],bold:1,boldfac:[0,5,16,27,28],boldsymbol:[0,1,2,3,5,6,7,8,10,11,13,14,15,16,17,18,21,26],boltzmann:[12,19,26],book1:25,book:[21,25,26,27],boost:[1,9,19,26],boostrap:10,bootstrap:[1,13,19,21,26],borrow:26,boston_dataset:0,bot:8,both:[0,1,4,5,6,8,9,10,13,14,15,16,19,20,21,23,24,26,27,28,29],bottl:7,bound:[0,8,12],boundari:[2,4,8,11,12],box:[4,9],boxed_arg:2,boyd:[8,13],bracket:[4,23],brain:[1,7,12],branch:9,breast:[5,7,11,28,29],breat:[],breviti:13,brew:[0,15,19,21,26],brg:8,brief:[21,27],briefli:[0,26],bring:[0,5,6,10,27,28],broad:0,broadcast_idx:2,brought:[13,19,26],brownle:4,browser:26,brute:[3,5,11,27,28],bs:[8,9,10],buffer_s:4,bui:4,build:[0,4,5,6,10,20,23,26,28,29],built:[0,1,3,4,6,29],bunch:11,busi:0,c1:[8,11],c2:[8,11],c:[0,1,2,4,5,6,7,8,9,10,11,12,13,14,15,16,19,20,23,24,25,27,28,29],c_0:23,c_1:12,c_2:12,c_3:12,c_4:12,c_:[0,8,9,10,13,23],c_i:[12,13],c_k:23,ca:[1,26],cach:10,cal:[0,8,10,12,13,15,16,26],calcul:[0,1,2,4,5,6,8,9,10,11,12,13,14,15,16,18,20,23,26,28,29],california:[],call:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,18,19,20,21,23,24,26,27,28,29],calor:[0,27],cambridg:[13,25],can:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,23,24,25,27,28],cancel:[0,13,26,27],cancer:[5,10,28,29],cancerpd:7,candid:[8,9,10],cannot:[0,1,4,5,6,7,8,9,22,23,27,28],canopi:[0,15,19,21,26],canva:[21,26],cap:[5,28,29],capabl:[0,1,8,13,19,26],capac:[2,24],capita:0,caption:21,captur:[4,11,12],car:[3,4],card:[0,7,26],cardin:1,care:11,carefulli:13,carlo:[0,6,19,23,25,26,29],carri:[2,6,7,21,29],cart:10,casella:25,cast:1,cat:[3,4],categor:[0,1,3,9,11,26],categori:[0,1,3,7,10,12,14,26],categorical_crossentropi:[1,3],caus:[0,5,6,7,23,26,27,28,29],causal:0,causat:[0,26],cax:1,cb:[6,26],cbar:1,cc:[0,1,5,13,26,27,28],ccc:[5,12,28],cd_fast:6,cdf:23,cdot:[0,2,6,12,13,14,20,23,26,29],celebr:13,cell:[0,1,2,3,4,6,7,8,9,10,13,14,21,28,29],center:[0,1,6,7,8,9,11,14,21,23,26,28,29],central:[0,3,5,6,8,16,20,26,27,28],centroid:[14,23],centroid_differ:14,centuri:3,certain:[0,3,6,7,9,23,26,27,29],certainti:29,cg:13,cha:0,chain:[0,1,13,19,23,26],chanc:[1,5,13,23,28,29],chang:[0,1,2,3,4,5,6,8,9,11,12,13,14,20,21,23,26,27,28,29],channel:3,chapter3:[0,21],chapter:[0,6,10,11,16,18,20,21,25,26,27,28,29],charact:[0,3,5,8,26,27],character:[8,9,10,12,23],characterist:[0,1,3,10,13,26],charg:[0,26],charl:0,chase:4,chatgpt:21,chd:7,chddata:7,cheap:[5,27],cheaper:[1,13],check:[0,1,3,4,5,6,11,13,20,26],checkmark:3,checkpoint:4,checkpoint_dir:4,checkpoint_prefix:4,chen:10,cheng:27,chiaramont:2,choic:[0,1,2,3,4,6,9,12,13,14,20,26,27,29],choleski:[5,20,27],choos:[2,3,6,9,10,11,13,14,21,29],chosen:[0,1,2,6,8,9,10,13,16,23,26,29],chosen_datapoint:1,christian:25,christoph:[22,25,26],cifar10:3,cifar:3,circ:[1,12],circl:[0,8,12,27],circuit:3,circumfer:9,circumv:[1,5,13,27],cite:21,ckpt:4,clariti:23,class_nam:[3,9],class_val:9,class_valu:9,classic:[7,9,13],classif:[0,3,5,6,7,8,11,12,19,21,25,26,27,28,29],classifi:[0,1,4,7,9,10,11,26],classificaton:1,classifii:10,clean:1,clear:[1,5,10,12,13],clearli:[0,3,5,6,7,8,18,23,27,28,29],clever:[1,10],clf3:0,clf:[0,6,8,9,10,26,27],clf_lasso:6,clf_ridg:[6,26],clip:[3,23],clone:24,close:[0,1,2,4,6,8,9,11,12,13,14,21,23,25,26,29],closer:[3,5,13,27],closest:[8,11,13,14],closur:[19,26],cloud:[19,26],cluster:[0,1,4,6,11,19,26,29],cluster_label:14,cm:[1,2,3,6,8,9,13,21],cmap:[0,1,2,3,4,6,8,9,10,21,26],cmap_arg:6,cmb:22,cmd:9,cn_:23,cnn:12,cnn_kera:3,cntk:[19,26],co:[0,2,3,6,9,13,26,29],code:[3,4,6,7,8,15,16,17,19,20,21,23,25],coef0:8,coef:[0,26],coef_:[0,5,6,8,9,13,26,27,28,29],coeff:5,coeffici:[0,3,5,6,7,8,9,13,15,16,20,26,27,28,29],coerc:[0,6,26,29],coin:[10,23],coin_toss:10,col:[0,11,26,27],colab:[19,26],cold:9,colinear:0,collabor:21,collaps:8,collect:[0,2,6,10,11,19,23,25,26,29],collinear:[5,27,28],color:[0,3,4,6,8,9,10,21,23],color_channel:3,color_cod:6,colorbar:[1,6,21],colsample_bytre:10,colsaobject:10,column:[0,1,2,5,6,7,8,9,11,12,17,20,26,27,28,29],columntransform:9,com:[4,6,16,19,21,25,26],combin:[1,2,5,6,7,10,23,28,29],come:[0,1,3,4,5,12,13,14,15,26,27,28,29],command:[0,1,26],comment:[0,4,5,6,15,16,21,26,27],commerci:[0,15,19,21,26],commod:[0,26],common:[0,1,3,5,6,7,9,11,13,14,21,23,26,27,29],commonli:[0,1,4,6,7,9,13,14,27,29],commun:[0,12,21],commut:3,commutatitav:3,compact:[0,1,3,5,6,7,9,11,12,13,14,26,27,28,29],compair:0,compar:[0,3,4,5,6,11,13,15,16,20,21,26,27,28,29],comparison:[2,4,13],compat:7,compet:0,competit:10,compil:[0,1,3,4,13,15,19,20,26],complet:[0,2,3,4,9,12,26],completenn:12,complex:[1,5,8,9,11,12,13,15,16,26,29],complic:[0,1,9,13,26,29],compoment:27,compon:[0,1,3,4,5,6,7,9,14,16,19,26,27,28,29],components_:11,compos:[9,12,13,14,19,26],compphys:[0,6,16,17,19,21,22,24,25,26],compress:[0,26,27],compris:6,compromis:[5,27],compulsori:[19,26],comput:[0,1,2,3,4,5,6,7,8,10,11,12,13,15,16,17,19,20,21,22,23,25,26,27,28,29],computation:[0,3,6,9,13,23,26],computationalscienceuio:26,computerlab:21,con:[],concaten:[2,4,6,14],concav:[1,9,13,27],concentr:[0,10],concept:[0,2,19,26,27],conceptu:[12,13],concern:[0,1,4,7,26],concic:26,conclud:[0,5,13,28,29],conclus:1,cond:2,conda:[0,1,15,19,21,26],condis:27,condit:[0,2,4,5,6,8,9,11,13,23,26,27],conduct:19,condwav:2,confid:[0,5,6,7,8,18,21,26,27,28],configur:3,confirm:[5,12,28,29],confus:[5,6,7,10,20,27,28,29],confusion_matrix:9,congruenti:23,conjug:[4,8],conjugaci:13,conjunct:3,connect:[0,1,3,4,9,11,12,13,20,26,27],consequ:[5,6,8,10,12,13,27,28,29],conserv:[5,14,27],consid:[0,1,2,3,5,6,7,8,9,10,12,13,16,20,21,23,26,27,28,29],consider:[0,1,5,13,26,27,28,29],consist:[0,1,2,3,4,6,12,13,21,23,27,29],constant:[0,2,4,5,6,8,12,13,23,26,27,28,29],constitu:[0,26],constitut:[2,6,29],constrain:[1,3,5,7,11,28],constraint:[5,6,8,13,27,28,29],construct:[0,1,2,3,5,6,7,8,9,10,11,20,23,26,27,28,29],consult:[],contact:[0,26],contain:[0,2,3,4,5,6,7,8,9,11,12,13,17,20,21,23,25,26,27,28,29],contemporari:26,content:[1,19,20,26],context:[6,10,13,21,29],contextlib:[0,7],contigu:20,continu:[0,1,2,3,4,5,6,7,8,9,10,12,13,17,18,20,21,23,26,27,28,29],contour:[9,10,13],contourf:[8,9,10],contrast:[1,4,9,10,12],contribut:[0,3,5,13,23,26,27,28],contributor:[0,21],control:[0,1,3,9,13,15,19,26],conv2d:[3,4],conv2dtranspos:4,conv:[3,4],convei:26,conveni:[0,5,6,12,13,20,21,26,28,29],convent:[12,27],converg:[1,2,4,5,6,8,9,11,13,14,27,28],convergencewarn:[1,6,8,9,11],convert:[0,1,4,5,9,11,13,20,26,27],converttomatrix:4,convex:[4,5,7,27],convinc:13,convolut:[1,4,19,26],cook:21,cool:[4,9],coolwarm:[6,21],coordin:[5,12,14,27,28],coorel:0,copi:[0,1,2,14,27],copyright:[],core:[0,2,7,10,26],corel:26,coronari:7,corr:[0,5,7,11,27],correalt:[11,19],correct:[0,1,2,3,4,5,7,13,20,23,26,27,28,29],correctli:[1,2,6,7,10,21],correl:[0,1,3,5,6,7,10,12,13,19,23,26,28,29],correlation_matrix:[0,5,7,11,27],correspond:[0,3,5,6,8,9,11,12,15,16,17,19,20,21,23,26,27,28,29],cortex:12,cosin:[3,6,29],cost:[0,2,3,5,6,7,8,9,12,13,16,17,21,26],cost_deep_grad:2,cost_funct:2,cost_function_deep:2,cost_function_deep_grad:2,cost_function_grad:2,cost_grad:2,cost_sum:2,costol:13,could:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,17,20,21,23,26,27,28,29],coulomb:[0,26],count:[0,2,9,22,23,24,26],countor:13,coupl:[4,5,6,28,29],cours:[0,1,3,5,11,21,22,24,27,29],courvil:[22,25,26,27],cov:[5,6,11,20,23,26,27,28,29],cov_xi:[5,11,27],cov_xx:[5,11,27],cov_yi:[5,11,27],covari:[0,7,19,20,26,28],covariance_matrix:[5,11,14,27],cover:[0,5,17,19,24,25,27,28],covert:[0,26],covxi:23,covxx:23,covxz:23,covyi:23,covyz:23,covzz:23,cpu:1,craft:3,creat:[1,2,3,4,5,9,10,11,12,19,26,28],create_biases_and_weight:1,create_convolutional_neural_network_kera:3,create_neural_network_kera:1,create_x:[5,11,27],credit:[0,7,24,26],crim:0,crime:0,criteria:[0,4,9,10,14,23,26],criterion:[9,10,13],critic:[6,21,27,28],critiqu:21,cross:[0,1,3,7,9,10,13,19,23,26,27,28],cross_entropi:4,cross_val_scor:[6,29],cross_valid:[7,10],crossvalid:[6,29],crucial:[1,23],cs231:3,cs:[22,24],csr_matrix:[20,26],csv:[0,4,6,7,9,26,29],ctnk:1,cubic:0,cumbersom:[5,28,29],cumsum:[10,11,26],cumul:[7,10,23],cumulative_heads_ratio:10,cup:[5,28,29],current:[1,2,3,4,13,14,25],curs:[0,27],curv:[6,7,10,12,21],curvatur:13,custom:[6,14,21],custom_cmap2:[9,10],custom_cmap:[9,10],cutpoint:9,cv:[6,7,10,29],cvxbook:13,cvxopt:[5,8,27],cycl:[1,12],cyclotron:28,d2_g_t:2,d:[1,2,3,4,5,6,7,8,9,10,11,13,14,20,23,24,26,27,28,29],d_f:13,d_g_t:2,d_net_out:2,da:3,dagger:[5,20,27],dai:[1,9,19],damp:3,daniel:24,darget:9,darkr:23,dat:[0,26],dat_id:[0,6,7,9,26,27,29],data1:14,data2:14,data3:14,data4:14,data:[2,4,5,8,10,12,13,14,17,18,20,25,29],data_id:[0,6,7,9,26,27,29],data_indic:1,data_panda:26,data_path:[0,6,7,9,26,27,29],databas:1,datafil:[0,6,7,9,21,26,27,29],datafram:[0,4,5,7,9,11,26,27],datapoint:[1,5,6,7,11,13,27,29],dataset:[0,4,6,7,8,9,10,11,13,14,15,16,21,26,29],datatyp:4,date:[15,16,17,18,21,26,27,28,29],daughter:10,david:[25,26],davison:29,dbh:1,dbo:1,dc5e85cd93c3:[],dcomposit:20,ddot:2,dead:1,deadlin:[16,17,18,22],deal:[0,1,3,5,6,8,11,13,14,17,18,20,21,23,26,27,28,29],dealt:0,debt:7,debug:[0,5,6,28,29],decad:[0,3],decai:[0,13,23,26],decemb:[22,24,26],decent:10,decid:[0,2,3,5,6,9,21,28,29],decim:[0,26],decis:[0,1,8,11,19,25,26],decision_funct:8,decision_tre:9,decisiontreeclassifi:[9,10],decisiontreeregressor:[0,9,10],decisiontreeregressordecisiontreeregressor:9,declar:[0,4,20,26],decompos:[5,6,20,27],decomposit:[0,6,12,21,26,28],decompost:[5,27],deconvolut:3,decorrel:[10,13],decreas:[1,2,4,5,6,10,11,13,28,29],deduc:[0,26],deep:[3,7,12,13,19,22,25,27],deep_neural_network:2,deep_param:2,deep_tree_clf1:9,deep_tree_clf2:9,deep_tree_clf:[9,10],deepen:[5,19,26],deeper:[0,3,4,26],deeplearningbook:[25,26,28],deer:3,def:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,21,23,26,27,28,29],def_covari:23,default_tim:4,defect:[5,27],defici:[5,27],defin:[0,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,27,28,29],definit:[1,2,5,6,7,8,10,11,12,13,17,20,23,27,28,29],defint:23,defvjp:2,degre:[3,5,6,8,9,10,11,15,16,17,21,23,26,28,29],deisenroth:27,del:1,delet:6,delimit:4,deliv:[22,26],delta:[0,2,3,6,8,12,13,14,26],delta_0:3,delta_1:3,delta_2:3,delta_3:3,delta_4:3,delta_5:3,delta_:[1,20],delta_h:[0,1,26],delta_j:[3,12],delta_k:12,delta_l:[1,3],delta_momentum:13,delta_n:[0,3,26],delug:19,delv:0,demand:13,demonstr:[0,3,5,6,7,11,12,19,26,27,28,29],den:4,denomin:[1,5,28,29],denot:[1,2,6,7,13,23],dens:[1,3,4],densiti:[0,2,6,23,26,29],depart:[24,26,27,28,29],depend:[0,1,2,4,5,6,7,8,11,12,13,15,19,20,21,23,26,27,28,29],depict:23,deploy:[0,15,19,21,26],deprec:2,depth:[0,3,9,10,20,29],deriv:[0,1,2,6,7,8,10,11,13,16,17,19,21,26],derivati:13,derivative_fn:13,descend:[5,9,11,27,28],descent:[0,1,3,7,8,12,26,27],describ:[0,2,4,5,6,8,10,11,12,13,18,20,21,26,28,29],descript:[0,8,9,26],design:[0,1,3,4,5,6,7,10,11,12,13,15,16,17,18,21,26,28,29],designmatrix:[0,26],desir:[0,2,4,5,13,14,26,27],despit:[1,12],destroi:20,det:[5,20,27],detail:[0,6,11,13,14,20,21,27],detect:[3,8,12],determin:[0,2,3,4,5,6,8,9,10,11,12,13,16,20,23,26,27,28,29],determinist:[7,13,23],dev:[1,21],develop:[0,3,5,8,10,11,12,19,20,21,26,27,28,29],deviat:[0,1,2,4,5,6,21,23,26,27,28,29],devic:[3,4,14],devis:12,df1:26,df:[4,8,11,13,26],di:0,diag:[5,8,27,28],diagnost:[1,10],diagon:[0,5,7,13,18,20,21,23,26,27,28],diagonaliz:[5,27],diagram:10,diagsvd:6,dice:[6,23,29],dict:[6,8],dictat:[],dictionari:0,did:[0,1,5,6,7,10,11,14,21,26,28,29],die:1,diff1:2,diff2:2,diff:2,diff_ag:2,diffeent:8,differ:[0,1,2,3,4,5,6,7,9,10,11,12,13,14,15,17,19,20,21,23,25,26,27,28,29],differenti:[0,3,16,19,20,26,27],differential_oper:2,difficult:[0,1,6,10,13,21,23,26,29],difficulti:[0,1,13,26],diffonedim:2,diffus:[],digit:[0,1,3,4,6,21,22,24,26],dilemma:13,dilut:1,dim:[4,11,14,20],dimens:[0,1,2,3,4,5,8,9,11,14,16,17,20,26,27,28],dimension:[0,4,5,6,9,11,13,14,19,20,26,28,29],dimensionless:[0,3,26],diment:20,dimnsion:4,diod:3,direct:[0,1,2,4,7,11,12,13,14,26,27],directli:[1,4,5,6,23,27],directori:[0,7],disadvantag:[0,26],disappear:[3,6,29],disc_loss:4,disc_tap:4,discard:[6,11,29],disciplin:[0,3,12],disclaim:23,discord:26,discourag:13,discov:[0,26],discover:[5,28],discret:[1,3,5,7,13,28,29],discrimin:[4,7,10,11],discriminator_loss:4,discriminator_loss_list:4,discriminator_model:4,discriminator_optim:4,discuss:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,23,25,26,27,28,29],diseas:7,disguis:[6,27,28,29],disord:[1,7],displai:[0,1,3,4,5,6,7,8,9,10,11,12,14,21,23,26,27,29],displaystyl:[0,5,17,26,27,28],disregard:[0,26],dissimilar:[11,14],dist:14,distanc:[0,8,9,11,14,23],distance_list:9,distinct:[3,7,8,9,10,14],distinctli:8,distinguish:[0,4,7,8,23,26],distplot:0,distribut:[0,1,4,6,7,10,11,13,14,15,16,18,19,20,21,26,27],distrubut:[0,15,19,21,26],ditto:[],dive:[0,8,20,26],diverg:[1,13],divid:[0,1,3,5,6,7,8,9,11,12,23,26,27,28,29],divis:[6,8,9,13,20,23,29],dlopen:[3,4,14],dna:7,dnn1:4,dnn2_gru2:4,dnn:[0,1,2,4,12,26],dnn_kera:1,dnn_model:1,dnn_numpi:1,dnn_scikit:[0,1,26],doamin:[28,29],doc:[0,6,16,17,19,21,22,24,25,26],document:[4,9,11,13],doe:[0,1,2,3,4,5,6,8,10,11,12,13,14,16,20,21,23,26,29],doesn:[3,9,12],dog:[1,3,4],domain:[5,8,13,21,28,29],domin:[0,26],don:[0,1,3,5,6,7,8,11,13,15,19,21,26,27],done:[0,2,3,4,5,6,9,10,11,13,17,20,21,26,27,28,29],dot:[0,2,3,5,6,7,8,9,10,11,12,13,20,21,23,26,27,28,29],doubl:[3,4,20,26],doubli:1,doubt:21,down:[0,3,6,9,11,12,13],download:[0,1,3,5,6,20,21,25,26],downsampl:3,dozen:1,dq:[6,29],drag:13,dramat:11,drastic:4,draw:[4,6,10,13,29],drawback:[0,1,3,13,27],drawn:[1,4,6,7,11,23,26,29],drive:[3,4],driven:3,drop:[0,1,5,6,11,13,23,26,27,28,29],dropna:[0,6,26,29],dropout:4,dt:[2,3,13,23],dtype:[0,1,2,3,4,14,20,26,27],dual:8,dualiti:6,dub:[0,26],due:[1,2,5,6,8,10,12,13,24,26,27,29],dummi:0,dure:[0,1,3,4,8,9,11,17,19,21,26,29],dwell:0,dwh:1,dwo:1,dx:[2,3,8,23],dx_1:23,dx_1p:[6,29],dx_2p:[6,29],dx_mp:[6,29],dx_n:23,dxp:[6,29],dy:[1,8,23],dylib:[3,4,14],dynam:4,dz:8,e:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,18,23,24,26,27,28,29],e_:[0,2,26],each:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,19,20,22,23,24,26,27,28,29],eager:29,eapprox:[0,26],earli:[1,13],earlier:[0,5,7,8,9,11,12,13,26,27],earthexplor:[6,21],eas:[6,9,14,29],easi:[0,5,6,7,8,9,10,11,12,13,18,19,20,26,27,28,29],easier:[5,6,8,9,13,23,26,27,29],easiest:13,easili:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,20,21,26,27,28,29],eastern:[24,26],ebind:[0,26],eblock:9,econometr:26,economi:5,ecosystem:[19,26],ect:22,edg:3,edgecolor:[6,29],edu:13,educ:[0,21,26],eff:23,effect:[1,4,10,13,23],effic:1,effici:[0,3,10,13,19,20,23,26],efron:[6,29],egrad:13,eig:[5,11,13,20,23,26,27],eigen:23,eigenpair:[5,11,27],eigenvalu:[0,5,8,11,13,20,26,27,28],eigenvector:[5,11,13,17,27,28],eight:[20,26],eigval:[20,23,26],eigvalu:[11,13],eigvec:[20,23,26],eigvector:[11,13],eispack:[20,26],either:[1,5,6,7,8,9,10,11,13,15,16,21,23,26,27,28,29],elabor:23,elarn:3,electr:[0,3,12,26],electron:26,eleg:11,element:[1,2,3,4,5,6,7,8,11,12,13,17,18,19,20,21,22,25,27,28,29],elementari:[10,13,20],elementwis:[3,13],elementwise_grad:[2,13],elessar:26,elif:[2,14],elim:20,elimin:[3,8],els:[1,2,3,4,7,9,12,13,14,20],elu:1,elus:[0,26],email:[22,24,26],embed:[0,11,27],embodi:[6,21,29],emit:23,emner:25,emphas:[0,10,19,26],emphasi:[0,19,25,26],empir:[1,11,23],emploi:[0,1,5,6,11,13,21,23,26,27,28,29],employ:0,empti:[6,10,29],emul:12,en:[19,25],enabl:11,enbodi:[6,29],encod:[0,3,5,7,9,11,14,17,26,27,28],encompass:[0,21,23],encount:[0,1,5,6,7,13,23,26,27,29],encourag:21,end:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,20,21,23,24,26,27,28,29],end_box:2,end_nod:2,end_valu:2,endpoint:[3,6],energi:[0,4,6,27,29],enet_coordinate_desc:6,enforc:12,eng:25,engin:[0,1,3,4,19,26],english:21,enocurag:21,enoent:[3,4,14],enorm:3,enough:[0,6,13,26,29],ensembl:[1,9],ensur:[0,1,2,3,5,6,11,13,23,27,28,29],entail:26,enter:[5,6,17,27,28],enthought:[0,15,19,21,26],entir:[1,3,7,9,19,23,26],entiti:[9,12,20,26],entri:[0,5,8,11,12,20,26,27,28,29],entropi:[1,3,7,10,13,26],enumer:[0,1,2,3,4,6,8,26,27,28],env:[0,1,2,3,4,6,7,8,9,11,14,23,26,28],environ:[2,9,19,21],eo:[0,6,26,29],eol:[0,26],eosfit:[0,26],epoch:[0,1,3,4,12,13,26],epsilon:[0,5,6,7,13,21,26,27,28,29],epsilon_0:[0,26],epsilon_1:[0,26],epsilon_2:[0,26],epsilon_:[0,26],epsilon_i:[0,26,27],eq:[3,13,14,20,23],eqnarrai:[3,5,6,28,29],equal:[0,1,2,3,4,5,6,8,9,11,12,13,14,15,20,21,23,26,27,28,29],equat:[1,3,4,5,6,7,8,9,10,11,13,14,16,17,18,20,21,23,29],equilibrium:[2,12],equiv:[3,13,20,23],equival:[0,1,5,7,8,11,13,19,20,26,27,28,29],erf:23,eriador:26,eridg:26,err:[0,7,10],err_:[6,29],err_sqr:2,errat:13,errno:[0,3,4,7,14],erron:2,error:[1,2,4,5,6,7,9,11,12,13,15,16,17,18,19,20,21,23,28],error_estimate_corr_tim:23,error_hidden:1,error_output:1,escap:13,especi:[1,3,9,12,13,21],essenti:[0,5,6,9,10,12,14,21,23,27],establish:[0,6,10,11,15,21,26],estim:[0,1,5,6,7,10,11,13,19,23,26,27],estimated_mse_fold:[6,29],estimated_mse_kfold:[6,29],estimated_mse_sklearn:[6,29],et:[0,2,4,15,16,17,21,25,26,27,28,29],eta0:[8,13],eta:[0,1,3,8,12,13,26],eta_:13,eta_t:13,eta_v:[0,1,3,26],etc:[0,1,3,5,7,8,9,11,12,13,14,15,19,20,21,23,27],ethic:19,etsim:29,euclidean:[0,14,27],euler:[],evalu:[0,2,3,4,5,6,9,13,21,23,26,27,28,29],evaluationform:21,evaluationgrad:21,evalut:13,even:[0,1,3,4,5,6,8,9,10,11,12,13,14,19,20,23,26,27,28,29],evenli:4,event:[5,7,10,23,28,29],eventu:[0,5,6,11,12,13,21,24,27,28,29],everi:[0,1,2,3,4,5,6,9,10,11,12,13,14,19,23,24,26,27,28,29],everyth:[4,12],everywher:[4,13],evolv:0,exact:[0,5,11,12,13,20,23,26,27],exactli:[0,3,4,6,12,19,21,27,29],exam:26,examin:[6,29],exampl:[5,11,12,13,15,16,19,20,21,23,25],exce:[1,12,13],excel:[0,1,4,5,10,26,27],except:[0,3,4,6,7,8,9,20],excess:[0,26],excit:0,exclud:[1,6,12,21,27,28,29],exclus:[0,1,3,6,23,26,29],execut:[2,5,13,27],exemplifi:13,exercic:[24,26],exercis:[5,19,21,22,24,28],exercisesweek35:17,exhaust:[6,29],exhibit:[0,5,6,8,26,27,29],exisit:[],exist:[0,1,2,3,4,5,6,7,8,9,13,14,18,20,21,26,27,28,29],exit:[5,20,27],exp:[0,1,2,5,6,7,8,10,11,12,13,15,16,21,23,26,27,28,29],exp_term:1,expand:[5,7,11,13,27],expandus:[0,7],expans:[0,3,5,8,10,12,13,26,27],expect:[0,1,5,6,7,11,12,13,15,16,19,21,26,27],expectation_value_of_h_wrt_p:23,expens:[6,10,13],experi:[0,1,6,8,13,19,21,26,27,29],experiment:[0,4,6,9,23,26,29],expert:[1,9],explain:[0,6,9,10,11,13,21,26],explained_variance_ratio_:11,explan:[],explanatori:[0,26],explicit:[0,3,6,13,20,21,26,27,28],explicitli:[0,4,8],explod:1,exploit:[0,3,12,13,26],explor:[1,4,6,8,13,19,21],expon:1,exponenti:[0,1,5,6,10,13,21,23,26,28,29],export_graphviz:9,export_text:9,exporttext:9,expos:19,express:[0,2,3,5,6,7,10,12,13,16,20,21,23,26,28,29],exptmean:23,exptvari:23,extend:[0,2,7,11,13,17,19,26],extens:[0,12,15,19,26],extent:[0,1,6,25,29],extern:[3,6,9],extra:[1,3,5,24,26,27],extract:[0,3,5,6,7,8,11,13,20,26,27],extrapol:[0,26],extrem:[0,1,4,5,6,7,8,9,13,16,20,27,28,29],extremum:13,extrins:11,ey:[0,5,6,13,14,20,26,28,29],f11:[0,26],f12:[0,26],f13:[0,26],f1:13,f1_grad:13,f1d:13,f2:13,f2_grad_x1:13,f2_grad_x1_analyt:13,f2_grad_x2:13,f2_grad_x2_analyt:13,f3:13,f3_grad:13,f3_grad_analyt:13,f4:13,f4_grad:13,f4_grad_analyt:13,f5:13,f5_grad:13,f6:13,f6_for:13,f6_for_grad:13,f6_grad_analyt:13,f6_while:13,f6_while_grad:13,f7:13,f7_grad:13,f7_grad_analyt:13,f8:13,f8_grad:13,f9:[0,13,26],f9_altern:13,f9_alternative_grad:13,f9_grad:13,f:[0,1,2,3,4,5,6,7,8,10,12,13,14,16,18,20,23,24,26,27,28,29],f_0:[3,10],f_1:[10,13],f_2:[12,13],f_3:12,f_:10,f_d:23,f_grad:13,f_grad_analyt:13,f_i:[0,6,12,16,29],f_m:[3,10],f_n:3,f_vec:2,f_wrap:2,face:13,facecolor:[6,8,23,29],facil:[0,15,19,26,29],facilit:12,fact:[0,1,3,5,9,11,12,13,26,27,28,29],factor:[0,1,3,5,6,9,10,11,13,20,23,26,27,28,29],factori:13,fade:6,fafab0:[9,10],fahimeh:[24,26],fail:[0,6,8,9,11,13,24,26,29],fail_on_error:[0,7],failur:7,fairli:[1,2,23],faisal:[16,27],fake:4,fake_loss:4,fake_output:4,fale:[],fall:[8,9,22],fals:[0,1,2,3,4,5,6,7,9,10,14,20,21,26,27,28,29],famili:[0,7,8,23],familiar:[0,3,5,6,8,15,19,20,21,23,26,28,29],famou:[6,12],far:[0,3,4,5,6,8,11,12,13,14,26,27],fashion:[0,9,10,26],fast:[1,3,6,10,12,13,19,23,26,29],faster:[1,11,13],fastest:[13,20],favor:7,favorit:23,fc:3,fd:[0,7],featur:[0,1,3,5,6,7,8,10,11,12,13,18,19,21,23,26,28,29],feature_nam:[0,1,7,9],feautur:9,fed:1,feed:[0,2,3,11,19,26],feed_forward:1,feed_forward_out:1,feed_forward_train:1,feedback:[4,26],feeddorward:4,feedforward:[1,4,12],feel:[0,5,6,11,13,15,16,19,21,24,26,28],feet:0,felt:21,fetch:[6,21],few:[1,3,4,5,9,23,26],fewer:[0,9,11,26],ffnn:[1,12],field:[0,3,6,12,19],fifth:[0,6,15,16,21,26],fig:[0,1,2,3,4,6,7,12,13,14,21,26],fig_id:[0,6,7,9,26,27,29],figaxi:23,figsiz:[0,1,2,3,4,6,7,8,9,10,26,29],figur:[0,1,2,3,4,5,6,7,8,9,10,12,13,14,15,16,19,21,26,27,28,29],figure_id:[0,6,7,9,26,27,29],figurefil:[0,6,7,9,26,27,29],file:[0,2,3,4,5,6,7,9,14,21,26,27,28,29],file_prefix:4,filenam:26,filenotfounderror:[0,7],fill:[5,9,27],filter:[0,3,4,7],financ:0,find:[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15,16,19,21,23,26,27,28],fine:[0,14,15,26],finish:2,finit:[3,5,6,12,13,17,23,27,28,29],first:[0,1,2,3,5,6,7,8,9,10,11,13,14,15,16,17,20,21,23,24,25,29],firsteigvector:11,fit:[1,3,4,5,6,7,8,9,11,12,13,15,16,17,21,23,27,29],fit_beta:[6,27,28],fit_intercept:[0,5,6,27,28,29],fit_mod:9,fit_transform:[0,6,8,9,11,29],fiti:[0,26],five:[0,9,17,21,26,27],fix:[0,3,4,6,10,11,12,13,21,26,29],fixedloc:6,flag:4,flat:[3,4,12,13,14],flatten:[1,3,4,5,20],flexibl:[1,6,8,10,12,26,29],flip:[24,26],float32:[4,9],float64:[4,20,26,27],flop:[5,20,27],flow:[1,4,12],fluctuat:[5,28],fluid:[],fly:11,fm:[0,26],fmax:3,fmesh:13,fn:7,fname:[0,7],focu:[0,3,4,5,6,15,19,25,26,27,28,29],focus:[1,6,7,20,27,28],fold:[6,9,21],folder:[0,1,3,4,6,14,21,26,29],follow:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29],font:[0,7,23,26],fontdict:23,fontsiz:[1,6,8,9,10,23],fontweight:1,footprint:3,foral:[8,27],forc:[0,5,6,10,11,26,27,28],forcast:4,forecast:[4,12],forest:[0,1,9,19,26],forget:11,form:[0,3,4,5,6,7,8,9,11,12,13,17,19,20,21,23,26,27,28,29],formal:[3,4,14,23],format:[0,1,2,3,4,6,7,8,9,10,11,19,23,25,27,28,29],format_data:4,formatstrformatt:[6,13,21],formul:[4,6,11,14],formula:[3,13,23],forth:[4,12],fortran2003:[19,26],fortran2008:21,fortran90:23,fortran:[0,15,19,20,26],fortun:[0,11,27],forward:[0,3,6,19,20,26,29],foster:26,found:[1,2,3,4,5,6,12,13,14,21,26,27,28,29],foundat:[19,26],four:[4,5,6,8,12,20,21,22,24,26,28],fourier:[0,26],fourierdef1:3,fourierdef2:3,fourierseriessign:3,fourth:[12,26,27],fp:7,frac:[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15,16,17,20,21,23,26,27,28,29],fractal:9,fraction:9,frame:7,framework:[1,3,4,8,10,14,23],frank:[5,11],frankefunct:[5,6,11,21,27],fredli:[24,26],free:[0,6,11,13,15,16,19,20,21,23,24,25,26],freecodecamp:19,freedom:[5,28],freeli:[0,21],frequenc:[3,6,7,23,29],frequent:[0,8,9,13],frequentist:19,fresh:10,fridai:[16,17,18,24,26],friedman:[6,18,21,22,25,26],friendli:4,frodo:26,frog:3,from:[0,1,2,3,4,6,7,8,9,11,13,14,16,17,18,19,20,21,23,24,25,26],from_cod:9,from_logit:[3,4],from_tensor_slic:4,fromnumer:2,front:[0,4,5,15,16,26,27,28],fulfil:[2,5,12,27],full:[0,1,3,5,7,9,10,13,23,26,27],full_matric:[5,27],fulli:[3,6,12,22,23,29],fun:[2,19,26],fun_nam:2,func:2,functionali:11,fundament:[0,6,19,26,29],funtion:2,further:[2,7,9,26],furthermor:[0,3,5,6,7,11,12,13,17,19,21,26,27,28,29],futur:[0,4,8,9,26],futurewarn:8,fy:[15,21,22,24,25,26],fys3150:[],fys4155:21,fys5419:[25,26],fys5429:[25,26],g0:2,g:[0,1,2,3,4,6,8,9,10,11,13,23,26,28,29],g_0:2,g_1:[2,10],g_2:[2,10],g_:[2,9,10],g_analyt:2,g_dnn_ag:2,g_euler:2,g_i:2,g_m:[3,10],g_n:3,g_re:2,g_t:2,g_t_d2t:2,g_t_d2x:2,g_t_dt:2,g_t_hessian:2,g_t_hessian_func:2,g_t_jacobian:2,g_t_jacobian_func:2,g_trial:2,g_trial_deep:2,g_vec:2,gain:[1,5,7,9,10,13,27,28,29],galleri:[0,26],game:4,gamge:26,gamma1:8,gamma2:8,gamma:[0,2,8,9,10,11,13,26],gamma_0:10,gamma_1:10,gamma_1x:10,gamma_:[0,26],gamma_i:[0,8,23,26],gamma_j:13,gamma_k:13,gamma_m:10,gamma_x:[0,26],gap:[6,8],gate:[4,12],gather:[0,1,12,27],gaug:12,gaussbacksub:20,gaussian:[4,5,6,8,14,23,28,29],gaussian_point:14,gaussian_rbf:8,gave:13,gavra:26,gbc:[22,26],gca:[2,6,8,13,21],gd:1,gd_clf:10,gdclassiffiercgain:10,gdclassiffierconfus:10,gdclassiffierroc:10,gdm:13,gdregress:10,ge:[1,5,7,23,27],gen:[0,7],gen_loss:4,gen_tap:4,gender:[0,26],genener:4,gener:[0,1,2,3,5,6,8,10,11,12,13,14,15,16,17,20,21,23,25,27,28,29],generallay:12,generate_and_save_imag:4,generate_imag:4,generate_latent_point:4,generate_simple_clustering_dataset:14,generated_imag:4,generator_loss:4,generator_loss_list:4,generator_model:4,generator_optim:4,genom:19,geodes:11,geometr:[0,13,26],geometri:[5,28,29],georg:25,geotif:[6,21],geq:[2,5,8,9,13,27,28],geron:[0,15,22,25,26],get:[0,1,2,3,4,5,6,7,9,10,11,13,15,19,20,21,23,24,26,27,28,29],get_dummi:9,get_paramet:2,get_split:9,get_yaxi:8,get_yticklabel:6,getattr:2,gg:26,gibb:[19,26],gif:4,gini:10,gini_index:9,ginvers:13,git:[0,15,19,26],giter:13,github:[0,6,9,15,16,17,19,21,22,24,25,26,27],gitlab:[0,15,19,21,26],give:[0,1,2,3,5,6,7,8,9,10,12,13,14,17,19,21,23,26,27,28,29],given:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,26,27,28,29],global:[6,7,13,21],gloriou:[],glorot:1,gnew:13,go:[0,1,3,5,6,8,9,11,12,13,21,26,27,28,29],goal:[0,7,9,26],goe:[0,1,2,5,6,13,14,18,20,26,27,28,29],golden:13,gone:[5,27,28],gong:1,good:[1,3,4,5,6,9,10,11,13,17,19,23,25,27,28,29],goodfellow:[4,22,25,26,27,28],googl:[1,4,19,26],got:[1,6,13,21],gotten:26,gov:[6,21],gp:25,gpu:[1,13,19,26],grad:[2,13],grad_analyt:13,grade:[21,22],gradient:[0,3,4,7,8,9,12,19,26,27],gradientboostingclassifi:10,gradientboostingregressor:10,gradients_of_discrimin:4,gradients_of_gener:4,gradienttap:4,gradual:[1,14],grai:[4,6,21],graph:[1,9,11,12,13],graph_from_dot_data:9,graphic:[0,1,9,26],grasp:0,gray_r:[1,3],grayscal:3,great:[5,13],greater:[1,7,23],greatli:13,greedi:9,green:[0,3,9,23],grei:4,grid:[1,3,6,7,8,12,23,27,28,29],grossli:13,ground:[0,26],group:[0,6,7,9,14,19,21,22,24,26,29],groupbi:[0,26],grow:[1,3,9,10],growth:[0,26],gru:4,guarante:[0,4,13,23,26,27],guess:[1,4,10,13,14],guestrin:10,guid:1,guidanc:21,h1:2,h:[0,1,5,6,8,13,16,18,21,23,24,25,26,27],h_1:[2,13],h_2:[2,13],h_:[0,13,26],h_m:10,ha:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,20,21,23,26,27,28,29],haa:24,habit:[0,27],had:[0,1,6,7,13,26,29],hadamard:[1,12,13],half:[1,8,9],halv:10,hand:[0,1,2,3,5,11,12,13,15,19,20,21,22,23,24,25,26,27,28,29],handi:[3,21],handl:[0,1,2,5,9,11,19,28],handle_unknown:9,handsid:12,handwrit:12,handwritten:[1,5],happen:[1,2,3,4,5,6,10,13,23,27],hard:[1,7,8,10,13],hardcopi:[19,26],harder:[0,1,27],harmon:3,hasn:1,hassl:[0,15,19,26],hast:[19,26],hasti:[0,6,15,16,17,18,21,22,25,26,27,29],hat:[0,1,5,6,7,9,10,11,12,13,17,18,20,21,27,28,29],have:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,23,24,26,27,28,29],haven:1,he:7,head:[0,4,10,23],header:[0,26],heads_proba:10,health:[0,27],hear:[0,13,26],heart:[0,7,26],heatmap:[0,1,3,7,26],heavili:0,heavisid:1,height:[1,3,6,27,28],held:13,help:[0,1,4,12,13,16,21,26],helper:[4,14],henc:[0,5,6,8,9,10,12,13,18,21,26,27,28,29],henrik:[24,26],her:7,here:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,23,26,27,28,29],hereaft:[0,8,12,26],hermitian:20,hessenberg:20,hessian:[0,2,5,13,16,28],heterogen:[9,10],hi:7,hidden:[1,3,4,12],hidden_bia:1,hidden_bias_gradi:1,hidden_layer_s:[0,1,26],hidden_neuron:4,hidden_weight:1,hidden_weights_gradi:1,hierarch:[5,27,28],high:[0,1,2,3,4,5,6,9,10,11,13,14,19,20,21,26,27,29],higher:[0,1,3,5,6,8,13,21,26,27,28,29],highest:[1,2],highli:[0,3,4,10,18,19,20,25,26,27,28,29],highwai:0,hing:8,hint:[13,16,27,28],hip:19,hire:0,hist:[4,6,7,23,29],histogram:[0,6,7,23],histor:[7,11],histori:[3,4,12],hitherto:[5,28],hjorth:[24,26,27,28,29],hobbi:23,hoc:[5,27],hoff:25,hold:[1,3,6,13,14,29],holder:[0,26],holomorphic_grad:2,home:0,homepag:[21,26],homework:[6,13],homogen:[1,3,9,10,13],honchar:2,hopefulli:[0,11,23,26],horizont:11,hors:[3,7],hot:[1,9],hour:[1,19,22,23,24,26,29],how:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,19,20,21,23,26,27,28,29],howev:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,19,20,21,23,26,27,28,29],hspace:[0,4,8,10,23,26],hstack:1,htf:[22,26],html:[0,9,11,17,19,21,22,24,25,26,28],http:[0,3,4,6,9,11,13,16,17,19,20,21,22,24,25,26,27,28],huang:[0,26],huber:[0,26],huge:[1,3,4,19],human:[0,1,3,6,9,12,27,28],humid:9,hundr:1,hungri:1,huse:26,hybrid:22,hydrogen:[0,26],hyper:[],hyperbol:[1,4,12],hyperparam:8,hyperparamet:[3,4,5,6,9,13,17,26,27,28,29],hyperplan:11,i0:[0,26],i1:[0,6,8,12,26,27,28,29],i2:[0,8,12,26],i3:[0,12,26],i5:[0,26],i:[0,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,24,26,27,28,29],i_1:[5,6,28,29],i_2:[5,6,28,29],i_:13,ian:25,ic:1,id:[7,13],ida:[24,26],idea:[0,1,2,3,4,6,9,10,12,13,20,21,27,28,29],ideal:[0,2,6,8,13,23,26,29],idem:[6,29],ident:[5,6,12,13,17,20,21,27],identical:[28,29],identifi:[0,1,7,9,11,12,13,14,26,27],ieor:23,ifi:25,ifs:[19,26],ignor:[0,1,3,9,27],ii:[20,23],iii:[20,26],ij:[0,1,3,6,8,12,14,16,18,20,21,23,26,27,28,29],ik:[0,20,26,27],illustr:[5,7,10,12,13,14,19,26,29],im:6,imag:[1,3,4,6,9,11,12,14,25,26],image_at_epoch_:4,image_batch:4,image_height:3,image_path:[0,6,7,9,26,27,29],image_width:3,imageio:[6,21],images_from_seed_imag:4,imagin:1,immedi:[0,3,4,6,15,19,26],implement:[0,2,3,4,5,6,8,9,10,11,12,13,14,17,21,23,26,27,28,29],impli:[3,5,6,7,13,20,27,28,29],implicit:3,implicitli:[11,23],importantli:3,impos:[0,6,11,12,26],imposs:[0,5,26,27],impress:[0,12,26],improv:[0,4,5,9,10,11,13,21,27],impur:9,imread:[6,21],imshow:[1,3,4,6,21],in3050:[25,26],in3310:26,in4080:[25,26],in4300:[25,26],in4310:25,in5400:3,in5550:25,in_out_neuron:4,inaccur:13,inact:12,inadequ:[0,26],inch:[6,27,28],includ:[0,1,2,3,4,5,6,7,11,12,15,16,17,19,21,23,24,25,26,27,28,29],include_bia:[6,9,29],inclus:17,incom:12,incorrect:1,incoveni:8,increas:[0,1,3,4,5,6,8,9,11,12,13,21,23,26,27,28,29],increasingli:23,ind:6,inde:[0,2,4,5,6,13,26,27],indefinit:4,independ:[0,5,6,7,8,12,13,23,26,27],index:[0,1,3,4,10,14,19,20,23,25,26],index_col:[0,26],indic:[0,1,3,4,5,6,9,10,11,13,16,21,26,27],indispens:[6,29],individu:[1,6,7,10,12,23,26,27,29],indu:0,indx1:2,indx2:2,indx3:2,indx:20,ineffici:[3,13],inequ:[8,13],inertia:13,inf1000:[19,26],inf1100:[19,26],inf1100l:[19,26],inf1110:[19,26],inf3000:26,infeas:9,infer:[0,1,4,6,25,26,29],inferenc:1,infil:[0,6,7,9,26,29],infin:[5,6,7,11,18,27,28,29],infinit:3,infinitesim:23,influenc:[6,10,29],influenti:1,info:26,inform:[0,1,3,4,6,9,11,12,13,14,20,21,25,26,29],infti:[3,6,13,23,29],ingeni:13,ingrad:2,ingredi:[0,9,26],inher:[6,29],inherit:[20,26],initi:[0,1,2,6,10,13,14,20,23,26,29],inject:14,inlin:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,21,23,26,27,28,29],inner:[0,13,27],inp:4,inplac:13,input:[0,1,3,4,5,6,7,8,12,13,14,21,23,26,27,28,29],input_dim:1,input_shap:[3,4],inputs:1,inputs_shuffl:[0,1,27],insert:[3,5,6,8,10,23,27,28,29],insid:[0,4,7],insight:[0,1,5,19,26,27,28,29],insist:[6,13,27,28],inspir:[0,1,12,26],instabl:2,instal:[0,1,5,6,9,15],instanc:[0,1,2,4,6,9,11,13,26,27,29],instanti:10,instead:[0,1,2,3,4,5,6,8,9,11,13,14,20,21,23,26,27,29],institut:1,instruct:[0,1,15,16,26],int32:10,int_0:23,int_:[3,6,23,29],int_a:23,intak:[0,27],integ:[1,2,13,14,20,23,26],integer_vector:1,integr:[3,6,23,26,29],intellig:[0,14,25,26],intend:10,intens:1,intention:14,interact:[0,6,9,12,19,21,26],intercept:[0,6,8,11,13,16,21,26,28,29],intercept_:[0,6,8,9,13,26,27,28,29],interceptol:29,interceptridg:29,interchang:[5,12,20,28,29],interconnect:1,interest:[0,1,2,3,4,5,6,7,8,9,12,19,21,23,26,27,28,29],interfac:[0,1,20,27],interior:[0,9,26],intermedi:[20,27],intern:[1,10,12],interpol:[1,3,4,6,12,21],interpr:[5,27,28],interpret:[0,1,6,9,10,12,13,17,20,21,23],interv:[0,3,5,6,7,13,18,21,23,26,27,28],intial:13,intract:[0,4,27],intrins:[3,11,20,23,26],intro:[19,25,26],introduc:[0,1,5,6,8,10,12,17,20,21,23,26,29],introduct:[1,2,4,13,17,25,27],introductori:[0,4,20,25,26,27],intuit:[0,5,6,8,12,13,21,26,28,29],inv:[0,5,13,26,27,28],invalid:[1,8],invalu:[0,13,15,19,26],invari:1,invd:[5,28],inver:8,invers:[0,3,6,13,15,16,17,21,26,27],inverse_transform:8,invert:[0,5,7,10,13,26,28],invh:13,invok:[0,8],involv:[0,2,6,7,11,12,27,29],io:[0,17,19,21,22,24,25,26,27],ip:[0,8,23,26],ipca:11,ipykernel_48918:1,ipykernel_49101:6,ipykernel_49291:26,ipykernel_85796:29,ipynb:[19,21,26],ipython:[0,5,7,9,11,14,15,19,21,26,27],iq:[6,29],iri:[8,9],irreduc:[6,29],irrelev:[5,27],irrespect:[0,26],irvin:[],isbox:2,iscomplexobj:2,isn:[5,28,29],isnul:0,isomap:11,isotop:[26,29],issu:[1,9,20,28],it_arrai:13,item:[0,13,26],items:[20,26],iter:[1,2,4,6,8,9,11,13,14,23,29],its:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,20,21,23,26,28,29],itself:[5,6,12,21,23,27,28,29],j1:20,j:[0,1,2,3,4,5,6,8,9,11,12,13,14,16,17,18,20,21,23,25,26,27,28,29],j_:6,j_lasso_sk:6,j_ridge_sk:6,j_sk:6,jackknif:[6,19,26,29],jacobian:[2,13],jacobian_shap:2,jakobsen:24,jason:4,jax:[19,26],jensen:[24,26,27,28,29],jerom:[18,21,25],ji:[12,20],jit:13,jj:[0,5,6,26,28,29],jk:[0,1,6,12,20,26],jl:[0,26],jm:20,jnp:13,job:[2,8,10],johansen:26,join:[0,3,4,6,7,9,14,26,27,29],joint:[4,5,28,29],judg:13,judgement:[6,21],julia:[19,20,21],jump:[21,23],junk:4,jupit:26,jupyt:[0,9,15,19,21,25,26,29],just:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,19,21,23,26,27,28,29],justif:0,justifi:[3,10],k0:7,k1:7,k:[0,1,3,5,6,7,8,9,10,11,12,13,14,19,20,21,23,24,26,27,28],kaggl:[6,21],kappa_d:23,karl:[24,26],karush:8,katrin:[24,26],keep:[0,1,4,5,6,11,13,14,20,21,26,27,28,29],keepdim:[1,2,6,10,20,29],kei:[0,1,3,6,12,27],kept:[4,6,14,29],kera:[0,4,19,21,26],kernel:[0,1,3,4,14,19,26,27],kernel_librari:[3,4,14],kernel_regular:[1,3],kernel_s:4,kernelpca:11,kev:[0,26],kevin:[25,26],keyboardinterrupt:2,keyword:[6,13,20,21,26],kfold:[6,29],kg:1,ki:20,kick:[1,13],kiener:2,kilomet:[6,27,28],kind:[0,2,3,4,8,12,13,14,26,27],kj:[6,12,20,27,28,29],kjm:[19,26],kkt:8,kl:23,km:[12,26],kmean:14,kmeanspoint:14,kn_k:14,know:[0,1,2,5,6,8,13,19,26,27,28],knowledg:[0,19,26],known:[1,3,4,5,6,7,8,9,12,20,21,23,25,27,28,29],kondev:[0,26],kp:23,kpca:11,kroneck:14,kuhn:8,kvalsund:[24,26],kwarg:2,kwown:[0,26],l0:7,l1:[0,1,3,7,26],l1_l2:[1,3],l1regl:5,l2:[1,3],l:[0,1,2,3,5,6,7,8,10,11,12,13,20,21,23,26,27,29],l_1:7,l_2:[7,13],l_:20,l_j:12,la:13,la_i:12,la_k:12,lab:[19,21,26],label:[0,1,2,3,4,5,6,7,8,9,10,12,13,14,19,20,21,23,26,27,28,29],labelencod:[7,10],labels:[6,8,9],labels_shuffl:[0,1,27],laboratori:[22,28],lack:[0,26],lagari:2,lagrang:[8,11],lambda:[0,1,2,3,5,6,7,8,10,12,13,17,18,21,23,26,27,28,29],lambda_0:11,lambda_1:[5,8,11,27],lambda_2:[8,11],lambda_:11,lambda_i:[8,11],lambda_iy_i:8,lambda_jy_iy_j:8,lambda_k:8,lambda_n:[5,8,27],lamda:1,land:[0,8],landmark:8,landscap:13,langl:[0,6,11,23,26,27],languag:[0,1,4,8,15,19,20,21,25,26],lapack:[20,26],laplac:[5,28,29],laptop:19,larg:[0,1,2,4,5,6,8,9,10,11,13,15,19,20,21,23,25,26,27,29],larger:[0,3,5,6,8,10,11,13,17,23,26,27,28,29],largest:[4,8,11],lasso:[0,7,19,26],lasso_sk:6,last:[0,1,2,3,4,5,6,7,8,10,12,13,14,17,18,20,21,23,24,26,28],latent:4,latent_dim:4,latent_point:4,latent_space_value_rang:4,later:[0,1,4,7,8,12,13,14,15,19,21,26],latest:[4,19],latest_checkpoint:4,latex:26,latter:[0,3,6,7,8,11,13,16,20,21,23,26,27,28,29],lattic:12,law:0,layer:[0,4,13,26],lbfg:[7,9,10,11],lcc:[5,6,28,29],lda:11,ldot:[0,6,11,21,26,29],le:[5,7,10,13,17,23,27,28],lead:[0,1,3,5,6,7,8,9,10,11,12,13,16,17,20,23,26,27,28,29],leaf:9,leaki:1,leakyrelu:4,lear:13,learn:[3,4,5,6,7,8,9,10,12,18,20,22,24,25],learnabl:3,learner:10,learnig:26,learning_r:[8,10],learning_rate_init:[0,1,26],learning_schedul:13,learnt:21,least:[0,2,7,8,10,11,15,16,17,19,20,23,29],leat:13,leav:[0,1,3,5,6,9,11,26,28,29],lectur:[0,1,5,10,11,12,13,15,17,19,20,21,22,24,25],lecturenot:[0,17,19,21,25,26],left:[0,1,2,3,5,6,7,8,9,10,11,12,13,14,16,17,20,21,23,26,27,28,29],leftarrow:[8,12],legend:[0,2,3,4,5,6,7,8,9,10,13,26,27,28,29],len:[0,1,2,3,4,5,6,8,9,10,11,12,20,26,27,28,29],length:[0,1,2,3,4,8,9,13,16,19,26,27],length_of_sequ:4,leq:[0,5,7,8,13,14,23,26,27,28],less:[0,1,3,4,5,6,8,9,13,19,23,26,27,28,29],lessen:1,let:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,23,26,27,28,29],letter:[0,16,20,23,26,27],level:[0,1,5,6,9,19,20,21,22,24,26,29],li:[8,11],lib:[0,1,2,3,4,6,7,8,9,11,14,26,28],liblinear:[8,10],libmetal_plugin:[3,4,14],librari:[0,1,2,3,4,5,6,7,9,10,11,14,15,20,21,23,25,27,28,29],library_loc:[3,4,14],licens:[0,1,15,19,21,26],lie:[0,6,11,23,26,27,29],life:[0,1,8,12,26],lifetim:13,like:[0,1,2,3,4,5,6,7,9,10,11,12,13,15,16,19,20,21,23,26,27,28,29],likelihood:[0,1,5,9,26,27],lim_:23,lima:24,limit:[0,5,6,8,9,11,12,20,21,26,27],lin_clf:8,lin_model:0,lin_reg:9,linalg:[0,2,5,6,8,11,13,20,23,26,27,28,29],line1:8,line2:8,line3:8,line:[0,1,2,3,4,6,7,8,10,11,13,14,21,26,27,28,29],linear:[1,3,5,6,7,9,10,11,12,17,18,19,21,23,29],linear_model:[0,5,6,7,8,9,10,11,13,26,27,28,29],linear_regress:[6,29],linearli:[5,27],linearloc:[6,13,21],linearregress:[0,6,7,9,26,27,28,29],linearsvc:8,lineat:28,liner:[1,3],linerar:10,linewidth:[0,2,4,6,8,9,10,21,29],link:[0,4,9,12,19,21,24,26],linlag:[5,28],linpack:[20,26],linreg:[0,26],linspac:[0,2,3,4,6,8,9,10,13,15,16,20,23,26,27,28,29],linu:4,linux:[0,1,15,19,21,26],liquid:[0,26],list:[0,1,2,3,4,7,9,19,21,26,27],listcomp:2,listedcolormap:[9,10],literatur:[1,7,14,25,29],littl:[1,3,9,12],live:8,ll:[0,23,26,27],lle:[0,27],lloyd:[4,14],lmb:[0,2,5,6,28,29],lmbd:[0,1,3,26],lmbd_val:[0,1,3,26],lmbda:13,ln:[1,13],load:[0,1,3,4,6,7,9,10,14,21],load_boston:0,load_breast_canc:[1,7,9,10,11],load_data:[3,4],load_digit:[1,3],load_iri:[8,9],load_librari:[3,4,14],load_pluggable_device_librari:[3,4,14],loc:[0,3,6,7,8,9,10,26,29],local:[0,1,2,3,7,12,13,27],locat:[2,3,8],log10:[0,5,6,28,29],log:[0,1,2,4,5,6,7,9,10,11,13,20,21,26,27,28,29],log_:[0,26],log_clf:10,logarithm:[0,5,7,20,26,28,29],logbook:21,logic:[0,1,9,26],logist:[0,1,2,8,9,10,11,12,13,19,27],logisticregress:[7,9,10,11],logit:7,logreg:[7,9,10,11],logspac:[0,1,3,5,6,26,28,29],longer:[2,3,8,10,14,20,23,26],loocv:[6,29],look:[0,1,2,3,4,5,6,7,8,9,10,11,13,16,18,20,21,23,26,27,28,29],loop:[1,4,6,10,12,14,19,20,26,29],lose:1,loss:[0,1,3,4,5,6,7,8,10,11,13,20,21,26,28,29],loss_fil:4,lossfil:4,lost:4,lot:[0,1,4,6,29],low:[0,6,9,10,11,21,26,27,29],lower:[0,1,3,6,9,10,16,20,27],lowercas:[20,26],lowest:[9,13,23],lr:[1,3,4,10],lstat:0,lstm:4,lstm_2layer:4,lstsq:[0,26,27],lt:[6,29],lu:[0,5,26,27],lubksb:20,luckili:2,ludcmp:20,lux:20,lvert:1,lw:[0,26],m:[0,1,2,3,5,6,8,9,10,11,12,13,20,22,23,24,25,26,27,28,29],m_1:14,m_:[9,12],m_h:[0,26],m_k:14,m_l:12,m_n:[0,26],m_p:[0,26],m_t:13,ma:11,machin:[1,3,4,5,6,7,9,10,11,12,15,20,22,25,27,28,29],machinelearn:[0,6,16,17,19,21,22,24,25,26],mackai:25,made:[0,1,3,4,5,6,7,9,11,12,18,21,26,27],mae:[0,26],magic:4,magnitud:[1,6,7,13,27,28],mai:[0,1,2,3,5,6,7,8,9,11,12,13,18,19,20,21,23,26,27,28,29],mail:[22,24],main:[0,1,3,4,5,6,7,9,20,21,25,27],mainli:[0,5,6,7,9,26,27,28,29],maintain:[6,29],major:[1,6,9,10,13,20,26,29],make:[1,2,3,4,5,6,7,8,11,12,13,19,20,21,23,25,28,29],make_axes_locat:6,make_moon:[8,9,10],make_pipelin:[0,6,10,27,29],make_vjp:2,makedir:[0,6,7,9,26,27,29],makeplot:[0,26],malcondit:20,malign:[1,7,9],mammographi:[5,28,29],manag:[0,2,3,15,19,21,26],mandatori:[24,26],mani:[0,1,3,4,5,6,7,8,9,11,13,14,19,20,21,23,25,26,27,28,29],manifold:11,manner:3,manual:[6,27,28],map:[0,1,2,6,7,8,11,12,14,21,23,26],marc:27,margin:[0,5,8],marit:[0,26],mark:26,marker:[0,7,20,26],markov:[19,26],marsaglia:23,mass:[0,1,5,13,27],massag:[0,26],masses2016:[0,26],masses2016ol:[0,26],masses2016tre:0,masseval2016:[0,26],master:[6,16,21,22,24],mat1100:[19,26],mat1110:[19,26],mat1120:[19,26],mat:[19,26],match:[1,4,5,13,14,27],materi:[4,5,7,13,17,20,24],math:[3,7,12,13,20,23,25,26,29],mathbb:[0,4,5,6,7,8,11,12,13,14,17,18,20,21,23,26,27,28,29],mathbf:[0,5,6,7,8,13,18,20,21,26,27,28,29],mathcal:[1,5,6,7,13,21,28,29],matheemat:3,mathemat:[0,6,11,12,13,17,19,20,23,25,26,28,29],mathemati:26,mathmlbook:16,mathrm:[0,1,3,4,5,6,7,8,9,10,11,12,13,14,17,18,21,23,26,27,28,29],matmul:[1,2,5,28],matnat:25,matplotlib:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,19,20,21,23,26,27,28,29],matric:[0,1,3,4,6,7,8,11,13,16,17,19,27],matrix:[0,2,3,4,6,7,8,10,13,15,16,17,18,21,23,29],matshow:1,matter:[2,3,13,27],max:[0,1,2,3,4,9,10,12,13,24,26],max_depth:[0,9,10],max_diff1:2,max_diff2:2,max_diff:2,max_it:[0,1,8,9,11,13,26],max_iter:14,max_leaf_nod:10,max_sampl:10,maxdegre:[0,6,10,27,29],maxdepth:10,maxim:[1,4,5,7,8,11,28,29],maximum:[0,1,2,3,5,7,8,9,10,13,14,26,27],maxpolydegre:[5,6,28,29],maxpooling2d:3,mbox:[5,6,18,21,27,28,29],mcculloch:12,md:[11,21],mdoel:4,mean:[1,2,3,4,5,6,7,9,10,11,12,13,14,15,16,17,18,19,20,21,23,26,29],mean_absolute_error:[0,26],mean_divisor:14,mean_i:23,mean_matrix:14,mean_squared_error:[0,4,6,7,10,26,27,29],mean_squared_log_error:[0,26],mean_vector:14,mean_x:23,meaning:[0,4,7,26],meansquarederror:[0,26],meant:[2,3,7,10,13],measur:[0,1,2,5,6,9,11,12,14,21,23,26,27,28,29],mechan:[0,4,23,26],median:[0,26,27],medicin:12,medium:[4,8,13],medv:0,meet:[0,24],mehta:[0,26,27,28],mek4270:[],memori:[3,4,11,12,13,20],mention:[0,12,13,21,23,26],mere:[0,15,26],meshgrid:[2,5,6,8,9,10,11,21,27],messag:[5,13],messi:2,met:[0,3,8,27],metadata:2,meteorolog:9,meter:[6,27,28],method:[0,1,2,3,4,5,7,8,11,12,14,16,17,18,19,20,23,25,27,28],metion:[6,21],metric:[0,1,3,6,7,9,10,14,15,16,26,27,29],metropoli:[19,26],mev:[0,23,26],mgd:13,mglearn:[19,26],mgrid:13,mhjensen:[1,2,3,4,6,8,9,11,14],mi:10,mia:[24,26],michael:[],michigan:[26,28,29],microsoft:25,mid:1,midel:4,midnight:[16,17,18],midpoint:9,might:[0,1,2,4,6,9,13,27,28],mild:9,millimet:[6,27,28],million:[0,26,27],mimic:12,min:[0,2,5,8,9],min_:[0,2,5,14,17,26,27,28],min_samples_leaf:9,mind:[0,6,13,21,26,27,28,29],mindboard:4,mine:[19,26],mini:[1,11,12,13],minibatch:[1,11,13],minibathc:13,miniforge3:[0,1,2,3,4,6,7,8,9,11,14,26,28],minim:[0,1,2,3,5,6,7,8,9,10,11,12,13,14,17,18,21,27,28,29],minima:[0,1,7,13,26],minimum:[0,1,2,6,8,9,11,13,27,29],minmaxscal:[0,27],minor:23,minst:1,minu:7,mirjalili:26,mirror:9,misc:[6,21],misclassif:[8,9,10],misclassifi:[8,10],miser:0,mismatch:1,miss:[0,7,10],mistak:4,mit:25,mix:[1,2,26],mixtur:13,mk:[9,20],mkdir:[0,6,7,9,26,27,29],ml:[0,1,10,13,20,21,27],mlab:23,mlbook:26,mle:[5,7],mlp:1,mlpclassifi:1,mlpregressor:[0,26],mm:20,mml:27,mn:[12,23],mnist:[1,11],mod:23,mode:[22,24,26],model:[2,3,5,7,8,9,10,11,13,14,15,16,18,19,21,23,25,27,28,29],model_select:[0,1,3,5,6,7,9,10,11,26,27,28,29],moder:10,modern:[0,6,7,19,26,29],modif:[2,12,13],modifi:[0,1,3,5,7,8,10,12,13,26,27,28],modul:[0,9,11,20,26,28],modular:23,modulo:23,moe:[11,27],moment:[5,6,13,23,28,29],momentum:[],mondai:[26,29],monitor:13,monoton:[5,12,23,28,29],mont:[0,6,19,23,25,26,29],moor:[5,6,28],more:[0,1,2,4,5,7,8,9,10,11,12,13,14,18,19,23],moreov:[0,3],morten:[24,26,27,28,29],most:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,19,21,23,26,27,28,29],mostli:[1,11],motion:[0,13],motiv:[1,4],move:[0,4,5,6,7,9,12,13,14,21,23,27,28,29],mpl:[0,7,26],mpl_toolkit:[2,6,13,21],mplot3d:[2,6,13,21],mplregressor:1,mse:[0,4,5,6,9,10,15,16,17,21,26,27,28,29],mse_simpletre:10,mselassopredict:[5,28],mselassotrain:[5,28],mseownridgepredict:[6,28],msepredict:[5,28],mseridgepredict:[0,5,6,28],msetrain:[5,28],msle:[0,26],mt:[7,12],mu0:23,mu1:23,mu2:23,mu:[0,6,11,13,23,26,29],mu_1:27,mu_:[6,23,27,28,29],mu_i:[6,27,28,29],mu_n:11,mu_x:23,much:[0,1,2,3,4,5,6,8,9,10,11,12,13,20,21,23,26,27,28,29],multi:[0,1,3,7,19,26],multiclass:[1,7],multidimension:[11,12,26],multilay:1,multinomi:7,multipl:[2,4,5,6,7,12,13,23,27,29],multipli:[3,5,6,11,13,20,23,27,28,29],multiplum:8,multivari:[0,2,10,11,19,23,26],multivariate_norm:[11,14],murphi:[11,25,26],must:[1,2,5,6,8,10,12,13,14,23,27,29],mutat:7,mutual:[1,3,6,13,29],mx_:23,myenv:[0,1,2,3,4,6,7,8,9,11,14,26,28],myriad:[0,15,19,26],mz1:23,mz2:23,n1:20,n2:20,n:[0,1,2,3,4,5,6,7,8,10,11,12,13,14,15,16,17,18,20,21,23,26,27,28,29],n_0:[12,23],n_:[1,2,3,8,12,23],n_boostrap:[6,10,29],n_bootstrap:[6,29],n_categori:[1,3],n_cluster:14,n_compon:11,n_epoch:13,n_estim:10,n_examples_to_gener:4,n_featur:1,n_filter:3,n_hidden:2,n_hidden_neuron:[0,1,26],n_i:23,n_input:[0,1,3,27],n_instanc:9,n_iter_i:[9,11],n_job:10,n_k:14,n_l:[12,23],n_layer:1,n_m:9,n_neuron:1,n_neurons_connect:3,n_neurons_layer1:1,n_neurons_layer2:1,n_point:14,n_sampl:[6,8,9,10,14,29],n_split:[6,29],n_step:4,n_t:2,n_x:2,nabla:[1,13],nabla_:[2,13],nabla_w:13,nag:13,naimi:[0,26],naiv:7,naive_kmean:14,najafi:[24,26],name:[0,1,3,4,5,6,7,8,9,10,12,13,14,15,19,20,21,23,24,26,27,28,29],nameerror:[10,29],namespac:[3,4,14],narrow:13,nary_f:2,nary_op_arg:2,nary_op_kwarg:2,nary_oper:2,nation:[1,5,28,29],nativ:[19,26],natur:[0,1,4,8,9,12,13,21,23,25,26],navier:12,nb:23,nb_:20,nbconvert:26,nbviewer:9,nd:14,ndarrai:[2,6],ne:[9,10,20,23,27],nearest:[1,3,6,11],nearli:13,neat:26,neccesari:[6,29],necess:2,necessari:[0,1,3,4,8,14,26],necessarili:[0,4,11,23,26],necesserali:[5,28,29],neck:7,need:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,20,23,27,28,29],neg:[0,1,3,5,6,7,10,13,20,23,26,28,29],neg_mean_squared_error:[6,29],neglect:23,neglig:23,neighbor:[3,6,11],neither:[4,13],neq:[13,14,23],nervou:12,nest:[2,9,12],nesterov:13,net:[2,4,12],netlib:[20,26],network:[0,9,13,19,25,27],networkd:[],neural:[0,13,19,25,27],neural_network:[0,1,2,26],neuralnetwork:1,neuron:[1,2,3,4,12],neutral:[0,26],neutron:[0,26],never:[1,4,6,9,23,29],new_box:2,new_chang:13,new_hobbit:26,new_root:2,new_trac:2,newaxi:[0,3,6,9,29],newli:[0,26],newton:[1,7,8,13,23],next:[0,1,2,3,4,5,6,7,8,9,13,14,26,27,28],next_guess:13,next_input:4,ng:1,ngini:9,ni:14,nice:[0,1,5,11,26,27,28,29],nielsen:[],niter:13,nitric:0,nlambda:[0,5,6,28,29],nlp:25,nm:23,nm_n:[0,26],nmse:[6,29],nn:[2,5,6,12,20,28,29],nn_model:1,nnmin:2,node:[1,2,3,9,10,12],node_constructor:2,nois:[0,4,5,6,8,9,10,13,15,16,21,26,27,28,29],noise_dimens:4,noisi:[1,6,21,29],non:[0,1,3,5,6,7,9,10,11,12,13,14,20,23,26,27,28,29],noncommerci:[],none:[0,1,2,4,5,9,10,13,23,26,27,28,29],nonlinear:[3,6,8,9,11,12,29],nonneg:[6,9,13,29],nonparametr:6,nonsens:23,nonsingular:20,nonumb:[3,7,8,13,20],nor:[1,4,13],norm:[0,1,5,6,8,11,13,17,26,27,28,29],normal:[3,4,5,6,7,8,9,10,11,12,13,15,16,18,19,20,21,23,26,27,28],normali:[20,26],norwai:[6,21,26],notat:[0,2,5,6,13,14,23,26,27,28,29],note:[0,1,2,3,4,5,6,7,8,11,12,13,14,15,16,19,20,23,25,26],notebook:[0,1,3,9,15,19,21,26,29],notfounderror:[3,4,14],noth:[1,2,5,8,12,14,23,27,28],notic:[4,5,12,13,20,23,26,28],notimplementederror:2,notion:3,nov:[],novel:[3,6,10],novemb:[1,22,24,26],now:[0,2,4,5,6,7,8,10,11,12,14,15,16,17,19,20,21,23,26,27],nowadai:[0,1,3,9,19,26],nox:0,np:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,15,16,20,21,23,26,27,28,29],npr:2,nsampl:[6,9,29],nt:2,ntrained_model:[6,28],nu:23,nuclear:[5,27],nuclei:[0,23,26],nucleon:[0,26],nucleu:[0,26],num:4,num_coordin:2,num_hidden_neuron:2,num_it:2,num_neuron:2,num_neurons_hidden:2,num_point:2,num_tre:10,num_valu:2,number:[1,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,20,21,22,24,26,28,29],numberid:7,numberparamet:3,numer:[0,5,6,9,10,11,12,13,17,19,20,25,26,27,28,29],numpi:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,19,21,23,27,28,29],numpy_vjp:2,numpy_wrapp:2,nunmpi:[5,27],nvalu:9,nx:2,nx_test:[6,28],nx_train:[6,28],nx_train_mean:[6,28],ny:23,ny_pr:[6,28],ny_train:[6,28],ny_train_mean:[6,28],o:[0,6,7,8,9,11,15,20,24,25,26],obei:[6,11,13,27,28,29],object:[0,1,2,4,6,8,10,20,26],obliqu:[5,27],observ:[0,1,3,5,6,7,8,9,10,11,12,13,14,23,28,29],obtain:[0,1,5,6,7,8,9,10,12,13,14,17,20,21,23,26,27,28,29],obviou:[5,6,11,23,27],obviouli:26,obvious:[0,4,5,6,20,26,28,29],oc:27,occupi:0,occur:[0,6,8,9,20,23,26],octob:[22,24,26],od:0,odd:[0,3,7,26,27],odenum:2,odesi:2,odin:26,oen:0,off:[1,3,4,5,9,13,23,28,29],offer:[6,11,19,20,22,24,26,29],offic:[24,26],offici:[22,26],often:[0,1,3,4,5,6,7,8,9,10,11,12,13,14,16,19,20,21,23,26,27,29],ofter:[20,26],ol:[0,13,15,16,17,18,26,27],old:[1,5,10,13,28,29],ols_fit:29,ols_fit_beta:29,ols_sk:6,ols_svd:6,olsbeta:[0,5,28],omega:[2,3,6],omega_0:3,omit:[0,5,26,27,28,29],onc:[1,6,9,11,13,29],one:[0,1,3,4,5,6,7,8,9,10,11,13,14,19,20,21,23,24,26,27,28,29],onehot:1,onehot_vector:1,onehotencod:9,ones:[0,2,5,6,8,9,10,11,13,20,21,26,27,28,29],ones_lik:4,ong:27,onl:3,onli:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,20,21,23,26,27,28,29],onlin:[11,22],onto:[5,11,27],open:[0,1,4,6,7,9,15,19,21,22,24,26,29],oper:[0,1,3,5,6,10,11,12,13,15,19,23,26,27,28,29],operation:23,oplu:23,opmiz:13,opportun:0,oppos:[6,13],opposit:[1,5,8,27],opt:[1,5,26,28],optim:[0,2,3,4,5,6,7,9,10,11,14,15,16,17,18,21,28,29],optimis:[1,3],option:[0,1,3,5,6,8,9,11,20,21,27,28,29],optmiz:[1,8,13,27],oral:26,orang:0,order:[0,1,2,3,5,6,7,8,9,10,11,12,15,16,20,21,23,26,27,28,29],ordinari:[0,2,3,7,11,13,15,16,17,19,29],ordinrari:29,oreilli:[25,26],org:[0,3,4,9,11,19,20,21,25,26,28],organ:[6,7,10,20,29],orient:[1,5,23,27,28],origin:[0,3,5,6,8,11,12,13,20,26,27,29],orthogn:[5,27],orthogon:[0,5,6,8,11,13,17,20,26,27,28],orthonorm:[5,27,28],os:[0,1,4,5,6,7,8,9,24,26,27,28,29],oscar:1,oscil:[3,13],oserror:[0,3,4,7,14],oslo:[0,15,19,21,22,24,26,27,28,29],osx:[0,15,19,21,26],other:[0,1,2,3,5,6,7,8,10,13,14,15,19,22,23,24,25,28,29],otherwis:[0,1,4,7,13,20,26],ouput:[5,7,12,28,29],our:[1,2,3,6,7,8,9,10,12,14,15,16,17,18,19,20,21,23,28,29],ourmodel:0,ourselv:[0,5,6,8,11,13,26,27,28,29],out:[0,1,2,4,5,6,7,8,9,10,11,12,13,16,19,20,21,23,26,27,28,29],out_fil:9,outcom:[0,7,9,10,12,23,27],outdoor:9,outer:[6,12,13],outfil:4,outgrad:2,outlier:[0,8,26,27],outlin:[6,10,11,29],outlook:9,outperform:10,output:[0,1,3,4,5,6,7,8,9,10,12,13,16,20,21,23,26,27,28,29],output_bia:1,output_bias_gradi:1,output_shap:4,output_weight:1,output_weights_gradi:1,outputlayer1:12,outputlayer2:12,outsid:4,over1:13,over:[0,1,3,4,5,6,9,10,12,13,20,21,26,27,28,29],overal:[1,10],overcast:9,overcom:[12,13],overdetermin:[0,26],overfit:[0,1,3,6,9,10,13,29],overflow:[1,5,28,29],overhead:12,overlap:[3,7,8,9],overlin:[0,5,6,9,10,11,14,20,26,27,28,29],overst:0,overtrain:4,overview:3,own:[4,5,6,8,12,13,19,20,21,28,29],owner:0,ownmsepredict:0,ownmsetrain:0,ownridgebeta:[0,6,28],ownypredictridg:0,ownytilderidg:0,oxid:0,p0:2,p1:2,p:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,17,20,23,26,27,28,29],p_:[2,4,8,9],p_hidden:2,p_i:[5,23,28,29],p_j:23,p_n:23,p_output:2,p_x:23,pack:[0,26],packag:[0,1,2,3,4,5,6,7,8,9,11,13,14,15,19,21,23,27,28],packtpub:26,packtpublish:26,pad:[3,4],page:[0,9,19,26,28],pai:[0,1,9,13],paid:[],pair:[0,2,3,9,19,23,26],panda:[0,4,5,6,7,9,11,15,19,21,28,29],panel:26,paper:1,paradigm:[0,26],parallel:[10,13,19,20,26],param:2,paramat:2,paramet:[0,1,2,3,4,5,6,7,8,9,10,12,13,16,17,18,21,23,28,29],parameter:[0,6,10,21,26,27],parametr:[0,6,15,16,26,27,29],paramt:[3,5,28,29],parent:2,parent_argnum:2,part:[0,1,3,5,6,10,17,18,20,22,23,24,26,28,29],partial:[0,1,5,6,7,8,10,11,12,13,16,23,26,27,28,29],particip:[19,22,24,26],particl:[0,4,13,23,26],particular:[0,1,2,3,5,6,9,10,11,12,13,16,21,23,25,26,27,28,29],particularli:[5,6,8,11,13,23,27,29],partit:[1,4,9],partli:[6,26],pass:[2,3,12,14],password:21,past:[10,23],patch:[6,23,29],path:[0,3,4,6,7,9,14,15,19,26,27,29],patient:7,patter:4,pattern:[0,3,4,12,22,25,26],pauli:[0,26],pc:[11,19],pca:[0,7,19,26,27],pd:[0,4,5,6,7,9,11,26,27,28,29],pde:2,pdf:[0,3,4,5,6,9,16,21,25,26,28,29],pedagog:[0,26,27],penal:[6,27,28,29],penalti:[6,13,21,27,28,29],penros:[5,6,28],pentagon:13,peopl:[0,1,9,13,19],per:[0,1,6,22,24,26,29],percentag:[0,10,11,24],perceptron:[0,1,7,26],peregrin:26,perfect:[0,1,13,26],perfectli:[4,6,29],perform:[0,2,3,4,5,6,8,10,11,12,13,14,15,16,17,19,20,21,23,26,27,28,29],performac:4,perhap:[0,5,13,26,27,28],perimet:[1,9],period:[1,4,23],permut:11,persist:13,person:[5,6,7,22,24,26,27,28,29],perspect:25,pertin:[12,26],petal:[8,9],peter:[25,27],phantom:23,phase:[6,12],phenomena:23,phi:8,phi_k:8,philosophi:13,phone:[24,26],photo:4,php:[],phrase:[0,26],physic:[0,1,4,7,12,13,23,24,25,26,27,28,29],physicist:[],pi:[2,3,5,6,7,9,12,13,23,28,29],pick:[1,9,10,11,13,14],pickl:1,pictur:[0,26],pie:[19,26],piec:[11,14],pillow:[0,15,19,21,26],pinv:[5,6,13,27,28,29],pip3:[0,1,15,21,26],pip:[0,1,15,19,21,26],pipelin:[0,6,8,10,27,29],pippin:26,pit:4,pitfal:[6,27,28],pitt:12,pixel:[1,3,4],pixel_height:[1,3],pixel_width:[1,3],place:[0,4,6,8,13,20,21,26,29],plai:[0,3,4,5,6,8,11,19,26,27,28,29],plain:[8,10,12,13,14],plan:[6,9,24,25,26],plane:[8,9],plateau:[5,28],platform:[19,26],plausibl:12,pleas:[9,11,13,21,24,26],plenti:1,plethora:[3,12],plot:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,19,20,21,23,26,27,28],plot_all_sc:[21,28],plot_confusion_matrix:[7,10],plot_count:6,plot_cumulative_gain:[7,10],plot_data:1,plot_dataset:8,plot_decision_boundari:[9,10],plot_import:10,plot_max:4,plot_min:4,plot_model:4,plot_numb:4,plot_predict:8,plot_regression_predict:9,plot_result:4,plot_roc:[7,10],plot_surfac:[2,6,13,21],plot_train:9,plot_tre:[9,10],plt:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,21,23,26,27,28,29],plu:[0,3,5,7,26,27,28],pluggabl:[3,4,14],plugin:[3,4,14],pm:[8,29],pmatrix:2,pml:25,pn:3,png:[0,4,6,7,9,26,27,29],point:[0,1,2,3,5,6,7,8,9,10,11,13,14,15,16,17,20,21,23,24,26,27,29],point_1:4,point_2:4,poisson:[19,23,26],poli:[6,8,29],poly100_kernel_svm_clf:8,poly3:0,poly3_plot:0,poly_featur:[8,9],poly_features10:9,poly_fit10:9,poly_fit:9,poly_kernel_svm_clf:8,polydegre:[0,5,6,10,27,28,29],polygon:13,polym:12,polymi:21,polynomi:[0,5,6,7,8,9,10,11,15,16,17,21,26,27,29],polynomial_featur:[6,29],polynomial_svm_clf:8,polynomialfeatur:[0,6,8,9,27,29],polytrop:[0,6,26,29],pool:3,pool_siz:3,poor:[1,13],poorli:[0,27],pop:2,popul:[0,5,26,27,28,29],popular:[0,1,3,6,7,8,9,11,12,15,19,20,21,23,27],popularli:[0,26],portabl:10,portion:[11,13],pose:[0,4,5,6,11,23,26,29],posit:[0,1,2,3,5,7,8,10,11,13,14,20,23,26,27,28,29],possibl:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,15,19,20,21,23,24,26,27,28,29],posterior:[5,28,29],postpon:[0,26,27],postscript:21,postul:[5,28,29],potenti:[0,3,5,6,12,13,26,27,28,29],pott:12,power:[0,1,5,6,8,9,12,13,26,27,29],pp:[5,6,18,28,29],practic:[0,5,6,7,8,21,23,27,28,29],practition:[0,1,3,26],pre:26,preced:[1,11,12,23],preceed:4,preceq:8,precis:[0,2,5,11,13,20,21,23,26,27,28,29],pred:[6,29],predicit:0,predict:[0,1,5,6,7,8,9,10,15,16,19,21,25,26,27,28,29],predict_prob:1,predict_proba:[7,10],predictor:[0,5,6,7,9,10,11,26,27,28],prefer:[0,1,6,8,9,11,13,15,19,21,26],prepar:[0,6,20,21,26,27],preprocess:[0,4,6,7,8,9,10,11,21,28,29],prerequisit:0,prescript:21,presenc:13,present:[0,5,6,7,9,12,13,20,21,23,26,27,28],preserv:[3,11,20],press:[13,25],pretrain:[1,4],pretti:[0,4,8,9,15,19,21,26],prev_centroid:14,prevent:[13,23],previou:[0,1,2,3,4,5,6,8,10,11,12,13,20,21,23,27],previous:[2,3,9,10,23],price:[0,4,9,13],primal:8,primari:[0,7,26],prime:23,primit:2,princip:[0,5,7,19,26,27],principl:[0,6,7,8,14,26,29],print:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,20,23,26,27,28,29],print_funct:[8,9],printout:[0,26],prior:[0,5,6,26,28,29],privat:0,pro:[],prob:[1,23],probabilist:[0,25,26,27],probabl:[0,1,3,4,6,7,10,13,19,26,27],problem:[0,3,4,5,6,7,8,9,10,11,12,16,17,19,20,21,23,29],probml:25,proce:[0,5,6,7,8,9,10,11,13,20,26,27,28,29],procedur:[2,4,5,6,8,10,11,13,21,27,28,29],proceed:20,process:[0,2,4,6,9,10,12,13,15,19,20,21,23,25,26,29],prod:25,prod_:[1,5,7,28,29],produc:[0,3,4,5,6,9,10,11,12,13,19,20,21,23,26,27,28,29],product:[0,1,3,5,6,7,8,12,13,16,19,20,26,27,28,29],profess:[0,26],progag:[],program:[0,1,4,5,6,8,12,14,15,16,19,20,22,23,24,26,27,29],programm:20,progress:[1,4,14],prohibit:[6,29],project1:[6,21],project:[0,1,2,3,5,11,13,17,18,19,22,24,27,28,29],project_root_dir:[0,6,7,9,26,27,29],projectwrit:21,promin:12,promis:8,promot:[24,26],prone:9,pronounc:[13,19,26],proof:[0,11,12,13,26,29],propag:[2,3,13],proper:[0,2,6,7,21,29],properli:[1,6,8,10,13,21],properti:[0,1,3,12,13,16,17,20,26,29],proport:[0,1,5,9,11,13,16,23,26,27],propos:[1,4,6,10,21,26],propto:[5,13,28,29],proton:[0,26],prove:[3,13],provid:[0,1,3,4,5,6,8,9,10,12,13,15,19,20,21,23,26,27,28,29],proxi:[1,13],prune:9,pseudo:[20,23],pseudocod:21,pseudoinv:[5,28],pseudoinvers:[5,6,28],pseudorandom:[6,23,29],psycholog:[0,26],pt:13,publish:[],punish:[0,1,26],pure:[3,9,23],purest:9,puriti:9,purpos:[0,3,10,12,14,26],put:1,py:[0,1,2,3,4,5,6,7,8,9,11,14,26,28,29],py_tf:[3,4,14],pycod:26,pydata:19,pydot:9,pyhton2:26,pylab:[0,7,26],pypi:19,pyplot:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,21,23,26,27,28,29],pythagora:[5,28,29],python2:[0,15,21,26],python3:[0,1,2,3,4,6,7,8,9,11,14,15,19,21,26,28],python:[1,2,3,4,5,6,8,11,12,13,14,16,21,23,27,28],pytorch:[0,19,21,26],q:[5,6,8,11,23,27,29],qp:8,qquad:[2,11,13,20],qr:[5,6,20,27],quad:[1,13,20],quadrat:[0,8,9,13,15,16,26],qualit:[4,9,21,23],qualiti:[0,9,15,16,19,26,27],quantifi:1,quantil:10,quantit:[0,6,9,21,26,29],quantiti:[0,2,5,6,7,9,10,11,12,14,16,20,23,26,27,28,29],quantum:[4,12,25,26],quartil:[0,27],quench:5,queri:9,question:[0,5,6,9,11,12,13,21,24,26,27,28,29],qugan:4,quick:[4,23],quickli:[1,3,9,11,13],quit:[1,5,6,9,10,12,27,29],quot:4,r2:[0,5,6,26,27,28],r2_score:[0,26],r2score:[0,26],r:[0,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,19,20,21,23,27,28,29],r_1:9,r_2:9,r_j:9,r_m:9,rad:0,radial:[0,8,12],radioact:23,radiu:[0,1,9,27],rag:2,rain:9,rais:[0,2,3,4,7,14,28],ramp:1,ran0:23,ran1:23,ran2:23,ran3:23,rand:[0,4,5,6,9,10,13,15,16,20,26,27,28,29],randint:[6,9,13,29],randn:[0,1,2,5,6,9,11,13,15,16,26,27,28,29],random:[0,1,2,3,4,5,6,8,9,13,14,15,16,19,20,21,26,27,28,29],random_forest_model:10,random_index:13,random_indic:[1,3],random_st:[0,7,8,9,10,11],randomforestclassifi:10,randomli:[1,6,9,13,14,29],rang:[0,1,2,3,4,5,6,7,9,10,11,12,13,14,20,23,26,27,28,29],rangl:[0,6,11,23,26,27],rangle_x:23,rank:[5,27],rankdir:4,raphson:[1,8,13],rapidli:0,rare:[1,13,26,29],raschka:[26,27,28],rasckha:26,rate:[0,1,2,3,4,8,9,10,12,13],rather:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,20,23,26,27,28,29],ratio:[4,7,9,10,11],rational:[0,26],ravel:[5,6,7,8,9,10,11,13,20,27,29],raw:3,rbf:[8,11,12],rbf_kernel_svm_clf:8,rbf_pca:11,rc:[0,23],rc_temp:[0,7],rcond:[0,26,27],rcparam:[0,1,3,7,8,9,10,23,26],re:[2,4,13],reach:[1,4,5,6,9,10,11,12,13,14,28,29],read:[0,2,3,4,5,6,7,8,11,12,16,20,21,22,23,25,28],read_csv:[0,6,7,9,26,29],read_fwf:[0,26],reader:[0,6,20,23,26,27,28],readi:[0,1,5,6,8,10,11,12,20,21,26,28,29],readili:1,readthedoc:19,real:[0,1,2,4,7,10,11,12,20,27,29],real_loss:4,real_output:4,realist:8,realiti:23,realiz:[1,12],realli:[0,1,26],rearrang:13,reason:[0,1,3,4,10,13,25,26],reassign:1,recal:[5,6,9,10,11,12,20,23,26,27,28,29],recast:3,receiv:[1,3,10,12,23],recent:[0,2,3,4,6,7,10,13,14,21,25,26,28,29],recept:[3,12],receptive_field:3,recip:[0,6,7,20,21,26,27],reciproc:[5,28],recogn:[0,4,5,10,26,28,29],recognit:[0,1,3,12,22,25,26],recommen:26,recommend:[0,2,3,4,5,6,8,13,15,18,19,20,21,25,28,29],reconsid:9,reconstruct:11,record:[10,21,22,24,26],rectangl:[9,13],rectangular:[5,27],rectifi:[1,3,12],recur:[0,19,26],recurr:[0,1,19,26],recurs:[9,19,20,26],red:[0,3,4,6,8,9,29],redefin:[0,10,26,27],redefinit:28,reduc:[1,3,5,6,9,10,11,13,26,28,29],reduct:[0,10,11,19,23,26,27],refer:[0,1,2,3,5,6,9,11,12,13,14,20,25,26,27,29],referenc:2,refin:12,refit:[6,29],reflect:[0,1,4,5,21,23,26],refresh:[19,26],refreshprogrammingskil:26,reg:[10,11],regard:[1,9,13],regardless:12,region:[3,4,6,9,12,21],regist:[6,21,23],reglasso:[5,28],regr_1:[0,9],regr_2:[0,9],regr_3:[0,9],regress:[1,8,11,12,15,16,19,20],regressor:[0,7,10,26],regridg:[0,5,6,28,29],regular:[0,3,4,5,6,7,9,13,17,24,26,27,28,29],regularis:6,reilli:[0,15,25,26],reinforc:[0,8,19,26],reiter:1,reject:7,rel:[0,4,6,7,9,12,13,23,26,27,29],relat:[0,1,3,4,5,11,13,14,20,23,26,28,29],relationship:[0,4,9,26],relativeerror:[0,26,27],releas:[1,19,26],relev:[0,1,5,7,11,15,19,21,23,26,28],reli:[0,6,8],reliabilti:21,reliabl:[7,23],relu:[3,4,26],remain:[1,2,4,6,12,20,23,27,28,29],remaind:23,reman:2,remark:1,rememb:[0,8,13,20,21,26],remind:[0,5,11,13,20,23,27,28,29],remov:[0,4,5,6,27,28,29],render:[0,9,26,27],reorder:[5,7,27,28],reorgan:[0,26],repeat:[0,1,3,4,5,6,9,10,11,13,14,20,21,23,26,28,29],repeated:26,repeatedli:[0,6,10,13,29],repet:3,repetit:[6,26,27,29],rephras:13,replac:[0,1,3,4,5,6,10,12,14,15,16,19,21,26,27,28,29],replica:[6,29],repo:21,report:26,reportexampl:21,repositori:[0,4,21,26],repres:[0,1,2,3,4,5,6,7,8,9,10,12,13,23,26,27,28,29],represent:[0,1,3,6,9,23,26,29],representd:3,reproduc:[0,5,6,9,12,15,16,19,21,23,26,27],repuls:[0,26],request:[0,13],requir:[0,1,3,4,5,6,8,9,11,12,13,17,20,26,27,28,29],rerun:9,res1:2,res2:2,res3:2,res_analyt:2,res_analytical1:2,res_analytical2:2,res_analytical3:2,resaml:[6,21],resampl:[0,7,10,19,26,27],rescal:[0,11,12,27],rescu:[5,28,29],reseach:[6,21],research:[0,4,13,19,25,26],resembl:[6,23,29],reserv:[1,5,6,23,28,29],reshap:[0,1,2,3,4,6,8,9,10,15,16,20,26,27,29],residenti:0,residu:[0,5,13,26],resiz:[5,27],resourc:26,respect:[0,1,2,3,5,6,7,8,10,11,12,13,14,16,17,21,23,26,27,28,29],respond:12,respons:[0,7,9,12,26,27],rest:[0,5,27],restat:[0,12,26],restor:4,restored_discrimin:4,restored_gener:4,restrict:[0,3,9,12,26],result:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,19,20,21,23,26,29],retail:0,retain:[5,6,27,29],rethink:29,retriev:26,return_data:14,return_sequ:4,return_x_i:9,reus:[1,3,6,18,21],reveal:[0,12,26],revers:[1,20],review:[19,20],revisit:14,revolut:26,reward:[0,4,26],rewrit:[0,3,5,6,7,8,10,11,12,13,20,21,23,28],rewritten:[2,6,8,10,23,29],rewrot:13,rf:10,rgb:3,rgoj5yh7evk:19,rh:[6,29],rho:[0,10,13,26],rho_1:10,rho_2:10,rho_m:10,rich:[0,26],ride:9,rideclass:9,ridedata:9,ridg:[7,11,13,19,26],ridge_fit:29,ridge_fit_beta:29,ridge_sk:6,ridgebeta:[5,28],right:[0,1,2,3,5,6,7,8,9,10,12,13,14,16,17,20,21,23,26,27,28,29],right_sid:2,rightarrow:[0,1,5,6,8,11,12,13,23,26,27,29],rigor:[0,26],ring:6,rise:[0,26],risk:[0,13,15,16,26],rival:4,river:0,rlm:26,rm:[0,23],rmse:0,rmsporp:13,rmsprop:[1,3,4,13],rnd_clf:10,rng:23,rnn1:4,rnn2:4,rnn:[4,12],rnn_2layer:4,rnn_input:4,rnn_output:4,rnn_train:4,rntrick1:23,rntrick2:23,rntrick3:23,rntrick4:23,ro:[0,13,26],robert:[18,21,25],robust:[0,26],robustscal:[0,27],roc:[7,10],rod:[],role:[0,2,5,6,8,19,26,27,28,29],roll:6,room:[0,24,26],root:[0,5,9,13,23,27,28],rot:26,rotat:[1,8,9,10],rotation_matrix:9,roughli:[1,3],round:[0,7,9,13],routin:[13,20,26],row:[0,1,2,5,6,9,11,20,26,27,28,29],rr:[5,27],rrr:[5,27],rug:13,rule:[0,1,5,6,13,21,26,27,28,29],run:[0,1,2,4,5,6,8,9,11,13,15,19,21,26,27,28,29],runtim:[1,6,14],runtimewarn:[1,6,29],rust:[0,15,19,20,26],rvert:1,rvert_2:1,s:[0,1,2,3,4,5,6,7,9,11,12,13,16,17,18,19,20,21,23,26,27,28],s_1:6,s_:[3,6],s_i:[6,7],s_j:6,s_k:6,saddl:13,sai:[0,1,2,3,4,5,6,7,8,9,10,11,12,20,21,23,26,27,28,29],said:[6,9,13],sake:[0,5,7,11,26,27,28],sale:[0,26],sam:26,same:[0,1,2,3,4,5,6,8,9,11,12,14,17,20,21,23,26,27,28],samm:10,sampl:[0,1,2,3,4,5,6,7,8,9,10,13,14,15,16,19,20,21,23,26,27,28,29],sample_vari:14,sampleexptvari:23,samwis:26,sastri:11,satisfactori:[0,26],satisfi:[1,2,3,6,8,13,20,23,29],satur:[1,6,29],save:[0,4,6,7,9,13,26,27,29],save_fig:[0,6,7,9,10,26,27,29],savefig:[0,4,6,7,9,23,26,27,29],savetxt:4,saw:[5,27],scalabl:10,scalar:[2,5,6,10,27,28,29],scale:[0,1,3,5,6,7,8,9,10,11,12,13,15,16,17,19,20,21,24,26,28],scale_mean:4,scale_std:4,scaler:[0,7,8,9,10,11,21,27,28,29],scan:[5,7,28,29],scari:[5,28,29],scatter:[0,1,6,7,8,9,14,26,27,28,29],scenario:[6,13],schedul:[13,24],scheme:[1,13],schrage:23,scienc:[0,1,10,12,13,19,22,23,24,25],scientif:[0,15,19,21,26],scientist:[0,26],scikit:[3,5,6,8,9,10,13,19,20,21,22,25],scikit_learn:[0,16],scikitlearn:26,scikitplot:[7,10],scipi:[0,3,5,6,13,15,19,20,21,26,27,28,29],scl:6,score:[0,1,3,6,7,9,10,11,15,16,21,24,26,27,29],scores_kfold:[6,29],scratch:[1,13],sdg:13,seaborn:[0,1,3,6,7,26],seamless:[0,15,19,21,26],search:[0,1,3,5,9,13,26,28],sebastian:26,sebastianraschka:26,sec:6,second:[0,2,3,4,5,6,7,8,9,11,12,14,15,16,19,20,21,23,24,26,27,29],secondeigvector:11,secondli:12,section:[4,11,16,17,20,21,23,27,28],sector:0,see:[0,1,2,3,4,5,6,7,8,10,11,12,13,15,16,17,18,19,20,21,23,26,27,28,29],seed:[0,1,2,3,4,5,6,8,9,11,13,14,15,16,21,23,26,27,28,29],seed_imag:4,seek:[1,2,8],seem:[1,3,4],seemingli:[0,26],seen:[0,1,3,5,10,12,17,23,26],segment:13,seismic:6,seldomli:[0,26],select:[1,5,6,8,9,10,11,17,21,22,23,24,25,26,27,28,29],self:[0,1,2,5,7,26,27,28],sell:4,semest:[7,22],semi:[8,13],semilogx:6,send:[5,12,13,24,26],senior:[22,24],sens:[0,4,6,8,21,26,29],sensibl:3,sensit:[0,5,6,9,13,26,27,28,29],sent:[2,29],sentenc:[4,12],separ:[0,1,2,4,6,8,9,12,14,15,19,21,23,26,29],septemb:[17,18,21,26],sequenc:[2,3,4,7,9,10,12,13,19,20,23,26],sequenti:[1,3,4,10,12,23],seri:[0,1,2,3,4,5,6,10,11,12,13,20,26,27,28,29],serif:[0,7,23,26],serv:[0,1,2,3,5,7,13,21,25,26,27,28,29],servic:21,session:[1,21,22,24,26,29],set:[1,4,5,6,7,8,10,11,13,14,16,17,19,20,21,23,24,28,29],set_major_formatt:[6,21],set_major_loc:[6,21],set_tick:[1,6,8],set_ticklabel:[1,6],set_titl:[0,1,2,3,7,12,14,26],set_xlabel:[0,1,2,3,7,12,26],set_xlim:[7,12],set_xticklabel:1,set_ylabel:[0,1,2,3,7,26],set_ylim:[7,12],set_ytick:7,set_yticklabel:[1,6],set_zlim:[6,21],seth:4,setminu:6,setosa:[8,9],setosa_or_versicolor:8,setp:[6,29],setup:[1,4,6,8,19,26,27],sever:[0,3,5,6,7,8,9,11,12,13,19,20,23,26,27,28,29],sgd:[1,3],sgd_clf:8,sgdclassifi:8,sgdreg:13,sgdregressor:13,sgn:[5,27,28],shallow:13,shape:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,15,16,20,26,27,28,29],shape_bas:2,share:[1,3,26],she:7,shift:[1,6,12,23,28],ship:3,shire:26,shortcom:13,shorten:4,shorter:23,shorthand:[26,29],shortli:[20,26],should:[0,2,3,5,6,8,9,11,12,15,16,20,21,23,26,27,28,29],show:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,17,18,20,21,23,26,27,28,29],show_shap:4,shown:[0,4,5,8,9,11,12,13,20,27],shrink:[3,5,6,8,11,21,27,28],shrinkag:[5,6,27,28],shrunk:11,shuffl:[0,1,4,6,13,27,29],side:[0,2,5,8,12,13,20,26,28,29],sigh:[19,26],sigma0:23,sigma1:23,sigma2:23,sigma:[0,1,5,6,7,10,11,12,13,17,18,20,21,23,26,27,28,29],sigma_0:[5,27,28],sigma_1:[5,27,28],sigma_2:[5,27,28],sigma_:[5,20,26,27,28,29],sigma_fn:[7,12],sigma_i:[0,5,26,27,28],sigma_j:[5,17,27,28],sigma_m:[6,23,29],sigma_n:[11,23],sigma_t:13,sigma_x:23,sigmoid:[1,2,4,7,8,10,12],sigmundson:[6,27,28],sign:[1,2,7,8,10,23,24],signal:[1,3,10,12],signifi:4,signific:1,significantli:[1,13,23],sigurd:26,sim:[4,5,6,13,18,21,23,28,29],similar:[0,1,2,3,4,5,6,7,8,9,10,11,14,17,19,20,21,26,28,29],similarli:[0,1,3,5,8,10,13,23,26,27,28],simpl:[1,2,3,5,6,7,8,10,11,12,14,15,16,17,19,20,21,23,29],simplepredict:10,simpler:[0,1,5,6,7,13,16,19,21,26,28],simplernn:4,simplest:[0,1,3,4,9,10,12,14,26],simpletre:10,simpli:[0,1,2,4,5,6,8,9,10,11,12,15,19,20,21,23,26,27,28,29],simplic:[2,5,6,7,8,9,10,11,12,14,27,28,29],simplicti:[5,27],simplifi:[0,6,9,15,19,21,26,27,28,29],simplist:[3,6,23,29],simul:[6,29],simultan:[6,29],sin:[0,1,2,3,4,9,12,13,20,26],sinc:[0,1,2,3,5,6,7,8,9,10,11,13,20,23,25,26,27,28,29],sine:[3,12],singl:[0,1,2,3,5,6,7,8,9,12,13,20,23,26,27,28,29],singular:[0,6,13,20,21,26,28,29],sinusoid:3,site:[0,1,2,3,4,6,7,8,9,11,14,21,22,26,28],situat:[0,4,5,7,13,23,26,27],six:[3,23],size:[0,1,2,3,4,5,6,8,9,10,11,13,20,21,23,26,28,29],sketch:10,ski:9,skill:0,skip:11,skl:[0,6,26,27,28],sklearn:[0,1,3,5,6,7,8,9,10,11,13,14,26,27,28,29],skplt:[7,10],sl:[6,28],slack:8,slice:[2,20,26],slide:[0,3,15,16,21,23,26,27,29],slight:[6,13,29],slightli:[1,2,3,5,6,7,10,23,27,28,29],slope:[8,11,12],slow:[0,2,8,13,27],slower:[5,20,26,27,28],slowest:20,slowli:12,slp:1,small:[0,1,2,3,5,6,8,9,10,11,12,13,19,20,21,23,26,27,28,29],smaller:[0,1,2,5,6,8,9,11,13,23,26,27,28,29],smallest:[0,4,14,15,16,26],smallest_row_index:14,smooth:[0,3,6,9,13,21,26],sn:[0,1,3,6,7,26],sne:11,sneak:26,so:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,18,19,20,21,23,24,26,27,28,29],soar:6,social:0,soft:[1,7,10,12],soften:8,softmax:[3,7],softwar:[0,8,15,19,20],sol:8,sole:[0,6,26],solid:[0,7],solut:[0,1,2,3,5,6,8,10,11,13,17,18,20,21,23,26,27,28,29],soluton:2,solv:[0,1,3,5,6,8,10,11,12,13,20,21,26,27],solve_expdec:2,solve_ode_deep_neural_network:2,solve_ode_neural_network:2,solve_pde_deep_neural_network:2,solveod:2,solveode_popul:2,solver:[2,7,8,9,10,11,20,26],some:[0,1,2,3,4,5,6,7,8,9,10,11,12,14,21,23,26,28,29],some_model:[6,27,28],somehow:4,someth:[0,1,3,4,7,9,11,21,23,26,27],sometim:[0,1,11,12,13,14,26,27],soon:[20,24,27],sophist:[0,26],sopt:13,sort:[5,6,9,11,23,27,29],sound:[3,5,28,29],sourc:[0,1,3,6,15,19,20,21,23,26,29],space:[0,1,4,5,8,9,11,12,13,14,23,27,28,29],span:[0,3,5,9,11,20,26,27],spare:1,spars:[3,6,20,26],sparse_mtx:[20,26],sparsecategoricalcrossentropi:3,sparsiti:10,spatial:[1,2,3,12],speak:23,special:[6,7,10,12,13,20,23,26,27,28,29],specif:[0,1,2,3,4,5,6,7,8,9,11,12,16,19,20,21,23,25,26,27,28,29],specifi:[0,3,5,6,7,9,11,13,14,21,23,26,28,29],specifici:[0,10,26],spectacular:3,spectral:1,speech:[0,1,3,4,12],speed:[1,2,4,13],spend:23,spent:21,sphere:[0,27],spin:6,spite:0,spline:8,split:[1,3,4,5,6,8,9,10,11,14,17,21,23,28,29],splite:0,splitter:[1,10],spontan:23,spot:3,spread:[0,11,23,26,27],springer:[18,21,25,26],spuriou:13,sqquar:28,sqrsignal:3,sqrt:[0,3,4,5,6,8,10,11,13,17,23,27,28,29],squar:[1,2,3,4,7,8,9,11,13,14,15,16,17,19,20,23,29],squarederror:10,squaredeuclidean:14,squash:12,srtm:[6,21],srtm_data_norway_1:[6,21],stabil:[5,21],stabl:[0,4,5,6,9,11,19,21,26,27,28],stack:[2,3,4],stage:[5,13,21],stai:[0,2,4,5,11,17,26,27],stand:[0,5,9,12,26,27,28],standadscal:27,standard:[0,1,4,5,6,7,8,10,12,15,16,17,20,21,23,26,28],standard_basi:2,standardscal:[0,6,7,8,9,10,11,27,28,29],stanford:13,start:[0,1,2,3,4,5,6,8,9,10,11,12,13,14,20,23,24,26,27,29],start_box:2,start_nod:2,start_tim:14,stat:[6,29],state:[1,2,4,5,6,7,8,10,11,12,13,19,23,27,28,29],statement:[0,7,20,26],stationari:[],statist:[0,1,3,4,7,9,10,11,12,13,14,18,20,21,22,25,27],statu:[0,7,9,11,26],stavang:[6,21],std:[0,4,6,26,27,29],steep:13,step:[0,1,2,4,6,7,9,10,11,12,13,14,20,21,26],step_fn:[7,12],step_length:13,steps_list:9,stereo:3,still:[0,2,3,5,6,11,13,17,23,29],stimuli:12,stk2100:[25,26],stk3155:[15,21,22,24],stk4021:[25,26],stk4051:[25,26],stk4155:[22,24],stk5000:25,stk:[25,26],stochast:[0,1,5,6,8,11,12,15,16,21,26,28,29],stock:4,stoke:12,stone:[0,7,21],stop:[1,4,9,11,13,14],stopiter:[0,7],storag:[5,27],store:[0,1,2,3,6,11,13,21,23,26],storehaug:[24,26],str:[1,3,4],straight:[0,6,8,13,26,29],straightforward:[0,2,3,5,6,8,9,10,13,20,26,27,28,29],strategi:[0,1,9,26],stratifi:[6,29],strength:[0,5,14,27],stretch:11,strict:[8,13],strictli:[8,13],stride:[4,20],strike:6,string:1,stroke:7,strong:[3,6,9,10,12,20,23,29],strongli:[0,8,19,20],stronli:0,structur:[0,1,2,3,6,9,10,12,19,26,29],stuck:[1,13],student:[0,21,22,24,25,26],studi:[0,3,4,5,6,7,8,11,12,13,17,19,21,25,26,27,28],studier:25,style:[0,7,9,20,26],sub:[9,12],subarg:2,subdivid:[0,20,26],subfield:0,subject:[6,8,23],submit:26,subplot:[0,1,3,4,6,7,8,9,10,14,26,29],subplots_adjust:[8,23],subprogram:[20,26],subract:[0,27],subroutin:[0,26],subscript:1,subsequ:[1,4,5,6,12,20,23,27,29],subset:[1,6,9,12,13,19,26,29],subspac:[0,8,11,27],substanti:[9,10],substep:11,substitut:[3,6,12,20,29],subsubset:9,subtask:6,subtl:1,subtract:[0,4,5,6,11,13,17,20,21,23,28,29],subtre:9,subval:2,succeed:[0,4,26],success:[3,7,9,13,23],successfulli:[4,9],sudo:[0,15,19,21,26],suffer:[0,1,2,5,10,26,27,28],suffici:[1,6,8,11,13,29],suggest:[1,13,21,25],suit:[8,12],suitabl:[0,23,27],sum:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,21,23,26,27,28],sum_:[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,26,27,28,29],sum_i:[0,2,5,6,8,13,17,21,27,28,29],sum_j:6,sum_ja_:0,sum_k:[6,8,12,20],sum_logist:13,sum_m:3,sum_n:3,sum_nx_:3,summar:[5,6,9,28,29],summari:[1,3,4,10,22,28],summat:[0,3,16,27],sunni:9,superconduct:28,superfici:3,superscript:[1,12],supervis:[0,5,6,7,9,12,19,26,27,29],supplement:[7,21],support:[0,1,9,10,11,13,19,26,27],suppos:[0,5,6,7,8,10,11,12,13,20,26,27,28,29],suppress:[5,8,13,28],sure:[0,1,4,6,21],surf:[6,21],surfac:[0,6,21,26],surpass:6,surpris:[0,26],surround:[3,19],survei:[0,5,6,26,27,28,29],svc:[8,9,10],svd:[0,6,11,17,26,29],svdinv:[5,28],svm:[8,9,10,11],svm_clf:[8,10],swath:[5,27],sy:13,symbol:[1,3,4,5,11,13,14,19,23,26,27,28],symmeteri:1,symmetr:[0,5,8,11,12,13,20,26,27],symmetri:[6,9],sympi:[0,15,19,21,26],synonim:23,syntax:[1,13],syntaxerror:[1,8],system:[0,1,3,4,6,7,9,10,12,13,15,19,20,21,26],systemat:[4,6,29],t0:[3,6,13],t1:[2,13],t2:2,t3:2,t:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,23,24,26,29],t_0:[2,9,13],t_1:13,t_2:[],t_:2,t_b:10,t_i:[1,2,5,12,27],t_j:12,t_k:9,tabl:[9,21,23,24,26],tabul:[0,26],tabular:26,tackl:4,tag:[2,3,4,5,6,7,12,13,14,20,23,27],taht:[0,26],tail:23,tailor:[2,8,11,26],taiwan:[0,26],take:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,16,19,20,23,26,27,28,29],taken:[0,1,3,6,10,13,20,26,29],tan:3,tangent:[1,4,12,13],tanh:[1,4,7,8,12],target:[0,1,2,3,4,5,6,7,8,9,10,11,12,26,27,28,29],target_iscomplex:2,target_meta:2,target_nam:9,target_shap:2,task:[0,1,3,6,9,11,12,14,21,26,29],tau:[3,5,23,28,29],taught:26,tax:0,taylor:[2,13],taylornr:13,tc:8,td:[1,6,26,29],teach:26,team:1,teaser:0,technic:[0,5,6,13,15,21,26,27],techniqu:[0,1,8,10,13,19,23,25,26,27,29],technolog:[0,1],tell:[0,4,6,10,11,13,23,29],temp1:1,temp2:1,temp:1,temperatur:[0,9,26],temporarili:1,ten:[3,17,26],tend:[3,5,6,8,9,10,12,13,14,27,28,29],tendenc:[0,26],tension:[6,29],tensor:3,tensorflow:[0,2,4,8,14,15,19,20,21,22,25,26,27],term1:[5,6,11,21,27],term2:[5,6,11,21,27],term3:[5,6,11,21,27],term4:[5,6,11,21,27],term:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,21,23,26,27,28],termin:[0,4,5,9,10,13,15,26,27],terrain1:[6,21],terrain:[6,21],test:[3,4,5,6,7,8,9,10,13,17,20,21,23,29],test_acc:3,test_accuraci:[1,3],test_error:6,test_imag:[3,4],test_ind:[6,29],test_input:4,test_label:[3,4],test_loss:3,test_pr:1,test_predict:1,test_rnn:4,test_scor:[7,10],test_siz:[0,1,3,5,6,10,26,27,28,29],test_split:9,tester:28,testerror:[0,6,27,29],testi:4,testpredict:4,testx:4,text:[0,1,2,4,5,8,9,11,13,15,20,23,25,27,29],textbook:[16,17,21,27,29],textual:9,textur:[1,9],tf:[1,3,4,13,14],tf_loadlibrari:[3,4,14],th:[0,1,2,5,6,7,9,12,13,14,15,16,20,21,23,26,27,28,29],than:[0,1,2,3,4,5,6,7,9,10,11,12,13,17,19,23,26,27,28,29],thank:[4,6,27,28],theano:[1,19,26],thei:[0,1,2,3,4,5,6,7,8,9,11,12,13,20,21,23,26,27,28,29],them:[0,1,3,4,6,8,9,10,11,12,13,20,21,26,27,28,29],theme:[0,26],themselv:[0,23,26],thenc:[6,29],theorem:[2,6,7,27],theoret:[0,4,10,26],theori:[0,1,3,8,9,12,13,18,19,21,25,26,28],thereaft:[0,5,6,11,12,15,16,17,20,21,26,29],therebi:[0,5,7,11,26,27,28],therefor:[0,1,2,3,4,6,7,8,11,13,23,26,27,29],therein:11,thereof:[0,6,13,26,29],theta:[1,4,13,23],theta_:[1,13],theta_i:1,theta_linreg:13,theta_t:13,thi:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,19,20,21,22,23,25,27,28,29],thing:[0,1,2,4,5,7,9,23,26,28,29],think:[0,1,3,4,6,9,12,13,14,23,26,29],third:[0,3,6,13,24,26,29],thirti:7,thorughout:26,those:[0,3,5,6,8,9,10,11,20,21,22,26,27,29],though:[1,2,3,4,13,20,23],thought:[6,14,21,23,29],thousand:[0,1,21,27],three:[0,1,3,5,6,8,9,12,15,20,21,22,23,24,26,27,28,29],threshold:[1,3,9,10,11,12,13],through:[0,1,2,3,4,5,6,8,11,12,13,14,19,20,23,26,27,28,29],throughout:[0,4,5,14,19,20,23,26],thu:[0,1,2,5,6,7,8,10,11,12,13,21,24,26,27,28,29],thumb:[0,6,21,26,27],thursdai:[24,27],tibshirani:[6,18,21,22,25,26,29],tick:6,tick_param:6,ticker:[6,13,21,23],tif:[6,21],tight_layout:[1,7],tightli:11,tild:[0,5,6,7,11,15,16,17,18,21,23,26,27,28,29],till:[0,4,7,8,9,10,12,20,26,27,28],time:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,17,19,20,21,23,26,27,28,29],timeit:4,timer:4,tini:1,tip:3,titl:[0,1,2,3,4,6,7,8,9,10,13,21,23,26,29],tmp:13,tn:[2,3,7],to_categor:[1,3,4],to_categorical_numpi:1,to_numer:[0,6,26,29],todai:3,togeth:[0,3,6,8,11,13,19,26],toi:14,told:13,toler:[2,6,14],tolist:4,tomographi:12,too:[0,2,4,5,6,9,11,13,15,23,25,26,27,29],took:[8,26],tool:[0,1,3,6,13,19,27,29],toolbox:8,top:[0,3,5,6,9,10,18,19,26,28,29],topic:[0,5,6,7,8,19,21,27,28],topograph:21,topolog:[1,3,12],toposort:2,torkjellsdatt:[24,26],toss:[10,23],total:[0,1,2,3,4,6,7,8,9,10,11,12,13,14,15,16,20,23,24,26,27,29],total_loss:4,totalclustervari:14,totalscatt:14,toward:[1,2,7,12,13,21],town:0,tp:[4,7],tpng:9,tpu:[13,19,26],tqdm:6,tr:27,trace:2,trace_stack:2,traceback:[0,2,3,4,6,7,10,13,14,21,26,28,29],tracer:2,track:[3,13,14,20,27],tract:0,tractabl:[0,26,27],trade:[5,9,28,29],tradeoff:[0,5,21,26,27,28],tradit:[0,1,4,6,26,29],train:[2,3,5,6,8,9,10,11,12,13,17,21,28,29],train_accuraci:[0,1,3,26],train_dataset:4,train_end:[0,1,27],train_error:6,train_imag:[3,4],train_ind:[6,29],train_label:[3,4],train_pr:1,train_siz:[0,1,3,27],train_step:4,train_test_split:[0,1,3,5,6,7,9,10,11,26,27,28,29],train_test_split_numpi:[0,1,27],trainable_vari:4,trained_model:[6,27,28],trainerror:[0,27],traini:4,training_checkpoint:4,training_dataset:4,training_gradi:13,trainingerror:[6,29],trainpredict:4,trainscor:4,trainx:4,trait:[0,26],trajectori:4,transfer:9,transform:[0,5,6,7,8,9,10,11,12,13,19,20,26,27,28,29],transit:[6,12],translat:[1,4,6,10,26,27,28],translate_vjp:2,transpos:[1,5,11,20,27,28],travers:[0,5],treat:[0,1,3,6,12,13,23,26,27,28,29],tree:[0,1,6,19,21,26],tree_clf:[9,10],tree_clf_:9,tree_clf_sr:9,tree_reg1:9,tree_reg2:9,tree_reg:9,trend:23,treue:7,trevor:[18,21,25],tri:[2,3,4,9,13],triain:0,trial:[0,2,4,6,13,23,26,29],triangl:13,triangular:20,trick:[3,4,8,11,13,23],trickier:23,tridiagon:20,trigger:[0,7],trillion:19,trivial:[0,1,5,11,23,26,28],troubl:[0,8,12,27],truck:3,true_beta:[6,27,28],true_divid:1,true_fun:[6,29],trust:9,tucker:8,tuesdai:[24,26,29],tumor:[7,9],tumour:7,tunabl:1,tune:[4,9,13,20,26],tupl:2,turbul:[],turn:[0,1,5,6,7,8,9,10,11,12,13,20,21,23,26,27,29],tutori:[1,4],tv:2,tveito:2,tweak:[1,4,10,23],twice:13,twist:11,two:[0,1,2,4,5,6,7,9,10,11,12,13,15,16,20,22,23,25,26,28,29],tx:13,tx_1:13,txt:4,ty:13,type:[0,1,3,6,8,10,13,16,20,21,23,27,28,29],typeerror:[2,6,13,21],typic:[0,1,2,3,4,5,7,9,10,12,13,23,26,27,28,29],typo:21,u:[0,2,5,6,8,10,11,12,17,20,26,27,28],u_:20,u_i:12,u_m:10,u_t:[],ua:[0,26],ubuntu:[0,15,19,21,26],uci:0,uio:[21,24,25],un:14,unabl:9,unari:[20,26],unary_f:2,unary_oper:2,unary_to_nari:2,unbalanc:[6,9,29],unbias:[0,5,6,26,28,29],unbroadcast:2,unbroadcast_f:2,uncent:[6,28],uncertainti:[0,5,26,28,29],uncertitud:23,unchang:[1,3],uncorrel:[10,23],undefin:[5,27],under:[0,1,5,6,10,13,15,16,19,21,26,27,28,29],underdetermin:[0,26],underfit:[1,6,29],underflowproblem:[5,28,29],undergo:[5,28,29],undergradu:[22,24],underli:[0,1,9,13,23,26],underset:[4,14],understand:[0,1,3,5,6,10,13,14,19,26,27,28],understood:[8,13],undesir:8,undetermin:[5,8,28,29],undo:4,unexpect:[6,13,21,29],unexpected:23,unfair:[6,27,28],unfortun:[1,8,9,10],unicode_liter:[8,9],uniform:[0,1,5,6,11,13,15,16,21,23,26,27],uniformli:[13,23],unifrompdf:23,unimport:13,union:[5,6,28,29],uniqu:[0,2,6,13,14,20,26,29],unique_cluster_label:14,unit:[0,1,3,4,5,10,12,23,26,27,28],unitari:[5,6,20,27],unitarili:[20,26],uniti:23,univari:23,univers:[0,1,2,13,15,19,21,22,24,26,27,28,29],unix:1,unknow:[0,20,26],unknown:[0,1,3,4,5,6,8,10,13,16,20,26,27,28,29],unknowwn:12,unlabel:1,unless:[0,3,6,11,13,21,26,29],unlik:[1,3,8,13],unnecessarili:9,unord:3,unravel:1,unrol:[3,11],unseen:[0,7,9],unstabl:1,unsupervis:[0,1,4,12,19,26],unsymmetr:[20,26],until:[1,2,4,9,12,13,14],untouch:0,unusu:12,up:[1,3,4,5,6,8,10,11,13,14,16,18,19,20,21,23,24,28],updat:[1,2,10,12,13,14],uploa:26,upload:[19,21,25],uploda:[],upon:[0,1,6,7,11,20,21],upper:[0,8,9,16,20,27],uppercas:[20,26],upsampl:4,upscal:4,url:[0,7,26,27],us:[4,5,6,8,9,10,11,12,14,15,16,17,18,20,22,23,25,28,29],usa:26,usag:[0,8,19,26,27],usd10000:0,usd:0,use_bia:4,usecol:[0,26],useless:1,user:[0,1,2,3,4,6,7,8,9,11,14,15,19,20,21,26,27,28],usernam:21,userwarn:6,usetex:23,usg:[6,21],usr:23,usual:[0,3,4,7,12,13,14,26],ut:[5,28],utf:[0,7],util:[1,3,4,6,7,10,14,26,29],ux:20,v0:23,v1:23,v2:23,v:[2,4,5,6,11,13,17,19,27,28],v_0:11,va:1,vahid:26,val:13,val_accuraci:3,val_loss:4,vale:2,valid:[0,1,4,7,9,10,13,19,23,26,27],validation_data:3,validation_split:4,valu:[0,1,2,3,4,6,7,8,9,10,12,13,14,15,16,19,20,21,26],valuat:9,valueerror:2,valy:4,van:[0,18,21,26,27,28,29],vandenbergh:[8,13],vandermond:[0,26],vanilla:[0,6,11,14,27,28],vanish:[1,4,13,23],var_x:23,varabl:8,varepsilon:[5,6,18,21,28,29],varepsilon_:[5,6,28,29],varepsilon_i:[5,6,28,29],vari:[0,1,3,5,6,10,15,16,26,29],variabl:[0,1,2,5,6,7,8,10,11,12,13,14,20,26,27,28,29],varianc:[0,1,5,7,9,10,11,13,14,18,19,20,23,26,27],variance_i:[5,11,27],variance_x:[5,11,27],variant:[0,1,6,8,12,13,26,27],variat:[3,4,11,26],varieti:[0,3,12,15,19,21,26],variou:[1,3,5,6,7,8,9,11,12,13,16,18,19,20,21,23,27,28],varydimens:4,vastli:3,vaue:1,vault:0,vdot:[2,13],ve:21,vec:[6,29],vector:[0,1,2,3,4,5,6,7,9,10,11,13,14,16,17,19,28,29],vector_mean:14,veloc:[],ventur:[0,8,15,19,26],verbos:[1,3,4],veri:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,21,23,25,26,27,28,29],verifi:[3,11,20,26],versatil:[8,26],versicolor:[8,9],version:[0,3,10,13,14,15,19,20,21,23,26],versu:1,vert:[0,1,5,6,7,8,9,11,13,17,26,27,28,29],vert_1:[5,6,27,28,29],vert_2:[5,6,11,17,27,28,29],via:[0,5,6,7,8,9,10,11,12,15,19,20,21,22,23,24,26,27,28,29],vidal:11,video:[0,1,12,19,22,24,26,27,28,29],view:[1,3,5,6,12,13,23,25,26,28,29],violat:8,virginica:9,viridi:[0,1,2,3,26],virtual:1,viscos:13,viscou:13,visibledeprecationwarn:2,vision:[0,3],visual:[0,3,11,12,15,19,26,27],visualis:1,viz:[6,8,23],vjp:2,vjp_argnum:2,vjpfun:2,vjpmaker:2,vjpnode:2,vjps_dict:2,vmap:13,vmax:[1,6],vmin:[1,6],voic:3,volum:[0,3,26],vote:10,voting_clf:10,votingclassifi:10,votingsimpl:10,vs:[0,4,6,29],vspace:2,vstack:[5,11,20,23,26,27],vt:[5,27,28],w1:8,w2:[8,11],w3:8,w:[0,1,2,3,4,5,6,7,8,10,11,12,13,14,20,23,26,27,29],w_1:[8,20],w_1x_1:8,w_1x_:8,w_2:[8,20],w_2x_2:8,w_2x_:8,w_3:20,w_4:20,w_:[1,12],w_hidden:2,w_i:[1,2,10],w_ix_i:12,w_j:20,w_m:20,w_output:2,w_px_:8,w_px_p:8,wa:[0,1,3,4,5,6,7,10,11,12,14,17,20,21,26,27,28,29],wai:[0,1,2,3,4,5,6,7,8,10,11,12,13,14,17,20,23,26,27,28],walk:9,walker:23,wang:[0,26],want:[0,1,2,3,4,5,6,8,9,10,11,12,13,14,15,16,17,19,21,23,26,27,28,29],warn:[1,4,8],warrant:[6,29],wast:3,watch:19,wave:3,wavelet:8,we:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,27,28,29],weak:[9,10,14],weather:[1,12],web:[19,22,24,26],weblink:[],webpag:26,websit:[6,20,21,22,26],wedg:[8,23],wednesdai:[24,26,29],wee:11,week:[0,5,6,7,21,22,24],weekli:[16,19,21,24,25,26],weight:[0,1,2,3,6,7,9,10,12,13,21,23],weigth:2,welcom:[8,19],well:[0,1,2,3,4,5,6,7,8,9,10,12,13,15,16,19,20,21,23,25,26,27,28,29],went:8,were:[0,1,3,4,5,6,7,8,10,11,12,14,23,26,27,29],wessel:[0,18,21,26,27,28,29],what:[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,19,20,21,23],whatev:3,when:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,17,18,20,21,23,26,27,28,29],whenev:[13,23],where:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,23,24,26,27,28,29],wherea:[6,23,29],wherefrom:21,wherein:[1,12],whether:[0,3,5,7,9,21,23,26,28],which:[0,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,26,27,28,29],whichev:[1,3],white:9,whiteboard:28,who:[0,22],whole:[1,3,4,5,9,11,13,28,29],whose:[0,6,10,23,27,29],whow:[11,27],whrn:29,why:[0,1,3,6,13,21,28],wide:[0,1,3,6,7,12,15,19,20,21,26,29],widehat:[6,29],width:[0,3,8,9,26],wieringen:[0,18,21,26,27,28,29],win:10,wind:9,wing:[24,26],winther:2,wiothout:[6,27],wiscons:7,wisconsin:10,wisdom:[6,28],wise:[0,1,5,12,13,27],wish:[0,2,5,7,8,11,13,14,20,21,26,27],with_std:[0,27,29],wither:6,within:[0,2,3,4,7,9,12,13,14,23,25,26],withinclust:14,without:[0,1,5,6,8,9,11,12,13,26,27,28,29],won:[0,26],wonder:8,word:[0,1,3,4,5,6,7,14,23,26,27,28,29],work:[0,1,4,6,7,8,9,13,15,19,21,22,23,24,26,27,29],workshop:26,world:[0,8,27],worldwid:[0,26],wors:[0,1,3,4,6,26,29],worst:9,worth:9,worthi:21,would:[0,1,3,5,6,7,8,9,10,11,12,13,20,21,23,26,27,28,29],wrap:[6,20,26],wrap_util:2,write:[0,1,2,3,5,6,7,8,12,13,15,16,17,20,26,27,29],written:[0,2,3,5,11,12,13,16,19,20,21,23,26,27,28,29],wrong:[1,8],wrongli:10,wrote:[5,11,27],wrt:[2,10,13],wth:[10,13],www:[19,20,21,25,26,28],wx_1:8,x0:8,x1:[4,8,9,10,13],x1_exampl:8,x1d:8,x2:[8,9,10,13],x2d:[8,11],x2d_train:11,x2dsl:11,x3:[8,9],x4:9,x:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,26,29],x_0:[0,5,11,20,26,27,28,29],x_1:[0,2,5,6,7,8,9,10,11,13,20,23,26,27,28,29],x_2:[0,2,5,6,7,8,9,10,11,13,20,23,26,27,29],x_3:[8,20,23],x_4:20,x_:[0,2,3,5,6,8,10,11,13,14,18,20,21,23,26,27,28,29],x_center:11,x_data:1,x_data_ful:1,x_hidden:2,x_i:[0,1,2,5,6,7,8,9,10,11,12,13,14,17,20,23,26,27,28,29],x_input:2,x_ix_:[0,26],x_iy_i:8,x_j:[0,2,8,9,12,16,23,27],x_jy_j:8,x_k:[12,14,20,23,27],x_l:23,x_m:[6,12,20,23,29],x_n:[0,2,3,6,8,11,12,13,20,23,26,29],x_new:[9,10],x_offset:[6,27,28],x_output:2,x_p:[3,7,9],x_poli:9,x_poly10:9,x_pred:4,x_prev:2,x_reduc:11,x_scale:8,x_small:13,x_test:[0,1,3,5,6,7,9,10,11,26,27,28,29],x_test_own:6,x_test_scal:[0,6,7,9,10,11,27,28,29],x_tot:4,x_train:[0,1,3,4,5,6,7,9,10,11,26,27,28,29],x_train_mean:[6,27,28,29],x_train_own:6,x_train_scal:[0,6,7,9,10,11,27,28,29],x_val:1,xarrai:[19,26],xavier:1,xbkjd4ccgq:26,xbnew:13,xcode:[0,15,19,21,26],xdclassiffierconfus:10,xdclassiffierroc:10,xg_clf:10,xgb:10,xgbclassifi:10,xgboost:9,xgboot:10,xgbregressor:10,xgparam:10,xgtree:10,xi:[8,13],xi_1:8,xi_:8,xi_i:8,xinv:28,xk:8,xla:[13,19,26],xlabel:[0,1,2,3,4,5,6,7,8,9,10,13,21,23,26,28,29],xlim:[6,10,29],xm:9,xmesh:13,xnew:[0,13,26],xp:23,xpanda:[0,27],xpd:[5,11,27],xplot:0,xs:9,xscale:[0,27],xsr:9,xt_x:13,xtest:[6,29],xtick:[3,6,8,9,29],xtrain:[6,29],xu:[0,26],xx:[0,20,26],xy:[0,6,8,20,21,26],xytext:8,xz:[20,26],y1:4,y2:4,y3:4,y:[0,1,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,26,27,28,29],y_0:[0,5,11,20,26,27,28,29],y_1:[0,5,8,9,11,13,20,26,27,28,29],y_1y_1:8,y_1y_1k:8,y_1y_2:8,y_1y_2k:8,y_1y_n:8,y_1y_nk:8,y_2:[0,5,8,9,11,20,26,27],y_2y_1:8,y_2y_1k:8,y_2y_2:8,y_2y_2k:8,y_3:[0,9,20],y_4:20,y_:[0,1,5,6,10,11,20,26,27,28,29],y_data:[0,1,5,6,26,27,28,29],y_data_ful:1,y_decis:8,y_fit:[0,27],y_i:[0,1,5,6,7,8,9,10,11,12,13,15,16,17,18,20,21,26,27,28,29],y_if_:10,y_ix_:[0,26],y_ix_i:[7,8,13,27],y_iy_jk:8,y_j:[6,8,12,21,29],y_k:12,y_m:20,y_model:[0,4,5,6,26,27,28,29],y_n:[8,13],y_ny_1:8,y_ny_1k:8,y_ny_2:8,y_ny_2k:8,y_ny_n:8,y_ny_nk:8,y_offset:[6,27,28],y_plot:9,y_pred1:9,y_pred2:9,y_pred:[0,1,4,6,7,8,9,10,27,28,29],y_pred_rf:10,y_pred_tre:10,y_proba:[7,10],y_scaler:[6,28,29],y_test:[0,1,3,4,5,6,7,9,10,11,26,27,28,29],y_test_onehot:1,y_test_predict:0,y_test_scal:29,y_tot:4,y_train:[0,1,3,4,5,6,7,9,10,11,26,27,28,29],y_train_mean:[6,27,28],y_train_onehot:1,y_train_predict:0,y_train_scal:[6,28,29],y_val:1,yadav:[],ye:[3,6,7,29],year:[0,19,26],yet:[0,1,6,8,11,13,26],yi:13,yield:[0,2,5,6,7,8,10,12,13,14,20,23,26,28,29],yk:8,ylabel:[0,1,2,3,4,5,6,7,8,9,10,13,21,23,26,28,29],ylim:[3,6,29],ym:9,ymesh:13,yn:0,yo:[8,9,10],yoshua:[1,25],you:[0,1,2,3,4,5,6,8,9,10,11,13,15,16,17,19,20,21,23,24,25,26,27,28,29],young:0,your:[1,2,4,5,6,8,11,13,17,19,20,21,28,29],yourself:[11,13,26],youtub:19,ypred:[6,29],ypredict2:13,ypredict:[0,13,26,27],ypredictlasso:[5,28],ypredictol:[0,5,28,29],ypredictown:[6,27,28],ypredictownridg:[6,28],ypredictridg:[0,5,6,28,29],ypredictskl:[6,27,28],yridg:26,ys:9,ytest:[6,29],ytick:[3,6,8,9,29],ytild:[0,6,26,27,29],ytilde_test_ol:29,ytilde_test_ridg:29,ytildelasso:[5,28],ytildenp:[0,26,27],ytildeol:[0,5,28],ytildeownridg:[6,28],ytilderidg:[5,6,28],ytrain:[6,29],yuxi:26,yx:[20,26],yy:[20,26],yz:[20,26],z:[0,1,2,3,4,5,6,7,8,9,11,12,13,20,21,23,26,27,29],z_0:[20,26],z_1:[20,26],z_2:[20,26],z_:[1,2,12,20,26],z_c:1,z_h:1,z_hidden:2,z_i:[1,12],z_j:[1,12],z_k:[12,27],z_m:1,z_mod:9,z_o:1,z_output:2,zaman:23,zaxi:[6,21],zero:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,18,20,21,23,26,27,28,29],zeros_lik:4,zeroth:27,zfill:4,zip:[2,4,6],zm_h:[0,26],zn:0,zone:0,zoom:26,zx:[20,26],zy:[20,26],zz:[20,26]},titles:["3. Linear Regression","14. Building a Feed Forward Neural Network","15. Solving Differential Equations with Deep Learning","16. Convolutional Neural Networks","17. Recurrent neural networks: Overarching view","4. Ridge and Lasso Regression","5. Resampling Methods","6. Logistic Regression","8. Support Vector Machines, overarching aims","9. Decision trees, overarching aims","10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods","11. Basic ideas of the Principal Component Analysis (PCA)","13. Neural networks","7. Optimization, the central part of any Machine Learning algortithm","12. Clustering and Unsupervised Learning","Exercises week 34","Exercises week 35","Exercises week 36","Exercises week 37","Applied Data Analysis and Machine Learning","2. Linear Algebra, Handling of Arrays and more Python Features","Project 1 on Machine Learning, deadline October 7 (midnight), 2024","Teaching schedule with links to material","1. Elements of Probability Theory and Statistical Data Analysis","Teachers and Grading","Textbooks","Week 34: Introduction to the course, Logistics and Practicalities","Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression","Week 36: Linear Regression and Statistical interpretations","Week 37: Statistical interpretations and Resampling Methods"],titleterms:{"1":[0,15,16,17,18,21,26,27],"14":29,"17":[],"18":[],"2":[0,15,16,17,18,26,27,28],"2023":24,"2024":21,"26":27,"3":[0,15,16,26,27],"34":[15,26],"35":[16,27],"36":[17,28],"37":[18,29],"4":[0,27],"5":0,"7":21,"9":[],"case":[8,10,23,27,28],"do":[1,28,29],"final":[12,27,28],"function":[0,1,6,7,8,10,11,12,13,21,23,26,27,28,29],"import":[5,20,26,27,28],"new":[4,28,29],A:[0,1,4,8,9,26,28,29],And:[26,27,28],For:27,In:24,Ising:6,The:[0,1,2,3,5,6,7,8,9,11,12,17,19,26,27,28,29],To:[26,27],With:[4,28],about:[26,27],abov:28,activ:[1,12,28],ad:[0,6,17,21,26,27],adaboost:10,adagrad:13,adam:13,adapt:10,adjust:1,adversari:4,again:[3,9],ai:[21,26],aim:[8,9,17,18,26],aka:[26,27],algebra:[20,26],algorithm:[9,10,11,12,27],algortithm:13,all:8,altern:[],an:[0,4,10,26],analys:[5,27],analysi:[0,5,6,11,19,21,23,26,27,28,29],analyt:[0,16,17],analyz:[],ani:13,anoth:[9,28,29],appli:19,approach:[0,8,14,26,29],approxim:12,architectur:1,arrai:[20,26],assist:24,assumpt:[28,29],august:27,autocorrel:23,autograd:[2,13],automat:13,b:[17,21],back:[1,11,12],background:[19,21,29],bag:10,base:[13,29],basic:[0,5,7,9,10,11,20,27,28,29],batch:1,bay:[5,28,29],befor:11,beta:[28,29],better:8,bia:[6,21,29],binari:1,bind:26,bird:10,boldsymbol:[27,28,29],boost:10,bootstrap:[6,10,29],boston:0,breast:1,brief:[26,29],bring:12,build:[1,3,9],c:[21,26],calcul:27,can:[26,29],cancer:[1,7,9,11],cart:9,center:27,central:[13,19,23,29],chain:12,chang:10,channel:26,chi:[0,26],choos:1,cifar01:3,classic:11,classif:[1,9,10],classifi:8,clip:1,cluster:14,cnn:3,code:[0,1,2,5,9,11,12,13,14,26,27,28,29],collect:[1,3],commun:26,compar:[2,10],comparison:28,complet:27,complex:[0,6,21,27],complic:6,compon:11,comput:9,computation:29,computerlab:26,con:9,concept:23,condit:[28,29],confid:29,conjug:13,contn:26,convex:[8,13],convolut:[3,12],correctli:[28,29],correl:[11,27],cost:[1,10,27,28,29],cours:[19,25,26],covari:[5,11,23,27],cover:26,critic:[],cross:[6,21,29],cython:26,d:21,data:[0,1,3,6,7,9,11,15,16,19,21,23,26,27,28],dataset:[1,3],deadlin:[21,26],decai:2,decemb:[],decis:[9,10],decomposit:[5,11,17,20,27],deep:[1,2,26],defin:[1,26],degre:[0,27],deliveri:21,delta:29,dens:[0,26],deriv:[5,12,27,28,29],descent:[2,10,13],descript:21,design:27,detail:[3,26],develop:1,diagon:11,differ:8,differenti:[2,13],diffus:2,dimension:[2,3,8,21,27],disadvantag:9,discret:23,distribut:[5,23,28,29],distrubut:29,doe:[27,28],domain:23,down:1,dropout:1,e:21,economi:27,electron:21,element:[0,23,26],elimin:20,energi:26,ensembl:10,entropi:9,environ:[0,15,26],equat:[0,2,12,26,27,28],error:[0,10,26,27,29],essenti:26,estim:[28,29],etc:26,euler:2,evalu:1,exampl:[0,1,2,3,4,6,7,8,9,10,26,27,28,29],exercis:[0,6,15,16,17,18,26,27,29],expect:[18,23,28,29],expens:29,experi:23,explor:[0,15,16,26],exponenti:2,express:[17,18,27],extrapol:4,extrem:[10,26],ey:10,f:21,fall:24,famili:[1,26,27],famou:20,fantast:27,featur:[9,20,27],feed:[1,12],find:29,fine:1,first:[4,12,26,27,28],fit:[0,10,26,28],fix:27,fold:29,forc:3,forest:10,format:[21,26],forward:[1,2,12],fourier:3,frank:[6,21,27],freedom:[0,27],frequent:27,frequentist:[0,26],from:[5,10,12,27,28,29],full:2,further:[3,5,27],g:21,gan:4,gaussian:20,gd:13,gener:[4,9,26],geometr:11,gini:9,good:[0,26],grade:[24,26],gradient:[1,2,10,13],growth:2,ha:19,handl:[20,26,27],happen:[28,29],hessian:27,hidden:2,histogram:29,hous:0,hyperparamet:1,hyperplan:8,i:1,id3:9,idea:11,ident:[28,29],identifi:29,ii:26,iid:[28,29],illustr:28,implement:1,implic:[5,27],improv:1,includ:13,increment:11,independ:[28,29],index:9,inform:24,input:2,instal:[19,21,26],instructor:24,intercept:27,interpret:[5,11,26,27,28,29],interv:29,introduc:[11,13,27],introduct:[0,6,19,20,21,26],invers:[5,20,28],invert:27,iter:10,its:27,jacobian:27,jax:13,julia:26,jungl:10,k:29,kera:[1,3],kernel:[8,11],lagrangian:8,lasso:[5,6,21,27,28,29],last:[27,29],later:[5,27],layer:[1,2,3,12],learn:[0,1,2,11,13,14,15,16,19,21,26,27,28,29],least:[5,6,18,21,26,27,28],lectur:[26,27,28,29],level:10,librari:[19,26],likelihood:[7,28,29],limit:[1,13,23,29],linear:[0,8,13,20,26,27,28],link:[5,11,22,25,27,28,29],literatur:21,logist:[7,26],loss:27,lu:20,machin:[0,8,13,19,21,26],made:[28,29],main:[23,26],make:[0,9,10,15,16,26,27],mani:[10,12],manipul:27,margin:[28,29],mass:26,materi:[21,22,26,27,28,29],math:[5,27],mathemat:[3,5,8,27],matric:[5,20,26,28],matrix:[1,5,11,12,20,26,27,28],matter:[0,26],max:27,maximum:[28,29],mean:[0,27,28],meet:[5,10,23,26,27],mercer:8,method:[6,9,10,13,21,26,29],midnight:21,min:27,minim:26,ml:26,mle:[28,29],mlp:12,mnist:[3,4],model:[0,1,4,6,12,26],momentum:13,mondai:[27,28],moon:[8,9],more:[3,6,20,21,26,27,28,29],multilay:12,multipl:[1,3],multipli:8,need:[21,26],network:[1,2,3,4,7,12,26],neural:[1,2,3,4,7,12,26],non:8,normal:[0,1,29],notat:12,note:[21,27,28,29],novemb:[],now:[1,9,13,28,29],nuclear:[0,26],numba:26,number:[0,2,23,27],numer:[2,21,23],numpi:[20,26],object:3,obtain:11,octob:21,od:2,off:[6,21],ol:[5,6,21,28,29],one:[2,12],oper:20,optim:[1,8,13,19,26,27],order:13,ordinari:[5,6,18,21,26,27,28],organ:[0,26],oslo:25,other:[4,9,11,12,20,21,26,27],our:[0,4,5,11,13,26,27],outcom:[19,26],output:2,overarch:[0,4,8,9,17,18,26,27],overview:[10,26],own:[0,10,11,15,16,26,27],packag:[20,26],panda:[26,27],paper:21,paramet:[26,27],part:[13,19,21,27],partial:2,pass:1,path:[],pca:11,pdf:23,pencil:21,perceptron:12,perform:[1,9],period:3,perspect:1,plan:[27,28,29],plot:29,point:4,poisson:2,polynomi:[3,28],popul:2,popular:26,practic:[13,24,26],pre:[1,3],preambl:21,predict:4,preprocess:27,prerequisit:[3,19,26],princip:11,principl:3,pro:9,probabl:[5,23,28,29],problem:[1,2,13,26,27,28],procedur:[9,26],process:[1,3],program:[2,13,21],project:[6,21,26],prop:13,propag:[1,12],properti:[5,23,27],python:[0,9,15,19,20,26],quick:8,r:26,random:[10,11,23],read:[9,26,27,29],real:[6,21,26],recommend:[26,27],rectangular:28,recurr:[4,12],reduc:[0,27],reduct:3,refer:21,reformul:2,regress:[0,5,6,7,9,10,13,17,18,21,26,27,28,29],regular:1,relat:27,relev:[25,27,29],relu:1,remark:3,remind:[6,8,26],repeat:27,replac:13,report:21,repositori:29,requir:[2,19],resampl:[6,21,29],rescal:[6,28],residu:27,resourc:2,result:[27,28],revisit:13,rewrit:[26,27,29],rgeress:[],ridg:[0,5,6,17,18,21,27,28,29],rm:13,rule:12,s:[8,10,29],same:[13,29],sampl:11,scale:[27,29],schedul:[22,26],schemat:9,scheme:2,scienc:26,scikit:[0,1,11,15,16,26,27,28,29],second:13,semest:24,septemb:[28,29],session:[27,28],set:[0,2,3,9,12,15,26,27],sgd:13,should:1,similar:13,simpl:[0,4,9,13,26,27,28],singl:10,singular:[5,11,17,27],size:27,soft:8,softmax:1,softwar:[21,26],solv:[2,28],solver:13,some:[13,20,27],specifi:2,split:[0,15,16,26,27],squar:[0,5,6,10,18,21,26,27,28],standard:[13,27,29],state:[0,26],statist:[5,6,19,23,26,28,29],steepest:[10,13],step:29,still:27,stochast:[13,23],strongli:26,structur:[],subtract:27,suggest:26,sum:29,summari:[24,26],superposit:3,supervis:1,support:8,svd:[5,27,28],systemat:3,t:[27,28],teach:[22,24],teacher:[24,26],technic:28,techniqu:[6,11,21],technolog:19,tensorflow:[1,3],tent:26,term:29,test:[0,1,15,16,26,27,28],text:26,textbook:[25,26],theorem:[5,8,11,12,23,28,29],theori:23,thi:[17,18,26],think:27,thursdai:29,tip:13,togeth:12,tool:[21,26],top:1,topic:26,toward:11,trade:[6,21],tradeoff:[6,29],train:[0,1,4,15,16,26,27],transform:3,tree:[9,10],tuesdai:28,tune:1,two:[3,8,19,21,27],type:[2,4,12,26],uio:26,understand:29,univers:[12,25],unsupervis:14,up:[0,2,9,12,15,26,27,29],us:[0,1,2,3,7,13,19,21,26,27],usag:[28,29],valid:[6,21,29],valu:[5,11,17,18,23,27,28,29],variabl:23,varianc:[6,21,28,29],variou:[0,15,26,29],vector:[8,12,20,26,27],view:[0,4,10,27],visual:[1,9],vs:3,wai:[9,29],wave:2,we:26,wednesdai:28,week:[15,16,17,18,26,27,28,29],weekli:[22,29],what:[0,26,27,28,29],which:1,why:[26,27,29],wisconsin:7,wrap:[27,29],write:[4,11,21,28],x:[27,28],xgboost:10,yet:28,your:[0,10,15,16,26,27],yourself:[]}})
\ No newline at end of file
+Search.setIndex({docnames:["chapter1","chapter10","chapter11","chapter12","chapter13","chapter2","chapter3","chapter4","chapter5","chapter6","chapter7","chapter8","chapter9","chapteroptimization","clustering","exercisesweek34","exercisesweek35","exercisesweek36","exercisesweek37","intro","linalg","project1","schedule","statistics","teachers","textbooks","week34","week35","week36","week37"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":4,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":3,"sphinx.domains.rst":2,"sphinx.domains.std":2,"sphinx.ext.intersphinx":1,sphinx:56},filenames:["chapter1.ipynb","chapter10.ipynb","chapter11.ipynb","chapter12.ipynb","chapter13.ipynb","chapter2.ipynb","chapter3.ipynb","chapter4.ipynb","chapter5.ipynb","chapter6.ipynb","chapter7.ipynb","chapter8.ipynb","chapter9.ipynb","chapteroptimization.ipynb","clustering.ipynb","exercisesweek34.ipynb","exercisesweek35.ipynb","exercisesweek36.ipynb","exercisesweek37.ipynb","intro.md","linalg.ipynb","project1.ipynb","schedule.md","statistics.ipynb","teachers.md","textbooks.md","week34.ipynb","week35.ipynb","week36.ipynb","week37.ipynb"],objects:{},objnames:{},objtypes:{},terms:{"0":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,24,26,27,28,29],"00":[0,1,5,6,9,11,26,27,28],"000":[1,3,9],"0000":9,"00000":[9,11],"000000":[5,11,27],"00000000e":[5,27],"0001":[1,17],"00019961646575285301":9,"00019998":5,"00024087":5,"00029012":5,"00031535148309577417":[6,28],"0003153514830958095":[6,28],"00034944":5,"000417932":2,"00042089":5,"000464088":2,"00050694":5,"00058016":[6,28],"00060704":6,"00061058":5,"00062592":6,"00063362":29,"00063866":29,"00066668":[6,28],"00068091":29,"00068734":[6,28],"00069123":6,"00073541":5,"00075590":29,"00076495":[6,28],"00076908":6,"0007698473260556325":[6,28],"0007698473260556343":[6,28],"00079125":6,"00079904":29,"00079968":5,"00083355":29,"00084707":6,"00085892":6,"00087697":[6,28],"00088573":5,"00089193":29,"00092678":6,"00092898":29,"00096314":5,"001":[1,2,8,13,17,28],"00100519":[6,28],"0010479245693919886":[6,29],"00105081":[6,28],"00106677":5,"00107405":[6,28],"00111756":[6,28],"0011526":[6,28],"00115999":5,"00118584":6,"00119688":29,"00128479":5,"001323":[6,28],"00137814":6,"00139705":5,"00140846":29,"00149311":[6,28],"00149956":[6,28],"00152117":[6,28],"00154733":5,"00154853":29,"00156374":6,"00168251":5,"00174276":[6,28],"00175331":[6,28],"00186347":5,"001880":5,"00200":8,"00202624":5,"00202756":[6,28],"00217499":[6,28],"00219195":29,"00224413":5,"00228740":6,"0023548":[6,28],"002381316302584885":[6,28],"0023813163025848865":[6,28],"00242954":29,"00242999":6,"00243186":[6,28],"0024401":5,"00249435":[6,28],"00266858":2,"00270244":5,"00274989":[6,28],"00289724":[6,28],"00293838":5,"00310113":2,"00312361":[6,28],"00315593":6,"0032153180657605116":[6,29],"00323332":[6,28],"0032542":5,"00325450":29,"003301":[6,28],"0033955154592040936":[6,29],"00353823":5,"0036237":[6,28],"0036367":[6,28],"00369758":[6,28],"0038335":[6,28],"003909404072811226":[6,29],"00391839":5,"0039987":[6,28],"004":[5,28,29],"004091940707753939":[6,29],"00410387":[6,28],"00410478":6,"004113634617443131":[6,27,28],"00411363461744314":[6,27,28],"004113634617443147":[6,27,28],"00415598634586135":[6,29],"00420072":29,"00424909":2,"00424967":[6,28],"00426027":5,"00433417":11,"00440346":[6,28],"00443743":[6,28],"00445655":11,"00445922":26,"004579219539673834":[6,29],"00458878":[6,28],"004610275230656212":[6,29],"00462287":[6,28],"00465099":29,"00471782":5,"00472199":6,"00472512":[6,28],"0049544":[6,28],"00497837":28,"004999999999999991":0,"00512927":5,"00517114":[6,28],"00526348":[6,28],"0053018":[6,28],"00554552":[6,28],"00556826":[6,28],"0056799":5,"00579953":[6,28],"00588657":[6,28],"00607783":6,"006162":[6,28],"00617499":5,"00630331":[6,28],"00642221":[6,28],"0064472":28,"00660427":[6,28],"00672607":[6,28],"00673407":[6,28],"00676387":[6,28],"0068011":[6,28],"00683748":5,"00683964":[6,28],"007024126888938232":[6,29],"00719176":[6,28],"00727646693":[0,26],"007315":27,"0074331":5,"00759119":6,"00779281":23,"00781918":29,"00784393":[6,28],"00803064":[6,28],"00805074":29,"00813803":6,"00817631":[6,28],"00823002":5,"00827728":[6,28],"00831018":[6,28],"00834567":[6,28],"00848904":[6,28],"0086649156":[0,26],"008675369724975977":5,"008831941890485846":27,"00890669":20,"00894639":5,"00902152e":27,"00905423":[6,28],"009163470508352218":5,"009164545680330616":[6,29],"00917248":[6,28],"00934327e":27,"00934499":[6,28],"009522910538005715":27,"0096208":[6,28],"00987022":26,"00990475":5,"00992331":[6,28],"00996754":[6,28],"00it":6,"01":[0,1,2,5,6,9,11,13,17,25,26,27,28],"010018312644137363":[6,29],"0100706":[6,28],"010331721306655127":[6,29],"010516485576645508":[6,29],"01066519":[6,28],"01076611":5,"0110":23,"011076219011379514":[6,29],"011225":2,"011265488125148788":27,"0113104":[6,28],"01173838":0,"01179792":[6,28],"01191824":5,"012073649446193166":[6,29],"01219292":[6,28],"01222822":27,"01223198":[6,28],"01231917":[6,28],"01265755":27,"012658":27,"01267007":28,"01290947":[6,28],"01295356":5,"013121574015585152":[6,29],"013121574015585155":[6,29],"01312254":26,"01318643":[6,28],"01347916":[6,28],"01348565":[6,28],"01367553":[6,28],"01384313":26,"01397146":[6,28],"01405935":[6,28],"01416528":[6,28],"01433809":5,"014436800088904942":[6,29],"01449782":[6,28],"01458337":[6,28],"014586":27,"01458611":27,"0146081":[6,28],"01463049":[6,28],"014891664497538563":26,"01496065":26,"01507238889517706":[6,28],"01507238889517717":[6,28],"01531845":[6,28],"01542292":[],"01549377":[6,28],"01549939":[],"01550546":[],"01558197":5,"015910":11,"01591047":11,"01594452":29,"01628578269596628":[6,29],"01633913":[6,28],"01640891":6,"01655318":[6,28],"016587414993043573":[6,29],"01691985":[6,28],"0169643":5,"01708781":[6,28],"01708852":[6,28],"01713366":[6,28],"01724499":5,"017355848195593347":[6,29],"01735584819559336":[6,29],"017665":5,"01783414e":27,"01813007e":27,"018232":27,"01831050e":[6,28],"01831190e":[6,28],"01846688":26,"01866537":[6,28],"01873344":11,"01873869":5,"01898855":[6,28],"01905883":[6,28],"01908936":[6,28],"01963611":[6,28],"01969145":[6,28],"01975416527185249":[6,29],"01975848":[6,28],"01it":6,"02":[0,4,6,7,12,26,27,28],"02024962":[6,28],"02054837e":[6,28],"02068067":[6,28],"02073509":5,"02077431":28,"02098261":[6,28],"02123176":[6,28],"021544346900318832":28,"021592704588025022":[6,29],"021592704588025025":[6,29],"02180258":26,"02198702e":[6,28],"02198703e":[6,28],"02208512":[6,28],"02228115":[6,28],"02229529":[6,28],"02252765":5,"022999498260366312":[6,29],"02348765":[6,28],"02365049":[6,28],"024217":11,"02421742":11,"02424794":29,"0245528":[6,28],"02492265":5,"02498832":[6,28],"02503753":[6,28],"02511518":6,"02522069":6,"02546675":29,"025709":[11,27],"02586427":[6,28],"0260906":[6,28],"02625193":8,"02629596":20,"02660572763718093":[6,29],"026605727637180936":[6,29],"02693858":28,"02707227":5,"02723445":[6,28],"027609773491022525":[6,29],"02760977349102253":[6,29],"027610":5,"02761005":5,"02857":4,"02895918":28,"029483":5,"029733":27,"02976145":[6,28],"02994311":5,"02f":[6,21],"02it":6,"03":[1,6,27,28],"03032441e":[6,28],"03077640549":4,"03099776":5,"031":[5,28,29],"031696741936652305":23,"03196357":[6,28],"03251863":5,"03256632e":1,"03267527":[6,28],"03279636":[6,28],"0330308045183194":[6,28],"0330308045187757":[6,28],"03308408":5,"0334336":26,"033657685071527665":[6,29],"03405654":11,"034057":11,"03447512":[6,28],"034557":27,"03562355":[6,28],"03568439":[6,28],"0359565":5,"03630548":[6,28],"03707133":11,"037299816041296285":9,"037813671417389005":[6,29],"03781367141738902":[6,29],"03814292":[6,28],"03815288":[6,28],"038300":[11,27],"03846169":28,"039039":5,"039967668952797":[6,28],"0399676689527975":[6,28],"04":[1,6,11,27,28],"040102":5,"04010697":[6,28],"0402":[],"04063602":[6,28],"041":9,"0411487294305088":[6,28],"0411487294305226":[6,28],"04193203":27,"04220758":[6,28],"04242885e":27,"04295757":29,"043":9,"04315108":5,"04321941e":27,"04346721":5,"04355837":[6,28],"04362":9,"04373024":26,"0437499":2,"04389027":[6,28],"04423486":[6,28],"04426744e":27,"044334":27,"044613":[6,28],"04537385":[6,28],"04543942":[6,28],"045548":11,"04554803":11,"04566964":[6,28],"0458":9,"04648335":5,"04664255":6,"04683565":5,"046842629321028326":11,"04784395":[6,28],"0481872773043029":[6,29],"04876305":26,"04892055":[6,28],"04909093":[6,28],"04912436":6,"04926746":29,"0494262":26,"04944931918420588":23,"049556996627824":[6,28],"0495569966278269":[6,28],"05":[1,4,6,21,27,28],"05009826":[6,28],"05061841":23,"05073207e":27,"05087888":11,"050879":11,"051006":11,"05100602":11,"05100875":[6,28],"05108705":26,"051418":5,"051649":[11,27],"0517473":5,"05210975":20,"052279218012056844":[6,29],"05227921801205686":[6,29],"052305":27,"05263":9,"05302":9,"053417":27,"05364854":8,"05383795":[6,28],"0538391":26,"053849":5,"05447415":[6,28],"054617":27,"054655":27,"05505310046363":2,"05519999":28,"05533":9,"055676":27,"055734":27,"055910":27,"055987":27,"05614483":5,"05623":9,"05648":9,"05651951":[6,28],"05667":9,"056683":27,"056767":5,"057076":5,"05716368155342608":[6,29],"057300":27,"057393":27,"057415":5,"057418":27,"057457":27,"057722":27,"057786":5,"057835":27,"05785343":[6,28],"05789007":[6,28],"05796251":[6,28],"05807125":[6,28],"058190":5,"058552":27,"058645":27,"05883":9,"05884":9,"058854":27,"058952":27,"058996":27,"059031":27,"059226":5,"059406":5,"059597":5,"05965043":20,"059711":5,"059797":5,"059807":27,"059951":27,"05999":9,"059993":27,"06":[6,27,28],"060003":5,"060043":5,"060083":27,"06020587":[6,28],"060254":27,"060340":5,"060422":5,"06043581":[6,28],"060612":5,"060716":27,"060756":27,"060764":5,"060864":5,"060872":27,"061076":5,"061146":5,"061163":27,"061359":27,"061367":5,"061443":27,"061452":27,"061496":5,"061614":27,"061822":5,"061882":5,"06200174":5,"062016":27,"06208238634231949":[6,29],"062242":5,"062292565":4,"062337":27,"062599":27,"062624":27,"062634":5,"062874":27,"063019":27,"063061":27,"063080":27,"063260":27,"063511":5,"063653":27,"063684":5,"063832":5,"063894":27,"063965":5,"064061":5,"064074":27,"064087":5,"064307":5,"06442194e":27,"06444":9,"064501":27,"064521":5,"064530":5,"06453579006728324":[6,29],"064606":27,"064609":27,"064699":27,"064713":5,"064793":27,"064841":5,"064889":5,"06491736":[6,28],"064933":5,"065024":5,"065077":27,"065101":5,"065115":5,"065186":5,"065273":5,"065299":5,"065304":5,"065359":5,"065361":5,"065378":27,"065390":27,"065410":27,"06547790180152355":[6,29],"06547790180152359":[6,29],"065588":5,"065633":5,"065860":5,"065930":5,"066066":27,"066204":5,"066349":5,"06637":9,"066380":5,"066395":5,"066402":5,"066406":5,"06644775":20,"066458":5,"066474":27,"066500":27,"066612":27,"066647":27,"0666807":2,"066787":27,"066837":[11,27],"066854":27,"06696769":20,"066999":27,"067097":5,"067149":5,"06719214":20,"06724062":5,"067284":5,"067392":5,"067429":5,"067482":5,"067560":5,"067637":27,"067707":27,"067729":5,"068022":5,"068264":27,"068301":5,"068307":27,"068441":27,"06844519414009445":[6,29],"06844519414009446":[6,29],"068605":11,"068629":27,"068727":27,"068906":27,"068963":5,"069028":27,"06926365":27,"069275":5,"0694169556368514":20,"069452":27,"069521":5,"069584":27,"069704":5,"069717":5,"07":[6,27,28],"070006":11,"070043":27,"070054":5,"070146":27,"07016":9,"07017":9,"070202":11,"070213":27,"070342":5,"07039":9,"070457":27,"070496":5,"070582":27,"07062318":[6,28],"070737":27,"070868":5,"070889":[11,27],"070986":27,"07115":9,"0713":[0,26],"071315":5,"07145103":11,"071462":11,"07160048164233102":[6,29],"07160048164233104":[6,29],"07172103":26,"071729":11,"072000":27,"072111":27,"072132":27,"072476":27,"072489":5,"072621":27,"072633":5,"072759":5,"07285":3,"072860":5,"072976":11,"073318":11,"073481":5,"073581":5,"073598":[11,27],"073627":5,"073634":5,"073644":27,"073934":5,"074067":[11,27],"074084":27,"074208":5,"074210":27,"07421084":5,"074323":27,"074407":5,"074548":5,"074552":11,"07456491":5,"074706":5,"07490892":[6,28],"074973":11,"075398":11,"075698":5,"075816":27,"075889":27,"075972":5,"076150":27,"076383":5,"07641937":29,"07643382":20,"0764924":[6,28],"076620":5,"076697":11,"076716":5,"07678":9,"07692307692307693":9,"076938":[11,27],"076964":5,"077082":11,"077168":27,"07745069":23,"077460":27,"077522":11,"07777777777777778":1,"078110":27,"07820":9,"078336":27,"078540":27,"07862755":23,"07864":9,"078704":5,"07871":9,"078785":11,"078824":11,"078974":27,"078986":27,"079381":11,"079412":5,"07944154":[20,26],"079578":5,"079642":5,"0796891867672603":[6,29],"0798590438380667":11,"079873":5,"079946":27,"08":[6,9,23,27],"080016":5,"080195":11,"080425":5,"08043851":5,"080616":27,"080630":11,"080764":27,"081276":27,"081308":11,"08131003":6,"081374":11,"081422":11,"08156108":[6,28],"081597":11,"08165104":27,"081655":27,"081798":11,"08251519":[6,28],"082592":11,"082619":11,"082653":27,"082661":11,"08277442":23,"0827744231113545":23,"082978":11,"08299273e":[6,28],"083096":11,"08318298e":1,"08323973e":27,"083309":11,"08333333333333333":[1,9],"08336233266":4,"083527":27,"083540":11,"08376632":[6,27,28],"083766322923899":[6,27,28],"0837663229239043":[6,27,28],"083851":11,"083906":11,"083943":11,"084022":11,"084076":27,"084157":11,"08426840630693411":[6,29],"08426840630693413":[6,29],"084362":11,"08455":9,"084574":11,"084650":11,"08474":9,"084809":27,"085152":11,"085237":11,"085259":11,"085265":11,"085346":11,"085383":11,"085431":11,"08551306":[6,28],"08557805981945521":5,"085693":11,"08576932":6,"08593216":[6,28],"086059":11,"08611111111111111":1,"086230":11,"086263":11,"086538":11,"086550":11,"086611":11,"086615":11,"086673":11,"086782":11,"086875":11,"086890":11,"08690":9,"087176":11,"08750220e":27,"08758":9,"087822":11,"087938":11,"087951":11,"088061":11,"088130":11,"088160":11,"0881981":5,"088222":11,"08823":[],"088361":11,"088402":11,"088422":11,"08860792":20,"08871404":5,"08881497884574564":27,"08888888888888889":1,"088893":11,"088908":11,"088986":11,"08902":9,"08939489":20,"0895387":27,"089539":27,"089629":11,"089654":11,"089682":11,"089702":11,"089739":11,"089892":11,"08996":9,"09":[1,6,27],"090000":11,"090025":11,"09004029":26,"090290":11,"09047191e":27,"090623":11,"090696":11,"090751":11,"091246":11,"091331":11,"091353":11,"091561":11,"091627":11,"09163947":6,"09166666666666666":1,"0917":9,"091700":11,"09221309":23,"092347":11,"09251":9,"092544":11,"092613":11,"092672":11,"09298511":23,"093078":11,"093132":11,"093429":11,"0940821989627646e":[6,28],"0940821989652176e":[6,28],"094426":11,"09444444444444444":1,"094577":11,"094659":11,"094671":11,"094970":11,"09609807":5,"09650726":26,"096803":11,"096837":11,"096928":11,"09715081560068299":26,"09744":9,"09780":9,"09791":9,"09861229":[20,26],"0988662":20,"09900923":26,"09903804":8,"09917303":20,"09919198949275869":[6,29],"099287":11,"09951287404314545":1,"09953815":20,"0n":[0,26],"0x0006":[3,4,14],"1":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,22,23,24,25,28,29],"10":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,20,21,22,23,24,26,27,28,29],"100":[0,1,2,3,4,5,6,8,9,10,11,13,14,15,16,17,20,23,24,26,27,28,29],"1000":[0,1,2,4,5,8,11,13,14,19,23,26,27],"10000":[2,5,6,10,11,13,23,28,29],"100000":8,"10001":10,"1001":[9,23],"1002":23,"1003":23,"10030":9,"10041812":28,"1005":23,"1007":29,"1007711427354898":[6,29],"1009":23,"1011":23,"1013":23,"1013904243":23,"10141413e":[6,28],"1015":23,"1015am":26,"1023":23,"10230":9,"1024":3,"1026":23,"1027":23,"103":1,"1030":23,"10340":9,"1037":23,"10378326e":1,"1038":23,"10391807":[6,28],"10398646080125032":[6,29],"10398646080125035":[6,29],"1040":23,"10405456":11,"10430":9,"104411":27,"1047":23,"105":28,"10520":9,"10555555555555556":1,"10572":29,"10589577":5,"106095":[11,27],"10694828":20,"1079":29,"10790125813226321":27,"108":[6,9],"10913":6,"10931453":[6,28],"10960":9,"10983159":11,"10th":9,"10x":[0,26],"11":[0,2,5,6,7,8,9,10,11,12,13,15,16,20,21,22,23,24,25,26,27,28,29],"1100":23,"1101":23,"11022072":11,"110221":11,"11022302e":5,"111":[1,7,12],"11100":9,"11202337":29,"112383":[11,27],"1124":9,"11368985":26,"11382896":28,"11388888888888889":1,"11390":9,"11400097":20,"114550":29,"11462415":5,"11482289e":[6,28],"11500354":20,"11507992e":1,"11547777218872497":[6,29],"115822":6,"11590":9,"11660":9,"11666666666666667":1,"11694":21,"117":8,"11744554e":[6,28],"11780":9,"118":[0,2,7],"11830757":28,"118318":27,"1184":29,"11840":9,"11890":9,"119":[0,2,7],"11938949":23,"11952661":23,"11967653":26,"119936":2,"12":[0,1,2,3,4,5,6,7,8,9,11,12,20,23,25,26,27,29],"120":[0,2,3,7],"1203":9,"1203284":8,"12044974":27,"120450":27,"121":[8,9,10],"12122786e":27,"1215pm":[24,26],"12182967":[6,28],"122":[2,8,9,10],"12222222222222222":1,"1225397":23,"123":2,"12318726e":[6,28],"12330033":27,"12333649":[6,28],"123711":6,"12380":9,"124":[0,26],"12400":9,"1242":8,"12552073e":[6,28],"12591227":27,"126":9,"1261":9,"12618549":5,"1265":9,"127":4,"12700133":20,"1271":6,"12777777777777777":1,"12790":9,"128":[3,4,13],"12871842":27,"129012":6,"12952339":28,"1297":9,"1298":9,"129963":29,"12pm":[24,26],"13":[0,2,5,6,7,9,11,12,18,20,21,23,26,27,29],"130":9,"13003291":[6,28],"130224":11,"13022421":11,"13055555555555556":1,"130694":27,"13069442":27,"131":9,"13118774":26,"13156239":20,"13179053e":27,"132":9,"13220608e":[6,28],"1325":6,"1326":9,"1326197715":26,"13280":9,"133":7,"13318759":20,"1332":29,"134":28,"135":9,"1350":29,"13535942":[6,28],"13588335":27,"136":[0,7],"13614217":20,"13633112":26,"13641587e":27,"13646574":5,"13658936":23,"13661243e":[6,28],"13679863":[6,28],"137":[0,7],"13740":9,"137652":[11,27],"138":[0,7],"1385":6,"13858317":26,"13865173":5,"13867518":26,"138775":[11,27],"1388888888888889":1,"13890":9,"139":[0,7],"1392559585020426e":[6,28],"139255958997547e":[6,28],"13929376":20,"139431112903922":[],"1394311129039245":[],"1395084586525954":[],"1395235273363669":[],"14":[0,2,4,5,6,7,8,9,10,11,12,20,21,23,25,27,29],"140":[0,2,7,9],"14021063":[6,28],"14092044":20,"141":[0,2,7],"14100":9,"1416398":[6,28],"14174745":[6,28],"1418":9,"142":[0,7,9],"14250":9,"14277718e":27,"143":[0,2,7,9],"1437":1,"144":[0,7],"14400":9,"1440501043841336":1,"14440":9,"1446729567":4,"145":2,"14568914":28,"146":2,"14629156":29,"14710":9,"14716":6,"14722222222222223":1,"147400":[11,27],"147420":[11,27],"1479":9,"148":[3,4,14],"14812206":6,"148282":6,"14932255":23,"149425":6,"14966323":20,"14984157":23,"14g":[6,29],"15":[0,2,4,6,7,8,9,12,13,15,16,17,20,23,26,29],"150":[3,4,8,9,14],"15005476":5,"150707":6,"15098090e":[6,28],"151":[3,4,14],"151068":29,"1512":[],"15130074e":[6,28],"152":9,"15200":9,"1527777777777778":1,"153":[3,4,14],"153036":27,"15313054":29,"15330788e":27,"15384615384615385":9,"154":[3,4,6,14],"15431202":20,"155":[3,4,9,14],"156":[3,4,14],"156956":5,"157":[3,4,14],"1575":9,"15792278643800434":5,"158":9,"15807529":26,"1587":9,"1590":9,"15990":9,"15am":26,"15g":[6,29],"16":[1,2,3,4,5,6,8,9,10,23,26,27,28,29],"1603":3,"16111111111111112":1,"16211139":5,"16220":9,"162246":5,"16231451":4,"1625":9,"1628":9,"16304863":27,"163049":27,"1630775253":1,"16342407":5,"16343471":[6,28],"16384":3,"16500":9,"16526258":20,"166":9,"16632351e":27,"16638104":20,"167787":5,"168":9,"16805821e":[6,28],"16855533":26,"17":[1,2,5,6,7,8,9,13,23,27,29],"1709":9,"171":28,"171525":27,"1715252":27,"17174962e":1,"17222222222222222":1,"1726":9,"17300":9,"1731":9,"17446471":6,"17483760":29,"17485935":20,"175":29,"1752":9,"175300":27,"17561484":20,"17576105":26,"17641709":[6,28],"17748443e":27,"17755779e":27,"17777777777777778":1,"17801022":5,"17861098":6,"17917768":5,"17949575":5,"17953942":11,"1797":[1,3],"17979265":5,"179793":5,"18":[2,6,7,8,9,10,13,23,29],"1801709":28,"18029127":5,"1803":[],"1807":4,"18075088":23,"1809":9,"181":9,"1812":9,"1821":9,"182925111119767":20,"18307821":28,"18333333333333332":1,"18383522":27,"18388":6,"184":9,"1840":29,"184519":27,"185":28,"1860":9,"18611111111111112":1,"18613217e":[6,28],"18660":9,"18670072e":27,"18673098":11,"189496":27,"189622":27,"19":[2,6,9,15,23,26,29],"19166666666666668":1,"192":29,"19207979":5,"19220":9,"1940":0,"194042826649355e":[6,28],"194042826815211e":[6,28],"1943":12,"1944":6,"1949983":26,"1956":9,"19569961":[6,27,28],"19688765":26,"1970":[20,26],"1973":9,"197370":[11,27],"19740":9,"1979":[6,29],"19800":9,"1999":[],"1_1":12,"1_2":12,"1_3":12,"1cm":[0,8,10,23,26],"1d":[1,2,3],"1e":[1,2,4,13,14],"1e10":14,"1e4":6,"1f":1,"1k":20,"1n":[0,26],"1x":[0,26],"2":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,19,20,21,22,23,25,29],"20":[0,1,2,6,7,8,9,13,23,24,26,28,29],"200":[0,2,3,4,8,9,10],"2000":[0,27],"20015436":6,"2004":13,"2006":25,"2008":26,"2010":1,"2011":1,"2014":4,"2015":1,"2016":[0,26],"2018":[0,6,27,29],"2019":[8,9],"2021":[6,14,27,28],"2022":26,"20222345":23,"2023":21,"2024":[15,16,17,18,26,27,28,29],"2027":9,"20277777777777778":1,"20488808e":27,"20500":9,"2060":9,"2069":9,"207545":[11,27],"20833333333333334":1,"20867052175034223":[6,29],"20916295":27,"20980":9,"21":[0,1,2,5,6,7,9,12,13,20,26,27,29],"210340":[11,27],"21058097":5,"21093775":6,"21117128e":27,"21130":9,"2116753732":4,"21169159e":[6,28],"213103":[11,27],"213743":[11,27],"21378153":26,"21517262":6,"21596432":6,"216290":[11,27],"216683":[11,27],"21733307e":27,"217962":11,"21796226":11,"21888121":20,"22":[0,1,2,5,6,9,12,13,20,23,26,27,29],"22019267":20,"22044605e":[5,27],"221":8,"2211108":20,"221180":27,"2216":9,"2218":9,"221805":2,"221921":5,"2222736":5,"222274":5,"222400":27,"22398902":26,"22467274":29,"225":4,"22668089":26,"226818":5,"22681817":5,"22690428":5,"22842468702219465":[6,29],"2284246870221947":[6,29],"22847924":5,"23":[1,2,6,9,12,15,20,23,26,29],"23002365e":[6,28],"23076923076923078":9,"23117504":26,"23167717":5,"23177156e":27,"232435":27,"23333333333333334":1,"2338675":27,"233868":27,"234":6,"23505507e":27,"2351":6,"23528337":29,"2364":9,"239":29,"2397":9,"23971032":27,"24":[0,1,2,6,9,20,26,29],"2401":21,"24140":9,"24155894":20,"24159785":27,"2416":9,"24175744e":[6,28],"2419":9,"2430":9,"24390":9,"24444444444444444":1,"24493666":29,"24502113":20,"2456969087678473":23,"24569691":23,"246":2,"24604925":20,"24829908":5,"24906604e":[6,28],"24975055":20,"25":[2,3,4,5,6,8,9,11,21,23,27,29],"250":[2,4,7,9],"25000":0,"250084":5,"2500845":5,"25185716":28,"251879":27,"252436":27,"25325161e":27,"25343155":29,"255":3,"255001":27,"25529356":26,"256":[2,4],"25617655e":[6,28],"25617658e":[6,28],"25695398":29,"2572":9,"2575":9,"259153":[11,27],"2597":9,"26":[2,6,9,16,29],"26093249":20,"26124692":6,"26292364":29,"26297455":27,"26301436":5,"26409315307910025":[6,28],"2640931530791004":[6,28],"2650":9,"265109911":4,"2654":9,"2662882":26,"26710969":5,"26773827":20,"26780278":5,"26792572":20,"268":9,"26814548":6,"26995402":27,"27":[0,1,2,6,29],"270350130579073":11,"27065633":28,"27486761":5,"274868":5,"2750":9,"27518808":26,"276263":[11,27],"27700":9,"27760":9,"27924636":5,"27n_":23,"28":[1,2,3,4,6,9,23,29],"280573":5,"280647":[11,27],"28205578e":27,"282075":5,"28207517":5,"28238649":20,"282727":[11,27],"2830637392":4,"28336218e":[6,28],"28390":9,"28391978":27,"28443039":29,"28475098":8,"2861":23,"2871":9,"2873":9,"2882":23,"2886":23,"2890":[0,26],"2892":23,"28986131":23,"29":[2,6,9,29],"2915":23,"29167186":5,"2931":26,"29313228":20,"29487549e":27,"296697":11,"29669702":11,"2968":26,"2980":26,"29822833":[6,28],"2990":26,"29920468":20,"299748":27,"2_":12,"2_1":12,"2_2":12,"2_3":12,"2_i":12,"2_m":[6,23,29],"2_t":13,"2_x":23,"2b":23,"2cm":8,"2d":[1,3,11,12,19,26],"2e":[6,29],"2f":[0,7,9,10,11,12,26,27],"2g":2,"2g_i":2,"2k":3,"2m":[6,29],"2n":[0,2,3,26,27],"2nd":9,"2p":23,"2pm":24,"2pt":4,"2u":[],"2x":[0,3,8,13,26],"2x_ix_jy_iy_j":8,"2x_j":8,"2y_i":10,"2y_j":8,"3":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,18,19,20,21,22,23,24,28,29],"30":[0,1,2,4,6,7,9,10,13,16,24,27,29],"30000":[0,26],"30010":9,"30119421":8,"30132427":20,"302":29,"30258509":[20,26],"30283188":28,"303":6,"30442964":27,"30466214e":[6,28],"30481195":20,"3072":3,"30764357e":27,"3076923076923077":9,"30787294":6,"30820040103372":11,"30873019":26,"30940":9,"3099":29,"31":[2,12,20,23],"310579":27,"3105791":27,"312":28,"3123314713548606":[6,29],"31276579e":[6,28],"313":28,"31318084":5,"313851":5,"31385129":5,"31457796":5,"31471398":26,"315":[6,28],"3155":[0,5,6,28,29],"31566127":26,"316":28,"31718909":11,"317367":11,"31896852":8,"31901056":23,"3199881":26,"32":[3,4,6,8,12,13,20,23],"3200":1,"3207":29,"32149601703519115":[6,29],"3214960170351912":[6,29],"3215":9,"32186558":26,"32238906":11,"322768":5,"3227683":5,"32351459e":27,"32399798":20,"324":2,"3245":2,"32450054":27,"3250":1,"3262":6,"326238":27,"32740721e":27,"32791346e":27,"33":[0,2,9,12,20,24],"33047051":5,"330471":5,"33066907e":[5,27],"33166055e":5,"331939":27,"333":7,"33435469":20,"33540382":20,"34":[2,9,20],"340071371496255":27,"3403":9,"340782":[11,27],"34114547":5,"34175429":23,"342680":27,"3436":[0,26],"34367970":6,"34369306":26,"3437":[0,26],"34412923":27,"34565973":11,"345660":11,"34569596":5,"34693879":26,"34782766":23,"3480159":20,"35":[0,2,6,17,21,23,24,26],"35140":9,"351636":[11,27],"35182854":5,"35389684e":27,"3544313922":6,"35444257":27,"35470445e":[5,27],"35514368":26,"35533773":6,"35577273":26,"356399":27,"357508":27,"35768161":26,"3581341341":4,"359":[5,28],"36":[0,2,5,6,21,23],"360":1,"36097931e":27,"3613":9,"361556":27,"3621311":5,"3632959111950474e":[6,28],"363295916366933e":[6,28],"36417623":26,"3655222":5,"367":[2,9],"36895644":29,"369139":[11,27],"36985812":26,"37":[9,21],"37040343":23,"370782966":4,"37239927e":27,"373465215701515":9,"37396662":[6,28],"37409975e":27,"37416502e":27,"37504943":20,"377372":27,"37737221":27,"3777801602":6,"37900111":[6,28],"38":[9,21,23],"380":9,"38135733e":[6,28],"38157698":26,"3817475779":[6,29],"3823545":20,"38267862":5,"382679":5,"38461538461538464":9,"386":9,"38629436":[20,26],"387":29,"38827994":6,"38916861e":[6,28],"3892":29,"38962192e":[6,28],"39":[0,9,24,26],"39197698":27,"391977":27,"39214397":27,"392144":27,"394":9,"394285":11,"39428522":11,"3952007":20,"39579407":5,"39706027":23,"39706038":5,"397700":[11,27],"39789527":[20,26],"3983":6,"39931051e":27,"399836":27,"39990572e":27,"3d":[2,3,4,6,13,21,29],"3f":[1,3,9],"3n":20,"3x":[2,8],"3x_i":2,"3y":8,"3yk470mj5p931p9dtkk0y6jw0000gn":[1,6,26,29],"4":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,20,21,23,26,28,29],"40":[1,6,9,21,24,26,29],"400":4,"4000":[9,26],"40030144":20,"40103689":11,"401037":11,"401842":[11,27],"40332158":23,"40420245":20,"405":9,"4050":[25,26],"405890":[11,27],"40620066":27,"406201":27,"407":9,"40720059":23,"408":9,"4087":9,"4087793":5,"40892859":26,"40927184e":[6,28],"40contain":[],"41":[2,9,20],"4107":9,"41237041":26,"41253101":23,"41383693":26,"41433969":5,"4150":[],"41511965e":1,"4155":[2,15],"4162706317":6,"4166666666666667":9,"41691626":20,"41761415e":27,"41790782":26,"41847076":23,"418506":[11,27],"41858649":20,"41882037e":[6,28],"41988809":20,"42":[1,4,8,9,10,20],"4230769230769231":9,"42379872":23,"42441033":5,"42450":9,"426":[6,9],"42742185":26,"42765083e":27,"429":29,"43":[0,1,2,7,9,20],"43054282":5,"4310":26,"43246775e":27,"43295279":20,"433":29,"43330971e":[6,28],"43442461":20,"435163":27,"43579948e":[6,28],"43639284e":27,"436462435":4,"43766686":11,"438060758":6,"438136":27,"439230":6,"44":[0,1,2,20],"44049151":26,"44089210e":[5,27],"441":[3,4,14],"442":[3,4,14],"442600":[11,27],"443":[3,4,14],"443217":27,"44395301":28,"444":[3,4,9,14],"4442873":23,"445":[3,4,14],"446033":29,"44624525e":27,"44757627":23,"44970586e":1,"45":[0,2,9,24,26],"450":9,"450257":[11,27],"4504":9,"45312744":26,"45422069":20,"455173":29,"4557763":11,"455947":27,"456":9,"457":2,"458027":27,"458078":[11,27],"45915671e":27,"45937581":26,"45960079":5,"46":[2,9,24,26],"460":[9,11],"4601":9,"46124943e":27,"46153846153846156":9,"462":7,"46260797e":27,"46383925e":[6,28],"46383926e":[6,28],"46415888336127775":28,"46608732e":27,"46659511":26,"46713267":28,"46736215":29,"46749552e":27,"4676059":27,"467606":27,"46922796":5,"469228":5,"469596964296967":5,"46984697e":[6,28],"47":[2,9,24,26],"47042744":5,"470478":11,"4704785":11,"47051432":29,"470714":27,"47075725":6,"47116868e":[6,28],"47125748":5,"47132891":5,"47176716":[],"47176783":[],"47179152":[],"47187014":23,"47313680":29,"47610036":6,"47647836":26,"4768597":20,"47734350e":27,"47780495":5,"477805":5,"47815203":11,"47826973":23,"479465113":4,"48":[9,29],"481979":6,"48257387":[24,26],"48329035":23,"48447474":20,"48476997":11,"48494803":28,"48593774":20,"48655388":20,"48688666":11,"486887":11,"48780047":23,"48898864":20,"48994188":5,"49":[5,6,9,11,21,27],"49152":3,"4940954":[0,26],"49510364":20,"4959161509357395e":[6,28],"4959161509366685e":[6,28],"49622615e":27,"49724059":23,"49775827":20,"4990":23,"49905638":26,"4992":23,"4997":23,"4c4c7f":[9,10],"4d":3,"4f":6,"4pm":[24,26],"4y":8,"4y_i":10,"5":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,20,21,23,26,27,28,29],"50":[1,2,3,4,6,7,8,9,10,13,26,27,29],"500":[1,3,4,6,9,10,13,29],"50000455":5,"50000553":5,"50000718":5,"50000855":5,"50000969":5,"50001063":5,"50001142":5,"50001207":5,"50001261":5,"50001306":5,"50001343":5,"50001374":5,"500014":5,"50001414":5,"50001422":5,"50001439":5,"50001454":5,"50001466":5,"50001476":5,"50001485":5,"50001492":5,"50001498":5,"50001502":5,"50001506":5,"5000151":5,"50001512":5,"50001515":5,"50001517":5,"50001518":5,"50001519":5,"50001521":5,"50001522":5,"50001523":5,"50001524":5,"50001525":5,"50066971":23,"5018":23,"50227564e":[6,28],"50321091":5,"50427787":29,"50470576":26,"50530832":29,"506":0,"50641483":23,"506553":27,"50655336":27,"507d50":[9,10],"50846111e":[6,28],"50846112e":[6,28],"50j":13,"50x10":1,"51":10,"510":1,"51084243":20,"51097872":5,"51178631e":27,"511888":5,"51198558e":27,"51374050":29,"51383260e":27,"514219":27,"51542953":27,"51566561":26,"51569907":20,"51631859":26,"51681181e":27,"5177783846":4,"519327774467847":20,"52018478":20,"5222222222222223":1,"52362157e":27,"52511377":20,"52554699e":27,"52656770e":27,"526744":[11,27],"52780600":29,"528698":5,"52869849":5,"52870818":11,"52874252":5,"53":[2,9],"5303329":11,"5305555555555556":1,"531280":27,"53541027":20,"53603254":20,"53607935":20,"53703498":6,"53738247":29,"5378811":11,"5384615384615384":9,"539261":[11,27],"54":[2,6,9,23],"540":9,"54039921":5,"54041041e":5,"541605":27,"54213195":26,"543939":27,"54393936":27,"54428380e":27,"544439":27,"54637219":29,"55":[1,2,9],"55119373":20,"55217048":26,"5545673":26,"5555555555555556":1,"55640817":26,"5567":6,"55757089e":27,"557795":[11,27],"55796518":20,"55854694":11,"55960462":26,"56":[1,2,9],"56033697":5,"56198284":5,"56239321":23,"564":9,"56413916":23,"564374":[11,27],"56466087e":27,"565":9,"56536":[0,26],"56589683":29,"566":9,"56636616e":[6,28],"567":9,"568":[9,29],"56823411":20,"569":[1,9],"56912044e":[6,28],"56939714":5,"56963365":26,"56965674":29,"57":[0,2,8,9,24,26],"570":9,"571":[5,28],"571105947979344e":[6,28],"571105947979412e":[6,28],"57169969":26,"57201944e":[6,28],"574465":[11,27],"576":29,"57652181":26,"5769230769230769":9,"58":[2,9,10,24,26],"58102999e":27,"58310989":26,"58314173e":27,"5833333333333334":9,"583595":27,"58384652":26,"58492636e":27,"58734026":11,"58827236":6,"58886887":20,"5888888888888889":1,"59":9,"591317992":4,"59161025":20,"59354079":6,"5944444444444444":1,"596190":11,"5961905":11,"5972962":26,"59831212":20,"5cm":23,"5f":8,"5x":8,"5y":8,"6":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,20,23,24,26,27,28,29],"60":[1,3,6,9],"60000":4,"60004008e":27,"60010693e":27,"6019067271":4,"60236121":20,"60293962":5,"60314816":26,"60337375":23,"60342978":29,"60393882":20,"60420593":5,"60482174":20,"60518617":20,"60543038":27,"60656897e":27,"60673226":11,"606760":5,"60929685":29,"60943791":[20,26],"60958647":26,"6111111111111112":1,"61124231":20,"612939":27,"613579":27,"614808":[11,27],"6153846153846154":9,"61542957":20,"61562855e":27,"61602252":23,"61775176":29,"61810248":26,"618982":27,"62":2,"6200":26,"6201":26,"6202":26,"6203":26,"6204":26,"62201698e":27,"62373464":11,"625":7,"62528192":26,"62554572":23,"626635268":[6,29],"62725226":28,"62765236":20,"62788541e":27,"628":6,"62894215":5,"629019":11,"62901905":11,"62958747":26,"63":[0,1,2,6,9,29],"63023737":26,"63025821e":[6,28],"63037846":6,"63162342":27,"63249532e":[6,28],"63376422e":27,"63498144":5,"63588621":26,"63619296":20,"6377013":26,"63811548":20,"638618":20,"64":[1,2,3,4,13,20,26],"64012627":5,"640x480":[6,13,21],"64286758":20,"64316192":29,"64403731":20,"64605756":26,"646283":[11,27],"6473418":20,"64742912e":[6,28],"647473":[11,27],"649382":[11,27],"64961094":26,"64x50":1,"65":[1,2,8,9],"651":2,"6515332649941654":5,"65198643":23,"652":2,"652089":11,"65208925":11,"653":2,"6536392":27,"65367685e":27,"654":2,"65442354":27,"654424":27,"655":2,"65571174e":27,"65599456":27,"65743689":27,"658":2,"65833132":27,"65885453":5,"65891389":27,"658914":27,"659":2,"65904730e":27,"66":2,"660":2,"66064822":27,"66080313":27,"66204648":6,"66294408":27,"6638":9,"6652177":27,"66560":9,"66617168469673":23,"667":9,"668172":27,"66865889":26,"67":[2,9],"67047975e":[6,28],"672721":27,"67314874e":5,"674":29,"67496658":23,"67509467":5,"67801834":27,"67851352e":27,"68":2,"68078357e":27,"68192193":5,"68286725":27,"68534263e":[6,28],"68542204":5,"685942":11,"68594205":11,"6869":9,"68772578":26,"6887363571":4,"68929213e":[6,28],"689519":[11,27],"69":[2,9,23],"690":9,"690617":27,"69069n_":23,"69087404e":27,"691":1,"6923076923076923":9,"69313903":23,"69475072":5,"694751":5,"69484813e":5,"69504801":6,"69634577e":[6,28],"69695259":5,"6999536":11,"6n_":23,"6pm":26,"7":[0,1,2,3,4,5,6,7,8,9,11,12,13,14,15,20,22,23,25,26,27,28,29],"70":[1,6,7,9],"700258285068488":23,"70025829":23,"70037324e":27,"701370":5,"70162272":23,"70435575e":27,"70653767":4,"70710678":[5,27],"70832814":5,"7086067479626619":27,"71":1,"7106484566285455":23,"71142161":27,"711422":27,"7119":9,"712018":[11,27],"71384474e":27,"71538320e":27,"71737253":27,"71761101":27,"718165":5,"72048339":20,"72174172":11,"72219105e":27,"72271878e":[6,28],"7236674":5,"724":3,"72532661":26,"72545392e":27,"72664028e":27,"72793290e":27,"72859758":5,"72981762":8,"73":[6,9,29],"731000":27,"73184876":20,"73185968e":27,"73223692e":27,"733096":27,"73435999":26,"7345483":20,"73537890e":27,"73600557":23,"73972932":26,"74":[6,29],"740":9,"74063343e":27,"74070824e":27,"74081822":8,"74222022e":27,"7423220054006":23,"74232201":23,"74303070e":27,"74384949":27,"74474996":26,"7458376":26,"74607851":27,"74647981e":27,"74720185":20,"74829661":27,"74830119e":27,"74840039":26,"74840212":5,"74947597":11,"749476":11,"7496":29,"749765":27,"75":[5,6,8,9,11,21,27,29],"7500735612987705":11,"7503":29,"750445":27,"75050135":27,"75079864":26,"751699":[11,27],"75170092":5,"75199355":20,"75269037":27,"75309603":20,"75437273":11,"754373":11,"75576555":27,"756352":27,"7577":29,"75770568":27,"76":[2,9,24,26],"76160572":0,"76187240e":27,"76190591":28,"76212914":26,"76428845":26,"76471048":20,"76478383":23,"765":7,"76681867":11,"76690383":29,"76887029e":27,"7692307692307693":9,"76936315":5,"7694444444444445":1,"77":[9,24,26],"77152076":5,"7718":9,"77519276":20,"77714169":8,"7782028952":4,"77921992":20,"77966116":23,"7796611648499043":23,"78":2,"78009660e":27,"78058232e":27,"7809358":23,"78156479e":5,"78184120e":[6,28],"78546777":26,"78584667":23,"78890834":23,"78941903":5,"79":2,"79036971":5,"790370":5,"79111643":5,"79189954":26,"793167":27,"794282":[11,27],"7953443":28,"79633065":29,"79689487":26,"79704553":20,"79778008":6,"797e":6,"79909592":[],"7993408651198877":[],"79934087":[],"7995707762668065":[],"7d7d58":[9,10],"8":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,18,20,21,23,26,27,28,29],"80":[0,1,2,5,6,8,9,27,28,29],"800":[4,7],"80006807":26,"80291466":26,"80354994":6,"80460179":29,"80469739":5,"8055555555555556":1,"80609615e":[6,28],"80609616e":[6,28],"80816668":5,"808167":5,"80847477e":[6,28],"80919738":20,"81":1,"81048318e":[6,28],"81160425":5,"81194394":26,"81203214":23,"81333808":6,"815am":[24,26],"81633628":11,"816454":27,"816847":27,"81700981":26,"82":2,"82168579e":27,"82198978":5,"82566059e":27,"8265786":5,"82924408":5,"83":2,"8305555555555556":1,"831048":5,"8310483":5,"83146596":29,"83223115":23,"83460016":20,"83512277":5,"83870189":6,"83954195":23,"84":2,"842":[0,7],"842436":27,"843":[0,7],"84355903e":1,"84388592":26,"844":[0,7],"84443254e":1,"8461538461538461":9,"84716164":23,"84780262":6,"84858":29,"84880366":11,"8492168":26,"84923989e":[6,28],"84963695":5,"849637":5,"84994524":5,"85":[1,2,9],"85013763":26,"850164":5,"85263220":6,"85278920e":5,"85533937":20,"85601992":5,"858":9,"8583333333333333":1,"86":[0,2],"86062977":6,"86117291":5,"86134827":5,"86145244":11,"86260426":20,"86273371":28,"8638888888888889":1,"865":[0,7],"86528841":26,"86557766e":27,"866":[0,7],"86630":9,"8666666666666667":1,"867":[0,7],"86810":9,"87":9,"8702784034":4,"87068171":28,"87072815e":27,"87078":11,"8722222222222222":1,"87381451":5,"875":1,"87599676":20,"87627342":29,"877":6,"87923449":26,"88":[2,26],"88046261":5,"8805555555555555":1,"88168312e":[6,28],"88182591":27,"88253432e":27,"88323026":27,"88336879":5,"88347937":20,"88354703":20,"8840785":20,"88422263":20,"884669":27,"88529063e":[6,28],"88613493":27,"8888888888888888":1,"89":[2,9],"89126914e":27,"89261577":11,"892616":11,"89288636":11,"89410423":5,"89442642":20,"8944444444444445":1,"89685175":23,"89757312":23,"89794988":26,"89800468":26,"89906633":26,"89954169e":27,"89971681e":27,"8f":[6,29],"8g":[6,29],"8n":20,"8x8":1,"9":[0,1,2,3,4,5,6,7,8,9,11,12,13,14,18,20,21,22,23,24,26,27,28],"90":[1,2,6,9],"90075537":5,"9011":6,"90220243":5,"90266948":5,"9027777777777778":1,"9040":9,"90475506e":[6,28],"9054":29,"9055555555555556":1,"9059370566788297":23,"90593706":23,"906747":5,"9090666":20,"90914803":26,"909446359461397":11,"91":[24,26],"910":9,"9106590586410548":23,"9111111111111111":1,"91128596":5,"91358186":20,"91402008e":27,"9149137":20,"91492986e":[6,28],"91561163":20,"91660026":5,"9166666666666666":9,"91760278":5,"91812702":5,"918992":27,"91924234":23,"9197998358799464":23,"92":[6,9,24,26],"9230769230769231":9,"92397145":26,"924251":11,"9242511":11,"924e":6,"925":1,"92507116e":1,"92578916":5,"926564304542535":26,"92674248":20,"92740215":23,"92756256":28,"9277777777777778":1,"92878175":23,"9305555555555556":1,"931":[0,26],"93155188":5,"93158979":5,"93176482":26,"933":[5,28],"93356853e":27,"93492130e":[6,28],"93601008e":27,"937":23,"937082":27,"93742122":26,"93792515":26,"93799826":5,"938":23,"9387":9,"939":[0,23,26],"9392":6,"94":[9,23],"94226022e":[6,28],"94273542":27,"94284104":5,"94320205":5,"9444444444444444":1,"94556653e":27,"94591015":[20,26],"94639099":11,"946957":5,"9472222222222222":1,"94781064":28,"9482527":5,"94897736":26,"9492":6,"95":[1,9,11,29],"95008046":6,"95154558":20,"95231424":5,"952387":27,"95274315":20,"9527777777777777":1,"95284275":5,"953065564":1,"95351665":5,"954":23,"95511792":27,"955118":27,"9555555555555556":1,"955820c21e8b":4,"956563":[11,27],"95684892":5,"95782166e":27,"958228616652075":5,"95980855e":27,"96":[6,9,11,29],"960":23,"9601304850035702e":[6,28],"9601304850073734e":[6,28],"96024953":5,"9603":6,"96084663":5,"961":23,"962":23,"9637117593816477":[6,28],"9640435":5,"9649652536":4,"965548":27,"96631321":27,"96688672":5,"9674916":5,"967809":[11,27],"97005689":5,"97056672":20,"97065359":29,"97108525e":27,"9722222222222222":1,"97243128":5,"972591":5,"97300836":5,"97497404e":[6,28],"975":1,"97507735":5,"97630233":29,"97705827":5,"97758848":5,"9777777777777777":1,"978":29,"9780387310732":25,"9780387848570":25,"9781098134174":26,"9781492032632":25,"9781801819312":26,"978553":5,"97898392":[6,27,28],"97906022e":27,"97926491":5,"97979891":6,"97992238e":27,"98":[0,1,9],"980":9,"9805555555555555":1,"98091621":5,"981321":27,"98139097":5,"98275501":5,"98279090e":27,"9828":[],"983310":27,"98368659":26,"9841":[],"98413059":5,"98454786":5,"985":23,"98566191":5,"986":23,"9861111111111112":1,"98625331e":27,"98650215":26,"98650823":26,"986699":5,"98680716":5,"98716878":5,"98808176":5,"98822371":6,"9888888888888889":1,"989":23,"9890348":5,"9893447":5,"98947894":27,"9898ff":[9,10],"99":[6,9,11,13,29],"990":9,"99009525":5,"99021575e":27,"99088801":5,"991":23,"99115119":5,"99176998":5,"992":23,"99217800e":27,"9923539830272697":27,"99242921":5,"99265097":5,"993":23,"99316252":5,"99371056":5,"99371466":23,"99389612":5,"9943201":5,"9947756":5,"99492986":5,"99509854":26,"9951502749739212":27,"99528218":5,"99539415":5,"99566069":5,"99578809":5,"996":[5,28,29],"99608161":5,"9963961":5,"9964486445275116":27,"99650061":5,"9967458":5,"99700706":5,"99709215":5,"99729756":5,"99751458":5,"99758326":5,"99762142":28,"99775587":5,"99793613":5,"99799099":5,"998100":5,"99810021":5,"99813653":5,"99828624":5,"9983295":5,"99845267":5,"99854557":27,"998577":5,"99861053":5,"99862019":27,"99869482":27,"99871521":5,"99876945":27,"99881845":5,"99883628":27,"99884384":5,"99884409":27,"99884986":28,"99891093":27,"99891873":27,"99893323":5,"99898558":27,"999":[9,23],"99901896":5,"99903755":5,"99906023":27,"9990927157696581":28,"99911427":5,"99912709":27,"99913489":27,"99918546":5,"99919837":5,"99920175":27,"99926459":5,"99927642":27,"9993237":5,"99933188":5,"9993511":27,"99938942":5,"99941797":27,"9994385":5,"99944272":5,"99949266":27,"99949306":5,"99953381":5,"99953475":5,"99956735":27,"99957911":5,"99961294":5,"99965056":5,"99967865":5,"99970894":27,"99970988":5,"9997332":5,"99975913":5,"99977849":5,"99978365":27,"99980002":5,"9998161":5,"99984732":5,"99987324":5,"99989476":5,"9999":29,"99991263":5,"99992746":5,"99993978":5,"99995":5,"999955585168597":[6,28],"9x":[6,21],"9y":[6,21],"\u00f8yvind":[6,27,28],"abstract":1,"boolean":4,"break":[0,4,6,11,14,26],"byte":[20,26],"case":[0,1,2,3,4,5,6,7,11,12,13,14,16,19,20,21,26,29],"catch":0,"class":[0,1,3,4,6,7,8,9,11,12,13,23,26,29],"default":[0,1,2,4,6,7,8,16,20,21,26,27,28],"do":[0,2,3,4,5,6,8,9,10,11,12,13,14,15,16,20,21,26,27],"ekstr\u00f8m":4,"export":9,"f\u00f8470":[24,26],"final":[0,1,2,3,4,5,6,7,8,9,10,11,13,14,16,18,21,22,23,24,26,29],"float":[0,3,4,5,9,11,13,14,20,26,27],"function":[2,3,4,5,9,14,15,16,17,18,19,20],"import":[0,1,2,3,4,6,7,8,9,10,11,12,13,14,15,16,17,21,23,29],"int":[0,1,2,3,4,5,6,11,13,14,20,23,27,29],"long":[0,1,3,4,12,13,26],"m\u00f8svatn":[6,21],"new":[0,1,2,3,5,6,7,8,9,10,11,13,14,17,20,26,27],"public":[0,19,26],"return":[0,1,2,3,4,5,6,7,8,9,11,13,14,20,21,23,26,27,28,29],"sch\u00f8yen":[6,27,28],"short":[4,5,21,28,29],"super":[5,27,28],"switch":0,"throw":[3,6,23,29],"true":[0,1,2,3,4,5,6,7,8,9,10,12,13,14,15,16,20,21,23,26,27,28,29],"try":[0,1,2,4,5,6,7,8,9,10,11,13,14,15,16,19,20,21,23,26,27,28,29],"var":[1,5,6,10,11,18,21,23,26,27,28,29],"while":[0,1,3,4,5,6,7,8,9,11,12,13,23,26,27,28,29],A:[2,3,5,6,7,10,11,12,13,16,18,19,20,21,22,23,24,25,27],AND:2,And:[0,3,4,5,6,9,13,19,21,23],As:[0,1,2,3,4,5,6,8,10,12,13,16,20,21,23,26,27,28,29],At:[0,4,6,13,21,26],BE:[0,26],Be:[2,19,26],Being:13,But:[0,1,2,3,5,6,9,10,23,27,28,29],By:[0,3,5,6,12,13,17,20,26,27,28,29],For:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,25,26,28,29],IF:[6,28],IN:25,If:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,15,16,19,20,21,23,26,27,28,29],In:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,19,20,21,23,25,26,27,28,29],Is:11,Ising:[5,12,27],It:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,19,20,21,23,26,27,28,29],Its:[1,2,4,11],NO:[9,11],No:[0,6,7,9,26,28],Not:[0,1,5,6,27,28,29],OR:23,Of:23,On:[0,3,9,15,21,22,23,24,25,26],One:[0,1,3,4,5,6,7,8,11,12,13,17,21,23,27,29],Or:[0,1,6,21,26],Such:[0,6,12,16,23,29],That:[0,5,7,10,11,12,14,21,23,26,28,29],The:[4,10,13,14,15,16,18,20,21,22,23,24,25],Then:[0,1,6,8,9,10,11,12,13,14,20,21,26,29],There:[0,3,4,5,6,8,9,11,12,14,20,21,22,23,24,26,27,28],These:[0,2,3,4,5,8,9,10,11,12,13,14,15,16,20,21,23,24,26,27,28],To:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,20,23,28,29],With:[0,5,6,8,9,10,11,12,14,18,20,21,23,26,27,29],_0:[5,8,10,11,13,27],_1:[2,5,6,8,10,11,12,13,14,20,27,28],_2:[2,5,8,11,12,13,20,27],_3:20,_4:20,_9:13,_:[0,1,2,4,5,6,7,8,9,10,11,12,13,17,18,20,21,26,27,28,29],__class__:10,__doc__:[6,29],__enter__:[0,7],__future__:[8,9],__getattr__:28,__getattribute__:26,__getitem__:[0,7],__init__:[0,1,2,3,4,7,14,28],__main__:2,__name__:[2,10,28],_accessor:26,_auto10:[6,12],_auto11:6,_auto12:6,_auto1:[2,3,4,5,6,7,12,13,20,23,27],_auto2:[2,3,4,5,6,12,13,20,23],_auto3:[3,4,5,6,12,13,20],_auto4:[4,6,12,13,20],_auto5:[4,6,12,13,20],_auto6:[4,6,12,20],_auto7:[4,6,12,20],_auto8:[6,12],_auto9:[6,12],_base:8,_build:[0,17,19,21,25,26],_c:1,_can_hold_identifiers_and_holds_nam:26,_check_optimize_result:[9,11],_class:8,_compon:11,_coordinate_desc:6,_depth:9,_fraction:9,_generatorcontextmanag:[0,7],_i:[0,1,2,5,6,7,8,11,12,13,15,16,17,21,26,27,28,29],_info_axi:26,_j:[0,1,2,3,5,6,8,13,17,21,27,28,29],_k:13,_l:12,_lambda:[6,26],_leaf:9,_ll:[3,4,14],_logist:[9,11],_m:10,_make_vjp:2,_multilayer_perceptron:1,_n:[2,5,8,11,13,27],_node:[2,9],_o:[3,4,14],_open_file_or_url:[0,7],_p:[5,8,27],_plugin_dir:[3,4,14],_ratio:11,_rc_params_in_fil:[0,7],_s:[3,4,14],_sampl:9,_split:[6,9,21],_t:13,_test:[6,21],_tf_getinputpropertieslist:[3,4,14],_trace:2,_valu:2,_varianc:11,_weight:9,a0:3,a0faa0:[9,10],a1:[0,26],a2:[0,26],a3:[0,26],a4:[0,26],a_0:[0,26],a_1a:[0,26],a_1x:[],a_2a:[0,26],a_2x:[],a_3:[0,26],a_3a:[0,26],a_4:[0,26],a_4a:[0,26],a_:[0,1,16,20,26,27],a_h:1,a_i:[0,1,2,12,26],a_j:[1,12],a_k:[0,1,12],aaron:25,ab:[0,2,5,13,14,21,26,27,28],ab_channel:19,abandon:1,abbrevi:22,abid:23,abil:[0,10],abl:[0,1,4,5,6,7,10,12,13,16,21,27],about:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,19,20,21,24,28,29],abov:[0,1,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,25,26,27,29],abovement:[6,26,29],abscissa:13,absolut:[0,2,5,6,13,26,27,28,29],absorb:[27,28],acceler:13,accept:[0,3,6,9,21,26,27],access:[0,3,11,23,26],accid:[4,6,29],accompani:[0,26,27],accomplish:[8,9,13],accord:[0,1,2,5,6,9,12,13,14,15,16,23,26,28,29],accordingli:11,account:[0,3,5,13,15,23,26,28,29],accumul:[12,13,23],accur:[0,3,4,6,10,13,29],accuraci:[0,1,3,4,5,6,7,9,10,11,12,26,27],accuracy_scor:[0,1,10,26],accuracy_score_numpi:1,achiev:[0,1,5,6,8,12,20,26,28,29],aco:23,acquaint:19,acquir:[1,19,26],acr:0,across:[1,3,6,9,19,26,29],act:[1,3,20],action:23,activ:[0,2,3,4,9,22,24,26],actual:[0,1,4,5,6,8,11,16,20,23,26,27,28,29],ad:[1,3,4,5,8,13,15,16,20,28,29],ada_clf:10,adaboostclassifi:10,adadelta:13,adagrad:[],adam:[1,3,4,24,26],adapt:[4,6,13,25,29],add:[0,1,2,3,4,5,6,8,10,11,12,15,16,17,21,23,26,27,28,29],add_outgrad:2,add_subplot:[1,7,12,14],addendum:5,addit:[0,2,3,5,6,7,8,9,10,12,13,15,17,19,20,21,23,24,25,26,27,28,29],addition:[12,13],address:[1,9,11,13,26],adjac:[3,12],adjoint:[5,27,28],adjust:[0,5,12,13],admir:[0,26],advanc:[4,6,12,25,26,29],advantag:[1,3,5,6,10,13,20,28,29],adversari:26,afecionado:26,affect:3,affin:[0,3,8,11,27],afford:3,aficionado:26,aforement:14,african:0,after:[0,1,2,4,5,6,9,11,12,13,15,19,20,21,23,26,27,28,29],afterward:[0,26],ag:[0,7,22,26,27],ag_0:2,again:[0,1,4,5,6,7,8,10,11,12,13,15,16,21,23,26,27,28,29],against:[1,4,7,10],agegroup:7,agegroupmean:7,aggreg:[9,10],agorithm:10,agre:[5,6,23,28,29],agreement:13,ahead:9,ai:[0,25],aid:11,aim:[0,1,4,6,7,11,14,15,16,19,20,21,27,29],ainv:5,airplan:3,aka:[5,28,29],al:[0,2,4,15,16,17,21,25,26,27,28,29],alarm:[5,7,28,29],aldo:27,algebra:[0,3,5,13,19,27,28,29],algorithm:[0,1,2,4,5,6,7,8,13,14,19,20,21,23,25,26,28,29],align:[0,2,5,6,7,8,13,21,23,26,27,28,29],all:[0,1,2,3,4,5,6,7,9,10,11,12,13,14,15,19,20,21,22,23,24,25,26,27,28,29],allevi:[1,13],alloc:[3,20],allow:[0,1,2,3,5,6,8,10,13,15,16,19,20,21,26,27,28,29],almost:[0,1,6,8,11,13,23,29],alon:[2,9],along:[2,3,4,5,6,9,10,11,19,20,26,27,28,29],alpha:[0,1,2,3,4,6,7,8,9,10,13,14,23,26,27,29],alpha_0:3,alpha_1:3,alpha_2:3,alpha_:10,alpha_i:[3,13],alpha_k:13,alpha_m:10,alpha_n:3,alpha_opt:13,alreadi:[2,3,4,5,6,10,12,19,20,23,26,27,28,29],also:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,26,27,28,29],alter:1,altern:[0,1,4,5,6,8,9,11,13,20,21,26,27,28,29],although:[0,1,5,6,8,10,13,16,26,28,29],alwai:[0,3,5,6,12,13,16,21,23,26,27,28,29],am:4,ame2016:[0,26],american:0,among:[0,3,5,9,10,12,20,26,27,28],amongst:[5,28,29],amount:[0,1,3,4,6,8,10,14,19,29],an:[1,2,3,5,6,7,8,9,11,12,13,14,15,16,17,19,20,21,23,24,25,27,28,29],an_:23,anaconda:[0,1,15,19,21,26],analog:13,analys:[6,28,29],analysi:[1,3,4,7,14,15,16,17,20,25],analyt:[2,3,5,6,7,12,13,15,19,21,26,27,28,29],analyz:[0,1,3,4,5,6,16,17,21,23,27,28],andrew:1,angl:[0,3,9,27],anharmon:3,ani:[0,1,2,3,4,5,6,7,8,9,10,12,14,18,23,26,27,28,29],anim:[4,12],ann:12,annot:[0,1,3,7,8,26],announc:26,anoth:[0,1,3,4,5,6,7,8,10,11,12,13,20,21,23,26,27],anp:2,ans_vspac:2,ansatz:[0,26],answer:[0,1,3,5,6,20,21,24,26,28,29],antialias:[2,6,21],anticip:4,anymor:[1,8],anyon:[4,8],anyth:[1,23],anytim:[24,26],apach:1,apart:[11,13],api:[1,19,26],appar:2,appear:[0,1,3,13,16,20,23],append:[1,3,4,8,9,13,26],appendix:21,appli:[0,1,2,3,4,6,7,8,9,10,11,12,13,15,16,21,23,25,26,27,28,29],applic:[0,1,3,4,5,6,7,9,12,13,20,21,23,25,26,27,29],apply_gradi:4,approach:[1,2,4,5,6,9,10,11,12,13,17,19,23,25,27],appropri:[2,6,9,12,13,19,23,29],approv:26,approx:[0,2,3,6,10,11,13,23,26,29],approxim:[0,1,2,3,4,5,6,7,10,11,13,18,21,23,26,27,28,29],apt:[0,15,19,21,26],aq:23,ar:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,19,20,21,22,23,24,25,26,27,28,29],aragorn:26,arang:[1,3,4,6,7,9,10,12,13,21,26],arbitrari:[1,4,6,8,12,13,23,29],arbitrarili:[0,1,11,26],arc:[6,21],architectur:[3,4,12],archiv:[],area:[0,3,6,9,21,25,26],arg:2,argmax:[1,11],argmin:[4,10,14],argnum:2,argnum_0:2,argnum_1:2,argsort:11,argu:[1,13],argument:[0,2,3,5,6,11,12,13,21,26,27,28,29],argval:2,aris:[0,6,12,13,23,26,29],arithmet:[0,13,20,26],arm:[6,28],armadillo:20,around:[0,1,4,5,6,11,23,26,28,29],arr:2,arrai:[0,1,2,3,4,5,6,7,8,9,11,12,13,14,19,21,23,27,28,29],arrang:[3,26],arraybox:13,arriv:[0,6,9,11,20,23,26,29],arrow:12,arrowprop:8,art:[0,1,15,19,26],articl:[0,3,4,6,10,18,21,26,27,28,29],artifici:[0,2,7,12,25,26],artificialneuron:12,arug:13,arxiv:[3,4,21],asarrai:[0,2,6,9,27,28],asid:27,ask:[5,6,11,12,21,28,29],aspect:[0,6,19,21,26,27,28],assembl:[0,3,26],assert:4,assess:[0,6,21,26,27,28,29],assici:4,assign:[0,7,8,9,12,13,14,22,24,25,26],associ:[0,6,9,12,14,23,26,29],assum:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,21,23,26,27,28,29],assumpt:[0,3,5,6,9,11,18,21,23,26,27],ast:[0,5,6,18,21,26,28,29],astronomi:[26,28],astyp:[4,9,10],asymmetri:[0,26],asymptot:[4,6,29],atom:[0,26],attempt:[0,4,6,7,8,10,26,28],attend:22,attent:[0,20,26],attr:28,attract:[0,10,26],attribut:[0,9,26,28],attributeerror:[26,28],audi:[0,26],audio:[3,4],august:[15,16,26],aurelien:[0,15,22,25,26],austfjel:[6,21],author:[0,1,10,23],authour:26,auto:[8,9,10,23],auto_exampl:[21,28],autocor:23,autocorrelation_tim:23,autocorrelform:23,autocovari:23,autoencod:[4,19,26],autoencond:19,autograd:[19,26],autom:[0,19,25,26],automac:20,automag:26,automat:[0,1,2,3,4,11,16,19,20,26],automobil:3,autonom:4,avail:[0,1,4,6,7,10,11,15,19,20,21,22,24,25,26,29],averag:[0,1,3,6,9,10,13,14,23,24,26,27,28,29],avoid:[0,4,5,6,9,11,13,20,26,27,29],awai:[2,3,6,27,28],awar:[2,10],award:[24,26],ax:[0,1,2,3,4,6,7,8,9,10,11,12,13,14,20,21,26,29],axes3d:[2,6,13,21],axes_grid1:6,axhlin:8,axi:[0,1,2,3,4,6,7,8,9,10,11,12,13,14,20,21,23,26,27,28,29],axiom:[5,28,29],axlabel:[0,26],axvlin:[4,8],axvspan:4,b1:8,b2:8,b3:8,b:[0,1,3,4,5,6,8,9,10,12,13,14,15,16,23,24,26,27,28,29],b_0:0,b_1:[0,2,12,13],b_2:[0,13],b_5:13,b_:[0,1,20],b_group:9,b_i:[0,1,2,12,26],b_ia_:[0,26],b_ia_i:0,b_index:9,b_j:[1,12],b_k:[0,1,12,13],b_m:12,b_score:9,b_valu:9,babcock:26,bachelor:[22,24],back:[0,3,4,5,6,8,9,10,15,20,23,26,27,28],backbon:20,backend:[0,1,4,7],background:[25,26,27],backpropag:1,backtrack:9,backup:20,backward:[1,2,4,12,20],backward_pass:2,bad:[6,27,28],badli:23,bag:[9,19,26],bag_clf:10,baggin:26,baggingboot:10,baggingclassifi:10,baggingtre:10,balanc:[6,29],band:20,bandwidth:20,bar:[0,6,11,15,16,21,26],barber:25,bare:[4,10],base:[0,1,3,4,5,7,8,9,10,14,19,23,24,25,26,27,28],basi:[5,7,8,10,11,12,13,20,27,28],basic:[2,6,8,12,13,14,19,21,23,26],batch:[3,4,11,12,13],batch_shap:4,batch_siz:[1,3,4],batchnorm:4,bay:7,bayesian:[5,19,25,26,28,29],beam:26,becattini:24,becaus:[0,1,2,3,4,5,6,8,9,12,13,14,26,28,29],becom:[0,1,2,5,6,7,9,12,13,23,26,27,28,29],been:[0,1,2,3,4,5,6,11,12,13,19,20,21,26,27,28,29],befor:[0,1,2,3,4,5,6,7,8,12,13,14,20,21,23,26,27,28,29],beforehand:[0,23,26],begin:[0,1,2,3,4,5,6,7,8,9,11,12,13,14,20,21,23,24,26,27,28,29],behav:[1,6,13,29],behavior:[0,1,13,26],behaviour:12,behind:[0,1,6,8,13,26],being:[0,1,2,3,4,5,7,8,10,11,12,13,17,23,26,27,28,29],believ:[9,20],belong:[7,8,9,13,14],below:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,20,21,23,26,27,28,29],benchmark:10,benefici:[1,13],benefit:[0,1,4,11,13,15,19,26],bengio:[1,22,25,26,27],benign:[1,7,9],besid:[4,5,28],bessel:[5,27,28,29],best:[0,1,2,3,4,5,6,7,8,9,10,12,13,21,24,26,27,29],beta:[0,1,3,5,6,7,10,11,13,16,17,18,21,26,27],beta_0:[0,1,3,5,6,7,13,26,27,28],beta_0x_:[0,26,27],beta_1:[0,1,3,5,6,7,10,13,26,27,28],beta_1x_0:[0,26],beta_1x_1:[0,7,26],beta_1x_2:[0,26],beta_1x_:[0,26,27],beta_1x_i:[7,13,27],beta_2:[0,3,13,26,27],beta_2x_0:[0,26],beta_2x_1:[0,26],beta_2x_2:[0,7,26],beta_2x_:[0,26,27],beta_2x_i:27,beta_3:3,beta_3x_i:27,beta_4x_i:27,beta_:[0,3,6,7,13,26,27,28],beta_i:[0,3,5,26,27,28],beta_j:[0,5,6,13,18,21,26,27,28,29],beta_k:13,beta_linreg:13,beta_m:10,beta_mg_m:10,beta_n:3,beta_ol:[],beta_p:7,beta_px_p:7,beta_ridg:[],betaol:[],betaridg:[],betavalu:5,better:[0,1,2,3,4,6,9,10,11,12,13,26,27,28,29],between:[0,1,2,3,4,5,6,7,8,9,11,12,13,14,17,21,23,26,27,28,29],beyond:[0,1,5,6,8,13,21,26,27],bf:[13,14,20,23],bg:26,bgd:13,bia:[0,1,2,3,5,8,9,10,12,13,26,27,28],bias:[1,2,3,5,6,9,12,28],bibliographi:21,big:[0,1,2,5,6,14,18,28,29],bigger:[1,6,27,28],bigr:12,bike:9,bilbo:26,billion:[3,12,19],bin:[0,7,23],binari:[0,3,5,7,9,10,12,26,28,29],binarycrossentropi:4,bind:[0,27],binomi:[19,23,26],binsboot:[6,29],bioinformat:0,biolog:[1,12],bios1100:[19,26],bird:[0,3],birth:26,bishop:[22,25,26],bit:[1,4,20,23,26],bitwis:23,bivari:2,bk:[0,13],bla:[20,26],black:[8,9,14],blob:[16,21],block:[6,10,19,20,23,26,29],blog:26,blogpost:4,blue:[0,3],bmatrix:[0,1,3,5,7,8,11,13,20,26,27,28],bmi:1,bodi:[0,1,4,12,26],bold:1,boldfac:[0,5,16,27,28],boldsymbol:[0,1,2,3,5,6,7,8,10,11,13,14,15,16,17,18,21,26],boltzmann:[12,19,26],book1:25,book:[21,25,26,27,29],boost:[1,9,19,26],boostrap:10,bootstrap:[1,13,19,21,26],borrow:26,boston_dataset:0,bot:8,both:[0,1,4,5,6,8,9,10,13,14,15,16,19,20,21,23,24,26,27,28,29],bottl:7,bound:[0,8,12],boundari:[2,4,8,11,12],box:[4,9],boxed_arg:2,boyd:[8,13],bracket:[4,23],brain:[1,7,12],branch:9,breast:[5,7,11,28,29],breat:[],breviti:13,brew:[0,15,19,21,26],brg:8,brief:[21,27],briefli:[0,26],bring:[0,5,6,10,27,28],broad:0,broadcast_idx:2,brought:[13,19,26],brownle:4,browser:26,brute:[3,5,11,27,28],bs:[8,9,10],buffer_s:4,bui:4,build:[0,4,5,6,10,20,23,26,28,29],built:[0,1,3,4,6,29],bunch:11,busi:0,c1:[8,11],c2:[8,11],c:[0,1,2,4,5,6,7,8,9,10,11,12,13,14,15,16,19,20,23,24,25,27,28,29],c_0:23,c_1:12,c_2:12,c_3:12,c_4:12,c_:[0,8,9,10,13,23],c_i:[12,13],c_k:23,ca:[1,26],cach:10,cal:[0,8,10,12,13,15,16,26],calcul:[0,1,2,4,5,6,8,9,10,11,12,13,14,15,16,18,20,23,26,28,29],california:[],call:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,18,19,20,21,23,24,26,27,28,29],calor:[0,27],cambridg:[13,25],can:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,23,24,25,27,28],cancel:[0,13,26,27],cancer:[5,10,28,29],cancerpd:7,candid:[8,9,10],cannot:[0,1,4,5,6,7,8,9,22,23,27,28],canopi:[0,15,19,21,26],canva:[21,26],cap:[5,28,29],capabl:[0,1,8,13,19,26],capac:[2,24],capita:0,caption:21,captur:[4,11,12],car:[3,4],card:[0,7,26],cardin:1,care:11,carefulli:13,carlo:[0,6,19,23,25,26,29],carri:[2,6,7,21,29],cart:10,casella:25,cast:1,cat:[3,4],categor:[0,1,3,9,11,26],categori:[0,1,3,7,10,12,14,26],categorical_crossentropi:[1,3],caus:[0,5,6,7,23,26,27,28,29],causal:0,causat:[0,26],cax:1,cb:[6,26],cbar:1,cc:[0,1,5,13,26,27,28],ccc:[5,12,28],cd_fast:6,cdf:23,cdot:[0,2,6,12,13,14,20,23,26,29],celebr:13,cell:[0,1,2,3,4,6,7,8,9,10,13,14,21,28],center:[0,1,6,7,8,9,11,14,21,23,26,28,29],central:[0,3,5,6,8,16,20,26,27,28],centroid:[14,23],centroid_differ:14,centuri:3,certain:[0,3,6,7,9,23,26,27,29],certainti:29,cg:13,cha:0,chain:[0,1,13,19,23,26],chanc:[1,5,13,23,28,29],chang:[0,1,2,3,4,5,6,8,9,11,12,13,14,20,21,23,26,27,28,29],channel:3,chapter3:[0,21],chapter:[0,6,10,11,16,18,20,21,25,26,27,28,29],charact:[0,3,5,8,26,27],character:[8,9,10,12,23],characterist:[0,1,3,10,13,26],charg:[0,26],charl:0,chase:4,chatgpt:21,chd:7,chddata:7,cheap:[5,27],cheaper:[1,13],check:[0,1,3,4,5,6,11,13,20,26],checkmark:3,checkpoint:4,checkpoint_dir:4,checkpoint_prefix:4,chen:10,cheng:27,chiaramont:2,choic:[0,1,2,3,4,6,9,12,13,14,20,26,27,29],choleski:[5,20,27],choos:[2,3,6,9,10,11,13,14,21,29],chosen:[0,1,2,6,8,9,10,13,16,23,26,29],chosen_datapoint:1,christian:25,christoph:[22,25,26],cifar10:3,cifar:3,circ:[1,12],circl:[0,8,12,27],circuit:3,circumfer:9,circumv:[1,5,13,27],cite:21,ckpt:4,clariti:23,class_nam:[3,9],class_val:9,class_valu:9,classic:[7,9,13],classif:[0,3,5,6,7,8,11,12,19,21,25,26,27,28,29],classifi:[0,1,4,7,9,10,11,26],classificaton:1,classifii:10,clean:1,clear:[1,5,10,12,13],clearli:[0,3,5,6,7,8,18,23,27,28,29],clever:[1,10],clf3:0,clf:[0,6,8,9,10,26,27],clf_lasso:6,clf_ridg:[6,26],clip:[3,23],clone:24,close:[0,1,2,4,6,8,9,11,12,13,14,21,23,25,26,29],closer:[3,5,13,27],closest:[8,11,13,14],closur:[19,26],cloud:[19,26],cluster:[0,1,4,6,11,19,26,29],cluster_label:14,cm:[1,2,3,6,8,9,13,21],cmap:[0,1,2,3,4,6,8,9,10,21,26],cmap_arg:6,cmb:22,cmd:9,cn_:23,cnn:12,cnn_kera:3,cntk:[19,26],co:[0,2,3,6,9,13,26,29],code:[3,4,6,7,8,15,16,17,19,20,21,23,25],coef0:8,coef:[0,26],coef_:[0,5,6,8,9,13,26,27,28],coeff:5,coeffici:[0,3,5,6,7,8,9,13,15,16,20,26,27,28,29],coerc:[0,6,26,29],coin:[10,23],coin_toss:10,col:[0,11,26,27],colab:[19,26],cold:9,colinear:0,collabor:21,collaps:8,collect:[0,2,6,10,11,19,23,25,26,29],collinear:[5,27,28],color:[0,3,4,6,8,9,10,21,23],color_channel:3,color_cod:6,colorbar:[1,6,21],colsample_bytre:10,colsaobject:10,column:[0,1,2,5,6,7,8,9,11,12,17,20,26,27,28,29],columntransform:9,com:[4,6,16,19,21,25,26,29],combin:[1,2,5,6,7,10,23,28,29],come:[0,1,3,4,5,12,13,14,15,26,27,28,29],command:[0,1,26],comment:[0,4,5,6,15,16,21,26,27],commerci:[0,15,19,21,26],commod:[0,26],common:[0,1,3,5,6,7,9,11,13,14,21,23,26,27,29],commonli:[0,1,4,6,7,9,13,14,27,29],commun:[0,12,21],commut:3,commutatitav:3,compact:[0,1,3,5,6,7,9,11,12,13,14,26,27,28,29],compair:0,compar:[0,3,4,5,6,11,13,15,16,20,21,26,27,28,29],comparison:[2,4,13],compat:7,compet:0,competit:10,compil:[0,1,3,4,13,15,19,20,26],complet:[0,2,3,4,9,12,26],completenn:12,complex:[1,5,8,9,11,12,13,15,16,26,29],complic:[0,1,9,13,26,29],compoment:27,compon:[0,1,3,4,5,6,7,9,14,16,19,26,27,28,29],components_:11,compos:[9,12,13,14,19,26],compphys:[0,6,16,17,19,21,22,24,25,26],compress:[0,26,27],compris:6,compromis:[5,27],compulsori:[19,26],comput:[0,1,2,3,4,5,6,7,8,10,11,12,13,15,16,17,19,20,21,22,23,25,26,27,28,29],computation:[0,3,6,9,13,23,26],computationalscienceuio:26,computerlab:21,con:[],concaten:[2,4,6,14],concav:[1,9,13,27],concentr:[0,10],concept:[0,2,19,26,27],conceptu:[12,13],concern:[0,1,4,7,26],concic:26,conclud:[0,5,13,28,29],conclus:1,cond:2,conda:[0,1,15,19,21,26],condis:27,condit:[0,2,4,5,6,8,9,11,13,23,26,27],conduct:19,condwav:2,confid:[0,5,6,7,8,18,21,26,27,28],configur:3,confirm:[5,12,28,29],confus:[5,6,7,10,20,27,28,29],confusion_matrix:9,congruenti:23,conjug:[4,8],conjugaci:13,conjunct:3,connect:[0,1,3,4,9,11,12,13,20,26,27],consequ:[5,6,8,10,12,13,27,28,29],conserv:[5,14,27],consid:[0,1,2,3,5,6,7,8,9,10,12,13,16,20,21,23,26,27,28,29],consider:[0,1,5,13,26,27,28,29],consist:[0,1,2,3,4,6,12,13,21,23,27,29],constant:[0,2,4,5,6,8,12,13,23,26,27,28,29],constitu:[0,26],constitut:[2,6,29],constrain:[1,3,5,7,11,28],constraint:[5,6,8,13,27,28,29],construct:[0,1,2,3,5,6,7,8,9,10,11,20,23,26,27,28,29],consult:[],contact:[0,26],contain:[0,2,3,4,5,6,7,8,9,11,12,13,17,20,21,23,25,26,27,28,29],contemporari:26,content:[1,19,20,26],context:[6,10,13,21,29],contextlib:[0,7],contigu:20,continu:[0,1,2,3,4,5,6,7,8,9,10,12,13,17,18,20,21,23,26,27,28,29],contour:[9,10,13],contourf:[8,9,10],contrast:[1,4,9,10,12],contribut:[0,3,5,13,23,26,27,28],contributor:[0,21],control:[0,1,3,9,13,15,19,26],conv2d:[3,4],conv2dtranspos:4,conv:[3,4],convei:26,conveni:[0,5,6,12,13,20,21,26,28,29],convent:[12,27],converg:[1,2,4,5,6,8,9,11,13,14,27,28],convergencewarn:[1,6,8,9,11],convert:[0,1,4,5,9,11,13,20,26,27],converttomatrix:4,convex:[4,5,7,27],convinc:13,convolut:[1,4,19,26],cook:21,cool:[4,9],coolwarm:[6,21],coordin:[5,12,14,27,28],coorel:0,copi:[0,1,2,14,27],copyright:[],core:[0,2,7,10,26],corel:26,coronari:7,corr:[0,5,7,11,27],correalt:[11,19],correct:[0,1,2,3,4,5,7,13,20,23,26,27,28,29],correctli:[1,2,6,7,10,21],correl:[0,1,3,5,6,7,10,12,13,19,23,26,28,29],correlation_matrix:[0,5,7,11,27],correspond:[0,3,5,6,8,9,11,12,15,16,17,19,20,21,23,26,27,28,29],cortex:12,cosin:[3,6,29],cost:[0,2,3,5,6,7,8,9,12,13,16,17,21,26],cost_deep_grad:2,cost_funct:2,cost_function_deep:2,cost_function_deep_grad:2,cost_function_grad:2,cost_grad:2,cost_sum:2,costol:13,could:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,17,20,21,23,26,27,28,29],coulomb:[0,26],count:[0,2,9,22,23,24,26],countor:13,coupl:[4,5,6,28,29],cours:[0,1,3,5,11,21,22,24,27,29],courvil:[22,25,26,27],cov:[5,6,11,20,23,26,27,28,29],cov_xi:[5,11,27],cov_xx:[5,11,27],cov_yi:[5,11,27],covari:[0,7,19,20,26,28],covariance_matrix:[5,11,14,27],cover:[0,5,17,19,24,25,27,28],covert:[0,26],covxi:23,covxx:23,covxz:23,covyi:23,covyz:23,covzz:23,cpu:1,craft:3,creat:[1,2,3,4,5,9,10,11,12,19,26,28],create_biases_and_weight:1,create_convolutional_neural_network_kera:3,create_neural_network_kera:1,create_x:[5,11,27],credit:[0,7,24,26],crim:0,crime:0,criteria:[0,4,9,10,14,23,26],criterion:[9,10,13],critic:[6,21,27,28],critiqu:21,cross:[0,1,3,7,9,10,13,19,23,26,27,28],cross_entropi:4,cross_val_scor:[6,29],cross_valid:[7,10],crossvalid:[6,29],crucial:[1,23],cs231:3,cs:[22,24],csr_matrix:[20,26],csv:[0,4,6,7,9,26,29],ctnk:1,cubic:0,cumbersom:[5,28,29],cumsum:[10,11,26],cumul:[7,10,23],cumulative_heads_ratio:10,cup:[5,28,29],current:[1,2,3,4,13,14,25],curs:[0,27],curv:[6,7,10,12,21],curvatur:13,custom:[6,14,21],custom_cmap2:[9,10],custom_cmap:[9,10],cutpoint:9,cv:[6,7,10,29],cvxbook:13,cvxopt:[5,8,27],cycl:[1,12],cyclotron:28,d2_g_t:2,d:[1,2,3,4,5,6,7,8,9,10,11,13,14,20,23,24,26,27,28,29],d_f:13,d_g_t:2,d_net_out:2,da:3,dagger:[5,20,27],dai:[1,9,19],damp:3,daniel:24,darget:9,darkr:23,dat:[0,26],dat_id:[0,6,7,9,26,27,29],data1:14,data2:14,data3:14,data4:14,data:[2,4,5,8,10,12,13,14,17,18,20,25,29],data_id:[0,6,7,9,26,27,29],data_indic:1,data_panda:26,data_path:[0,6,7,9,26,27,29],databas:1,datafil:[0,6,7,9,21,26,27,29],datafram:[0,4,5,7,9,11,26,27],datapoint:[1,5,6,7,11,13,27,29],dataset:[0,4,6,7,8,9,10,11,13,14,15,16,21,26,29],datatyp:4,date:[15,16,17,18,21,26,27,28,29],daughter:10,david:[25,26],davison:29,dbh:1,dbo:1,dc5e85cd93c3:[],dcomposit:20,ddot:2,dead:1,deadlin:[16,17,18,22],deal:[0,1,3,5,6,8,11,13,14,17,18,20,21,23,26,27,28],dealt:0,debt:7,debug:[0,5,6,28,29],decad:[0,3],decai:[0,13,23,26],decemb:[22,24,26],decent:10,decid:[0,2,3,5,6,9,21,28,29],decim:[0,26],decis:[0,1,8,11,19,25,26],decision_funct:8,decision_tre:9,decisiontreeclassifi:[9,10],decisiontreeregressor:[0,9,10],decisiontreeregressordecisiontreeregressor:9,declar:[0,4,20,26],decompos:[5,6,20,27],decomposit:[0,6,12,21,26,28],decompost:[5,27],deconvolut:3,decorrel:[10,13],decreas:[1,2,4,5,6,10,11,13,28,29],deduc:[0,26],deep:[3,7,12,13,19,22,25,27],deep_neural_network:2,deep_param:2,deep_tree_clf1:9,deep_tree_clf2:9,deep_tree_clf:[9,10],deepen:[5,19,26],deeper:[0,3,4,26],deeplearningbook:[25,26,28],deer:3,def:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,21,23,26,27,28,29],def_covari:23,default_tim:4,defect:[5,27],defici:[5,27],defin:[0,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,27,28,29],definit:[1,2,5,6,7,8,10,11,12,13,17,20,23,27,28,29],defint:23,defvjp:2,degre:[3,5,6,8,9,10,11,15,16,17,21,23,26,28,29],deisenroth:27,del:1,delet:6,delimit:4,deliv:[22,26],delta:[0,2,3,6,8,12,13,14,26],delta_0:3,delta_1:3,delta_2:3,delta_3:3,delta_4:3,delta_5:3,delta_:[1,20],delta_h:[0,1,26],delta_j:[3,12],delta_k:12,delta_l:[1,3],delta_momentum:13,delta_n:[0,3,26],delug:19,delv:0,demand:13,demonstr:[0,3,5,6,7,11,12,19,26,27,28,29],den:4,denomin:[1,5,28,29],denot:[1,2,6,7,13,23],dens:[1,3,4],densiti:[0,2,6,23,26,29],depart:[24,26,27,28,29],depend:[0,1,2,4,5,6,7,8,11,12,13,15,19,20,21,23,26,27,28,29],depict:23,deploy:[0,15,19,21,26],deprec:2,depth:[0,3,9,10,20,29],deriv:[0,1,2,6,7,8,10,11,13,16,17,19,21,26],derivati:13,derivative_fn:13,descend:[5,9,11,27,28],descent:[0,1,3,7,8,12,26,27],describ:[0,2,4,5,6,8,10,11,12,13,18,20,21,26,28,29],descript:[0,8,9,26],design:[0,1,3,4,5,6,7,10,11,12,13,15,16,17,18,21,26,28,29],designmatrix:[0,26],desir:[0,2,4,5,13,14,26,27],despit:[1,12],destroi:20,det:[5,20,27],detail:[0,6,11,13,14,20,21,27],detect:[3,8,12],determin:[0,2,3,4,5,6,8,9,10,11,12,13,16,20,23,26,27,28,29],determinist:[7,13,23],dev:[1,21],develop:[0,3,5,8,10,11,12,19,20,21,26,27,28,29],deviat:[0,1,2,4,5,6,21,23,26,27,28,29],devic:[3,4,14],devis:12,df1:26,df:[4,8,11,13,26],di:0,diag:[5,8,27,28],diagnost:[1,10],diagon:[0,5,7,13,18,20,21,23,26,27,28],diagonaliz:[5,27],diagram:10,diagsvd:6,dice:[6,23,29],dict:[6,8],dictat:[],dictionari:0,did:[0,1,5,6,7,10,11,14,21,26,28,29],die:1,diff1:2,diff2:2,diff:2,diff_ag:2,diffeent:8,differ:[0,1,2,3,4,5,6,7,9,10,11,12,13,14,15,17,19,20,21,23,25,26,27,28,29],differenti:[0,3,16,19,20,26,27],differential_oper:2,difficult:[0,1,6,10,13,21,23,26,29],difficulti:[0,1,13,26],diffonedim:2,diffus:[],digit:[0,1,3,4,6,21,22,24,26],dilemma:13,dilut:1,dim:[4,11,14,20],dimens:[0,1,2,3,4,5,8,9,11,14,16,17,20,26,27,28],dimension:[0,4,5,6,9,11,13,14,19,20,26,28,29],dimensionless:[0,3,26],diment:20,dimnsion:4,diod:3,direct:[0,1,2,4,7,11,12,13,14,26,27],directli:[1,4,5,6,23,27],directori:[0,7],disadvantag:[0,26],disappear:[3,6,29],disc_loss:4,disc_tap:4,discard:[6,11,29],disciplin:[0,3,12],disclaim:23,discord:26,discourag:13,discov:[0,26],discover:[5,28],discret:[1,3,5,7,13,28,29],discrimin:[4,7,10,11],discriminator_loss:4,discriminator_loss_list:4,discriminator_model:4,discriminator_optim:4,discuss:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,23,25,26,27,28,29],diseas:7,disguis:[6,27,28],disord:[1,7],displai:[0,1,3,4,5,6,7,8,9,10,11,12,14,21,23,26,27,29],displaystyl:[0,5,17,26,27,28],disregard:[0,26],dissimilar:[11,14],dist:14,distanc:[0,8,9,11,14,23],distance_list:9,distinct:[3,7,8,9,10,14],distinctli:8,distinguish:[0,4,7,8,23,26],distplot:0,distribut:[0,1,4,6,7,10,11,13,14,15,16,18,19,20,21,26,27],distrubut:[0,15,19,21,26],ditto:[],dive:[0,8,20,26],diverg:[1,13],divid:[0,1,3,5,6,7,8,9,11,12,23,26,27,28,29],divis:[6,8,9,13,20,23,29],dlopen:[3,4,14],dna:7,dnn1:4,dnn2_gru2:4,dnn:[0,1,2,4,12,26],dnn_kera:1,dnn_model:1,dnn_numpi:1,dnn_scikit:[0,1,26],doamin:[28,29],doc:[0,6,16,17,19,21,22,24,25,26],document:[4,9,11,13],doe:[0,1,2,3,4,5,6,8,10,11,12,13,14,16,20,21,23,26,29],doesn:[3,9,12],dog:[1,3,4],domain:[5,8,13,21,28,29],domin:[0,26],don:[0,1,3,5,6,7,8,11,13,15,19,21,26,27],done:[0,2,3,4,5,6,9,10,11,13,17,20,21,26,27,28,29],dot:[0,2,3,5,6,7,8,9,10,11,12,13,20,21,23,26,27,28,29],doubl:[3,4,20,26],doubli:1,doubt:21,down:[0,3,6,9,11,12,13],download:[0,1,3,5,6,20,21,25,26],downsampl:3,dozen:1,dq:[6,29],drag:13,dramat:11,drastic:4,draw:[4,6,10,13,29],drawback:[0,1,3,13,27],drawn:[1,4,6,7,11,23,26,29],drive:[3,4],driven:3,drop:[0,1,5,6,11,13,23,26,27,28,29],dropna:[0,6,26,29],dropout:4,dt:[2,3,13,23],dtype:[0,1,2,3,4,14,20,26,27],dual:8,dualiti:6,dub:[0,26],due:[1,2,5,6,8,10,12,13,24,26,27,29],dummi:0,dure:[0,1,3,4,8,9,11,17,19,21,26,29],dwell:0,dwh:1,dwo:1,dx:[2,3,8,23],dx_1:23,dx_1p:[6,29],dx_2p:[6,29],dx_mp:[6,29],dx_n:23,dxp:[6,29],dy:[1,8,23],dylib:[3,4,14],dynam:4,dz:8,e:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,18,23,24,26,27,28,29],e_:[0,2,26],each:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,19,20,22,23,24,26,27,28,29],eager:29,eapprox:[0,26],earli:[1,13],earlier:[0,5,7,8,9,11,12,13,26,27],earthexplor:[6,21],eas:[6,9,14,29],easi:[0,5,6,7,8,9,10,11,12,13,18,19,20,26,27,28,29],easier:[5,6,8,9,13,23,26,27,29],easiest:13,easili:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,20,21,26,27,28,29],eastern:[24,26],ebind:[0,26],eblock:9,econometr:26,economi:5,ecosystem:[19,26],ect:22,edg:3,edgecolor:[6,29],edu:13,educ:[0,21,26],eff:23,effect:[1,4,10,13,23],effic:1,effici:[0,3,10,13,19,20,23,26],efron:[6,29],egrad:13,eig:[5,11,13,20,23,26,27],eigen:23,eigenpair:[5,11,27],eigenvalu:[0,5,8,11,13,20,26,27,28],eigenvector:[5,11,13,17,27,28],eight:[20,26],eigval:[20,23,26],eigvalu:[11,13],eigvec:[20,23,26],eigvector:[11,13],eispack:[20,26],either:[1,5,6,7,8,9,10,11,13,15,16,21,23,26,27,28,29],elabor:23,elarn:3,electr:[0,3,12,26],electron:26,eleg:11,element:[1,2,3,4,5,6,7,8,11,12,13,17,18,19,20,21,22,25,27,28,29],elementari:[10,13,20],elementwis:[3,13],elementwise_grad:[2,13],elessar:26,elif:[2,14],elim:20,elimin:[3,8],els:[1,2,3,4,7,9,12,13,14,20],elu:1,elus:[0,26],email:[22,24,26],embed:[0,11,27],embodi:[6,21,29],emit:23,emner:25,emphas:[0,10,19,26],emphasi:[0,19,25,26],empir:[1,11,23],emploi:[0,1,5,6,11,13,21,23,26,27,28,29],employ:0,empti:[6,10,29],emul:12,en:[19,25],enabl:11,enbodi:[6,29],encod:[0,3,5,7,9,11,14,17,26,27,28],encompass:[0,21,23],encount:[0,1,5,6,7,13,23,26,27,29],encourag:21,end:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,20,21,23,24,26,27,28,29],end_box:2,end_nod:2,end_valu:2,endpoint:[3,6],energi:[0,4,6,27,29],enet_coordinate_desc:6,enforc:12,eng:25,engin:[0,1,3,4,19,26],english:21,enocurag:21,enoent:[3,4,14],enorm:3,enough:[0,6,13,26,29],ensembl:[1,9],ensur:[0,1,2,3,5,6,11,13,23,27,28,29],entail:26,enter:[5,6,17,27,28],enthought:[0,15,19,21,26],entir:[1,3,7,9,19,23,26],entiti:[9,12,20,26],entri:[0,5,8,11,12,20,26,27,28,29],entropi:[1,3,7,10,13,26],enumer:[0,1,2,3,4,6,8,26,27,28],env:[0,1,2,3,4,6,7,8,9,11,14,23,26,28],environ:[2,9,19,21],eo:[0,6,26,29],eol:[0,26],eosfit:[0,26],epoch:[0,1,3,4,12,13,26],epsilon:[0,5,6,7,13,21,26,27,28,29],epsilon_0:[0,26],epsilon_1:[0,26],epsilon_2:[0,26],epsilon_:[0,26],epsilon_i:[0,26,27],eq:[3,13,14,20,23],eqnarrai:[3,5,6,28,29],equal:[0,1,2,3,4,5,6,8,9,11,12,13,14,15,20,21,23,26,27,28,29],equat:[1,3,4,5,6,7,8,9,10,11,13,14,16,17,18,20,21,23,29],equilibrium:[2,12],equiv:[3,13,20,23],equival:[0,1,5,7,8,11,13,19,20,26,27,28,29],erf:23,eriador:26,eridg:26,err:[0,7,10],err_:[6,29],err_sqr:2,errat:13,errno:[0,3,4,7,14],erron:2,error:[1,2,4,5,6,7,9,11,12,13,15,16,17,18,19,20,21,23,28],error_estimate_corr_tim:23,error_hidden:1,error_output:1,escap:13,especi:[1,3,9,12,13,21],essenti:[0,5,6,9,10,12,14,21,23,27],establish:[0,6,10,11,15,21,26],estim:[0,1,5,6,7,10,11,13,19,23,26,27],estimated_mse_fold:[6,29],estimated_mse_kfold:[6,29],estimated_mse_sklearn:[6,29],et:[0,2,4,15,16,17,21,25,26,27,28,29],eta0:[8,13],eta:[0,1,3,8,12,13,26],eta_:13,eta_t:13,eta_v:[0,1,3,26],etc:[0,1,3,5,7,8,9,11,12,13,14,15,19,20,21,23,27],ethic:19,etsim:29,euclidean:[0,14,27],euler:[],evalu:[0,2,3,4,5,6,9,13,21,23,26,27,28,29],evaluationform:21,evaluationgrad:21,evalut:13,even:[0,1,3,4,5,6,8,9,10,11,12,13,14,19,20,23,26,27,28,29],evenli:4,event:[5,7,10,23,28,29],eventu:[0,5,6,11,12,13,21,24,27,28,29],everi:[0,1,2,3,4,5,6,9,10,11,12,13,14,19,23,24,26,27,28,29],everyth:[4,12],everywher:[4,13],evolv:0,exact:[0,5,11,12,13,20,23,26,27],exactli:[0,3,4,6,12,19,21,27,29],exam:26,examin:[6,29],exampl:[5,11,12,13,15,16,19,20,21,23,25],exce:[1,12,13],excel:[0,1,4,5,10,26,27],except:[0,3,4,6,7,8,9,20],excess:[0,26],excit:0,exclud:[1,6,12,21,27,28,29],exclus:[0,1,3,6,23,26,29],execut:[2,5,13,27],exemplifi:13,exercic:[24,26],exercis:[5,19,21,22,24,28,29],exercisesweek35:17,exhaust:[6,29],exhibit:[0,5,6,8,26,27,29],exisit:[],exist:[0,1,2,3,4,5,6,7,8,9,13,14,18,20,21,26,27,28,29],exit:[5,20,27],exp:[0,1,2,5,6,7,8,10,11,12,13,15,16,21,23,26,27,28,29],exp_term:1,expand:[5,7,11,13,27],expandus:[0,7],expans:[0,3,5,8,10,12,13,26,27],expect:[0,1,5,6,7,11,12,13,15,16,19,21,26,27],expectation_value_of_h_wrt_p:23,expens:[6,10,13],experi:[0,1,6,8,13,19,21,26,27,29],experiment:[0,4,6,9,23,26,29],expert:[1,9],explain:[0,6,9,10,11,13,21,26],explained_variance_ratio_:11,explan:[],explanatori:[0,26],explicit:[0,3,6,13,20,21,26,27,28],explicitli:[0,4,8],explod:1,exploit:[0,3,12,13,26],explor:[1,4,6,8,13,19,21],expon:1,exponenti:[0,1,5,6,10,13,21,23,26,28,29],export_graphviz:9,export_text:9,exporttext:9,expos:19,express:[0,2,3,5,6,7,10,12,13,16,20,21,23,26,28,29],exptmean:23,exptvari:23,extend:[0,2,7,11,13,17,19,26],extens:[0,12,15,19,26],extent:[0,1,6,25,29],extern:[3,6,9],extra:[1,3,5,24,26,27],extract:[0,3,5,6,7,8,11,13,20,26,27],extrapol:[0,26],extrem:[0,1,4,5,6,7,8,9,13,16,20,27,28,29],extremum:13,extrins:11,ey:[0,5,6,13,14,20,26,28],f11:[0,26],f12:[0,26],f13:[0,26],f1:13,f1_grad:13,f1d:13,f2:13,f2_grad_x1:13,f2_grad_x1_analyt:13,f2_grad_x2:13,f2_grad_x2_analyt:13,f3:13,f3_grad:13,f3_grad_analyt:13,f4:13,f4_grad:13,f4_grad_analyt:13,f5:13,f5_grad:13,f6:13,f6_for:13,f6_for_grad:13,f6_grad_analyt:13,f6_while:13,f6_while_grad:13,f7:13,f7_grad:13,f7_grad_analyt:13,f8:13,f8_grad:13,f9:[0,13,26],f9_altern:13,f9_alternative_grad:13,f9_grad:13,f:[0,1,2,3,4,5,6,7,8,10,12,13,14,16,18,20,23,24,26,27,28,29],f_0:[3,10],f_1:[10,13],f_2:[12,13],f_3:12,f_:10,f_d:23,f_grad:13,f_grad_analyt:13,f_i:[0,6,12,16,29],f_m:[3,10],f_n:3,f_vec:2,f_wrap:2,face:13,facecolor:[6,8,23,29],facil:[0,15,19,26],facilit:12,fact:[0,1,3,5,9,11,12,13,26,27,28,29],factor:[0,1,3,5,6,9,10,11,13,20,23,26,27,28,29],factori:13,fade:6,fafab0:[9,10],fahimeh:[24,26],fail:[0,6,8,9,11,13,24,26,29],fail_on_error:[0,7],failur:7,fairli:[1,2,23],faisal:[16,27],fake:4,fake_loss:4,fake_output:4,fale:[],fall:[8,9,22],fals:[0,1,2,3,4,5,6,7,9,10,14,20,21,26,27,28,29],famili:[0,7,8,23],familiar:[0,3,5,6,8,15,19,20,21,23,26,28,29],famou:[6,12],far:[0,3,4,5,6,8,11,12,13,14,26,27],fashion:[0,9,10,26],fast:[1,3,6,10,12,13,19,23,26,29],faster:[1,11,13],fastest:[13,20],favor:7,favorit:23,fc:3,fd:[0,7],featur:[0,1,3,5,6,7,8,10,11,12,13,18,19,21,23,26,28,29],feature_nam:[0,1,7,9],feautur:9,fed:1,feed:[0,2,3,11,19,26],feed_forward:1,feed_forward_out:1,feed_forward_train:1,feedback:[4,26],feeddorward:4,feedforward:[1,4,12],feel:[0,5,6,11,13,15,16,19,21,24,26,28],feet:0,felt:21,fetch:[6,21],few:[1,3,4,5,9,23,26],fewer:[0,9,11,26],ffnn:[1,12],field:[0,3,6,12,19],fifth:[0,6,15,16,21,26],fig:[0,1,2,3,4,6,7,12,13,14,21,26],fig_id:[0,6,7,9,26,27,29],figaxi:23,figsiz:[0,1,2,3,4,6,7,8,9,10,26,29],figur:[0,1,2,3,4,5,6,7,8,9,10,12,13,14,15,16,19,21,26,27,28,29],figure_id:[0,6,7,9,26,27,29],figurefil:[0,6,7,9,26,27,29],file:[0,2,3,4,5,6,7,9,14,21,26,27,28,29],file_prefix:4,filenam:26,filenotfounderror:[0,7],fill:[5,9,27],filter:[0,3,4,7],financ:0,find:[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15,16,19,21,23,26,27,28],fine:[0,14,15,26],finish:2,finit:[3,5,6,12,13,17,23,27,28,29],first:[0,1,2,3,5,6,7,8,9,10,11,13,14,15,16,17,20,21,23,24,25,29],firsteigvector:11,fit:[1,3,4,5,6,7,8,9,11,12,13,15,16,17,21,23,27,29],fit_beta:[6,27,28],fit_intercept:[0,5,6,27,28,29],fit_mod:9,fit_transform:[0,6,8,9,11,29],fiti:[0,26],five:[0,9,17,21,26,27],fix:[0,3,4,6,10,11,12,13,21,26,29],fixedloc:6,flag:4,flat:[3,4,12,13,14],flatten:[1,3,4,5,20],flexibl:[1,6,8,10,12,26,29],flip:[24,26],float32:[4,9],float64:[4,20,26,27],flop:[5,20,27],flow:[1,4,12],fluctuat:[5,28],fluid:[],fly:11,fm:[0,26],fmax:3,fmesh:13,fn:7,fname:[0,7],focu:[0,3,4,5,6,15,19,25,26,27,28,29],focus:[1,6,7,20,27,28],fold:[6,9,21],folder:[0,1,3,4,6,14,21,26,29],follow:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29],font:[0,7,23,26],fontdict:23,fontsiz:[1,6,8,9,10,23],fontweight:1,footprint:3,foral:[8,27],forc:[0,5,6,10,11,26,27,28],forcast:4,forecast:[4,12],forest:[0,1,9,19,26],forget:11,form:[0,3,4,5,6,7,8,9,11,12,13,17,19,20,21,23,26,27,28,29],formal:[3,4,14,23],format:[0,1,2,3,4,6,7,8,9,10,11,19,23,25,27,28,29],format_data:4,formatstrformatt:[6,13,21],formul:[4,6,11,14],formula:[3,13,23],forth:[4,12],fortran2003:[19,26],fortran2008:21,fortran90:23,fortran:[0,15,19,20,26],fortun:[0,11,27],forward:[0,3,6,19,20,26,29],foster:26,found:[1,2,3,4,5,6,12,13,14,21,26,27,28,29],foundat:[19,26],four:[4,5,6,8,12,20,21,22,24,26,28],fourier:[0,26],fourierdef1:3,fourierdef2:3,fourierseriessign:3,fourth:[12,26,27],fp:7,frac:[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15,16,17,20,21,23,26,27,28,29],fractal:9,fraction:9,frame:7,framework:[1,3,4,8,10,14,23],frank:[5,11],frankefunct:[5,6,11,21,27],fredli:[24,26],free:[0,6,11,13,15,16,19,20,21,23,24,25,26],freecodecamp:19,freedom:[5,28],freeli:[0,21],frequenc:[3,6,7,23,29],frequent:[0,8,9,13],frequentist:19,fresh:10,fridai:[16,17,18,24,26],friedman:[6,18,21,22,25,26],friendli:4,frodo:26,frog:3,from:[0,1,2,3,4,6,7,8,9,11,13,14,16,17,18,19,20,21,23,24,25,26],from_cod:9,from_logit:[3,4],from_tensor_slic:4,fromnumer:2,front:[0,4,5,15,16,26,27,28],fulfil:[2,5,12,27],full:[0,1,3,5,7,9,10,13,23,26,27],full_matric:[5,27],fulli:[3,6,12,22,23,29],fun:[2,19,26],fun_nam:2,func:2,functionali:11,fundament:[0,6,19,26,29],funtion:2,further:[2,7,9,26],furthermor:[0,3,5,6,7,11,12,13,17,19,21,26,27,28,29],futur:[0,4,8,9,26],futurewarn:8,fy:[15,21,22,24,25,26],fys3150:[],fys4155:21,fys5419:[25,26],fys5429:[25,26],g0:2,g:[0,1,2,3,4,6,8,9,10,11,13,23,26,28,29],g_0:2,g_1:[2,10],g_2:[2,10],g_:[2,9,10],g_analyt:2,g_dnn_ag:2,g_euler:2,g_i:2,g_m:[3,10],g_n:3,g_re:2,g_t:2,g_t_d2t:2,g_t_d2x:2,g_t_dt:2,g_t_hessian:2,g_t_hessian_func:2,g_t_jacobian:2,g_t_jacobian_func:2,g_trial:2,g_trial_deep:2,g_vec:2,gain:[1,5,7,9,10,13,27,28,29],galleri:[0,26],game:4,gamge:26,gamma1:8,gamma2:8,gamma:[0,2,8,9,10,11,13,26],gamma_0:10,gamma_1:10,gamma_1x:10,gamma_:[0,26],gamma_i:[0,8,23,26],gamma_j:13,gamma_k:13,gamma_m:10,gamma_x:[0,26],gap:[6,8],gate:[4,12],gather:[0,1,12,27],gaug:12,gaussbacksub:20,gaussian:[4,5,6,8,14,23,28,29],gaussian_point:14,gaussian_rbf:8,gave:13,gavra:26,gbc:[22,26],gca:[2,6,8,13,21],gd:1,gd_clf:10,gdclassiffiercgain:10,gdclassiffierconfus:10,gdclassiffierroc:10,gdm:13,gdregress:10,ge:[1,5,7,23,27],gen:[0,7],gen_loss:4,gen_tap:4,gender:[0,26],genener:4,gener:[0,1,2,3,5,6,8,10,11,12,13,14,15,16,17,20,21,23,25,27,28,29],generallay:12,generate_and_save_imag:4,generate_imag:4,generate_latent_point:4,generate_simple_clustering_dataset:14,generated_imag:4,generator_loss:4,generator_loss_list:4,generator_model:4,generator_optim:4,genom:19,geodes:11,geometr:[0,13,26],geometri:[5,28,29],georg:25,geotif:[6,21],geq:[2,5,8,9,13,27,28],geron:[0,15,22,25,26],get:[0,1,2,3,4,5,6,7,9,10,11,13,15,19,20,21,23,24,26,27,28,29],get_dummi:9,get_paramet:2,get_split:9,get_yaxi:8,get_yticklabel:6,getattr:2,gg:26,gibb:[19,26],gif:4,gini:10,gini_index:9,ginvers:13,git:[0,15,19,26],giter:13,github:[0,6,9,15,16,17,19,21,22,24,25,26,27],gitlab:[0,15,19,21,26],give:[0,1,2,3,5,6,7,8,9,10,12,13,14,17,19,21,23,26,27,28,29],given:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,26,27,28,29],global:[6,7,13,21],gloriou:[],glorot:1,gnew:13,go:[0,1,3,5,6,8,9,11,12,13,21,26,27,28,29],goal:[0,7,9,26],goe:[0,1,2,5,6,13,14,18,20,26,27,28,29],golden:13,gone:[5,27,28],gong:1,good:[1,3,4,5,6,9,10,11,13,17,19,23,25,27,28,29],goodfellow:[4,22,25,26,27,28],googl:[1,4,19,26],got:[1,6,13,21],gotten:26,gov:[6,21],gp:25,gpu:[1,13,19,26],grad:[2,13],grad_analyt:13,grade:[21,22],gradient:[0,3,4,7,8,9,12,19,26,27],gradientboostingclassifi:10,gradientboostingregressor:10,gradients_of_discrimin:4,gradients_of_gener:4,gradienttap:4,gradual:[1,14],grai:[4,6,21],graph:[1,9,11,12,13],graph_from_dot_data:9,graphic:[0,1,9,26],grasp:0,gray_r:[1,3],grayscal:3,great:[5,13],greater:[1,7,23],greatli:13,greedi:9,green:[0,3,9,23],grei:4,grid:[1,3,6,7,8,12,23,27,28,29],grossli:13,ground:[0,26],group:[0,6,7,9,14,19,21,22,24,26,29],groupbi:[0,26],grow:[1,3,9,10],growth:[0,26],gru:4,guarante:[0,4,13,23,26,27],guess:[1,4,10,13,14],guestrin:10,guid:1,guidanc:21,h1:2,h:[0,1,5,6,8,13,16,18,21,23,24,25,26,27],h_1:[2,13],h_2:[2,13],h_:[0,13,26],h_m:10,ha:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,20,21,23,26,27,28,29],haa:24,habit:[0,27],had:[0,1,6,7,13,26,29],hadamard:[1,12,13],half:[1,8,9],halv:10,hand:[0,1,2,3,5,11,12,13,15,19,20,21,22,23,24,25,26,27,28,29],handi:[3,21],handl:[0,1,2,5,9,11,19,28],handle_unknown:9,handsid:12,handwrit:12,handwritten:[1,5],happen:[1,2,3,4,5,6,10,13,23,27],hard:[1,7,8,10,13],hardcopi:[19,26],harder:[0,1,27],harmon:3,hasn:1,hassl:[0,15,19,26],hast:[19,26],hasti:[0,6,15,16,17,18,21,22,25,26,27,29],hat:[0,1,5,6,7,9,10,11,12,13,17,18,20,21,27,28,29],have:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,23,24,26,27,28,29],haven:1,he:7,head:[0,4,10,23],header:[0,26],heads_proba:10,health:[0,27],hear:[0,13,26],heart:[0,7,26],heatmap:[0,1,3,7,26],heavili:0,heavisid:1,height:[1,3,6,27,28],held:13,help:[0,1,4,12,13,16,21,26],helper:[4,14],henc:[0,5,6,8,9,10,12,13,18,21,26,27,28,29],henrik:[24,26],her:7,here:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,23,26,27,28,29],hereaft:[0,8,12,26],hermitian:20,hessenberg:20,hessian:[0,2,5,13,16,28],heterogen:[9,10],hi:7,hidden:[1,3,4,12],hidden_bia:1,hidden_bias_gradi:1,hidden_layer_s:[0,1,26],hidden_neuron:4,hidden_weight:1,hidden_weights_gradi:1,hierarch:[5,27,28],high:[0,1,2,3,4,5,6,9,10,11,13,14,19,20,21,26,27,29],higher:[0,1,3,5,6,8,13,21,26,27,28,29],highest:[1,2],highli:[0,3,4,10,18,19,20,25,26,27,28,29],highwai:0,hing:8,hint:[13,16,27,28],hip:19,hire:0,hist:[4,6,7,23,29],histogram:[0,6,7,23],histor:[7,11],histori:[3,4,12],hitherto:[5,28],hjorth:[24,26,27,28,29],hobbi:23,hoc:[5,27],hoff:25,hold:[1,3,6,13,14,29],holder:[0,26],holomorphic_grad:2,home:0,homepag:[21,26],homework:[6,13],homogen:[1,3,9,10,13],honchar:2,hopefulli:[0,11,23,26],horizont:11,hors:[3,7],hot:[1,9],hour:[1,19,22,23,24,26],how:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,19,20,21,23,26,27,28,29],howev:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,19,20,21,23,26,27,28,29],hspace:[0,4,8,10,23,26],hstack:1,htf:[22,26],html:[0,9,11,17,19,21,22,24,25,26,28],http:[0,3,4,6,9,11,13,16,17,19,20,21,22,24,25,26,27,28,29],huang:[0,26],huber:[0,26],huge:[1,3,4,19],human:[0,1,3,6,9,12,27,28],humid:9,hundr:1,hungri:1,huse:26,hybrid:22,hydrogen:[0,26],hyper:[],hyperbol:[1,4,12],hyperparam:8,hyperparamet:[3,4,5,6,9,13,17,26,27,28],hyperplan:11,i0:[0,26],i1:[0,6,8,12,26,27,28],i2:[0,8,12,26],i3:[0,12,26],i5:[0,26],i:[0,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,24,26,27,28,29],i_1:[5,6,28,29],i_2:[5,6,28,29],i_:13,ian:25,ic:1,id:[7,13],ida:[24,26],idea:[0,1,2,3,4,6,9,10,12,13,20,21,27,28,29],ideal:[0,2,6,8,13,23,26,29],idem:[6,29],ident:[5,6,12,13,17,20,21,27],identical:[28,29],identifi:[0,1,7,9,11,12,13,14,26,27],ieor:23,ifi:25,ifs:[19,26],ignor:[0,1,3,9,27],ii:[20,23],iii:[20,26],ij:[0,1,3,6,8,12,14,16,18,20,21,23,26,27,28],ik:[0,20,26,27],illustr:[5,7,10,12,13,14,19,26,29],im:6,imag:[1,3,4,6,9,11,12,14,25,26],image_at_epoch_:4,image_batch:4,image_height:3,image_path:[0,6,7,9,26,27,29],image_width:3,imageio:[6,21],images_from_seed_imag:4,imagin:1,immedi:[0,3,4,6,15,19,26],implement:[0,2,3,4,5,6,8,9,10,11,12,13,14,17,21,23,26,27,28],impli:[3,5,6,7,13,20,27,28,29],implicit:3,implicitli:[11,23],importantli:3,impos:[0,6,11,12,26],imposs:[0,5,26,27],impress:[0,12,26],improv:[0,4,5,9,10,11,13,21,27],impur:9,imread:[6,21],imshow:[1,3,4,6,21],in3050:[25,26],in3310:26,in4080:[25,26],in4300:[25,26],in4310:25,in5400:3,in5550:25,in_out_neuron:4,inaccur:13,inact:12,inadequ:[0,26],inch:[6,27,28],includ:[0,1,2,3,4,5,6,7,11,12,15,16,17,19,21,23,24,25,26,27,28,29],include_bia:[6,9,29],inclus:17,incom:12,incorrect:1,incoveni:8,increas:[0,1,3,4,5,6,8,9,11,12,13,21,23,26,27,28,29],increasingli:23,ind:6,inde:[0,2,4,5,6,13,26,27],indefinit:4,independ:[0,5,6,7,8,12,13,23,26,27],index:[0,1,3,4,10,14,19,20,23,25,26],index_col:[0,26],indic:[0,1,3,4,5,6,9,10,11,13,16,21,26,27],indispens:[6,29],individu:[1,6,7,10,12,23,26,27,29],indu:0,indx1:2,indx2:2,indx3:2,indx:20,ineffici:[3,13],inequ:[8,13],inertia:13,inf1000:[19,26],inf1100:[19,26],inf1100l:[19,26],inf1110:[19,26],inf3000:26,infeas:9,infer:[0,1,4,6,25,26,29],inferenc:1,infil:[0,6,7,9,26,29],infin:[5,6,7,11,18,27,28,29],infinit:3,infinitesim:23,influenc:[6,10,29],influenti:1,info:26,inform:[0,1,3,4,6,9,11,12,13,14,20,21,25,26,29],infti:[3,6,13,23,29],ingeni:13,ingrad:2,ingredi:[0,9,26],inher:[6,29],inherit:[20,26],initi:[0,1,2,6,10,13,14,20,23,26,29],inject:14,inlin:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,21,23,26,27,28,29],inner:[0,13,27],inp:4,inplac:13,input:[0,1,3,4,5,6,7,8,12,13,14,21,23,26,27,28,29],input_dim:1,input_shap:[3,4],inputs:1,inputs_shuffl:[0,1,27],insert:[3,5,6,8,10,23,27,28,29],insid:[0,4,7],insight:[0,1,5,19,26,27,28,29],insist:[6,13,27,28],inspir:[0,1,12,26],instabl:2,instal:[0,1,5,6,9,15],instanc:[0,1,2,4,6,9,11,13,26,27,29],instanti:10,instead:[0,1,2,3,4,5,6,8,9,11,13,14,20,21,23,26,27,29],institut:1,instruct:[0,1,15,16,26],int32:10,int_0:23,int_:[3,6,23,29],int_a:23,intak:[0,27],integ:[1,2,13,14,20,23,26],integer_vector:1,integr:[3,6,23,26,29],intellig:[0,14,25,26],intend:10,intens:1,intention:14,interact:[0,6,9,12,19,21,26],intercept:[0,6,8,11,13,16,21,26,28,29],intercept_:[0,6,8,9,13,26,27,28],interceptol:[],interceptridg:[],interchang:[5,12,20,28,29],interconnect:1,interest:[0,1,2,3,4,5,6,7,8,9,12,19,21,23,26,27,28,29],interfac:[0,1,20,27],interior:[0,9,26],intermedi:[20,27],intern:[1,10,12],interpol:[1,3,4,6,12,21],interpr:[5,27,28],interpret:[0,1,6,9,10,12,13,17,20,21,23],interv:[0,3,5,6,7,13,18,21,23,26,27,28],intial:13,intract:[0,4,27],intrins:[3,11,20,23,26],intro:[19,25,26],introduc:[0,1,5,6,8,10,12,17,20,21,23,26,29],introduct:[1,2,4,13,17,25,27],introductori:[0,4,20,25,26,27],intuit:[0,5,6,8,12,13,21,26,28,29],inv:[0,5,13,26,27,28],invalid:[1,8],invalu:[0,13,15,19,26],invari:1,invd:[5,28],inver:8,invers:[0,3,6,13,15,16,17,21,26,27],inverse_transform:8,invert:[0,5,7,10,13,26,28],invh:13,invok:[0,8],involv:[0,2,6,7,11,12,27,29],io:[0,17,19,21,22,24,25,26,27],ip:[0,8,23,26],ipca:11,ipykernel_48918:1,ipykernel_49101:6,ipykernel_49291:26,ipykernel_85796:[],ipykernel_87621:29,ipynb:[19,21,26],ipython:[0,5,7,9,11,14,15,19,21,26,27],iq:[6,29],iri:[8,9],irreduc:[6,29],irrelev:[5,27],irrespect:[0,26],irvin:[],isbox:2,iscomplexobj:2,isn:[5,28,29],isnul:0,isomap:11,isotop:26,issu:[1,9,20,28],it_arrai:13,item:[0,13,26],items:[20,26],iter:[1,2,4,6,8,9,11,13,14,23,29],its:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,20,21,23,26,28,29],itself:[5,6,12,21,23,27,28,29],j1:20,j:[0,1,2,3,4,5,6,8,9,11,12,13,14,16,17,18,20,21,23,25,26,27,28,29],j_:6,j_lasso_sk:6,j_ridge_sk:6,j_sk:6,jackknif:[6,19,26,29],jacobian:[2,13],jacobian_shap:2,jakobsen:24,jason:4,jax:[19,26],jensen:[24,26,27,28,29],jerom:[18,21,25],ji:[12,20],jit:13,jj:[0,5,6,26,28,29],jk:[0,1,6,12,20,26],jl:[0,26],jm:20,jnp:13,job:[2,8,10],johansen:26,join:[0,3,4,6,7,9,14,26,27,29],joint:[4,5,28,29],judg:13,judgement:[6,21],julia:[19,20,21],jump:[21,23],junk:4,jupit:26,jupyt:[0,9,15,19,21,25,26],just:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,19,21,23,26,27,28,29],justif:0,justifi:[3,10],k0:7,k1:7,k:[0,1,3,5,6,7,8,9,10,11,12,13,14,19,20,21,23,24,26,27,28],kaggl:[6,21],kappa_d:23,karl:[24,26],karush:8,katrin:[24,26],keep:[0,1,4,5,6,11,13,14,20,21,26,27,28,29],keepdim:[1,2,6,10,20,29],kei:[0,1,3,6,12,27],kept:[4,6,14,29],kera:[0,4,19,21,26],kernel:[0,1,3,4,14,19,26,27],kernel_librari:[3,4,14],kernel_regular:[1,3],kernel_s:4,kernelpca:11,kev:[0,26],kevin:[25,26],keyboardinterrupt:2,keyword:[6,13,20,21,26],kfold:[6,29],kg:1,ki:20,kick:[1,13],kiener:2,kilomet:[6,27,28],kind:[0,2,3,4,8,12,13,14,26,27],kj:[6,12,20,27,28],kjm:[19,26],kkt:8,kl:23,km:[12,26],kmean:14,kmeanspoint:14,kn_k:14,know:[0,1,2,5,6,8,13,19,26,27,28],knowledg:[0,19,26],known:[1,3,4,5,6,7,8,9,12,20,21,23,25,27,28,29],kondev:[0,26],kp:23,kpca:11,kroneck:14,kuhn:8,kvalsund:[24,26],kwarg:2,kwown:[0,26],l0:7,l1:[0,1,3,7,26],l1_l2:[1,3],l1regl:5,l2:[1,3],l:[0,1,2,3,5,6,7,8,10,11,12,13,20,21,23,26,27],l_1:7,l_2:[7,13],l_:20,l_j:12,la:13,la_i:12,la_k:12,lab:[19,21,26],label:[0,1,2,3,4,5,6,7,8,9,10,12,13,14,19,20,21,23,26,27,28,29],labelencod:[7,10],labels:[6,8,9],labels_shuffl:[0,1,27],laboratori:[22,28],lack:[0,26],lagari:2,lagrang:[8,11],lambda:[0,1,2,3,5,6,7,8,10,12,13,17,18,21,23,26,27,28,29],lambda_0:11,lambda_1:[5,8,11,27],lambda_2:[8,11],lambda_:11,lambda_i:[8,11],lambda_iy_i:8,lambda_jy_iy_j:8,lambda_k:8,lambda_n:[5,8,27],lamda:1,land:[0,8],landmark:8,landscap:13,langl:[0,6,11,23,26,27],languag:[0,1,4,8,15,19,20,21,25,26],lapack:[20,26],laplac:[5,28,29],laptop:19,larg:[0,1,2,4,5,6,8,9,10,11,13,15,19,20,21,23,25,26,27,29],larger:[0,3,5,6,8,10,11,13,17,23,26,27,28,29],largest:[4,8,11],lasso:[0,7,19,26],lasso_sk:6,last:[0,1,2,3,4,5,6,7,8,10,12,13,14,17,18,20,21,23,24,26,28,29],latent:4,latent_dim:4,latent_point:4,latent_space_value_rang:4,later:[0,1,4,7,8,12,13,14,15,19,21,26],latest:[4,19],latest_checkpoint:4,latex:26,latter:[0,3,6,7,8,11,13,16,20,21,23,26,27,28,29],lattic:12,law:0,layer:[0,4,13,26],lbfg:[7,9,10,11],lcc:[5,6,28,29],lda:11,ldot:[0,6,11,21,26,29],le:[5,7,10,13,17,23,27,28],lead:[0,1,3,5,6,7,8,9,10,11,12,13,16,17,20,23,26,27,28,29],leaf:9,leaki:1,leakyrelu:4,lear:13,learn:[3,4,5,6,7,8,9,10,12,18,20,22,24,25],learnabl:3,learner:10,learnig:26,learning_r:[8,10],learning_rate_init:[0,1,26],learning_schedul:13,learnt:21,least:[0,2,7,8,10,11,15,16,17,19,20,23,29],leat:13,leav:[0,1,3,5,6,9,11,26,28,29],lectur:[0,1,5,10,11,12,13,15,17,19,20,21,22,24,25],lecturenot:[0,17,19,21,25,26],left:[0,1,2,3,5,6,7,8,9,10,11,12,13,14,16,17,20,21,23,26,27,28,29],leftarrow:[8,12],legend:[0,2,3,4,5,6,7,8,9,10,13,26,27,28,29],len:[0,1,2,3,4,5,6,8,9,10,11,12,20,26,27,28,29],length:[0,1,2,3,4,8,9,13,16,19,26,27],length_of_sequ:4,leq:[0,5,7,8,13,14,23,26,27,28],less:[0,1,3,4,5,6,8,9,13,19,23,26,27,28,29],lessen:1,let:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,23,26,27,28,29],letter:[0,16,20,23,26,27],level:[0,1,5,6,9,19,20,21,22,24,26,29],li:[8,11],lib:[0,1,2,3,4,6,7,8,9,11,14,26,28],liblinear:[8,10],libmetal_plugin:[3,4,14],librari:[0,1,2,3,4,5,6,7,9,10,11,14,15,20,21,23,25,27,28],library_loc:[3,4,14],licens:[0,1,15,19,21,26],lie:[0,6,11,23,26,27,29],life:[0,1,8,12,26],lifetim:13,like:[0,1,2,3,4,5,6,7,9,10,11,12,13,15,16,19,20,21,23,26,27,28,29],likelihood:[0,1,5,9,26,27],lim_:23,lima:24,limit:[0,5,6,8,9,11,12,20,21,26,27],lin_clf:8,lin_model:0,lin_reg:9,linalg:[0,2,5,6,8,11,13,20,23,26,27,28],line1:8,line2:8,line3:8,line:[0,1,2,3,4,6,7,8,10,11,13,14,21,26,27,28,29],linear:[1,3,5,6,7,9,10,11,12,17,18,19,21,23,29],linear_model:[0,5,6,7,8,9,10,11,13,26,27,28,29],linear_regress:[6,29],linearli:[5,27],linearloc:[6,13,21],linearregress:[0,6,7,9,26,27,28,29],linearsvc:8,lineat:28,liner:[1,3],linerar:10,linewidth:[0,2,4,6,8,9,10,21,29],link:[0,4,9,12,19,21,24,26],linlag:[5,28],linpack:[20,26],linreg:[0,26],linspac:[0,2,3,4,6,8,9,10,13,15,16,20,23,26,27,28,29],linu:4,linux:[0,1,15,19,21,26],liquid:[0,26],list:[0,1,2,3,4,7,9,19,21,26,27],listcomp:2,listedcolormap:[9,10],literatur:[1,7,14,25,29],littl:[1,3,9,12],live:8,ll:[0,23,26,27],lle:[0,27],lloyd:[4,14],lmb:[0,2,5,6,28,29],lmbd:[0,1,3,26],lmbd_val:[0,1,3,26],lmbda:13,ln:[1,13],load:[0,1,3,4,6,7,9,10,14,21],load_boston:0,load_breast_canc:[1,7,9,10,11],load_data:[3,4],load_digit:[1,3],load_iri:[8,9],load_librari:[3,4,14],load_pluggable_device_librari:[3,4,14],loc:[0,3,6,7,8,9,10,26,29],local:[0,1,2,3,7,12,13,27],locat:[2,3,8],log10:[0,5,6,28,29],log:[0,1,2,4,5,6,7,9,10,11,13,20,21,26,27,28,29],log_:[0,26],log_clf:10,logarithm:[0,5,7,20,26,28,29],logbook:21,logic:[0,1,9,26],logist:[0,1,2,8,9,10,11,12,13,19,27],logisticregress:[7,9,10,11],logit:7,logreg:[7,9,10,11],logspac:[0,1,3,5,6,26,28,29],longer:[2,3,8,10,14,20,23,26],loocv:[6,29],look:[0,1,2,3,4,5,6,7,8,9,10,11,13,16,18,20,21,23,26,27,28,29],loop:[1,4,6,10,12,14,19,20,26,29],lose:1,loss:[0,1,3,4,5,6,7,8,10,11,13,20,21,26,28,29],loss_fil:4,lossfil:4,lost:4,lot:[0,1,4,6,29],low:[0,6,9,10,11,21,26,27,29],lower:[0,1,3,6,9,10,16,20,27],lowercas:[20,26],lowest:[9,13,23],lr:[1,3,4,10],lstat:0,lstm:4,lstm_2layer:4,lstsq:[0,26,27],lt:[6,29],lu:[0,5,26,27],lubksb:20,luckili:2,ludcmp:20,lux:20,lvert:1,lw:[0,26],m:[0,1,2,3,5,6,8,9,10,11,12,13,20,22,23,24,25,26,27,28,29],m_1:14,m_:[9,12],m_h:[0,26],m_k:14,m_l:12,m_n:[0,26],m_p:[0,26],m_t:13,ma:11,machin:[1,3,4,5,6,7,9,10,11,12,15,20,22,25,27,28,29],machinelearn:[0,6,16,17,19,21,22,24,25,26],mackai:25,made:[0,1,3,4,5,6,7,9,11,12,18,21,26,27],mae:[0,26],magic:4,magnitud:[1,6,7,13,27,28],mai:[0,1,2,3,5,6,7,8,9,11,12,13,18,19,20,21,23,26,27,28,29],mail:[22,24],main:[0,1,3,4,5,6,7,9,20,21,25,27],mainli:[0,5,6,7,9,26,27,28,29],maintain:[6,29],major:[1,6,9,10,13,20,26,29],make:[1,2,3,4,5,6,7,8,11,12,13,19,20,21,23,25,28,29],make_axes_locat:6,make_moon:[8,9,10],make_pipelin:[0,6,10,27,29],make_vjp:2,makedir:[0,6,7,9,26,27,29],makeplot:[0,26],malcondit:20,malign:[1,7,9],mammographi:[5,28,29],manag:[0,2,3,15,19,21,26],mandatori:[24,26],mani:[0,1,3,4,5,6,7,8,9,11,13,14,19,20,21,23,25,26,27,28,29],manifold:11,manner:3,manual:[6,27,28],map:[0,1,2,6,7,8,11,12,14,21,23,26],marc:27,margin:[0,5,8],marit:[0,26],mark:26,marker:[0,7,20,26],markov:[19,26],marsaglia:23,mass:[0,1,5,13,27],massag:[0,26],masses2016:[0,26],masses2016ol:[0,26],masses2016tre:0,masseval2016:[0,26],master:[6,16,21,22,24],mat1100:[19,26],mat1110:[19,26],mat1120:[19,26],mat:[19,26],match:[1,4,5,13,14,27],materi:[4,5,7,13,17,20,24],math:[3,7,12,13,20,23,25,26],mathbb:[0,4,5,6,7,8,11,12,13,14,17,18,20,21,23,26,27,28,29],mathbf:[0,5,6,7,8,13,18,20,21,26,27,28,29],mathcal:[1,5,6,7,13,21,28,29],matheemat:3,mathemat:[0,6,11,12,13,17,19,20,23,25,26,28],mathemati:26,mathmlbook:16,mathrm:[0,1,3,4,5,6,7,8,9,10,11,12,13,14,17,18,21,23,26,27,28,29],matmul:[1,2,5,28],matnat:25,matplotlib:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,19,20,21,23,26,27,28,29],matric:[0,1,3,4,6,7,8,11,13,16,17,19,27],matrix:[0,2,3,4,6,7,8,10,13,15,16,17,18,21,23,29],matshow:1,matter:[2,3,13,27],max:[0,1,2,3,4,9,10,12,13,24,26],max_depth:[0,9,10],max_diff1:2,max_diff2:2,max_diff:2,max_it:[0,1,8,9,11,13,26],max_iter:14,max_leaf_nod:10,max_sampl:10,maxdegre:[0,6,10,27,29],maxdepth:10,maxim:[1,4,5,7,8,11,28,29],maximum:[0,1,2,3,5,7,8,9,10,13,14,26,27],maxpolydegre:[5,6,28,29],maxpooling2d:3,mbox:[5,6,18,21,27,28,29],mcculloch:12,md:[11,21],mdoel:4,mean:[1,2,3,4,5,6,7,9,10,11,12,13,14,15,16,17,18,19,20,21,23,26,29],mean_absolute_error:[0,26],mean_divisor:14,mean_i:23,mean_matrix:14,mean_squared_error:[0,4,6,7,10,26,27,29],mean_squared_log_error:[0,26],mean_vector:14,mean_x:23,meaning:[0,4,7,26],meansquarederror:[0,26],meant:[2,3,7,10,13],measur:[0,1,2,5,6,9,11,12,14,21,23,26,27,28,29],mechan:[0,4,23,26],median:[0,26,27],medicin:12,medium:[4,8,13],medv:0,meet:[0,24],mehta:[0,26,27,28],mek4270:[],memori:[3,4,11,12,13,20],mention:[0,12,13,21,23,26],mere:[0,15,26],meshgrid:[2,5,6,8,9,10,11,21,27],messag:[5,13],messi:2,met:[0,3,8,27],metadata:2,meteorolog:9,meter:[6,27,28],method:[0,1,2,3,4,5,7,8,11,12,14,16,17,18,19,20,23,25,27,28],metion:[6,21],metric:[0,1,3,6,7,9,10,14,15,16,26,27,29],metropoli:[19,26],mev:[0,23,26],mgd:13,mglearn:[19,26],mgrid:13,mhjensen:[1,2,3,4,6,8,9,11,14],mi:10,mia:[24,26],michael:[],michigan:[26,28],microsoft:25,mid:1,midel:4,midnight:[16,17,18],midpoint:9,might:[0,1,2,4,6,9,13,27,28],mild:9,millimet:[6,27,28],million:[0,26,27],mimic:12,min:[0,2,5,8,9],min_:[0,2,5,14,17,26,27,28],min_samples_leaf:9,mind:[0,6,13,21,26,27,28,29],mindboard:4,mine:[19,26],mini:[1,11,12,13],minibatch:[1,11,13],minibathc:13,miniforge3:[0,1,2,3,4,6,7,8,9,11,14,26,28],minim:[0,1,2,3,5,6,7,8,9,10,11,12,13,14,17,18,21,27,28,29],minima:[0,1,7,13,26],minimum:[0,1,2,6,8,9,11,13,27,29],minmaxscal:[0,27],minor:23,minst:1,minu:7,mirjalili:26,mirror:9,misc:[6,21],misclassif:[8,9,10],misclassifi:[8,10],miser:0,mismatch:1,miss:[0,7,10],mistak:4,mit:25,mix:[1,2,26],mixtur:13,mk:[9,20],mkdir:[0,6,7,9,26,27,29],ml:[0,1,10,13,20,21,27],mlab:23,mlbook:26,mle:[5,7],mlp:1,mlpclassifi:1,mlpregressor:[0,26],mm:20,mml:27,mn:[12,23],mnist:[1,11],mod:23,mode:[22,24,26],model:[2,3,5,7,8,9,10,11,13,14,15,16,18,19,21,23,25,27,28,29],model_select:[0,1,3,5,6,7,9,10,11,26,27,28,29],moder:10,modern:[0,6,7,19,26,29],modif:[2,12,13],modifi:[0,1,3,5,7,8,10,12,13,26,27,28],modul:[0,9,11,20,26,28],modular:23,modulo:23,moe:[11,27],moment:[5,6,13,23,28,29],momentum:[],mondai:26,monitor:13,monoton:[5,12,23,28,29],mont:[0,6,19,23,25,26,29],moor:[5,6,28],more:[0,1,2,4,5,7,8,9,10,11,12,13,14,18,19,23],moreov:[0,3],morten:[24,26,27,28,29],most:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,19,21,23,26,27,28,29],mostli:[1,11],motion:[0,13],motiv:[1,4],move:[0,4,5,6,7,9,12,13,14,21,23,27,28,29],mpl:[0,7,26],mpl_toolkit:[2,6,13,21],mplot3d:[2,6,13,21],mplregressor:1,mse:[0,4,5,6,9,10,15,16,17,21,26,27,28,29],mse_simpletre:10,mselassopredict:[5,28],mselassotrain:[5,28],mseownridgepredict:[6,28],msepredict:[5,28],mseridgepredict:[0,5,6,28],msetrain:[5,28],msle:[0,26],mt:[7,12],mu0:23,mu1:23,mu2:23,mu:[0,6,11,13,23,26,29],mu_1:27,mu_:[6,23,27,28,29],mu_i:[6,27,28],mu_n:11,mu_x:23,much:[0,1,2,3,4,5,6,8,9,10,11,12,13,20,21,23,26,27,28,29],multi:[0,1,3,7,19,26],multiclass:[1,7],multidimension:[11,12,26],multilay:1,multinomi:7,multipl:[2,4,5,6,7,12,13,23,27,29],multipli:[3,5,6,11,13,20,23,27,28],multiplum:8,multivari:[0,2,10,11,19,23,26],multivariate_norm:[11,14],murphi:[11,25,26],must:[1,2,5,6,8,10,12,13,14,23,27,29],mutat:7,mutual:[1,3,6,13,29],mx_:23,myenv:[0,1,2,3,4,6,7,8,9,11,14,26,28],myriad:[0,15,19,26],mz1:23,mz2:23,n1:20,n2:20,n:[0,1,2,3,4,5,6,7,8,10,11,12,13,14,15,16,17,18,20,21,23,26,27,28,29],n_0:[12,23],n_:[1,2,3,8,12,23],n_boostrap:[6,10,29],n_bootstrap:[6,29],n_categori:[1,3],n_cluster:14,n_compon:11,n_epoch:13,n_estim:10,n_examples_to_gener:4,n_featur:1,n_filter:3,n_hidden:2,n_hidden_neuron:[0,1,26],n_i:23,n_input:[0,1,3,27],n_instanc:9,n_iter_i:[9,11],n_job:10,n_k:14,n_l:[12,23],n_layer:1,n_m:9,n_neuron:1,n_neurons_connect:3,n_neurons_layer1:1,n_neurons_layer2:1,n_point:14,n_sampl:[6,8,9,10,14,29],n_split:[6,29],n_step:4,n_t:2,n_x:2,nabla:[1,13],nabla_:[2,13],nabla_w:13,nag:13,naimi:[0,26],naiv:7,naive_kmean:14,najafi:[24,26],name:[0,1,3,4,5,6,7,8,9,10,12,13,14,15,19,20,21,23,24,26,27,28,29],nameerror:10,namespac:[3,4,14],narrow:13,nary_f:2,nary_op_arg:2,nary_op_kwarg:2,nary_oper:2,nation:[1,5,28,29],nativ:[19,26],natur:[0,1,4,8,9,12,13,21,23,25,26],navier:12,nb:23,nb_:20,nbconvert:26,nbviewer:9,nd:14,ndarrai:[2,6],ne:[9,10,20,23,27],nearest:[1,3,6,11],nearli:13,neat:26,neccesari:[6,29],necess:2,necessari:[0,1,3,4,8,14,26],necessarili:[0,4,11,23,26],necesserali:[5,28,29],neck:7,need:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,20,23,27,28,29],neg:[0,1,3,5,6,7,10,13,20,23,26,28,29],neg_mean_squared_error:[6,29],neglect:23,neglig:23,neighbor:[3,6,11],neither:[4,13],neq:[13,14,23],nervou:12,nest:[2,9,12],nesterov:13,net:[2,4,12],netlib:[20,26],network:[0,9,13,19,25,27],networkd:[],neural:[0,13,19,25,27],neural_network:[0,1,2,26],neuralnetwork:1,neuron:[1,2,3,4,12],neutral:[0,26],neutron:[0,26],never:[1,4,6,9,23,29],new_box:2,new_chang:13,new_hobbit:26,new_root:2,new_trac:2,newaxi:[0,3,6,9,29],newli:[0,26],newton:[1,7,8,13,23],next:[0,1,2,3,4,5,6,7,8,9,13,14,26,27,28],next_guess:13,next_input:4,ng:1,ngini:9,ni:14,nice:[0,1,5,11,26,27,28,29],nielsen:[],niter:13,nitric:0,nlambda:[0,5,6,28,29],nlp:25,nm:23,nm_n:[0,26],nmse:[6,29],nn:[2,5,6,12,20,28,29],nn_model:1,nnmin:2,node:[1,2,3,9,10,12],node_constructor:2,nois:[0,4,5,6,8,9,10,13,15,16,21,26,27,28,29],noise_dimens:4,noisi:[1,6,21,29],non:[0,1,3,5,6,7,9,10,11,12,13,14,20,23,26,27,28,29],noncommerci:[],none:[0,1,2,4,5,9,10,13,23,26,27,28,29],nonlinear:[3,6,8,9,11,12,29],nonneg:[6,9,13,29],nonparametr:6,nonsens:23,nonsingular:20,nonumb:[3,7,8,13,20],nor:[1,4,13],norm:[0,1,5,6,8,11,13,17,26,27,28,29],normal:[3,4,5,6,7,8,9,10,11,12,13,15,16,18,19,20,21,23,26,27,28],normali:[20,26],norwai:[6,21,26,29],notat:[0,2,5,6,13,14,23,26,27,28,29],note:[0,1,2,3,4,5,6,7,8,11,12,13,14,15,16,19,20,23,25,26,29],notebook:[0,1,3,9,15,19,21,26],notfounderror:[3,4,14],noth:[1,2,5,8,12,14,23,27,28],notic:[4,5,12,13,20,23,26,28],notimplementederror:2,notion:3,nov:[],novel:[3,6,10],novemb:[1,22,24,26],now:[0,2,4,5,6,7,8,10,11,12,14,15,16,17,19,20,21,23,26,27],nowadai:[0,1,3,9,19,26],nox:0,np:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,15,16,20,21,23,26,27,28,29],npr:2,nsampl:[6,9,29],nt:2,ntrained_model:[6,28],nu:23,nuclear:[5,27],nuclei:[0,23,26],nucleon:[0,26],nucleu:[0,26],num:4,num_coordin:2,num_hidden_neuron:2,num_it:2,num_neuron:2,num_neurons_hidden:2,num_point:2,num_tre:10,num_valu:2,number:[1,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,20,21,22,24,26,28,29],numberid:7,numberparamet:3,numer:[0,5,6,9,10,11,12,13,17,19,20,25,26,27,28,29],numpi:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,19,21,23,27,28,29],numpy_vjp:2,numpy_wrapp:2,nunmpi:[5,27],nvalu:9,nx:2,nx_test:[6,28],nx_train:[6,28],nx_train_mean:[6,28],ny:23,ny_pr:[6,28],ny_train:[6,28],ny_train_mean:[6,28],o:[0,6,7,8,9,11,15,20,24,25,26],obei:[6,11,13,27,28],object:[0,1,2,4,6,8,10,20,26],obliqu:[5,27],observ:[0,1,3,5,6,7,8,9,10,11,12,13,14,23,28,29],obtain:[0,1,5,6,7,8,9,10,12,13,14,17,20,21,23,26,27,28,29],obviou:[5,6,11,23,27],obviouli:26,obvious:[0,4,5,6,20,26,28,29],oc:27,occupi:0,occur:[0,6,8,9,20,23,26],octob:[22,24,26],od:0,odd:[0,3,7,26,27],odenum:2,odesi:2,odin:26,oen:0,off:[1,3,4,5,9,13,23,28,29],offer:[6,11,19,20,22,24,26,29],offic:[24,26],offici:[22,26],often:[0,1,3,4,5,6,7,8,9,10,11,12,13,14,16,19,20,21,23,26,27,29],ofter:[20,26],ol:[0,13,15,16,17,18,26,27],old:[1,5,10,13,28,29],ols_fit:[],ols_fit_beta:[],ols_sk:6,ols_svd:6,olsbeta:[0,5,28],omega:[2,3,6],omega_0:3,omit:[0,5,26,27,28,29],onc:[1,6,9,11,13,29],one:[0,1,3,4,5,6,7,8,9,10,11,13,14,19,20,21,23,24,26,27,28,29],onehot:1,onehot_vector:1,onehotencod:9,ones:[0,2,5,6,8,9,10,11,13,20,21,26,27,28,29],ones_lik:4,ong:27,onl:3,onli:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,20,21,23,26,27,28,29],onlin:[11,22],onto:[5,11,27],open:[0,1,4,6,7,9,15,19,21,22,24,26,29],oper:[0,1,3,5,6,10,11,12,13,15,19,23,26,27,28,29],operation:23,oplu:23,opmiz:13,opportun:0,oppos:[6,13],opposit:[1,5,8,27],opt:[1,5,26,28],optim:[0,2,3,4,5,6,7,9,10,11,14,15,16,17,18,21,28,29],optimis:[1,3],option:[0,1,3,5,6,8,9,11,20,21,27,28,29],optmiz:[1,8,13,27],oral:26,orang:0,order:[0,1,2,3,5,6,7,8,9,10,11,12,15,16,20,21,23,26,27,28,29],ordinari:[0,2,3,7,11,13,15,16,17,19,29],ordinrari:[],oreilli:[25,26],org:[0,3,4,9,11,19,20,21,25,26,28],organ:[6,7,10,20,29],orient:[1,5,23,27,28],origin:[0,3,5,6,8,11,12,13,20,26,27,29],orthogn:[5,27],orthogon:[0,5,6,8,11,13,17,20,26,27,28],orthonorm:[5,27,28],os:[0,1,4,5,6,7,8,9,24,26,27,28,29],oscar:1,oscil:[3,13],oserror:[0,3,4,7,14],oslo:[0,15,19,21,22,24,26,27,28,29],osx:[0,15,19,21,26],other:[0,1,2,3,5,6,7,8,10,13,14,15,19,22,23,24,25,28,29],otherwis:[0,1,4,7,13,20,26],ouput:[5,7,12,28,29],our:[1,2,3,6,7,8,9,10,12,14,15,16,17,18,19,20,21,23,28,29],ourmodel:0,ourselv:[0,5,6,8,11,13,26,27,28,29],out:[0,1,2,4,5,6,7,8,9,10,11,12,13,16,19,20,21,23,26,27,28,29],out_fil:9,outcom:[0,7,9,10,12,23,27],outdoor:9,outer:[6,12,13],outfil:4,outgrad:2,outlier:[0,8,26,27],outlin:[6,10,11,29],outlook:9,outperform:10,output:[0,1,3,4,5,6,7,8,9,10,12,13,16,20,21,23,26,27,28,29],output_bia:1,output_bias_gradi:1,output_shap:4,output_weight:1,output_weights_gradi:1,outputlayer1:12,outputlayer2:12,outsid:4,over1:13,over:[0,1,3,4,5,6,9,10,12,13,20,21,26,27,28,29],overal:[1,10],overcast:9,overcom:[12,13],overdetermin:[0,26],overfit:[0,1,3,6,9,10,13,29],overflow:[1,5,28,29],overhead:12,overlap:[3,7,8,9],overlin:[0,5,6,9,10,11,14,20,26,27,28],overst:0,overtrain:4,overview:3,own:[4,5,6,8,12,13,19,20,21,28],owner:0,ownmsepredict:0,ownmsetrain:0,ownridgebeta:[0,6,28],ownypredictridg:0,ownytilderidg:0,oxid:0,p0:2,p1:2,p:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,17,20,23,26,27,28,29],p_:[2,4,8,9],p_hidden:2,p_i:[5,23,28,29],p_j:23,p_n:23,p_output:2,p_x:23,pack:[0,26],packag:[0,1,2,3,4,5,6,7,8,9,11,13,14,15,19,21,23,27,28],packtpub:26,packtpublish:26,pad:[3,4],page:[0,9,19,26,28,29],pai:[0,1,9,13],paid:[],pair:[0,2,3,9,19,23,26],panda:[0,4,5,6,7,9,11,15,19,21,28,29],panel:26,paper:1,paradigm:[0,26],parallel:[10,13,19,20,26],param:2,paramat:2,paramet:[0,1,2,3,4,5,6,7,8,9,10,12,13,16,17,18,21,23,28,29],parameter:[0,6,10,21,26,27],parametr:[0,6,15,16,26,27,29],paramt:[3,5,28,29],parent:2,parent_argnum:2,part:[0,1,3,5,6,10,17,18,20,22,23,24,26,28,29],partial:[0,1,5,6,7,8,10,11,12,13,16,23,26,27,28],particip:[19,22,24,26],particl:[0,4,13,23,26],particular:[0,1,2,3,5,6,9,10,11,12,13,16,21,23,25,26,27,28,29],particularli:[5,6,8,11,13,23,27,29],partit:[1,4,9],partli:[6,26,29],pass:[2,3,12,14],password:21,past:[10,23],patch:[6,23,29],path:[0,3,4,6,7,9,14,15,19,26,27,29],patient:7,patter:4,pattern:[0,3,4,12,22,25,26],pauli:[0,26],pc:[11,19],pca:[0,7,19,26,27],pd:[0,4,5,6,7,9,11,26,27,28,29],pde:2,pdf:[0,3,4,5,6,9,16,21,25,26,28,29],pedagog:[0,26,27],penal:[6,27,28],penalti:[6,13,21,27,28],penros:[5,6,28],pentagon:13,peopl:[0,1,9,13,19],per:[0,1,6,22,24,26,29],percentag:[0,10,11,24],perceptron:[0,1,7,26],peregrin:26,perfect:[0,1,13,26],perfectli:[4,6,29],perform:[0,2,3,4,5,6,8,10,11,12,13,14,15,16,17,19,20,21,23,26,27,28,29],performac:4,perhap:[0,5,13,26,27,28],perimet:[1,9],period:[1,4,23],permut:11,persist:13,person:[5,6,7,22,24,26,27,28,29],perspect:25,pertin:[12,26],petal:[8,9],peter:[25,27],phantom:23,phase:[6,12],phenomena:23,phi:8,phi_k:8,philosophi:13,phone:[24,26],photo:4,php:[],phrase:[0,26],physic:[0,1,4,7,12,13,23,24,25,26,27,28,29],physicist:[],pi:[2,3,5,6,7,9,12,13,23,28,29],pick:[1,9,10,11,13,14],pickl:1,pictur:[0,26],pie:[19,26],piec:[11,14],pillow:[0,15,19,21,26],pinv:[5,6,13,27,28],pip3:[0,1,15,21,26],pip:[0,1,15,19,21,26],pipelin:[0,6,8,10,27,29],pippin:26,pit:4,pitfal:[6,27,28],pitt:12,pixel:[1,3,4],pixel_height:[1,3],pixel_width:[1,3],place:[0,4,6,8,13,20,21,26,29],plai:[0,3,4,5,6,8,11,19,26,27,28,29],plain:[8,10,12,13,14],plan:[6,9,24,25,26],plane:[8,9],plateau:[5,28],platform:[19,26],plausibl:12,pleas:[9,11,13,21,24,26],plenti:1,plethora:[3,12],plot:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,19,20,21,23,26,27,28],plot_all_sc:[21,28],plot_confusion_matrix:[7,10],plot_count:6,plot_cumulative_gain:[7,10],plot_data:1,plot_dataset:8,plot_decision_boundari:[9,10],plot_import:10,plot_max:4,plot_min:4,plot_model:4,plot_numb:4,plot_predict:8,plot_regression_predict:9,plot_result:4,plot_roc:[7,10],plot_surfac:[2,6,13,21],plot_train:9,plot_tre:[9,10],plt:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,21,23,26,27,28,29],plu:[0,3,5,7,26,27,28],pluggabl:[3,4,14],plugin:[3,4,14],pm:[8,29],pmatrix:2,pml:25,pn:3,png:[0,4,6,7,9,26,27,29],point:[0,1,2,3,5,6,7,8,9,10,11,13,14,15,16,17,20,21,23,24,26,27,29],point_1:4,point_2:4,poisson:[19,23,26],poli:[6,8,29],poly100_kernel_svm_clf:8,poly3:0,poly3_plot:0,poly_featur:[8,9],poly_features10:9,poly_fit10:9,poly_fit:9,poly_kernel_svm_clf:8,polydegre:[0,5,6,10,27,28,29],polygon:13,polym:12,polymi:21,polynomi:[0,5,6,7,8,9,10,11,15,16,17,21,26,27,29],polynomial_featur:[6,29],polynomial_svm_clf:8,polynomialfeatur:[0,6,8,9,27,29],polytrop:[0,6,26,29],pool:3,pool_siz:3,poor:[1,13],poorli:[0,27],pop:2,popul:[0,5,26,27,28,29],popular:[0,1,3,6,7,8,9,11,12,15,19,20,21,23,27],popularli:[0,26],portabl:10,portion:[11,13],pose:[0,4,5,6,11,23,26,29],posit:[0,1,2,3,5,7,8,10,11,13,14,20,23,26,27,28,29],possibl:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,15,19,20,21,23,24,26,27,28,29],posterior:[5,28,29],postpon:[0,26,27],postscript:21,postul:[5,28,29],potenti:[0,3,5,6,12,13,26,27,28,29],pott:12,power:[0,1,5,6,8,9,12,13,26,27,29],pp:[5,6,18,28,29],practic:[0,5,6,7,8,21,23,27,28,29],practition:[0,1,3,26],pre:26,preced:[1,11,12,23],preceed:4,preceq:8,precis:[0,2,5,11,13,20,21,23,26,27,28,29],pred:[6,29],predicit:0,predict:[0,1,5,6,7,8,9,10,15,16,19,21,25,26,27,28,29],predict_prob:1,predict_proba:[7,10],predictor:[0,5,6,7,9,10,11,26,27,28],prefer:[0,1,6,8,9,11,13,15,19,21,26],prepar:[0,6,20,21,26,27],preprocess:[0,4,6,7,8,9,10,11,21,28,29],prerequisit:0,prescript:21,presenc:13,present:[0,5,6,7,9,12,13,20,21,23,26,27,28],preserv:[3,11,20],press:[13,25],pretrain:[1,4],pretti:[0,4,8,9,15,19,21,26],prev_centroid:14,prevent:[13,23],previou:[0,1,2,3,4,5,6,8,10,11,12,13,20,21,23,27],previous:[2,3,9,10,23],price:[0,4,9,13],primal:8,primari:[0,7,26],prime:23,primit:2,princip:[0,5,7,19,26,27],principl:[0,6,7,8,14,26,29],print:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,20,23,26,27,28,29],print_funct:[8,9],printout:[0,26],prior:[0,5,6,26,28,29],privat:0,pro:[],prob:[1,23],probabilist:[0,25,26,27],probabl:[0,1,3,4,6,7,10,13,19,26,27],problem:[0,3,4,5,6,7,8,9,10,11,12,16,17,19,20,21,23,29],probml:25,proce:[0,5,6,7,8,9,10,11,13,20,26,27,28,29],procedur:[2,4,5,6,8,10,11,13,21,27,28,29],proceed:20,process:[0,2,4,6,9,10,12,13,15,19,20,21,23,25,26,29],prod:25,prod_:[1,5,7,28,29],produc:[0,3,4,5,6,9,10,11,12,13,19,20,21,23,26,27,28,29],product:[0,1,3,5,6,7,8,12,13,16,19,20,26,27,28,29],profess:[0,26],progag:[],program:[0,1,4,5,6,8,12,14,15,16,19,20,22,23,24,26,27],programm:20,progress:[1,4,14],prohibit:[6,29],project1:[6,21],project:[0,1,2,3,5,11,13,17,18,19,22,24,27,28,29],project_root_dir:[0,6,7,9,26,27,29],projectwrit:21,promin:12,promis:8,promot:[24,26],prone:9,pronounc:[13,19,26],proof:[0,11,12,13,26,29],propag:[2,3,13],proper:[0,2,6,7,21,29],properli:[1,6,8,10,13,21],properti:[0,1,3,12,13,16,17,20,26,29],proport:[0,1,5,9,11,13,16,23,26,27],propos:[1,4,6,10,21,26],propto:[5,13,28,29],proton:[0,26],prove:[3,13],provid:[0,1,3,4,5,6,8,9,10,12,13,15,19,20,21,23,26,27,28,29],proxi:[1,13],prune:9,pseudo:[20,23],pseudocod:21,pseudoinv:[5,28],pseudoinvers:[5,6,28],pseudorandom:[6,23,29],psycholog:[0,26],pt:13,publish:[],punish:[0,1,26],pure:[3,9,23],purest:9,puriti:9,purpos:[0,3,10,12,14,26],put:1,py:[0,1,2,3,4,5,6,7,8,9,11,14,26,28,29],py_tf:[3,4,14],pycod:26,pydata:19,pydot:9,pyhton2:26,pylab:[0,7,26],pypi:19,pyplot:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,21,23,26,27,28,29],pythagora:[5,28,29],python2:[0,15,21,26],python3:[0,1,2,3,4,6,7,8,9,11,14,15,19,21,26,28],python:[1,2,3,4,5,6,8,11,12,13,14,16,21,23,27,28],pytorch:[0,19,21,26],q:[5,6,8,11,23,27,29],qp:8,qquad:[2,11,13,20],qr:[5,6,20,27],quad:[1,13,20],quadrat:[0,8,9,13,15,16,26],qualit:[4,9,21,23],qualiti:[0,9,15,16,19,26,27],quantifi:1,quantil:10,quantit:[0,6,9,21,26,29],quantiti:[0,2,5,6,7,9,10,11,12,14,16,20,23,26,27,28,29],quantum:[4,12,25,26],quartil:[0,27],quench:5,queri:9,question:[0,5,6,9,11,12,13,21,24,26,27,28,29],qugan:4,quick:[4,23],quickli:[1,3,9,11,13],quit:[1,5,6,9,10,12,27,29],quot:4,r2:[0,5,6,26,27,28],r2_score:[0,26],r2score:[0,26],r:[0,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,19,20,21,23,27,28,29],r_1:9,r_2:9,r_j:9,r_m:9,rad:0,radial:[0,8,12],radioact:23,radiu:[0,1,9,27],rag:2,rain:9,rais:[0,2,3,4,7,14,28],ramp:1,ran0:23,ran1:23,ran2:23,ran3:23,rand:[0,4,5,6,9,10,13,15,16,20,26,27,28,29],randint:[6,9,13,29],randn:[0,1,2,5,6,9,11,13,15,16,26,27,28,29],random:[0,1,2,3,4,5,6,8,9,13,14,15,16,19,20,21,26,27,28,29],random_forest_model:10,random_index:13,random_indic:[1,3],random_st:[0,7,8,9,10,11],randomforestclassifi:10,randomli:[1,6,9,13,14,29],rang:[0,1,2,3,4,5,6,7,9,10,11,12,13,14,20,23,26,27,28,29],rangl:[0,6,11,23,26,27],rangle_x:23,rank:[5,27],rankdir:4,raphson:[1,8,13],rapidli:0,rare:[1,13,26],raschka:[26,27,28,29],rasckha:26,rate:[0,1,2,3,4,8,9,10,12,13],rather:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,20,23,26,27,28,29],ratio:[4,7,9,10,11],rational:[0,26],ravel:[5,6,7,8,9,10,11,13,20,27,29],raw:3,rbf:[8,11,12],rbf_kernel_svm_clf:8,rbf_pca:11,rc:[0,23],rc_temp:[0,7],rcond:[0,26,27],rcparam:[0,1,3,7,8,9,10,23,26],re:[2,4,13],reach:[1,4,5,6,9,10,11,12,13,14,28,29],read:[0,2,3,4,5,6,7,8,11,12,16,20,21,22,23,25,28],read_csv:[0,6,7,9,26,29],read_fwf:[0,26],reader:[0,6,20,23,26,27,28],readi:[0,1,5,6,8,10,11,12,20,21,26,28,29],readili:1,readthedoc:19,real:[0,1,2,4,7,10,11,12,20,27,29],real_loss:4,real_output:4,realist:8,realiti:23,realiz:[1,12],realli:[0,1,26],rearrang:13,reason:[0,1,3,4,10,13,25,26],reassign:1,recal:[5,6,9,10,11,12,20,23,26,27,28,29],recast:3,receiv:[1,3,10,12,23],recent:[0,2,3,4,6,7,10,13,14,21,25,26,28,29],recept:[3,12],receptive_field:3,recip:[0,6,7,20,21,26,27],reciproc:[5,28],recogn:[0,4,5,10,26,28,29],recognit:[0,1,3,12,22,25,26],recommen:26,recommend:[0,2,3,4,5,6,8,13,15,18,19,20,21,25,28,29],reconsid:9,reconstruct:11,record:[10,21,22,24,26],rectangl:[9,13],rectangular:[5,27],rectifi:[1,3,12],recur:[0,19,26],recurr:[0,1,19,26],recurs:[9,19,20,26],red:[0,3,4,6,8,9,29],redefin:[0,10,26,27],redefinit:28,reduc:[1,3,5,6,9,10,11,13,26,28,29],reduct:[0,10,11,19,23,26,27],refer:[0,1,2,3,5,6,9,11,12,13,14,20,25,26,27,29],referenc:2,refin:12,refit:[6,29],reflect:[0,1,4,5,21,23,26],refresh:[19,26],refreshprogrammingskil:26,reg:[10,11],regard:[1,9,13],regardless:12,region:[3,4,6,9,12,21],regist:[6,21,23],reglasso:[5,28],regr_1:[0,9],regr_2:[0,9],regr_3:[0,9],regress:[1,8,11,12,15,16,19,20],regressor:[0,7,10,26],regridg:[0,5,6,28],regular:[0,3,4,5,6,7,9,13,17,24,26,27,28,29],regularis:6,reilli:[0,15,25,26],reinforc:[0,8,19,26],reiter:1,reject:7,rel:[0,4,6,7,9,12,13,23,26,27,29],relat:[0,1,3,4,5,11,13,14,20,23,26,28,29],relationship:[0,4,9,26],relativeerror:[0,26,27],releas:[1,19,26],relev:[0,1,5,7,11,15,19,21,23,26,28],reli:[0,6,8],reliabilti:21,reliabl:[7,23],relu:[3,4,26],remain:[1,2,4,6,12,20,23,27,28,29],remaind:23,reman:2,remark:1,rememb:[0,8,13,20,21,26],remind:[0,5,11,13,20,23,27,28,29],remov:[0,4,5,6,27,28],render:[0,9,26,27],reorder:[5,7,27,28],reorgan:[0,26],repeat:[0,1,3,4,5,6,9,10,11,13,14,20,21,23,26,28,29],repeated:26,repeatedli:[0,6,10,13,29],repet:3,repetit:[6,26,27,29],rephras:13,replac:[0,1,3,4,5,6,10,12,14,15,16,19,21,26,27,28,29],replica:[6,29],repo:21,report:26,reportexampl:21,repositori:[0,4,21,26],repres:[0,1,2,3,4,5,6,7,8,9,10,12,13,23,26,27,28,29],represent:[0,1,3,6,9,23,26,29],representd:3,reproduc:[0,5,6,9,12,15,16,19,21,23,26,27],repuls:[0,26],request:[0,13],requir:[0,1,3,4,5,6,8,9,11,12,13,17,20,26,27,28,29],rerun:9,res1:2,res2:2,res3:2,res_analyt:2,res_analytical1:2,res_analytical2:2,res_analytical3:2,resaml:[6,21],resampl:[0,7,10,19,26,27],rescal:[0,11,12,27],rescu:[5,28,29],reseach:[6,21],research:[0,4,13,19,25,26],resembl:[6,23,29],reserv:[1,5,6,23,28,29],reshap:[0,1,2,3,4,6,8,9,10,15,16,20,26,27,29],residenti:0,residu:[0,5,13,26],resiz:[5,27],resourc:26,respect:[0,1,2,3,5,6,7,8,10,11,12,13,14,16,17,21,23,26,27,28,29],respond:12,respons:[0,7,9,12,26,27],rest:[0,5,27],restat:[0,12,26],restor:4,restored_discrimin:4,restored_gener:4,restrict:[0,3,9,12,26],result:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,19,20,21,23,26,29],retail:0,retain:[5,6,27,29],rethink:29,retriev:26,return_data:14,return_sequ:4,return_x_i:9,reus:[1,3,6,18,21],reveal:[0,12,26],revers:[1,20],review:[19,20],revisit:14,revolut:26,reward:[0,4,26],rewrit:[0,3,5,6,7,8,10,11,12,13,20,21,23,28],rewritten:[2,6,8,10,23,29],rewrot:13,rf:10,rgb:3,rgoj5yh7evk:19,rh:[6,29],rho:[0,10,13,26],rho_1:10,rho_2:10,rho_m:10,rich:[0,26],ride:9,rideclass:9,ridedata:9,ridg:[7,11,13,19,26],ridge_fit:[],ridge_fit_beta:[],ridge_sk:6,ridgebeta:[5,28],right:[0,1,2,3,5,6,7,8,9,10,12,13,14,16,17,20,21,23,26,27,28,29],right_sid:2,rightarrow:[0,1,5,6,8,11,12,13,23,26,27,29],rigor:[0,26],ring:6,rise:[0,26],risk:[0,13,15,16,26],rival:4,river:0,rlm:26,rm:[0,23],rmse:0,rmsporp:13,rmsprop:[1,3,4,13],rnd_clf:10,rng:23,rnn1:4,rnn2:4,rnn:[4,12],rnn_2layer:4,rnn_input:4,rnn_output:4,rnn_train:4,rntrick1:23,rntrick2:23,rntrick3:23,rntrick4:23,ro:[0,13,26],robert:[18,21,25],robust:[0,26],robustscal:[0,27],roc:[7,10],rod:[],role:[0,2,5,6,8,19,26,27,28,29],roll:6,room:[0,24,26],root:[0,5,9,13,23,27,28],rot:26,rotat:[1,8,9,10],rotation_matrix:9,roughli:[1,3],round:[0,7,9,13],routin:[13,20,26],row:[0,1,2,5,6,9,11,20,26,27,28,29],rr:[5,27],rrr:[5,27],rug:13,rule:[0,1,5,6,13,21,26,27,28,29],run:[0,1,2,4,5,6,8,9,11,13,15,19,21,26,27,28,29],runtim:[1,6,14],runtimewarn:[1,6,29],rust:[0,15,19,20,26],rvert:1,rvert_2:1,s:[0,1,2,3,4,5,6,7,9,11,12,13,16,17,18,19,20,21,23,26,27,28],s_1:6,s_:[3,6],s_i:[6,7],s_j:6,s_k:6,saddl:13,sai:[0,1,2,3,4,5,6,7,8,9,10,11,12,20,21,23,26,27,28,29],said:[6,9,13],sake:[0,5,7,11,26,27,28],sale:[0,26],sam:26,same:[0,1,2,3,4,5,6,8,9,11,12,14,17,20,21,23,26,27,28],samm:10,sampl:[0,1,2,3,4,5,6,7,8,9,10,13,14,15,16,19,20,21,23,26,27,28,29],sample_vari:14,sampleexptvari:23,samwis:26,sastri:11,satisfactori:[0,26],satisfi:[1,2,3,6,8,13,20,23,29],satur:[1,6,29],save:[0,4,6,7,9,13,26,27,29],save_fig:[0,6,7,9,10,26,27,29],savefig:[0,4,6,7,9,23,26,27,29],savetxt:4,saw:[5,27],scalabl:10,scalar:[2,5,6,10,27,28,29],scale:[0,1,3,5,6,7,8,9,10,11,12,13,15,16,17,19,20,21,24,26,28],scale_mean:4,scale_std:4,scaler:[0,7,8,9,10,11,21,27,28],scan:[5,7,28,29],scari:[5,28,29],scatter:[0,1,6,7,8,9,14,26,27,28,29],scenario:[6,13],schedul:[13,24],scheme:[1,13],schrage:23,scienc:[0,1,10,12,13,19,22,23,24,25],scientif:[0,15,19,21,26],scientist:[0,26],scikit:[3,5,6,8,9,10,13,19,20,21,22,25],scikit_learn:[0,16],scikitlearn:26,scikitplot:[7,10],scipi:[0,3,5,6,13,15,19,20,21,26,27,28,29],scl:6,score:[0,1,3,6,7,9,10,11,15,16,21,24,26,27,29],scores_kfold:[6,29],scratch:[1,13],sdg:13,seaborn:[0,1,3,6,7,26],seamless:[0,15,19,21,26],search:[0,1,3,5,9,13,26,28],sebastian:26,sebastianraschka:26,sec:6,second:[0,2,3,4,5,6,7,8,9,11,12,14,15,16,19,20,21,23,24,26,27,29],secondeigvector:11,secondli:12,section:[4,11,16,17,20,21,23,27,28],sector:0,see:[0,1,2,3,4,5,6,7,8,10,11,12,13,15,16,17,18,19,20,21,23,26,27,28,29],seed:[0,1,2,3,4,5,6,8,9,11,13,14,15,16,21,23,26,27,28,29],seed_imag:4,seek:[1,2,8],seem:[1,3,4],seemingli:[0,26],seen:[0,1,3,5,10,12,17,23,26],segment:13,seismic:6,seldomli:[0,26],select:[1,5,6,8,9,10,11,17,21,22,23,24,25,26,27,28,29],self:[0,1,2,5,7,26,27,28],sell:4,semest:[7,22],semi:[8,13],semilogx:6,send:[5,12,13,24,26],senior:[22,24],sens:[0,4,6,8,21,26,29],sensibl:3,sensit:[0,5,6,9,13,26,27,28,29],sent:2,sentenc:[4,12],separ:[0,1,2,4,6,8,9,12,14,15,19,21,23,26,29],septemb:[17,18,21,26],sequenc:[2,3,4,7,9,10,12,13,19,20,23,26],sequenti:[1,3,4,10,12,23],seri:[0,1,2,3,4,5,6,10,11,12,13,20,26,27,28,29],serif:[0,7,23,26],serv:[0,1,2,3,5,7,13,21,25,26,27,28,29],servic:21,session:[1,21,22,24,26],set:[1,4,5,6,7,8,10,11,13,14,16,17,19,20,21,23,24,28,29],set_major_formatt:[6,21],set_major_loc:[6,21],set_tick:[1,6,8],set_ticklabel:[1,6],set_titl:[0,1,2,3,7,12,14,26],set_xlabel:[0,1,2,3,7,12,26],set_xlim:[7,12],set_xticklabel:1,set_ylabel:[0,1,2,3,7,26],set_ylim:[7,12],set_ytick:7,set_yticklabel:[1,6],set_zlim:[6,21],seth:4,setminu:6,setosa:[8,9],setosa_or_versicolor:8,setp:[6,29],setup:[1,4,6,8,19,26,27],sever:[0,3,5,6,7,8,9,11,12,13,19,20,23,26,27,28,29],sgd:[1,3],sgd_clf:8,sgdclassifi:8,sgdreg:13,sgdregressor:13,sgn:[5,27,28],shallow:13,shape:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,15,16,20,26,27,28,29],shape_bas:2,share:[1,3,26],she:7,shift:[1,6,12,23,28],ship:3,shire:26,shortcom:13,shorten:4,shorter:23,shorthand:[26,29],shortli:[20,26],should:[0,2,3,5,6,8,9,11,12,15,16,20,21,23,26,27,28,29],show:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,17,18,20,21,23,26,27,28,29],show_shap:4,shown:[0,4,5,8,9,11,12,13,20,27],shrink:[3,5,6,8,11,21,27,28],shrinkag:[5,6,27,28],shrunk:11,shuffl:[0,1,4,6,13,27,29],side:[0,2,5,8,12,13,20,26,28,29],sigh:[19,26],sigma0:23,sigma1:23,sigma2:23,sigma:[0,1,5,6,7,10,11,12,13,17,18,20,21,23,26,27,28,29],sigma_0:[5,27,28],sigma_1:[5,27,28],sigma_2:[5,27,28],sigma_:[5,20,26,27,28,29],sigma_fn:[7,12],sigma_i:[0,5,26,27,28],sigma_j:[5,17,27,28],sigma_m:[6,23,29],sigma_n:[11,23],sigma_t:13,sigma_x:23,sigmoid:[1,2,4,7,8,10,12],sigmundson:[6,27,28],sign:[1,2,7,8,10,23,24],signal:[1,3,10,12],signifi:4,signific:1,significantli:[1,13,23],sigurd:26,sim:[4,5,6,13,18,21,23,28,29],similar:[0,1,2,3,4,5,6,7,8,9,10,11,14,17,19,20,21,26,28,29],similarli:[0,1,3,5,8,10,13,23,26,27,28],simpl:[1,2,3,5,6,7,8,10,11,12,14,15,16,17,19,20,21,23,29],simplepredict:10,simpler:[0,1,5,6,7,13,16,19,21,26,28],simplernn:4,simplest:[0,1,3,4,9,10,12,14,26],simpletre:10,simpli:[0,1,2,4,5,6,8,9,10,11,12,15,19,20,21,23,26,27,28,29],simplic:[2,5,6,7,8,9,10,11,12,14,27,28],simplicti:[5,27],simplifi:[0,6,9,15,19,21,26,27,28,29],simplist:[3,6,23,29],simul:[6,29],simultan:[6,29],sin:[0,1,2,3,4,9,12,13,20,26],sinc:[0,1,2,3,5,6,7,8,9,10,11,13,20,23,25,26,27,28,29],sine:[3,12],singl:[0,1,2,3,5,6,7,8,9,12,13,20,23,26,27,28,29],singular:[0,6,13,20,21,26,28,29],sinusoid:3,site:[0,1,2,3,4,6,7,8,9,11,14,21,22,26,28],situat:[0,4,5,7,13,23,26,27],six:[3,23],size:[0,1,2,3,4,5,6,8,9,10,11,13,20,21,23,26,28,29],sketch:10,ski:9,skill:0,skip:11,skl:[0,6,26,27,28],sklearn:[0,1,3,5,6,7,8,9,10,11,13,14,26,27,28,29],skplt:[7,10],sl:[6,28],slack:8,slice:[2,20,26],slide:[0,3,15,16,21,23,26,27,29],slight:[6,13,29],slightli:[1,2,3,5,6,7,10,23,27,28,29],slope:[8,11,12],slow:[0,2,8,13,27],slower:[5,20,26,27,28],slowest:20,slowli:12,slp:1,small:[0,1,2,3,5,6,8,9,10,11,12,13,19,20,21,23,26,27,28,29],smaller:[0,1,2,5,6,8,9,11,13,23,26,27,28,29],smallest:[0,4,14,15,16,26],smallest_row_index:14,smooth:[0,3,6,9,13,21,26],sn:[0,1,3,6,7,26],sne:11,sneak:26,so:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,18,19,20,21,23,24,26,27,28,29],soar:6,social:0,soft:[1,7,10,12],soften:8,softmax:[3,7],softwar:[0,8,15,19,20],sol:8,sole:[0,6,26],solid:[0,7],solut:[0,1,2,3,5,6,8,10,11,13,17,18,20,21,23,26,27,28,29],soluton:2,solv:[0,1,3,5,6,8,10,11,12,13,20,21,26,27],solve_expdec:2,solve_ode_deep_neural_network:2,solve_ode_neural_network:2,solve_pde_deep_neural_network:2,solveod:2,solveode_popul:2,solver:[2,7,8,9,10,11,20,26],some:[0,1,2,3,4,5,6,7,8,9,10,11,12,14,21,23,26,28,29],some_model:[6,27,28],somehow:4,someth:[0,1,3,4,7,9,11,21,23,26,27],sometim:[0,1,11,12,13,14,26,27],soon:[20,24,27],sophist:[0,26],sopt:13,sort:[5,6,9,11,23,27,29],sound:[3,5,28,29],sourc:[0,1,3,6,15,19,20,21,23,26,29],space:[0,1,4,5,8,9,11,12,13,14,23,27,28,29],span:[0,3,5,9,11,20,26,27],spare:1,spars:[3,6,20,26],sparse_mtx:[20,26],sparsecategoricalcrossentropi:3,sparsiti:10,spatial:[1,2,3,12],speak:23,special:[6,7,10,12,13,20,23,26,27,28],specif:[0,1,2,3,4,5,6,7,8,9,11,12,16,19,20,21,23,25,26,27,28,29],specifi:[0,3,5,6,7,9,11,13,14,21,23,26,28,29],specifici:[0,10,26],spectacular:3,spectral:1,speech:[0,1,3,4,12],speed:[1,2,4,13],spend:23,spent:21,sphere:[0,27],spin:6,spite:0,spline:8,split:[1,3,4,5,6,8,9,10,11,14,17,21,23,28,29],splite:0,splitter:[1,10],spontan:23,spot:3,spread:[0,11,23,26,27],springer:[18,21,25,26,29],spuriou:13,sqquar:28,sqrsignal:3,sqrt:[0,3,4,5,6,8,10,11,13,17,23,27,28,29],squar:[1,2,3,4,7,8,9,11,13,14,15,16,17,19,20,23,29],squarederror:10,squaredeuclidean:14,squash:12,srtm:[6,21],srtm_data_norway_1:[6,21],stabil:[5,21],stabl:[0,4,5,6,9,11,19,21,26,27,28],stack:[2,3,4],stage:[5,13,21],stai:[0,2,4,5,11,17,26,27],stand:[0,5,9,12,26,27,28],standadscal:27,standard:[0,1,4,5,6,7,8,10,12,15,16,17,20,21,23,26,28],standard_basi:2,standardscal:[0,6,7,8,9,10,11,27,28],stanford:13,start:[0,1,2,3,4,5,6,8,9,10,11,12,13,14,20,23,24,26,27,29],start_box:2,start_nod:2,start_tim:14,stat:[6,29],state:[1,2,4,5,6,7,8,10,11,12,13,19,23,27,28,29],statement:[0,7,20,26],stationari:[],statist:[0,1,3,4,7,9,10,11,12,13,14,18,20,21,22,25,27],statu:[0,7,9,11,26],stavang:[6,21],std:[0,4,6,26,27,29],steep:13,step:[0,1,2,4,6,7,9,10,11,12,13,14,20,21,26],step_fn:[7,12],step_length:13,steps_list:9,stereo:3,still:[0,2,3,5,6,11,13,17,23,29],stimuli:12,stk2100:[25,26],stk3155:[15,21,22,24],stk4021:[25,26],stk4051:[25,26],stk4155:[22,24],stk5000:25,stk:[25,26],stochast:[0,1,5,6,8,11,12,15,16,21,26,28,29],stock:4,stoke:12,stone:[0,7,21],stop:[1,4,9,11,13,14],stopiter:[0,7],storag:[5,27],store:[0,1,2,3,6,11,13,21,23,26],storehaug:[24,26],str:[1,3,4],straight:[0,6,8,13,26,29],straightforward:[0,2,3,5,6,8,9,10,13,20,26,27,28,29],strategi:[0,1,9,26],stratifi:[6,29],strength:[0,5,14,27],stretch:11,strict:[8,13],strictli:[8,13],stride:[4,20],strike:6,string:1,stroke:7,strong:[3,6,9,10,12,20,23,29],strongli:[0,8,19,20],stronli:0,structur:[0,1,2,3,6,9,10,12,19,26,29],stuck:[1,13],student:[0,21,22,24,25,26],studi:[0,3,4,5,6,7,8,11,12,13,17,19,21,25,26,27,28],studier:25,style:[0,7,9,20,26],sub:[9,12],subarg:2,subdivid:[0,20,26],subfield:0,subject:[6,8,23],submit:26,subplot:[0,1,3,4,6,7,8,9,10,14,26,29],subplots_adjust:[8,23],subprogram:[20,26],subract:[0,27],subroutin:[0,26],subscript:1,subsequ:[1,4,5,6,12,20,23,27,29],subset:[1,6,9,12,13,19,26,29],subspac:[0,8,11,27],substanti:[9,10],substep:11,substitut:[3,6,12,20,29],subsubset:9,subtask:6,subtl:1,subtract:[0,4,5,6,11,13,17,20,21,23,28,29],subtre:9,subval:2,succeed:[0,4,26],success:[3,7,9,13,23],successfulli:[4,9],sudo:[0,15,19,21,26],suffer:[0,1,2,5,10,26,27,28],suffici:[1,6,8,11,13,29],suggest:[1,13,21,25],suit:[8,12],suitabl:[0,23,27],sum:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,20,21,23,26,27,28],sum_:[0,1,2,3,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,26,27,28,29],sum_i:[0,2,5,6,8,13,17,21,27,28,29],sum_j:6,sum_ja_:0,sum_k:[6,8,12,20],sum_logist:13,sum_m:3,sum_n:3,sum_nx_:3,summar:[5,6,9,28,29],summari:[1,3,4,10,22,28],summat:[0,3,16,27],sunni:9,superconduct:28,superfici:3,superscript:[1,12],supervis:[0,5,6,7,9,12,19,26,27,29],supplement:[7,21],support:[0,1,9,10,11,13,19,26,27],suppos:[0,5,6,7,8,10,11,12,13,20,26,27,28,29],suppress:[5,8,13,28],sure:[0,1,4,6,21],surf:[6,21],surfac:[0,6,21,26],surpass:6,surpris:[0,26],surround:[3,19],survei:[0,5,6,26,27,28,29],svc:[8,9,10],svd:[0,6,11,17,26,29],svdinv:[5,28],svm:[8,9,10,11],svm_clf:[8,10],swath:[5,27],sy:13,symbol:[1,3,4,5,11,13,14,19,23,26,27,28],symmeteri:1,symmetr:[0,5,8,11,12,13,20,26,27],symmetri:[6,9],sympi:[0,15,19,21,26],synonim:23,syntax:[1,13],syntaxerror:[1,8],system:[0,1,3,4,6,7,9,10,12,13,15,19,20,21,26],systemat:[4,6,29],t0:[3,6,13],t1:[2,13],t2:2,t3:2,t:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,23,24,26,29],t_0:[2,9,13],t_1:13,t_2:[],t_:2,t_b:10,t_i:[1,2,5,12,27],t_j:12,t_k:9,tabl:[9,21,23,24,26],tabul:[0,26],tabular:26,tackl:4,tag:[2,3,4,5,6,7,12,13,14,20,23,27],taht:[0,26],tail:23,tailor:[2,8,11,26],taiwan:[0,26],take:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,16,19,20,23,26,27,28,29],taken:[0,1,3,6,10,13,20,26,29],tan:3,tangent:[1,4,12,13],tanh:[1,4,7,8,12],target:[0,1,2,3,4,5,6,7,8,9,10,11,12,26,27,28,29],target_iscomplex:2,target_meta:2,target_nam:9,target_shap:2,task:[0,1,3,6,9,11,12,14,21,26,29],tau:[3,5,23,28,29],taught:26,tax:0,taylor:[2,13],taylornr:13,tc:8,td:[1,6,26,29],teach:26,team:1,teaser:0,technic:[0,5,6,13,15,21,26,27],techniqu:[0,1,8,10,13,19,23,25,26,27,29],technolog:[0,1],tell:[0,4,6,10,11,13,23,29],temp1:1,temp2:1,temp:1,temperatur:[0,9,26],temporarili:1,ten:[3,17,26],tend:[3,5,6,8,9,10,12,13,14,27,28,29],tendenc:[0,26],tension:[6,29],tensor:3,tensorflow:[0,2,4,8,14,15,19,20,21,22,25,26,27],term1:[5,6,11,21,27],term2:[5,6,11,21,27],term3:[5,6,11,21,27],term4:[5,6,11,21,27],term:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,21,23,26,27,28],termin:[0,4,5,9,10,13,15,26,27],terrain1:[6,21],terrain:[6,21],test:[3,4,5,6,7,8,9,10,13,17,20,21,23,29],test_acc:3,test_accuraci:[1,3],test_error:6,test_imag:[3,4],test_ind:[6,29],test_input:4,test_label:[3,4],test_loss:3,test_pr:1,test_predict:1,test_rnn:4,test_scor:[7,10],test_siz:[0,1,3,5,6,10,26,27,28,29],test_split:9,tester:28,testerror:[0,6,27,29],testi:4,testpredict:4,testx:4,text:[0,1,2,4,5,8,9,11,13,15,20,23,25,27,29],textbook:[16,17,21,27,29],textual:9,textur:[1,9],tf:[1,3,4,13,14],tf_loadlibrari:[3,4,14],th:[0,1,2,5,6,7,9,12,13,14,15,16,20,21,23,26,27,28,29],than:[0,1,2,3,4,5,6,7,9,10,11,12,13,17,19,23,26,27,28,29],thank:[4,6,27,28],theano:[1,19,26],thei:[0,1,2,3,4,5,6,7,8,9,11,12,13,20,21,23,26,27,28,29],them:[0,1,3,4,6,8,9,10,11,12,13,20,21,26,27,28],theme:[0,26],themselv:[0,23,26],thenc:[6,29],theorem:[2,6,7,27],theoret:[0,4,10,26],theori:[0,1,3,8,9,12,13,18,19,21,25,26,28],thereaft:[0,5,6,11,12,15,16,17,20,21,26,29],therebi:[0,5,7,11,26,27,28],therefor:[0,1,2,3,4,6,7,8,11,13,23,26,27,29],therein:11,thereof:[0,6,13,26,29],theta:[1,4,13,23],theta_:[1,13],theta_i:1,theta_linreg:13,theta_t:13,thi:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,19,20,21,22,23,25,27,28,29],thing:[0,1,2,4,5,7,9,23,26,28,29],think:[0,1,3,4,6,9,12,13,14,23,26,29],third:[0,3,6,13,24,26],thirti:7,thorughout:26,those:[0,3,5,6,8,9,10,11,20,21,22,26,27,29],though:[1,2,3,4,13,20,23],thought:[6,14,21,23,29],thousand:[0,1,21,27],three:[0,1,3,5,6,8,9,12,15,20,21,22,23,24,26,27,28,29],threshold:[1,3,9,10,11,12,13],through:[0,1,2,3,4,5,6,8,11,12,13,14,19,20,23,26,27,28,29],throughout:[0,4,5,14,19,20,23,26],thu:[0,1,2,5,6,7,8,10,11,12,13,21,24,26,27,28,29],thumb:[0,6,21,26,27],thursdai:[24,27],tibshirani:[6,18,21,22,25,26,29],tick:6,tick_param:6,ticker:[6,13,21,23],tif:[6,21],tight_layout:[1,7],tightli:11,tild:[0,5,6,7,11,15,16,17,18,21,23,26,27,28,29],till:[0,4,7,8,9,10,12,20,26,27,28],time:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,17,19,20,21,23,26,27,28,29],timeit:4,timer:4,tini:1,tip:3,titl:[0,1,2,3,4,6,7,8,9,10,13,21,23,26,29],tmp:13,tn:[2,3,7],to_categor:[1,3,4],to_categorical_numpi:1,to_numer:[0,6,26,29],todai:3,togeth:[0,3,6,8,11,13,19,26],toi:14,told:13,toler:[2,6,14],tolist:4,tomographi:12,too:[0,2,4,5,6,9,11,13,15,23,25,26,27,29],took:[8,26],tool:[0,1,3,6,13,19,27,29],toolbox:8,top:[0,3,5,6,9,10,18,19,26,28,29],topic:[0,5,6,7,8,19,21,27,28],topograph:21,topolog:[1,3,12],toposort:2,torkjellsdatt:[24,26],toss:[10,23],total:[0,1,2,3,4,6,7,8,9,10,11,12,13,14,15,16,20,23,24,26,27,29],total_loss:4,totalclustervari:14,totalscatt:14,toward:[1,2,7,12,13,21],town:0,tp:[4,7],tpng:9,tpu:[13,19,26],tqdm:6,tr:27,trace:2,trace_stack:2,traceback:[0,2,3,4,6,7,10,13,14,21,26,28],tracer:2,track:[3,13,14,20,27],tract:0,tractabl:[0,26,27],trade:[5,9,28,29],tradeoff:[0,5,21,26,27,28],tradit:[0,1,4,6,26,29],train:[2,3,5,6,8,9,10,11,12,13,17,21,28,29],train_accuraci:[0,1,3,26],train_dataset:4,train_end:[0,1,27],train_error:6,train_imag:[3,4],train_ind:[6,29],train_label:[3,4],train_pr:1,train_siz:[0,1,3,27],train_step:4,train_test_split:[0,1,3,5,6,7,9,10,11,26,27,28,29],train_test_split_numpi:[0,1,27],trainable_vari:4,trained_model:[6,27,28],trainerror:[0,27],traini:4,training_checkpoint:4,training_dataset:4,training_gradi:13,trainingerror:[6,29],trainpredict:4,trainscor:4,trainx:4,trait:[0,26],trajectori:4,transfer:9,transform:[0,5,6,7,8,9,10,11,12,13,19,20,26,27,28,29],transit:[6,12],translat:[1,4,6,10,26,27,28],translate_vjp:2,transpos:[1,5,11,20,27,28],travers:[0,5],treat:[0,1,3,6,12,13,23,26,27,28,29],tree:[0,1,6,19,21,26],tree_clf:[9,10],tree_clf_:9,tree_clf_sr:9,tree_reg1:9,tree_reg2:9,tree_reg:9,trend:23,treue:7,trevor:[18,21,25],tri:[2,3,4,9,13],triain:0,trial:[0,2,4,6,13,23,26,29],triangl:13,triangular:20,trick:[3,4,8,11,13,23],trickier:23,tridiagon:20,trigger:[0,7],trillion:19,trivial:[0,1,5,11,23,26,28],troubl:[0,8,12,27],truck:3,true_beta:[6,27,28],true_divid:1,true_fun:[6,29],trust:9,tucker:8,tuesdai:[24,26,29],tumor:[7,9],tumour:7,tunabl:1,tune:[4,9,13,20,26],tupl:2,turbul:[],turn:[0,1,5,6,7,8,9,10,11,12,13,20,21,23,26,27,29],tutori:[1,4],tv:2,tveito:2,tweak:[1,4,10,23],twice:13,twist:11,two:[0,1,2,4,5,6,7,9,10,11,12,13,15,16,20,22,23,25,26,28,29],tx:13,tx_1:13,txt:4,ty:13,type:[0,1,3,6,8,10,13,16,20,21,23,27,28,29],typeerror:[2,6,13,21],typic:[0,1,2,3,4,5,7,9,10,12,13,23,26,27,28,29],typo:21,u:[0,2,5,6,8,10,11,12,17,20,26,27,28],u_:20,u_i:12,u_m:10,u_t:[],ua:[0,26],ubuntu:[0,15,19,21,26],uci:0,uio:[21,24,25],un:14,unabl:9,unari:[20,26],unary_f:2,unary_oper:2,unary_to_nari:2,unbalanc:[6,9,29],unbias:[0,5,6,26,28,29],unbroadcast:2,unbroadcast_f:2,uncent:[6,28],uncertainti:[0,5,26,28,29],uncertitud:23,unchang:[1,3],uncorrel:[10,23],undefin:[5,27],under:[0,1,5,6,10,13,15,16,19,21,26,27,28,29],underdetermin:[0,26],underfit:[1,6,29],underflowproblem:[5,28,29],undergo:[5,28,29],undergradu:[22,24],underli:[0,1,9,13,23,26],underset:[4,14],understand:[0,1,3,5,6,10,13,14,19,26,27,28],understood:[8,13],undesir:8,undetermin:[5,8,28,29],undo:4,unexpect:[6,13,21,29],unexpected:23,unfair:[6,27,28],unfortun:[1,8,9,10],unicode_liter:[8,9],uniform:[0,1,5,6,11,13,15,16,21,23,26,27],uniformli:[13,23],unifrompdf:23,unimport:13,union:[5,6,28,29],uniqu:[0,2,6,13,14,20,26,29],unique_cluster_label:14,unit:[0,1,3,4,5,10,12,23,26,27,28],unitari:[5,6,20,27],unitarili:[20,26],uniti:23,univari:23,univers:[0,1,2,13,15,19,21,22,24,26,27,28,29],unix:1,unknow:[0,20,26],unknown:[0,1,3,4,5,6,8,10,13,16,20,26,27,28,29],unknowwn:12,unlabel:1,unless:[0,3,6,11,13,21,26,29],unlik:[1,3,8,13],unnecessarili:9,unord:3,unravel:1,unrol:[3,11],unseen:[0,7,9],unstabl:1,unsupervis:[0,1,4,12,19,26],unsymmetr:[20,26],until:[1,2,4,9,12,13,14],untouch:0,unusu:12,up:[1,3,4,5,6,8,10,11,13,14,16,18,19,20,21,23,24,28],updat:[1,2,10,12,13,14],uploa:26,upload:[19,21,25],uploda:[],upon:[0,1,6,7,11,20,21],upper:[0,8,9,16,20,27],uppercas:[20,26],upsampl:4,upscal:4,url:[0,7,26,27],us:[4,5,6,8,9,10,11,12,14,15,16,17,18,20,22,23,25,28,29],usa:26,usag:[0,8,19,26,27],usd10000:0,usd:0,use_bia:4,usecol:[0,26],useless:1,user:[0,1,2,3,4,6,7,8,9,11,14,15,19,20,21,26,27,28],usernam:21,userwarn:6,usetex:23,usg:[6,21],usr:23,usual:[0,3,4,7,12,13,14,26],ut:[5,28],utf:[0,7],util:[1,3,4,6,7,10,14,26,29],ux:20,v0:23,v1:23,v2:23,v:[2,4,5,6,11,13,17,19,27,28],v_0:11,va:1,vahid:26,val:13,val_accuraci:3,val_loss:4,vale:2,valid:[0,1,4,7,9,10,13,19,23,26,27],validation_data:3,validation_split:4,valu:[0,1,2,3,4,6,7,8,9,10,12,13,14,15,16,19,20,21,26],valuat:9,valueerror:2,valy:4,van:[0,18,21,26,27,28,29],vandenbergh:[8,13],vandermond:[0,26],vanilla:[0,6,11,14,27,28],vanish:[1,4,13,23],var_x:23,varabl:8,varepsilon:[5,6,18,21,28,29],varepsilon_:[5,6,28,29],varepsilon_i:[5,6,28,29],vari:[0,1,3,5,6,10,15,16,26,29],variabl:[0,1,2,5,6,7,8,10,11,12,13,14,20,26,27,28,29],varianc:[0,1,5,7,9,10,11,13,14,18,19,20,23,26,27],variance_i:[5,11,27],variance_x:[5,11,27],variant:[0,1,6,8,12,13,26,27],variat:[3,4,11,26],varieti:[0,3,12,15,19,21,26],variou:[1,3,5,6,7,8,9,11,12,13,16,18,19,20,21,23,27,28],varydimens:4,vastli:3,vaue:1,vault:0,vdot:[2,13],ve:21,vec:[6,29],vector:[0,1,2,3,4,5,6,7,9,10,11,13,14,16,17,19,28,29],vector_mean:14,veloc:[],ventur:[0,8,15,19,26],verbos:[1,3,4],veri:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,21,23,25,26,27,28,29],verifi:[3,11,20,26],versatil:[8,26],versicolor:[8,9],version:[0,3,10,13,14,15,19,20,21,23,26],versu:1,vert:[0,1,5,6,7,8,9,11,13,17,26,27,28,29],vert_1:[5,6,27,28,29],vert_2:[5,6,11,17,27,28,29],via:[0,5,6,7,8,9,10,11,12,15,19,20,21,22,23,24,26,27,28,29],vidal:11,video:[0,1,12,19,22,24,26,27,28,29],view:[1,3,5,6,12,13,23,25,26,28,29],violat:8,virginica:9,viridi:[0,1,2,3,26],virtual:1,viscos:13,viscou:13,visibledeprecationwarn:2,vision:[0,3],visual:[0,3,11,12,15,19,26,27],visualis:1,viz:[6,8,23],vjp:2,vjp_argnum:2,vjpfun:2,vjpmaker:2,vjpnode:2,vjps_dict:2,vmap:13,vmax:[1,6],vmin:[1,6],voic:3,volum:[0,3,26],vote:10,voting_clf:10,votingclassifi:10,votingsimpl:10,vs:[0,4,6,29],vspace:2,vstack:[5,11,20,23,26,27],vt:[5,27,28],w1:8,w2:[8,11],w3:8,w:[0,1,2,3,4,5,6,7,8,10,11,12,13,14,20,23,26,27,29],w_1:[8,20],w_1x_1:8,w_1x_:8,w_2:[8,20],w_2x_2:8,w_2x_:8,w_3:20,w_4:20,w_:[1,12],w_hidden:2,w_i:[1,2,10],w_ix_i:12,w_j:20,w_m:20,w_output:2,w_px_:8,w_px_p:8,wa:[0,1,3,4,5,6,7,10,11,12,14,17,20,21,26,27,28,29],wai:[0,1,2,3,4,5,6,7,8,10,11,12,13,14,17,20,23,26,27,28],walk:9,walker:23,wang:[0,26],want:[0,1,2,3,4,5,6,8,9,10,11,12,13,14,15,16,17,19,21,23,26,27,28,29],warn:[1,4,8],warrant:[6,29],wast:3,watch:19,wave:3,wavelet:8,we:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,27,28,29],weak:[9,10,14],weather:[1,12],web:[19,22,24,26],weblink:[],webpag:26,websit:[6,20,21,22,26],wedg:[8,23],wednesdai:[24,26,29],wee:11,week:[0,5,6,7,21,22,24],weekli:[16,19,21,24,25,26],weight:[0,1,2,3,6,7,9,10,12,13,21,23],weigth:2,welcom:[8,19],well:[0,1,2,3,4,5,6,7,8,9,10,12,13,15,16,19,20,21,23,25,26,27,28,29],went:8,were:[0,1,3,4,5,6,7,8,10,11,12,14,23,26,27,29],wessel:[0,18,21,26,27,28,29],what:[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,19,20,21,23],whatev:3,when:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,17,18,20,21,23,26,27,28,29],whenev:[13,23],where:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,23,24,26,27,28,29],wherea:[6,23,29],wherefrom:21,wherein:[1,12],whether:[0,3,5,7,9,21,23,26,28],which:[0,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,26,27,28,29],whichev:[1,3],white:9,whiteboard:28,who:[0,22],whole:[1,3,4,5,9,11,13,28,29],whose:[0,6,10,23,27,29],whow:[11,27],whrn:[],why:[0,1,3,6,13,21,28],wide:[0,1,3,6,7,12,15,19,20,21,26,29],widehat:[6,29],width:[0,3,8,9,26],wieringen:[0,18,21,26,27,28,29],win:10,wind:9,wing:[24,26],winther:2,wiothout:[6,27],wiscons:7,wisconsin:10,wisdom:[6,28],wise:[0,1,5,12,13,27],wish:[0,2,5,7,8,11,13,14,20,21,26,27],with_std:[0,27],wither:6,within:[0,2,3,4,7,9,12,13,14,23,25,26],withinclust:14,without:[0,1,5,6,8,9,11,12,13,26,27,28,29],won:[0,26],wonder:8,word:[0,1,3,4,5,6,7,14,23,26,27,28,29],work:[0,1,4,6,7,8,9,13,15,19,21,22,23,24,26,27,29],workshop:26,world:[0,8,27],worldwid:[0,26],wors:[0,1,3,4,6,26,29],worst:9,worth:9,worthi:21,would:[0,1,3,5,6,7,8,9,10,11,12,13,20,21,23,26,27,28,29],wrap:[6,20,26],wrap_util:2,write:[0,1,2,3,5,6,7,8,12,13,15,16,17,20,26,27,29],written:[0,2,3,5,11,12,13,16,19,20,21,23,26,27,28,29],wrong:[1,8],wrongli:10,wrote:[5,11,27],wrt:[2,10,13],wth:[10,13],www:[19,20,21,25,26,28],wx_1:8,x0:8,x1:[4,8,9,10,13],x1_exampl:8,x1d:8,x2:[8,9,10,13],x2d:[8,11],x2d_train:11,x2dsl:11,x3:[8,9],x4:9,x:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,26,29],x_0:[0,5,11,20,26,27,28,29],x_1:[0,2,5,6,7,8,9,10,11,13,20,23,26,27,28,29],x_2:[0,2,5,6,7,8,9,10,11,13,20,23,26,27,29],x_3:[8,20,23],x_4:20,x_:[0,2,3,5,6,8,10,11,13,14,18,20,21,23,26,27,28,29],x_center:11,x_data:1,x_data_ful:1,x_hidden:2,x_i:[0,1,2,5,6,7,8,9,10,11,12,13,14,17,20,23,26,27,28,29],x_input:2,x_ix_:[0,26],x_iy_i:8,x_j:[0,2,8,9,12,16,23,27],x_jy_j:8,x_k:[12,14,20,23,27],x_l:23,x_m:[6,12,20,23,29],x_n:[0,2,3,6,8,11,12,13,20,23,26,29],x_new:[9,10],x_offset:[6,27,28],x_output:2,x_p:[3,7,9],x_poli:9,x_poly10:9,x_pred:4,x_prev:2,x_reduc:11,x_scale:8,x_small:13,x_test:[0,1,3,5,6,7,9,10,11,26,27,28,29],x_test_own:6,x_test_scal:[0,6,7,9,10,11,27,28],x_tot:4,x_train:[0,1,3,4,5,6,7,9,10,11,26,27,28,29],x_train_mean:[6,27,28],x_train_own:6,x_train_scal:[0,6,7,9,10,11,27,28],x_val:1,xarrai:[19,26],xavier:1,xbkjd4ccgq:26,xbnew:13,xcode:[0,15,19,21,26],xdclassiffierconfus:10,xdclassiffierroc:10,xg_clf:10,xgb:10,xgbclassifi:10,xgboost:9,xgboot:10,xgbregressor:10,xgparam:10,xgtree:10,xi:[8,13],xi_1:8,xi_:8,xi_i:8,xinv:28,xk:8,xla:[13,19,26],xlabel:[0,1,2,3,4,5,6,7,8,9,10,13,21,23,26,28,29],xlim:[6,10,29],xm:9,xmesh:13,xnew:[0,13,26],xp:23,xpanda:[0,27],xpd:[5,11,27],xplot:0,xs:9,xscale:[0,27],xsr:9,xt_x:13,xtest:[6,29],xtick:[3,6,8,9,29],xtrain:[6,29],xu:[0,26],xx:[0,20,26],xy:[0,6,8,20,21,26],xytext:8,xz:[20,26],y1:4,y2:4,y3:4,y:[0,1,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21,23,26,27,28,29],y_0:[0,5,11,20,26,27,28,29],y_1:[0,5,8,9,11,13,20,26,27,28,29],y_1y_1:8,y_1y_1k:8,y_1y_2:8,y_1y_2k:8,y_1y_n:8,y_1y_nk:8,y_2:[0,5,8,9,11,20,26,27],y_2y_1:8,y_2y_1k:8,y_2y_2:8,y_2y_2k:8,y_3:[0,9,20],y_4:20,y_:[0,1,5,6,10,11,20,26,27,28,29],y_data:[0,1,5,6,26,27,28],y_data_ful:1,y_decis:8,y_fit:[0,27],y_i:[0,1,5,6,7,8,9,10,11,12,13,15,16,17,18,20,21,26,27,28,29],y_if_:10,y_ix_:[0,26],y_ix_i:[7,8,13,27],y_iy_jk:8,y_j:[6,8,12,21,29],y_k:12,y_m:20,y_model:[0,4,5,6,26,27,28],y_n:[8,13],y_ny_1:8,y_ny_1k:8,y_ny_2:8,y_ny_2k:8,y_ny_n:8,y_ny_nk:8,y_offset:[6,27,28],y_plot:9,y_pred1:9,y_pred2:9,y_pred:[0,1,4,6,7,8,9,10,27,28,29],y_pred_rf:10,y_pred_tre:10,y_proba:[7,10],y_scaler:[6,28],y_test:[0,1,3,4,5,6,7,9,10,11,26,27,28,29],y_test_onehot:1,y_test_predict:0,y_test_scal:[],y_tot:4,y_train:[0,1,3,4,5,6,7,9,10,11,26,27,28,29],y_train_mean:[6,27,28],y_train_onehot:1,y_train_predict:0,y_train_scal:[6,28],y_val:1,yadav:[],ye:[3,6,7,29],year:[0,19,26],yet:[0,1,6,8,11,13,26],yi:13,yield:[0,2,5,6,7,8,10,12,13,14,20,23,26,28,29],yk:8,ylabel:[0,1,2,3,4,5,6,7,8,9,10,13,21,23,26,28,29],ylim:[3,6,29],ym:9,ymesh:13,yn:0,yo:[8,9,10],yoshua:[1,25],you:[0,1,2,3,4,5,6,8,9,10,11,13,15,16,17,19,20,21,23,24,25,26,27,28,29],young:0,your:[1,2,4,5,6,8,11,13,17,19,20,21,28,29],yourself:[11,13,26],youtub:19,ypred:[6,29],ypredict2:13,ypredict:[0,13,26,27],ypredictlasso:[5,28],ypredictol:[0,5,28],ypredictown:[6,27,28],ypredictownridg:[6,28],ypredictridg:[0,5,6,28],ypredictskl:[6,27,28],yridg:26,ys:9,ytest:[6,29],ytick:[3,6,8,9,29],ytild:[0,6,26,27,29],ytilde_test_ol:[],ytilde_test_ridg:[],ytildelasso:[5,28],ytildenp:[0,26,27],ytildeol:[0,5,28],ytildeownridg:[6,28],ytilderidg:[5,6,28],ytrain:[6,29],yuxi:26,yx:[20,26],yy:[20,26],yz:[20,26],z:[0,1,2,3,4,5,6,7,8,9,11,12,13,20,21,23,26,27,29],z_0:[20,26],z_1:[20,26],z_2:[20,26],z_:[1,2,12,20,26],z_c:1,z_h:1,z_hidden:2,z_i:[1,12],z_j:[1,12],z_k:[12,27],z_m:1,z_mod:9,z_o:1,z_output:2,zaman:23,zaxi:[6,21],zero:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,17,18,20,21,23,26,27,28,29],zeros_lik:4,zeroth:27,zfill:4,zip:[2,4,6],zm_h:[0,26],zn:0,zone:0,zoom:26,zx:[20,26],zy:[20,26],zz:[20,26]},titles:["3. Linear Regression","14. Building a Feed Forward Neural Network","15. Solving Differential Equations with Deep Learning","16. Convolutional Neural Networks","17. Recurrent neural networks: Overarching view","4. Ridge and Lasso Regression","5. Resampling Methods","6. Logistic Regression","8. Support Vector Machines, overarching aims","9. Decision trees, overarching aims","10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods","11. Basic ideas of the Principal Component Analysis (PCA)","13. Neural networks","7. Optimization, the central part of any Machine Learning algortithm","12. Clustering and Unsupervised Learning","Exercises week 34","Exercises week 35","Exercises week 36","Exercises week 37","Applied Data Analysis and Machine Learning","2. Linear Algebra, Handling of Arrays and more Python Features","Project 1 on Machine Learning, deadline October 7 (midnight), 2024","Teaching schedule with links to material","1. Elements of Probability Theory and Statistical Data Analysis","Teachers and Grading","Textbooks","Week 34: Introduction to the course, Logistics and Practicalities","Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression","Week 36: Linear Regression and Statistical interpretations","Week 37: Statistical interpretations and Resampling Methods"],titleterms:{"1":[0,15,16,17,18,21,26,27],"14":[],"17":[],"18":[],"2":[0,15,16,17,18,26,27,28],"2023":24,"2024":21,"26":27,"3":[0,15,16,26,27],"34":[15,26],"35":[16,27],"36":[17,28],"37":[18,29],"4":[0,27],"5":0,"7":21,"9":29,"case":[8,10,23,27,28],"do":[1,28,29],"final":[12,27,28],"function":[0,1,6,7,8,10,11,12,13,21,23,26,27,28,29],"import":[5,20,26,27,28],"new":[4,28,29],A:[0,1,4,8,9,26,28,29],And:[26,27,28],For:27,In:24,Ising:6,The:[0,1,2,3,5,6,7,8,9,11,12,17,19,26,27,28,29],To:[26,27],With:[4,28],about:[26,27],abov:28,activ:[1,12,28],ad:[0,6,17,21,26,27],adaboost:10,adagrad:13,adam:13,adapt:10,adjust:1,adversari:4,again:[3,9],ai:[21,26],aim:[8,9,17,18,26],aka:[26,27],algebra:[20,26],algorithm:[9,10,11,12,27],algortithm:13,all:8,altern:[],an:[0,4,10,26],analys:[5,27],analysi:[0,5,6,11,19,21,23,26,27,28,29],analyt:[0,16,17],analyz:[],ani:13,anoth:[9,28,29],appli:19,approach:[0,8,14,26,29],approxim:12,architectur:1,arrai:[20,26],assist:24,assumpt:[28,29],august:27,autocorrel:23,autograd:[2,13],automat:13,b:[17,21],back:[1,11,12],background:[19,21,29],bag:10,base:[13,29],basic:[0,5,7,9,10,11,20,27,28,29],batch:1,bay:[5,28,29],befor:11,beta:[28,29],better:8,bia:[6,21,29],binari:1,bind:26,bird:10,boldsymbol:[27,28,29],boost:10,bootstrap:[6,10,29],boston:0,breast:1,brief:[26,29],bring:12,build:[1,3,9],c:[21,26],calcul:27,can:[26,29],cancer:[1,7,9,11],cart:9,center:27,central:[13,19,23,29],chain:12,chang:10,channel:26,chi:[0,26],choos:1,cifar01:3,classic:11,classif:[1,9,10],classifi:8,clip:1,cluster:14,cnn:3,code:[0,1,2,5,9,11,12,13,14,26,27,28,29],collect:[1,3],commun:26,compar:[2,10],comparison:28,complet:27,complex:[0,6,21,27],complic:6,compon:11,comput:9,computation:29,computerlab:26,con:9,concept:23,condit:[28,29],confid:29,conjug:13,contn:26,convex:[8,13],convolut:[3,12],correctli:[28,29],correl:[11,27],cost:[1,10,27,28,29],cours:[19,25,26],covari:[5,11,23,27],cover:26,critic:[],cross:[6,21,29],cython:26,d:21,data:[0,1,3,6,7,9,11,15,16,19,21,23,26,27,28],dataset:[1,3],deadlin:[21,26],decai:2,decemb:[],decis:[9,10],decomposit:[5,11,17,20,27],deep:[1,2,26],defin:[1,26],degre:[0,27],deliveri:21,delta:29,dens:[0,26],deriv:[5,12,27,28,29],descent:[2,10,13],descript:21,design:27,detail:[3,26],develop:1,diagon:11,differ:8,differenti:[2,13],diffus:2,dimension:[2,3,8,21,27],disadvantag:9,discret:23,distribut:[5,23,28,29],distrubut:29,doe:[27,28],domain:23,down:1,dropout:1,e:21,economi:27,electron:21,element:[0,23,26],elimin:20,energi:26,ensembl:10,entropi:9,environ:[0,15,26],equat:[0,2,12,26,27,28],error:[0,10,26,27,29],essenti:26,estim:[28,29],etc:26,euler:2,evalu:1,exampl:[0,1,2,3,4,6,7,8,9,10,26,27,28,29],exercis:[0,6,15,16,17,18,26,27],expect:[18,23,28,29],expens:29,experi:23,explor:[0,15,16,26],exponenti:2,express:[17,18,27],extrapol:4,extrem:[10,26],ey:10,f:21,fall:24,famili:[1,26,27],famou:20,fantast:27,featur:[9,20,27],feed:[1,12],find:29,fine:1,first:[4,12,26,27,28],fit:[0,10,26,28],fix:27,fold:29,forc:3,forest:10,format:[21,26],forward:[1,2,12],fourier:3,frank:[6,21,27],freedom:[0,27],frequent:27,frequentist:[0,26],from:[5,10,12,27,28,29],full:2,further:[3,5,27],g:21,gan:4,gaussian:20,gd:13,gener:[4,9,26],geometr:11,gini:9,good:[0,26],grade:[24,26],gradient:[1,2,10,13],growth:2,ha:19,handl:[20,26,27],happen:[28,29],hessian:27,hidden:2,histogram:29,hous:0,hyperparamet:1,hyperplan:8,i:1,id3:9,idea:11,ident:[28,29],identifi:29,ii:26,iid:[28,29],illustr:28,implement:1,implic:[5,27],improv:1,includ:13,increment:11,independ:[28,29],index:9,inform:24,input:2,instal:[19,21,26],instructor:24,intercept:27,interpret:[5,11,26,27,28,29],interv:29,introduc:[11,13,27],introduct:[0,6,19,20,21,26],invers:[5,20,28],invert:27,iter:10,its:27,jacobian:27,jax:13,julia:26,jungl:10,k:29,kera:[1,3],kernel:[8,11],lab:29,lagrangian:8,lasso:[5,6,21,27,28,29],last:27,later:[5,27],layer:[1,2,3,12],learn:[0,1,2,11,13,14,15,16,19,21,26,27,28,29],least:[5,6,18,21,26,27,28],lectur:[26,27,28,29],level:10,librari:[19,26],likelihood:[7,28,29],limit:[1,13,23,29],linear:[0,8,13,20,26,27,28],link:[5,11,22,25,27,28,29],literatur:21,logist:[7,26],loss:27,lu:20,machin:[0,8,13,19,21,26],made:[28,29],main:[23,26],make:[0,9,10,15,16,26,27],mani:[10,12],manipul:27,margin:[28,29],mass:26,materi:[21,22,26,27,28,29],math:[5,27],mathemat:[3,5,8,27],matric:[5,20,26,28],matrix:[1,5,11,12,20,26,27,28],matter:[0,26],max:27,maximum:[28,29],mean:[0,27,28],meet:[5,10,23,26,27],mercer:8,method:[6,9,10,13,21,26,29],midnight:21,min:27,minim:26,ml:26,mle:[28,29],mlp:12,mnist:[3,4],model:[0,1,4,6,12,26],momentum:13,mondai:[27,28,29],moon:[8,9],more:[3,6,20,21,26,27,28,29],multilay:12,multipl:[1,3],multipli:8,need:[21,26],network:[1,2,3,4,7,12,26],neural:[1,2,3,4,7,12,26],non:8,normal:[0,1,29],notat:12,note:[21,27,28],novemb:[],now:[1,9,13,28,29],nuclear:[0,26],numba:26,number:[0,2,23,27],numer:[2,21,23],numpi:[20,26],object:3,obtain:11,octob:21,od:2,off:[6,21],ol:[5,6,21,28,29],one:[2,12],oper:20,optim:[1,8,13,19,26,27],order:13,ordinari:[5,6,18,21,26,27,28],organ:[0,26],oslo:25,other:[4,9,11,12,20,21,26,27],our:[0,4,5,11,13,26,27],outcom:[19,26],output:2,overarch:[0,4,8,9,17,18,26,27],overview:[10,26],own:[0,10,11,15,16,26,27],packag:[20,26],panda:[26,27],paper:21,paramet:[26,27],part:[13,19,21,27],partial:2,pass:1,path:[],pca:11,pdf:23,pencil:21,perceptron:12,perform:[1,9],period:3,perspect:1,plan:[27,28,29],plot:29,point:4,poisson:2,polynomi:[3,28],popul:2,popular:26,practic:[13,24,26],pre:[1,3],preambl:21,predict:4,preprocess:27,prerequisit:[3,19,26],princip:11,principl:3,pro:9,probabl:[5,23,28,29],problem:[1,2,13,26,27,28],procedur:[9,26],process:[1,3],program:[2,13,21],project:[6,21,26],prop:13,propag:[1,12],properti:[5,23,27],python:[0,9,15,19,20,26],quick:8,r:26,random:[10,11,23],read:[9,26,27,29],real:[6,21,26],recommend:[26,27],rectangular:28,recurr:[4,12],reduc:[0,27],reduct:3,refer:21,reformul:2,regress:[0,5,6,7,9,10,13,17,18,21,26,27,28,29],regular:1,relat:27,relev:[25,27],relu:1,remark:3,remind:[6,8,26],repeat:27,replac:13,report:21,repositori:29,requir:[2,19],resampl:[6,21,29],rescal:[6,28],residu:27,resourc:2,result:[27,28],revisit:13,rewrit:[26,27,29],rgeress:[],ridg:[0,5,6,17,18,21,27,28,29],rm:13,rule:12,s:[8,10,29],same:[13,29],sampl:11,scale:27,schedul:[22,26],schemat:9,scheme:2,scienc:26,scikit:[0,1,11,15,16,26,27,28,29],second:13,semest:24,septemb:[28,29],session:[27,28,29],set:[0,2,3,9,12,15,26,27],sgd:13,should:1,similar:13,simpl:[0,4,9,13,26,27,28],singl:10,singular:[5,11,17,27],size:27,soft:8,softmax:1,softwar:[21,26],solv:[2,28],solver:13,some:[13,20,27],specifi:2,split:[0,15,16,26,27],squar:[0,5,6,10,18,21,26,27,28],standard:[13,27,29],state:[0,26],statist:[5,6,19,23,26,28,29],steepest:[10,13],step:29,still:27,stochast:[13,23],strongli:26,structur:[],subtract:27,suggest:26,sum:29,summari:[24,26],superposit:3,supervis:1,support:8,svd:[5,27,28],systemat:3,t:[27,28],teach:[22,24],teacher:[24,26],technic:28,techniqu:[6,11,21],technolog:19,tensorflow:[1,3],tent:26,term:29,test:[0,1,15,16,26,27,28],text:26,textbook:[25,26],theorem:[5,8,11,12,23,28,29],theori:23,thi:[17,18,26],think:27,thursdai:[],tip:13,togeth:12,tool:[21,26],top:1,topic:26,toward:11,trade:[6,21],tradeoff:[6,29],train:[0,1,4,15,16,26,27],transform:3,tree:[9,10],tuesdai:28,tune:1,two:[3,8,19,21,27],type:[2,4,12,26],uio:26,understand:29,univers:[12,25],unsupervis:14,up:[0,2,9,12,15,26,27,29],us:[0,1,2,3,7,13,19,21,26,27],usag:[28,29],valid:[6,21,29],valu:[5,11,17,18,23,27,28,29],variabl:23,varianc:[6,21,28,29],variou:[0,15,26,29],vector:[8,12,20,26,27],view:[0,4,10,27],visual:[1,9],vs:3,wai:[9,29],wave:2,we:26,wednesdai:28,week:[15,16,17,18,26,27,28,29],weekli:22,what:[0,26,27,28,29],which:1,why:[26,27,29],wisconsin:7,wrap:[27,29],write:[4,11,21,28],x:[27,28],xgboost:10,yet:28,your:[0,10,15,16,26,27],yourself:[]}})
\ No newline at end of file
diff --git a/doc/LectureNotes/_build/html/week37.html b/doc/LectureNotes/_build/html/week37.html
index 32a4cc488..cd478a242 100644
--- a/doc/LectureNotes/_build/html/week37.html
+++ b/doc/LectureNotes/_build/html/week37.html
@@ -372,41 +372,18 @@ const thebe_selector_output = ".output, .cell_output"
-
+
@@ -2436,12 +2284,12 @@ Mean squared error on test data: 1.53738247
Degree of polynomial: 8
Mean squared error on training data: 0.04926746
Mean squared error on test data: 0.14629156
-Degree of polynomial: 9
-Mean squared error on training data: 0.02546675
-Mean squared error on test data: 0.11202337
-
Degree of polynomial: 10
+
Degree of polynomial: 9
+Mean squared error on training data: 0.02546675
+Mean squared error on test data: 0.11202337
+Degree of polynomial: 10
Mean squared error on training data: 0.02424794
Mean squared error on test data: 0.22467274
Degree of polynomial: 11
@@ -2450,21 +2298,23 @@ Mean squared error on test data: 1.07641937
Degree of polynomial: 12
Mean squared error on training data: 0.00805074
Mean squared error on test data: 0.04295757
-Degree of polynomial: 13
+
+
+
Degree of polynomial: 13
Mean squared error on training data: 0.00781918
Mean squared error on test data: 0.56965674
Degree of polynomial: 14
Mean squared error on training data: 0.00465099
Mean squared error on test data: 0.28443039
-
-
-
Degree of polynomial: 15
+Degree of polynomial: 15
Mean squared error on training data: 0.00420072
Mean squared error on test data: 568.47051432
Degree of polynomial: 16
Mean squared error on training data: 0.00325450
Mean squared error on test data: 48.97630233
-Degree of polynomial: 17
+
+
+
Degree of polynomial: 17
Mean squared error on training data: 0.00242954
Mean squared error on test data: 2.52780600
Degree of polynomial: 18
@@ -2473,12 +2323,12 @@ Mean squared error on test data: 429.25695398
Degree of polynomial: 19
Mean squared error on training data: 0.00154853
Mean squared error on test data: 239.97065359
-
-
-
Degree of polynomial: 20
+Degree of polynomial: 20
Mean squared error on training data: 0.00140846
Mean squared error on test data: 1350.24493666
-Degree of polynomial: 21
+
+
+
Degree of polynomial: 21
Mean squared error on training data: 0.00119688
Mean squared error on test data: 1840.50530832
Degree of polynomial: 22
@@ -2487,12 +2337,12 @@ Mean squared error on test data: 1184.60929685
Degree of polynomial: 23
Mean squared error on training data: 0.00089193
Mean squared error on test data: 3892.17483760
-
-
-
Degree of polynomial: 24
+Degree of polynomial: 24
Mean squared error on training data: 0.00083355
Mean squared error on test data: 1332.46736215
-Degree of polynomial: 25
+
+
+
Degree of polynomial: 25
Mean squared error on training data: 0.00079904
Mean squared error on test data: 7577.76690383
Degree of polynomial: 26
@@ -2501,23 +2351,23 @@ Mean squared error on test data: 1079.36895644
Degree of polynomial: 27
Mean squared error on training data: 0.00068091
Mean squared error on test data: 3207.25343155
-Degree of polynomial: 28
-Mean squared error on training data: 0.00063362
-Mean squared error on test data: 674.79633065
-
Degree of polynomial: 29
+
Degree of polynomial: 28
+Mean squared error on training data: 0.00063362
+Mean squared error on test data: 674.79633065
+Degree of polynomial: 29
Mean squared error on training data: 0.00063866
Mean squared error on test data: 3099.60342978
-
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_85796/626635268.py:73: RuntimeWarning: divide by zero encountered in log10
+
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87621/626635268.py:73: RuntimeWarning: divide by zero encountered in log10
plt.plot(polynomial, np.log10(trainingerror), label='Training Error')
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_85796/626635268.py:74: RuntimeWarning: divide by zero encountered in log10
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87621/626635268.py:74: RuntimeWarning: divide by zero encountered in log10
plt.plot(polynomial, np.log10(testerror), label='Test Error')
-
+
Note that we kept the intercept column in the fitting here. This means that we need to set the intercept in the call to the Scikit-Learn function as False. Alternatively, we could have set up the design matrix \(X\) without the first column of ones.
@@ -2598,368 +2448,165 @@ Mean squared error on test data: 3099.60342978
-
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_85796/3817475779.py:63: RuntimeWarning: divide by zero encountered in log10
+
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87621/3817475779.py:63: RuntimeWarning: divide by zero encountered in log10
plt.plot(polynomial, np.log10(estimated_mse_sklearn), label='Test Error')
The programs here use both ordinrary least squares (OLS) and Ridge
-regression with one value only for the hyperparameter \(\lambda\). The
-first example has no scaling and includes the intercept as well and we
-are trying to fit a second-order polynomial. The second code takes out
-the intercept and subtracts the mean values of each column of the
-design matrix and the mean value of the outputs.
-
The third and final code uses Scikit-Learn as library in order to
-calculate the optimal parameters for OLS and Ridge regression. Note
-that it is highly recommended to not include the intercept in Ridge
-and Lasso regression, in order to avoid penalizing the optimization by
-the intercept. The second and third codes do thus not include the
-intercept. In the second code we do the scaling ourselves while the
-last code uses the standard scaler option included in Scikit-Learn, known as centering (where
-we subtract the mean values).
-
-
-
importmatplotlib.pyplotasplt
-importnumpyasnp
-fromsklearn.linear_modelimportLinearRegression
-fromsklearn.preprocessingimportPolynomialFeatures
-fromsklearn.model_selectionimporttrain_test_split
-fromsklearn.preprocessingimportStandardScaler
-
-defMSE(y_data,y_model):
- n=np.size(y_model)
- returnnp.sum((y_data-y_model)**2)/n
-
-defOLS_fit_beta(X,y):
- returnnp.linalg.pinv(X.T@X)@X.T@y
-
-defRidge_fit_beta(X,y,L,d):
- I=np.eye(d,d)
- returnnp.linalg.pinv(X.T@X+L*I)@X.T@y
-
-# Same random numbers for each test.
-np.random.seed(2018)
-n=100
-d=3
-# hyperparameter lambda
-Lambda=0.01
-
-# Make data set, simple second-order polynomial
-x=np.linspace(-3,3,n)
-y=2.0+0.5*x+5.0*(x**2)+np.random.randn(n)
-
-# The design matrix X includes the intercept and no scaling is made
-X=np.zeros((len(x),d))
-forpinrange(d):
- X[:,p]=x**(p)
-
-
-#Split data, no scaling is used and we include the intercept
-X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2)
-
-
-#Calculate beta, own code
-beta_OLS=OLS_fit_beta(X_train,y_train)
-beta_Ridge=Ridge_fit_beta(X_train,y_train,Lambda,d)
-print(beta_OLS)
-print(beta_Ridge)
-#predict value
-ytilde_test_OLS=X_test@beta_OLS
-ytilde_test_Ridge=X_test@beta_Ridge
-
-#Calculate MSE
-print(" ")
-print("test MSE of OLS:")
-print(MSE(y_test,ytilde_test_OLS))
-print(" ")
-print("test MSE of Ridge")
-print(MSE(y_test,ytilde_test_Ridge))
-
-plt.scatter(x,y,label='Data')
-plt.plot(x,X@beta_OLS,'*',label="OLS_Fit")
-plt.plot(x,X@beta_Ridge,label="Ridge_Fit")
-plt.grid()
-plt.legend()
-plt.show()
-
Linking the regression analysis with a statistical interpretation¶
+
We will now couple the discussions of ordinary least squares, Ridge
+and Lasso regression with a statistical interpretation, that is we
+move from a linear algebra analysis to a statistical analysis. In
+particular, we will focus on what the regularization terms can result
+in. We will amongst other things show that the regularization
+parameter can reduce considerably the variance of the parameters
+\(\beta\).
+
The
+advantage of doing linear regression is that we actually end up with
+analytical expressions for several statistical quantities.
+Standard least squares and Ridge regression allow us to
+derive quantities like the variance and other expectation values in a
+rather straightforward way.
+
It is assumed that \(\varepsilon_i
+\sim \mathcal{N}(0, \sigma^2)\) and the \(\varepsilon_{i}\) are
+independent, i.e.:
The randomness of \(\varepsilon_i\) implies that
+\(\mathbf{y}_i\) is also a random variable. In particular,
+\(\mathbf{y}_i\) is normally distributed, because \(\varepsilon_i \sim
+\mathcal{N}(0, \sigma^2)\) and \(\mathbf{X}_{i,\ast} \, \boldsymbol{\beta}\) is a
+non-random scalar. To specify the parameters of the distribution of
+\(\mathbf{y}_i\) we need to calculate its first two moments.
+
Recall that \(\boldsymbol{X}\) is a matrix of dimensionality \(n\times p\). The
+notation above \(\mathbf{X}_{i,\ast}\) means that we are looking at the
+row number \(i\) and perform a sum over all values \(p\).
-
-
[1.79934087 0.47179152 5.01549939]
-[1.79909592 0.47176716 5.01550546]
-
-test MSE of OLS:
-1.139431112903922
-
-test MSE of Ridge
-1.1395235273363669
-
-
-
-
-
-
In this example we do not include the intercept and we scale the data by subtracting the mean values. This follows the discussion in the lecture material.
-see also the weekly slides for week 36.
-It is recommended whrn we use Ridge and Lasso regression to not include the intercept in the optimization process.
-
Before we discuss the code, we repeat some of the basic math from the slides of week 36.
-
Let us try to understand what this may imply mathematically when we
-subtract the mean values, also known as zero centering or simply centering. For
-simplicity, we will focus on ordinary regression, as done in the above example.
The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)
+that there exists a function \(f(\boldsymbol{x})\) and a normal distributed error \(\boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2)\)
+which describe our data
Recall also that we use the squared value. This expression can lead to an
-increased penalty for higher differences between predicted and
-output/target values.
-
What we have done is to single out the \(\beta_0\) term in the
-definition of the mean squared error (MSE). The design matrix \(X\)
-does in this case not contain any intercept column. When we take the
-derivative with respect to \(\beta_0\), we want the derivative to obey
+
We approximate this function with our model from the solution of the linear regression equations, that is our
+function \(f\) is approximated by \(\boldsymbol{\tilde{y}}\) where we want to minimize \((\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\), our MSE, with
the mean value for all elements of the column vector \(\boldsymbol{x}_j\).
-
Replacing \(y_i\) with \(y_i - y_i - \overline{\boldsymbol{y}}\) and centering also our design matrix results in a cost function (in vector-matrix disguise)
np.random.seed(2018)
-n=100
-# we do not include the intercept
-d=2
-Lambda=0.01
-
-# Make data set.
-x=np.linspace(-3,3,n)
-y=2.0+0.5*x+5.0*(x**2)+np.random.randn(n)
-
-#Design matrix X does not include the intercept.
-X=np.zeros((len(x),d))
-forpinrange(d):
- X[:,p]=x**(p+1)
-
-
-#Split data in train and test
-X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2)
-
-# Scale data by subtracting mean value,own implementation
-#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable
-X_train_mean=np.mean(X_train,axis=0)
-#Center by removing mean from each feature
-X_train_scaled=X_train-X_train_mean
-X_test_scaled=X_test-X_train_mean
-#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered, note)
-y_scaler=np.mean(y_train)
-y_train_scaled=y_train-y_scaler
-
-
-#Calculate beta
-beta_OLS=OLS_fit_beta(X_train_scaled,y_train_scaled)
-beta_Ridge=Ridge_fit_beta(X_train_scaled,y_train_scaled,Lambda,d)
-print(beta_OLS)
-print(beta_Ridge)
-# calculate intercepts and print them
-interceptOLS=y_scaler-X_train_mean@beta_OLS
-interceptRidge=y_scaler-X_train_mean@beta_Ridge
-print(interceptOLS)
-print(interceptRidge)
-
-#predict value with intercept
-ytilde_test_OLS=X_test_scaled@beta_OLS+y_scaler
-ytilde_test_Ridge=X_test_scaled@beta_Ridge+y_scaler
-
-
-#Calculate MSE
-
-print(" ")
-print("test MSE of OLS:")
-print(MSE(y_test,ytilde_test_OLS))
-print(" ")
-print("test MSE of Ridge")
-print(MSE(y_test,ytilde_test_Ridge))
-
-plt.scatter(x,y,label='Data')
-plt.plot(x,X@beta_OLS+interceptOLS,'*',label="OLS_Fit")
-plt.plot(x,X@beta_Ridge+interceptRidge,label="Ridge_Fit")
-plt.grid()
-plt.legend()
-plt.show()
-
-
-
-
-
[0.47179152 5.01549939]
-[0.47176783 5.01542292]
-1.7993408651198877
-1.7995707762668065
-
-test MSE of OLS:
-1.1394311129039245
-
-test MSE of Ridge
-1.1395084586525954
-
-
-
-
-
-
Finally, instead of using our own function we repeat the same example
-using the standardscaler functionality of the library
-Scikit-Learn. Here we limit ourselves to Ridge regression only.
-
-
-
fromsklearnimportlinear_model
-np.random.seed(2018)
-n=10
-d=2
-Lambda=0.01
-
-# Make data set.
-x=np.linspace(-3,3,n)
-y=2.0+0.5*x+5.0*(x**2)+np.random.randn(n)
-
-# Design matrix X does not include the intercept.
-X=np.zeros((n,d))
-forpinrange(d):
- X[:,p]=x**(p+1)
-
-#Split data in train and test
-X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2)
-# Scale data by subtracting mean value of the input using scikit-learn
-scaler=StandardScaler(with_std=False)
-scaler.fit(X_train)
-X_train_mean=np.mean(X_train,axis=0)
-X_train_scaled=scaler.transform(X_train)
-X_test_scaled=scaler.transform(X_test)
-# We scale also the output, here by our own code
-y_scaler=np.mean(y_train)
-y_train_scaled=y_train-y_scaler
-y_test_scaled=y_test-y_scaler
-
-#Calculate beta
-OLS=LinearRegression()
-betaOLS=OLS.fit(X_train_scaled,y_train_scaled)
-ypredictOLS=OLS.predict(X_test_scaled)
-linear_model.Ridge(Lambda)
-RegRidge.fit(X_train_scaled,y_train_scaled)
-ypredictRidge=RegRidge.predict(X_test_scaled)
-betaOLS=OLS.coef_
-betaRidge=RegRidge.coef_
-print(betaOLS)
-print(betaRidge)
-interceptOLS=np.mean(y_train)-X_train_mean@betaOLS
-interceptRidge=y_scaler-X_train_mean@betaRidge
-print(interceptOLS)
-print(interceptRidge)
-#predict value
-ytilde_test_Ridge=X_test_scaled@betaRidge+y_scaler
-ytilde_test_OLS=X_test_scaled@betaOLS+y_scaler
-
-#Calculate MSE
-print(" ")
-print("test MSE of OLS")
-print(MSE(y_test,ytilde_test_OLS))
-print(" ")
-print("test MSE of Ridge")
-print(MSE(y_test,ytilde_test_Ridge))
-plt.scatter(x,y,label='Data')
-plt.plot(x,X@RegRidge.coef_+RegRidge.intercept_,label="Ridge_Fit")
-plt.grid()
-plt.legend()
-plt.show()
-
-
-
-
-
---------------------------------------------------------------------------
-NameErrorTraceback (most recent call last)
-CellIn[11],line34
-32ypredictOLS=OLS.predict(X_test_scaled)
-33linear_model.Ridge(Lambda)
----> 34RegRidge.fit(X_train_scaled,y_train_scaled)
-35ypredictRidge=RegRidge.predict(X_test_scaled)
-36betaOLS=OLS.coef_
-
-NameError: name 'RegRidge' is not defined
-
Hence, \(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2)\), that is \(\boldsymbol{y}\) follows a normal distribution with
+mean value \(\boldsymbol{X}\boldsymbol{\beta}\) and variance \(\sigma^2\) (not be confused with the singular values of the SVD).
+
+
Expectation value and variance for \(\boldsymbol{\beta}\)¶
+
With the OLS expressions for the optimal parameters \(\boldsymbol{\hat{\beta}}\) we can evaluate the expectation value
where we have used that \(\mathbb{E} (\mathbf{y} \mathbf{y}^{T}) =
+\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
+\sigma^2 \, \mathbf{I}_{nn}\). From \(\mbox{Var}(\boldsymbol{\beta}) = \sigma^2
+\, (\mathbf{X}^{T} \mathbf{X})^{-1}\), one obtains an estimate of the
+variance of the estimate of the \(j\)-th regression coefficient:
+\(\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to
+construct a confidence interval for the estimates.
+
In a similar way, we can obtain analytical expressions for say the
+expectation values of the parameters \(\boldsymbol{\beta}\) and their variance
+when we employ Ridge regression, allowing us again to define a confidence interval.
We see clearly that
+\(\mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big] \not= \hat{\boldsymbol{\beta}}^{\mathrm{OLS}}\) for any \(\lambda > 0\).
The difference is non-negative definite since each component of the
+matrix product is non-negative definite.
+This means the variance we obtain with the standard OLS will always for \(\lambda > 0\) be larger than the variance of \(\boldsymbol{\beta}\) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
+
For more discussions of Ridge regression and calculation of averages, Wessel van Wieringen’s article is highly recommended.
diff --git a/doc/LectureNotes/_build/jupyter_execute/week37.ipynb b/doc/LectureNotes/_build/jupyter_execute/week37.ipynb
index 02de1a2cf..e421a77b9 100644
--- a/doc/LectureNotes/_build/jupyter_execute/week37.ipynb
+++ b/doc/LectureNotes/_build/jupyter_execute/week37.ipynb
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
- "id": "672fea2a",
+ "id": "7c0ddab1",
"metadata": {
"editable": true
},
@@ -14,13 +14,13 @@
},
{
"cell_type": "markdown",
- "id": "479a6913",
+ "id": "6c033bf4",
"metadata": {
"editable": true
},
"source": [
"# Week 37: Statistical interpretations and Resampling Methods\n",
- "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n",
+ "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo, Norway\n",
"\n",
"Date: **September 9, 2024**\n",
"\n",
@@ -29,419 +29,69 @@
},
{
"cell_type": "markdown",
- "id": "10b8f456",
+ "id": "e4cd5fa1",
"metadata": {
"editable": true
},
"source": [
- "## Plans for week 37\n",
+ "## Plans for week 37, lecture Monday\n",
"\n",
"**Material for the lecture on Monday September 9.**\n",
"\n",
"\n",
"\n",
- " * Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff\n",
+ " * Statistical interpretation of Ridge and Lasso regression, see also slides from last week\n",
"\n",
- " * Statistical interpretation of Ridge and Lasso regression\n",
+ " * Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff (this may partly be discussed during the exercise sessions as well.\n",
"\n",
" * Readings and Videos:\n",
"\n",
- " * Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap). \n",
+ " * Raschka et al, pages 175-192\n",
+ "\n",
+ " * Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap). See .\n",
"\n",
" * [Video on cross validation](https://www.youtube.com/watch?v=fSytzGwwBVw)\n",
"\n",
" * [Video on Bootstrapping](https://www.youtube.com/watch?v=Xz0x-8-cgaQ)\n",
"\n",
- " * [Video on bias-variance tradeoff](https://www.youtube.com/watch?v=EuBBz3bI-aA)\n",
+ " * [Video on bias-variance tradeoff](https://www.youtube.com/watch?v=EuBBz3bI-aA)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bf80a276",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Plans for week 37, lab sessions\n",
"\n",
- "**Material for the active learning sessions on Tuesday and Wednesday.**\n",
+ "**Material for the lab sessions on Tuesday and Wednesday.**\n",
"\n",
" * Calculations of expectation values\n",
"\n",
+ " * Discussion of resampling techniques\n",
+ "\n",
" * Exercise set for week 37\n",
"\n",
" * Work on project 1\n",
"\n",
- " * See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session. This note is added at the end of these slides.\n",
- "\n",
" * For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended."
]
},
{
"cell_type": "markdown",
- "id": "f7e43b1b",
+ "id": "0e1f0a90",
"metadata": {
"editable": true
},
"source": [
- "## Material from last week and relevant for the weekly exercises"
+ "## Material for lecture Monday September 9"
]
},
{
"cell_type": "markdown",
- "id": "af6a447a",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Linking the regression analysis with a statistical interpretation\n",
- "\n",
- "We will now couple the discussions of ordinary least squares, Ridge\n",
- "and Lasso regression with a statistical interpretation, that is we\n",
- "move from a linear algebra analysis to a statistical analysis. In\n",
- "particular, we will focus on what the regularization terms can result\n",
- "in. We will amongst other things show that the regularization\n",
- "parameter can reduce considerably the variance of the parameters\n",
- "$\\beta$.\n",
- "\n",
- "The\n",
- "advantage of doing linear regression is that we actually end up with\n",
- "analytical expressions for several statistical quantities. \n",
- "Standard least squares and Ridge regression allow us to\n",
- "derive quantities like the variance and other expectation values in a\n",
- "rather straightforward way.\n",
- "\n",
- "It is assumed that $\\varepsilon_i\n",
- "\\sim \\mathcal{N}(0, \\sigma^2)$ and the $\\varepsilon_{i}$ are\n",
- "independent, i.e.:"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b6b7ea38",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\begin{align*} \n",
- "\\mbox{Cov}(\\varepsilon_{i_1},\n",
- "\\varepsilon_{i_2}) & = \\left\\{ \\begin{array}{lcc} \\sigma^2 & \\mbox{if}\n",
- "& i_1 = i_2, \\\\ 0 & \\mbox{if} & i_1 \\not= i_2. \\end{array} \\right.\n",
- "\\end{align*}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e53d8d41",
- "metadata": {
- "editable": true
- },
- "source": [
- "The randomness of $\\varepsilon_i$ implies that\n",
- "$\\mathbf{y}_i$ is also a random variable. In particular,\n",
- "$\\mathbf{y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n",
- "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\beta}$ is a\n",
- "non-random scalar. To specify the parameters of the distribution of\n",
- "$\\mathbf{y}_i$ we need to calculate its first two moments. \n",
- "\n",
- "Recall that $\\boldsymbol{X}$ is a matrix of dimensionality $n\\times p$. The\n",
- "notation above $\\mathbf{X}_{i,\\ast}$ means that we are looking at the\n",
- "row number $i$ and perform a sum over all values $p$."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "401644e7",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Assumptions made\n",
- "\n",
- "The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)\n",
- "that there exists a function $f(\\boldsymbol{x})$ and a normal distributed error $\\boldsymbol{\\varepsilon}\\sim \\mathcal{N}(0, \\sigma^2)$\n",
- "which describe our data"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0603387c",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "eb2b6edf",
- "metadata": {
- "editable": true
- },
- "source": [
- "We approximate this function with our model from the solution of the linear regression equations, that is our\n",
- "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we want to minimize $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, our MSE, with"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f1a08194",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ad40fa0b",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Expectation value and variance\n",
- "\n",
- "We can calculate the expectation value of $\\boldsymbol{y}$ for a given element $i$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d3a4907d",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\begin{align*} \n",
- "\\mathbb{E}(y_i) & =\n",
- "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}) + \\mathbb{E}(\\varepsilon_i)\n",
- "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n",
- "\\end{align*}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d7908c9c",
- "metadata": {
- "editable": true
- },
- "source": [
- "while\n",
- "its variance is"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2528f74e",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n",
- "- \\mathbb{E}(y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( y_i^2 ) -\n",
- "[\\mathbb{E}(y_i)]^2 \\\\ & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n",
- "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \\\\ &\n",
- "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2 \\varepsilon_i\n",
- "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
- "\\ast} \\, \\beta)^2 \\\\ & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2\n",
- "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} +\n",
- "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \n",
- "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n",
- "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2. \n",
- "\\end{align*}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "525f3a59",
- "metadata": {
- "editable": true
- },
- "source": [
- "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
- "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD)."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ead1d419",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Expectation value and variance for $\\boldsymbol{\\beta}$\n",
- "\n",
- "With the OLS expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}}$ we can evaluate the expectation value"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e471f7ab",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mathbb{E}(\\boldsymbol{\\hat{\\beta}}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7886e97f",
- "metadata": {
- "editable": true
- },
- "source": [
- "This means that the estimator of the regression parameters is unbiased.\n",
- "\n",
- "We can also calculate the variance\n",
- "\n",
- "The variance of the optimal value $\\boldsymbol{\\hat{\\beta}}$ is"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "49c32ca5",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\begin{eqnarray*}\n",
- "\\mbox{Var}(\\boldsymbol{\\hat{\\beta}}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n",
- "\\\\\n",
- "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}]^{T} \\}\n",
- "\\\\\n",
- "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "% \\\\\n",
- "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} \\, \\mathbf{y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "% \\\\\n",
- "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{y} \\, \\mathbf{y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "\\\\\n",
- "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "% \\\\\n",
- "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^T \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
- "% \\\\\n",
- "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\boldsymbol{\\beta}^T\n",
- "\\\\\n",
- "& = & \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "\\, \\, \\, = \\, \\, \\, \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1},\n",
- "\\end{eqnarray*}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9bd27980",
- "metadata": {
- "editable": true
- },
- "source": [
- "where we have used that $\\mathbb{E} (\\mathbf{y} \\mathbf{y}^{T}) =\n",
- "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n",
- "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n",
- "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n",
- "variance of the estimate of the $j$-th regression coefficient:\n",
- "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $. This may be used to\n",
- "construct a confidence interval for the estimates.\n",
- "\n",
- "In a similar way, we can obtain analytical expressions for say the\n",
- "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n",
- "when we employ Ridge regression, allowing us again to define a confidence interval. \n",
- "\n",
- "It is rather straightforward to show that"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "184f9c3b",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d0f46be1",
- "metadata": {
- "editable": true
- },
- "source": [
- "We see clearly that \n",
- "$\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}$ for any $\\lambda > 0$.\n",
- "\n",
- "We can also compute the variance as"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "03e3ac5f",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "13ec2524",
- "metadata": {
- "editable": true
- },
- "source": [
- "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n",
- "\n",
- "With this, we can compute the difference"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "58c3f236",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}]-\\mbox{Var}(\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "bfcfc2a0",
- "metadata": {
- "editable": true
- },
- "source": [
- "The difference is non-negative definite since each component of the\n",
- "matrix product is non-negative definite. \n",
- "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. \n",
- "\n",
- "For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f3a9f084",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Material for lecture Thursday September 14"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a4338936",
+ "id": "1b290d64",
"metadata": {
"editable": true
},
@@ -464,7 +114,7 @@
},
{
"cell_type": "markdown",
- "id": "d7566301",
+ "id": "ceed3a82",
"metadata": {
"editable": true
},
@@ -476,7 +126,7 @@
},
{
"cell_type": "markdown",
- "id": "c8af7fae",
+ "id": "1d7e8f98",
"metadata": {
"editable": true
},
@@ -489,7 +139,7 @@
},
{
"cell_type": "markdown",
- "id": "959e33e8",
+ "id": "58722d7d",
"metadata": {
"editable": true
},
@@ -501,7 +151,7 @@
},
{
"cell_type": "markdown",
- "id": "c847b78b",
+ "id": "45120edc",
"metadata": {
"editable": true
},
@@ -513,7 +163,7 @@
},
{
"cell_type": "markdown",
- "id": "75c188ba",
+ "id": "4f3b680a",
"metadata": {
"editable": true
},
@@ -525,7 +175,7 @@
},
{
"cell_type": "markdown",
- "id": "60b89ba3",
+ "id": "08489797",
"metadata": {
"editable": true
},
@@ -536,7 +186,7 @@
},
{
"cell_type": "markdown",
- "id": "fd5d5a54",
+ "id": "a1c81f06",
"metadata": {
"editable": true
},
@@ -548,7 +198,7 @@
},
{
"cell_type": "markdown",
- "id": "56f25582",
+ "id": "4e092776",
"metadata": {
"editable": true
},
@@ -559,7 +209,7 @@
},
{
"cell_type": "markdown",
- "id": "9974b303",
+ "id": "79e5358a",
"metadata": {
"editable": true
},
@@ -571,7 +221,7 @@
},
{
"cell_type": "markdown",
- "id": "4112e053",
+ "id": "e5a4f865",
"metadata": {
"editable": true
},
@@ -581,7 +231,7 @@
},
{
"cell_type": "markdown",
- "id": "7be4a14c",
+ "id": "10b81ba7",
"metadata": {
"editable": true
},
@@ -612,7 +262,7 @@
},
{
"cell_type": "markdown",
- "id": "35078f48",
+ "id": "3349cb27",
"metadata": {
"editable": true
},
@@ -624,7 +274,7 @@
},
{
"cell_type": "markdown",
- "id": "cb25ead1",
+ "id": "0fd1502e",
"metadata": {
"editable": true
},
@@ -636,7 +286,7 @@
},
{
"cell_type": "markdown",
- "id": "5cc80036",
+ "id": "43cba0e8",
"metadata": {
"editable": true
},
@@ -646,7 +296,7 @@
},
{
"cell_type": "markdown",
- "id": "8ebc6099",
+ "id": "d6f4e7dc",
"metadata": {
"editable": true
},
@@ -658,7 +308,7 @@
},
{
"cell_type": "markdown",
- "id": "3160dff4",
+ "id": "50c2d673",
"metadata": {
"editable": true
},
@@ -668,7 +318,7 @@
},
{
"cell_type": "markdown",
- "id": "8a3aaa69",
+ "id": "30315a10",
"metadata": {
"editable": true
},
@@ -680,7 +330,7 @@
},
{
"cell_type": "markdown",
- "id": "7af6708c",
+ "id": "fd8cdf51",
"metadata": {
"editable": true
},
@@ -690,7 +340,7 @@
},
{
"cell_type": "markdown",
- "id": "c82e095e",
+ "id": "7323296b",
"metadata": {
"editable": true
},
@@ -702,7 +352,7 @@
},
{
"cell_type": "markdown",
- "id": "9431eca4",
+ "id": "e21334ab",
"metadata": {
"editable": true
},
@@ -712,7 +362,7 @@
},
{
"cell_type": "markdown",
- "id": "1db6c222",
+ "id": "82974124",
"metadata": {
"editable": true
},
@@ -732,7 +382,7 @@
},
{
"cell_type": "markdown",
- "id": "98c457ce",
+ "id": "e053e43c",
"metadata": {
"editable": true
},
@@ -744,7 +394,7 @@
},
{
"cell_type": "markdown",
- "id": "b6a3893f",
+ "id": "94098e13",
"metadata": {
"editable": true
},
@@ -754,7 +404,7 @@
},
{
"cell_type": "markdown",
- "id": "17f0e52e",
+ "id": "27625b1f",
"metadata": {
"editable": true
},
@@ -766,7 +416,7 @@
},
{
"cell_type": "markdown",
- "id": "c2bfaea6",
+ "id": "8f3cc8b4",
"metadata": {
"editable": true
},
@@ -778,7 +428,7 @@
},
{
"cell_type": "markdown",
- "id": "377eefdf",
+ "id": "d7d5464d",
"metadata": {
"editable": true
},
@@ -790,7 +440,7 @@
},
{
"cell_type": "markdown",
- "id": "6da6253e",
+ "id": "c292afee",
"metadata": {
"editable": true
},
@@ -802,7 +452,7 @@
},
{
"cell_type": "markdown",
- "id": "e5906b3e",
+ "id": "cdbbd444",
"metadata": {
"editable": true
},
@@ -814,7 +464,7 @@
},
{
"cell_type": "markdown",
- "id": "792d292c",
+ "id": "100a8014",
"metadata": {
"editable": true
},
@@ -826,7 +476,7 @@
},
{
"cell_type": "markdown",
- "id": "c3bc0674",
+ "id": "d2e40c4d",
"metadata": {
"editable": true
},
@@ -838,7 +488,7 @@
},
{
"cell_type": "markdown",
- "id": "0faadf3f",
+ "id": "1e48456e",
"metadata": {
"editable": true
},
@@ -850,7 +500,7 @@
},
{
"cell_type": "markdown",
- "id": "a2d478a7",
+ "id": "38ced5b6",
"metadata": {
"editable": true
},
@@ -860,7 +510,7 @@
},
{
"cell_type": "markdown",
- "id": "d51ac6ce",
+ "id": "1163c07b",
"metadata": {
"editable": true
},
@@ -872,7 +522,7 @@
},
{
"cell_type": "markdown",
- "id": "91d489d9",
+ "id": "becc087f",
"metadata": {
"editable": true
},
@@ -882,7 +532,7 @@
},
{
"cell_type": "markdown",
- "id": "ebe26b44",
+ "id": "5ca2b09d",
"metadata": {
"editable": true
},
@@ -901,7 +551,7 @@
},
{
"cell_type": "markdown",
- "id": "a67660e2",
+ "id": "6572c77d",
"metadata": {
"editable": true
},
@@ -922,7 +572,7 @@
},
{
"cell_type": "markdown",
- "id": "40b7eac0",
+ "id": "2a3f65e5",
"metadata": {
"editable": true
},
@@ -934,7 +584,7 @@
},
{
"cell_type": "markdown",
- "id": "16a3b6ea",
+ "id": "2d264f77",
"metadata": {
"editable": true
},
@@ -945,7 +595,7 @@
},
{
"cell_type": "markdown",
- "id": "f41ccc6d",
+ "id": "06b8a13c",
"metadata": {
"editable": true
},
@@ -958,7 +608,7 @@
},
{
"cell_type": "markdown",
- "id": "ae81305b",
+ "id": "6d958ed3",
"metadata": {
"editable": true
},
@@ -970,7 +620,7 @@
},
{
"cell_type": "markdown",
- "id": "f4829b81",
+ "id": "6330f3f2",
"metadata": {
"editable": true
},
@@ -980,7 +630,7 @@
},
{
"cell_type": "markdown",
- "id": "812273af",
+ "id": "ed6a9f67",
"metadata": {
"editable": true
},
@@ -992,7 +642,7 @@
},
{
"cell_type": "markdown",
- "id": "31ecba0e",
+ "id": "de42ab8a",
"metadata": {
"editable": true
},
@@ -1002,7 +652,7 @@
},
{
"cell_type": "markdown",
- "id": "ca6682ec",
+ "id": "4f36b126",
"metadata": {
"editable": true
},
@@ -1014,7 +664,7 @@
},
{
"cell_type": "markdown",
- "id": "662eaffb",
+ "id": "0a7eef00",
"metadata": {
"editable": true
},
@@ -1024,7 +674,7 @@
},
{
"cell_type": "markdown",
- "id": "08d5551d",
+ "id": "29940f94",
"metadata": {
"editable": true
},
@@ -1038,7 +688,7 @@
},
{
"cell_type": "markdown",
- "id": "03f3de9c",
+ "id": "f2204679",
"metadata": {
"editable": true
},
@@ -1050,7 +700,7 @@
},
{
"cell_type": "markdown",
- "id": "c57ace49",
+ "id": "d3f4597f",
"metadata": {
"editable": true
},
@@ -1060,7 +710,7 @@
},
{
"cell_type": "markdown",
- "id": "cb014b2b",
+ "id": "c9e9ca01",
"metadata": {
"editable": true
},
@@ -1072,7 +722,7 @@
},
{
"cell_type": "markdown",
- "id": "1054d47e",
+ "id": "1c3062b6",
"metadata": {
"editable": true
},
@@ -1082,7 +732,7 @@
},
{
"cell_type": "markdown",
- "id": "1485d44c",
+ "id": "b455f707",
"metadata": {
"editable": true
},
@@ -1094,7 +744,7 @@
},
{
"cell_type": "markdown",
- "id": "f457f909",
+ "id": "5ac8fdd3",
"metadata": {
"editable": true
},
@@ -1104,7 +754,7 @@
},
{
"cell_type": "markdown",
- "id": "59447b2a",
+ "id": "827927db",
"metadata": {
"editable": true
},
@@ -1116,17 +766,17 @@
},
{
"cell_type": "markdown",
- "id": "4fe26100",
+ "id": "0c740a13",
"metadata": {
"editable": true
},
"source": [
- "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$!"
+ "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta})$!"
]
},
{
"cell_type": "markdown",
- "id": "d73c2d9c",
+ "id": "a855fc3c",
"metadata": {
"editable": true
},
@@ -1142,7 +792,7 @@
},
{
"cell_type": "markdown",
- "id": "8cdf2c97",
+ "id": "9dda8b9e",
"metadata": {
"editable": true
},
@@ -1154,7 +804,7 @@
},
{
"cell_type": "markdown",
- "id": "7272ef9f",
+ "id": "d160205c",
"metadata": {
"editable": true
},
@@ -1164,7 +814,7 @@
},
{
"cell_type": "markdown",
- "id": "0dc5b2b7",
+ "id": "1b7501a6",
"metadata": {
"editable": true
},
@@ -1176,7 +826,7 @@
},
{
"cell_type": "markdown",
- "id": "b6c8c684",
+ "id": "d75a8c65",
"metadata": {
"editable": true
},
@@ -1189,7 +839,7 @@
},
{
"cell_type": "markdown",
- "id": "f681ed50",
+ "id": "c9f718df",
"metadata": {
"editable": true
},
@@ -1201,7 +851,7 @@
},
{
"cell_type": "markdown",
- "id": "5c3bc898",
+ "id": "b07b6425",
"metadata": {
"editable": true
},
@@ -1211,7 +861,7 @@
},
{
"cell_type": "markdown",
- "id": "a3cea046",
+ "id": "215a8730",
"metadata": {
"editable": true
},
@@ -1223,7 +873,7 @@
},
{
"cell_type": "markdown",
- "id": "68276410",
+ "id": "7013d0f0",
"metadata": {
"editable": true
},
@@ -1233,7 +883,7 @@
},
{
"cell_type": "markdown",
- "id": "dfd55611",
+ "id": "d56b4ce0",
"metadata": {
"editable": true
},
@@ -1245,7 +895,7 @@
},
{
"cell_type": "markdown",
- "id": "83656dd8",
+ "id": "f9c6b14d",
"metadata": {
"editable": true
},
@@ -1257,7 +907,7 @@
},
{
"cell_type": "markdown",
- "id": "b62e29fd",
+ "id": "0bde16cb",
"metadata": {
"editable": true
},
@@ -1267,7 +917,7 @@
},
{
"cell_type": "markdown",
- "id": "1de8855c",
+ "id": "0a2869fc",
"metadata": {
"editable": true
},
@@ -1279,7 +929,7 @@
},
{
"cell_type": "markdown",
- "id": "212152f1",
+ "id": "75b83b9b",
"metadata": {
"editable": true
},
@@ -1291,7 +941,7 @@
},
{
"cell_type": "markdown",
- "id": "afb042c4",
+ "id": "1e76d916",
"metadata": {
"editable": true
},
@@ -1303,7 +953,7 @@
},
{
"cell_type": "markdown",
- "id": "ab1fd3c5",
+ "id": "09dfe26d",
"metadata": {
"editable": true
},
@@ -1313,7 +963,7 @@
},
{
"cell_type": "markdown",
- "id": "43979108",
+ "id": "4037962f",
"metadata": {
"editable": true
},
@@ -1325,7 +975,7 @@
},
{
"cell_type": "markdown",
- "id": "96622b66",
+ "id": "39a85ba0",
"metadata": {
"editable": true
},
@@ -1335,7 +985,7 @@
},
{
"cell_type": "markdown",
- "id": "e17a5b6d",
+ "id": "b263a1fe",
"metadata": {
"editable": true
},
@@ -1354,7 +1004,7 @@
},
{
"cell_type": "markdown",
- "id": "24e35401",
+ "id": "41730165",
"metadata": {
"editable": true
},
@@ -1382,7 +1032,7 @@
},
{
"cell_type": "markdown",
- "id": "a37f1d06",
+ "id": "4076cee3",
"metadata": {
"editable": true
},
@@ -1408,7 +1058,7 @@
},
{
"cell_type": "markdown",
- "id": "58f54a23",
+ "id": "11139cb3",
"metadata": {
"editable": true
},
@@ -1425,7 +1075,7 @@
},
{
"cell_type": "markdown",
- "id": "6ed7c809",
+ "id": "b2bead9b",
"metadata": {
"editable": true
},
@@ -1445,7 +1095,7 @@
},
{
"cell_type": "markdown",
- "id": "bed9bb2f",
+ "id": "8df8364e",
"metadata": {
"editable": true
},
@@ -1474,7 +1124,7 @@
},
{
"cell_type": "markdown",
- "id": "2cafa0a3",
+ "id": "6b5273f1",
"metadata": {
"editable": true
},
@@ -1499,7 +1149,7 @@
},
{
"cell_type": "markdown",
- "id": "fefbb980",
+ "id": "a260b371",
"metadata": {
"editable": true
},
@@ -1519,7 +1169,7 @@
},
{
"cell_type": "markdown",
- "id": "4560153d",
+ "id": "2ff1e63c",
"metadata": {
"editable": true
},
@@ -1531,7 +1181,7 @@
},
{
"cell_type": "markdown",
- "id": "ffcbdd5a",
+ "id": "ab4b2e8c",
"metadata": {
"editable": true
},
@@ -1541,7 +1191,7 @@
},
{
"cell_type": "markdown",
- "id": "d055ea55",
+ "id": "85b6fb62",
"metadata": {
"editable": true
},
@@ -1556,7 +1206,7 @@
},
{
"cell_type": "markdown",
- "id": "004f0161",
+ "id": "42449344",
"metadata": {
"editable": true
},
@@ -1569,7 +1219,7 @@
},
{
"cell_type": "markdown",
- "id": "039f4ad5",
+ "id": "35e4f511",
"metadata": {
"editable": true
},
@@ -1582,7 +1232,7 @@
},
{
"cell_type": "markdown",
- "id": "413459f1",
+ "id": "c787234f",
"metadata": {
"editable": true
},
@@ -1594,7 +1244,7 @@
},
{
"cell_type": "markdown",
- "id": "bef022c8",
+ "id": "183711dc",
"metadata": {
"editable": true
},
@@ -1607,7 +1257,7 @@
},
{
"cell_type": "markdown",
- "id": "3d53693d",
+ "id": "2351078d",
"metadata": {
"editable": true
},
@@ -1618,7 +1268,7 @@
},
{
"cell_type": "markdown",
- "id": "e682c32a",
+ "id": "408ba3b5",
"metadata": {
"editable": true
},
@@ -1632,7 +1282,7 @@
},
{
"cell_type": "markdown",
- "id": "f44053ea",
+ "id": "3ea79849",
"metadata": {
"editable": true
},
@@ -1642,7 +1292,7 @@
},
{
"cell_type": "markdown",
- "id": "f10de930",
+ "id": "69439aad",
"metadata": {
"editable": true
},
@@ -1656,7 +1306,7 @@
},
{
"cell_type": "markdown",
- "id": "1f91402a",
+ "id": "e6c29fd8",
"metadata": {
"editable": true
},
@@ -1669,7 +1319,7 @@
},
{
"cell_type": "markdown",
- "id": "2d7a8e36",
+ "id": "4bb52549",
"metadata": {
"editable": true
},
@@ -1682,7 +1332,7 @@
},
{
"cell_type": "markdown",
- "id": "4c756ca2",
+ "id": "3430d548",
"metadata": {
"editable": true
},
@@ -1692,7 +1342,7 @@
},
{
"cell_type": "markdown",
- "id": "908f782e",
+ "id": "9ba636b3",
"metadata": {
"editable": true
},
@@ -1705,7 +1355,7 @@
},
{
"cell_type": "markdown",
- "id": "f12152da",
+ "id": "cb19e98f",
"metadata": {
"editable": true
},
@@ -1715,7 +1365,7 @@
},
{
"cell_type": "markdown",
- "id": "85dd290d",
+ "id": "3cc840d7",
"metadata": {
"editable": true
},
@@ -1728,7 +1378,7 @@
},
{
"cell_type": "markdown",
- "id": "57631d10",
+ "id": "a5062213",
"metadata": {
"editable": true
},
@@ -1740,7 +1390,7 @@
},
{
"cell_type": "markdown",
- "id": "2d4d5ccb",
+ "id": "8d555244",
"metadata": {
"editable": true
},
@@ -1759,7 +1409,7 @@
},
{
"cell_type": "markdown",
- "id": "a29c8551",
+ "id": "c4356604",
"metadata": {
"editable": true
},
@@ -1772,7 +1422,7 @@
},
{
"cell_type": "markdown",
- "id": "6509ba36",
+ "id": "09dc08a1",
"metadata": {
"editable": true
},
@@ -1784,7 +1434,7 @@
},
{
"cell_type": "markdown",
- "id": "733307b1",
+ "id": "dab5056a",
"metadata": {
"editable": true
},
@@ -1797,7 +1447,7 @@
},
{
"cell_type": "markdown",
- "id": "a69b7331",
+ "id": "8a1f8583",
"metadata": {
"editable": true
},
@@ -1817,7 +1467,7 @@
},
{
"cell_type": "markdown",
- "id": "ced659da",
+ "id": "1be08ebb",
"metadata": {
"editable": true
},
@@ -1831,16 +1481,16 @@
"With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ we found \n",
"$\\mathbb{E}(\\boldsymbol{\\beta}) = \\boldsymbol{\\beta}$, which means that the estimator of the regression parameters is unbiased.\n",
"\n",
- "We found also that the variance of the estimate of the $j$-th regression coefficient is\n",
+ "In the exercises this week we show that the variance of the estimate of the $j$-th regression coefficient is\n",
"$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $.\n",
"\n",
- "This quantity will be used to\n",
+ "This quantity can be used to\n",
"construct a confidence interval for the estimates."
]
},
{
"cell_type": "markdown",
- "id": "2c0d0a71",
+ "id": "2a368353",
"metadata": {
"editable": true
},
@@ -1855,7 +1505,7 @@
},
{
"cell_type": "markdown",
- "id": "d3a4d693",
+ "id": "6fee4a1a",
"metadata": {
"editable": true
},
@@ -1867,7 +1517,7 @@
},
{
"cell_type": "markdown",
- "id": "3ed3d852",
+ "id": "1a97c9ef",
"metadata": {
"editable": true
},
@@ -1887,7 +1537,7 @@
},
{
"cell_type": "markdown",
- "id": "e826e70b",
+ "id": "0a70afb2",
"metadata": {
"editable": true
},
@@ -1907,7 +1557,7 @@
},
{
"cell_type": "markdown",
- "id": "10b5d4d5",
+ "id": "dc32eaea",
"metadata": {
"editable": true
},
@@ -1931,7 +1581,7 @@
},
{
"cell_type": "markdown",
- "id": "1645ce3c",
+ "id": "016d0ab3",
"metadata": {
"editable": true
},
@@ -1952,7 +1602,7 @@
},
{
"cell_type": "markdown",
- "id": "4986483a",
+ "id": "3f371f0d",
"metadata": {
"editable": true
},
@@ -1982,7 +1632,7 @@
},
{
"cell_type": "markdown",
- "id": "fe853f3b",
+ "id": "abc56fe7",
"metadata": {
"editable": true
},
@@ -2006,7 +1656,7 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "81ec18f8",
+ "id": "a3d2885b",
"metadata": {
"collapsed": false,
"editable": true
@@ -2018,7 +1668,7 @@
"text": [
"Bootstrap Statistics :\n",
"original bias std. error\n",
- " 99.9841 15.0402 99.9828 0.1512\n"
+ " 99.7503 14.9999 99.7496 0.151068\n"
]
}
],
@@ -2055,7 +1705,7 @@
},
{
"cell_type": "markdown",
- "id": "f0b5b588",
+ "id": "5db67125",
"metadata": {
"editable": true
},
@@ -2065,7 +1715,7 @@
},
{
"cell_type": "markdown",
- "id": "262a95e8",
+ "id": "6f1b5417",
"metadata": {
"editable": true
},
@@ -2076,7 +1726,7 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "d235b516",
+ "id": "9d98136a",
"metadata": {
"collapsed": false,
"editable": true
@@ -2084,14 +1734,14 @@
"outputs": [
{
"data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkkAAAGwCAYAAAC99fF4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABXIElEQVR4nO3deVhUZf8G8HuAYQBl31HEfV9C3FBRccFUUNPXMrOsX/pmYqVkJm1qpfb2+pqpqWmWqblU7ogL7vuGmLhh5hqLyCIoCDMM5/fHJIWMss3MM8v9ua65YM6cmXOf55qB7zznOc+RSZIkgYiIiIhKsRIdgIiIiMgYsUgiIiIi0oJFEhEREZEWLJKIiIiItGCRRERERKQFiyQiIiIiLVgkEREREWlhIzqAoRUXFyMlJQWOjo6QyWSi4xAREVEFSJKE+/fvw8/PD1ZWhunjsbgiKSUlBf7+/qJjEBERURXcvn0btWvXNsi2LK5IcnR0BKBpZCcnp6euq1KpsGvXLoSFhUEulxsinllh+1Uf27D62IbVw/arPrZh9alUKmzatAmjR48u+T9uCBZXJD06xObk5FShIsnBwQFOTk58Y1cB26/62IbVxzasHrZf9bENq+9RGwIw6FAZDtwmIiIi0oJFEhEREZEWLJKIiIiItGCRRERERKQFiyQiIiIiLVgkEREREWnBIomIiIhICxZJRERERFqwSCIiIiLSgkUSERERkRYskoiIiIi0YJFEREREpAWLJCIiIiItWCQRERERacEiiYiIiEgLG9EBiIgAAKGh5a+zb5/+cxAR/YU9SURERERasEgiIiIi0oJFEhEREZEWHJNERKaD45aIyIDYk0RERESkBYskIiIiIi1YJBERERFpwSKJiIiISAsO3CYik/VQbYukfH9czK+Li3kBuJQfgJzewNSpQEiI6HREZOpYJBGRSSgsluPXu92R+KAeLuYH4GJeXVwr8IX0V4e4n+1dNK9xE/fuAd27A+PHA7NmAba2YnMTkelikURERi9frcDg858hLrs96ijS0LzGTQzyOILmNW6iucMNNHO4CRd5HgBAvXsf5s8HPvgAiIkBFi+WCU5PRKaKRRIRGbU8tR0iEmfgRG4z7GkThZ6uCU9d39oamDABCA8HRo8G+va1Qd++rdG1K+DubpjMRGQeOHCbiIzW/SJ7PHvuPzh1vyl2tH6/3ALpnxo2BPbuBebNU+PAAX8EBtpg5049hiUis8MiiYiMUk5RDYSd+y/OPaiPXa3fQ4hLYqVfw8oKGDu2GF9/vReNG0t49lng//4PuHdP93mJyPywSCIio5Otqonev83G5fw62N1mEoKdL1br9by9HyI2Vo2lS4H164GWLYE//9RRWCIyWyySiMioZCid0PO3Obj+0Bd720ShvVOSTl5XJtOMUTp/HpAk4NVXgeJinbw0EZkpFklEZDTSlS7o+dscJBd6YN8zExHoeFXn2/D3B5YvB/bsARYs0PnLE5EZYZFEREYhtdANPc5+hbsqF+x/ZiJa1byut2316QO89Rbw/vvApUt62wwRmTgWSUQk3J07QI+zXyG3qAYOPDMBzWvc1Ps2v/gCqFsXGDkSUCr1vjkiMkEskohIuEmTgKwiJxwInIDGDoYZUe3gAKxcCZw7B3z2mUE2SUQmhkUSEQl17BiwahUwq95SNLBPMei227UDPvkEmDlTk4OI6J9YJBGRMMXFwDvvAIGBwGu+O4RkiI4G2rcHXnkFePBASAQiMlIskohImBUrgFOngK+/BqxlYs7Ht7HRHHZLSdEc9iMiekRokTRr1iy0b98ejo6O8PLywuDBg5GU9PQ5Ufbv3w+ZTFbmdvnyZQOlJiJdyM0FpkwBhg8HQkLEZmnUCJgzB/j2W2DbNrFZiMh4CC2SDhw4gMjISBw/fhxxcXEoKipCWFgY8vLyyn1uUlISUlNTS26NGjUyQGIi0pWZMzWF0n/+IzqJxr//DfTvD7z+OnD3rug0RGQMbERufMeO0mMQfvjhB3h5eSE+Ph7dunV76nO9vLzg4uKix3REpC9XrwJffQV88AFQp47oNBoyGbBsmeaSJW+8obl8iUwmOhURiSS0SHpcTk4OAMDNza3cdQMDA1FQUIDmzZvjo48+QmhoqNb1CgsLUVhYWHI/NzcXAKBSqaBSqZ66jUePl7ceacf2qz5zbcOJE63h7S3DhAlFKNk1W1vdvPhjbaW1DcPDtT7VHcDC2l3wwsZP8P33RXjlFUk3mUyYub4HDYltWH2i2k4mSZJR/BWQJAmDBg1CdnY2Dh069MT1kpKScPDgQQQFBaGwsBArV67E4sWLsX//fq29T9OmTcP06dPLLF+9ejUcHBx0ug9EVL6EBE9Mn94ZkyadQteuhj3lv6K+/joQx4/7YuHCPXB1LSz/CUSkd/n5+RgxYgRycnLg5ORkkG0aTZEUGRmJbdu24fDhw6hdu3alnhsREQGZTIYtW7aUeUxbT5K/vz8yMjLKbWSVSoW4uDj06dMHcrm8UpmI7acL5taGKhUQFGQDDw8Je/aoSx/OekLvTqXFxDy2TS1tWM62slU10fD0rxg7thgzZlj2VXDN7T0oAtuw+lQqFTZv3mzwIskoDre99dZb2LJlCw4ePFjpAgkAOnXqhFWrVml9TKFQQKFQlFkul8sr/GatzLpUFtuv+sylDRctApKSgNWrZbC1fey8EV1dG+QJ7VSqDcvZlheyMHasDN9+a40PP7SGs7Nuopkyc3kPisQ2ND1Cz26TJAnjx4/Hhg0bsHfvXtSrV69Kr5OQkABfX18dpyMiXcrIAKZOBUaP1kweaewmTgQKCoDFi0UnISJRhPYkRUZGYvXq1di8eTMcHR2RlpYGAHB2doa9vT0AIDo6GsnJyVixYgUAYO7cuahbty5atGgBpVKJVatWYf369Vi/fr2w/SCi8n38MSBJwIwZopNUjK8v8OqrmrPw3n4b+OtPEhFZEKE9SYsWLUJOTg569OgBX1/fktu6detK1klNTcWtW7dK7iuVSkyaNAmtW7dGSEgIDh8+jG3btmHIkCEidoGIKuC334AlSzQ9SZ6eotNU3HvvaeZM+vFH0UmISAShPUkVGTO+fPnyUvcnT56MyZMn6ykREVXaE6bfeESSgAmyfWjUCIiMNFAmHWnYEPjXv4D//ldzmNDGKEZxEpGh8NptRKRXMZnB2L9fc9hKV1MhGdKUKcC1a8Cvv4pOQkSGxiKJiPTq6z+HomNHoF8/0UmqJjAQ6NsX+OILTa8YEVkOFklEpDeX8upgz70gvPWW6CTVM2WKZlzVY1dSIiIzxyKJiPTmm+TB8JZnYdgw0Umqp3t3oGNHTW8SEVkODkMkIr3ILXLAj3f6YmLtX2Hb9wfRcapFJgOio4HBg4GjR4HOnUUnIiJDYJFERHrxY1pfPFQr8IbfVsNu+PGz7WxtgXHjNJciqcas3hERQLNmmt4kLVdAIiIzxMNtRKRzxZIMC5IHY4jnIdRSZIiOoxNWVsD77wNbtwLnz4tOQ0SGwCKJiHRuT3ZbXHlYB+NrbRQdRadefBHw9we+/FJ0EiIyBB5uIyKdW5D8HFrX+AMhzudER6k6LZNk2gKYpBiCqJXj8OmVl1D3+FrD5yIig2FPEhHp1I2H3tiaGYzxtTZCJhOdRvde942Fi80D/O/P50VHISI9Y5FERDq1KGUQnG3yMMJ7j+goelHDugBv196A71IHID1ddBoi0icWSUSkMw/VtvgutT/+z2c7algXiI6jN5G1NsEaxZg/X3QSItInFklEpDNr03siu8gRb/ptFh1Fr9zlufi3XwwWLADu3xedhoj0hUUSEemEJAHzk59DP7eTaOiQIjqO3k2o/StycnjhWyJzxiKJiHTieG5zJDxojLdqbRAdxSDq2KWjZ0/gxx9FJyEifWGRREQ6sSD5OTS0/xNhbqdFRzGYUaOAAweA69dFJyEifWCRRETVllboil/udkek3yZYySTRcQxmyBCgZk1gxQrRSYhIH1gkEVG1LU0Nh1xWhFd9doiOYlA1agDDhmmKJMlyakMii8EiiYiqRVVsjcUpEXjZOw4u8jzRcQxu1Cjg2jXg8GHRSYhI11gkEVG1bMwIQYrSE5G1NomOIkRICFC3LrB8uegkRKRrLJKIqFoWJA9GD5cEtKppmaOXraw0vUm//ALk54tOQ0S6xCKJiKrs3DngUE4bjLfQXqRHXnlFM6nkxo2ikxCRLrFIIqIqW74c8JZnYZC7ZQ/IqV9fc9iNcyYRmRcWSURUJWo1sG4d8ILXPthYFYuOI9yoUcDu3cDt26KTEJGu2IgOQESm6dAhICUFeDFwj+go4oSGlvw6rMgBb8nWY1X3lYgOWP33Ovv2CQhGRLrAniQiqpI1a4B69YCOTpdERzEKTjb5GOJxCD+m9eWcSURmgkUSEVWaUqk5m2v4cEAmE53GeIzy2Ymkh3Vw8n4z0VGISAdYJBFRpe3aBWRnAyNGiE5iXHq6JqCW7V0sT+srOgoR6QCLJCKqtDVrgJYtNTf6m7WsGK/47MLa9J4oUMtFxyGiamKRRESVkpcHbNoEvPii6CTGaZTPTtwrcsTWzM6ioxBRNbFIIqJK2bpVM7M0iyTtmjjcRkfHi/iRh9yITB6LJCKqlDVrgE6dNGe2kXajfHZiR1YHpBW6io5CRNXAIomIKiwrC9i+nb1I5RnutRfWMjV+Su8tOgoRVQOLJCKqsA0bNDNtP/+86CTGzVX+AIM8jnLOJCITxyKJiCpszRqgZ0/Ax0d0EuM3ymcnEvMa4OxZ0UmIqKpYJBFRhaSkaK6wwUNtFdPX9SS85VlYvlx0EiKqKhZJRFQhP/8MyOXAkCGik5gGG6tijPSOw+rVmhnKicj0sEgiogpZswbo1w9wcRGdxHS84rMLGRnA7t2ikxBRVbBIIqJyXb0KnDzJy5BUVqsa19CokWbAOxGZHhZJRFSutWuBmjWB8HDRSUyLTKY5PLlpE1BUJDoNEVUWiyQieipJAlavBgYNAhwcRKcxPUOHApmZwMGDopMQUWWxSCKipzp3Drh0iWe1VVW7dkCdOsD69aKTEFFlsUgioqdaswZwdwfCwkQnMU2PDrlt3AgUF4tOQ0SVYSM6ABEZr+IePbH2+Gr8y+0k5GFfiY5jmkJDMfReS8xNnY/j7cajs/OFsuvs22f4XERULvYkEdETHcttgZuFPnjRe6/oKCats/MF+NhmYv3dbqKjEFElsEgioidac6cnatneRYjzOdFRTJqVTMJgj8NYf7cbr+VGZEJYJBGRVkVFwM93e2C4115YyfifvbqGehzEzUIfnHnQWHQUIqogFklEpNXevcBdlSsPtelId5ff4GaTw0NuRCaERRIRabVhA1DfLhlta14RHcUsyK3UGORxFOvvhvCQG5GJYJFERGUUFwObNwODPY5AJhOdxnwM9TyIKw/r4GJ+XdFRiKgCWCQRURknTwJpacBgj8Oio5iV3q7xcLJ+wENuRCaCRRIRlbFpE+DhAe1z+lCVKaxUCHc/ziKJyESwSCKiMjZtAgYOBKxlnCJa14Z4HsK5vAa4mu8nOgoRlYNFEhGVcvkykJQEDB4sOol5etbtJOytCrA+g71JRMZOaJE0a9YstG/fHo6OjvDy8sLgwYORlJRU7vMOHDiAoKAg2NnZoX79+li8eLEB0hJZhk2bAAcHoHdv0UnMUw3rAvRzO4kNPORGZPSEFkkHDhxAZGQkjh8/jri4OBQVFSEsLAx5eXlPfM7169fRv39/hISEICEhAR988AHefvttrOcltol0YtMm4NlnAXt70UnM11DPgzh5vxluF3iKjkJETyH0Arc7duwodf+HH36Al5cX4uPj0a2b9m9ZixcvRp06dTB37lwAQLNmzXD69GnMnj0bQ4cOLbN+YWEhCgsLS+7n5uYCAFQqFVQq1VPzPXq8vPVIO7Zf9Rm6DVNSgBMn5Pj++yKoVBJga2uQ7eqTSi4v9dMY9PWNh+1lJX7JDsVbAZsAI/6M8HNcfWzD6hPVdjJJMp5pza5evYpGjRohMTERLVu21LpOt27dEBgYiK+//rpk2caNG/H8888jPz8f8sf+EE6bNg3Tp08v8zqrV6+Gg4ODbneAyMTt3BmAb79tjR9/3AFHR/5B16fPP++I/HwbzJx5RHQUIpOQn5+PESNGICcnB05OTgbZptCepH+SJAlRUVHo2rXrEwskAEhLS4O3t3epZd7e3igqKkJGRgZ8fX1LPRYdHY2oqKiS+7m5ufD390dYWFi5jaxSqRAXF4c+ffqUKb6ofGy/6jN0Gy5aZI3u3YEXXuijWRAervdt6ptKLkfcmDHos3Qp5Eb0TT794VW8cXEi2n61Bj5xK0XHeSJ+jquPbVh9KpUKmzdvNvh2jaZIGj9+PM6dO4fDh8ufvE722BTAjzrDHl8OAAqFAgqFosxyuVxe4TdrZdalsth+1WeINszN1VyvbfZsQC7/a7iiUqnXbRqSXKWC3Ij2Z4jLQbyJCdiW0gFjTeDzwc9x9bENTY9RTAHw1ltvYcuWLdi3bx9q16791HV9fHyQlpZWall6ejpsbGzg7u6uz5hEZm37ds3QmEGDRCexDO7yXPRwOYsNGSGioxDREwgtkiRJwvjx47Fhwwbs3bsX9erVK/c5wcHBiIuLK7Vs165daNeuHSt0omrYtAkIDAQCAkQnsRxDPQ9i371AZGWJTkJE2ggtkiIjI7Fq1SqsXr0ajo6OSEtLQ1paGh4+fFiyTnR0NF555ZWS+2PHjsXNmzcRFRWFS5cu4fvvv8eyZcswadIkEbtAZBYKC4Ft2ziBpKE953EIaskKW7aITkJE2ggdk7Ro0SIAQI8ePUot/+GHH/Dqq68CAFJTU3Hr1q2Sx+rVq4fY2FhMnDgR33zzDfz8/DBv3jytp/8T0VOEhpb8uj+rPe7f/xKDt74O7LsmMJRl8VFko4vzeaxf3xp//ckjIiMitEiqyOwDy5cvL7Ose/fuOHPmjB4SEVmmTRldUM8uBa1qsEAytCEehzBlV2vk5gIGOquZiCrIKAZuE5E4xZIMmzO6YLDHEWg5QZT0bIjnISiVwM6dopMQ0eNYJBFZuFP3myJV6YHBHuVPv0G6F2B3B61aAVu3ik5CRI9jkURk4TZndIGH/B46O50XHcViRUQAsbGAWi06CRH9E4skIgu3KaMLItyPwcaqWHQUixURAWRmAseOiU5CRP9kNDNuE5HhJeX741J+XXxRf6noKBatw5Se8JL/iq0jdqJrgyVPXnHfPsOFIiL2JBFZss0ZXeBg9RB9XE+LjmLRrGQSBrifwNbMYNFRiOgfWCQRWbBNGV3Q1+0U7K2N55pmlirC/Sgu5dfFHw/9REchor+wSCKyUKmFbjie2xyDPY6IjkIA+riehq1Mia0Z7E0iMhYskogs1NbMzrCChAHux0VHIQA1bQrQ0zUBWzM7i45CRH9hkURkoTZldEGIyzm4y3NFR6G/RLgfw8Gc1sgpqiE6ChGBRRKRRcrNBfZkt+WhNiMT7n4MRZINdmR1EB2FiMAiicgi7dgBKCVbDHLnLNvGpI5dOtrUuMpxSURGgkUSkQWKiQFa1fgDde3viI5Cj4nwOIrYrI4oKuafZyLR+CkksjBqteYSGBHunN7ZGEW4H0N2kROO5rYUHYXI4rFIIrIwJ05oLoERziLJKLVzTIKPbSYnliQyAiySiCxMTAzg4QF0cLosOgppYSWTMMDtOLZmcCoAItFYJBFZmJgYoH9/wFrGC9oaqwiPY0h6WAe/59cSHYXIorFIIrIgt24BiYlAeLjoJPQ0vV3joZApObEkkWAskogsyLZtgI0NEBYmOgk9TQ3rAvRyPYOYzE6ioxBZNBZJRBYkJgbo1g1wdhadhMoT4X4Uh3Ja456Ks28TicIiichC5OUBe/YAAwaITkIVwdm3icRjkURkIfbuBQoLOR7JVNS2y0BgzSscl0QkEIskIgsREwM0agQ0biw6CVVUhPsxbM/qwNm3iQThJ4/IAkiSpkhiL5JpiXA/iuwiJxzJbSU6CpFFYpFEZAHOngVSUlgkmZq2jr/D1zaDF7wlEoRFEpEFiIkBnJyArl1FJ6HKsJJJCHc/zkuUEAnCIonIAsTEAH37Ara2opNQZUW4H8WVh3VwJb+26ChEFodFEpGZu3MHOHWKh9pMVS/XM7CzKmRvEpEANqIDEJEehIaW/Lo9tS8gTUa/b4cAP+QIDEVV4WBdiF4uZ7A1ozPeFR2GyMKwJ4nIzMVkBqOj0yV42rJAMlURHsdwOKcVsrJEJyGyLCySiMyYstgGO7PbI9z9mOgoVA3h7seghjV27hSdhMiysEgiMmMH77XGA7UDwt2Pi45C1VBLoZl9e9s20UmILAvHJBGZmn+MNypPTGYwaivS0brGH3oMRIYwwP04Fm5vDLUasLYWnYbIMrAnichMSRKwNTMY4e7HIJOJTkPVNcDtOLKygOPsFCQymCoVScuXL0d+fr6usxCRDiXl++NaQS0eajMT7Z2S4OkJHnIjMqAqFUnR0dHw8fHB66+/jqNHj+o6ExHpQExmMOytCtDT5YzoKKQD1rJi9OunmRiUiAyjSkXSn3/+iVWrViE7OxuhoaFo2rQp/vOf/yAtLU3X+YioimIyg9HL9QzsrZWio5CODBgAJCYCt26JTkJkGapUJFlbW2PgwIHYsGEDbt++jX//+9/46aefUKdOHQwcOBCbN29GcXGxrrMSUQXdU9XA4ZxWGODGQ23mJCwMsLEBYmNFJyGyDNUeuO3l5YUuXbogODgYVlZWSExMxKuvvooGDRpg//79OohIRJW1M7sD1LDGAI5HMisuLpqLFPOQG5FhVLlIunPnDmbPno0WLVqgR48eyM3NRUxMDK5fv46UlBQMGTIEo0aN0mVWIqqgmMxOaFPjKvzt7oqOQjo2YACwZw/Ac2eI9K9KRVJERAT8/f2xfPlyjBkzBsnJyVizZg169+4NALC3t8e7776L27dv6zQsEZVPLVkhNrMjZ9k2U+HhQEEBsG+f6CRE5q9Kk0l6eXnhwIEDCA5+8lWpfX19cf369SoHI6KqOZ7bHFlFziySzFSTJkD9+pqpAAYMEJ2GyLxVqSepe/fuaNu2bZnlSqUSK1asAADIZDIEBARULx0RVVpMZid4yO+hvVOS6CikBzKZpjjatk0zYSgR6U+ViqTXXnsNOTllryh+//59vPbaa9UORURVF5MZjP5uJ2At4xmm5io8XDMNwPnzopMQmbcqFUmSJEGm5ToHf/75J5ydnasdioiq5maBN87n1eehNjPXvTtQowZn3ybSt0qNSQoMDIRMJoNMJkOvXr1gY/P309VqNa5fv45nn31W5yGJqGK2ZXaCjawIYW6nRUchPVIogN69NUXSlCmi0xCZr0oVSYMHDwYAnD17Fn379kXNmjVLHrO1tUXdunUxdOhQnQYkoorbltkJIc7n4GyTJzoK6dmAAcDYsUBmJuDuLjoNkXmqVJE0depUAEDdunXxwgsvwM7OTi+hiKjy8tR22JPdFjPrfyc6ChlA//5AcTGwcycwYoToNETmqUpjkkaNGsUCicjI7M0ORKFky/FIFqJWLSAwkOOSiPSpwj1Jbm5uuHLlCjw8PODq6qp14PYjWVlZOglHRBW3LbMTGtr/icYOf4qOQgYyYADwzTdAUZHmmm5EpFsV/lh99dVXcHR0LPn9aUUSERmWJGlO/R/mtV90FDKg8HDg88+B48c113QjIt2qcJH0z+uwvfrqq/rIQkRV9NuDBkhWeiKcF7S1KO3bA56emkNuLJKIdK/CRVJubm6FX9TJyalKYYioarZldYKjdR5CnM+JjkIGZGUF9OsHxMQAs2aJTkNkfipcJLm4uJR7iO3RJJNqtbrawYio4mIygxHmehq2VkWio5CBhYcDK1YAN28CvBIUkW5VuEjap4dLTh88eBD//e9/ER8fj9TUVGzcuLFkLiZt9u/fj9DQ0DLLL126hKZNm+o8H5EpuKt0xoncZvi+yZeio5AAYWGaQdvbtgHjxolOQ2ReKlwkde/eXecbz8vLQ5s2bfDaa69VahLKpKSkUof0PD09dZ6NyFRsz+oICVbo535SdBTSNy1fEp0BdK05B9s+LsS4X6Ir9jp6+NJLZI4qXCSdO3cOLVu2hJWVFc6de/q4h9atW1foNfv164d+/fpVNEIJLy8vuLi4VGjdwsJCFBYWltx/NLZKpVJBpVI99bmPHi9vPdKO7Vd9WtvQ1rbUOluyu6C902W41cyDCqUfI0All5f6aY76eZ3C1KujkGPtCAfrwvKfUInPJD/H1cc2rD5RbSeTJEmqyIpWVlZIS0uDl5cXrKysIJPJoO2pVR2TJJPJKny4rW7duigoKEDz5s3x0UcfaT0E98i0adMwffr0MstXr14NBweHSuckMiZFRTK88ko/DBp0FS+8cEV0HBLkzz9rYvz4Xvjww+No3/6O6DhEepGfn48RI0YgJyfHYCeIVbgn6fr16yWHta5fv663QE/j6+uLJUuWICgoCIWFhVi5ciV69eqF/fv3o1u3blqfEx0djaioqJL7ubm58Pf3R1hYWLmNrFKpEBcXhz59+kBuxt9C9YXtV31a2zA8vOTx/VltkJ8/EBOuL0DgwquCUho3lVyOuDFj0GfpUsjN9Ju8JAFz7Jvh7o8Z6H9qYflPiImp8Gvzc1x9bMPqU6lU2Lx5s8G3W+EiKeAfp00ECDqFokmTJmjSpEnJ/eDgYNy+fRuzZ89+YpGkUCigUCjKLJfL5RV+s1ZmXSqL7Vd9pdpQqSxZviOtHfxs76K94iJkyic8mQAAcpUKcqX5NlK42zFsvNsVNoVKlDvXbxU+j/wcVx/b0PRU6dptgGbw9Pjx49GrVy/07t0b48ePR1JSki6zVUinTp3w+++/G3y7RMZgW2Yn9Hc/Uf4/RTJ7A9yP43ahNxLz6ouOQmQ2qlQk/frrr2jZsiXi4+PRpk0btG7dGmfOnEHLli3xyy+/6DrjUyUkJMDX19eg2yQyBlfz/ZD0sA5n2SYAQHeX31DTOh9bM4NFRyEyG1W6JOLkyZMRHR2NTz/9tNTyqVOn4v3338ewYcMq9DoPHjzA1at/j6O4fv06zp49Czc3N9SpUwfR0dFITk7GihUrAABz585F3bp10aJFCyiVSqxatQrr16/H+vXrq7IbRCZtW1YwFDIlernEi45CRkBhpUKY62lszeiMDwN+Eh2HyCxUqScpLS0Nr7zySpnlI0eORFpaWoVf5/Tp0wgMDERgYCAAICoqCoGBgfjkk08AAKmpqbh161bJ+kqlEpMmTULr1q0REhKCw4cPY9u2bRgyZEhVdoPIpMVkdkIPl7OoaVMgOgoZiQj3ozh5vynuKF1FRyEyC1XqSerRowcOHTqEhg0bllp++PBhhISEVOp1njYDwfLly0vdnzx5MiZPnlyprETm6H6RPQ7ca4M5DSpwJhNZjP7uJwBoxqr9n+92wWmITF+Fi6QtW7aU/D5w4EC8//77iI+PR6dOnQAAx48fxy+//KJ1TiIi0q247HZQSXIM4Hgk+gcv23vo5HQRWzODWSQR6UCFiyRtkzwuXLgQCxeW/iYbGRmJsWPHVjsYET1ZTGYwmjvcQD37ih/eJssw0P0oPrv5MgrUcthZm+e8UESGUuExScXFxRW6VWW2bSKquGJJhm2ZHRHufkx0FDJCER7HkF9sj333AkVHITJ5VZ4niYjEiL/fGOkqNxZJpFVzhxuoZ5eCrZmdRUchMnlVGrgNAHl5eThw4ABu3boF5WOz2L799tvVDkZE2sVkBsPVJhfBThdERyEjJJMBEe7HsDGjK76RwIlGiaqhSkVSQkIC+vfvj/z8fOTl5cHNzQ0ZGRlwcHCAl5cXiyQiPdqW2QnPup2CjVWx6ChkpCLcj2Je8lD89qABnnH8Q3QcIpNVpcNtEydOREREBLKysmBvb4/jx4/j5s2bCAoKwuzZs3WdkYj+kpICxD9owkNt9FTdXM7ByfoBD7kRVVOViqSzZ8/i3XffhbW1NaytrVFYWAh/f398+eWX+OCDD3SdkYj+EhsLWEGNvm6nREchI2ZrVYS+bqd5iRKiaqpSkSSXyyH760C3t7d3yazYzs7OpWbIJiLd2rIF6OJ8Hu7yXNFRyMhFuB/FqfvNkFroJjoKkcmqUpEUGBiI06dPAwBCQ0PxySef4KeffsKECRPQqlUrnQYkIo28PCAuDhjkcUR0FDIB/d1PwApqbMvsJDoKkcmqUpE0c+ZM+Pr6AgA+++wzuLu7480330R6ejqWLFmi04BEpLF7N1BQoJkskKg87vJcdHa+wHFJRNVQpbPb2rVrV/K7p6cnYmNjdRaIiLTbvBlo1gxo5JAsOgqZiAj3Y5h2YxQeqm1hb60s/wlEVEq1JpNMT0/HoUOHcPjwYdy9e1dXmYjoMWo1EBMDDBwoOgmZkgj3o3hYbIc92W1FRyEySVUqknJzc/Hyyy+jVq1a6N69O7p16wY/Pz+MHDkSOTk5us5IZPFOnJDh7l1g0CDRSciUNHW4hQZ2yTzkRlRFVSqSRo8ejRMnTiAmJgb37t1DTk4OYmJicPr0aYwZM0bXGYks3pYtMnh5AR07ik5CpkQmAyI8jiImsxMkSXQaItNTpSJp27Zt+P7779G3b184OTnB0dERffv2xdKlS7Ft2zZdZySyeDExVoiIAKx4tUWqpIHuR5Gi9MSZB41FRyEyOVX6k+vu7g5nZ+cyy52dneHq6lrtUET0t+TkmrhyRcZDbVQlXZ0T4Wz9AFszOLEkUWVVqUj66KOPEBUVhdTU1JJlaWlpeO+99/Dxxx/rLBwRASdP+sDeXkKvXqKTkCmSW6nRz/0ExyURVUGFpwAIDAwsmWUbAH7//XcEBASgTp06AIBbt25BoVDg7t27eOONN3SflMhCnTzpg969JTg48HLuVDUR7sfwUnovJBd6oJYiQ3QcIpNR4SJp8ODBeoxBRNqkpwOXL7th4kQ1qjljB1mwfm4nYA01YjKD8YbfVtFxiExGhYukqVOn6jMHEWmxfbum96h/f56aRFXnKn+Ars6J2JrBIomoMqo04/Yj8fHxuHTpEmQyGZo3b47AwEBd5SIiAFu2WKFJkyx4eTmJjkImLsLjKD66/jry1Qo4iA5DZCKq1H+fnp6Onj17on379nj77bcxfvx4BAUFoVevXpx5m0hHHj4Edu+WoUOHNNFRyAxEuB9DQbECu7ODREchMhlVKpLeeust5Obm4sKFC8jKykJ2djbOnz+P3NxcvP3227rOSGSRdu8GHj6UoWNHFklUfY0d/kRj+1vYmsmpAIgqqkqH23bs2IHdu3ejWbNmJcuaN2+Ob775BmFhYToLR2TJtmwBGjWSUKvWA9FRyExEuB/DT+m9UVzMiUmJKqJKH5Pi4mLI5fIyy+VyOYqLi6sdisjSFRcDW7cCERH8PJHuRHgcQ5rSHadPi05CZBqqVCT17NkT77zzDlJSUkqWJScnY+LEiejFGe+Iqi40FAgNxYl2kbhzB4g49J5meXh4yWNEVdXFKRHuNjnYtEl0EiLTUKUiacGCBbh//z7q1q2LBg0aoGHDhqhXrx7u37+P+fPn6zojkcXZktkZHvJ76ORySXQUMiM2VsUY5HEE69eDF7wlqoAqjUny9/fHmTNnEBcXh8uXL0OSJDRv3hy9e/fWdT4ii7Q5owvC3Y/BWsbDbaRbQzwP4fvE/rh0CWjeXHQaIuNW6SKpqKgIdnZ2OHv2LPr06YM+ffroIxeRxfo9vxYu5dfFzHrfiY5CZqiXSzwcHYENG1gkEZWn0ofbbGxsEBAQALVarY88RBZvS2Zn2FkVoo9bvOgoZIbsrFUYMEBTJBHR01VpTNJHH32E6OhoZGVl6ToPkcXbnNEFvV3jUcO6QHQUMlNDhgAJCcD166KTEBm3Ko1JmjdvHq5evQo/Pz8EBASgRo0apR4/c+aMTsIRWZoMpROO5LTEt43niI5CZqxfP0ChADZuBKKiRKchMl5VKpIGDx4MmUwGiadHEOnUtqxgSJAh3P2Y6ChkxmrWBMLCNIfcWCQRPVmliqT8/Hy899572LRpE1QqFXr16oX58+fDw8NDX/mILMqWjM7o6HQJPops0VHIzA0ZAvzf/wFpaYCPj+g0RMapUmOSpk6diuXLl2PAgAF48cUXsXv3brz55pv6ykZkUQoKgJ1Z7THQ/ajoKGQBIiI0lybZvFl0EiLjVamepA0bNmDZsmUYPnw4AOCll15Cly5doFarYW1trZeARJZi714gr9gegzyOiI5CFsDdHejRQ3PI7Y03RKchMk6V6km6ffs2QkJCSu536NABNjY2pS5PQkRVs2ED0ND+TzRzuCk6ClmIIUM0xXk2j+4SaVWpIkmtVsPW1rbUMhsbGxQVFek0FJGlUak0ZxoN8zwAmUx0GrIUgwYBRUVATIzoJETGqVKH2yRJwquvvgqFQlGyrKCgAGPHji01DcAGzlJGVCn79gFZWcC/6h0QHYUsSK1aQKdOml7Ml18WnYbI+FSqSBo1alSZZSNHjtRZGCJL9csvQP36QGDN30VHIQszZAjwySdAXh7w2JR3RBavUkXSDz/8oK8cRBarqEhzqG30aEB2QnQasjTPPQdMngzs2AEMHSo6DZFxqdJlSYhId/bvBzIzgWHDRCchS9SwIdC6Na/lRqQNiyQiwX75BahbF2jbVnQSslRDhmgGbyuVopMQGRcWSUQCPTrUNmwYeFYbCfPcc0BurmY6ACL6W5Wu3UZEVRAaWmbRwexA3L07B8P2jAVCkwSEIgJatQIaNNAccnv2WdFpiIwHe5KIBPrlbncEKNLQzpEFEokjk2kOuW3aBKjVotMQGQ8WSUSCqCUrbLgbgn9xAkkyAkOGAHfvAkd4VRyiEiySiAQ5dK8V0lVuGObFCSRJvA4dAD8/nuVG9E8skogE+eVuD/gr7qCD4yXRUYhgZaUZwL1hAyBJotMQGQcWSUQCqCUrbMgIwb88D/JQGxmN554Dbt8G4uNFJyEyDiySiAQ4ktMSaUp3DPPcLzoKUYlu3QA3N820FETEIolIiF/udkdtRTo6OvFQGxkPuRwYOJDjkogeEVokHTx4EBEREfDz84NMJsOmTZvKfc6BAwcQFBQEOzs71K9fH4sXL9Z/UCIdKpZkWH+3G4Z6HISVjIM/yLgMGQJcvgxcvCg6CZF4QoukvLw8tGnTBgsWLKjQ+tevX0f//v0REhKChIQEfPDBB3j77bexfv16PScl0p2jOS2QqvTgWW1klPr0AZycgLVrRSchEk/ojNv9+vVDv379Krz+4sWLUadOHcydOxcA0KxZM5w+fRqzZ8/GUF6+mkzEL3d7oJbtXQQ7XRAdhagMOztg6FBg9Wpg+nTRaYjEMqnLkhw7dgxhYWGllvXt2xfLli2DSqWCXC4v85zCwkIUFhaW3M/NzQUAqFQqqFSqp27v0ePlrUfasf0eY2urOdSW0Q3P+RyGWiFHeZMbq/56T6u0vLepYtiGWpTzmXz+eRl++MEGx44V4Zln+DmuLv4trD5RbWdSRVJaWhq8vb1LLfP29kZRUREyMjLg6+tb5jmzZs3CdC1fh3bt2gUHB4cKbTcuLq5qgQkA26/EuHG4fNkVyXGe8H3DA7EtxlX4qXFjxugxmGVgG/5DbOxTH1arAReXvpg1KxmjR58HwM+xLrANTY9JFUkAIHtsUhnpr1nPHl/+SHR0NKKiokru5+bmwt/fH2FhYXBycnrqtlQqFeLi4tCnTx+tvVT0dGy/x4SHY+/lN+CryMS7+2fA6kD5g7ZVcjnixoxBn6VLIee30CphG2oRE1PuKi+/bIWff66P1at9sXcvP8fVwb+F1adSqbB582aDb9ekiiQfHx+kpaWVWpaeng4bGxu4u7trfY5CoYBCoSizXC6XV/jNWpl1qSy2n0ZxoQob7nTFUI8DUKgKy3/CP8hVKsiVSj0lswxsw3+owOdx5Ehg/nzgyBHbv57Cz3F1sQ1Nj0nNkxQcHFymu3LXrl1o164d33hk9E7mNsWfhV74l+dB0VGIytW+PdCgAbB2rUn9myDSKaE9SQ8ePMDVq1dL7l+/fh1nz56Fm5sb6tSpg+joaCQnJ2PFihUAgLFjx2LBggWIiorCmDFjcOzYMSxbtgxr1qwRtQtEFfbL3R7wlmehq3Oi6Chk6UJDy11Ftm8fRowAvv5ahgEDWCiRZRL6zj99+jQCAwMRGBgIAIiKikJgYCA++eQTAEBqaipu3bpVsn69evUQGxuL/fv345lnnsFnn32GefPm8fR/MnqSBPx6txuGeh6EtaxYdByiChkxAsjNlSE+3rv8lYnMkNCepB49epQMvNZm+fLlZZZ1794dZ86c0WMqIt07dQq4VejDa7WRSWnaFAgMlHDwYG3RUYiEYB8qkQGsXQt4ybMQ4sJDbWRahg8vxunT3sjJEZ2EyPBYJBHpWVER8NNPwAjvPTzURiZn2LBiFBVZYfNm7dOsEJkzFklEerZzJ5CeDozy3ik6ClGl1a4NtGiRwbPcyCLxXU+kZytWAK1aAW1q/iE6ClGVdOuWjL17ZXhsmjois8ciiUiP7t0DNm8GRo0CnjApPJHRCw5OgbU18PPPopMQGRaLJCI9+vlnzbVER4wQnYSo6hwdVejbV8Lq1aKTEBkWiyQiPVqxAggLA7Rce5nIpAwfXowTJ4A/eNSYLAiLJCI9uXoVOHJEc6iNyNSFh0uoUQPgBQ7IkrBIItKTlSsBJydg0CDRSYiqz8EBeO45zXQWT5kDmMissEgi0oPiYs2htuefB+ztRach0o0RI4DLl4HffhOdhMgwWCQR6cHhw8CNG8Arr4hOQqQ7vXsDHh6a3iQiS8AiiUgPVqwA6tUDunQRnYRId+RyTe/omjWa3lIicyf0ArdEZiM0tOTXh2pb/Hx0A6Jq/wKrXj8KDEWkeyNGAAsXAocOAd27i05DpF8skoh0bFNGV9xX18DLPnGioxBV3aPC39YWGDcOCA8HlEp0loC6dqux+sXT6J4SITYjkZ7xcBuRjq24E4YuToloYJ8iOgqRzslkwItee/HL3e5QKkWnIdIvFklEOpRS6I5dWe0wyocXsyXzNcJrN7KLnBAbKzoJkX6xSCLSodV3ekEuU2OY537RUYj0pmXNG2jveAlLlohOQqRfLJKIdESSgB/v9MVgj8NwkeeJjkOkV2/4xWDHDuDmTdFJiPSHRRKRjpx90BDn8+rjFZ9doqMQ6d0LnntRsyawbJnoJET6wyKJSEdW3AmDtzwLYa6nREch0ruaNgV46SVNkVRUJDoNkX6wSCLSAVWxNVbf6YWXvHfDxoqz7JFleOMNICUF2LZNdBIi/WCRRKQDO7PaI13lxkNtZFGeeQZo3x4cwE1mi0USkQ6suNMXbWpcRZuaf4iOQmRQ//43sH07B3CTeWKRRFRN2dnA5ozO7EUiizR8ODiAm8wWiySialq3DlBL1hjhtVt0FCKDq1kTHMBNZotFElE1SJLmYp8D3I/DR5EtOg6REBzATeaKF7glqoaDB4HERGBO6w2ioxAZ3l8XwX0GQHvHhVjyfzkY1Dq69Dr79hk8FpGusCeJqBoWLACaNQN6uZ4RHYVIqH/7xmB7VgfcKvASHYVIZ1gkEVXR7dvAxo3A+PGaK6MTWbLhXntR0/ohvksdIDoKkc6wSCKqom+/BRwcgJdfFp2ESLyaNgV4yXs3lqX2Q1Ex/7WQeeA7magKCgo0E+i99hrg6Cg6DZFx+LdvDFKUnojN6iQ6CpFOsEgiqoKffwbu3gUiI0UnITIegY5X0d7xEr5NiRAdhUgnWCQRVcGCBUDfvkDjxqKTEBkXDuAmc8IiiaiSTpwATp3SDNgmotIeDeBeltpfdBSiamORRFRJCxYA9esD/fqJTkJkfB4N4P4utT8HcJPJ4zuYqBLu3NFchiQyErC2Fp2GyDhxADeZCxZJRJWwZAkgl2vOaiMi7R4N4F6SEi46ClG1sEgiqiCVCli8GBg5EnB1FZ2GyLj92zcGsVkdcfOm6CREVcciiaiCNm7UXMSTA7aJyjfcay+cbfIwb57oJERVxyKJqIIWLAC6dwdatRKdhMj41bQpQKTfJnz7LZCVJToNUdWwSCKqgN9+Aw4dYi8SUWW8XXsD1Grgm29EJyGqGhZJRBUwfz5QuzYweLDoJESmw8v2Hl5/HZg3D8jPF52GqPJsRAcgMnZZWcBPPwEffQTY8BNDVCnvnn4RizNW4fs2CzC+9ibtK+3bZ9BMRBXFniSicixbBhQXA2PGiE5CZHrq2adhuNdezL79AlTFnFyMTAu/FxM9hVoNLPwkFcNdz8HrhS9ExyEySe/XWYOfTvfBuvRQjPTZLToOUYWxJ4noKbZtA24U+GJ8rY2ioxCZrFY1r2OA2zF8cWsEiiWZ6DhEFcYiiegJJAmYORPo7HQe7Z2SRMchMmlT6qzGhfx6iM3sKDoKUYWxSCJ6gh07gBMngKl1fxQdhcjkdXU5jy5Oifji1gjRUYgqjEUSkRaSBEybBnTuDPRxPS06DpFZmFJnNY7ktsLhey1FRyGqEBZJRFps3w6cPKkplGQcQkGkE/3dT6BljWvsTSKTwSKJ6DGSBEydCnTpAvTuLToNkfmwkkl4338NtmUF49yD+qLjEJWLRRLRY2JjgdOngenT2YtEpGsveO1DHUUavrw1XHQUonKxSCL6h0djkUJCgJ49RachMj9yKzUm+f+Mtek9cf2hj+g4RE/FIonoH2JiNL1IHItEpD+v+8bCVX4f/7v9vOgoRE8lvEhauHAh6tWrBzs7OwQFBeHQoUNPXHf//v2QyWRlbpcvXzZgYjIroaElN6lHKKaNuIJuzr8h9NO/lxORbjlYF+LtWhuwLK0/0pUuouMQPZHQImndunWYMGECPvzwQyQkJCAkJAT9+vXDrVu3nvq8pKQkpKamltwaNWpkoMRkzrZmdsaZB40xve4P7EUi0rPIWptgjWLMT35OdBSiJxJaJM2ZMwevv/46Ro8ejWbNmmHu3Lnw9/fHokWLnvo8Ly8v+Pj4lNysrXnRRKoeSQKm3RiF7s5n0cP1N9FxiMyem/w+3vDbigXJz+H+fdFpiLQTdoFbpVKJ+Ph4TJkypdTysLAwHD169KnPDQwMREFBAZo3b46PPvoIoU85JFJYWIjCwsKS+7m5uQAAlUoFlUr11O08ery89Ug7k2g/W1sAwOb0YCQ8aIzd7SZB9dcyY6CSy0v9pMpjG1aPPttvfP3NWJA8GP/9rxoff1ys89c3Fibxt9DIiWo7mSRJkogNp6SkoFatWjhy5Ag6d+5csnzmzJn48ccfkZRU9lpZSUlJOHjwIIKCglBYWIiVK1di8eLF2L9/P7p166Z1O9OmTcP06dPLLF+9ejUcHBx0t0NksoqLgaioHnB0VOKzz55eoBORbv34Y3PExtbDwoV74O5eIDoOGbH8/HyMGDECOTk5cHJyMsg2hfUkPSJ7bPCHJElllj3SpEkTNGnSpOR+cHAwbt++jdmzZz+xSIqOjkZUVFTJ/dzcXPj7+yMsLKzcRlapVIiLi0OfPn0g57fQSjOJ9gsPx6Y7nXHjxiDsafcuQhaeF52oFJVcjrgxY9Bn6VLI+S20StiG1aPv9uuickBzpw3Yv78Pli1T6/z1jYFJ/C00ciqVCps3bzb4doUVSR4eHrC2tkZaWlqp5enp6fD29q7w63Tq1AmrVq164uMKhQIKhaLMcrlcXuE3a2XWpbKMuf2KC1X4/OpI9HQ5g541zwBK0Ym0k6tUkCuNNJyJYBtWj77azwNKfPqpDG++KcM771ghKEjnmzAaxvy3kLQTNnDb1tYWQUFBiIuLK7U8Li6u1OG38iQkJMDX11fX8chCbMwIwbm8Bphe9wfRUYgs1ujRQIsWQFSU5iQKImMh9HBbVFQUXn75ZbRr1w7BwcFYsmQJbt26hbFjxwLQHCpLTk7GihUrAABz585F3bp10aJFCyiVSqxatQrr16/H+vXrRe4GmajiYmD6jVfQ2/U0uroY12E2IktiYwP873/As88CGzcCQ4aITkSkIbRIeuGFF5CZmYlPP/0UqampaNmyJWJjYxEQEAAASE1NLTVnklKpxKRJk5CcnAx7e3u0aNEC27ZtQ//+/UXtApmw9euBxLwGWNT4LdFRiCxe375Av37A5MnAgAGAllESRAYnfOD2uHHjMG7cOK2PLV++vNT9yZMnY/LkyQZIReYuLw+YNAno73YcXZzZi0RkDGbPBlq3BhYsAN59V3QaIiO4LAmRCJ99BqSnA/MazRcdhYj+0rw58MYbms/n3bui0xCxSCILdOGCZvzDBx8ADexTRMchon+YNk3zU8v0dkQGxyKJLIokAePGAfXqacY+EJFx8fQEPvoIWLwYuHhRdBqydCySyKKsWAEcPAgsXMiBoUTG6q23gIAA4L33RCchS8ciiSxGVpZmsPbw4UDv3qLTENGTKBTAl18CsbHArl2i05AlY5FEFiM6GlAqgTlzRCchovIMGQJ066aZYLKoSHQaslQsksgiHD8OLFkCfP45wAnaiYyfTKb5QnPhArBsmeg0ZKmEz5NEpG9FRcDYsUDbtppB20RkZEJDtS4OAvCK9xR8/FYHDB/uCmdnw8YiYk8Smb0FC4Bz5zRny1hbi05DRJUxs/53yFfbYdIk0UnIErFIIrOWnAx8/LGmJ6l9e9FpiKiyaikyMLfhAnz3HbBpk+g0ZGlYJJFZi4oCHByAmTNFJyGiqnrdNxYDBwJjxgBpaaLTkCVhkURma9cu4OefNbNru7iITkNEVSWTAd99pzlc/vrrmklhiQyBA7fJLBUUAJGRmvGgL70kOg0RVZfn86FY5tUJ4bGzsLjJV3iz1payK+3bZ/hgZNZYJJFZmjYNuPmHCltcR0PW85boOESkAwPcj+NNv81494830dM1AU0cbouORGaOh9vI7MTEAP/5D/Bp3R/QrAYLJCJzMrvBIvgr0jHy0gdQFfN0VdIvFklkVq5fB15+GQgPBybXWSs6DhHpmIN1IVY1m4mzDxri05uviI5DZo5FEpmNggJg2DDA1VVzIVsrGUd3Epmj9k5JmBrwI2befAlHc1qIjkNmjEUSmY0JE4Dz54Fff9UUSkRkvqbUWY2OTpfw8qVo3C+yFx2HzBSLJDILK1cC334LzJ+vufwIEZk3G6tirGw2C+kqV0y4Ol50HDJTLJLI5J0/D7zxBjBqFDB6tOg0RGQoDexT8HXDBfg+rT823e0iOg6ZIRZJZNJyc4GhQ4GGDYGFCzWTzhGR5XjNZzsGexzC6KT3kJIiOg2ZGxZJZLIkSTP7bmqqZhySg4PoRERkaDIZsLTxbNhZKREervniRKQrnEySTNa8eZriaP16oHFj0WmISBQP21xsb/0+uibMw9C6l7GtVTRsrYq0r8xZuakS2JNEJunoUWDSJM0FbIcMEZ2GiERrVfM6NrX8GAfvtcbrSe+hWOKxd6o+9iSRyUlPB54PvYtONVLxxekoIFQtOhIRGYFQ17NY0ewLDL/4CfxsM/GfBktERyITxyKJTEpODhARAagka6xr/inkViyQiOhvL3jtQ2qhGyb+MR61FBl4u/YG0ZHIhLFIIpORmws8+yxw5Qqwu3U0/BSZoiMRkRGa4L8efxZ6YsLVSPjaZmKY1wHRkchEcUwSmYT794F+/YBLl4C4OCDI8YroSERkxL5s8C2Ge+3DyEsf4MC9NqLjkIliTxIZn9DQUncfFNmhf+J/cP5BfcS1eQ/t3rssKBgRmQormYQfmv4Hd5SuGJT4OQ4HvoWWNW+IjkUmhj1JZNTy1HYYkPgFfnvQALvaTEYHJxZIRFQxCisVNrb8GHXt0vDsuS9xu8BTdCQyMSySyGjlqe0QnjgTZx40ws7Wk9HR6ZLoSERkYpxs8hHbegpsZGo8e+5LZGWJTkSmhEUSGaV8tQIRiTNwKrcpdrR+H8HOF0VHIiIT5afIxI7Wk3FH6YrOnYE//hCdiEwFxySR0XmotsWg85/jZG4zbG/9Pro4nxcdiYhMXNMat3GsbST6J36BTs1qYlPLj5/8t4WzctNf2JNERuXhQ2DQ+c9xNKcFYltPQYhLouhIRGQmGjkk43jbSDRzuIleZ/+HNXd6io5ERo5FEhmNP/8EevcGDue0wrZW0ejmck50JCIyM+7yXMS1eQ/DvPZjxKWP8fmNkZAk0anIWPFwGxmFuDhgxAjAzg7Y2yYKnZw5SJuI9ENhpcKKprPQ0D4ZH994HVcf1sKSJv978kVxyWKxJ4mEUquB6dOBvn2Btm2BhASwQCIivZPJgKl1V2BVsxlYk94TYb/9F1kqR9GxyMiwSCJh7t4F+vfXFEnTpgGxsYCHh+hURGRJXvLejd1tJuF8Xj0En1mAq/l+oiOREWGRREIcPQoEBgJnzgA7dwKffAJYW4tORUSWKMQlEcfbjgMAdDqzELGxggOR0eCYJNKdxy4nAltbYNw4IDwcUCoBAJIEfP3nULx3bSw6OF7GuubTUXtmBjBTQF4ior80dEjBsbbj8dKlDzFgQEcMHw58/TXg5SU6GYnEniQymJyiGhh2YRom/jEe79Raj/3PTEBtuwzRsYiIAABu8vuIbTUFK1ZoTiZp1gxYvhw8+82CsUgivVNLVliW2h9NT/6IuOwgrG/xCWY3XAy5lVp0NCKiUmQy4OWXgUuXNGMmX3sN6NOHs3RbKhZJpFe//eaBDse+weik99DTJQGJ7V/HEM9DomMRET2VpyewciWwYwdw9SrQsiXw5ZdAEWcJsCgck0R6kZTvj3cvvIltu4IR7HIBx9uO4wVqicg0/GN8ZV8A5/3t8In6/xD9/hCs+ewavmsyG0GnvxWXjwyGRRLpVKbKCdNvjMKilIGorcjApEmnMOO3j2CrUoqORkRUJTVtCjCn4UK86LUHY5ImoUP8Qoz02YnJddaiRY0bT34irwFn8ni4jXRCqQS+uv0vNDyxCsvT+uLzestwrstodO2aAplMdDoioupr75SEU0Fj8VXDhdh7LxAtT/2AiMQZOHSvFQd3mykWSVQtN24AH34IBAQAk/4Yi+Fee3G140i8X2ct7KxVouMREemU3EqNt2tvwB8dX8KPTWfh+kNfdDs7D10S5mPT3S4olvit0JywSKJKU6uBrVuBAQOA+vWBBQuAoUOB8+3/D4saz4WX7T3REYmI9MrWqgiv+OzCufavY2vLaNjI1HjuwudofnI5lqX2R2GxXHRE0gEWSVRhycnAp58CdesCAwcC6enA0qVASoqmUGpW45boiEREBmUlkxDucRwHAyfgaGAkmtW4idFJ76He8dWIjgbi4znPkinjwG16qtxcYM8eYOW/D2JLRhcorJR4yXsP3gjaiqCaV4BV0NyIiCxcsPNFbHT+BJfz/DH3z39hyezu+OILZ9S1y8QzzzaHx8x30dk+sfQ4TQ7uNmoskqgUSQISE4Ht2zW3I0c084K0rlEL8xrNx0veu+Fskyc6JhGR0Wpa4zYWN/kK8xvNw4F7bfBzVk/8vK83NuXMg7/iDoZ6HsRQj4Po7HyBh3OMHIskwr2u4didHYTtWR2xI6s9UpSecLB6iJ6uCfi63kk863YS9e1TRcckIjIpcis1erudQXef83j2jSI4zzqETamdsS49FHP/HAZf2wz0ehno3Flza9mSF/o2NiySLMzDh5qeooQE4MwZzS3h9GaoYY3mDjcw3Gsf+rmdQIhLIhRWPDuNiEgXrK2Bbm6J6FUzHl83XIBjuS2w4W4IDiU9j7VrNT32jo5Ap05/F00dOwLOzqKTWzYWSWZKkoDsbODChb8Kob+KoosXNWenWVsDLVoAbdsCo3O/xrNuJxFgd0d0bCIis2clk9DF+Ty6OJ8H9j2P/Hzg9Gng6FHNbcECYPp0zXXkWrQA2rQBmjT5+9aoEeDgIHovLIPwImnhwoX473//i9TUVLRo0QJz585FSEjIE9c/cOAAoqKicOHCBfj5+WHy5MkYO3asARMbj4cPNfMUXb8OXLum+fnP33NzNespFEDr1kBwMBAZqSmMWrUC7Oz+eqHQraJ2gYjI4jk4AN26aW6A5kvulSuagunYMeDCuvPYsc4fmUV/dyvVUaShicNtzc3+Nup+9Q58fQE/P8DLi4ftdEVokbRu3TpMmDABCxcuRJcuXfDtt9+iX79+uHjxIurUqVNm/evXr6N///4YM2YMVq1ahSNHjmDcuHHw9PTE0KFDBeyB7qjVwP37msImNxfIytKcYn/3ruantt8zMv5+vlymQoDdHdSzS0Unu1S86JGKerXT0GT1VDRvDsg5ZQcRkfH5x3XiHpEBaPLX7TUAaKtZnqlyQlK+/9+3h/7Yk90W36ZEQBXx9/OtrAAfH03B5OeHkuLJwwNwcQFcXTW3R7+7uGi+TFNZQoukOXPm4PXXX8fo0aMBAHPnzsXOnTuxaNEizJo1q8z6ixcvRp06dTB37lwAQLNmzXD69GnMnj3bKIqkLVuAG+/OR0GxLR4W22p+qhV/3VeULM9r06WkGMrN1RRHeU84YczaWvOtwNNT89PPD3jmmb9+X/EF6tmnob5dCvwUmbCWFZd9gQn79bnLRERkIO7yXHR2voDOzhdKLVdLVkhfswcpKSi5pab+/fvJk5qfmZmasU/a2NtriiVHR03PVo0amp/aflcoNDdb279/Pv57w4ZA06b6bxN9E1YkKZVKxMfHY8qUKaWWh4WF4ejRo1qfc+zYMYSFhZVa1rdvXyxbtgwqlQpyLd0lhYWFKCwsLLmfk5MDAMjKyoJK9fSBySqVCvn5+cjMzNT62o+bPdsaJ/8YDnsrJRTWKthbFcLWSgV7ayXsrDQ3hdVDOJ+PQW2bh3C0zkdN+wI41syHo00+alrnw9GmAI42+XCe/zk8PTWD9qyedI7onriSX+8BMLa5QVWApv0AyJ+4E/Q0bMPqYxtWD9uv+gzRhrYje6EugLraHnTV3CQJyFfb4l6RI3JUDshR10SOqgbuFdVErsoB99Q1kddHM0YqP1+Ghw81X+AzMlBqWWFaNpTF1lAV26CwWA4VbMts8s031fjsMy1f3Kvo0f9jAJAMOTunJEhycrIEQDpy5Eip5TNmzJAaN26s9TmNGjWSZsyYUWrZkSNHJABSSkqK1udMnTpVAsAbb7zxxhtvvJnB7fbt27opRCpA+MBt2WOXiJckqcyy8tbXtvyR6OhoREVFldwvLi5GVlYW3N3dn7odAMjNzYW/vz9u374NJyenp65LZbH9qo9tWH1sw+ph+1Uf27D6HrXhxYsX4efnZ7DtCiuSPDw8YG1tjbS0tFLL09PT4e3trfU5Pj4+Wte3sbGBu7u71ucoFAooHhuR5uLiUqmsTk5OfGNXA9uv+tiG1cc2rB62X/WxDauvVq1asDLgYV9hB5htbW0RFBSEuLi4Usvj4uLQuXNnrc8JDg4us/6uXbvQrl27Co0ZIiIiIqoooaPwoqKi8N133+H777/HpUuXMHHiRNy6datk3qPo6Gi88sorJeuPHTsWN2/eRFRUFC5duoTvv/8ey5Ytw6RJk0TtAhEREZkpoWOSXnjhBWRmZuLTTz9FamoqWrZsidjYWAQEBAAAUlNTcevWrZL169Wrh9jYWEycOBHffPMN/Pz8MG/ePL2d/q9QKDB16tQyh+uoYth+1cc2rD62YfWw/aqPbVh9otpQJkmGPJeOiIiIyDRw0gsiIiIiLVgkEREREWnBIomIiIhICxZJRERERFpYTJF0//59TJgwAQEBAbC3t0fnzp1x6tSpksfv3LmDV199FX5+fnBwcMCzzz6L33//vcKvv3btWshkMgwePFgP6Y2Dvtrw3r17iIyMhK+vL+zs7NCsWTPExsbqc1eE0Ff7zZ07F02aNIG9vT38/f0xceJEFBQU6HNXDOLgwYOIiIiAn58fZDIZNm3aVOpxSZIwbdo0+Pn5wd7eHj169MCFC6Uv/FlYWIi33noLHh4eqFGjBgYOHIg///yz3G0vXLgQ9erVg52dHYKCgnDo0CFd7prBiGrDWbNmoX379nB0dISXlxcGDx6MpKQkXe+eQYh8Hz4ya9YsyGQyTJgwQQd7ZFgi2y85ORkjR46Eu7s7HBwc8MwzzyA+Pr5S+S2mSBo9ejTi4uKwcuVKJCYmIiwsDL1790ZycjIkScLgwYNx7do1bN68GQkJCQgICEDv3r2Rl5dX7mvfvHkTkyZNQkhIiAH2RBx9tKFSqUSfPn1w48YN/Prrr0hKSsLSpUtRq1YtA+6ZYeij/X766SdMmTIFU6dOxaVLl7Bs2TKsW7cO0dHRBtwz/cjLy0ObNm2wYMECrY9/+eWXmDNnDhYsWIBTp07Bx8cHffr0wf3790vWmTBhAjZu3Ii1a9fi8OHDePDgAcLDw6FWq5+43XXr1mHChAn48MMPkZCQgJCQEPTr16/UdCSmQlQbHjhwAJGRkTh+/Dji4uJQVFSEsLCwCv09NTai2vCRU6dOYcmSJWjdurXO9smQRLVfdnY2unTpArlcju3bt+PixYv43//+V+krbgi7wK0h5efnS9bW1lJMTEyp5W3atJE+/PBDKSkpSQIgnT9/vuSxoqIiyc3NTVq6dOlTX7uoqEjq0qWL9N1330mjRo2SBg0apI9dEE5fbbho0SKpfv36klKp1Ft2Y6Cv9ouMjJR69uxZallUVJTUtWtX3e6AYACkjRs3ltwvLi6WfHx8pC+++KJkWUFBgeTs7CwtXrxYkiRJunfvniSXy6W1a9eWrJOcnCxZWVlJO3bseOK2OnToII0dO7bUsqZNm0pTpkzR0d6IYcg2fFx6eroEQDpw4ED1d0QgQ7fh/fv3pUaNGklxcXFS9+7dpXfeeUen+2Nohmy/999/Xyd/By2iJ6moqAhqtRp2dnalltvb2+Pw4cMoLCwEgFKPW1tbw9bWFocPH37qa3/66afw9PTE66+/rvvgRkRfbbhlyxYEBwcjMjIS3t7eaNmyJWbOnFmhb1imRF/t17VrV8THx+PkyZMAgGvXriE2NhYDBgzQw14Yj+vXryMtLQ1hYWElyxQKBbp3746jR48CAOLj46FSqUqt4+fnh5YtW5as8zilUon4+PhSzwGAsLCwJz7HVOmrDbXJyckBALi5uekovXHQdxtGRkZiwIAB6N27t352QDB9tt+WLVvQrl07DBs2DF5eXggMDMTSpUsrndEiiiRHR0cEBwfjs88+Q0pKCtRqNVatWoUTJ04gNTUVTZs2RUBAAKKjo5GdnQ2lUokvvvgCaWlpSE1NfeLrHjlyBMuWLatSw5safbXhtWvX8Ouvv0KtViM2NhYfffQR/ve//2HGjBkG3Dv901f7DR8+HJ999hm6du0KuVyOBg0aIDQ0FFOmTDHg3hneowtdP34xbG9v75LH0tLSYGtrC1dX1yeu87iMjAyo1eqnvq650FcbPk6SJERFRaFr165o2bKlDpIbD3224dq1a3HmzBnMmjVLx6mNhz7b79q1a1i0aBEaNWqEnTt3YuzYsXj77bexYsWKSmW0iCIJAFauXAlJklCrVi0oFArMmzcPI0aMgLW1NeRyOdavX48rV67Azc0NDg4O2L9/P/r16wdra2utr3f//n2MHDkSS5cuhYeHh4H3RgxdtyEAFBcXw8vLC0uWLEFQUBCGDx+ODz/8EIsWLTLgnhmGPtpv//79mDFjBhYuXIgzZ85gw4YNiImJwWeffWbAPRNHJpOVui9JUpllj6vIOlV5XVOlrzZ8ZPz48Th37hzWrFlT5YzGTtdtePv2bbzzzjtYtWpVmd5nc6SP92BxcTHatm2LmTNnIjAwEG+88QbGjBlT6f8tFlMkNWjQAAcOHMCDBw9w+/ZtnDx5EiqVCvXq1QMABAUF4ezZs7h37x5SU1OxY8cOZGZmljz+uD/++AM3btxAREQEbGxsYGNjgxUrVmDLli2wsbHBH3/8YcjdMwhdtyEA+Pr6onHjxqUKgWbNmiEtLQ1KpVLv+2RI+mi/jz/+GC+//DJGjx6NVq1a4bnnnsPMmTMxa9YsFBcXG2rXDM7HxwcAynyTTE9PL/lW6uPjA6VSiezs7Ceu8zgPDw9YW1s/9XXNhb7a8J/eeustbNmyBfv27UPt2rV1lNx46KsN4+PjkZ6ejqCgoJL/LwcOHMC8efNgY2NjNsMR9Pke9PX1RfPmzUsta9asWaVPwLCYIumRGjVqwNfXF9nZ2di5cycGDRpU6nFnZ2d4enri999/x+nTp8s8/kjTpk2RmJiIs2fPltwGDhyI0NBQnD17Fv7+/obYHSF01YYA0KVLF1y9erXUP/QrV67A19cXtra2etsHkXTZfvn5+bCyKv0xtra2hiRJkMz4soz16tWDj48P4uLiSpYplUocOHAAnTt3BqApOuVyeal1UlNTcf78+ZJ1Hmdra4ugoKBSzwGAuLi4Jz7HVOmrDQHNt/zx48djw4YN2Lt371MLfVOmrzbs1atXmf8v7dq1w0svvYSzZ88+tXfZlOjzPdilS5cy005cuXIFAQEBlQtZ7aHfJmLHjh3S9u3bpWvXrkm7du2S2rRpI3Xo0KHkrKqff/5Z2rdvn/THH39ImzZtkgICAqQhQ4aUeo2XX375qWe4mPPZbZKknza8deuWVLNmTWn8+PFSUlKSFBMTI3l5eUmff/65QffNEPTRflOnTpUcHR2lNWvWlLxugwYNpOeff96g+6YP9+/flxISEqSEhAQJgDRnzhwpISFBunnzpiRJkvTFF19Izs7O0oYNG6TExETpxRdflHx9faXc3NyS1xg7dqxUu3Ztaffu3dKZM2eknj17Sm3atJGKiopK1unZs6c0f/78kvtr166V5HK5tGzZMunixYvShAkTpBo1akg3btww3M7riKg2fPPNNyVnZ2dp//79UmpqasktPz/fcDuvI6La8HGmenabqPY7efKkZGNjI82YMUP6/fffpZ9++klycHCQVq1aVan8FlMkrVu3Tqpfv75ka2sr+fj4SJGRkdK9e/dKHv/666+l2rVrS3K5XKpTp4700UcfSYWFhaVeo3v37tKoUaOeuA1zL5L01YZHjx6VOnbsKCkUCql+/frSjBkzSr35zYU+2k+lUknTpk2TGjRoINnZ2Un+/v7SuHHjpOzsbAPtlf7s27dPAlDm9mj/i4uLpalTp0o+Pj6SQqGQunXrJiUmJpZ6jYcPH0rjx4+X3NzcJHt7eyk8PFy6detWqXUCAgKkqVOnllr2zTffSAEBAZKtra3Utm1bkz11XVQbatsmAOmHH37Q8x7rnsj34T+ZapEksv22bt0qtWzZUlIoFFLTpk2lJUuWVDq/TJLMuE+eiIiIqIosbkwSERERUUWwSCIiIiLSgkUSERERkRYskoiIiIi0YJFEREREpAWLJCIiIiItWCQRERERacEiiYiIiEgLFklEREREWrBIIiIiItKCRRIRERGRFiySiMjk3b17Fz4+Ppg5c2bJshMnTsDW1ha7du0SmIyITBkvcEtEZiE2NhaDBw/G0aNH0bRpUwQGBmLAgAGYO3eu6GhEZKJYJBGR2YiMjMTu3bvRvn17/Pbbbzh16hTs7OxExyIiE8UiiYjMxsOHD9GyZUvcvn0bp0+fRuvWrUVHIiITxjFJRGQ2rl27hpSUFBQXF+PmzZui4xCRiWNPEhGZBaVSiQ4dOuCZZ55B06ZNMWfOHCQmJsLb21t0NCIyUSySiMgsvPfee/j111/x22+/oWbNmggNDYWjoyNiYmJERyMiE8XDbURk8vbv34+5c+di5cqVcHJygpWVFVauXInDhw9j0aJFouMRkYliTxIRERGRFuxJIiIiItKCRRIRERGRFiySiIiIiLRgkURERESkBYskIiIiIi1YJBERERFpwSKJiIiISAsWSURERERasEgiIiIi0oJFEhEREZEWLJKIiIiItPh/48hUlYWQ61sAAAAASUVORK5CYII=",
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAksAAAGwCAYAAAC5ACFFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABWNUlEQVR4nO3deViUVf8G8HsYhk3ZdxRxyd3cUHNXlMU907JsszLfLLSMzOLNMtvsLSszU9M0f2oupWYmpAwKqLkCmvuWC4YgsggIwgzD8/tjkkQQB5iZM8v9ua65bJ55ZuZ+TjPDd85z5hyZJEkSiIiIiKhaNqIDEBEREZkyFktERERENWCxRERERFQDFktERERENWCxRERERFQDFktERERENWCxRERERFQDW9EBjK28vBxXr16Fs7MzZDKZ6DhERESkA0mSUFhYiICAANjYGLevx+qKpatXryIwMFB0DCIiIqqDK1euoHHjxkZ9TqsrlpydnQFoG9vFxUVwGv1Qq9WIi4tDeHg4FAqF6Dgmi+10f2wj3bCddMN20g3bSTe5ublo1qxZxd9xY7K6Yun2qTcXFxeLKpacnJzg4uLCN1oN2E73xzbSDdtJN2wn3bCddKNWqwFAyBAaDvAmIiIiqgGLJSIiIqIasFgiIiIiqgGLJSIiIqIasFgiIiIiqgGLJSIiIqIasFgiIiIiqgGLJSIiIqIasFgiIiIiqgGLJSIiIqIasFgiIiIiqgGLJSIiIqIasFgiIiIiqgGLJSIiIqIasFgiIiIiqoGt6ABERMZ06RJwatRbCHNPhq1N+b13TEgwWiYiMm3sWSIiiyZJwMmTwEcfAV27As2aAcOO/Q/9j3yNc8WNRMcjIjPAYomILI4kAYcOAdHRQJs2QPv2wP/+B7RuDaxfD+zoFIXrajd0Tl6KhekPQ5JEJyYiU8ZiiYgsRl4eMG0a0KQJ0KMHsHQp0LcvsHUrcP06sHYtMG4cMMj9MI50m4QJfnGIPDcNQ45+hr9LvETHJyITxTFLRGQRysuBp54C/vgDmDABGDNGWyjZ3uNTroG8BAtbzcPDXn/ghdNv4sHk5fi25dcY77MDMpmOTzpiBKBS1bwPxz4RmT0WS0RkEebMAbZtA2I7zMCQY4eAY7rdL8LjEI51n4gp517DU6dmYnN2Xyxs+RXYz0REt/E0HBGZvR07gPfeA959FxjieajW9/dQFGJNu4+wvt1s7Mjrgg6HfkBMjAGCEpFZYrFERGYtPR0YPx4YPFhbMNXHOJ9EHOs+EV2dz2LECCAxUS8RicjMsVgiIrOlVmsHbNvbAz/+CMjl9X/MAPscbH3wv+jdG3jllfsPSSIiy8diiYjM1ltvAQcPAj/9BHh76+9xbWQSFi0Czp4FvvpKf49LROaJxRIRmaUNG7SFzBdfAL166f/xO3YEXn0V+OAD4PJl/T8+EZkPFktEZHbOngVeeEF7Cm7qVMM9z/vvA25u2rmbiMh6sVgiIrNSVASMHQsEBADffw/d50SqAxcXbe/V5s3aiS2JyDqxWCIisyFJwMsvAxcuABs3As7Ohn/Oxx4DwsK0PVjFxYZ/PiIyPSyWiMhsLF0KrFoFLFmiXe/NGGQyYMEC4OpV7cSXRGR9WCwRkVk4cULbu/Pyy9plTYypVSvtL+8++ww4c8a4z01E4gktlubMmYPu3bvD2dkZPj4+GD16NM7c55MoMTERMpmsyuX06dNGSk1EInz8MeDvL+6n/NHRQOPGwJQp2tOBRGQ9hBZLSUlJiIyMxP79+6FUKlFWVobw8HAUFRXd975nzpxBRkZGxaVly5ZGSExEIly+rJ1L6Y03tBNQiuDoqD0dFx8P/PyzAUeVE5HJEbqQ7rZt2ypd/+GHH+Dj44OUlBT079+/xvv6+PjAzc3NgOmIyFR89RXg6qqdLkCkoUOBMWOAN9+U44svuA45kbUwqXd7fn4+AMDDw+O++3bp0gUlJSVo164dZs6ciZCQkGr3Ky0tRWlpacX1goICAIBarYZardZDavFuH4elHI+hsJ3uzxTbKDcX+P57W7z6ajns7Mpx32h2dvp54ns80eefAx072mLt2jZ4WKGo8+NYA1N8PZkitpNuRLaPTJJM4+y7JEl4+OGHkZeXh927d99zvzNnzmDXrl0IDg5GaWkpVq1ahcWLFyMxMbHa3qj3338fs2fPrrJ9zZo1cHJy0usxEJH+/fxzK/z0UyssXaqEm1vp/e9gBL/88gBWrWqHuXMT0bx5geg4RFahuLgYTz75JPLz8+Hi4mLU5zaZYikyMhIxMTHYs2cPGjduXKv7jhw5EjKZDFu2bKlyW3U9S4GBgcjOzjZ6YxuKWq2GUqlEWFgYFLp807VSbKf7M7U2KikBWra0xahR5fj223Ld7jRihGFDASiWO+DB46vgX3gVu7pPg42sho9RK57N0tReT6aK7aSbnJwc+Pv7CymWTOI03NSpU7Flyxbs2rWr1oUSAPTs2ROrV6+u9jZ7e3vYVzMiVKFQWNyL0hKPyRDYTvdnKm20YgWQlaUdI6RQyHW7k0pl0EwA4GQH/Oc/x/Duu32wI7MzhnkeuPfOJtCOopnK68nUsZ1qJrJthP4aTpIkTJkyBZs2bcLOnTvRrFmzOj3O4cOH4e/vr+d0RCRSeTkwdy4wejRgij927dAhG52cz+O7qyNFRyEiAxPasxQZGYk1a9bg119/hbOzMzIzMwEArq6ucHR0BABER0cjPT0dK1euBADMmzcPTZs2Rfv27aFSqbB69Wps3LgRGzduFHYcRKR/W7ZoF8z9v/8TnaR6MhkwqXEsXj0Vib9LvNDYIVt0JCIyEKE9S4sWLUJ+fj4GDhwIf3//isv69esr9snIyEBaWlrFdZVKhenTp6Njx47o168f9uzZg5iYGIwZM0bEIRCRgXz+OdC3L9Czp+gk9zbefyec5KX4PmO46ChEZEBCe5Z0GVu+YsWKStdnzJiBGTNmGCgREZmCvXu1l19/FZ2kZs62t/CUTzyWZgzHzKBVsLXRcRA6EZkVrg1HRCbn88+BNm2M8sO2ensp4DdcVXkjJreX6ChEZCAslojIpJw5o+1Rmj4dsDGDT6guzufRw/kUFnOgN5HFMoOPIiKyJl98Afj6Ak8/LTqJ7l4K+A3bc7vj4i0/0VGIyABYLBGRybh2DVi5Enj1VXEL5tbF4z4JcJEXY2mGGZw3JKJaY7FERCbjm2+0czhOniw6Se00kJfgWb84LMsYClW5Scz1S0R6xGKJiEzCzZvAwoXApEmAu7voNLX3UsBvyFJ74NfsPqKjEJGe8SsQEZmEZcuAggJg2rQadgoJMVacWmvf4BL6uh7F4quj8JhPkug4RKRH7FkiIuHKyoCvvgKeeAJo0kR0mrp7yf837LzRFWeLa7/GJRGZLhZLRCTcr78Cly9rpwswZ496J8HDNh9LrnKgN5ElYbFERMKtWQN06wZ07iw6Sf04yNV43m8bfsgcghINV48nshQslohIqIICICZGewrOEvwnYCtyy1yxMXuA6ChEpCcslohIqF9/BUpLgXHjRCfRj1ZOf2OQWypn9CayICyWiEiodeuAvn2BwEDRSfTnpYDfsCe/I04UNRUdhYj0gMUSEQmTkwPExVnOKbjbRnvtgY8iF9+xd4nIIrBYIiJhNm0CysuBRx8VnUS/7GzKMNH/d6zMDEdRkeg0RFRfLJaISJh164DBg7UL51qaSf5bUaBxwvr1opMQUX2xWCIiITIygIQEyzsFd1szx0xEeBzCd9+JTkJE9cViiYiE+PlnwNYWeOQR0UkM5yX/33DwIHDsmOgkRFQfLJaISIh164AhQ8xz0VxdDfM8ABcX7dgsIjJfLJaIyOguXQL27bPcU3C32dmUYeRIFktE5o7FEhEZ3U8/AY6OwKhRopMY3pgxwNGjwPnzopMQUV3Zig5ARFYgJKTS1XXJ32FEw6toOHL2vxsTEowcyjgiIrSF4aZNwIwZotMQUV2wWCIiozpTHIjDN1thZtDqyjfcVVBZigYNtGOzWCwRmS+ehiMio1qXFQJneRGGehwQHcVoxowBDhwA/v5bdBIiqgsWS0RkNJIErMsahEe89sBRrhIdx2hGjNBOk7B5s+gkRFQXLJaIyGiOFrXA6eIgPOGzU3QUo3Jz085Uzl/FEZknFktEZDTrskLgYZuPUPcU0VGMbswYICkJyM4WnYSIaovFEhEZxe1TcI9674LCRiM6jtE9/LC2DbZsEZ2EiGqLxRIRGcXBwra4VOJvdafgbvP1Bfr25ak4InPEYomIjGJdVgj87HLQ3+2o6CjCjBkDKJVAQYHoJERUGyyWiMjgNJIN1meFYJx3IuSyctFxhHnkEUClAmJjRSchotpgsUREBrf7xoPIUHlZ7Sm424KCgOBgnoojMjcslojI4NZlDUKQfSZ6upwUHUW4MWO0PUu3bolOQkS64nInRGRQajWw4foAvOgfA5lMdBoB7lrGZUxRIN4pWgllj3cwymuvdqOFrotHZCnYs0REBrVzJ5BT5mr1p+Bua9PgCto6XcKm6/1ERyEiHbFYIiKD+u03oKlDBjo1/Et0FJMxxms3tuT0hrpcLjoKEemAxRIRGYwkATExwHCP/dZ5Cu4exnjvRl6ZC5JudBIdhYh0wGKJiAzm1Cng0iVguOd+0VFMSpeG59DUIQObsvuLjkJEOmCxREQGExMDODoCA92OiI5iUmQy7am4X7L7olxilxuRqWOxREQGExMDDBoEOMpVoqOYnDHeu5Gp8sT+gnaioxDRfbBYIiKDyM8H9uwBhg8XncQ09XI5AT+7HP4qjsgMsFgiIoOIiwM0GmDYMNFJTJONTMJorz3YlN0PkiQ6DRHVhMUSERlETAzQvr12iQ+q3hiv3bhYEoA//xSdhIhqwmKJiPSuvBz4/XeegrufgW5H4GZbiI0bRSchopqwWCIivUtOBrKyWCzdj8JGg1Gee7mwLpGJY7FERHoXGwu4uQG9e4tOYvrGeO/GyZPA2bOikxDRvbBYIiK9i4kBIiIAWy7VfV+h7imws9OetiQi08SPMiKqn5CQSlczS92RnLwJU2/OAULiBIUyHw3kJejfH9i2DXjtNdFpiKg67FkiIr36PfchyFCOoR4HREcxG0OGAImJwK1bopMQUXVYLBGRXsXmPoQezqfhbZcvOorZGDIEKCkBdu0SnYSIqsNiiYj0Rl0uR1xuNy6cW0vt2gGNG2tPxRGR6WGxRER6syf/QRRoGmKYJ0/B1YZMpu1dYrFEZJo4wJuI9CYmpyf87HLQpeE50VHMS0gIhlzvj+9Pz8blXk8gyOFa9fslJBg3FxEBYM8SEelRTG5PDPM4ABsZFzurrcFuKZBDg+253UVHIaK7sFgiIr24cMsfp4uDOF6pjtwURejlegLbWCwRmRwWS0SkF7E5D0EhUyPUPUV0FLM1xOMg4vOCoS6Xi45CRHcQWizNmTMH3bt3h7OzM3x8fDB69GicOXPmvvdLSkpCcHAwHBwc0Lx5cyxevNgIaYmoJjG5PdHP9RhcbItFRzFbQzwOoVDTAPsK2ouOQkR3EFosJSUlITIyEvv374dSqURZWRnCw8NRVFR0z/tcvHgRw4YNQ79+/XD48GH897//xauvvoqNXLabSJgijQMS8rrwFFw9dWl4Dt6KPGzL7SE6ChHdQeiv4bbd9TvZH374AT4+PkhJSUH//v2rvc/ixYvRpEkTzJs3DwDQtm1bJCcnY+7cuRg7dmyV/UtLS1FaWlpxvaCgAACgVquhVqv1dCRi3T4OSzkeQ2E73V+d2sjODnFZ3VEq2SHcLwVqOzsDpTMdaoWi0r/6FOaVgt/zemC23cpqnti8Xrt8z+mG7aQbke0jkyTJZH62cv78ebRs2RLHjh1Dhw4dqt2nf//+6NKlC77++uuKbb/88gvGjRuH4uJiKO768Hr//fcxe/bsKo+zZs0aODk56fcAiKzU4sUdceSINxYt2gGZTHQa85aU1BhffRWMH37YBnf30vvfgchKFBcX48knn0R+fj5cXFyM+twmM8+SJEmIiopC375971koAUBmZiZ8fX0rbfP19UVZWRmys7Ph7+9f6bbo6GhERUVVXC8oKEBgYCDCw8ON3tiGolaroVQqERYWVqVYpH+xne6vLm0kDR+BqbtXYaxPPIYvWmTghKZBrVBAOWkSwpYuhULP33a7lbriK/yE8i+OYVhAfOUbt27V63MZGt9zumE76SYnJ0fYc5tMsTRlyhQcPXoUe/bsue++sru+ut7uHLt7OwDY29vD3t6+ynaFQmFxL0pLPCZDYDvdX23a6FheI1wp8cFIt71QqFQGTmZaFGq13o+5kew6ghueQXxWV7zgFXvXE5rn65bvOd2wnWomsm1MYuqAqVOnYsuWLUhISEDjxo1r3NfPzw+ZmZmVtmVlZcHW1haenp6GjElE1YjJ6Qknm1sY4HpEdBSLMcTjIOJyu0EjmcRHNJHVE/pOlCQJU6ZMwaZNm7Bz5040a9bsvvfp1asXlEplpW1xcXHo1q0bK3IiAWJyeyLUPRUOcg5O1ZchHgeRU+aKlMJWoqMQEQQXS5GRkVi9ejXWrFkDZ2dnZGZmIjMzE7du3arYJzo6Gs8++2zF9cmTJ+Py5cuIiorCqVOnsHz5cixbtgzTp08XcQhEVi0vD9ib355TBuhZT5eTcJXf5BQCRCZCaLG0aNEi5OfnY+DAgfD396+4rF+/vmKfjIwMpKWlVVxv1qwZYmNjkZiYiM6dO+PDDz/E/Pnzq502gIgMKy4OKIccQz0OiI5iUWxtyhHqnsKlT4hMhNAB3rrMWrBixYoq2wYMGIDU1FQDJCKi2oiLA9o5XUKgw3XRUSzOEI+DeOlsFHLVzvBQFIqOQ2TVOHqQiOpEkgClEghzTxYdxSJFeBxCOeSIzwsWHYXI6rFYIqI6OXsWuHIFCPPgwrmGEOhwHe2dLnLcEpEJYLFERHUSH6+d9odTBhjOEI+D2JbbHaazzgKRdWKxRER1olQCvXoBDW1LREexWEM8DiJD5YVjRc1FRyGyaiyWiKjWysqAhAQgNFR0EsvW1/UYnGxu8VQckWAsloio1g4eBAoKgLAw0Uksm4NcjRC3I9jOKQSIhGKxRES1plQCrq5At26ik1i+IR4HsTv/QdwscxAdhchqsVgiolqLjwcGDQJsTWYpbss1xOMg1JICCTe6iI5CZLVYLBFRrRQWAvv38xScsTzgdBUtHNI5bolIIBZLRFQriYnaAd4sloxniMdB/J7bg1MIEAnCYomIakWpBIKCgBYtRCexHkM8DuJiSQDOnROdhMg6sVgiolpRKrW9SjKZ6CTWY6DbEdjKyhAfLzoJkXVisUREOvv7b+D0aZ6CM7aGtiXo6XKSxRKRICyWiEhn8fHaHqXBg0UnsT6h7ilISAA0GtFJiKwPiyUi0plSCXTtCnh6ik5ifULdU3HjBpCaKjoJkfXhLClEdG8hIRX/WS7JEL93A17w/x0I+V5gKOvUw/kUGjbU9u5154TeREbFniUi0smxoubIUnsg1J1dGyIobDQYOBAct0QkAIslItKJMjcYDjal6ONyTHQUqxUaCuzZAxQXi05CZF1YLBGRTuLzgtHf9Sgc5GrRUaxWaCigUgF//CE6CZF14ZglIrqvEo0Cu/I74oOmP4iOYtXaRYbAz24DdkzcjrAWS6vfKSHBuKGIrAB7lojovvYWdMCtcgeEeaSIjmLVZDJgsFsq4vOCRUchsioslojovpR5wfBW5OHBBhdER7F6oe4pSL3ZEjlqF9FRiKwGiyUiui9lbjeEuqfCRsaVXEUb7J4KCTZIyOssOgqR1WCxREQ1ylG7IPVmS4S5J4uOQgACHa6jtWMaT8URGRGLJSKq0c68LpBgw2LJhIS6p7BYIjIiFktEVCNlXje0cbqMxg7ZoqPQPwa7p+Kvkka4dMtXdBQiq8BiiYjuSZK0g7vD3PkrOFMy0O0IbKDBjhvsXSIyBhZLRHRPf90KwKUSf4SyWDIp7oqb6OZ8FvF5XUVHIbIKLJaI6J6Ued0ghwYD3Y6IjkJ3CXVPwY68riiXZKKjEFk8FktEdE/xecHo6XISLrZcjMzUhLqn4LraHceKmouOQmTxWCwRUbU0GmDnjS4I8+Cv4ExRL5cTcLQp4ak4IiNgsURE1UpOBm6UOXNwt4lykKvR1/UYpxAgMgIWS0RUrfh4wFlehB7Op0RHoXsIdU/FrhsdoSrnmuhEhsRiiYiqtWOH9ifqtjbloqPQPYS6p6C43BH7C9qJjkJk0fh1hMhahYRU3WZnB7zyCoqHjsUfiRvwWYvDxs9FOuvc8Dw8bPMRnxeM/m5HRcchsljsWSKiKvbeaA+VZMf5lUycjUzCYPfDHORNZGAsloioih05XeBnl4N2TpdER6H7CHVPwcGCtsgvayA6CpHFYrFERFUk5HbGYLdUyDjfockLdU+BBnIk3egkOgqRxWKxRESVFBYqcLjgAQx2TxUdhXTQ3DEDTR0yOIUAkQGxWCKiSo4d84IEGxZLZuT20idEZBh1KpZWrFiB4mIuf0BkiY4e9cYDTn+jiUOW6Ciko1D3VJwsboqrpZ6ioxBZpDoVS9HR0fDz88PEiROxd+9efWciIoGOHvXGII8jomNQLQxy0/YCsneJyDDqVCz9/fffWL16NfLy8hASEoI2bdrgf//7HzIzM/Wdj4iMKO2WN65ebYhBnpxfyZx42+Wjc8NzHLdEZCB1KpbkcjlGjRqFTZs24cqVK/jPf/6DH3/8EU2aNMGoUaPw66+/orycs/4SmZuE3C6QySQM9PhTdBSqpVD3FMTndYUkiU5CZHnqPcDbx8cHffr0Qa9evWBjY4Njx47hueeeQ4sWLZCYmKiHiERkLDtzO6NZs3x4KApFR6FaGuyWiqsqb5ziUn5EelfnYunatWuYO3cu2rdvj4EDB6KgoABbt27FxYsXcfXqVYwZMwYTJkzQZ1YiMiBJAhJyOqNTp+uio1Ad9HM7BoVMjR07RCchsjx1KpZGjhyJwMBArFixApMmTUJ6ejrWrl2L0NBQAICjoyPeeOMNXLlyRa9hichwThY3RabKEx07slgyRw3kJejtcoLFEpEB1GkhXR8fHyQlJaFXr1733Mff3x8XL16sczAiMq4deV1hJ1OhXbtcYJ/oNFQXoe4p+DyhM8rKAFsuk06kN3XqWRowYAC6dq36E1WVSoWVK1cCAGQyGYKCguqXjoiMJj6vK3q5nYK9vUZ0FKqjUPcUFBQAycmikxBZljoVS88//zzy8/OrbC8sLMTzzz9f71BEZFxl5TZIutEJIZwywKx1cz4DFxcgPl50EiLLUqdiSZIkyKpZYfPvv/+Gq6trvUMRkXElF7ZGgaYhBnuwWDJntjblGDgQHLdEpGe1OqvdpUsXyGQyyGQyDB48GLZ3nBTXaDS4ePEihgwZoveQRGRY8XnBcJHfRLDLWcRhkOg4VA+hocD06UBREdCggeg0RJahVsXS6NGjAQBHjhxBREQEGjZsWHGbnZ0dmjZtirFjx+o1IBEZ3o4bXTHQ7U/Y2nAyWXMXGgqoVMCePUBEhOg0RJahVsXSrFmzAABNmzbF448/DgcHB4OEIiLjKdbYY29+e3ze4jvRUUgP2rQB/P2145ZYLBHpR51+XMrJJoksx578B6GS7DDYPVV0FNIDmUzbu8RxS0T6o/MAbw8PD2RnZwMA3N3d4eHhcc+Lrnbt2oWRI0ciICAAMpkMmzdvrnH/xMTEijFTd15Onz6t83MSUWU78rrCzy4H7ZwuiY5CehIaChw+DPzzkU1E9aRzz9JXX30FZ2fniv+u7tdwtVVUVIROnTrh+eefr9VYpzNnzsDFxaXiure3d72zEFmr+LxgDHZLhR7e0mQiBg/W/rtzJzBunNgsRJZA52LpzlNvzz33nF6efOjQoRg6dGit7+fj4wM3Nze9ZCCyZjlqFxy++QCmNtokOgrpUaNGQNu22nFLLJaI6k/nYqmgoEDnB72z18cQunTpgpKSErRr1w4zZ85ESEjIPfctLS1FaWlpxfXbx6FWq6FWqw2a01huH4elHI+hsJ3uYmcHZW43SLBBf9/jUNvZQa1QAEDFv1Q9k26nf17fISE22LbNBmp1mcAofM/pgu2kG5HtI5MkSdJlRxsbm/ueers9WaVGU/vlEmQyGX755ZeK6Qmqc+bMGezatQvBwcEoLS3FqlWrsHjxYiQmJqJ///7V3uf999/H7Nmzq2xfs2YNnJycap2TyJIsXtwRR496Y+FCjga2NAcO+GHOnIeweLESfn7FouMQ1VtxcTGefPJJ5OfnG7xT5m46F0tJSUk6P+iAAQNqH0SHYqk6I0eOhEwmw5YtW6q9vbqepcDAQGRnZxu9sQ1FrVZDqVQiLCwMClP8pmsi2E53GTEC7fYsx2CPVHzTbgEAbU+JctIkhC1dCgW/5d6TSbfT1q0AgPx8wNfXFt9+q8HEiTp9zOsd33O6YTvpJicnB/7+/kKKJZ1Pw9WlADKGnj17YvXq1fe83d7eHvb29lW2KxQKi3tRWuIxGQLbSSutwA3nixvh02ZLoFCpKt2mUKurbKOqTLKd/nlte3kBPXoACQm2mDxZdCS+53TBdqqZyLbRuVg6evQoOnToABsbGxw9erTGfTt27FjvYLo6fPgw/P39jfZ8RJZiR15XyFCOEDeuB2epBg8GFi8GyssBmzqtBEpEQC2Kpc6dOyMzMxM+Pj7o3LkzZDIZqjuDV5sxSzdv3sT58+crrl+8eBFHjhyBh4cHmjRpgujoaKSnp2PlypUAgHnz5qFp06Zo3749VCoVVq9ejY0bN2Ljxo26HgYR/SM+LxhdG56Dh6JQdBQykNBQ4OOPgaNHgc6dRachMl86F0sXL16smM/o4sWLenny5OTkSr9ki4qKAqCdpmDFihXIyMhAWlpaxe0qlQrTp09Heno6HB0d0b59e8TExGDYsGF6yUNkLSRJ27M0wW+76Cikb3d8pvYqV8DRZgvix/6Azk1++nefhAQBwYjMl87FUlBQULX/XR8DBw6stnfqthUrVlS6PmPGDMyYMUMvz01kzU6cAK6pPRDqniI6ChmQvY0a/V2PIj4vGNPvLJaIqFbqtDYcoP0Z/zfffINTp05BJpOhTZs2mDp1Klq3bq3PfERkADt2AHYyFfq4HhcdhQws1D0F7116HqXlCtjbmNgv94jMRJ2G/G3YsAEdOnRASkoKOnXqhI4dOyI1NRUdOnTAzz//rO+MRKRncXFAX9fjcJKX3n9nMmuD3VNxq9wB+wvaiY5CZLbq1LM0Y8YMREdH44MPPqi0fdasWXjrrbfw2GOP6SUcEelfaSmQmAi853dIdBQygk4N/4KnbT7i87pigNufouMQmaU69SxlZmbi2WefrbL96aefRmZmZr1DEZHh7NsHFBcDYRyvZBVsZBIGu6ciPi9YdBQis1WnYmngwIHYvXt3le179uxBv3796h2KiAxHqdROWNi54fn770wWIdQ9BQcL2iC/rIHoKERmSefTcHcuJzJq1Ci89dZbSElJQc+ePQEA+/fvx88//1ztOmxEZDri4rTz79hkilkCg4xvsHsqyiFH0o1OGOW1V3QcIrOjc7FU3ZptCxcuxMKFCytti4yMxGTRc+sTUbVycoCUFOCVVwCsFJ2GjKW5YwaaOVxFfF4wiyWiOtD5NFx5eblOF11n7yYi49uxQzshZViY6CRkbKHuqYjP6yo6BpFZ4mpBRFZEqQTatgUaNxadhIwt1D0Fp4qbIr3US3QUIrNT50kpi4qKkJSUhLS0NKjuWnX71VdfrXcwItIvSdKOV6rmjDpZgUFuqQCAnXld8IzgLETmpk7F0uHDhzFs2DAUFxejqKgIHh4eyM7OhpOTE3x8fFgsEZmgc+eAtDQgPFx0EhLBy64AnRueQ3xeMIslolqq02m4119/HSNHjkRubi4cHR2xf/9+XL58GcHBwZg7d66+MxKRHsTFAQoFMGCA6CQkSqh7CuLzuqKGJTmJqBp1KpaOHDmCN954A3K5HHK5HKWlpQgMDMRnn32G//73v/rOSER6oFQCvXsDDRuKTkKihLqn4qrKG6dOiU5CZF7qVCwpFArIZDIAgK+vL9LS0gAArq6uFf9NRKZDrQYSEvgrOGvXz/Uo7GUqKJWikxCZlzoVS126dEFycjIAICQkBO+99x5+/PFHTJs2DQ8++KBeAxJR/R04ABQWcryStXOSl6Kf21Fs2yY6CZF5qVOx9Mknn8Df3x8A8OGHH8LT0xMvv/wysrKysGTJEr0GJKL6i4sD3N2Brpxmx+pFuB9CUhJQUiI6CZH5qNOv4bp161bx397e3oiNjdVbICLSP6USGDwYkMtFJyHRIjwO4c0LL2P3bp6WJdJVvSalzMrKwu7du7Fnzx5cv35dX5mISI9u3AAOHuQpONLq0OAiAgKA7dtFJyEyH3UqlgoKCvDMM8+gUaNGGDBgAPr374+AgAA8/fTTyM/P13dGIqqHnTuB8nL2IpCWTKYtnFksEemuTsXSiy++iAMHDmDr1q24ceMG8vPzsXXrViQnJ2PSpEn6zkhE9aBUAi1bAk2bik5CpiIiAjh+HEhPF52EyDzUacxSTEwMtm/fjr59+1Zsi4iIwNKlSzFkyBC9hSOi+ouLA/i2pDuFhWl7mOLigOefF52GyPTVqWfJ09MTrq6uVba7urrC3d293qGISD/++gu4cIHjlagyT0+gWzeeiiPSVZ2KpZkzZyIqKgoZGRkV2zIzM/Hmm2/i3Xff1Vs4IqofpVL7C7iBA0UnIVMTEaF9fWg0opMQmT6dT8N16dKlYtZuADh37hyCgoLQpEkTAEBaWhrs7e1x/fp1vPTSS/pPSkS6CwkBACiPz0bPBu5wHc3FramyiAjgo4+AlBSgRw/RaYhMm87F0ujRow0Yg4j0razcBjvyuuL1wJ9FRyET9NBDgIuL9lQciyWimulcLM2aNcuQOYhIz5ILWyNf0xDh7smio5AJUii0E5Vu3w5w9ARRzer0a7jbUlJScOrUKchkMrRr1w5dunTRVy4iqqe4vO5wld9Ed+fToqOQiYqIACIjtROXurmJTkNkuupULGVlZeGJJ55AYmIi3NzcIEkS8vPzERISgnXr1sHb21vfOYmolpR5wRjkngpbm3LRUchERURoB3jv2AGMHSs6DZHpqtOv4aZOnYqCggKcOHECubm5yMvLw/Hjx1FQUIBXX+VAUiLRCsqcsC+/PcLcU0RHIRPWtCnQujWnECC6nzr1LG3btg3x8fFo27ZtxbZ27drh22+/RTgndCESLvFGZ2ggR7gHxytRzSIigM2bAUnSTlRJRFXVqWepvLwcCoWiynaFQoHycnb5E4kWl9sNzRyuooXjVdFRyMRFRABpacCZM6KTEJmuOhVLgwYNwmuvvYarV//9IE5PT8frr7+OwYMH6y0cEdWNMi+Yv4IjnQwYANjZ8VQcUU3qVCwtWLAAhYWFaNq0KVq0aIEHHngAzZo1Q2FhIb755ht9ZySiWrh8GTh7qwnCPDheie6vQQOgXz8WS0Q1qdOYpcDAQKSmpkKpVOL06dOQJAnt2rVDaGiovvMRUS3FxQE20GCQW6roKGQmIiKAWbOAkhLAwUF0GiLTU+tiqaysDA4ODjhy5AjCwsIQFhZmiFxEVEexsUBPl1NwV9wUHYXMREQEMGMGsGcPwO+8RFXV+jScra0tgoKCoOHqi0Qmp7RUuzjqcM/9oqOQGXnwQcDfn6fiiO6lTqfhZs6ciejoaKxevRoeHh76zkREdbRrF1BUxGKJ7uOfhZZvkwEIL38L2xe1xOfJL2o3JiQYPxeRiapTsTR//nycP38eAQEBCAoKQoMGDSrdnprKsRJEIsTEAI0bAx0b/CU6CpmZCI9D+L9rQ5Be6oVG9tmi4xCZlDoVS6NHj4ZMJoMkSfrOQ0T1EBMDDBsGyM6KTkLmJsw9GTKUIy63G5733yY6DpFJqVWxVFxcjDfffBObN2+GWq3G4MGD8c0338DLy8tQ+YhIR2fPAufPA198AYDFEtWSl10Bgp3PYntudxZLRHep1QDvWbNmYcWKFRg+fDjGjx+P+Ph4vPzyy4bKRkS1EBOjnVyQ88JSXUW4H4IyLxgaqU5T8BFZrFr1LG3atAnLli3DE088AQB46qmn0KdPH2g0GsjlcoMEJCLdxMYCAwdqJxkkqosIj0P4OO0ZpBS2Qg/RYYhMSK2+Ply5cgX9+vWruN6jRw/Y2tpWWvaEiIyvsBBISgKGDxedhMxZT5eTcJYXYXtud9FRiExKrYoljUYDOzu7SttsbW1RVlam11BEVDvx8YBazWKJ6kdho8Fg91QWS0R3qdVpOEmS8Nxzz8He3r5iW0lJCSZPnlxp+oBNmzbpLyER3VdMDNC6NdCihegkZO4i3A9hyrnXkJ8PuLqKTkNkGmpVLE2YMKHKtqefflpvYYio9iRJO15p/HjRScgSDPE4CA3kiIsDHntMdBoi01CrYumHH34wVA4iqqPDh4GMDJ6CI/1o6ngNHRpcwG+/NWexRPSPOk1KSUSmIyYGcHYG+vYVnYQsxSjPvVi81hNll8fA1qb83jtySRSyEpxMg8jMxcQA4eHaOZaI9GGU117klrliX0F70VGITAKLJSIzdv06cPAgT8GRfnV3Pg1fRS625PQWHYXIJPA0HJG5uWPF+N8zwyFJ0Ri6dAywMk9gKLIkNjIJI732Ykt2b3ze4jvRcYiEY88SkRmLyemJbs6n4WfPQon0a6TnPpy91QRnigNFRyESjsUSkZkqK7fB9tzuGO6xX3QUskCh7ilwsCnFlmyeiiNisURkpvYWdEC+piGGeR4QHYUskJO8FGHuyfgtp5foKETCsVgiMlMxOT3ho8hFN+czoqOQhRrluRd/5HdAtspFdBQioYQWS7t27cLIkSMREBAAmUyGzZs33/c+SUlJCA4OhoODA5o3b47FixcbPiiRCYrJ6YmhHgdhI5NERyELNcJzH8ohR2xuT9FRiIQSWiwVFRWhU6dOWLBggU77X7x4EcOGDUO/fv1w+PBh/Pe//8Wrr76KjRs3GjgpkWm5XOKLE8XNMNyT45XIcPzs8/CQ80mOWyKrJ3TqgKFDh2Lo0KE677948WI0adIE8+bNAwC0bdsWycnJmDt3LsaOHWuglESmJyanJ2xlZQj3SBYdhSzcKK+9+OTyUyjRKOAgV4uOQySEWc2ztG/fPoSHh1faFhERgWXLlkGtVkOhUFS5T2lpKUpLSyuuFxQUAADUajXUast4498+Dks5HkOxmHays8PWvF7o43YCTk5qqKG/qbvV/7yH1NW8l+hf1tROQ/wO4Z2LLyK+qDsivO4qzu/zXrKY95yBsZ10I7J9zKpYyszMhK+vb6Vtvr6+KCsrQ3Z2Nvz9/avcZ86cOZg9e3aV7XFxcXBycjJYVhGUSqXoCGbB3NupdOJU7EjshiefPIXY0a8Y5DmUkyYZ5HEtjTW0kyQBPi8VYaHvi9C81KPyjbGxOj2Gub/njIXtVLPi4mJhz21WxRIAyGSyStclSap2+23R0dGIioqquF5QUIDAwECEh4fDxcUyfuGhVquhVCoRFhZWbe8aaVlKO8U+9AFUqhF4/ew8tF14Ra+PrVYooJw0CWFLl0LBb7n3ZG3tNM6hHJt39sXQsoWo9FG7dWuN97OU95yhsZ10k5OTI+y5zapY8vPzQ2ZmZqVtWVlZsLW1haenZ7X3sbe3h729fZXtCoXC4l6UlnhMhmDu7bT9Wjc0c7iKBxV/QaYyzHMo1GooVAZ6cAtiLe002n0PFqQ9ghO5TdDF+fy/N+j4PjL395yxsJ1qJrJtzGqepV69elXppoyLi0O3bt34AiOrIElATM5DGO65H/foTCXSu/6uf8JVfpML65LVElos3bx5E0eOHMGRI0cAaKcGOHLkCNLS0gBoT6E9++yzFftPnjwZly9fRlRUFE6dOoXly5dj2bJlmD59uoj4REZ34gSQVurHJU7IqBQ2Ggz1PIAt2X1ERyESQmixlJycjC5duqBLly4AgKioKHTp0gXvvfceACAjI6OicAKAZs2aITY2FomJiejcuTM+/PBDzJ8/n9MGkNX47TfAyeYWBrj9KToKWZlRnnuRerMV/i7xEh2FyOiEjlkaOHBgxQDt6qxYsaLKtgEDBiA1NdWAqYhM18aNwDDPA3CUW/44GTItQzwOQg4Ntub0wuRGv4mOQ2RUZjVmiciaXbwIpKQAj3rvEh2FrJC74ib6u/3JcUtklVgsEZmJjRsBBwdgGMcrkSCjPPdiR15X3CxzEB2FyKhYLBGZiQ0bgCFDAGfbW6KjkJUa6bUPKskOcXndRUchMioWS0RmIC0NOHAAePRR0UnImrVwvIr2The5sC5ZHRZLRGZg0ybAzg4YOVJ0ErJ2o7z2Iia3JzQS/3yQ9eCrncgMbNgAREQAFrJCD5mxUZ5/IFvthv0F7URHITIaFktEJi49HfjjD56CI9PQw+U0fBS5PBVHVsWs1oYjsnghIVU2bfr7EShkL2PkkjHADzcFhCL6l41MwgjP/diS0xv/Ex2GyEjYs0Rk4jZcH4BQ9xS4K1gokWkY5bUXp4uDcPas6CRExsFiiciEZZa6Y3f+g5yIkkxKmHsyGtjcwoYNopMQGQeLJSIT9kt2P9hAwsNef4iOQlTBSV6KUV57sXat6CRExsFiiciEbbg+AIPcU+GpKBAdhaiS8T47cPw4cPy46CREhsdiichEXVe5IvFGJzzmnSQ6ClEV4R7JcHMD1q0TnYTI8FgsEZmozdl9AQCjvfYITkJUlb2NGmPHAmvXApIkOg2RYbFYIjJRG64PwEC3P+Ftly86ClG1nngCuHABSE4WnYTIsFgsEZmgHLULduR1xaM8BUcmLCQE8PUFB3qTxWOxRGSCfs3ug3LI8IjXbtFRiO5JLgfGjQPWrwc0GtFpiAyHxRKRCdpwvT/6uR6Dn32e6ChENXriCeDqVWAPh9aRBWOxRGRi8tQNEZ8XzFNwZBZ69QKCgngqjiwbiyUiE/NbTm+oJQXGePMUHJk+mQx4/HFgwwZArRadhsgwWCwRmZgN1/ujj8sxNLLPFh2FSCfjxwM5OUB8vOgkRIbBYonIhBSUOWF7bneegiOz0qkT0KYNT8WR5WKxRGRCtub0gkqy4yk4MisymXag9+bNwK1botMQ6R+LJSITsuF6fzzkfBJNHLJERyGqlfHjgcJCIDZWdBIi/WOxRGQibt4Efs99iKfgyCy1agV07cpTcWSZWCwRmYiYGKCk3B5jvXeJjkJUJ088oX0dFxSITkKkX7aiAxCR1sqVwEPOJ9HMMVN0FCLdhIRUuvp4iQ9mlKzHrw99gmf8lICdHfDKK4LCEekPe5aITEB6OrBtGzDRnwM+yHw1cchCH5djWJs1SHQUIr1isURkAlauBOztgcd9EkRHIaqX8b47oczrhmyVi+goRHrDYolIMEkCli8HHnsMcLEtFh2HqF4e805EuSTDxuwBoqMQ6Q2LJSLBdu8Gzp8HXnhBdBKi+vOxu4HB7qlYlxVy/52JzASLJSLBli8HWrQA+vcXnYRIP8b77ETSjU5IL/EUHYVIL1gsEQlUUAD8/DPw/PPaWZCJLMEjXruhkJVhwzV+AyDLwGKJSKCfftIuDzFhgugkRPrjpijCUI+D+CljoOgoRHrBYolIoOXLgYgIoHFj0UmI9Gu8704cKmiDjAwn0VGI6o3FEpEgp04B+/YBEyeKTkKkfyM898HFtgg7dgSJjkJUbyyWiAT54QfA0xMYOVJ0EiL9ayAvwdP+8VAqm0ClEp2GqH5YLBEJoFZrJ6J8+mntZJRElmhSYAzy8x2weTN/vUDmjcUSkQC//w5cu6b9FRyRpWrf8DLatcvG0qX8U0Pmja9gIgGWLweCg4FOnUQnITKsIUMuISnJBqdOiU5CVHe2ogMQWZvMTGDrVmD+fNFJiAyvV68MeCtuYHHYDnzdckH1OyVwTUQybSyWiIwlRLv8w6q0x2Fb/gLGrxkL/HxTcCgiw1IoyjGh0XYsvTIcc5ovhZO8VHQkolrjaTgiI5IkYHnmUIzx3g13BQslsg6TGsegQOOEdVmDREchqhMWS0RGtL+gHU4XB2Gif6zoKERG08zpGoZ4HMSiq6NERyGqExZLREa0PHMoguwzEeJ2WHQUIqOaHPAbkgvbILmglegoRLXGYonISIo0DliXNQjP+/8OG5kkOg6RUQ333I9A+2tYzN4lMkMsloiMZMP1ASjSOOA5v+2ioxAZnVxWjv/4b8WarMG4oW4gOg5RrbBYIjKS5RlDMdg9FUEO10RHIRJion8s1JItVl6LEB2FqFZYLBEZwblzwK78TnjB73fRUYiE8bfPxWivPVh8dSQknokmM8JiicgIliwB3GwLMdprj+goREK9HLAFp4qbYlc+p68n88FiicjAbtwAvvtO+0fCUc7l18m6hbgdRmvHNCxK50BvMh8slogMbPFiQKUCXm20UXQUIuFkMmBywBZsyu6Hayp30XGIdMJiiciASkqAr78GJkwA/OzzRMchMgnP+sVBLivH8oyhoqMQ6YTFEpEBrVoFXLsGvPGG6CREpsNDUYgnfHbiu6sjoZH4Z4hMH1+lRAai0QBz5wJjxgCtOGkxUSUvB2zB5VI/bMvtIToK0X2xWCIykF9/Bc6eBWbMEJ2EyPR0dz6NLg3PYvHVkaKjEN2X8GJp4cKFaNasGRwcHBAcHIzdu3ffc9/ExETIZLIql9OnTxsxMVE1QkIqXaSBIfjfc6cw0O0werz1z3YiqiCTaXuXYnJ64tIl0WmIaia0WFq/fj2mTZuGd955B4cPH0a/fv0wdOhQpKWl1Xi/M2fOICMjo+LSsmVLIyUm0s2u/E44WNgWMwLXiY5CZLKe9N0Bd9tCzJ0rOglRzWxFPvmXX36JiRMn4sUXXwQAzJs3D9u3b8eiRYswZ86ce97Px8cHbm5uOj1HaWkpSktLK64XFBQAANRqNdRqdd3Dm5Dbx2Epx2MoBm0nO7tKVz/9+0l0aHgBg/2OQC2zu8edTI9aoaj0L1WP7aSb+7WTHcoxrekmfLT0ObzxRhkaNzZmOtPBz3DdiGwfmSSJmXRepVLByckJP//8Mx555JGK7a+99hqOHDmCpKSkKvdJTExESEgImjZtipKSErRr1w4zZ85ESA2nON5//33Mnj27yvY1a9bAyclJPwdDdIdLl1wwbVoIpk1LwcCBf4uOQ2TSiott8dJLYejX72/85z/HRMchE1ZcXIwnn3wS+fn5cHFxMepzC+tZys7Ohkajga+vb6Xtvr6+yMzMrPY+/v7+WLJkCYKDg1FaWopVq1Zh8ODBSExMRP/+/au9T3R0NKKioiquFxQUIDAwEOHh4UZvbENRq9VQKpUICwuDgt9078mg7TRiRMV/PnfsTTRxuIaPj78LxUmNfp/HwNQKBZSTJiFs6VIo+C33nthOutG1nc6/OQQffdQMCxYEWmXvEj/DdZOTkyPsuYWehgMAmUxW6bokSVW23da6dWu0bt264nqvXr1w5coVzJ07957Fkr29Pezt7atsVygUFveitMRjMgSDtJNKu4zJ5RJfrM8IwRcPLIJT2S39PocRKdRqKFRcmuV+2E66uV87vfaaHPPmAV98ocCCBcbLZWr4GV4zkW0jbIC3l5cX5HJ5lV6krKysKr1NNenZsyfOnTun73hEdfLVlUfhansTL/rHiI5CZDZcXIDp04GlS4G/eeaaTJCwYsnOzg7BwcFQKpWVtiuVSvTu3Vvnxzl8+DD8/f31HY+o1nLULliaMRyRjX5FA3mJ6DhEZmXKFKBhQ+DTT0UnIapK6Gm4qKgoPPPMM+jWrRt69eqFJUuWIC0tDZMnTwagHW+Unp6OlStXAtD+Wq5p06Zo3749VCoVVq9ejY0bN2LjRi5QSuItTH8Y5bDB1EabREchMjvOztrepfffB956CwgMFJ2I6F9Ci6XHH38cOTk5+OCDD5CRkYEOHTogNjYWQUFBAICMjIxKcy6pVCpMnz4d6enpcHR0RPv27RETE4Nhw4aJOgQiAMAtjR3mp4/BC36/w9suX3QcIrM0ZYp2iaBPPwW+/VZ0GqJ/CR/g/corr+CVV16p9rYVK1ZUuj5jxgzM4NoRZIJ+yByKXLUz3gj8SXQUIrN1Z+/S22+zd4lMh/DlTojMXVkZ8MWVx/CYdxKaO2aIjkNk1jh2iUwRiyWietqwAbhQ0ggzmnBpE6L6ut279P33wJUrotMQabFYIqqHkhIgOhoY7rEPXZ05hQWRPkyZoi2a2LtEpoLFElE9fPmldl6YLx5YJDoKkcVg7xKZGhZLRHWUng588gnw6qtAayd+ohPpU2SktmiqYU11IqNhsURUR9HRgJMT8O67opMQWR72LpEpYbFEVAf79wOrVgEffwy4uYlOQ2SZIiO1S6Gwd4lEY7FEVEvl5cBrrwGdOwMvvCA6DZHlurN36Y75iYmMjsUSUS2tXg0cPAjMnw/I5aLTEFm2yEjAwwN4/XXRSciasVgiqoXCQu3Mwo8/DvTrJzoNkeVzdga+/hrYtAnYvFl0GrJWLJaIamHOHCAvD/jsM9FJiKzHuHHA8OHa+ZcKCkSnIWskfG04IpMXEgIAuHDLH18cXIHoJmvQZML/CQ5FZD1kMu3Cuu3bA++8A3zzjehEZG3Ys0Sko+l/TYaP3Q0ua0IkQFAQ8NFH2qJp/37RacjasGeJSAc78rril+z+WNv2AzjJS0XHIbIs//Te1ighAVOnAj/+CEyaBKSmAgqF4aMRAexZIrqvsnIbTDsfiT4ux/C4T4LoOERWSy4HliwBTp0CPv9cdBqyJiyWiO7ju4xROFHUFF+3XACZTHQaIuvWpQsQFQV88AFwjmtXk5HwNBxRDXJygHcvPo8X/H5HsPNZ0XGIrNcdp+pmaRywQbYML/XIxI5Ob/z7JSaBPb9kGOxZIqpBVBRQJsnxcfNloqMQ0T8ayEuwuNVXSLjRFf+XGSE6DlkBFktE97BiBbByJbCg5Xz42uWJjkNEdwj3SMbTvnF446+XkaVyEx2HLByLJaJqnDypXWbh+eeBZ/3iRMchomp82WIhACDq/CuCk5ClY7FEdJfiYu2MwU2bcvI7IlPmbZePLx9YhB+zwrA9t7voOGTBWCwR3WXqVODCBeCnn4AGDUSnIaKaPOu7HYPcUjH57OsoKhKdhiwVfw1H1u2uyfBWZ4Zi+el3sLz1/9B+yjZBoYhIVzIZ8F2rL9AxeRkmT9aOM+QUH6Rv7Fki+sfpokBMPhuFZ3234zk/FkpE5uIBp6v4oc1nWL2ai1yTYbBYIgJwS2OHcSdnIdAhC9+2nMdvpkRm5nGfBMycCURHA7/9JjoNWRqehiMC8Nr5qTh3qzEOdn0ZDW1LRMchojqYPRs4cQJ48klg3z6gQwfRichSsGeJrN7aa4OwNGMEvnlgPh5seFF0HCKqIxsb7Zil5s2BkSOB69dFJyJLwWKJrNrZ4sb4z9k38JSPEhP9Y0XHIaJ6atgQ2LIFKCoCHn0UUKlEJyJLwGKJrFZJCTDuxCwE2GVjUauvOE6JyEIEBQG//KI9FTdlCiBJohORuWOxRFaprAx47jngdHET/NT+Azjb3hIdiYj0qE8fYMkSYOlSYMEC0WnI3HGAN1mdsjJg4kRgwwZgfbuP0anhX6IjEZE+3DVv2nMAjjV+GdNeHYs237+FMI8UICFBSDQyb+xZIqui0QAvvCDHTz8B69YBY713iY5ERAb0WYvvEO6RjHEnZ+FscWPRcchMsVgiq6HRAPPnd8XPP8uwdq128CcRWTa5rBzr2n0IP7tcjDz2MXJzRScic8RiiayCRgNMnCjH7t2NsHKlBo89JjoRERmLq20RfuvwX+SoXTFgAPD336ITkblhsUQWT6MBnn8eWL9ehqioFDz2GH8aQ2RtHnC6it1dXkVBAdC7N3DypOhEZE5YLJFF045RAn78Efi//9Ogb9+roiMRkSBtG6Rh717AzQ3o2xf44w/RichcsFgii6U99QasXq0tlsaNY48SkbVr1AjYtQvo2BEIDQU2bxadiMwBiyWySBoN8OKLwKpV2mLpiSdEJyIiU+HmBmzbpl0SZexYYPFi0YnI1LFYIouTlwc89ph2jahVq4Dx40UnIiJT4+CgnT4kMhJ4+WXgvfc40zfdGyelJIuyZ492xfHCQmDTJuDhh0UnIiJTZWMDfP219tTc228DV69qe5ls+ZeR7sKXBFkEjQaYMweYNQvo1QtYswZoMiEEmHfHTnZ2wCuvACNGcHVNIgIAyGTAW28BAQHaH4NcuwasXatdkJfoNhZLZJ7uWNYgvdQLT5/6L5JudMLMoNV4T/5/sJ1QLjAcEZmsu5ZEue0ZAD7tumNs7Gw8+KAjvvsOCA83bjQyXRyzRGZta3ZPdDr0Pc4WN8bOTlH4oNkPsLVhoUREtRfhcQh/dn8RzZsDERHaxbY54zcB7FkiU3SPb353Ki1X4K2//oOv0x/FSM8/sLz1Z/CyKzBCOCKyZC0cryI+Hli+HHjjDeD334EFC7TLI8lkotORKOxZIrNz/GZT9EpdgEVXR+HrB77Brx1mslAiIr2RybRztJ06pZ28ctw44JFHtAPAyTqxWCKzcb44AE+f/C86Ji9DscYB+7tG4tXGm/htj4gMwt8f2LgR2LABOHAAaNsWWLIEKOeZfqvDYolM3uUSX7x4ejraHFyJhBud8W3Lr3G0+0R0cT4vOhoRWYGxY7VryT36KPDSS8DgwUBqquhUZEwcs0Qm62qpJz65/BSWZIyAm+1NfN5iMSYHbIGjnD/7JyIDqmbcpDuAZQCe7NQVkw+8juDgxhg4EJg+HRg6VDtnE1kuFktkcq6rXPHZlSewIP0RONioMLvpCkxttAkNbUtERyMiKzfYPRWnuk/AL9n98EXKOIwY0Q5tnC4jqvHPeMY3Dg5y9b87JySIC0p6xWKJTIIkAQcPahe8/eHAGsggYUbgOrze+Ge4KYpExyMiqmBrU47HfJLwqHcS9hZ0wBdXHsNLZ6Mw8+ILiGz0K14J2MwfnVgYFksk1OnT2tm216wB/vpLO6BySqNf8Ebjn/hhQ0QmTSYD+rgeRx/X4zhX3Ajz/n4Un6aNx5y0JzHBdzue2w/06MFTdJaA/wvJ6NLTgS++AIKDtb8umT8fGDAA2LEDuHIFmNP8exZKRGRWWjql49tWX+NKr8cxM2gVfs3pg169tMuoTJoE/PYbUFwsOiXVFXuWyODKyoA//wR27wa2bAESE7XLtI0YAbzzDjBsmHYFcCIic+epKMA7QT/i7SZrsS+/Hbbk9MGWH3vj+++bwNGmBGHuKRjltRcjPPfB1y6P45rMBIsl0ruiImD/fmDPHu1l3z7tNnt7oJ9TMpa12oFHvHbDLacI+AbaCxGRBZHLytHX7Tj6uh3HZy2+w5niQPyW3Qu/5vTBf85EQYIMD7mcQv+3gM6dZSgocIIkiU5N98JiieqloAA4c0Z7SY3+GXvyOyC1sBU0kMPdtgB9XI/jPd9j6Ot6DMHOZ2Fvo77/gxIRWZjWTlfQuskVTG/yE66rXBGb2xNbc3pi7Vrgs89sAYQhOlpCcDDQrZt2mEJwMNC0KZdZMQXCi6WFCxfi888/R0ZGBtq3b4958+ahX79+99w/KSkJUVFROHHiBAICAjBjxgxMnjzZiImtj0oFpKX9WxTdecnM/He/IPt+6Ot6DBP9f0df12No63QZNjJ+VSIiupO3XT4m+G3HBL/tAGYjvbk3vuv/OmzWnMWRwy2waldrfKryBgB42OajdXdXNGuGKpfAQMBW+F9x6yC0mdevX49p06Zh4cKF6NOnD7777jsMHToUJ0+eRJMmTarsf/HiRQwbNgyTJk3C6tWr8ccff+CVV16Bt7c3xo4dK+AIzFN5OVBYCOTnay85OUBGhrbwqe7fnJx/7+vkBLRqpb307w+0bq29tGoFuDw8XtxBERGZKR/7fAQHZ2HYgTVQqLST7maWuiPlZmukFrbEuQdewMWLQFKSdn2626fr5HJtwdS0qfaXxN7egJdX9f96eLCwqg+hTffll19i4sSJePHFFwEA8+bNw/bt27Fo0SLMmTOnyv6LFy9GkyZNMG/ePABA27ZtkZycjLlz55pEsaRSaQczS5K2ILnXv+XlgEZT86WsDFCra76UlgK3bgE3b9rg6NFW+OMPG6jV2m23Lzd37EdBWQPkaxogv0x7KdQ4Qarmh5ANGmjfcH5+2kvbttp//f2BwHlvoLXTFTSyz9b2FmVBe9lt9GYmIrJ4fvZ5GG6/H8M99wNXVmn/WrcESlsokFbig4sl/trLLT9cPOGPq1IIjh4Frl/XfsHVaKo+ZsOGgLOz9t97/bejo3Z86e2Lg0PV6wqFtvC6fbn7uq0t4OmpLdQshbBiSaVSISUlBW+//Xal7eHh4di7d2+199m3bx/Cw8MrbYuIiMCyZcugVquhUCiq3Ke0tBSlpaUV1/Pz8wEAubm5UKv1O35m1Cg59u417mwMCoUER0fAxsYTLi6FcHCQwdFRgr299kXvZFOAxk4ZaG9bjIa2xXCR34KLbTGcbYvgalsMF9siuNsWwtc+Dw1t/20n5P5zOfHP9X9e9HmQATDPE+hqAMXFxcgBoODEJ9ViG+mG7aQbtpNuatVONhp4NMyAR8MMBFe64SPAF4AvUC7JkF/WADkqZ+SoXZCjdkWOygWFZY4oKnfEzVJ7FBU5oeiqPW5qHHFN44C/NI4oLHOEqlyB0nIFSsoVFf9dhqp/W+/n5Zc1+PBD/a44nJubCwCQBIyEF1YsZWdnQ6PRwNfXt9J2X19fZN45EOYOmZmZ1e5fVlaG7Oxs+Pv7V7nPnDlzMHv27CrbmzVrVo/0puN2LxMA3LghNIp5iIsTncD0sY10w3bSDdtJNxbWTosWaS+GkJOTA1dXV8M8+D0IP4Mpu2uYvyRJVbbdb//qtt8WHR2NqKioiuvl5eXIzc2Fp6dnjc9jTgoKChAYGIgrV67AxcVFdByTxXa6P7aRbthOumE76YbtpJv8/Hw0adIEHh4eRn9uYcWSl5cX5HJ5lV6krKysKr1Ht/n5+VW7v62tLTw9Pau9j729Pezt7Sttc3Nzq3twE+bi4sI3mg7YTvfHNtIN20k3bCfdsJ10YyPglK6wk8h2dnYIDg6GUqmstF2pVKJ3797V3qdXr15V9o+Li0O3bt2qHa9EREREVF9CR9xFRUXh+++/x/Lly3Hq1Cm8/vrrSEtLq5g3KTo6Gs8++2zF/pMnT8bly5cRFRWFU6dOYfny5Vi2bBmmT58u6hCIiIjIwgkds/T4448jJycHH3zwATIyMtChQwfExsYiKCgIAJCRkYG0tLSK/Zs1a4bY2Fi8/vrr+PbbbxEQEID58+ebxLQBItnb22PWrFlVTjdSZWyn+2Mb6YbtpBu2k27YTroR2U4yScRv8IiIiIjMBCe+ICIiIqoBiyUiIiKiGrBYIiIiIqoBiyUiIiKiGrBYMjGFhYWYNm0agoKC4OjoiN69e+PQoUMVt1+7dg3PPfccAgIC4OTkhCFDhuDcuXM1PubSpUvRr18/uLu7w93dHaGhoTh48KChD8WgDNFOd1q3bh1kMhlGjx5tgPTGY6h2unHjBiIjI+Hv7w8HBwe0bdsWsbGxhjwUgzJUO82bNw+tW7eGo6MjAgMD8frrr6OkpMSQh6I3u3btwsiRIxEQEACZTIbNmzdXul2SJLz//vsICAiAo6MjBg4ciBMnTlTap7S0FFOnToWXlxcaNGiAUaNG4e+//77vcy9cuBDNmjWDg4MDgoODsXu36a7YLaqd5syZg+7du8PZ2Rk+Pj4YPXo0zpw5o+/D0xuRr6fb5syZA5lMhmnTptU6P4slE/Piiy9CqVRi1apVOHbsGMLDwxEaGor09HRIkoTRo0fjwoUL+PXXX3H48GEEBQUhNDQURUVF93zMxMREjB8/HgkJCdi3bx+aNGmC8PBwpKenG/HI9MsQ7XTb5cuXMX36dPTr188IR2JYhmgnlUqFsLAwXLp0CRs2bMCZM2ewdOlSNGrUyIhHpl+GaKcff/wRb7/9NmbNmoVTp05h2bJlWL9+PaKjo414ZHVXVFSETp06YcGCBdXe/tlnn+HLL7/EggULcOjQIfj5+SEsLAyFhYUV+0ybNg2//PIL1q1bhz179uDmzZsYMWIENBrNPZ93/fr1mDZtGt555x0cPnwY/fr1w9ChQytNI2NKRLVTUlISIiMjsX//fiiVSpSVlSE8PFynzzgRRLXTbYcOHcKSJUvQsWPHuh2ARCajuLhYksvl0tatWytt79Spk/TOO+9IZ86ckQBIx48fr7itrKxM8vDwkJYuXarz85SVlUnOzs7S//3f/+ktuzEZsp3KysqkPn36SN9//700YcIE6eGHHzbEIRiFodpp0aJFUvPmzSWVSmWw7MZkqHaKjIyUBg0aVGlbVFSU1LdvX/0egBEAkH755ZeK6+Xl5ZKfn5/06aefVmwrKSmRXF1dpcWLF0uSJEk3btyQFAqFtG7duop90tPTJRsbG2nbtm33fK4ePXpIkydPrrStTZs20ttvv62nozEcY7bT3bKysiQAUlJSUv0PxMCM3U6FhYVSy5YtJaVSKQ0YMEB67bXXap2ZPUsmpKysDBqNBg4ODpW2Ozo6Ys+ePSgtLQWASrfL5XLY2dlhz549Oj9PcXEx1Gq1kMUI9cGQ7fTBBx/A29sbEydO1H9wIzNUO23ZsgW9evVCZGQkfH190aFDB3zyySc6fbszRYZqp759+yIlJaXilPeFCxcQGxuL4cOHG+AojOvixYvIzMxEeHh4xTZ7e3sMGDAAe/fuBQCkpKRArVZX2icgIAAdOnSo2OduKpUKKSkple4DAOHh4fe8jykzVDtVJz8/HwDM8nPd0O0UGRmJ4cOHIzQ0tM4ZWSyZEGdnZ/Tq1Qsffvghrl69Co1Gg9WrV+PAgQPIyMhAmzZtEBQUhOjoaOTl5UGlUuHTTz9FZmYmMjIydH6et99+G40aNarXC0ckQ7XTH3/8gWXLlmHp0qVGPBrDMVQ7XbhwARs2bIBGo0FsbCxmzpyJL774Ah9//LERj05/DNVOTzzxBD788EP07dsXCoUCLVq0QEhICN5++20jHp1h3F7Q/O5Fz319fStuy8zMhJ2dHdzd3e+5z92ys7Oh0WhqfFxzYqh2upskSYiKikLfvn3RoUMHPSQ3LkO207p165Camoo5c+bUKyOLJROzatUqSJKERo0awd7eHvPnz8eTTz4JuVwOhUKBjRs34uzZs/Dw8ICTkxMSExMxdOhQyOVynR7/s88+w9q1a7Fp06Yq36TNib7bqbCwEE8//TSWLl0KLy8vIx+N4Rji9VReXg4fHx8sWbIEwcHBeOKJJ/DOO+9g0aJFRjwy/TJEOyUmJuLjjz/GwoULkZqaik2bNmHr1q348MMPjXhkhiWTySpdlySpyra76bJPXR7XlBmqnW6bMmUKjh49irVr19Y5oynQdztduXIFr732GlavXl3vv3cslkxMixYtkJSUhJs3b+LKlSs4ePAg1Go1mjVrBgAIDg7GkSNHcOPGDWRkZGDbtm3IycmpuL0mc+fOxSeffIK4uLi6D3IzEfpup7/++guXLl3CyJEjYWtrC1tbW6xcuRJbtmyBra0t/vrrL2Ment4Y4vXk7++PVq1aVSoU2rZti8zMTKhUKoMfkyEYop3effddPPPMM3jxxRfx4IMP4pFHHsEnn3yCOXPmoLy83FiHZhB+fn4AUOUbfVZWVkXvgJ+fH1QqFfLy8u65z928vLwgl8trfFxzYqh2utPUqVOxZcsWJCQkoHHjxnpKblyGaqeUlBRkZWUhODi44nM9KSkJ8+fPh62tba2GDrBYMlENGjSAv78/8vLysH37djz88MOVbnd1dYW3tzfOnTuH5OTkKrff7fPPP8eHH36Ibdu2oVu3boaMblT6aqc2bdrg2LFjOHLkSMVl1KhRCAkJwZEjRxAYGGiMwzEYfb6e+vTpg/Pnz1f6g3/27Fn4+/vDzs7OYMdgDPpsp+LiYtjYVP6IlcvlkCQJkpkvydmsWTP4+flBqVRWbFOpVEhKSkLv3r0BaAtMhUJRaZ+MjAwcP368Yp+72dnZITg4uNJ9AECpVN7zPqbMUO0EaHtUpkyZgk2bNmHnzp06fWE2VYZqp8GDB1f5XO/WrRueeuopHDlyROczMgD4azhTs23bNun333+XLly4IMXFxUmdOnWSevToUfHLo59++klKSEiQ/vrrL2nz5s1SUFCQNGbMmEqP8cwzz1T65cj//vc/yc7OTtqwYYOUkZFRcSksLDTqsemTIdrpbub+azhJMkw7paWlSQ0bNpSmTJkinTlzRtq6davk4+MjffTRR0Y9Nn0yRDvNmjVLcnZ2ltauXVvxuC1atJDGjRtn1GOrq8LCQunw4cPS4cOHJQDSl19+KR0+fFi6fPmyJEmS9Omnn0qurq7Spk2bpGPHjknjx4+X/P39pYKCgorHmDx5stS4cWMpPj5eSk1NlQYNGiR16tRJKisrq9hn0KBB0jfffFNxfd26dZJCoZCWLVsmnTx5Upo2bZrUoEED6dKlS8Y7+FoQ1U4vv/yy5OrqKiUmJlb6XC8uLjbewdeCqHa6W11/DcdiycSsX79eat68uWRnZyf5+flJkZGR0o0bNypu//rrr6XGjRtLCoVCatKkiTRz5kyptLS00mMMGDBAmjBhQsX1oKAgCUCVy6xZs4x0VPpniHa6myUUS4Zqp71790oPPfSQZG9vLzVv3lz6+OOPK31gmRtDtJNarZbef/99qUWLFpKDg4MUGBgovfLKK1JeXp6Rjqp+EhISqv3cuH2M5eXl0qxZsyQ/Pz/J3t5e6t+/v3Ts2LFKj3Hr1i1pypQpkoeHh+To6CiNGDFCSktLq7RPUFBQlc+ib7/9VgoKCpLs7Oykrl27mvTP4UW1U3XPCUD64YcfDHzEdSPy9XSnuhZLMkky8/5gIiIiIgPimCUiIiKiGrBYIiIiIqoBiyUiIiKiGrBYIiIiIqoBiyUiIiKiGrBYIiIiIqoBiyUiIiKiGrBYIiIiIqoBiyUiIiKiGrBYIiIiIqoBiyUiIiKiGrBYIiKzd/36dfj5+eGTTz6p2HbgwAHY2dkhLi5OYDIisgRcSJeILEJsbCxGjx6NvXv3ok2bNujSpQuGDx+OefPmiY5GRGaOxRIRWYzIyEjEx8eje/fu+PPPP3Ho0CE4ODiIjkVEZo7FEhFZjFu3bqFDhw64cuUKkpOT0bFjR9GRiMgCcMwSEVmMCxcu4OrVqygvL8fly5dFxyEiC8GeJSKyCCqVCj169EDnzp3Rpk0bfPnllzh27Bh8fX1FRyMiM8diiYgswptvvokNGzbgzz//RMOGDRESEgJnZ2ds3bpVdDQiMnM8DUdEZi8xMRHz5s3DqlWr4OLiAhsbG6xatQp79uzBokWLRMcjIjPHniUiIiKiGrBniYiIiKgGLJaIiIiIasBiiYiIiKgGLJaIiIiIasBiiYiIiKgGLJaIiIiIasBiiYiIiKgGLJaIiIiIasBiiYiIiKgGLJaIiIiIasBiiYiIiKgG/w9wmUOKZB15aQAAAABJRU5ErkJggg==",
"text/plain": [
""
]
},
"metadata": {
"filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_144_0.png"
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_121_0.png"
}
},
"output_type": "display_data"
@@ -2111,7 +1761,7 @@
},
{
"cell_type": "markdown",
- "id": "c1620bb7",
+ "id": "c87eac4e",
"metadata": {
"editable": true
},
@@ -2129,7 +1779,7 @@
},
{
"cell_type": "markdown",
- "id": "fd3bd4b5",
+ "id": "2f0f4313",
"metadata": {
"editable": true
},
@@ -2141,7 +1791,7 @@
},
{
"cell_type": "markdown",
- "id": "4a129d86",
+ "id": "7895ee8b",
"metadata": {
"editable": true
},
@@ -2158,7 +1808,7 @@
},
{
"cell_type": "markdown",
- "id": "ff57b405",
+ "id": "be454b28",
"metadata": {
"editable": true
},
@@ -2170,7 +1820,7 @@
},
{
"cell_type": "markdown",
- "id": "5a13919e",
+ "id": "7475f8f9",
"metadata": {
"editable": true
},
@@ -2180,7 +1830,7 @@
},
{
"cell_type": "markdown",
- "id": "45d4a7b6",
+ "id": "fd472029",
"metadata": {
"editable": true
},
@@ -2192,7 +1842,7 @@
},
{
"cell_type": "markdown",
- "id": "c837442b",
+ "id": "b43c8f67",
"metadata": {
"editable": true
},
@@ -2209,7 +1859,7 @@
},
{
"cell_type": "markdown",
- "id": "4b3c7b8e",
+ "id": "c47cfb6d",
"metadata": {
"editable": true
},
@@ -2221,7 +1871,7 @@
},
{
"cell_type": "markdown",
- "id": "7e6f977d",
+ "id": "593188cd",
"metadata": {
"editable": true
},
@@ -2231,7 +1881,7 @@
},
{
"cell_type": "markdown",
- "id": "adeab0fd",
+ "id": "76de4d81",
"metadata": {
"editable": true
},
@@ -2243,7 +1893,7 @@
},
{
"cell_type": "markdown",
- "id": "63b18db5",
+ "id": "afc38c97",
"metadata": {
"editable": true
},
@@ -2253,7 +1903,7 @@
},
{
"cell_type": "markdown",
- "id": "9c7eb00d",
+ "id": "b0f2971c",
"metadata": {
"editable": true
},
@@ -2265,7 +1915,7 @@
},
{
"cell_type": "markdown",
- "id": "c3f4530b",
+ "id": "5a65e495",
"metadata": {
"editable": true
},
@@ -2275,7 +1925,7 @@
},
{
"cell_type": "markdown",
- "id": "90792236",
+ "id": "31419298",
"metadata": {
"editable": true
},
@@ -2291,7 +1941,7 @@
},
{
"cell_type": "markdown",
- "id": "d73c7cd4",
+ "id": "217ddd53",
"metadata": {
"editable": true
},
@@ -2302,7 +1952,7 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "58124207",
+ "id": "ece05002",
"metadata": {
"collapsed": false,
"editable": true
@@ -2327,7 +1977,7 @@
},
"metadata": {
"filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_160_1.png"
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_137_1.png"
}
},
"output_type": "display_data"
@@ -2392,7 +2042,7 @@
},
{
"cell_type": "markdown",
- "id": "4b4e7bf5",
+ "id": "de6dc4e2",
"metadata": {
"editable": true
},
@@ -2403,7 +2053,7 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "47b138a9",
+ "id": "da218205",
"metadata": {
"collapsed": false,
"editable": true
@@ -2442,13 +2092,7 @@
"Error: 0.05227921801205686\n",
"Bias^2: 0.0481872773043029\n",
"Var: 0.004091940707753939\n",
- "0.05227921801205686 >= 0.0481872773043029 + 0.004091940707753939 = 0.052279218012056844\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
+ "0.05227921801205686 >= 0.0481872773043029 + 0.004091940707753939 = 0.052279218012056844\n",
"Polynomial degree: 6\n",
"Error: 0.037813671417389005\n",
"Bias^2: 0.033657685071527665\n",
@@ -2468,18 +2112,18 @@
"Error: 0.02660572763718093\n",
"Bias^2: 0.010018312644137363\n",
"Var: 0.016587414993043573\n",
- "0.02660572763718093 >= 0.010018312644137363 + 0.016587414993043573 = 0.026605727637180936\n",
- "Polynomial degree: 10\n",
- "Error: 0.021592704588025025\n",
- "Bias^2: 0.010516485576645508\n",
- "Var: 0.011076219011379514\n",
- "0.021592704588025025 >= 0.010516485576645508 + 0.011076219011379514 = 0.021592704588025022\n"
+ "0.02660572763718093 >= 0.010018312644137363 + 0.016587414993043573 = 0.026605727637180936\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
+ "Polynomial degree: 10\n",
+ "Error: 0.021592704588025025\n",
+ "Bias^2: 0.010516485576645508\n",
+ "Var: 0.011076219011379514\n",
+ "0.021592704588025025 >= 0.010516485576645508 + 0.011076219011379514 = 0.021592704588025022\n",
"Polynomial degree: 11\n",
"Error: 0.07160048164233104\n",
"Bias^2: 0.014436800088904942\n",
@@ -2489,13 +2133,7 @@
"Error: 0.11547777218872497\n",
"Bias^2: 0.01628578269596628\n",
"Var: 0.09919198949275869\n",
- "0.11547777218872497 >= 0.01628578269596628 + 0.09919198949275869 = 0.11547777218872497\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
+ "0.11547777218872497 >= 0.01628578269596628 + 0.09919198949275869 = 0.11547777218872497\n",
"Polynomial degree: 13\n",
"Error: 0.22842468702219465\n",
"Bias^2: 0.01975416527185249\n",
@@ -2512,7 +2150,7 @@
},
"metadata": {
"filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_162_4.png"
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_139_2.png"
}
},
"output_type": "display_data"
@@ -2569,7 +2207,7 @@
},
{
"cell_type": "markdown",
- "id": "37b66761",
+ "id": "5ea5df06",
"metadata": {
"editable": true
},
@@ -2607,7 +2245,7 @@
},
{
"cell_type": "markdown",
- "id": "5b5c601e",
+ "id": "40f41d74",
"metadata": {
"editable": true
},
@@ -2618,7 +2256,7 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "9bc32416",
+ "id": "03bb50ae",
"metadata": {
"collapsed": false,
"editable": true
@@ -2660,7 +2298,7 @@
},
"metadata": {
"filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_165_1.png"
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_142_1.png"
}
},
"output_type": "display_data"
@@ -2742,7 +2380,7 @@
},
{
"cell_type": "markdown",
- "id": "eb934dbe",
+ "id": "cd11e9fd",
"metadata": {
"editable": true
},
@@ -2767,7 +2405,7 @@
},
{
"cell_type": "markdown",
- "id": "c65f802b",
+ "id": "6cf68ff2",
"metadata": {
"editable": true
},
@@ -2795,7 +2433,7 @@
},
{
"cell_type": "markdown",
- "id": "9accabd4",
+ "id": "c445bfb2",
"metadata": {
"editable": true
},
@@ -2808,7 +2446,7 @@
{
"cell_type": "code",
"execution_count": 6,
- "id": "d5ed3f5c",
+ "id": "8e26035a",
"metadata": {
"collapsed": false,
"editable": true
@@ -2823,7 +2461,7 @@
},
"metadata": {
"filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_169_0.png"
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_146_0.png"
}
},
"output_type": "display_data"
@@ -2923,7 +2561,7 @@
},
{
"cell_type": "markdown",
- "id": "db1976f4",
+ "id": "86a6b012",
"metadata": {
"editable": true
},
@@ -2934,7 +2572,7 @@
{
"cell_type": "code",
"execution_count": 7,
- "id": "3c35540c",
+ "id": "4a0e1afd",
"metadata": {
"collapsed": false,
"editable": true
@@ -2973,16 +2611,16 @@
"Mean squared error on test data: 1.53738247\n",
"Degree of polynomial: 8\n",
"Mean squared error on training data: 0.04926746\n",
- "Mean squared error on test data: 0.14629156\n",
- "Degree of polynomial: 9\n",
- "Mean squared error on training data: 0.02546675\n",
- "Mean squared error on test data: 0.11202337\n"
+ "Mean squared error on test data: 0.14629156\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
+ "Degree of polynomial: 9\n",
+ "Mean squared error on training data: 0.02546675\n",
+ "Mean squared error on test data: 0.11202337\n",
"Degree of polynomial: 10\n",
"Mean squared error on training data: 0.02424794\n",
"Mean squared error on test data: 0.22467274\n",
@@ -2991,25 +2629,31 @@
"Mean squared error on test data: 1.07641937\n",
"Degree of polynomial: 12\n",
"Mean squared error on training data: 0.00805074\n",
- "Mean squared error on test data: 0.04295757\n",
- "Degree of polynomial: 13\n",
- "Mean squared error on training data: 0.00781918\n",
- "Mean squared error on test data: 0.56965674\n",
- "Degree of polynomial: 14\n",
- "Mean squared error on training data: 0.00465099\n",
- "Mean squared error on test data: 0.28443039\n"
+ "Mean squared error on test data: 0.04295757\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
+ "Degree of polynomial: 13\n",
+ "Mean squared error on training data: 0.00781918\n",
+ "Mean squared error on test data: 0.56965674\n",
+ "Degree of polynomial: 14\n",
+ "Mean squared error on training data: 0.00465099\n",
+ "Mean squared error on test data: 0.28443039\n",
"Degree of polynomial: 15\n",
"Mean squared error on training data: 0.00420072\n",
"Mean squared error on test data: 568.47051432\n",
"Degree of polynomial: 16\n",
"Mean squared error on training data: 0.00325450\n",
- "Mean squared error on test data: 48.97630233\n",
+ "Mean squared error on test data: 48.97630233\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
"Degree of polynomial: 17\n",
"Mean squared error on training data: 0.00242954\n",
"Mean squared error on test data: 2.52780600\n",
@@ -3018,16 +2662,16 @@
"Mean squared error on test data: 429.25695398\n",
"Degree of polynomial: 19\n",
"Mean squared error on training data: 0.00154853\n",
- "Mean squared error on test data: 239.97065359\n"
+ "Mean squared error on test data: 239.97065359\n",
+ "Degree of polynomial: 20\n",
+ "Mean squared error on training data: 0.00140846\n",
+ "Mean squared error on test data: 1350.24493666\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
- "Degree of polynomial: 20\n",
- "Mean squared error on training data: 0.00140846\n",
- "Mean squared error on test data: 1350.24493666\n",
"Degree of polynomial: 21\n",
"Mean squared error on training data: 0.00119688\n",
"Mean squared error on test data: 1840.50530832\n",
@@ -3036,16 +2680,16 @@
"Mean squared error on test data: 1184.60929685\n",
"Degree of polynomial: 23\n",
"Mean squared error on training data: 0.00089193\n",
- "Mean squared error on test data: 3892.17483760\n"
+ "Mean squared error on test data: 3892.17483760\n",
+ "Degree of polynomial: 24\n",
+ "Mean squared error on training data: 0.00083355\n",
+ "Mean squared error on test data: 1332.46736215\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
- "Degree of polynomial: 24\n",
- "Mean squared error on training data: 0.00083355\n",
- "Mean squared error on test data: 1332.46736215\n",
"Degree of polynomial: 25\n",
"Mean squared error on training data: 0.00079904\n",
"Mean squared error on test data: 7577.76690383\n",
@@ -3054,16 +2698,16 @@
"Mean squared error on test data: 1079.36895644\n",
"Degree of polynomial: 27\n",
"Mean squared error on training data: 0.00068091\n",
- "Mean squared error on test data: 3207.25343155\n",
- "Degree of polynomial: 28\n",
- "Mean squared error on training data: 0.00063362\n",
- "Mean squared error on test data: 674.79633065\n"
+ "Mean squared error on test data: 3207.25343155\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
+ "Degree of polynomial: 28\n",
+ "Mean squared error on training data: 0.00063362\n",
+ "Mean squared error on test data: 674.79633065\n",
"Degree of polynomial: 29\n",
"Mean squared error on training data: 0.00063866\n",
"Mean squared error on test data: 3099.60342978\n"
@@ -3073,9 +2717,9 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_85796/626635268.py:73: RuntimeWarning: divide by zero encountered in log10\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87621/626635268.py:73: RuntimeWarning: divide by zero encountered in log10\n",
" plt.plot(polynomial, np.log10(trainingerror), label='Training Error')\n",
- "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_85796/626635268.py:74: RuntimeWarning: divide by zero encountered in log10\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87621/626635268.py:74: RuntimeWarning: divide by zero encountered in log10\n",
" plt.plot(polynomial, np.log10(testerror), label='Test Error')\n"
]
},
@@ -3088,7 +2732,7 @@
},
"metadata": {
"filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_171_8.png"
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_148_9.png"
}
},
"output_type": "display_data"
@@ -3177,7 +2821,7 @@
},
{
"cell_type": "markdown",
- "id": "e60fabdf",
+ "id": "bb1e0f15",
"metadata": {
"editable": true
},
@@ -3187,7 +2831,7 @@
},
{
"cell_type": "markdown",
- "id": "9b6750d6",
+ "id": "83c87d33",
"metadata": {
"editable": true
},
@@ -3200,7 +2844,7 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "5e6b91e1",
+ "id": "74eaac59",
"metadata": {
"collapsed": false,
"editable": true
@@ -3210,7 +2854,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_85796/3817475779.py:63: RuntimeWarning: divide by zero encountered in log10\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87621/3817475779.py:63: RuntimeWarning: divide by zero encountered in log10\n",
" plt.plot(polynomial, np.log10(estimated_mse_sklearn), label='Test Error')\n"
]
},
@@ -3223,7 +2867,7 @@
},
"metadata": {
"filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_174_1.png"
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_151_1.png"
}
},
"output_type": "display_data"
@@ -3301,694 +2945,364 @@
},
{
"cell_type": "markdown",
- "id": "e83d7b10",
+ "id": "aa703354",
"metadata": {
"editable": true
},
"source": [
- "## Notes on scaling with examples\n",
- "\n",
- "The programs here use both ordinrary least squares (OLS) and Ridge\n",
- "regression with one value only for the hyperparameter $\\lambda$. The\n",
- "first example has no scaling and includes the intercept as well and we\n",
- "are trying to fit a second-order polynomial. The second code takes out\n",
- "the intercept and subtracts the mean values of each column of the\n",
- "design matrix and the mean value of the outputs.\n",
- "\n",
- "The third and final code uses **Scikit-Learn** as library in order to\n",
- "calculate the optimal parameters for OLS and Ridge regression. Note\n",
- "that it is highly recommended to not include the intercept in Ridge\n",
- "and Lasso regression, in order to avoid penalizing the optimization by\n",
- "the intercept. The second and third codes do thus not include the\n",
- "intercept. In the second code we do the scaling ourselves while the\n",
- "last code uses the standard scaler option included in **Scikit-Learn**, known as centering (where\n",
- "we subtract the mean values)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "id": "73807393",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[1.79934087 0.47179152 5.01549939]\n",
- "[1.79909592 0.47176716 5.01550546]\n",
- " \n",
- "test MSE of OLS:\n",
- "1.139431112903922\n",
- " \n",
- "test MSE of Ridge\n",
- "1.1395235273363669\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAACDY0lEQVR4nO3dd3gUVRfA4d/MbElCCiQBEkILTYSggEhXQARBaVYEROxIUbFhQSVYQLChINg+EUWKjaZSpYmA1AABQUooCYmUkEJCkt2d+f4Iu5BKAkk25bzPwwM7c3f25mbZPXPLuYphGAZCCCGEECVEdXcFhBBCCFGxSPAhhBBCiBIlwYcQQgghSpQEH0IIIYQoURJ8CCGEEKJESfAhhBBCiBIlwYcQQgghSpQEH0IIIYQoUSZ3VyA7Xdc5ceIEPj4+KIri7uoIIYQQogAMwyA5OZkaNWqgqvn3bZS64OPEiRPUqlXL3dUQQgghxBU4fvw4NWvWzLdMqQs+fHx8gMzK+/r6Fum1bTYby5cvp3v37pjN5iK9dnkjbVVw0lYFJ21VONJeBSdtVXDF1VZJSUnUqlXL9T2en1IXfDiHWnx9fYsl+PDy8sLX11fenJchbVVw0lYFJ21VONJeBSdtVXDF3VYFmTIhE06FEEIIUaIKFXyEh4ejKEqWP0FBQa7zhmEQHh5OjRo18PT0pHPnzuzZs6fIKy2EEEKIsqvQPR9NmzYlNjbW9Wf37t2uc5MmTeLDDz9k6tSpbNmyhaCgILp160ZycnKRVloIIYQQZVehgw+TyURQUJDrT9WqVYHMXo/JkyczZswY7rrrLsLCwpg5cyapqanMnj27yCsuhBBCiLKp0BNODxw4QI0aNbBarbRp04bx48dTr149oqKiiIuLo3v37q6yVquVTp06sWHDBoYOHZrr9dLT00lPT3c9TkpKAjInxNhstsJWL1/O6xX1dcsjaauCk7YqOGmrwpH2Kjhpq4IrrrYqzPUUwzCMghZesmQJqampNGrUiP/++4+3336bffv2sWfPHvbv30+HDh2IiYmhRo0aruc88cQTHD16lGXLluV6zfDwcMaNG5fj+OzZs/Hy8irwDyKEEEII90lNTWXgwIEkJiZedrVqoYKP7FJSUqhfvz6jR4+mbdu2dOjQgRMnThAcHOwq8/jjj3P8+HGWLl2a6zVy6/moVasWp0+fLpaltitWrKBbt26yFOsypK0KTtqq4KStCkfaq+CkrQquuNoqKSmJwMDAAgUfV5Xno1KlSjRr1owDBw7Qr18/AOLi4rIEHydPnqR69ep5XsNqtWK1WnMcN5vNxfYGKs5rlzfSVgUnbVVw0laFI+1VcNJWBVfUbVWYa11Vno/09HT++ecfgoODCQ0NJSgoiBUrVrjOZ2RksHbtWtq3b381LyOEEEKIIuDQDTZHxQOwOSoeh37Fgx9XpVA9Hy+88AK9e/emdu3anDx5krfffpukpCSGDBmCoiiMGjWK8ePH07BhQxo2bMj48ePx8vJi4MCBxVV/IYQQQhTA0shYxi3eS/y580xqDY/M3IK/tydjezehR1jw5S9QhAoVfERHRzNgwABOnz5N1apVadu2LZs2baJOnToAjB49mvPnzzN8+HDOnj1LmzZtWL58eYHyvAshhBCieCyNjGXYrO0YgFW7eDwuMY1hs7Yz/YGWJRqAFCr4mDt3br7nFUUhPDyc8PDwq6mTEEIIIYqIQzcYt3gvuQ2wGIACjFu8l25NgtDUy+/LUhRkbxchhBCiHNscFU9sYprrsW7A1D0qzukeBhCbmOaaC1ISJPgQQgghyrGTyWlZHusGHEi6GHzkVa44SfAhhBBClGPVfDyyPHYYWf/Oq1xxkuBDCCGEKMdah/pf8sjgffNnDNJW4kVaPuWKlwQfQgghRDmmqQoPd6gLwLXKMe7R/uQN07eYcLjKPNyhbolNNoWrzHAqhBBCiNJvbO+mBPt6wIrvAfhDb0kS3gC82rMxT3SqX6L1keBDCCGEqADa16tCVe0vABY4OqCQudKlfYPAEq+LBB9CCCFEBRB8djMBSgJJig8Bda4jLMOXuMR0ArwtJV4XCT6EEEKICiDg0AIAKrW4h7aGxriebTBUDatJy/+JxUAmnAohhBDlXUYK7F0EgNHsXiAzK7k7Ag+Qng8hhBCiXHLuYHsyOY0mp5fR0JYCVepihNwIu5a4tW4SfAghhBDljHMHW2da9W/M39BQg4NBt1NHKbkltXmRYRchhBCiHHHuYOsMPAJJ5CZ1FwCPR9Rn5T//ubN6gAQfQgghRLmR2w62fbQNaIrBDr0BR4xg3l2yz231c5LgQwghhCgnsu9gC9BPWw/AL46OGEBcUsltIJcXCT6EEEKIciL7zrT1lRiuU6OwGRq/OtoCYBi5PbNkSfAhhBBClBPZd6a980Kvxxr9es7iC0CGLhNOhRBCCFFEWof6U9nTDICC7go+Fjg6usoEeLl/oasEH0IIIUQ5oakK797dDIC26j+EKGdIMrxYqbd0lXm7X1N3Vc9Fgg8hhBCiHOkRFsyrPRtzt/YnAL862pJO5v4tr/ZszK3XVndn9YAKFnzsjklk6h6V3TGJ7q6KEEIIUWz6NPHjdm0zAGcb3U29wEoEelvp3byGm2uWyf0DPyXEoRt8ti6KA0kqn6+LYlrtADTV/ZNuhBBCiKIWdGIlkIZRJZQRgwcxHMhw6FhNGjabzd3VK//BR/TZVH7bFcuXfx4m+VwyzZQYlu01aDN+JY/fVI87rgumZhUvd1dTCCGEKDoRswFQrh8AioICbttELjflPvjoOHE1kLnWeZX1DQwUWqd/yulzChOW7GPCkn0cefcON9dSCCGEKCKJ0RC1LvPf1/d3b13yUK7nfDh0w7XkKMoI5hye+CmpdFV3uMpU9jTj0EtBxhUhhBCiKOyaBxhQpwNUqevu2uSqXAcfm6PiSTifObalo/KL4yYA7tHWucoknLexOSreLfUTQgghitKu42eJWTsj88H1A9xbmXyU6+Aje5rZXy4kWblZ3UVVzuZZTgghhCiLNq1fSYj9ODbFCk36urs6eSrXwUf2NLNRRg2OmBtgUnT6aX+5jh/47xwbD52R4RchhBBlTvTZVHZHJ7LzeAI++38EYLlxIztP6eyOTiT6bKqba5hTuZ5w2jrUn2A/D+IS0zAAswoJQTfB8YPcrf3Jl447AIWpqw8ydfVBgnytDGhdm7qBlajm40HrUH9ZjiuEEKJUcy6sMGNns/UvUGBeRgfWfXrxJru0Lawo1z0fmqowtncTABRAUSCmcmvSDTON1eOEKVFZysclpfPRygM8MzeCAV9uouPEVSyNjHVDzYUQQoiCebhDXQBuUbdTRTnHf0Zl/tLDcpwvTcp18AGZaWanP9CSIL/MIRi7qRIrjFYArtSzeYlLTGPYrO0SgAghhCiVHLrB0sg44OJiivmOm3BwMafH0si4UjetoNwHH5AZgKx/6Ra+HnIjAAv0zFUvfbW/MGPP83nOX9W4xXtL3S9OCCGE2BwVT2xiGoEk0kWNAOBHx81ZysQmppW6VZ0VIviAzCGY1qH+APxlNCPOqIK/co5bLsn5kRuD0vmLE0IIIZyrNftp6zEpOjv0BhwyQvIsV1pUmODjUjoqCy4su70050d+StsvTgghhMhc1Wm4vst+ytbrkbVc6VEhg4/ASlZ+upBwrLMaQQCX3+W2tP3ihBBCiNah/nT2OUFj9TjphpnFjnZZzitAsJ+Hq+e/tKiQwUdMUgYHjZpE6PUwKw76ahvyLFtaf3FCCCGEpiqMq505fWCZ3ookKrnOORNFjO3dpNSljaiQwcf79zRDVeAnRycA7tXWcnF66UWl+RcnhBBCYE+nTszvAKzyuDXLqSA/D6Y/0JIeYcHuqFm+ynWSsbz0vT6YxsF+DJxyjtdNs7hWPUZT5Qh7jNAs5YL8PBjbu0mp/MUJIYQQ7F8CaQngU4MPnhlF/6OJnExOK/WJMitk8OGUhDfL9Fb00TbS37SGN2yhjL+zGZWsWqn/xQkhhKjYdkUn4Fj0KS0Arr8fzWSiXf0Ad1erQCrksAtAgLeFqt5Wtla5HYA7TRsJqaTQpXFV+jYPoV39AAk8hBBClFrL/97JdWlbMh80H+TeyhRShe35CPbzZP3LXbAoneHjz/FJimZNnxTMfp7urpoQQgiRq+izqZxNsaEoYI78AU0xiOAaTOlVMaITqVLJTM0qXu6u5mVV2OADwGq6kH62+QBY9x7mXXPg+nvdWykhhBAiD85N5MBgpWUVqPCDrSOzp6x3lSltm8jlpsIOu2TRfGDm34dWQWK0e+sihBBC5GFy/+aYVIWWygEaqCc4b1hYdCG3h0lVmNy/uXsrWEASfAD414M6HQEDds5xd22EEEKIXPVrEcKCER0upIiA3/U2nCNzmGXBiA70a5EztXppJMGHU4sHAEjf+h0DPt/IrugE99ZHCCGEyIVqS6G3thGAefbOKGVwbYQEH05N+oDFB2vSURxH/uKX7THurpEQQgiRQ40Ty/BW0jih1qBv33toFuJHVW8rAd4Wd1etwCr0hFOnzNnDdkJC78B//1zuM61lws7rueeGmhgGZWb2sBBCiPKv8r55AAR3fpRBbesysE0dMhz6xUUUZYAEH1ycPdxSacwvVrhd/ZvwlER6lbHZw0IIIcq50wfh2EYMRWV7lZ5ER8S4kmKWJTLswsXZw9uNhhzUa+ClpHOHtgkoW7OHhRBClHM7vgNgg9Kcu78/wjNzIxjw5SY6TlzF0shYN1eu4CT44OLsYVCY5+gMwP1aZm9IWZo9LIQQovzadew0ZzbMBODbtJuznItLTGPYrO1lJgCR4COb+fpN2AyNFupBGinH3V0dIYQQAodusPa3OQQYZzlt+LJKb5nlvHNf9nGL9+LQc+7SXtpI8HGBc6+XGiG1iQ3qDMBDHn+WqdnDQgghypfos6l8vvYQbcavpNGJBQDMd3TElsuUTQOITUxjc1R8yVbyCsiE0wtce71oKsqBVJj9BwM8/kKpJPGZEEII93AuiAgkkVusOwD44cL0gLycTE4r7mpdNflmvYTVpKEoCjToCj41UM6fhX2/urtaQgghKiCHblDZ0wzAXdo6zIqDCL0+B4ya+T6vmo9HSVTvqkjwkRtVc2U8Zft37q2LEEKICmlzVDwJ522AQX9tDQBzHLfkWV4Bgv3KxrJbCT7y0mJQ5t+HV8PZI26tihBCiIrHOXzSWtlHfTWWFMPKr462uZZ1Zlgf27sJmlr6861L8JGXKnWhXmcA/lv3PwZ8sUn2exFCCFFinMMn95sy530scrQnBc9cywb5eTD9gZb0CAsusfpdDZlwmp8Wg+HwGjwi5/J3cmt+2e7DdTUru7tWQgghKoDWof408rVze/rfAMx1dMlRprKnmU8HtaRtvYAy0ePhdFU9HxMmTEBRFEaNGuU6ZhgG4eHh1KhRA09PTzp37syePXuutp4lLvpsKpE+N2G3VsbPdpKb1V0s3nmCyJhEdkcnEn021d1VFEIIUY5pqsLkJv/iodjYp9dip1HfdU658Ofdu5vRoUFgmQo84CqCjy1btvDFF19w3XXXZTk+adIkPvzwQ6ZOncqWLVsICgqiW7duJCcnX3VlS1LHiavp9dlWvk3JHF+7X1tNfEoGvaasp/fU9a7lT0IIIUSxMAyaxC4A4DdLdy7O7Ch7wyzZXVHwce7cOQYNGsSXX35JlSpVXMcNw2Dy5MmMGTOGu+66i7CwMGbOnElqaiqzZ88uskqXBOd+L85urq7qdgI5C8h+L0IIIUpAzHb4LxI0K6OefZ05j7fl4/ubM+fxtqx/6ZYyG3jAFc75GDFiBHfccQe33norb7/9tut4VFQUcXFxdO/e3XXMarXSqVMnNmzYwNChQ3NcKz09nfT0dNfjpKQkAGw2Gzab7Uqqlyfn9Qpy3TvCqlHXvw39phts0xtyg3qAe7V1THP05aehbWhaw7fI61eaFKatKjppq4KTtiocaa+CK09t5dANftwWjf/qd7kDcDTujW7xplVtAF8AdIcd3XFl1y+utirM9QodfMydO5ft27ezZcuWHOfi4uIAqF69epbj1atX5+jRo7leb8KECYwbNy7H8eXLl+Pl5VXY6hXIihUrClTu+DkAE3McXbhBPUB/bTXTHb1Zv349R72LpWqlTkHbSkhbFYa0VeFIexVceWmrdYcz+ChjHSiwMb0RZ37/vchfo6jbKjW14HMhCxV8HD9+nGeeeYbly5fj4ZF3BjVFyTrxxTCMHMecXnnlFZ577jnX46SkJGrVqkX37t3x9fUtTPUuy2azsWLFCrp164bZbL5s+djENL6J2sQR3+5kJM2mjv0kPb3+pXaT+zAMg0BvKzfUqVLmJvoURGHbqiKTtio4aavCkfYquPLQVvO2HufNxXsxgLuUTXib0zisB/HI3iaAwhu9m9C/Va2rfp3iaivnyEVBFCr42LZtGydPnuSGG25wHXM4HKxbt46pU6eyf/9+ILMHJDj44ljUyZMnc/SGOFmtVqxWa47jZrO52N5ABb127UAz60Z3IeJYAjF/9SY0ag53GSt4bF5jV5lgPw/G9m5Spsfe8lOcv4fyRtqq4KStCkfaq+DKals5dIPXFv6Dc1LpAEvmooZ5ji5k6JnTM19b+A8D2oQW2Q1vUbdVYa5VqAmnXbt2Zffu3URERLj+tGrVikGDBhEREUG9evUICgrK0pWTkZHB2rVrad++fWFeqlRYGhlL1w/WMvCrvxm+L3NVz836ZgJIdJWJS0xj2KztLI2MdVc1hRBClHGX7kR7rXKU5uohbIbGz46b8yxXlhWq58PHx4ewsLAsxypVqkRAQIDr+KhRoxg/fjwNGzakYcOGjB8/Hi8vLwYOHFh0tS4BSyNjGTZrO8aFx/8YdYjQ69NcPcTd2jq+cPQGMrcwVoBxi/fSrUlQuRyCEUIIUbwu3Yl2oPYHAMv0VpzGL89yZVmRp1cfPXo0o0aNYvjw4bRq1YqYmBiWL1+Oj49PUb9UsXHoBuMujLtdavaFDX0GaKvgkrMGmfNDyktEKoQQomQ5U6l7kUY/7S8AZju65lmurLvq9Opr1qzJ8lhRFMLDwwkPD7/aS7vN5qh4YhNzRpe/OtrxumkWoep/tFP3slFvmuV8eYlIhRBClKzWof4E+3lw07nV+CjnidKrs1FvkqVMWdmxtiBkY7lc5BVEpOLBIkfm3JXM3o+syktEKoQQomRpqsLY3k0YdGHIZY7jFoxLvqIVys6OtQUhwUcu8gsinEMvt6lb8CfrsqLyEpEKIYQoeT0CTnK9ehgbpiwTTYPLeCr13Miutrlwdn/FJablmPexxwhllx7KdWoUd2vr+NLRC4CHO9QtNxGpEEIIN9g6AwCtaR+mtryNk8lpVPPJHGopb98v0vORC2f3F1y6jc9F3ztuBTJnJCvovNzjGro3CWJhRAwbD53BoWcPWYQQQoi8RR6O5vz2uQCorR6mXf0A+jYPoV39gHIXeID0fOSpR1gw0x9oybjFe3NMPl3saMeYCxNP26t7+Gq9B6fP7XedL++Jx4QQQhQdh24QseR/hBnniTPXpGrtjmjurlQxk56PfPQIC2b9S7e4dhKcOqAFgd5WGtSsTmydvgAM0v7g9LmMLM+TxGNCCCEuJ/psKp+vPUSb8Su57r/5AHyV2ok2E/7g87WHiD5b8L1Syhrp+bgMTVVoVz/A9bhb0+pYNBX9vyrw2Vy6q1upxllOUsVVRhKPCSGEuJyOEzNTqIcph7nOGkW6YeJnx02cPZfBhCX7mLBkH0fevcPNtSwe0vNRSFaThqIobE4JYrN+DSZFp7+2Okc5STwmhBAiO4dusPHQGeZvj6aSNXNwxbm8dqnemrNc3FC1sqe53M4hlJ6PK3QyOY1V9q60tuxngGkV0xx9ceQySieJx4QQQkDmth3Z5xH6kuLKaDrLfmuW8gnnbWyOis/S+15eSM/HFarm48FSvTXxhjc1lHg6qxF5lhNCCFGxOfcLy76A4S7tTzyVDPbrNdliXJPjeeX1BlaCjyvUOtQffz9ffnJ0Ai52mzkplK9UuEIIIa5MXvuFgeH67pjluJXckjuU1xtYCT6ukKYqjOxS35XxtLO6k5rKKdd5AxjZpb5MNhVCiAour/3C2qr/0FCNIcWwMt/RMcu58n4DK8HHVRizYA9HjGD+dIShKgb3Z9vvZcyCPW6qmRBCiNIir6GTB7SVACxwdOQcXq7jzlvW8rSXS3YSfFyFyf2bY1KVC91l0F9bjQUbACZVYXL/5m6snRBCiNIgt6GTqpzlNnULgOs7xCmoHO7lkp2sdrkK/VqE0KCaN/2m2IgzqhCknKWHuoVFensWjOhAWIifu6sohBDCzXLbL6y/tgaz4mCb3pB/jDr4VzLzeq+mBPmWz71cspOejyJgx8RsR1cABpuWu7k2QgghSpPs+4Wp6AwwZQ7Tf2+/FQUYf2cz7mxRfvdyyU6Cj6sU4G2hqreVndX6oismblT/pX2lWAK8Le6umhBCiFIiLMSPl3s2JsDbwi3qDkKUM8Qb3mzyvImXezaucD3lMuxylYL9PFn/chcsmoryU2/YM5/vrtuF5veYu6smhBCilHCmUgd4wJw50fRHRydOpFPuU6nnRno+ioAz5To3ZgYcWuSPkJbo5loJIYQoLZwLFOoocdys7gJwDddXxAUKEnwUpTodoOq1YEuFiDnuro0QQohSol+LEBaM6MAD2kpUxWC143qOGkEALBjRgX4tQtxcw5IlwUdRUhS48VEAEv+cTrcP1jBr09FyuzGQEEKIglPs57lPWwPAt47uKOV/XmmeJPgoYivMnUnBA7+UI1Q98zevLYik48RVLI2MdXfVhBBClCDnDrYLI2LYeOgMwccW46ekEqsG0a3PQJqF+FHV21ohFyjIhNMi9P2mI4xZ8C9vmm7iQdMKBmsr2KCHEZuYxpOztvNOv6YMalvX3dUUQghRzHLuYGuw3HMq/kBQ1xEMbFuPAW1CyXDoWE05d0Qv76Tno4g4dMOVTv07RzcAuqnbCOKMq8yYBXtkCEYIIcq53HawbaXsp5FxhDTDzCrPzO8IRVEqZOABEnwUmc1R8a5/HzBqstHRBJOiM8j0R57lhBBClC957WD7oGkFAAsdHXht2YkKfyMqwUcRyb5x0ExHdwAGaKuwkpFnOSGEEOVHbjvYVuUsPdXNQOZE09jEtAp/IyrBRxHJvnHQCv0GYowAApUk7lA35VlOCCFE+ZHbDeZAbRVmxcFWvRF7jLp5lqtIJPgoIs6Ng5wcaHxvz9ypcIhpOWAQ7Je5YZAQQojyKfsNpgk7Ay8Mv39r755nuYpGgo8i4tw46NJl23McXUg3zFyvHqaFcpCxvZtUiA2DhBCiosp+g3mbupXqSgKnDD+W6K3zLFfRSPBRhHqEBTP9gZauHpCz+LLI0Q6AqQ224Odpca33ruiTjYQQojzSVIWHO9R1PR5iWgbAbMct2C5kt3i4Q90KfyMqeT6KWI+wYLo1CWJzVDwnk9MIzXgBfl9HteNL6fflUk5RGYBgPw/G9m5Cj7Bg91ZYCCFEkRrbuynBvh4sXLqE1up+bMbFYfhXezbmiU713VxD95Oej2KgqQrt6gfQt3kIJ7wasVVvhBk7A7WLy27jEtMYNmu7ZD4VQohyqH2DQB6+0Ovxu96GU1RxHRcSfBQr53rvmRcmGQ0y/YEZO4BrDfi4xXtlCEYIIcqZqmoifbQNAFg7DqdZzYqbSj03MuxSjJzrvZfQmjijCkFK5lrvRXp7IDMAca73blc/wL2VFUIIUWSq/zsPsGOE3ECP23pzW3ejwqZSz430fBQj5zpuOya+t3cF4CHT0jzLCSGEKAccNtjyFQBKmycz/67AqdRzI8FHMbp0HfccR1fSDRMt1YNcrxzMs5wQQogybu9COBcH3tWhST9316ZUkuCjGF2aeOw0fiy+MNzy8CW9H5J4TAghyo9d0Qn8u+j9zAetHgWTzPHIjQQfxciZeMxphr0HAHeof1OdzLz+knhMCCHKj7//XEEj2z84FBO0etjd1Sm1JPgoZj3CgplwZxiqAnuMuvytN8asOHjQtIIJd4ZJng8hhCjjos+msjs6kciYRGr8+y0AS2hPZKKV3dGJRJ9NdXMNSx9Z7VICBrSpQ98WIUQcS0D5dxhsfobhPn+itKzm7qoJIYS4Sh0nrgYyd6/9y/oXKPBZWncip6x3lTny7h3uql6pJD0fJcTLYqJ9g0Ba9xgMlWujnI+HXT+4u1pCCCGu0uT+zVEVGGxageXC7rWRRj0AVCXzvMhKgo+SpmrQemjmvzdNB0MSjAkhRFnmYVYxGxkMupDF+n/2nq5zupF5XmQlLeIOLR4AcyU49Q9vT/2MXdEJ7q6REEKIK+DMZH2ntp4AJZloI5DleqssZSSTdU4SfLiDZ2VoMQiAtifn8cv2GPfWRwghxBXJzGR9nke0JQDMsN+Gg6zJxJyZrMVFEnyUMOes6H/rDATgVm0HERHbiIxJlFnRQghRSjl0g42HzrAwIoaNh864ejJOJqdxs7qLRmoM5wwPfnB0yfX5ksk6K1ntUsKcs6IB/mduQVdtB3dmLKLXlIt7u8isaCGEKD2WRsYybvFeYhMvBhDBfh6M7d2Eaj4ePHqh1+MHR2eS8cr1GpLJOivp+ShhzlnRAP9zZE5Kuldbhx/nZFa0EEKUMksjYxk2a3uWwAMgLjGNJ2dt53+//E4nbRe6oTDDcVuO5ytIJuvcSPBRwjzMKs55Rxv0pvyj18ZLSWeQ9ge6AUfPpOTo1hNCCFHynJNJc/skdh67JfFnAJbrrYg2qmcp48xdLZmsc5LgowQ538gXKXxpvx2AIaZlmLHz0coDPDM3ggFfbqLjxFUsjYx1T2WFEKKCy5xMmvdcjSokcZeWmUgso9VQAryz7uMS5OfB9AdaSibrXEjwUYJyeyMv1tvzn1GZ6koCvdUNWc7FJaYxbNZ2CUCEEMINLjdJdJD2Bx6KjV16KE9v9OT0uQwAPr6/OXMeb8v6l26RwCMPEnyUoNzeyDZMzLRnjhM+ZloCl3TwOf8la8SFEKLk5TdJ1IKNIablAHxt7wkomFSFyf2b07d5CO3qB8hQSz4k+ChBeb2Rv3d0JdWw0kQ9Sjt1b5ZzBrJGXAghSpJzWW1c4nn8K1nILYToq/1FVSWRE4Y/v+ptAVgwogP9WoSUbGXLKFlqW4Jah/oT7OdBXGJalglMiXjzo+NmhphW8Lj2Gxv1pjmeK2vEhRCi+OW2rDY7BZ0ntN8AmGHvgUMxkeusVJEn6fkoQZqqMLZ3E4AckfTXjp7ohsItWgT1lZwZT2WNuBBCFK+8ltVm10ndSUM1hhQ8adhzBM1C/Kjqbc0x4VTkTYKPEtYjLJjpD7QkyC9rMHHUCGKFfgMAj2q/ZzlX1ccqa8SFEKIY5bes1sm/kpmP+jfn41qZK1y82j3KfR3DWDiiA+tf7kKwn2fJVLYckODDDXqEBbP+pVuY83hbPr6/ueu4c9nt3dp6Akl0HT+VnC4Tl4QQohhdblktQHyKjfq2g/j9txFDNbE9+H4WRsSw6XA8JlW+TgtD5ny4iaYqtKufmVLdMOC5HyLYalxDhF6f5uohHjQt40P7fagKfHhfc/dWVgghyrmCzqs7tfwDAJbTjqGzjwHHgIvp1mVpbcEUKlSbPn061113Hb6+vvj6+tKuXTuWLFniOm8YBuHh4dSoUQNPT086d+7Mnj17irzS5U2/FiEsGtkRUPjM3huAwdpKPElj/vAOVPf1kKynQghRjAoyr64Gp+lk+xOAT873yHJO8jIVTqF6PmrWrMm7775LgwYNAJg5cyZ9+/Zlx44dNG3alEmTJvHhhx/yzTff0KhRI95++226devG/v378fHxKZYfoLxZYbQiSq9OqPof92lreXSmrytxDUh0LYQQxSGv1YiXesS0FJOi85ejKXuM0CznDDIXEoxbvJduTYJkqPwyCtXz0bt3b26//XYaNWpEo0aNeOedd/D29mbTpk0YhsHkyZMZM2YMd911F2FhYcycOZPU1FRmz55dXPUvNwK8LVT1ttI0pAqnmj0OwGPa75w9dz5LOYmuhRCi6OW3GhHAh1T6a5m7kn/pyH3nccnLVHBXPEPG4XAwd+5cUlJSaNeuHVFRUcTFxdG9e3dXGavVSqdOndiwYUM+VxIAwX6erH+5CwtHdOCGPiOIx5da6iluV//OUk6yngohRPHIazUiwCBtJT7Kef7VQ1ijX5/vdSQv0+UVesLp7t27adeuHWlpaXh7ezN//nyaNGniCjCqV8+6q1/16tU5evRontdLT08nPT3d9TgpKQkAm82GzWYrbPXy5bxeUV+3qKiA3a6z+WgyWx3deUb7iSfNv7Lc3pbssXj8ufNsOniy2Jbglva2Kk2krQpO2qpwpL0Krqjaqus1gXRueBPbjp7l9Ll0Ar2teKp2an43HIDP7b1xfh6bVYPcRlcCvUyl+ndWXO+rwlxPMQyjULfPGRkZHDt2jISEBH7++We++uor1q5dS0JCAh06dODEiRMEB1+cj/D4449z/Phxli5dmuv1wsPDGTduXI7js2fPxsvLqzBVK1fM9mS673kWk57B/JCXmHaqGX3q6NT2dnfNhBCiYvGJWcMtJ7/mhOFPp/TJ2NEwUHihmZ1a8pnskpqaysCBA0lMTMTX1zffsoUOPrK79dZbqV+/Pi+99BL169dn+/bttGjRwnW+b9++VK5cmZkzZ+b6/Nx6PmrVqsXp06cvW/nCstlsrFixgm7dumE2m4v02kVpc1Q8j8zcwhh1Jg9qy1mnX8eDGS+jKQamSwbKvh5yY7H2fJSFtioNpK0KTtqqcKS9Cq7Y2kp3wPR2mBMO85XXoxyuP4Q5W45jABYVlAs9H84OkI/6N+fWa6vndbVSobjaKikpicDAwAIFH1ed58MwDNLT0wkNDSUoKIgVK1a4go+MjAzWrl3LxIkT83y+1WrFarXmOG42m4vtP1txXrsotG1QDV8vD744dzuD1BXcrO7iWuUo/xh1cDgyy1T1sdK2QbVin1Fd2tuqNJG2Kjhpq8KR9iq4Im+rvUsg4TCGhx+PPh2O4uHLTY2qMm7xXuKSLt44l8WViEXdVoW5VqGCj1dffZWePXtSq1YtkpOTmTt3LmvWrGHp0qUoisKoUaMYP348DRs2pGHDhowfPx4vLy8GDhxY6B+iItNUhVPJ6UA1ftfb0FvbxFDTYkbZRrrKSNZTIYQoZoYBf00GQLnxcfDIvJvv2awG3ZsGszkqnpPJaVTz8aB1qL98JhdCoYKP//77j8GDBxMbG4ufnx/XXXcdS5cupVu3bgCMHj2a8+fPM3z4cM6ePUubNm1Yvny55Pi4ApP7N+e5HyL4zN6H3tomeqmbeF+5j2ijmmQ9FUKIYrYrOoEF83/gjTPbQLNCm6FZzl+apVoUXqGCj//973/5nlcUhfDwcMLDw6+mToLMrKcNqnnTawqsdVxHJ20XT2i/8Yb9YRaN7EhYiJ+7qyiEEOXWL9tj6HRyFmhAi0HgXc3dVSpXZCecMmC6ow8A92lrsmw4J4QQ4uo5dIONh84w468ovt90lJ3HE/gnYgNdtJ04UPm3/kPsjk4k+myqu6tabsjGcqWYM+tpql9bTjmuo2rCLkZ4riDA+053V00IIcqFpZGxjFu8N8eOth+Z54MGSxytGTkzGogG4Mi7uWc3FYUjwUcp5sx6atFUlP0vw9yBPGRZgWLNADzdXT0hhCjTlkbGMmzW9hx7udRS/qO3uhGAz+y9ADCpCu/fm39mU1FwMuxSyllNGoqiQKOeULUxSnoy23/5kG4frmXWpqOSYl0IIa6AQzcYt3hvrpvIDdV+xaTorHVcR6RRD4AFIzrQr0VIyVayHJPgo6xQVXbVfQiAWvtncOxkPK8tiKTjxFWyyZwQQhSQc37HRyv25xhqAajKWe7V1gLwqb1vrpvMiasnwUcZ8f2mI9z1Zw2ijUCqKonco60DMndQfHLWdr7fdMS9FRRCiFJuaWQsHSeuYsCXm5i6+lCuZR4z/Y5VsbNVb8RmozE1/T2p6m0lwNtSwrUt3yT4KAMcusGYBXuwY+JLe+Zkp6HaYjQcrjJjFuyRIRghhMiDc35Hbr0dTn6c4wFtJZDZ6wEKE++6jvUvdyHYT+bZFSUJPsqAzVHxrn/Pc3TmjOFDbfUUvS5MiMqtnBBCiEz5ze+41EPaMiop6fyj12aN3pxgPw/a1AvAatJKpJ4ViQQfZcDJ5IuRehpW/mfvCcBI00IU9FzLCSGEyLQ5Kj7fHg8AL9J42JS5+7qz12Ns7yaSMr2YSPBRBlTz8cjy+DtHd5IMLxqqMdymbs2znBBCiILdmA3Q/qCykkKUXp0d3jcz/YGWZWqTuLJGgo8yoHWoP8F+FwOLZLyY4bgNgJGmBYBBsF/mxkZCCCGyutyNmQUbT5h+A+Ab9U6mDW5NSGUvyWhajCT4KAM0NbP779LOvxn2HqQYVsLUI3RRI6R7UAgh8uC8gcvrE/JebS3VlQROGP7MSWtP30//ovfU9XScuLpE61mRSPBRRvQIC2b6Ay1dPSAJ+DDLcSsAHwUtp0fTIHdWTwghSi3nDRyQIwAxYWeYaREAX9h7kXEh8bdJVZjcv3kJ1rJikfTqZUiPsGC6NQlic1Q8J5PTCDHVx5i/ksrxOzm05Xdeiwjgldsbc13Nyu6uqhBClCrOG7js+7g86LWRmvppThl+zHHc4jq+YEQH2T28GEnwUcZoqkK7+gEXDxwbApu/QPnzAzaeep5ftsdI8CGEELnIfgNXrZKJlotegWT4wtGLdCwoChiSMqnYybBLGRZ9NpV99R7GUEzUS95GS+VfFu88QWRMomz/LIQQuXDewPVtHkK71DVYk49yFl8iqt/FO3eG0SzETzKalgDp+SjDnJOh3jV15H7TGp4yzeeRlEb0mrLeVUa2fxZCiFzoDlj3HgA+XZ7hh5u7oigKA1vXJsOhS2KxYiY9H2XY5P7NMakK0x19sBsqXbSdXKccBEBTFAa3rc3GQ2ck7boQQlxiV3QCH095D84cAI/KmNo8kbl7OKAoigQeJUCCjzKsX4sQFozowFEjiIV6BwCeNs0HwGEYfLfpGAO+3CQ73wohxCXmbzvObWdmZT5oOxw8fN1boQpIgo9yYqqjHw5Doau2g2bK4Szn4hLTGDZruwQgQogKK/psKrujE4mMSSR550Iaq8c5hxd7aw+QOXJuIHM+yrgAbwtVva14+zXm9zMd6c2fPG36hcdtL7jKGGSubR+3eC/dmgRJMjIhRIVzMWGYwa+WH0GFb+zdef+L3a4yMkeu5EjPRxkX7OfJ2tGd6Xt9MB+l98VhKHTTttNUicpSzgBiE9P4aMW/Mg9ECFHhOOfIdVO3EaYe4ZzhwVcXNumUhGIlT3o+yrilkbGXJM2pwSK9PXdqfzHK9AuP257PUX7q6oNMXX2QYD8PxvZuIhsnCSEqhH4tQmhQtRLKFy8D8I3jNhLwASShmDtIz0cZtjQylmGztmfJ1jfV3g/dUOimbaOpciTP58o8ECFEReNzZBlN1aMkG558Zb8dRUag3UaCjzLKoRuMW7yX7IMnh4wQFuvtAHja9Euez3c+b9zivTIEI4Qo/3SdGhGTAfjdqw8v3tlOEoq5kQy7lFGbo+Kz9Hhc6hP7nfRWN3KbtpUm9iPsNermWs45D2RzVHzWlO1CCFHe7PsV8+m9GFYf7nvqXRQvf0ko5kbS81FGnUzOPfCAzN6PX/W2ADxr+vmqriWEEGXdruPxHPvlDQCUNsNQvPwz/y0JxdxGgo8yqpqPR77nP7bfdWHly7YceT8Key0hhCjL9q36ntr2KNLUStBuuLurI5Dgo8xqHepPsJ8Hec2XOmSEsOBC1tPnTD/mWkYBgv08aB3qXzyVFEIIN3ElFYs+S8uoLwD41ridyHhVkoqVAjLno4zSVIWxvZswbNZ2FMgx8RQgqskI7Ps30EXbSUv7v2w3GrnOOYOWsb2bSNIxIUS540wqdoe6iU8tx0gyvJia1p3xsvFmqSA9H2VYj7Bgpj/QkiC/rMMmQb5Wpg9qwQsDbkdrMQiAlz2yrnwJ8vNg+gMtJc+HEKJcmty/ORbV4FnTTwB87ehBEpUASSpWGkjPRxnXIyyYbk2C2BwVz8nkNKr5ZA6jOHszlE6jYedcWuu7+OLmNN7bF8iQ9nW5r1Utth09y8KImBzPEUKIsq5fixBaJSyl5toTnDW8+Z/9dtc5SSrmfhJ8lAOaquS9VLZybWj5IGz9H432fsKB0y/yxz//8enqg1mW6krGUyFEWeLQjSw3XS1q+mQtYM+g2vaPAPjc0YtkvFAUMCStUakgwUc5F302leSGT3DNjlnUPRdBBzWS1fub5SjnzHgqQzFCiNIu67YSmepUsfJc40sKRczCknycM1RmW/V7ead1Q+ZtOU5sQpokFSsFJPgo55yTrsaaOvOwaRnPm37kr4wwyLZOJvvOt0IIURo5t5XI3oHxX1JmILLyn//oeW0grH0PAL/uL/FDu64oiiJJxUoRmXBazjl3cpxm78t5w0JL9SBd1e25lr0046kQQpQ2eW0rAVm3jJg55Q1IPgG+NTG1fhTlwiYuklSs9JDgo5zr1yKEBSM6cIrKzHD0AOAF0w8o6Hk+RzKeCiFKo/y2lXBKOZfEHYlzMh90ehFM1hKomSgsCT4qkM/tvUgyvLhWPU4fdUOe5STjqRCiNMrvxkg34Pg5GKwuI1BJ4jhBRFbrJQnFSikJPiqAAG8LVb2t1A4J4RulDwDPmX7ChD1LOcl4KoQozfK7MbLpCl/sTmeo6VcAPsi4i16f/k3vqetdc99E6SHBRwUQ7OfJ+pe7sGhkB67tN5pThi911JP019a4ykjGUyFEaZf/thIGT5oW46eksl+vySK9PQCqgiQUK4Uk+KggrCYNRVHo1rw+p1o8BcDTpl/wIB2QjKdCiNLPua0EZF+vByHqWR7WlgLwnr0/+oWvN90AD7N81ZU28hupgJr0egbDrxbVlQQe1JbTo2kQ61+6RQIPIUSpl9e2EiO0+XgqGWzVG7JSb5nl3KvzdzN/RwwbD53BoUuWsdJA8nxUMNFnUzmbYqNK81HUXPs8w0yL6RfVg39iG2AYUKWSmZpVvNxdTSGEyNOl20r8dfAUv65Zz31q5ryOSbb7yd4vEp9i49l5EYBkcy4tJPioYJwTr1Sqs8wSQkM1hvsyfqHXFLOrjOz0KIQo7TRVoZJVY9me/3je9CMmRSfW53oiMhrnvs33BZLNuXSQYZcKxpl0TEflPft9ADyiLaEaZ2WnRyFEmfLL9hgsp3bTW9uEbijsq3EvymXmy1+ajEyGYNxHgo8Kxpl0DGC53oqteiM8lQxGmX5mwYgO9GsR4uYaCiFE3qLPprI7OpHImEQW7zzBaNM8ABbr7dmj1y7QxnGSzdn9ZNilAlMUhXdt9/OT9U3u09ZwOOEghNzg7moJIUQOzl1sB3y5yXWsnbqHTpZd2AyN9+33cHx34b7SJJuz+0jPRwXkTDrWLMSPO/vdw9+WNpgUndo73nd31YQQIoelkbF0nLgqS+ChoPOyKTON+mzHLRw3qqMqBial4EMpks3ZfaTnowJyJh2zaCqKomCEfowxvT0eB3/n4LaVvL7Nmxe7N3B3NYUQIs9dbHupm7hePcw5w4Mp9rsAeC7MQYsb23LynI23fvuHsykZuc49VcjMbSTZnN1Hej4qKGfSMQCl2rUoLR4AwLJqHBsPn2ZBRKw7qyeEEHnuYmvB5prr8Zm9N2cUP9e51qH+3NmyJuPvDANyJiOTbM6lgwQfguizqfxzzUh0zYPaKbu4Vd3Ob7vjOH4OImOSZFMmIYRb5LWL7WBtBbXUU8QZVfjKcTuhAZUI9LbgczFjQJ7JyCSbc+kgwy7ClfvjRVN3RpgW8ZJpLj1SmvP+bhPv784cY5XcH0KIkpbbhFBfzvGUaT4AH9rvIQ0rT3dtwK2Nq/LH8qVZyl6ajOxkchrVfDKHWqTHw/2k50O4cn98Zu9DvOFNQzWG+7XMgEST3B9CCDfJbULocNMiKisp7Ndr8pOjEwDVfT2xmnL/OtNUhXb1A+jbPIR29QMk8CglJPgQrtwfyXgx2X43AM+afsKbVH4e2kZyfwgh3CL7LrYhnOJhbRkAE+wDMFAJlomjZZIEHyKLOXpXDunBBCpJDDMtcnd1hBAVkEM32HjoDL/uOsHtYcGuCafPm3/Eqtj4y9GUNXpzDGBkl/rSm1EGyZwPAWTm/gj0tuLrUYk//UdS/9gYHtWWcEYfAwS4u3pCiApiaWQs4xbvzTHRtJlymLu09UBmr4dz3cqYBXsY1LZuCddSXC3p+RAA7DyegEmFw6dTCP+3Ln/r1+Kh2EhZMpYBX2xiV3SCu6sohCjnnDk9cq5wMXjd/B0Avzg6EmnUA5D9qMowCT6E6z98XFL6hSMKEx2DAGh0cikpUZuZtvqQbMIkhCg2eeX0AOipbqa1up/zhoVJtv6u47IfVdlVqOBjwoQJ3Hjjjfj4+FCtWjX69evH/v37s5QxDIPw8HBq1KiBp6cnnTt3Zs+ePUVaaVF08voPv9sIZZ9P5gZ0Y8zfs3RPLG3Gr+TztYck74cQosjlldPDSgavmGYD8IWjF3EEXHbnWlH6FSr4WLt2LSNGjGDTpk2sWLECu91O9+7dSUlJcZWZNGkSH374IVOnTmXLli0EBQXRrVs3kpOTi7zy4url9R/epis8fOp+0gwzbdR93KZu5fS5DCYs2efKCyKEEEUlr03eHtKWUftCQrHP7L24r1VNmoX4UdXbSoC3pYRrKYpKoSacLl2aNYHLjBkzqFatGtu2bePmm2/GMAwmT57MmDFjuOuuzFz7M2fOpHr16syePZuhQ4cWXc1Fkch7V0eDWAL40nEHT5kW8Krpe1ZnNCcDM5U9zTh0Q2aYCyGKTG45PQJIZKRpAQDv2fpzHg/ubFGTtvX8yXDoWE1aCddSFJWrWu2SmJgIgL9/5hrrqKgo4uLi6N69u6uM1WqlU6dObNiwIdfgIz09nfT0dNfjpKQkAGw2Gzab7Wqql4PzekV93bIs0MuEVcs5ympSDFLsCtPtfbhPW0Md9SSPaEv4n9Gb8xkZbDp4UtbWXyDvq4KTtiqcitReLWr6UKeKlf+S0jAAuw7PaT/ho5wn0qjL70oH6lSx0qKmD3a7HRWw2XTX8ytSW12t4mqrwlxPMQzjimYRGoZB3759OXv2LH/++ScAGzZsoEOHDsTExFCjRg1X2SeeeIKjR4+ybNmyHNcJDw9n3LhxOY7Pnj0bLy+vK6maKALHz8H7u00oGNyp/smHls84Z3jwQ71JBFSu7O7qCSHKofh0SLnw/fXHPzH8pL2Kphj8UvNVTng2ppIZ/K3uraPIW2pqKgMHDiQxMRFfX998y15xz8fIkSPZtWsX69evz3FOyTYbyDCMHMecXnnlFZ577jnX46SkJGrVqkX37t0vW/nCstlsrFixgm7dumE2my//hApi5T//8ey8CADXxFOzYuBjNkixwWKjAw/qK2iuHsL34M+M1p/g6yE3Ss/HBfK+Kjhpq8KpaO3V8PXlF/5l8J35ezTFYInjRp47GOYqc+Ct7rk+t6K11dUorrZyjlwUxBUFH0899RSLFi1i3bp11KxZ03U8KCgIgLi4OIKDL+4YePLkSapXr57rtaxWK1ZrzlDWbDYX2xuoOK9dFvW8riaKqmVN7KPB+BscvLJFI0PXeNM2mF+s4dylrmOBuSdT1gTwqpeV62pWdmvdSxN5XxWctFXhVJT2mty/OS/8uJOubOEmLZJ0w8x4+0AgM6fH+/def9l2qChtVRSKuq0Kc61CrXYxDIORI0fyyy+/sGrVKkJDQ7OcDw0NJSgoiBUrVriOZWRksHbtWtq3b1+YlxIlrEdYMOtfuoU5j7flkQ51ATCpuJa0bTcascDRAVUxeMP8HZsOn+GX7THuq7AQotzp1yKEhUNv4DXTLAC+cNzBcSPzxlVyepQvher5GDFiBLNnz2bhwoX4+PgQFxcHgJ+fH56eniiKwqhRoxg/fjwNGzakYcOGjB8/Hi8vLwYOHFgsP4AoOs7dH9vVD+DGOn5kRG3Lcv4Ly4PcYWyjUfoeeqsbWbzTyj031MQwoEolMzWryBwdIcTVCdz1BdXVU8Qa/kyz90FR4MpmJorSrFDBx/Tp0wHo3LlzluMzZszgoYceAmD06NGcP3+e4cOHc/bsWdq0acPy5cvx8fEpkgqLknHrtdX5PQq+HnIjp1PtPDM3gr0pPnyi9eJ580+8Yp7NrSkt6TXl4pyfj+9vTjWfzB0mZRmuEKIwdkUn8PnidXxyeioA3/s+ymsdWzFvy3FiE9Ikp0c5U6jgoyALYxRFITw8nPDw8CutkyhFWof6YzabMQx44cedfOHoRX/TGmoqpxlmWsQH9vtcZZ+ZGwFAsJ8HY3s3oUdYcB5XFUKIrH7ZHsNtJ6ahaWnotdry/MOvoqgqA1vXlpwe5ZDs7SIKpF+LEBaM6EA6Ft6yDQbgCe1X6ihxOcrGJaYxbNZ2lkbGlnQ1hRBlSPTZVHZHJxIZk0h0xEr6aBvRUTjc6g0iTyQTfTYVRVEk8CiHrirJmKiYluutWOu4jk7aLsaavuUR24s4t7eGzOW6CjBu8V66NQmSIRghRK6cWzWo6Pxq+R+oMMd+C2PmJACZQ7pH3r3DfRUUxUZ6PkSB7Y5OQFXAQCHcPoQMQ+MWLYKu6vYcZQ0gNjGNzVHxJV9RIUSZMLl/c0yqwiBtJU3UoyQaXrxvvxfIXFo7uX9z91ZQFBsJPkSBLI2M5dX5kegXpv1EGcF85ci8Ixlr+hYrGbk+L++9Y4QQFV2/FiGM7RLIi6YfAHjP3p+zZCaXHH3bNbK0thyT4ENclkM3GLd4L9mnG0+19yPW8Ke2eoqh2q+5Pje3zaKEEBWXQzfYeOgMCyNi+HjlAbzWvYmvkspuvS6zHV1d5cYv2SfzxsoxmfMhLmtzVPzFzKeXSMWDd2yDmGqZwnDTQn7RbyLaqOo6X9XHKinYhRAuSyNjs2RSvlHZx4/WP9ENhddtj6Bnux+WeWPll/R8iMvKb+jkV70tGxxN8FBsjDV9m+XcqeR0NkfF49AlQ5AQFd3SyFiGzdruCjw0HLxpngHAXEdnIowGOZ4j88bKLwk+xGXlP3Si8Ib9IWyGRjdtG93UrVnODvhyEx0nruL3XSdcXa0bD52RgESICiS3oduHtGVcqx7nrOHNe/b+eT5X5o2VTzLsIi6rdag/wX4exCWm5Zj3AXDQqMmXjjsYblrEWPO3rE8P4zwXA5bYxDSGz96R5TmSiEyIiiP70G01zjLK9DMAE+33uyaZ5kbmjZVP0vMhLktTFcb2bgJcms0jqyn2OzmuV6WmcppnTPMve01JRCZExZG99+I18yx8lPNE6PWZ5+ic63MUMm9SZN5Y+STBhyiQHmHBTH+gJUF+We9CqvlY8fUwERzoz1j7EAAe1X6nkXI83+s5e1DGLd4rQzBClHOX9l7crO6kj7YRh6Hwmu1hjFy+hpw3OWN7N5HJpuWUDLuIAusRFky3JkFsjornZHKaaxM5u66zZHcso+alsNRxIz20Lbxt/pr+Ga/n+sHidGkisnb1A0ruBxFClCjn0O3ZxETeNn0NwEzHbUQa9XItHyTDsuWeBB+iUDRVyREoaKpGdV9PAMbZHuQmdRet1f3co63jxzy6VC8lE8qEKN+cQ7eH5o6mtnqKWMOfDy5kMnV69taG1A2sJDtjVxAy7CKKhHNcNpYAPrLfA8Crptn4k3TZ58qEMiHKt13RCfyxdi1Pmn8DYKxtCClk3rCoCky4M4xnbm1E3+YhtKsfIIFHBSA9H6JIaKrCwx3qMuOvI3zjuI27tT+5Vj3Ga+ZZPGcbnutzFDK7V2VCmRDl2/xtx+n/3wdoqgN7o5481Hokd5xLp6q3lea1K+Nlka+iikZ6PkSRGdu7Ka/2bIwdEy/bHkM3FO7S1nOzujNHWZlQJkT5Fn02ld3RiUTGJKJFfEcr9V9S8eDQDWPx8TBzQ50qtG8QKIFHBSW/dVGk2jcIBGCX0YBvHLfxiGkp75i+ZoD5I6JTLsa6MqFMiPKt48TVAFQlgZXW70CB92338vWMw8BhAI68e4cbayjcSXo+RJEK8LZQ1dtKs5p+VOoZzkm1KrXUUyy57k/e7hdGw2revN0vjPUv3SKBhxDl2OT+zTGpCmPNM/G7sHHcTEd3AEyqwuT+zd1bQeFW0vMhilSwnyfrX+6CRVNRFAWj6jSYfS8+EV+S2qg1B056cvDkORlqEaKc69cihBYp66mz8m/shsrLtidwoAGwYEQHwkL83FxD4U7S8yGKnNWkoSiZwUVM1Y4k1O8Lhs7N+9/ChJ3FO08QGZPI7uhEos+murm2Qohicf4sNf4aA8Dnjl7sMeqiyD2HuEB6PkSx6jhxNQH0YKX1DxorR3hc+53PUvrQa8p6VxkZ9xWi/NgVncCE3/fxqffX+J8/xVFqsKr6w7zTuj7zthwnNiGNAG+Lu6sp3EyCD1GsJvdvzgs/7uRt2wN8YPmMUaafWa7fwCEjBJOq8P6917u7ikKIIvTL9hiUI2vxt8wDIPjBr/gptD2KojCwdW0yHDpWk+bmWgp3k+BDFKt+LUJoUM2bXlN0+jg20EnbxSTzF9ybMZYFI26WcV8hyoHos6mcTbGhGwZLth/kR9OXAJy+9kFiLWFUSThPzSpeKIoigYcAJPgQJUbhFfvjLFNHc4N6gIe1pcDN7q6UEKIIOJfVArxmmkNt0ylijAC67+hEyo7MIVYZXhWXkgmnotg5l98GhtRjb9iLALxo/oHkE/sZ8MUmdkUnuLeCQoir8nCHugDcoOznEW0pAGNsj7pSqDvPC+EkwYcods7ltwtHdKDNPc9h1OuMBxnUWPsCmw6f4pftMe6uohDiCjl0g6WRcXiQzvvmz1AVgx/tN7NGb+4qszQyDoduuK+SotSR4EOUCOfy2+iE8+y/8R0cJi/qnNvJg9oKWXorRBm2OSqe2MQ0RpvmEar+R6zhz1v2wVnKxCamsTkq3k01FKWRzPkQJco5NvyA1p+3zTN4yTSXNanN6TUlw1VGxoaFKBscusEv26NprfzDI6bM4ZaXbY+TRKUcZU8mp5V09UQpJj0fokQ5Uy5/7+jKRkcTvJR03jN/hoouKZeFKEOWRsbSceIqft12kPfMnwMwx96FtXruy+cP/HeOjYfOyPCLACT4ECWsX4sQFozogIHKi/ahnDM8aK3u5zHtNxaM6EC/FiHurqIQ4jK+33SEJ2dtJzYxjZdMc6mjniTGCOAd+6A8nzN19UEGfLmJjhNXsTQytgRrK0ojCT6E28QYVXnzwtjw86Yfscbvd3ONhBCX49ANxizYA0A7dQ8PmZYDMNr2BOfwuuzz4xLTGDZruwQgFZwEH6LE7Y5OQFXAAH5wdGalowVWxU7g8qfYffSkLL8VohRzThz1JYUPzNMBmGXvyl96swI93znoMm7xXhmCqcAk+BAlamlkLK/Oj+TiZ47CK7bHiTe8qZK8jyM/j2Xj4TOy/FaIUso5cfRN8wxqKPFE6dUZn89wS24MZAVMRSfBhygxDt1g3OK9ZL/XOUVlxtgeBeD2xDm0UA7I8lshShmHbrDx0BkO/JdMb3UD/bQN2A2VZ20jSMXjiq4pK2AqLllqK0qMMx9AbpbobZjv6MCd2l98aJ5Gr5QJsvOtEKXE0shYxi3eS2xiGsGcYan1awA+dfQjwmiQo7x/JTPxKbbLXreaz5UFLaLsk54PUWIud5cz1jaEE4Y/oep/vG76FkCW3wrhJs6ejjcX73GtbFHQec/8GX5KKhF6PabY++V4ngK83TeMYD8PlDyurQDBfh60DvUvxp9AlGbS8yFKzOXucpLw5tmMEcyxvM39pjWs0Ztzc99HURTYeOgMrUP90dS8Ps6EEEXl0p6OSz2sLaOjtofzhoVnbSOwZ/sKCfbzYGzvJvQIC0ZVFYbN2o4CWYZanf+Dx/ZuIv+fKzAJPkSJaR3qT7CfB3GJaTnmfTj9bVzLdEdvRpgW8a75S3rOr08sAUDWDzYhRPFYGhnLsFnbc/wfvVY5ykumuQC8Yx9ElHHx/+HILg3o0CAwyw1Cj7Bgpj/QMkcQEyT/jwUSfIgSpKkKY3s3yfVuyKlOgBdxoaPYuSuS69XDfGieziDbq+iorvwA0x9oKR9cQhSDvCaFe5LGFPMUrIqNPxwtmOW4Ncv5htW9aVc/IMf1eoQF061JEJuj4jmZnEY1Hw/pwRSAzPkQJcx5NxTkl3UIJtjPgykDmvPHc51Yuv8sz9hGkGJYaaft5QntVyAzWDGA1xfukfwAQhSDvCaFv276jgbqCf4zKvOibShkm82R35Cqpiq0qx9A3+YhtKsfIIGHAKTnQ7hBfndDGw+d4VRyOqcIJtw+hPfMX/C86Uc26E3ZZdQH4FRyOpuj4nO90xJCXLncJoXfrm5ioGk1uqHwrG048fhmOS8TR8WVkJ4P4Ra53Q05dIO/Dp52lfnR0YnfHK0xKw6mmKfgw8V8H0siY2WTKiGKWPYejBBO8a75KwCmO3qzQQ/L8RyZOCquhAQfolRw7pA5dfXBS45mZj89rleljnqSCeavcM4U+XbjUdmkSogi1jrUn6o+VgA0HHxs+RRfJZUdegM+st+TpWyQr5XPZP6VuEISfAi3c86uz22sOYlKPGV7Cpuh0UvbxEBtVZbzskmVEEVHUxVOJacDMMr0M63Uf0kyPHnKNjLLsto5j7flr5e7SuAhrpgEH8Kt8ppdf6kIowET7fcDMNb0LdcqR13nZJMqIYrW5P7N6azu5CnTAgDG2B4l2qgGgKpknpeJo+JqSfAh3Cq/lOuX+p+jJ384WmBVbEw1f4IXF5/j3KTqm7+iJAAR4ir1qwdf+XwBZO5Wu1hv7zq3aGRH+rUIcVfVRDkiwYdwq4JuLGWg8oJtKLGGP/XVWN42f032TCFv/faPzAER4grtik7ggc/XkzL7QUzpZ4nU6/KWYzAAinRyiCImwYdwq8JsLHUWX57KGIndULlLW59j/gfIHBAhroRDN/h09SE6Hp9Opf+2olt8eM38IteEVOWdO8NoFuJHVW8rAd4Wd1dVlBOS50O41eVSritAdV8roBCXlMZWozGT7P151TyHsaaZROp1Xfk/ILMvRCFzDki3JkEyLi1EPqLPpvLbrli+/PMw16du5ElLZkK/l+1D6daxDX2a16CWfyUGtq5NhkPHatLcXGNRXkjPh3ArZ8p1yJ4z8eLj8D5NCe/TxHX8C0cvljlaYVXsTLN8TGWSszzPOQdkc1R88VVciHKg48TVTFiyD8+U43xg/gyAr+09+CG1Je8t/5ebJq0BQFEUCTxEkZLgQ7hdXinXg/w8XPu49AgLZsKdYWR2ZCi8YHuSKL06NZXTfGz+FBU9x3ULOp9EiIrIoRtU9jTjQTqfmydTWUlhh96ACfaBrjKVPc0yiVsUCxl2EaVCQTagGtCmDn1bhDB701He/n0fw2zPMt/yBp20XTxt/MLkbEmQCjOfRIiKZnNUPAnnM/jI/BVN1KOcMnwZlvEMtku+FhLO22QrA1EspOdDlBoF2YDKy2Li4Y71ANhn1OYV22MAPK3Np7O6I0tZ2W9CiNztik7gjYWRPKQt407tL+yGysiMZ4gjZ5AhPYiiOEjwIcocTVV4uENdABboHfnOfiuqYvCJ+VNClcxVLg93qCuTTYXIwy/bY6hyagtjTN8DMN4+iL+Na3MtKz2IojjIsIsok8b2bkqwrwfjl+zjTfuDNFaPcaP6L1+aP+DPTnN4uGtTd1dRiFIl+mwqZ1NsKApsitjNd5ZPMCsOFjja87WjR47yCpnzrqQHURQHCT5EmdW+QSAAdkwMy3iWRdYxNFBPUO3om+w6/hUTlvzLK7c35rqald1bUSFKgY4TVwNgJYN5lklUVRP5R3cOXWbtJXQ+kh1rRXGRYRdRZgV4W6jqbaVZTT+evbMD71d+gzQs+B77g4Tfwtl4+Ay/bI9xdzWFKBUm92+OSYWJ5i9orh7irOHNUNuznCfnsMqlK82EKA7S8yHKrGA/T9a/3AWLpqIoCjc37E9shE7ouue4OW4mt6t+LN5p4Z4bamIYUKWSmZpVvNxdbSGKjUM3cl0xtis6gXlbjvNZ3XXcemIDNkNjuO0ZjhnVAXi5xzVcX6tKnivNhChqhe75WLduHb1796ZGjRooisKCBQuynDcMg/DwcGrUqIGnpyedO3dmz549RVVfIbKwmjSUCxtP3DRpNV2WB/Gl/XYAPjB/Ro3UffSasp7eU9e7up2FKI+WRsbSceIqBny5iWfmRjDgy02uvY5+2R5DpSPLuSXmcwDG2R9ko35xXtS7S/eTeD4j35VmQhSlQgcfKSkpXH/99UydOjXX85MmTeLDDz9k6tSpbNmyhaCgILp160ZycnKu5YUoKpndygrv2gew1nEdnkoGX1o+oDrxmFSFyf2bu7uKQhSLpZGxDJu1PccO0bGJaTw5azvbt/zFZPOnqIrBt/ZuzHJ0y3GNcYv3SkIxUWIKHXz07NmTt99+m7vuuivHOcMwmDx5MmPGjOGuu+4iLCyMmTNnkpqayuzZs4ukwkLkpV+LEBaM6IADjZG2p/lXDyFIOcv/LO+z6Inm1KtaiQFfbGJXdIK7qypEkXHoBuMW7811bySAABKZqkzCW0njL0dT3rQPzrWcbEkgSlKRzvmIiooiLi6O7t27u45ZrVY6derEhg0bGDp0aI7npKenk56e7nqclJQEgM1mw2azFWX1XNcr6uuWR2W1rex2OwDnFC8esb3IAssbhKlHSFz1NB9UeY2Nh8/w09bjNKrqxbajZzl9Lp1Abys31KlyxV3NZbWt3EHaqnAK0l6bo+KJP3cea7atVxw6aEYGX1k+oLZ6iiN6dYbbnsGOCTAwKaBlu/08mZiCzeZbxD9FyZD3VsEVV1sV5nqKYRhX3M+mKArz58+nX79+AGzYsIEOHToQExNDjRo1XOWeeOIJjh49yrJly3JcIzw8nHHjxuU4Pnv2bLy8ZHKgKJyEdHh/t0ZlC7SrrnP2xCGmGu9gVWzM0O9gXMYgvE0GT17rAKCSGfytbq60EEXo2DlYdFSlT207PeM+pcG5LSQYlbgrYxyHjczP5Rea2anl7eaKinInNTWVgQMHkpiYiK9v/kFssax2cU4AdDIMI8cxp1deeYXnnnvO9TgpKYlatWrRvXv3y1a+sGw2GytWrKBbt26YzeYivXZ5U5bb6u7eOhZNQVEUGr6+nBfVoXximcrD6m/8qwUxx34L7++++Na3aoYrr8FH/Ztz67XVC/V6ZbmtSpq0VeHk114r//mPd5fsIy4p6zwPuw4OQ8Fj3480MG0h3TDxRMazFwIPA1D4eI/GpR19ClDd14Nlo24us5NN5b1VcMXVVs6Ri4Io0uAjKCgIgLi4OIKDL64PP3nyJNWr5/6BbrVasVpz3nqazeZiewMV57XLm7LYVqp2cbnhiC4N+HytQqgtlmfNP/O26Wv+M6qwSm/pKp/uyPywVYA3f9tP97CQK/oALott5S7SVoWTvb2WRsYyfPbOC/M8cr5XH9BW8LjpNwBetA1ls9HkwpnMsjb94nOc/3rljqZ4WC1FX/kSJu+tgivqtirMtYo0yVhoaChBQUGsWLHCdSwjI4O1a9fSvn37onwpIXKVfbnhp6sPUtnLzMeOu/jB3glNMfjU/AnNlYM5nmsgk+5E6Xe5CaZd1B2MM30DwHu2+1ikd8j3epJQTLhDoXs+zp07x8GDFz+4o6KiiIiIwN/fn9q1azNq1CjGjx9Pw4YNadiwIePHj8fLy4uBAwcWacWFyM653DD7h/LpcxmAwqv2R6mmJNBZ28n/LO9xd0Y4R4ycH7iyi6cozTZHxedYUuvUQjnANPPHaIrBPHtnPnX0zfM6lT3NfDqoJW3rSV4PUfIK3fOxdetWWrRoQYsWLQB47rnnaNGiBW+88QYAo0ePZtSoUQwfPpxWrVoRExPD8uXL8fHxKdqaC3GJy90NAhiqieG2Z9ilhxKgJDPTPJFAEnOUk108RWmWV3BcX4nha8t7eCoZrHFczxj7I+Q2JKNc+PPu3c3o0CBQAg/hFoUOPjp37oxhGDn+fPPNN0DmZNPw8HBiY2NJS0tj7dq1hIWFFXW9hcgiv7tBJ4cOqXjwSMZojurVqKOe5GvLJCpxPks52cVTlGa5BcfBnOE7ywSqKOfYoTdgmO0ZHBc6tgO9s87jkGEWURrI3i6iXCjoUEmnhoGsPQBDbC/xsyWc69QovjR/wMO20aRj4eEOdeVOUJRqrUP9qepj5VRyZn6kyiTzreVdaijxHNRr8HDGi2hWb8ICKxGXmMb8Ee05Hn9e9m0RpYoEH6JcKOhQyZOdG9ChQSDjl8BDGS8x2/IO7bW9TGUKUV0+xcvLi24frmVI+7oMaF1bPqRFqaOpiivw8CSNry3v0VCNIdbw58GMl0nAB9LtLBrZgQyHjtWkyYaKotQp0tUuQrhL61B/gv08chnhzqQAwX6Zd33tGwQCEGnU47GMF0gzzHTTtlHzz9G8vmAXB06e47UFka5NuYQobSb3b46HksEX5g9pqR4kwajEgxkvc4JAVCXzvKIoWE3a5S8mhBtI8CHKBU1VGNs7M5dB9gDE+Xhs7yZoqkKAt4Wq3laa1fSjT797ed3yIjZD43ZjHeGmmXBh2qpzU67vNx0pqR9DiHztik5gwBebqO9vZnPD77hJi+Sc4cHDGaM5YNQEYNHIjvRrEeLmmgqRvzI77OJwOAqdl95ms2EymUhLS8PhcBRTzcqH4mors9mMphXP3ViPsGCmP9CScYv3Zpl8GuTnwet3XIufp4WFETFU8/Fg7ejOeJo1dAPGzA8jXR3GZPOnDDGtIAVPJtn74wxbxizYw/2t68gQjHC7X7bHsPnwSdT5E/FNWEWaYeYx24vsMBqiKHDlm2UIUbLKXPBhGAZxcXEkJCRc0XODgoI4fvx4nuneRabibKvKlSsTFBRULL+DHmHBdGsS5MpwWs3Hg7MpGbz1W9aAJNjPg7G9m+DnmbkSYJHeHh97Ku+Yv2a4aRE2ND6y3+sqvzkqnnb1A3DoRpZry8oYUdzi0yEyJgmz2cSvEdFMMn9B04Q/0VUzzxkvklCtNe+0q8O8LceJTUgjwLvsZykV5V+ZCz6cgUe1atXw8vIq1BeYruucO3cOb29vVFVGnPJTHG1lGAapqamcPHkSIEsK/qKkqQrt6gcAmYnHRszOmXgsLjGNYbO280iHuq5j3ztuxYKNsebveMY0H91Q+dhxNwBbj8aTeD4jR69KsJ8Hb9xxTbH8HEIAjNtugu2bUNB52zSDu01/YjdUhqeNZLkeBueTGdSmDgNb13ZNMBWitCtTwYfD4XAFHgEBAYV+vq7rZGRk4OHhIcHHZRRXW3l6egKZ+/1Uq1at2IZgIP/EY5nba8H8iJgsx2c4eqKi87r5e541/4wDlamOO/njn5N8uPzfHNeKTUxj1LwIJrXO7B1p26CaDM+IIuHsZRvcwMHsQwpvaf9joGk1uqHwnG04y/UbURX48L7mADLBVJQpZSr4cM7x8PKSZWNlmfP3Z7PZijX4uFziMQOIT7HhX8lCfEqG6/j/HHdgQucV8xxeMP+Ijsr0433zzZ4K8MjMLfh7ezK2dxNJ4CSuytLIWMYt3svp5POEVjKYYPof92lrcBgKz9uGsUjP3CtLN8DDLDdSouwpk+9ama9RtpXU76+gicf6Na+RY4XM547eTLL1B2C0eR4jtV/gsuHHxeEcWaIrCsOhG2w8dIaFETF8vPIAw2ZtzwycDZ3Hz3/lCjyetQ1ngd4xy3PHLd6LQ5eZpqJsKVM9H0IURkETj3VrEkTrUP8c8zmmOfoCBqPNP/C8+Se8lHQm2u8nt/0ynJzDOeMW76VbkyAZghGX5ezlyN5Lp6Lzrulz7tbWYzdURtlG8KveLsfznTsxO+c5CVEWSPAhyi1n4rG4xLRc+ywUMpfhOtNNd2sSxKZDZxgxezsJ5zOH+KY5+pGGlTfM3zHMtBhP0hlnfxAjn05DA/lCEBfltkLKGZTmtROzBRsfmqfRS/sbu6HytG0kv+tt83wN2YlZlDVlctilLHrooYdQFAVFUTCbzVSvXp1u3brx9ddfo+t6ga/zzTffULly5eKraDlSmMRjzvKqqrgCD6evHT151fYouqHwkGk575q+QuXi70w3MpdDZidfCGJpZCwdJ65iwJebeGZuBAO+3OTKnJvXhGhP0vjK/D69tL/JMDRG2J7ON/AA2YlZlD0VNvi4dIx146EzJTJm2qNHD2JjYzly5AhLliyhS5cuPPPMM/Tq1Qu73V7sr18ROROPBfll/XDOa2fPvAKG2Y6uPG97Eoeh0N+0hk/MU7GQGaTYdCVzOWQ28oVQsTl7NbIPpzjnBU1ddSDHOV/O8Z3lXW7WdpNqWHnENppleus8X+PSbQOEKEsq5LDL0sg43vrtn1yTThXnKgWr1UpQUBAAISEhtGzZkrZt29K1a1e++eYbHnvsMT788ENmzJjB4cOH8ff3p3fv3kyaNAlvb2/WrFnDww8/DFyctDl27FjCw8OZNWsWkydPZv/+/VSqVIlbbrmFyZMnU61atWL7ecqK3BKP5bWzZ34Bw3z9Js7brHxinkIvbRNVSGao7VnO4Ul1z4u9H5cO54iKqSDLvGf8dSTL8UAS+c4ygWvVYyQaXjycMZrtRsNsz7oot947IcqKCtfz8cf+M4yYvSPPu5GSXqVwyy23cP311/PLL78AoKoqn3zyCZGRkcycOZNVq1YxevRoANq3b8/kyZPx9fUlNjaW2NhYXnjhBQAyMjJ466232LlzJwsWLCAqKoqHHnqoRH+W0syZeKxv8xDa1Q/I88P6chvULdVb84htNOcMDzpoe5hreZtqJPDfeQXdkC+EiiavHtSCLPO+dHivjhLHT5ZwrlWPccrwo3/GG2w3GgEK99Vz5Pp+zKv3ToiyoEL1fDh0g0krD+d7N+KOVQqNGzdm165dAIwaNcp1PDQ0lLfeeothw4Yxbdo0LBYLfn5+KIri6kFxeuSRR1z/rlevHp988gmtW7d2ZSkVBeOcJzJs1nYUcl9c+5fejPszXmOGZRJh6hF+soTzoO1ljhhBBHpbeKxjKBl2w7WPzA11qrDt6NnL9rqIsiW3VSrOHtR0e8HmcSkKtOBfvrR8QICSzHG9Kg/YXuGokfn/W8OgQ3WDBUegSiULA9vUoW5gJXkfiTKvQgUfW47E819yRp7n3bVKwTAM1zDK6tWrGT9+PHv37iUpKQm73U5aWhopKSlUqlQpz2vs2LGD8PBwIiIiiI+Pd01iPXbsGE2aNCmRn6O8yGuDumo+VtJsDuoGVmJXdD3uzhjHd+YJ1FFP8pNlHI9lvEDEuQa8u3R/luupSuakVKeSGOITxSuvVSrOHtRRtzbM9XnZdVe28LF5Kh6KjV16KI9mvMgpKrvOP9O1PqT+y4yHbpTsuaJcqVDDLieTc1mSkGu5kl2l8M8//xAaGsrRo0e5/fbbCQsL4+eff2bbtm18+umnAPnu4JuSkkL37t3x9vZm1qxZbNmyhfnz5wOZwzGi8HqEBbP+pVuY83hbPr6/OXMeb8vGV7qy5bVbWTiiA5P7N+eEEsTdGePYrdclUElinuUteqkbc1wr+1xmSURWtl1uPgfAnM3HCPLNe/gO4GHTUqabJ+Oh2FjpaMH9Ga9nCTxe7dmYYZ0bAEgvhyh3KlTwUc3HWsByJbdKYdWqVezevZu7776brVu3Yrfb+eCDD2jbti2NGjXixIkTWcpbLJYcW9zv27eP06dP8+6773LTTTfRuHFj1+Zt4srlNk/EatJQFIV+LUJYMKIDp8kcn1/paIFVsTHVMoWnLpMN1XlGMlOWTQWZzxGXlM6A1rWBnMu8TdgJN33DWNO3qIrBLHtXhtqeI5XMz516gZUI9LbSu3mNYvoJhHC/ChV83FjXn+o+ljzvRop72Vp6ejpxcXHExMSwfft2xo8fT9++fenVqxcPPvgg9evXx263M2XKFA4fPsx3333HZ599luUadevW5dy5c/zxxx+cPn2a1NRUateujcVicT1v0aJFvPXWW8XyM4iczisePGF7ni/tPQF43vwTH5qnu5bi5ubSIT5RthS0Z7RuoFeOZd5+nGOmeSIPmZYDMNF2P6/ZH8GBhqbA5P7N+eP5Tvz1cheC/TyLpf5ClAYVKvjQVIXRt9YDCpZ0qqgtXbqU4OBg6tatS48ePVi9ejWffPIJCxcuRNM0mjdvzocffsjEiRMJCwvj+++/Z8KECVmu0b59e5588kn69+9P1apVmTRpElWrVuWbb77hxx9/pEmTJrz77ru8//77xfIziIsCvC1U9bYSVsOXe+oZvGMfzCu2R7EbKndp65ljeZtqnM33GpKIrOwpaM/o6eR0ujUJYv1Lt3Bb0yAaKNGs8A6ng7aHFMPKExnPMt3RB+enz8KRHenXIkR2pxUVgmIYRqnq901KSsLPz4/ExER8fX2znEtLSyMqKorQ0FA8PAo/NKLrOklJSWw4luqWPB9libOtfH19UdWijVGv9vdYmqTbHSi6gyVLlvDcJg2HodBB3c0088f4KamcNCozPONpthqNc33+yC4N6NAgsMKM6dtsNn7//Xduv/12zGazu6tzRRy6QceJq/JM23+pQG8LT9xUj71rf+Atx8f4KOc5awnm/uRn2G/UzlL21Z6NeaJT/SzHykN7lRRpq4IrrrbK7/s7uwq12sWpR1gQt4UFFyjplBD5sZo0bLbMlUU1K3tw9Gw6f+nN6JPxNp+bP6Kxepw5lnd40z6Y7xzdyN7nNnX1QaauPkiwnwev33EtVSpZ5T1ZyhVkObZT/Lk00la8yWTTAlBgk34tw5Ke4Sw5P5jHL9lH7QAvuQESFUKFDD7g4mRCIYrKyz0bM2z2TgCOGkHclTGOieYv6K1t4i3zNzRXDzHG9ghp5Jz4HJuYxvDZO7Icy603Lr9NykTJyWs59qUCSORj81Q6ansA+Nbejbfsg7Hl8bEruyGLiqRCzfkQojjdem11JtwZhvN7IxUPnrI9xVu2QdgNlbu1P1loeZ2GSnSBrpd9SW5+m5SJkudcjv36HdfmONdK2cdv1lfpqO0h1bDydMYI3rA/nGfgATIJWVQsEnwIUYQGtKlD5LjbmP1YGwa3rY2mKPzPcQeDba9wyvDjGjWaRZbXuF9bRf4d9lmX5P6+K/9NyiQAcQ9NVUi1XVz6rqLzpLaIuZa3CVLOclCvQZ+Mt1ikdyjwNWUSsqgIJPgQooh5WUy0bxDIW/2asXBkRwA26k3pmf4u6xzN8FQyeNf8FVPNU/AhNd9rOe+GX1sYmW9SK8kZUvzy2sclMiYJgGDO8L15PC+b52JSdBY52tEn420OGjUBuD0sKM9rX0p2QxYVQYWd8yFESVIUOG34McT2Ek/ov/GC6Qd6aZu4XjnEC7Yn+dvI2XV/qfiU0rctQEWS2z4uzpUsW4/Ec7u6iQnmr/BTUkkxrIyzP8gPjs44JxgH+VqZfH8Ltk9aRVxS7pmWZTdkUZFIz4cQxciZC6RZiB9v9Q1DVVQ+d/Tmvow3OK5XpZZ6innWt3jd9B1Wri4VvnTXFw/nPi7Zh7xOn8tg6pJtvJLxCdMsn+CnpBKh1+OOjPH84OjCpSubwvs0xWJSCe/TFAX35BkSojSR4EOIYhTs58n6l7uwcEQHBrerw9SBLQDYYTSkZ8YEZtu7APCoaQm/W16huXLwil9LuuuLXn77uHRVt7HcOpp7tHU4DIUp9n7ckxHOEePi6iRVgQl3hrlWLDlXyVya9RQyezymP9BSltmKCkOGXYQoZpdmq+zZrAav9jzP+CX7OIcXr9ofZ7l+IxPNX1BfjeVny1i+ctzOZPvdnKfgwURlTzO6YeDQDblzvgJ5LWHObR+XKiQRbv6WvtoGAA7rQYy2PeFKJPdOvzC8PUxU9bbSvHZlvCxZP2Z7hAXTrUmQLJkWFZoEH8Llm2++YdSoUSQkJLi7KuVa7+Y1+OLPKHw9TLSqW4XV+6x0P9eAceZv6KdtYKjpN+7Q/uYN20Os0lsW6JoJ520M+upvydR7BXKbz+FsxwMnz11S0qCPuoE3zN8RqCThMBS+dPTiI/vdpGNBUcAw4PpalQkL8cv3NSXPkKjoZNilhB0/fpxHH32UGjVqYLFYqFOnDs888wxnzpxxlencuTOjRo3K8xqrV6+mS5cu+Pv74+XlRcOGDRkyZAh2u/2yr79mzRoURcnx57XXXqN///78+++/rrLvvvsuLVsW7MtPFFywnyd/vdyFP57vxKR7rmfzmK7MGN6NUbaRPJzxItFGIDWV03xteZ9p5slUp+B5H2Tpbd6rUnKT13wOZzv+8c9/ADRSjjPX8jafWD4lUElin16LOzPe5F37ANKx8GjHUJqF+FHV20qAt6VYfz4hyoMK3fOxKzqBCb/v45XbG3NdzcrF/nqHDx+mXbt2NGrUiDlz5hAaGsqePXt48cUXWbJkCZs2bcLfP/+Z7nv27KFnz548/fTTTJkyBU9PTw4cOMBPP/2ErusFrsv+/fuz5N739vbG09MTT0/ZSbMkXDoUoygKFi3z8Rq9Bd3Sr+UZ0y88pv3O7dpmblZ3Mc3el/85epJO5hebqkBu36kGFTtTZn69GNl7g/Kbz+E8dig6ltdMP/OQtgyTonPesDDV3o8vHL1cCcOCfK28evu1qApkOHTZFE6IAqjQPR+/bI9h4+Ez/LI9pkReb8SIEVgsFpYvX06nTp2oXbs2PXv2ZOXKlcTExDBmzJjLXmPFihUEBwczadIkwsLCqF+/Pj169OCrr77CYin4HVe1atUICgpy/fH29uabb76hcuXKQOYQzMSJE9m5c6erd+Sbb765wp9cXI5zVUxoYCXO48G79oH0zniH7XoDvJU0RpvnsdLyIneomwAj18DD6UoyZRamt6C0ulwvxqW9QQ7d4Ju/ovJMja7h4H5tFX9YXuAx0xJMis5Sx43cmv4enzr6ZclUGt6nKZqqyG60QhRChev5OJGYxtFkA01VWbzzBACLd57gnhtqYhhQpZKZmlW8ivx14+PjWbZsGe+8806O3oWgoCAGDRrEvHnzmDZtWr7XCQoKIjY2lnXr1nHzzTcXeT2d+vfvz44dO1i9ejUrV64EwM8v/3FsceWcq2I0ReGmSauJTUzjH6MOd2eE00fdwMvmudRST/Gp5RMe0hsx3jaIHUbDfK9Z0KW3hektKK0u14txaW/Qir1x+ezJYtBN3cZo0zwaqpk3JVF6dcLtD7FWvz5LSVXJnFxaVtpIiNKkwgUft0/f5vq3s0M6PiWDXlPWu44fefeOIn/dAwcOYBgG116bezKpa6+9lrNnz3Lq1Kl8r3PvvfeybNkyOnXqRFBQEG3btqVr1648+OCDl93C+FI1a9bM8vjo0aNZHnt6elKpUiVMJhNBQQXLzCiujvOueWzvJjw5azsABioL9Y4sT2/FE9pvDDX9yo3qv8y3jmWVozkf2u8h0qiX6/UKsvTW2VuQ/Uvb2VtQVpZ/5rYq5VLO3qCpqw4yeeW/uQYprZR9vGSey41q5ryneMObKfY7+d5xKxlkbjs+4MaatK0fmOdKFiFEwVS4YZd3ejfEdGEc3PkB5PzbpCpM7t/cHdXCMDJroSj5j9FrmsaMGTOIjo5m0qRJ1KhRg3feeYemTZsSG1vwSYZ//vknERERrj9VqlS5qvqLotMjLDjLBnUA5/HgY8fddEn/gHn2ztgNlVu0CH61vsbn5g9prBzLco3KXmY+/uNfdkUn5Pk6BZnzUBRp2y8d0imuTdMK2ssz46+obD+vQTt1D3PMb/OT9U1uVP8lzTDzqb0PndInM8PR0xV4AMzZEo3VpNK+QaAEHkJchQr3v+eOptVoVrsqfT7dkOPcghEdLrtE7ko1aNAARVHYu3cv/fr1y3F+3759VKlShcDAwAJdLyQkhMGDBzN48GDefvttGjVqxGeffca4ceMK9PzQ0FDX/A5R+gxoU4e+LUKIOJbAqXPpVPW2YjGp3PPZRl6yP8F0R2+eNs2nn/oXt2lbuU3byh+OFnxh78XfRmMSUm1sOhzPL9tjcp1Mfbk5D1A0aduzD+lYNYNJrWHlP//R87qal3l2wRU0wVrCeduFfxl0UnfxlGk+rS70dGQYGj85buYT+13EkffPW1En8wpRlCpc8HEp57p859/FKSAggG7dujFt2jSeffbZLPM+4uLi+P7773nwwQcv2/ORmypVqhAcHExKSkpRVhmz2YzD4bh8QVEsnBvUOUXGJAKZ79cjRjDP2YYzTenDKNPP3K5upqu2g67aDnbq9Zip9GVB+g2u+Uz/xiUz6++jhPdpyomE8/nMecgpv16FvJJzQd5DOgDPzotAUbUiG9JpHepPsJ8HcYlp+e4V7EE6d2rreUhbxjVqNADphpm5js58bu/NCS4f/Ms+OkJcvQoZfDhXFgRX9qD/jbWYt+U4sQlpxb4+f+rUqbRv357bbruNt99+O8tS25CQEN555x1X2VOnThEREZHl+UFBQSxcuJCIiAjuvPNO6tevT1paGt9++y179uxhypQpRVrf2rVrExUVRUREBDVr1sTHxwer1VqkryEKLvv7dsZfRzh4siYjbc8QqsTymPY792jruF49zId8xPPWAOaldeHRKXH8R+YS7o9W/Mua/afy/YLO7sB/59h46EyOLJz5TVTt1iQozyEdp6LsQdBUhbG9mzBs1nYUyPG6IZxisGkl92urqKxkBukphpU5jlv43N6LUxRu2FH20RHi6lTI4MO5ssCiqSiKwsDWtUtkfX7Dhg3ZunUr4eHh9O/fnzNnzhAUFES/fv0YO3Zslhwfs2fPZvbs2VmeP3bsWPr27cv69et58sknOXHiBN7e3jRt2pQFCxbQqVOnIq1vnz59WLp0KV26dCEhIYEZM2bw0EMPFelriILL/r69LsSP3lP/QlEgyghmjP1RPrTfw4Om5QzWVhCinOE58088bfqFVXoLZjtuYf2/12FQuPf51NUHmbr6YJYVMJebqDrq1oYFGtL5aMW/dGgQeFXpxS/N1zP9gZaugMiDdHqoW7hXW0sHbY+r/DG9KjMdt/GDozPJZF3Z5mVRSc24fL4c2UdHiKtTIYMPyJnkqaTW59epU4cZM2bkW2bNmjX5nv/uu++u+PU7d+7smtya3UMPPZQluLBarfz444+oaoWbl1xqXfo+DfSxZukJ+WLdYY6egY/s9zLN3pee6mYGmFbRRt1Hd20b3bVtnDZ8+d3RhsWOdmw1GmEUYs65M7D4dGAL3vrtn3wnqn7x5+ECXTO3wCY/uSUGdObr+XbDUYa0DWFWl/NELP0f3Y2N+CjnXc9d72jKTMdt/KG3RM/j537/nut589e9su29EMWswgYfQpR1l/aE6AZM+eOA61w6FhboHVmQ0ZEGSjQDtVX01f4iUEniQdMKHjSt4IThzzLHjazSW/C3fm2WVR25cebLeG1hJPEptnzLpqQXbq5QXkt7swcbWQKN9gqKAssijtBV3Ubb3Z9Ra882Kisp1Cezssf0qvzk6MQv+k1EG1XzfH1nzo7br6uBqioMcy11vki2vRei6EjwUc707NmTP//8M9dzr776Kq+++moJ10gUJ2dPyObDZ/K8Wz9o1ORN+4O8Yx9EBzWSPtpGuqtbqKHE87BpGQ+zjBTDygY9jDX69WzSr+WQUYOLX7cXGXDZwCM/ea3adR5+es4O5g31oEXtKjh0g09XH2Lj4TNM+H0fL/dszOKdJ1DQ+WfHn/y6czod1EhWq/vwsFys02nDl+WOG1jo6Mhm45p8e3dGdmlA+/oBWXJ2OLe9zz6fJaiMJV4TojST4KOc+eqrrzh//nyu5y63b4wouwoyAdKBxjr9etbp12PlEW5Wd9FV3U4XLYLqSgLdtG100zKT8J0xfNimN2KLfg0RegP2GbVzzI+4EpdLGZLhMJi84gDtGwTw5Z+HOX0ug8okYzmykz8++5JJ6iFaWA/gr5zL8jxnL84SR2u2GtfkOaySXcPq3llWFDnJtvdCFC8JPsqZkJAQd1dBuEFhJ0CmY2GF3ooVeiuwGzRVjtJZjeBmbRfXK4cIUJJd80Scoo1A/tFrs9+oxVGjOtFGVWKMQGKNgCx7neTHYcCWU+DINqfTkzSqKQnUUM5Q6/AJzFGxfKCcoL71BDWV0zmuk2x4skm/lr/0MNbrYRw0Qsitp+Zy8ms32fZeiOIjwYcQ5UBB81zkTmGPUZc9jrp86uiHGTvNlMO0Uvdzo/ovTdQjhChnqKmcpqZ2mm5sz/Js3VA4gy+JRiUSqUSC4U0ilbAZJhwoONBwoKJi4KWk43k0ja6mdLyUdAJJpKqSgK+Se2+d0yE9mJ1GfXbp9dip12e3EYo9n48vk6rgZdFISrPn8RPLxFEh3EmCDyHKgcvlucjLyC4NCPbzYMyCSFeyPTsmthuN2O5oxBcX5o36co7GynEaq8dopERnBiLKKWoqp/BQbFQlkapK4lX9DKmGlTijClFGMIeMGhw2gjmk12C/UZMkvAt1Lbtu8GjHeny08t8c52TiqBDuJ8GHEOVEXhMlc+O883+2WyNOJqdRdWXWpHuHTp4jJePiipUkvNlsXMtmR/aNEQ0CSaKqkoCfkoIf5y78nYIZOxo6mqKjomOgcN6wkoqV81hJNTyIx4eTRmVOGlVIxpMrGTrJS91ALz6TiaNClEoSfAhRjlw6UXLF3ji+/utIjp6Q7Hf+uSXdOxafyt3TN+LrYaJV3SpsPXKW+JQMEs7bsm1HoHAGP04bfijG5XtcSmIrA6dqPh60qx8gE0eFKIUk+BCinHFOlGxXP4DWof4FuvPPnnSvTkAl/rokIDEMg2PxqdwzfaOrh+S7jUfZ/18yjYN8eKBtHeZtOc7hU5k9JtkDjJDKngzvUp+5m49x+L9ErBYLlb0stKpbhU2HznA84TyNg3wY2LoOb/66B5vjyiOU7PM5ZOKoEKWPBB+liKIozJ8/P9ddbwGOHDlCaGgoO3bsoHnz5iVat9zUrVuXUaNGMWrUKHdXReThapaM5haQZO8hOZdux9tqyrJNgUM3suzG26SGL36eZhRF4d4WwSz+bQm39ehEJQ+LK7BJPG9j74kkTp1LZ0TnBkz+40Ch5q646nnhb5nPIUTpJsFHCXnooYeYOXMmAJqmUaNGDe644w7Gjx9PlSqZm1rFxsa6/l1a3HLLLaxduzbHcZvNxpYtW6hUqZLr2OWCJ+EeRXnnnz0g8fEwZ3nsPJ9b7gxnGZMKVpPq2sF52Z64HL0zlb0yr5uQWriEZjKfQ4iyQYKPEtSjRw9mzJiB3W5n7969PPLIIyQkJDBnzhwgc9fa0ujxxx/nzTffzHLMZDJRtWre6aqFKIi8NqhLTLVhAM90bcDMDUdJOJ93EOJfyczrvZoS5CvzOYQoK2THsBJktVoJCgqiZs2adO/enf79+7N8+XLXeUVRWLBggevx5s2badGiBR4eHrRq1YodO3bkuOaiRYto2LAhnp6edOnShZkzZ6IoCgkJCa4yGzZs4Oabb8bT05NatWrx9NNPk5KSUuB6e3l5ERQUlOUPZA67TJ482fVvgDvvvBNFUVyPhciLQzcYt3hvnhvUKcAPW6MZf2czFHKug3EeG39nM+5sEUK7+gESeAhRRpT94MMwICOl4H9sqYUrn9+fq5i2f/jwYZYuXYrZnPtmXikpKfTq1YtrrrmGbdu2ER4ezgsvvJClzJEjR7jnnnvo168fERERDB06lDFjxmQps3v3bm677Tbuuusudu3axbx581i/fj0jR4684rrnZsuWLQDMmDGD2NhY12NRvjh0g42HzrAwIoaNh87guFy+9HxsjorPd0mwAcQmplGlkoXpD7QkyC9rNtIgP48cG9EJIcqGYht2mTZtGu+99x6xsbE0bdqUyZMnc9NNNxX9C9lSYXyNAhVVgcpF+dqvngBLpcuXu+DXX3/F29sbh8NBWlrmh+6HH36Ya9nvv/8eh8PB119/jZeXF02bNiU6Opphw4a5ynz22Wdcc801vPfeewBcc801REZG8s4777jKvPfeewwcONA1KbRhw4Z88skndOrUienTp+Phcfm03NOmTeOrr75yPR46dCgffPBBljLOIZjKlSuX2uEjcXWWRsbmmJsRfBVzLAqyH42zXN/mIbJkVohypFiCj3nz5jFq1CimTZtGhw4d+Pzzz+nZsyd79+6ldu3axfGSZUKXLl2YPn06qampfPXVV/z777889dRTuZb9559/uP766/HyuriZV7t27bKU2b9/PzfeeGOWY61bt87yeNu2bRw8eJDvv//edcwwDHRdJyoqimuvzZ40KqdBgwZl6VGpXLnyZZ8jype85mbEJaYxbNb2K+qBKOh+NM5ysmRWiPKjWIKPDz/8kEcffZTHHnsMgMmTJ7Ns2TKmT5/OhAkTivbFzF6ZPRAFoOs6ScnJ+Pr4oKpFMOJkLtwun5UqVaJBgwYAfPLJJ3Tp0oVx48bx1ltv5ShrFGBIxzAM14qBvJ6n6zpDhw7l6aefzvH8ggaCfn5+rnqLiqcgczPGLd5LtyZBheqJuNx+NLL/ihDlV5EHHxkZGWzbto2XX345y/Hu3buzYcOGHOXT09NJT093PU5KSgIyl3LabFlnuNtsNtddu65fsi2mybNAdTMMA8wODLMXulIE3bWGUeB5H4ZhuOru9Prrr3PHHXcwdOhQatTIHDpy/myNGzfmu+++IyUlBU/PzJ/P2X7OMtdccw1LlizJck3nXAtnmRYtWrBnzx7q1auXa72ytGO2+uZV7+zlnOfMZjM2my3Pspe+pmEY2Gw2NE3Lt2xZ4HyfZn+/lhebo+KJP3ceaz6/qvhz59l08ORlA4VL28oMvHHHNTw7LwLIPQvrG3dcg+6wozuokMr7e6soSVsVXHG1VWGuV+TBx+nTp3E4HFSvXj3L8erVqxMXF5ej/IQJExg3blyO48uXL88y5ACZyzuDgoI4d+4cGRkZV1zH5OTkK37ulbLZbNjtdldwBdCyZUsaN27MuHHjXPM2zp8/T1JSEr169eK1115jyJAhvPDCCxw7doz3338fyJyMmpSUxMCBA/noo4949tlnGTx4MLt372bGjBmun1FVVYYPH0737t154oknGDJkCF5eXuzfv581a9YwadKky9bb4XCQkZGRpd5Ouq6TlpbmOle7dm2WLl3Kddddh9VqzXN4JiMjg/Pnz7Nu3Trs9tx3HS2LVqxY4e4qFJtJrS9f5vQ/m/j9n4Jd79K2mpjPtTOitvF7VMGuWZ6V5/dWUZO2KriibqvU1NQCly22Cae5DQdkPwbwyiuv8Nxzz7keJyUlUatWLbp3746vr2+WsmlpaRw/fhxvb+8CTZTMzjAMkpOT8fHxybUuxclsNmMymXL8TM8//zyPPvoor732GgCenp74+vri6+vLokWLGD58OJ06daJJkyZMnDiRe++9l0qVKuHr60uzZs344YcfePHFF/n8889p164dY8aMYcSIEVStWhUPDw/at2/P6tWree2117j99tsxDIP69etz33335ajLpZxtpWkaFosl17KqquLh4eE698EHH/DCCy/w7bffEhISwuHDh3O9dlpaGp6entx8881X9HssbWw2GytWrKBbt255rl4qyzZHxfPIzMuvXvp6yI0F6vnIra0cusG2o2c5fS6dQG8rN9SpIpNJKf/vraIkbVVwxdVWud2k5qXIg4/AwEA0TcvRy3Hy5MkcvSGQmfvCarXmOG42m3M0isPhQFEUVFW9ojkbzuEA5zVKkjO7aXYPPPAADzzwAJBzvkb79u2JiIjIcix7mX79+mXJKPrOO+9Qs2bNLL1Gbdq0KXSE62yr1atX59lWR44cyfK4b9++9O3b97LXVtXM7Ja5/Y7LsvL28zi1bVANf2/Py87NaNugWoEDhuxtZQY6NMr5+SAyldf3VnGQtiq4om6rwlyryL+BLRYLN9xwQ44vuxUrVtC+ffuifrkKb9q0aWzZsoXDhw/z3Xff8d577zFkyBB3V0uUI5qqMLZ3EyD3RF8ge6kIIQqnWIZdnnvuOQYPHkyrVq1o164dX3zxBceOHePJJ58sjper0A4cOMDbb79NfHw8tWvX5vnnn+eVV14p0HP//PNPevbsmef56OjooqqmKON6hAUz/YGWBdohVwghLqdYgo/+/ftz5swZ3nzzTWJjYwkLC+P333+nTp06xfFyFdpHH33ERx99dEXPbdWqVY5hHafLrVgRFc/V7JArhBCXKrYJp8OHD2f48OHFdXlRBDw9PfPM36HreqEmD4mKQRJ9CSGKQtnf20UIIYQQZUqZDD5kSKBsk9+fEEJUbMU27FIcLBYLqqpy4sQJqlatisViKVS+Dl3XycjIIC0trcSX2pY1xdFWhmGQkZHBqVOnUFUVi8VSJNcVQghRtpSp4ENVVUJDQ4mNjeXEiYLt53IpwzA4f/48np6eJZ5krKwpzrby8vKidu3aEgAKIUQFVaaCD8js/ahduzZ2ux2Ho3AbPthsNtatW8fNN98sSWguo7jaStM0TCaTBH9CCFGBlbngA7ji7JiapmG32/Hw8JDg4zKkrYQQQhQX6fcWQgghRImS4EMIIYQQJUqCDyGEEEKUqFI358O5a2txZNe02WykpqaSlJQk8xguQ9qq4KStCk7aqnCkvQpO2qrgiqutnN/b2Xdfz02pCz6Sk5MBqFWrlptrIoQQQojCSk5Oxs/PL98yilGQEKUE6brOiRMn8PHxKfLlmElJSdSqVYvjx4/j6+tbpNcub6StCk7aquCkrQpH2qvgpK0KrrjayjAMkpOTqVGjxmXzOJW6ng9VValZs2axvoavr6+8OQtI2qrgpK0KTtqqcKS9Ck7aquCKo60u1+PhJBNOhRBCCFGiJPgQQgghRImqUMGH1Wpl7NixWK1Wd1el1JO2Kjhpq4KTtiocaa+Ck7YquNLQVqVuwqkQQgghyrcK1fMhhBBCCPeT4EMIIYQQJUqCDyGEEEKUKAk+hBBCCFGiKmzw0adPH2rXro2HhwfBwcEMHjyYEydOuLtapc6RI0d49NFHCQ0NxdPTk/r16zN27FgyMjLcXbVS65133qF9+/Z4eXlRuXJld1enVJk2bRqhoaF4eHhwww038Oeff7q7SqXSunXr6N27NzVq1EBRFBYsWODuKpVKEyZM4MYbb8THx4dq1arRr18/9u/f7+5qlVrTp0/nuuuucyUXa9euHUuWLHFLXSps8NGlSxd++OEH9u/fz88//8yhQ4e455573F2tUmffvn3ous7nn3/Onj17+Oijj/jss8949dVX3V21UisjI4N7772XYcOGubsqpcq8efMYNWoUY8aMYceOHdx000307NmTY8eOubtqpU5KSgrXX389U6dOdXdVSrW1a9cyYsQINm3axIoVK7Db7XTv3p2UlBR3V61UqlmzJu+++y5bt25l69at3HLLLfTt25c9e/aUfGUMYRiGYSxcuNBQFMXIyMhwd1VKvUmTJhmhoaHurkapN2PGDMPPz8/d1Sg1WrdubTz55JNZjjVu3Nh4+eWX3VSjsgEw5s+f7+5qlAknT540AGPt2rXurkqZUaVKFeOrr74q8detsD0fl4qPj+f777+nffv2shVzASQmJuLv7+/uaogyJCMjg23bttG9e/csx7t3786GDRvcVCtR3iQmJgLI51MBOBwO5s6dS0pKCu3atSvx16/QwcdLL71EpUqVCAgI4NixYyxcuNDdVSr1Dh06xJQpU3jyySfdXRVRhpw+fRqHw0H16tWzHK9evTpxcXFuqpUoTwzD4LnnnqNjx46EhYW5uzql1u7du/H29sZqtfLkk08yf/58mjRpUuL1KFfBR3h4OIqi5Ptn69atrvIvvvgiO3bsYPny5WiaxoMPPohRQRK+FratAE6cOEGPHj249957eeyxx9xUc/e4kvYSOSmKkuWxYRg5jglxJUaOHMmuXbuYM2eOu6tSql1zzTVERESwadMmhg0bxpAhQ9i7d2+J18NU4q9YjEaOHMn999+fb5m6deu6/h0YGEhgYCCNGjXi2muvpVatWmzatMktXVAlrbBtdeLECbp06UK7du344osvirl2pU9h20tkFRgYiKZpOXo5Tp48maM3RIjCeuqpp1i0aBHr1q2jZs2a7q5OqWaxWGjQoAEArVq1YsuWLXz88cd8/vnnJVqPchV8OIOJK+Hs8UhPTy/KKpVahWmrmJgYunTpwg033MCMGTNQ1XLVYVYgV/PeEpkfeDfccAMrVqzgzjvvdB1fsWIFffv2dWPNRFlmGAZPPfUU8+fPZ82aNYSGhrq7SmWOYRhu+d4rV8FHQW3evJnNmzfTsWNHqlSpwuHDh3njjTeoX79+hej1KIwTJ07QuXNnateuzfvvv8+pU6dc54KCgtxYs9Lr2LFjxMfHc+zYMRwOBxEREQA0aNAAb29v91bOjZ577jkGDx5Mq1atXD1ox44dk/lDuTh37hwHDx50PY6KiiIiIgJ/f39q167txpqVLiNGjGD27NksXLgQHx8fV8+an58fnp6ebq5d6fPqq6/Ss2dPatWqRXJyMnPnzmXNmjUsXbq05CtT4utrSoFdu3YZXbp0Mfz9/Q2r1WrUrVvXePLJJ43o6Gh3V63UmTFjhgHk+kfkbsiQIbm21+rVq91dNbf79NNPjTp16hgWi8Vo2bKlLInMw+rVq3N9Dw0ZMsTdVStV8vpsmjFjhrurVio98sgjrv9/VatWNbp27WosX77cLXVRDKOCzLAUQgghRKlQ8QbvhRBCCOFWEnwIIYQQokRJ8CGEEEKIEiXBhxBCCCFKlAQfQgghhChREnwIIYQQokRJ8CGEEEKIEiXBhxBCCCFKlAQfQgghhChREnwIIYQQokRJ8CGEEEKIEiXBhxBCCCFK1P8BtqD5pxfNaEwAAAAASUVORK5CYII=",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_176_1.png"
- }
- },
- "output_type": "display_data"
- }
- ],
- "source": [
- "import matplotlib.pyplot as plt\n",
- "import numpy as np\n",
- "from sklearn.linear_model import LinearRegression\n",
- "from sklearn.preprocessing import PolynomialFeatures\n",
- "from sklearn.model_selection import train_test_split\n",
- "from sklearn.preprocessing import StandardScaler\n",
- "\n",
- "def MSE(y_data,y_model):\n",
- " n = np.size(y_model)\n",
- " return np.sum((y_data-y_model)**2)/n\n",
- "\n",
- "def OLS_fit_beta(X, y):\n",
- " return np.linalg.pinv(X.T @ X) @ X.T @ y\n",
- "\n",
- "def Ridge_fit_beta(X, y,L,d):\n",
- " I = np.eye(d,d)\n",
- " return np.linalg.pinv(X.T @ X + L*I) @ X.T @ y\n",
- "\n",
- "# Same random numbers for each test.\n",
- "np.random.seed(2018)\n",
- "n = 100\n",
- "d = 3\n",
- "# hyperparameter lambda\n",
- "Lambda = 0.01\n",
- "\n",
- "# Make data set, simple second-order polynomial\n",
- "x = np.linspace(-3, 3, n)\n",
- "y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)\n",
- "\n",
- "# The design matrix X includes the intercept and no scaling is made\n",
- "X = np.zeros((len(x), d))\n",
- "for p in range(d): \n",
- " X[:, p] = x ** (p) \n",
- "\n",
- "\n",
- "#Split data, no scaling is used and we include the intercept\n",
- "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
- "\n",
- "\n",
- "#Calculate beta, own code\n",
- "beta_OLS = OLS_fit_beta(X_train, y_train)\n",
- "beta_Ridge = Ridge_fit_beta(X_train, y_train,Lambda,d)\n",
- "print(beta_OLS)\n",
- "print(beta_Ridge)\n",
- "#predict value\n",
- "ytilde_test_OLS = X_test @ beta_OLS\n",
- "ytilde_test_Ridge = X_test @ beta_Ridge\n",
- "\n",
- "#Calculate MSE\n",
- "print(\" \")\n",
- "print(\"test MSE of OLS:\")\n",
- "print(MSE(y_test,ytilde_test_OLS))\n",
- "print(\" \")\n",
- "print(\"test MSE of Ridge\")\n",
- "print(MSE(y_test,ytilde_test_Ridge))\n",
- "\n",
- "plt.scatter(x,y,label='Data')\n",
- "plt.plot(x, X @ beta_OLS,'*', label=\"OLS_Fit\")\n",
- "plt.plot(x, X @ beta_Ridge, label=\"Ridge_Fit\")\n",
- "plt.grid()\n",
- "plt.legend()\n",
- "plt.show()"
+ "## Material for the lab sessions"
]
},
{
"cell_type": "markdown",
- "id": "b09d20a2",
+ "id": "f8bffca6",
"metadata": {
"editable": true
},
"source": [
- "In this example we do not include the intercept and we scale the data by subtracting the mean values. This follows the discussion in the [lecture material](https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#more-on-rescaling-data).\n",
- "see also the weekly slides [for week 36](https://compphysics.github.io/MachineLearning/doc/pub/week36/html/._week36-bs029.html).\n",
- "It is recommended whrn we use Ridge and Lasso regression to not include the intercept in the optimization process.\n",
+ "## Linking the regression analysis with a statistical interpretation\n",
"\n",
- "Before we discuss the code, we repeat some of the basic math from the slides of week 36.\n",
+ "We will now couple the discussions of ordinary least squares, Ridge\n",
+ "and Lasso regression with a statistical interpretation, that is we\n",
+ "move from a linear algebra analysis to a statistical analysis. In\n",
+ "particular, we will focus on what the regularization terms can result\n",
+ "in. We will amongst other things show that the regularization\n",
+ "parameter can reduce considerably the variance of the parameters\n",
+ "$\\beta$.\n",
"\n",
- "Let us try to understand what this may imply mathematically when we\n",
- "subtract the mean values, also known as *zero centering* or simply *centering*. For\n",
- "simplicity, we will focus on ordinary regression, as done in the above example.\n",
+ "The\n",
+ "advantage of doing linear regression is that we actually end up with\n",
+ "analytical expressions for several statistical quantities. \n",
+ "Standard least squares and Ridge regression allow us to\n",
+ "derive quantities like the variance and other expectation values in a\n",
+ "rather straightforward way.\n",
"\n",
- "The cost/loss function for regression is"
+ "It is assumed that $\\varepsilon_i\n",
+ "\\sim \\mathcal{N}(0, \\sigma^2)$ and the $\\varepsilon_{i}$ are\n",
+ "independent, i.e.:"
]
},
{
"cell_type": "markdown",
- "id": "ef1cec97",
+ "id": "109c07d6",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n",
+ "\\begin{align*} \n",
+ "\\mbox{Cov}(\\varepsilon_{i_1},\n",
+ "\\varepsilon_{i_2}) & = \\left\\{ \\begin{array}{lcc} \\sigma^2 & \\mbox{if}\n",
+ "& i_1 = i_2, \\\\ 0 & \\mbox{if} & i_1 \\not= i_2. \\end{array} \\right.\n",
+ "\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "e43c422b",
+ "id": "1ea48f22",
"metadata": {
"editable": true
},
"source": [
- "Recall also that we use the squared value. This expression can lead to an\n",
- "increased penalty for higher differences between predicted and\n",
- "output/target values.\n",
+ "The randomness of $\\varepsilon_i$ implies that\n",
+ "$\\mathbf{y}_i$ is also a random variable. In particular,\n",
+ "$\\mathbf{y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n",
+ "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\beta}$ is a\n",
+ "non-random scalar. To specify the parameters of the distribution of\n",
+ "$\\mathbf{y}_i$ we need to calculate its first two moments. \n",
"\n",
- "What we have done is to single out the $\\beta_0$ term in the\n",
- "definition of the mean squared error (MSE). The design matrix $X$\n",
- "does in this case not contain any intercept column. When we take the\n",
- "derivative with respect to $\\beta_0$, we want the derivative to obey"
+ "Recall that $\\boldsymbol{X}$ is a matrix of dimensionality $n\\times p$. The\n",
+ "notation above $\\mathbf{X}_{i,\\ast}$ means that we are looking at the\n",
+ "row number $i$ and perform a sum over all values $p$."
]
},
{
"cell_type": "markdown",
- "id": "852577bc",
+ "id": "212f90ec",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Assumptions made\n",
+ "\n",
+ "The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)\n",
+ "that there exists a function $f(\\boldsymbol{x})$ and a normal distributed error $\\boldsymbol{\\varepsilon}\\sim \\mathcal{N}(0, \\sigma^2)$\n",
+ "which describe our data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a019e7cb",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C}{\\partial \\beta_j} = 0,\n",
+ "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "4edd229a",
+ "id": "2ca46c43",
"metadata": {
"editable": true
},
"source": [
- "for all $j$. For $\\beta_0$ we have"
+ "We approximate this function with our model from the solution of the linear regression equations, that is our\n",
+ "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we want to minimize $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, our MSE, with"
]
},
{
"cell_type": "markdown",
- "id": "bb93d608",
+ "id": "c61ed9c8",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n",
+ "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "5ef162e3",
+ "id": "96d84043",
"metadata": {
"editable": true
},
"source": [
- "Multiplying away the constant $2/n$, we obtain"
+ "## Expectation value and variance\n",
+ "\n",
+ "We can calculate the expectation value of $\\boldsymbol{y}$ for a given element $i$"
]
},
{
"cell_type": "markdown",
- "id": "b7c5467d",
+ "id": "e53b66df",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n",
+ "\\begin{align*} \n",
+ "\\mathbb{E}(y_i) & =\n",
+ "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}) + \\mathbb{E}(\\varepsilon_i)\n",
+ "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n",
+ "\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "493c75d6",
+ "id": "71d542a3",
"metadata": {
"editable": true
},
"source": [
- "Let us specialize first to the case where we have only two parameters $\\beta_0$ and $\\beta_1$.\n",
- "Our result for $\\beta_0$ simplifies then to"
+ "while\n",
+ "its variance is"
]
},
{
"cell_type": "markdown",
- "id": "862f2222",
+ "id": "01b18637",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n",
+ "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n",
+ "- \\mathbb{E}(y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( y_i^2 ) -\n",
+ "[\\mathbb{E}(y_i)]^2 \\\\ & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n",
+ "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \\\\ &\n",
+ "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2 \\varepsilon_i\n",
+ "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
+ "\\ast} \\, \\beta)^2 \\\\ & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2\n",
+ "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} +\n",
+ "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \n",
+ "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n",
+ "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2. \n",
+ "\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "e0d0a4e0",
+ "id": "e35bdcc1",
"metadata": {
"editable": true
},
"source": [
- "We obtain then"
+ "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
+ "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD)."
]
},
{
"cell_type": "markdown",
- "id": "73d327de",
+ "id": "15b7ea3c",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Expectation value and variance for $\\boldsymbol{\\beta}$\n",
+ "\n",
+ "With the OLS expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}}$ we can evaluate the expectation value"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a4c5f625",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n",
+ "\\mathbb{E}(\\boldsymbol{\\hat{\\beta}}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "8be27cbd",
+ "id": "450007e4",
"metadata": {
"editable": true
},
"source": [
- "If we define"
+ "This means that the estimator of the regression parameters is unbiased.\n",
+ "\n",
+ "We can also calculate the variance\n",
+ "\n",
+ "The variance of the optimal value $\\boldsymbol{\\hat{\\beta}}$ is"
]
},
{
"cell_type": "markdown",
- "id": "d49a9bac",
+ "id": "3bcf81c1",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\mu_{\\boldsymbol{x}_1}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1},\n",
+ "\\begin{eqnarray*}\n",
+ "\\mbox{Var}(\\boldsymbol{\\hat{\\beta}}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n",
+ "\\\\\n",
+ "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}]^{T} \\}\n",
+ "\\\\\n",
+ "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "% \\\\\n",
+ "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} \\, \\mathbf{y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "% \\\\\n",
+ "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{y} \\, \\mathbf{y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "\\\\\n",
+ "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "% \\\\\n",
+ "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^T \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
+ "% \\\\\n",
+ "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\boldsymbol{\\beta}^T\n",
+ "\\\\\n",
+ "& = & \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "\\, \\, \\, = \\, \\, \\, \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1},\n",
+ "\\end{eqnarray*}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "dd964823",
+ "id": "f69a019e",
"metadata": {
"editable": true
},
"source": [
- "and the mean value of the outputs as"
+ "where we have used that $\\mathbb{E} (\\mathbf{y} \\mathbf{y}^{T}) =\n",
+ "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n",
+ "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n",
+ "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n",
+ "variance of the estimate of the $j$-th regression coefficient:\n",
+ "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $. This may be used to\n",
+ "construct a confidence interval for the estimates.\n",
+ "\n",
+ "In a similar way, we can obtain analytical expressions for say the\n",
+ "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n",
+ "when we employ Ridge regression, allowing us again to define a confidence interval. \n",
+ "\n",
+ "It is rather straightforward to show that"
]
},
{
"cell_type": "markdown",
- "id": "09a87e67",
+ "id": "95b37da0",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\mu_y=\\frac{1}{n}\\sum_{i=0}^{n-1}y_i,\n",
+ "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "9b8797a0",
+ "id": "241c2335",
"metadata": {
"editable": true
},
"source": [
- "we have"
+ "We see clearly that \n",
+ "$\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}$ for any $\\lambda > 0$.\n",
+ "\n",
+ "We can also compute the variance as"
]
},
{
"cell_type": "markdown",
- "id": "30aeafdb",
+ "id": "cf794384",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\beta_0 = \\mu_y - \\beta_1\\mu_{\\boldsymbol{x}_1}.\n",
+ "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "dc867cbb",
+ "id": "4cd5260a",
"metadata": {
"editable": true
},
"source": [
- "In the general case with more parameters than $\\beta_0$ and $\\beta_1$, we have"
+ "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n",
+ "\n",
+ "With this, we can compute the difference"
]
},
{
"cell_type": "markdown",
- "id": "f154321e",
+ "id": "7361e846",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n",
+ "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}]-\\mbox{Var}(\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "6015f267",
+ "id": "6a07837b",
"metadata": {
"editable": true
},
"source": [
- "We can rewrite the latter equation as"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f181e88f",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\beta_j,\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a6e6c0e7",
- "metadata": {
- "editable": true
- },
- "source": [
- "where we have defined"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "708652ea",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mu_{\\boldsymbol{x}_j}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{ij},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b360788c",
- "metadata": {
- "editable": true
- },
- "source": [
- "the mean value for all elements of the column vector $\\boldsymbol{x}_j$.\n",
+ "The difference is non-negative definite since each component of the\n",
+ "matrix product is non-negative definite. \n",
+ "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. \n",
"\n",
- "Replacing $y_i$ with $y_i - y_i - \\overline{\\boldsymbol{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8f473602",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "70f518a7",
- "metadata": {
- "editable": true
- },
- "source": [
- "If we minimize with respect to $\\boldsymbol{\\beta}$ we have then"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "674d02d9",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6715261c",
- "metadata": {
- "editable": true
- },
- "source": [
- "where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n",
- "and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n",
- "\n",
- "For Ridge regression we need to add $\\lambda \\boldsymbol{\\beta}^T\\boldsymbol{\\beta}$ to the cost function and get then"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "17600ee3",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a8c6f068",
- "metadata": {
- "editable": true
- },
- "source": [
- "Now we try to implement this."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "1d89f80d",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[0.47179152 5.01549939]\n",
- "[0.47176783 5.01542292]\n",
- "1.7993408651198877\n",
- "1.7995707762668065\n",
- " \n",
- "test MSE of OLS:\n",
- "1.1394311129039245\n",
- " \n",
- "test MSE of Ridge\n",
- "1.1395084586525954\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAACDZUlEQVR4nO3dd3hURffA8e+925KQAkmAhNBCABGiAiJdAREE6TYERexIeRV97fhK+Kk0GyqKHVEErDSVQJQmTWqogpRQEhJaekKS3b3390fYhVQ2kGRTzud5eGB3Z+9OJiF7dubMGUXXdR0hhBBCiHKiursDQgghhKheJPgQQgghRLmS4EMIIYQQ5UqCDyGEEEKUKwk+hBBCCFGuJPgQQgghRLmS4EMIIYQQ5UqCDyGEEEKUK6O7O5CfpmmcPHkSHx8fFEVxd3eEEEII4QJd10lLS6NevXqoavFzGxUu+Dh58iQNGjRwdzeEEEIIcQVOnDhB/fr1i21T4YIPHx8fILfzvr6+pXptq9XKihUr6N27NyaTqVSvXdXIWLlOxsp1MlYlI+PlOhkr15XVWKWmptKgQQPn+3hxKlzw4Vhq8fX1LZPgw8vLC19fX/nhvAwZK9fJWLlOxqpkZLxcJ2PlurIeK1dSJiThVAghhBDlqkTBR0REBIqi5PkTFBTkfFzXdSIiIqhXrx6enp50796dvXv3lnqnhRBCCFF5lXjmo1WrVsTHxzv/7N692/nY9OnTeffdd5k5cyZbtmwhKCiIXr16kZaWVqqdFkIIIUTlVeLgw2g0EhQU5PxTu3ZtIHfWY8aMGUyYMIE777yT8PBw5syZQ2ZmJvPmzSv1jgshhBCicipxwunBgwepV68eFouFDh06MHnyZJo0aUJMTAwJCQn07t3b2dZisdCtWzc2bNjAqFGjCr1ednY22dnZztupqalAbkKM1WotafeK5bheaV+3KpKxcp2MletkrEpGxst1MlauK6uxKsn1FF3XdVcbL1u2jMzMTJo3b86pU6d444032L9/P3v37uXAgQN06dKFuLg46tWr53zOE088wbFjx1i+fHmh14yIiGDSpEkF7p83bx5eXl4ufyFCCCGEcJ/MzEyGDx9OSkrKZXerlij4yC8jI4OwsDBeeOEFOnbsSJcuXTh58iTBwcHONo8//jgnTpwgMjKy0GsUNvPRoEEDzp49WyZbbaOioujVq5dsxboMGSvXyVi5TsaqZGS8XCdj5bqyGqvU1FQCAwNdCj6uqs5HjRo1uO666zh48CCDBw8GICEhIU/wcfr0aerWrVvkNSwWCxaLpcD9JpOpzH6AyvLaVY2MletkrFwnY1UyMl6uk7FyXWmPVUmudVV1PrKzs/nnn38IDg4mNDSUoKAgoqKinI/n5OSwZs0aOnfufDUvI4QQQohSYNd0NsckArA5JhG7dsWLH1elRDMfzz33HAMGDKBhw4acPn2aN954g9TUVEaOHImiKIwfP57JkyfTrFkzmjVrxuTJk/Hy8mL48OFl1X8hhBBCuCByTzyTlu4jMf0809vDI3O24O/tycQBLekTHnz5C5SiEgUfsbGxDBs2jLNnz1K7dm06duzIpk2baNSoEQAvvPAC58+fZ8yYMSQlJdGhQwdWrFjhUp13IYQQQpSNyD3xjJ67HR2wGC7en5CSxei525n1QNtyDUBKFHwsWLCg2McVRSEiIoKIiIir6ZMQQgghSold05m0dB+FLbDogAJMWrqPXi2DMKiXP5elNMjZLkIIIUQVtjkmkfiULOdtTYeZe1Uc6R46EJ+S5cwFKQ8SfAghhBBV2Om0rDy3NR0Opl4MPopqV5Yk+BBCCCGqsDo+Hnlu2/W8fxfVrixJ8CGEEEJUYe1D/fPcfsv4KcMNf+JJVrHtypIEH0IIIUQVZlAVHu7SGIBrlWPcY1zLROMcTNidbR7u0rjckk3hKiucCiGEEKLimzigFcG+HuhR3wHwp9aWVLwBeKVvC57oFlau/ZHgQwghhKgGOjepRaBhAwCL7F1QyN3p0rlpYLn3RYIPIYQQohoITtpCgJJEquJDQKPrCc/xJSElmwBvc7n3RYIPIYQQohoIOLwQgBqt76IjBib17YCuGrAYDZd5ZumThFMhhBCiqsvJgH+WAqBffy+QW5XcHYEHyMyHEEIIUSU5TrA9nZZFy7PLaZaTDrUao4fcBLuWubVvEnwIIYQQVYzjBFtHWfXZpq9pZoBDQXfQSCm/LbVFkWUXIYQQogpxnGDrCDwCSeFmdTcAj0eH8cc/p9zZPUCCDyGEEKLKKOwE2wGGDRgVjR1aU47qwUxdtt9t/XOQ4EMIIYSoIvKfYAswxLAOgIX2LuhAQmr5HSBXFAk+hBBCiCoi/8m0YUoc16sxWHUDv9o7AaDrhT2zfEnwIYQQQlQR+U+mdcx6rNGuJxFfAHI0STgVQgghRClpH+pPTU8TAAoagw3rAVhov9nZJsDL/RtdJfgQQgghqgiDqjD1rusA6Kj+Q33lLKm6F39obZ1t3hjcyl3dc5LgQwghhKhC+oQH80rfFtxl+AuAX+0dySb3/JZX+rbgtmvrurN7QDULPnbHpTBzr8ruuBR3d0UIIYQoMwNb+XGHYTMASc3uoklgDQK9LQxoXc/NPcvl/oWfcmLXdD5ZG8PBVJVP18bwccMADKr7k26EEEKI0hYU9weQhV4rlLEP3s8YIMeuYTEasFqt7u5e1Q8+YpMy+W1XPJ//dYS09DTClTiW79PpMPkPHr+5Cf2uD6Z+LS93d1MIIYQoPTvnA6DcMAwUBQXcdohcYap88NF12iogd6/zSstENBQ6ZM/kbLrClGX7mbJsP0en9nNzL4UQQohSkhILR9bk/vuGoe7tSxGqdM6HXdOdW45i9GDS8aCmkkFPdYezTU1PE3atAlRcEUIIIUrDrh8AHRp1gVqN3d2bQlXp4GNzTCLJ53PXtjRUfrmwz9mRAQyQfN7K5phEt/RPCCGEKE27TiQRt+ar3Bs3DHNvZ4pRpYOP/GVmf7F3BaCbupPaJBXZTgghhKiMNq37gxDbCayKGVoOcnd3ilSlg4/8ZWZj9HocNTXFqFys+gZw8FQ6Gw+fk+UXIYQQlU5sUia7Y1PYeSIZnwM/AhCl38TOMxq7Y1OITcp0cw8LqtIJp+1D/Qn28yAhJQsdMKmQHNQVThziLsNffG7vByjMXHWImasOEeRrYVj7hjQOrEEdHw/ah/rLdlwhhBAVmmNjhQkbmy3rQYEFOV1Z+9HFD9kVbWNFlZ75MKgKEwe0BEABFAXianYgWzfRQj1BuBKTp31Cajbv/XGQpxdEM+zzTXSdtpLIPfFu6LkQQgjhmoe7NAbgVnUHtZR0Tuk1Wa+FF3i8IqnSwQfklpmd9UBbgvxyl2Bsxhr8od8I5E08LUxCShaj526XAEQIIUSFZNd0IvckAHC3IXd77UL7zdi5WNMjck9ChUsrqPLBB+QGIOtevJWvRt4EwELtFgAGGdZjwlbk8xzfqklL91W4b5wQQgixOSaR+JQsAkmhhxoNwI/2W/K0iU/JqnC7OqtF8AG5SzDtQ/0BWK9fxym9Jv5KOrdeUvOjMDoV8xsnhBBCOHZrDjKsw6ho7NCaclgPKbJdRVFtgo9LaagsvFDz427DWpeeU9G+cUIIIUTurk6dey68l/2Ub9Yjb7uKo1oGH4E1LPx0IfjorkYTwOVPua1o3zghhBCifag/3X1O0kI9QbZuYqm9U57HFSDYz8M5819RVMvgIy41h0N6faK1JpgUO4MMG4psW1G/cUIIIYRBVZjUMBqA5Vo7UqnhfMxRKGLigJYVrmxEtQw+3r77OlQFfrJ3A+AewxouppdeVJG/cUIIIQS2bBqd/B2AlR635XkoyM+DWQ+0pU94sDt6VqwqXWSsKINuCKZFsB/DP0znf8a5XKsep5VylL16aJ52QX4eTBzQskJ+44QQQggOLIPzSeBTj3eeHs/QYymcTsuq8IUyq2Xw4ZCKN8u1dgw0bGSocTWvWUOZPOQ6algMFf4bJ4QQonrbFZuMbclHtAW44T4MRiOdwgLc3S2XVMtlF4AAbzO1vS1srXUHAEOMGwmpodCjRW0GtQ6hU1iABB5CCCEqrBV/7+SGrC25N1rf797OlFC1nfkI9vNk3Us9MCvd4f1P8UmNZfXADEx+nu7umhBCCFGo2KRMkjKsKAoY9/yIQdGJ5hqM2bXRY1OoVcNE/Vpe7u7mZVXb4APAYrxQfrb1MFj7FqZd8+GGe9zbKSGEEKIIjkPkQCfKvBJU+MHalXkfrnO2qWiHyBWm2i675NF6eO7fh1dCSqx7+yKEEEIUYcbQ1hhVhbbKQZqpcZzXzSy5UNvDqCrMGNravR10kQQfAP5NoFFXQIed893dGyGEEKJQg9uEsGhslwslIuB3rQPp5C6zLBrbhcFtCpZWr4gk+HBo8wAA2Vu/ZdinG9kVm+ze/gghhBCFUK0ZDDBsBOB7W3eUSrg3QoIPh5YDweyDJfUY9qPr+WV7nLt7JIQQQhRQ7+RyvJUsTqr1GDTobq4L8aO2t4UAb7O7u+ayap1w6pCbPWwjJLQf/gcWcK9xDVN23sDdN9ZH16k02cNCCCGqvpr7fwAguPuj3N+xMcM7NCLHrl3cRFEJSPDBxezhtkoLfrHAHerfRGSk0L+SZQ8LIYSo4s4eguMb0BWV7bX6Ehsd5yyKWZnIsgsXs4e36804pNXDS8mmn2ETULmyh4UQQlRx0XMB2KC05q7vjvL0gmiGfb6JrtNWErkn3s2dc50EH1zMHgaF7+3dAbjPkDsbUpmyh4UQQlRdu46f5dz6OQB8k3VLnscSUrIYPXd7pQlAJPjIZ6F2M1bdQBv1EM2VE+7ujhBCCIFd01nz23wC9ETO6r6s1NrmedxxLvukpfuwawVPaa9oJPi4wHHWS72QhsQHdQfgIY+/KlX2sBBCiKolNimTT9ccpsPkP2h+chEAC+1dsRaSsqkD8SlZbI5JLN9OXgFJOL3AedaLQUU5mAnz/mSYx3qUGhKfCSGEcA/HhohAUrjVsgOAHy6kBxTldFpWWXfrqsk76yUsRgOKokDTnuBTD+V8Euz/1d3dEkIIUQ3ZNZ2aniYA7jSsxaTYidbCOKjXL/Z5dXw8yqN7V0WCj8KoBmhz4Xji7d+6ty9CCCGqpc0xiSSftwI6Qw2rAZhvv7XI9goQ7Fc5tt1K8FGUC+XWObIKko66tStCCCGqH8fySXtlP2FqPBm6hV/tHQtt66iwPnFASwxqxa+3LsFHUWo1hibdATi19kuGfbZJznsRQghRbhzLJ/cZc/M+ltg7k4FnoW2D/DyY9UBb+oQHl1v/roYknBanzQg4shqPPQv4O609v2z34fr6Nd3dKyGEENVA+1B/mvvauCP7bwAW2HsUaFPT08RH97elY5OASjHj4XBVMx9TpkxBURTGjx/vvE/XdSIiIqhXrx6enp50796dvXv3Xm0/y11sUiZ7fG7GZqmJn/U0t6i7WLrzJHviUtgdm0JsUqa7uyiEEKIKM6gKM1r+i4diZb/WgJ16mPMx5cKfqXddR5emgZUq8ICrCD62bNnCZ599xvXXX5/n/unTp/Puu+8yc+ZMtmzZQlBQEL169SItLe2qO1ueuk5bRf9PtvJtRgcgt+JpYkYO/T9cx4CZ65zbn4QQQogyoeu0jF8EwG/m3lzM7Kh8yyz5XVHwkZ6ezv3338/nn39OrVq1nPfrus6MGTOYMGECd955J+Hh4cyZM4fMzEzmzZtXap0uD47zXhzTXD3V7QSSBMh5L0IIIcrBye1wag8YLIx/5lXmP96R9+9rzfzHO7LuxVsrbeABV5jzMXbsWPr168dtt93GG2+84bw/JiaGhIQEevfu7bzPYrHQrVs3NmzYwKhRowpcKzs7m+zsbOft1NRUAKxWK1ar9Uq6VyTH9Vy5br/wOjT278DgWTrbtGbcqB7kHsNaPrYP4qdRHWhVz7fU+1eRlGSsqjsZK9fJWJWMjJfrqtJY2TWdH7fF4r9qKv0Ae4sBaGYf2jUE8AVAs9vQ7Fd2/bIaq5Jcr8TBx4IFC9i+fTtbtmwp8FhCQgIAdevWzXN/3bp1OXbsWKHXmzJlCpMmTSpw/4oVK/Dy8ipp91wSFRXlUrsT6QBGFth7cKN6kKGGVcyyD2DdunUc8y6TrlU4ro6VkLEqCRmrkpHxcl1VGau1R3J4L2ctKLAxuznnfv+91F+jtMcqM9P1XMgSBR8nTpzg6aefZsWKFXh4FF1BTVHyJr7oul7gPoeXX36ZZ5991nk7NTWVBg0a0Lt3b3x9fUvSvcuyWq1ERUXRq1cvTCbTZdvHp2TxdcwmYnx7k5M6j0a20/T1+peGLe9F13UCvS3c2KhWpUv0cUVJx6o6k7FynYxVych4ua4qjNX3W0/wf0v3oQNDlL/xNmVxRAvikX0tAYXXBrRkaLsGV/06ZTVWjpULV5Qo+Ni2bRunT5/mxhtvdN5nt9tZu3YtM2fO5MCBA0DuDEhw8MW1qNOnTxeYDXGwWCxYLJYC95tMpjL7AXL12g0DTax9oQfRx5OJWz+A0Jj53KlH8dj3LZxtgv08mDigZaVeeytOWX4fqhoZK9fJWJWMjJfrKutY2TWdVxf/gyOpdJg5d1PD9/Ye5Gi56ZmvLv6HYR1CS+0Db2mPVUmuVaKE0549e7J7926io6Odf9q1a8f9999PdHQ0TZo0ISgoKM9UTk5ODmvWrKFz584leakKIXJPPD3fWcPwL/5mzP7cXT23aJsJIMXZJiEli9FztxO5J95d3RRCCFHJXXoS7bXKMdqoh7DqBn6231Jku8qsRDMfPj4+hIeH57mvRo0aBAQEOO8fP348kydPplmzZjRr1ozJkyfj5eXF8OHDS6/X5SByTzyj525Hv3D7H70R0VoYrdXD3GVYy2f2AUDuEcYKMGnpPnq1DKqSSzBCCCHK1qUn0Q43/AnAcq0dZ/Ersl1lVurl1V944QXGjx/PmDFjaNeuHXFxcaxYsQIfH5/SfqkyY9d0Jl1Yd7vUvAsH+gwzrIRLHtXJzQ+pKhGpEEKI8uUope5JFoMN64HCD5GrDCfWuuKqy6uvXr06z21FUYiIiCAiIuJqL+02m2MSiU8pGF3+au/E/4xzCVVP0Undx0atVZ7Hq0pEKoQQony1D/Un2M+Dm9NX4aOc56hWlw353mMqy4m1rpCD5QpRVBCRiQdL7Lm5K7mzH3lVlYhUCCFE+TKoChMHtOT+C0su8+23ol/yFq1QeU6sdYUEH4UoLohwLL3crm7Bn7zbiqpKRCqEEKL89Qk4zQ3qEawY+emSRNPgSl5KvTByqm0hHNNfCSlZBfI+9uqh7NJCuV6N4S7DWj639wfg4S6Nq0xEKoQQwg22zgbA0GogM9vezum0LOr45C61VLX3F5n5KIRj+gsuPcbnou/stwG5GckKGi/1uYbeLYNYHB3HxsPnsGv5QxYhhBCiaHuOxHJ++wIA1HYP0yksgEGtQ+gUFlDlAg+QmY8i9QkPZtYDbZm0dF+B5NOl9k5MuJB42lndyxfrPDibfsD5eFUvPCaEEKL02DWdHcu+JFw/T4KpPrUbdsXg7k6VMZn5KEaf8GDWvXir8yTBmcPaEOhtoWn9usQ3GgTA/YY/OZuek+d5UnhMCCHE5cQmZfLpmsN0mPwHN5xaCMAXmd3oMOVPPl1zmNgk189KqWxk5uMyDKpCp7AA5+1erepiNqhop2rBJwvorW6lDkmcppazjRQeE0IIcTldp+WWUA9XjnC9JYZs3cjP9ptJSs9hyrL9TFm2n6NT+7m5l2VDZj5KyGI0oCgKmzOC2Kxdg1HRGGpYVaCdFB4TQgiRn13T2Xj4HAu3x1LDkru44theG6m1J4mLB6rW9DRV2RxCmfm4QqfTslhp60l78wGGGVfysX0Q9kJW6aTwmBBCCMg9tiN/HqEvGc6KpnNtt+Vpn3zeyuaYxDyz71WFzHxcoTo+HkRq7UnUvamnJNJdjS6ynRBCiOrNcV5Y/g0Mdxr+wlPJ4YBWny36NQWeV1U/wErwcYXah/rj7+fLT/ZuwMVpMweFqlUKVwghxJUp6rww0J3vHXPtt1FYcYeq+gFWgo8rZFAVxvUIc1Y87a7upL5yxvm4DozrESbJpkIIUc0VdV5YR/UfmqlxZOgWFtq75nmsqn+AleDjKkxYtJejejB/2cNRFZ378p33MmHRXjf1TAghREVR1NLJA4Y/AFhk70o6Xs77HR9Zq9JZLvlJ8HEVZgxtjVFVLkyXwVDDKsxYATCqCjOGtnZj74QQQlQEhS2d1CaJ29UtAM73EIegKniWS36y2+UqDG4TQtM63gz+0EqCXosgJYk+6haWaJ1ZNLYL4SF+7u6iEEIINyvsvLChhtWYFDvbtGb8ozfCv4aJ//VvRZBv1TzLJT+Z+SgFNozMs/cEYIRxhZt7I4QQoiLJf16YisYwY+4y/Xe221CAyUOuY0ibqnuWS34SfFylAG8ztb0t7KwzCE0xcpP6L51rxBPgbXZ314QQQlQQ4SF+vNS3BQHeZm5VdxCinCNR92aT58281LdFtZspl2WXqxTs58m6l3pgNqgoPw2AvQv59vpdGPwec3fXhBBCVBCOUuoAD5hyE01/tHfjZDZVvpR6YWTmoxQ4Sq5zU27AYdjzI2SluLlXQgghKgrHBoVGSgK3qLsAnMv11XGDggQfpalRF6h9LVgzIXq+u3sjhBCighjcJoRFY7vwgOEPVEVnlf0GjulBACwa24XBbULc3MPyJcFHaVIUuOlRAFL+mkWvd1Yzd9OxKnswkBBCCNcptvPca1gNwDf23ihVP6+0SBJ8lLIoU3cy8MAv4yi1z/3Nq4v20HXaSiL3xLu7a0IIIcqR4wTbxdFxbDx8juDjS/FTMolXg+g1cDjXhfhR29tSLTcoSMJpKfpu01EmLPqX/zPezIPGKEYYotighROfksWTc7fz5uBW3N+xsbu7KYQQoowVPMFWZ4XnTPyBoJ5jGd6xCcM6hJJj17AYC56IXtXJzEcpsWu6s5z6t/ZeAPRStxHEOWebCYv2yhKMEEJUcYWdYNtOOUBz/ShZuomVnrnvEYqiVMvAAyT4KDWbYxKd/z6o12ejvSVGReN+459FthNCCFG1FHWC7YPGKAAW27vw6vKT1f6DqAQfpST/wUFz7L0BGGZYiYWcItsJIYSoOgo7wbY2SfRVNwO5iabxKVnV/oOoBB+lJP/BQVHajcTpAQQqqfRTNxXZTgghRNVR2AfM4YaVmBQ7W7Xm7NUbF9muOpHgo5Q4Dg5ysGPgO1vuSYUjjSsAnWC/3AODhBBCVE35P2AasTH8wvL7N7beRbarbiT4KCWOg4Mu3bY9396DbN3EDeoR2iiHmDigZbU4MEgIIaqr/B8wb1e3UldJ5ozuxzKtfZHtqhsJPkpRn/BgZj3Q1jkDkoQvS+ydAJjZdAt+nmbnfu/qnmwkhBBVkUFVeLhLY+ftkcblAMyz34r1QnWLh7s0rvYfRKXORynrEx5Mr5ZBbI5J5HRaFqE5z8Hva6lzIpLBn0dyhpoABPt5MHFAS/qEB7u3w0IIIUrVxAGtCPb1YHHkMtqrB7DqF5fhX+nbgie6hbm5h+4nMx9lwKAqdAoLYFDrEE56NWer1hwTNoYbLm67TUjJYvTc7VL5VAghqqDOTQN5+MKsx+9aB85Qy3m/kOCjTDn2e8+5kGR0v/FPTNgAnHvAJy3dJ0swQghRxdRWUxho2ACApesYrqtffUupF0aWXcqQY7/3MtpzSq9JXSWZvupmlmidgdwAxLHfu1NYgHs7K4QQotTU/fd7wIYeciN9bh/A7b31altKvTAy81GGHPu4bRid630PGSOLbCeEEKIKsFthyxcAKB2ezP27GpdSL4wEH2Xo0n3c8+w9ydaNtFUPcYNyqMh2QgghKrl9iyE9AbzrQsvB7u5NhSTBRxm6tPDYWfz4VcvddvvwJbMfUnhMCCGqjl2xyfy75O3cG+0eBaPkeBRGgo8y5Cg85vCVrQ8A/dS/qUtuXX8pPCaEEFXH339F0dz6D3bFCO0ednd3KiwJPspYn/BgpgwJR1Vgrx7K31oLTIqdB41RTBkSLnU+hBCikotNymR3bAp74lKo9+83ACyjM3tSLOyOTSE2KdPNPax4ZLdLORjWoRGD2oQQfTwZ5d/RsPlpxvj8hdK2jru7JoQQ4ip1nbYKyD29dr1lPSjwaVZvdn+4ztnm6NR+7upehSQzH+XEy2ykc9NA2vcZATUbopxPhF0/uLtbQgghrtKMoa1RFRhhjMJ84fTa3XoTAFQl93GRlwQf5U01QPsncv+9aRboUmBMCCEqMw+TiknP4f4LVay/tPV1PqbpuY+LvGRE3KHNCDDVgDP/8MbMT9gVm+zuHgkhhLgCjkrWQwzrCFDSiNUDWaG1y9NGKlkXJMGHO3jWhDb3A9Dx9Pf8sj3Ovf0RQghxRXIrWZ/nEcMyAGbbbsdO3mJijkrW4iIJPsqZIyv630bDAbjNsIPo6G3siUuRrGghhKig7JrOxsPnWBwdx8bD55wzGafTsrhF3UVzNY503YMf7D0Kfb5Uss5LdruUM0dWNMCXpjb0NOxgSM4S+n948WwXyYoWQoiKI3JPPJOW7iM+5WIAEeznwcQBLanj48GjF2Y9frB3Jw2vQq8hlazzkpmPcubIigb40p6blHSPYS1+pEtWtBBCVDCRe+IZPXd7nsADICEliyfnbufLX36nm2EXmq4w2357gecrSCXrwkjwUc48TCqOvKMNWiv+0RripWRzv+FPNB2OncsoMK0nhBCi/DmSSQv7Tey479aUnwFYobUjVq+bp42jdrVUsi5Igo9y5PhBvkjhc9sdAIw0LseEjff+OMjTC6IZ9vkmuk5bSeSeePd0VgghqrncZNKiczVqkcqdhtxCYjntRhHgnfcclyA/D2Y90FYqWRdCgo9yVNgP8lKtM6f0mtRVkhmgbsjzWEJKFqPnbpcARAgh3OBySaL3G/7EQ7GyW2vMUxs9OZueA8D797Vm/uMdWffirRJ4FEGCj3JU2A+yFSNzbLnrhI8Zl8ElE3yOf8kecSGEKH/FJYmasTLSuAKAL213AApGVWHG0NYMah1Cp7AAWWophgQf5aioH+Tv7D3J1C20VI/RSd2X5zEd2SMuhBDlybGtNiHlPP41zBQWQgwyrKe2ksJJ3Z9ftY4ALBrbhcFtQsq3s5WUbLUtR+1D/Qn28yAhJStPAlMK3vxov4WRxigeN/zGRq1VgefKHnEhhCh7hW2rzU9B4wnDbwDMtvXBrhgpNCtVFElmPsqRQVWYOKAlQIFI+it7XzRd4VZDNGFKwYqnskdcCCHKVlHbavPrpu6kmRpHBp406zuW60L8qO1tKZBwKoomwUc56xMezKwH2hLklzeYOKYHEaXdCMCjht/zPFbbxyJ7xIUQogwVt63Wwb+GifeGtub9Brk7XLw6Pcq9XcNZPLYL617qQbCfZ/l0tgqQ4MMN+oQHs+7FW5n/eEfev6+1837Httu7DOsIJMV5/5m0bElcEkKIMnS5bbUAiRlWwqyH8Du1EV01sj34PhZHx7HpSCJGVd5OS0JyPtzEoCp0Csstqa7r8OwP0WzVryFaC6O1epgHjct513YvqgLv3tvavZ0VQogqztW8ujMr3gFgBZ0YNe84cBy4WG5dtta6pkSh2qxZs7j++uvx9fXF19eXTp06sWzZMufjuq4TERFBvXr18PT0pHv37uzdu7fUO13VDG4TwpJxXQGFT2wDABhh+ANPslg4pgt1fT2k6qkQQpQhV/Lq6nGWbta/APjgfJ88j0ldppIp0cxH/fr1mTp1Kk2bNgVgzpw5DBo0iB07dtCqVSumT5/Ou+++y9dff03z5s1544036NWrFwcOHMDHx6dMvoCqJkpvR4xWl1D1FPca1vDoHF9n4RqQ6FoIIcpCUbsRL/WIMRKjorHe3oq9emiex3RyNxJMWrqPXi2DZKn8Mko08zFgwADuuOMOmjdvTvPmzXnzzTfx9vZm06ZN6LrOjBkzmDBhAnfeeSfh4eHMmTOHzMxM5s2bV1b9rzICvM3U9rbQKqQWZ657HIDHDL+TlH4+TzuJroUQovQVtxsRwIdMhhpyTyX/3F74yeNSl8l1V5whY7fbWbBgARkZGXTq1ImYmBgSEhLo3bu3s43FYqFbt25s2LChmCsJgGA/T9a91IPFY7tw48CxJOJLA/UMd6h/52knVU+FEKJsFLUbEeB+wx/4KOf5VwthtXZDsdeRukyXV+KE0927d9OpUyeysrLw9vZm4cKFtGzZ0hlg1K2b91S/unXrcuzYsSKvl52dTXZ2tvN2amoqAFarFavVWtLuFctxvdK+bmlRAZtNY/OxNLbae/O04SeeNP3KCltH8sfiienn2XTodJltwa3oY1WRyFi5TsaqZGS8XFdaY9XzmkC6N7uZbceSOJueTaC3BU/VRv1vxwDwqW0Ajt/HJlWnsNWVQC9jhf6eldXPVUmup+i6XqKPzzk5ORw/fpzk5GR+/vlnvvjiC9asWUNycjJdunTh5MmTBAdfzEd4/PHHOXHiBJGRkYVeLyIigkmTJhW4f968eXh5eZWka1WKyZZG773PYNRyWBjyIh+fuY6BjTQaeru7Z0IIUb34xK3m1tNfcVL3p1v2DGwY0FF47jobDeR3slNmZibDhw8nJSUFX1/fYtuWOPjI77bbbiMsLIwXX3yRsLAwtm/fTps2bZyPDxo0iJo1azJnzpxCn1/YzEeDBg04e/bsZTtfUlarlaioKHr16oXJZCrVa5emzTGJPDJnCxPUOTxoWMFa7XoezHkJg6JjvGSh7KuRN5XpzEdlGKuKQMbKdTJWJSPj5boyGyvNDrM6YUo+whdej3IkbCTzt5xAB8wqKBdmPhwTIO8Nbc1t19Yt6moVQlmNVWpqKoGBgS4FH1dd50PXdbKzswkNDSUoKIioqChn8JGTk8OaNWuYNm1akc+3WCxYLJYC95tMpjL7z1aW1y4NHZvWwdfLg8/S7+B+NYpb1F1cqxzjH70Rdntum9o+Fjo2rVPmGdUVfawqEhkr18lYlYyMl+tKfaz2LYPkI+gefjz6VASKhy83N6/NpKX7SEi9+MG5Mu5ELO2xKsm1ShR8vPLKK/Tt25cGDRqQlpbGggULWL16NZGRkSiKwvjx45k8eTLNmjWjWbNmTJ48GS8vL4YPH17iL6I6M6gKZ9KygTr8rnVggGETo4xLGW8d52wjVU+FEKKM6TqsnwGActPj4JH7ab7vdfXo3SqYzTGJnE7Loo6PB+1D/eV3cgmUKPg4deoUI0aMID4+Hj8/P66//noiIyPp1asXAC+88ALnz59nzJgxJCUl0aFDB1asWCE1Pq7AjKGtefaHaD6xDWSAYRP91U28rdxLrF5Hqp4KIUQZ2xWbzKKFP/DauW1gsECHUXkev7RKtSi5EgUfX375ZbGPK4pCREQEERERV9MnQW7V06Z1vOn/IayxX083wy6eMPzGa7aHWTKuK+Ehfu7uohBCVFm/bI+j2+m5YADa3A/eddzdpSpFTsKpBGbZBwJwr2F1ngPnhBBCXD27prPx8Dlmr4/hu03H2HkimX+iN9DDsBM7Kv+GPcTu2BRikzLd3dUqQw6Wq8AcVU8z/Tpyxn49tZN3MdYzigDvIe7umhBCVAmRe+KZtHRfgRNt3zMtBAMss7dn3JxYIBaAo1MLr24qSkaCjwrMUfXUbFBRDrwEC4bzkDkKxZIDeLq7e0IIUalF7oln9NztBc5yaaCcYoC6EYBPbP0BMKoKb99TfGVT4TpZdqngLEYDiqJA875QuwVKdhrbf3mXXu+uYe6mY1JiXQghroBd05m0dF+hh8iNMvyKUdFYY7+ePXoTABaN7cLgNiHl28kqTIKPykJV2dX4IQAaHJjN8dOJvLpoD12nrZRD5oQQwkWO/I73og4UWGoBqE0S9xjWAPCRbVChh8yJqyfBRyXx3aaj3PlXPWL1QGorKdxtWAvknqD45NztfLfpqHs7KIQQFVzknni6TlvJsM83MXPV4ULbPGb8HYtiY6vWnM16C+r7e1Lb20KAt7mce1u1SfBRCdg1nQmL9mLDyOe23GSnUYalGLA720xYtFeWYIQQogiO/I7CZjsc/EjnAcMfQO6sByhMu/N61r3Ug2A/ybMrTRJ8VAKbYxKd//7e3p1zug8N1TP0v5AQVVg7IYQQuYrL77jUQ4bl1FCy+UdryGqtNcF+HnRoEoDFaCiXflYnEnxUAqfTLkbqWVj40tYXgHHGxShohbYTQgiRa3NMYrEzHgBeZPGwMff0dcesx8QBLaVkehmR4KMSqOPjkef2t/bepOpeNFPjuF3dWmQ7IYQQrn0wG2b4k5pKBjFaXXZ438KsB9pWqkPiKhsJPiqB9qH+BPtdDCzS8GK2/XYAxhkXATrBfrkHGwkhhMjrch/MzFh5wvgbAF+rQ/h4RHtCanpJRdMyJMFHJWBQc6f/Lp38m23rQ4ZuIVw9Sg81WqYHhRCiCI4PcEX9hrzHsIa6SjIndX/mZ3Vm0EfrGTBzHV2nrSrXflYnEnxUEn3Cg5n1QFvnDEgyPsy13wbAe0Er6NMqyJ3dE0KICsvxAQ4oEIAYsTHauASAz2z9yblQ+NuoKswY2roce1m9SHn1SqRPeDC9WgaxOSaR02lZhBjD0Bf+Qc3EnRze8juvRgfw8h0tuL5+TXd3VQghKhTHB7j857g86LWR+tpZzuh+zLff6rx/0dgucnp4GZLgo5IxqAqdwgIu3nF8JGz+DOWvd9h45r/8sj1Ogg8hhChE/g9wdWoYabvkZUiDz+z9ycaMooAuJZPKnCy7VGKxSZnsb/IwumKkSdo22ir/snTnSfbEpcjxz0IIUQjHB7hBrUPolLkaS9oxkvAluu6dvDkknOtC/KSiaTmQmY9KzJEMNdXYlfuMq/mPcSGPZDSn/4frnG3k+GchhCiEZoe1bwHg0+NpfrilJ4qiMLx9Q3LsmhQWK2My81GJzRjaGqOqMMs+EJuu0sOwk+uVQwAYFIURHRuy8fA5KbsuhBCX2BWbzPsfvgXnDoJHTYwdnsg9PRxQFEUCj3IgwUclNrhNCIvGduGYHsRirQsATxkXAmDXdb7ddJxhn2+Sk2+FEOISC7ed4PZzc3NvdBwDHr7u7VA1JMFHFTHTPhi7rtDTsIPrlCN5HktIyWL03O0SgAghqq3YpEx2x6awJy6FtJ2LaaGeIB0v9jUcJjlybiA5H5VcgLeZ2t4WvP1a8Pu5mxnAWp4y/sLj1uecbXRy97ZPWrqPXi2DpBiZEKLauVgwTOdX84+gwte23rz92W5nG8mRKz8y81HJBft5suaF7gy6IZh3swdh1xV6GbbTSonJ004H4lOyeC/qX8kDEUJUO44cuV7qNsLVo6TrHnxx4ZBOKShW/mTmo5KL3BN/SdGcYJZonRliWM944y88bv1vgfYzVx1i5qpDBPt5MHFASzk4SQhRLQxuE0LT2jVQPnsJgK/tt5OMDyAFxdxBZj4qscg98Yyeuz1Ptb6ZtsFoukIvwzZaKUeLfK7kgQghqhufo8tppR4jTffkC9sdKLIC7TYSfFRSdk1n0tJ95F88OayHsFTrBMBTxl+KfL7jeZOW7pMlGCFE1adp1IueAcDvXgN5fkgnKSjmRrLsUkltjknMM+NxqQ9sQxigbuR2w1Za2o6yT29caDtHHsjmmMS8JduFEKKq2f8rprP70C0+3PufqShe/lJQzI1k5qOSOp1WeOABubMfv2odAXjG+PNVXUsIISq7XScSOf7LawAoHUajePnn/lsKirmNBB+VVB0fj2Iff99254WdL9sK1P0o6bWEEKIy27/yOxraYshSa0CnMe7ujkCCj0qrfag/wX4eFJUvdVgPYdGFqqfPGn8stI0CBPt50D7Uv2w6KYQQbuIsKhabRNuYzwD4Rr+DPYmqFBWrACTno5IyqAoTB7Rk9NztKFAg8RQgpuVYbAc20MOwk7a2f9muN3c+5ghaJg5oKUXHhBBVjqOoWD91Ex+Zj5OqezEzqzeT5eDNCkFmPiqxPuHBzHqgLUF+eZdNgnwtzLq/Dc8NuwNDm/sBeMkj786XID8PZj3QVup8CCGqpBlDW2NWdZ4x/gTAl7a+pFIDkKJiFYHMfFRyfcKD6dUyiM0xiZxOy6KOT+4yimM2Q+n2AuxcQHttF5/dksVb+wMZ2bkx97ZrwLZjSSyOjivwHCGEqOwGtwmhXXIk9decJEn35kt7X+djUlTM/ST4qAIMqlL0VtmaDaHtg7D1S5rv+4CDZ5/nz39O8dGqQ3m26krFUyFEZWLX9DwfutrU98nbwJZDne3vAfCpfQDpeKEooEtZowpBgo8qLjYpk7RmT3DNjrk0To+mi7qHVQeuK9DOUfFUlmKEEBVd3mMlcjWqZeHZFpc0ip6LOe0E56jJtrp382b7Zny/5QTxyVlSVKwCkOCjinMkXU00dudh43L+a/yR9TnhkG+fTP6Tb4UQoiJyHCuRfwLjVGpuIPLHP6foe20grHkLAN/bX+KHjj1RFEWKilUgknBaxTlOcvzYNojzupm26iF6qtsLbXtpxVMhhKhoijpWAvIeGTHnw9cg7ST41sd00yMoFw5xkaJiFYcEH1Xc4DYhLBrbhTPUZLa9DwDPGX9AQSvyOVLxVAhRERV3rIRDRnoq/VLm597o9jwYLeXQM1FSEnxUI5/a+pOqe3GteoKB6oYi20nFUyFERVTcByNNhxPpMEJdTqCSygmC2FOnvxQUq6Ak+KgGArzN1Pa20DAkhK+VgQA8a/wJI7Y87aTiqRCiIivug5FVU/hsdzajjL8C8E7OnfT/6G8GzFznzH0TFYcEH9VAsJ8n617qwZJxXbh28Auc0X1ppJ5mqGG1s41UPBVCVHTFHyuh86RxKX5KJge0+izROgOgKkhBsQpIgo9qwmI0oCgKvVqHcabNfwB4yvgLHmQDUvFUCFHxOY6VgPz79SBETeJhQyQAb9mGol14e9N08DDJW11FI9+Raqhl/6fR/RpQV0nmQcMK+rQKYt2Lt0rgIYSo8Io6VmKsYSGeSg5btWb8obXN89grC3ezcEccGw+fw65JlbGKQOp8VDOxSZkkZVip1Xo89df8l9HGpQyO6cM/8U3RdahVw0T9Wl7u7qYQQhTp0mMl1h86w6+r13GvmpvXMd16H/nnRRIzrDzzfTQg1ZwrCgk+qhlH4pVKXZabQ2imxnFvzi/0/9DkbCMnPQohKjqDqlDDYmD53lP81/gjRkUj3ucGonNaFH7M9wVSzblikGWXasZRdExD5S3bvQA8YlhGHZLkpEchRKXyy/Y4zGd2M8CwCU1X2B9yD8pl8uUvLUYmSzDuI8FHNeMoOgawQmvHVq05nkoO440/s2hsFwa3CXFzD4UQomixSZnsjk1hT1wKS3ee5AXj9wAs1Tqz197QpYPjpJqz+8mySzWmKApTrffxk+X/uNewmiPJhyDkRnd3SwghCnCcYjvs803O+zqpe+lm3kWObuBt292c2F2ytzSp5uw+MvNRDTmKjl0X4seQwXfzt7kDRkWj4Y633d01IYQoIHJPPF2nrcwTeChovGTMLaM+334rJ/S6qIqOUXF9KUWqObuPzHxUQ46iY2aDiqIo6KHvo8/qjMeh3zm07Q/+t82b53s3dXc3hRCiyFNs+6ubuEE9QrruwQe2OwF4NtxOm5s6cjrdyuu//UNSRk6huacKubWNpJqz+8jMRzXlKDoGoNS5FqXNAwCYV05i45GzLIqOd2f3hBCiyFNszViduR6f2AaQqPg5H2sf6s+QtvWZPCQcKFiMTKo5VwwSfAhikzL555pxaAYPGmbs4jZ1O7/tTuBEOuyJS5VDmYQQblHUKbYjDFE0UM+QoNfiC/sdhAbUINDbjM/FigFFFiOTas4Vgyy7CGftj+eNvRlrXMKLxgX0yWjN27uNvL07d41Van8IIcpbYQmhvqTzH+NCAN613U0WFp7q2ZTbWtTmzxWRedpeWozsdFoWdXxyl1pkxsP9ZOZDOGt/fGIbSKLuTTM1jvsMuQGJQWp/CCHcpLCE0DHGJdRUMjig1ecnezcA6vp6YjEW/nZmUBU6hQUwqHUIncICJPCoICT4EM7aH2l4McN2FwDPGH/Cm0x+HtVBan8IIdwi/ym2IZzhYcNyAKbYhqGjEiyJo5WSBB8ij/laTw5rwQQqqYw2LnF3d4QQ1ZBd09l4+By/7jrJHeHBzoTT/5p+xKJYWW9vxWqtNTowrkeYzGZUQpLzIYDc2h+B3hZ8PWrwl/84wo5P4FHDMs5pE4AAd3dPCFFNRO6JZ9LSfQUSTa9TjnCnYR0Ak23DcexbmbBoL/d3bFzOvRRXS2Y+BAA7TyRjVOHI2Qwi/m3M39q1eChWMpZNZNhnm9gVm+zuLgohqjhHTY+CO1x0/mf6FoBf7F3Zq4cCyHlUlZgEH8L5Hz4hNfvCPQrT7PcD0Px0JBkxm/l41WE5hEkIUWaKqukB0FfdTHv1AOd1M9OtQ533y3lUlVeJgo8pU6Zw00034ePjQ506dRg8eDAHDhzI00bXdSIiIqhXrx6enp50796dvXv3lmqnRekp6j/8bj2U/T65B9BNMH1H5N54Okz+g0/XHJa6H0KIUldUTQ8LObxsnAfAZ/b+JBBw2ZNrRcVXouBjzZo1jB07lk2bNhEVFYXNZqN3795kZGQ420yfPp13332XmTNnsmXLFoKCgujVqxdpaWml3nlx9Yr6D2/VFB4+cx9ZuokO6n5uV7dyNj2HKcv2O+uCCCFEaSnqkLeHDMtpeKGg2Ce2/tzbrj7XhfhR29tCgLe5nHspSkuJEk4jI/MWcJk9ezZ16tRh27Zt3HLLLei6zowZM5gwYQJ33plba3/OnDnUrVuXefPmMWrUqNLruSgVRZ/qqBNPAJ/b+/Ef4yJeMX7HqpzW5GCipqcJu6ZLhrkQotQUVtMjgBTGGRcB8JZ1KOfxYEib+nRs4k+OXcNiNJRzL0VpuardLikpKQD4++fusY6JiSEhIYHevXs721gsFrp168aGDRsKDT6ys7PJzs523k5NTQXAarVitVqvpnsFOK5X2tetzAK9jFgMBVdZjYpOhk1hlm0g9xpW00g9zSOGZXypD+B8Tg6bDp2WvfUXyM+V62SsSqY6jVeb+j40qmXhVGoWOmDT4FnDT/go59mjhfK70oVGtSy0qe+DzWZDBaxWzfn86jRWV6usxqok11N0Xb+iLEJd1xk0aBBJSUn89ddfAGzYsIEuXboQFxdHvXr1nG2feOIJjh07xvLlywtcJyIigkmTJhW4f968eXh5eV1J10QpOJEOb+82oqAzRP2Ld82fkK578EOT6QTUrOnu7gkhqqDEbMi48P715z9x/GR4BYOi83P9CcR7XkMNE/hb3NtHUbTMzEyGDx9OSkoKvr6+xba94pmPcePGsWvXLtatW1fgMSVfNpCu6wXuc3j55Zd59tlnnbdTU1Np0KABvXv3vmznS8pqtRIVFUWvXr0wmUyXf0I18cc/p3jm+2gAZ+KpSdHxMelkWGGp3oUHtShaq4fxPfQzL2hP8NXIm2Tm4wL5uXKdjFXJVLfxava/FRf+pTPXNBeDovO7/Sb+e6iVs83B13sX+tzqNlZXo6zGyrFy4YorCj7+85//sGTJEtauXUv9+vWd9wcFBQGQkJBAcPDFEwNPnz5N3bp1C72WxWLBYikYyppMpjL7ASrLa1dGfa+vj6Ia8hb2McDkG+28vMVAjmZgkvVBFlomcqe6lkWmvny4OoBXvCxcX7+mW/tekcjPletkrEqmuozXjKGtee7HnfRkC10Ne8nWTUyxDQdya3q8fc8Nlx2H6jJWpaG0x6ok1yrRbhdd1xk3bhy//PILK1euJDQ0NM/joaGhBAUFERUV5bwvJyeHNWvW0Llz55K8lChnfcKDWffircx/vCOPdGkMgFHFuaVth96MhfauqIrOa6Zv2XTkHL9sj3Nfh4UQVc7gNiEsHnUjrxrnAvCZvR8n9NwPrlLTo2op0czH2LFjmTdvHosXL8bHx4eEhAQA/Pz88PT0RFEUxo8fz+TJk2nWrBnNmjVj8uTJeHl5MXz48DL5AkTpcZz+2CksgJsa+ZETsy3P45+ZR9Bf30rz7L0MUDeydKeFu2+sj65DrRom6teSHB0hxNUJ3PUZddUzxOv+fGwbiKLAlWUmioqsRMHHrFmzAOjevXue+2fPns1DDz0EwAsvvMD58+cZM2YMSUlJdOjQgRUrVuDj41MqHRbl47Zr6/J7DHw18ibOZtp4ekE0/2T48L5hAM+ZfuRl0zxuy2hL/w8v5vy8f19r6vjknjAp23CFECWxKzaZT5eu5YOzMwH4zvdRXu3aju+3nCA+OUtqelQxJQo+XNkYoygKERERREREXGmfRAXSPtQfk8mErsNzP+7kc3s/hhpW00A9w2jjEt6x3ets+/SCaACC/TyYOKAlfcKDC7+oEELk88v2OPqc/AiDIQutQUf++/ArKKrK8PYNpaZHFSRnuwiXDG4TwqKxXcjGzOu2BwB4wvArjZX4Am0TUrIYPXc7kXsKPiaEEA6xSZnsjk1hT1wKsdF/MMCwCTsqR26ayJ6TacQmZaIoigQeVdBVFRkT1VOU1o419uvpZtjFROM3PGx9Acfx1pC7XVcBJi3dR6+WQbIEI4QolOOoBhWNX81fggoLbD2YMC8JyF3SPTq1nxt7KMqKzHwIl+2OTUZVQEchwjaSHN1AD8NOeqrbC7TVgfiULDbHJJZ/R4UQlcKMoa0xqgoPGKJoqR4jRffibds9QO7W2hlDW7u3g6LMSPAhXBK5J55XFu5Bu5D2E6MH84U99xPJROM3WMgp9HlFnx0jhKjuBrcJYWKPQJ4z/gjAW7ahJJFbXPKF26+RrbVVmAQf4rLsms6kpfvIn2480zaYeN2fhuoZRhl+LfS5hR0WJYSovuyazsbD51gcHcf7fxzEa+3/4atksltrzDx7T2e7ycv2S95YFSY5H+KyNsckXqx8eolMPHjTej8zzR8yxriYX7SuxOp1nI/X9rFICXYhhFPknvg8lZRvUvbzo+UvNF3hf9ZH0PJ9Hpa8sapLZj7EZRW3dPKr1pEN9pZ4KFYmGr/J89iZtGw2xyRi16RCkBDVXeSeeEbP3e4MPIzYeN00G4AF9u5E600LPEfyxqouCT7EZRW/dKLwmu0hrLqBXobt9FK35nl02Oeb6DptJb/vOumcat14+JwEJEJUI4Ut3Y40rKCFeoIk3ZvptvuKfK7kjVVNsuwiLqt9qD/Bfh4kpGQVyPsAOKTX53N7P8YYlxBhmsP67HAyuRiwxKdkMWbejjzPkUJkQlQf+Zdu65DEeOPPAEyz3UcyRVfAlryxqklmPsRlGVSFiQNaApdW88jrQ9sQTmi1CVHO8ZTxl8teUwqRCVF95J+9eNU0Fx/lPNFaGN/buxf6HIXcDymSN1Y1SfAhXNInPJhZD7QlyC/vp5A6PhZ8PYwEB/rzmu0hAB41LOMa5Xix13PMoExauk+WYISo4i6dvbhF3clAw0bsusKr1ofRC3kbcnzImTigpSSbVlGy7CJc1ic8mF4tg9gck8jptCznIXI2TWPZ7njGf59BpP0m+hi28IbpK+7Nea3QXywOlxYi6xQWUH5fiBCiXDmWbpNSUnjD+BUAX9v7sEdvUmj7IFmWrfIk+BAlYlCVAoGCQTVQ19cTgEnWB7lZ3cVN6r/cY1jDD/Yel72mJJQJUbU5lm4PL3iBhuoZTur+vGu7O0+bZ25rRuPAGnIydjUhyy6iVDjWZeMJcP5Sedk4H39SL/tcSSgTomrbFZvMn2vW8KTpNwAmWh8ig9wPLKoCU4aE8/RtzRnUOoROYQESeFQDMvMhSoVBVXi4S2Nmrz/K1/Y+3GVYR0v1GP8zfcsz1rGFPkchd3pVEsqEqNoWbjvB0FPvYFDt2JrfwcPtx9I/PZva3hZaN6yJl1neiqobmfkQpWbigFa80rcFdgy8bH0Uu64wxLCeW9SdBdpKQpkQVVtsUia7Y1PYE5eCIfpb2qn/koEHh298DR8PEzc2qkXnpoESeFRT8l0Xpapz00AAdulN+dreh0eNy3jT+BXDTO8Rm3Ex1pWEMiGqtq7TVgFQm2T+sHwLCrxtvYfZs48ARwA4OrWfG3so3ElmPkSpCvA2U9vbwnX1/fDuO5FTah0aqGdYdv1fvDE4nGZ1vHljcDjrXrxVAg8hqrAZQ1tjVBUmmubgp2SySwtljv12AIyqwoyhrd3bQeFWMvMhSlWwnyfrXuqB2aCiKAp67Y9g3j34RH9OZvP2HDztyaHT6bLUIkQVN7hNCG0y1tHoj7+x6SovWR93Hhy3aGwXwkP83NxD4U4y8yFKncVoQFFyg4u42l1JDhsMuka3A69jxMbSnSfZE5fC7tgUYpMy3dtZIUTZOJ9EvfUTAPjUPoB9emMU+cwhLpCZD1Gmuk5bRQC384flD65RjvK44Xc+yRhI/w/XOdvIuq8QVceu2GSm/L6fj7y/wv/8GY5Sj5V1H+LN9mF8v+UE8clZBHib3d1N4WYSfIgyNWNoa577cSevWx/gXfMnjDf+zArtRg7rIRhVhbfvucHdXRRClKJftsehHF2Dv/l7QKHeyC/4qXFnFEVhePuG5Ng1LEaDu7sp3EyCD1GmBrcJoWkdb/p/qDHAvpEehp1MN33GPTkTWTT2Fln3FaIKiE3KJCnDiqbrLNt+iB+NnwNw9toRxJvCqZV8nvq1vFAURQIPAUjwIcqNwgTbYyxXX+BG9SAPGyKBW9zdKSFEKXBsqwV41TifhsYzxOkB9N7RjYwduUussrwqLiUJp6LMObbfBoY0Ye91zwPwvOkH0k4eYNhnm9gVm+zeDgohrsrDXRoDcKNygEcMkQBMsD7qLKHueFwIBwk+RJlzbL9dPLYLHe96Fr1JdzzIod6a59h05Ay/bI9zdxeFEFfIrulE7knAg2zeNn2Cquj8YOvGaq21s03kngTsmu6+TooKR4IPUS4c229jk89z4KY3sRu9aJS+kwcNUbL1VohKbHNMIvEpWbxg/J5Q9RQndX/esD2Qp018ShabYxLd1ENREUnOhyhXjrXhBwxDecM0mxeNC1id2Zr+H+Y428jasBCVg13T+WV7LB2Uf3jEmLvc8pL1cVKpUaDt6bSs8u6eqMBk5kOUK0fJ5e/sPdlob4mXks1bpk9Q0aTkshCVSOSeeLpOW8lv2w4x3fQpAPNtPVirFb59/uCpdDYePifLLwKQ4EOUs8FtQlg0tgs6Ks/bRpGue9BePcBjht9YNLYLg9uEuLuLQojL+G7TUZ6cu534lCxeNM6nkXqaOD2AN233F/mcmasOMezzTXSdtpLIPfHl2FtREUnwIdwmTq/NJNuDAPzX+COWxP1u7pEQ4nLsms6ERXsB6KTuZaQxCoAXrE+Qjtdln5+QksXoudslAKnmJPgQ5W53bDKqAjrwo70bf9jbYFFsBK54it3HTsv2WyEqMEfiqC/pvGOaBcBcW0/Wa9e59HzHosukpftkCaYak+BDlKvIPfG8snAPF3/nKLxsfZxE3Ztaafs5+vNENh45J9tvhaigHImjr5u+pp6SSIxWl8nFLLcURkd2wFR3EnyIcmPXdCYt3Uf+zzpnqMkr1scAuCNlPm2Ug7L9VogKxq7pbDx8joOn0hiobmCQYQM2XeUZ61gy8biia8oOmOpLttqKcuOoB1CYSK09v9i7cqdhHe+aPqZ/xhQ5+VaICiJyTzyTlu4jPiWLYM4RafkKgJn2wUTrTQu0969hIjHDetnr1vG5sqBFVH4y8yHKzeU+5URYRxKnBxCqnuI14zcAsv1WCDdxzHT839K9zp0tChpvmz7BT8kkWmvCTNvgAs9TgDcGhRPs54FSxLUVINjPg/ah/mX4FYiKTGY+RLm53KecVGrwbM4Y5pvfYKhxNau1G7h50GMoCmw8fI72of4Y1KJ+nQkhSsulMx2XesQQSRfDXjJ1C89Yx2LL9xYS7OfBxAEt6RMejKoqjJ67HQXyLLU6/gdPHNBS/j9XYxJ8iHLTPtSfYD8PElKyCuR9OPytX8vH9oGMMy5mqulz+ixsSjwBQN5fbEKIshG5J57Rc7cX+D96rXKMF4zfA/Cm7X5i9Iv/D8f1aEqXpoF5PiD0CQ9m1gNtCwQxQfL/WCDBhyhHBlVh4oCWhX4acmgU4EVCk/FE79xNa/UI75pmcb/1FTRUZ32AWQ+0lV9cQpSBopLCPcniQ9OHWBQrf9jb8J29Z57Hm9X1plNYQIHr9QkPplfLIDbHJHI6LYs6Ph4ygykAyfkQ5czxaSjIL+8STLCfBx8Oa82fz3Zj+f5ExlvHkqFb6GTYxxOGX4HcYEUH/rd4r9QHEKIMFJUU/j/jtzRVT3JKr8nz1lGQL5ujuCVVg6rQKSyAQa1D6BQWIIGHAGTmQ7hBcZ+GNh4+x5m0bM4QTIRtJG+ZPuO/xh/ZoLVilx4GwJm0bDbHJBb6SUsIceUKSwq/Q93EcOMqNF1hvHUsSfjmeVwSR8WVkJkP4RaFfRqyazrrD511tvnR3o1f7R0wKXZmmj7Ah4v1PpbtiZdDqoQoZflnMEI4w1TTFwB8bB/IRq1VgedI4qi4EhJ8iArBcULmzFWHLrlX4RXrY5zQatNQPcNU02c4MkW+2XhMDqkSopS1D/Wnto8FAAN23jd/hK+SyXatKTNsd+VpG+Rr4RPJvxJXSIIP4XaO7PrC1ppTqcE463+w6gb6GTZzv+HPPI/LIVVClB6DqnAmLRuA8cafaaf+S6ruyVPWcXm21c5/vCPrX+opgYe4YhJ8CLcqKrv+Ujv1pky13QfAa8ZvuVY55nxMDqkSonTNGNqa7upO/mNcBMAr1seI1esAoCq5j0viqLhaEnwItyqu5PqlvrTfceH0WyszTR/gxcXnOA6p+np9jAQgQlylwU10vvD5DIBvbbfxq9bJ+diScV0Z3CbEXV0TVYgEH8KtXD9YSuE565Oc1P0JU+N50/Ql+SuFvP7bP5IDIsQV2hWbzAOfriPjuwcxZiexR2vMG/YHAFBkkkOUMgk+hFuV5GCpZHx4KmccNl1liGF9gfwPkBwQIa6EXdP5aNVhup6YRY3T29DMPrxqep5rQmrz5pBwrgvxo7a3hQBvs7u7KqoIqfMh3OpyJdcVoK6vBVBISM1iq96Cabb7mGCax2vGb9ijNWbnJadq6heeM2npPnq1DJJ1aSGKEZuUyW+74vn8ryO0ztzIk+bcgn4v2kbRq2sHBrauRwP/Ggxv35Acu4bFaHBzj0VVITMfwq0cJdchf83Ei7cjBrYiYmBL5/2f2/uxzH4TFsXGR+YPqEVqnuc5ckA2xySWXceFqAK6TlvFlGX78cw4wTumWQB8ZevDj5lteWvFv9w8fTUAiqJI4CFKlQQfwu2KKrke5OfhPMelT3gwU4aEkzuRofCCdRRHtCDqK2d53/QRKlqB67qeTyJE9WPXdGp6mvAgm89M7+GnZLJDa8oU23Bnm5qeJkniFmVCll1EheDKAVTDOjRiUJsQ5m06xhu/72e0dTyLzK9xi2E3T+u/8J7t7jzXLEk+iRDVzeaYRJLP5zDD9DnXqsc5o/syOudprJe8LSSft8pRBqJMyMyHqDBcOYDKy2zk4a5NADigN+Rl62MAPG38hR7qjjxt5bwJIQq3KzaZ1xbv4WFDJIMNG7DpKuNyniaBgkGGzCCKsiDBh6h0DKrCw10aA7BI68o3tl4AvG+aSRPlJAAPd2ksyaZCFOGX7XH4n9nCBON3ALxpu5+/9WsLbSsziKIsyLKLqJQmDmhFsK8Hk5ft53XbCFqox2mvHuBz0zus7baAh3sWPABLiOosNimTpAwrigKbonfzrfl9jIrGIntnZtv7FGivkJt3JTOIoixI8CEqrc5NAwGwYWRMzniWWCYQpsZT++gkdh3/gimRB3n5jhZcX7+mezsqRAXQddoqACzk8IN5GrXVVP7RGvKS9XHy7zVz3JITa0VZkWUXUWkFeJup7W3huvp+PDOkC2/XfI0szPieWEnybxFsPHKOX7bHububQlQIM4a2xqjCNNNn3KAeIUn35gnrM2RhKdD20p1mQpQFmfkQlVawnyfrXuqB2aCiKAq3NBtKfLRO6NpnuOXUN9yh1mTpTjN331gfXYdaNUzUr+Xl7m4LUWbsml7ojrFdscl8v+UEnzZeQ8+TG7DqBsZYn+aEXheAl/pcww0NahW500yI0lbimY+1a9cyYMAA6tWrh6IoLFq0KM/juq4TERFBvXr18PT0pHv37uzdu7e0+itEHhajAeXCwRM3T19FjxV1+czWD4B3TJ9QL3M//T9cx4CZ65zTzkJURZF74uk6bSXDPt/E0wuiGfb5JudZR79sj8P76HJ6xOUeGBdhG8lG7WJe1NTIA6Sczyl2p5kQpanEwUdGRgY33HADM2fOLPTx6dOn8+677zJz5ky2bNlCUFAQvXr1Ii0t7ao7K0RxcqeVFabZ7mON/Xo8lRw+N79DXRIxqgozhrZ2dxeFKBORe+IZPXd7gROi41OyeHLudrZvWc8M00eois4cWy++s99W4BqTlu6TgmKi3JQ4+Ojbty9vvPEGd955Z4HHdF1nxowZTJgwgTvvvJPw8HDmzJlDZmYm8+bNK5UOC1GUwW1CWDS2C3YMjLM+xb9aCEFKEl+a32bJE61pUrsGwz7bxK7YZHd3VYhSY9d0Ji3dV+jZSAABpPCRMp0aSjbr7a143Tai0HZyJIEoT6Wa8xETE0NCQgK9e/d23mexWOjWrRsbNmxg1KhRBZ6TnZ1Ndna283Zqau45HVarFavVWprdc16vtK9bFVXWsbLZbACkK148Yn2eRebXCFePkrLyKd6p9Sobj5zjp60naF7bi23Hkjibnk2gt4UbG9W64qnmyjpW7iBjVTKujNfmmEQS089jyXf0il0Dg57DF+Z3aKCe4ahWlzHWp7FhBHSMChjyffw8nZKB1epbyl9F+ZCfLdeV1ViV5HqKrutXPM+mKAoLFy5k8ODBAGzYsIEuXboQFxdHvXr1nO2eeOIJjh07xvLlywtcIyIigkmTJhW4f968eXh5SXKgKJnkbHh7t4GaZuhUVyPp5GFm6m9iUazM1voxKed+vI06T15rB6CGCfwLJvsLUWkdT4clx1QGNrRxR/xMwjK2kqR7c2fOJGL03N0rz11no4G3mzsqqpzMzEyGDx9OSkoKvr7FB7FlstvFkQDooOt6gfscXn75ZZ599lnn7dTUVBo0aEDv3r0v2/mSslqtREVF0atXL0wmU6leu6qpzGN11wANs0FBURSa/W8Fz6mj+NA8k4fV3/jXEMR82628vfvij77FoDvrGrw3tDW3XVu3RK9XmceqvMlYlUxx4/XHP6eYumw/Cal58zxsGth1Bc/9PxBm3Eq2buSJnGcuBB46oPD+XgOXTvQpQF1fD5aPv6XSJpvKz5brymqsHCsXrijV4CMoKAiAhIQEgoMv7g8/ffo0desW/gvdYrFgsRT86GkymcrsB6gsr13VVMaxUg0XtxuO7dGUT9covGtN4FnTT7xh/IpTei1Wam2d7bPtub9sFeD/fjtA7/CQK/oFXBnHyl1krEom/3hF7olnzLydF/I8Cv6sPmCI4jHj7wA8bx3FFmfp9Ny2Vu3icxz/erlfKzws5tLvfDmTny3XlfZYleRapVpkLDQ0lKCgIKKiopz35eTksGbNGjp37lyaLyVEofJvN/xo1SFqepn4wD6E723dMSg6H5k+oLVyqMBzdSTpTlR8l0sw7aHuYJLxawCmW+9lidal2OtJQTHhDiWe+UhPT+fQoYu/uGNiYoiOjsbf35+GDRsyfvx4Jk+eTLNmzWjWrBmTJ0/Gy8uL4cOHl2rHhcjPsd0w/y/ls+k5gMIE2yPUUZLoYdjJl+a3uCsngqN6wV+4coqnqMg2xyQW2FLr0EY5yMem9zEoOgts3fnYPqjI69T0NPHR/W3p2ETqeojyV+KZj61bt9KmTRvatGkDwLPPPkubNm147bXXAHjhhRcYP348Y8aMoV27dsTFxbFixQp8fHxKt+dCXOJynwYBdNXIWOvT7NSaEKCkMcc0jUBSCrSTUzxFRVZUcBymxPGV+S08lRxW22/gVdsjFLYko1z4M/Wu6+jSNFACD+EWJQ4+unfvjq7rBf58/fXXQG6yaUREBPHx8WRlZbFmzRrCw8NLu99C5FHcp0EHuwaZePBIzvMc0+rQSD3NV+bp1OB8nnZyiqeoyAoLjoM5x7fmKdRS0tmhNWW09WnsFya2A73z5nHIMouoCORsF1EluLpU0q1ZIGsOwkjri/xsjuB6NYbPTe/wsPUFsjHzcJfG8klQVGjtQ/2p7WPhTFpufaSapPGNeSr1lEQOafV4OOd5DBZvwgNrkJCSxcKxnTmReF7ObREVigQfokpwdankye5N6dI0kMnL4KGcF5lnfpPOhn3M5ENienyEl5cXvd5dw8jOjRnWvqH8khYVjkFVnIGHJ1l8ZX6LZmoc8bo/D+a8RDI+kG1jybgu5Ng1LEaDHKgoKpxS3e0ihLu0D/Un2M+jkBXuXAoQ7Jf7qa9z00AA9uhNeCznObJ0E70M26j/1wv8b9EuDp5O59VFe5yHcglR0cwY2hoPJYfPTO/SVj1Esl6DB3Ne4iSBqEru44qiYDEaLn8xIdxAgg9RJRhUhYkDWgIFU+wctycOaIlBVQjwNlPb28J19f0YOPgeXjU/j1U3cIe+lgjjHLiQtuo4lOu7TUfL68sQoli7YpMZ9tkmwvxNbG76DTcb9pCue/Bwzgsc1OsDsGRcVwa3CXFzT4UoXqVddrHb7SWuS2+1WjEajWRlZWG328uoZ1VDWY2VyWTCYCibT2N9woOZ9UBbJi3dlyf5NMjPg//1uxY/TzOLo+Oo4+PBmhe642kyoOkwYWE4OepoZpg+YqQxigw8mW4biiNsmbBoL/e1byRLMMLtftkex+Yjp1EXTsU3eRVZuonHrM+zQ2+GosCVH5YhRPmqdMGHruskJCSQnJx8Rc8NCgrixIkTRZZ7F7nKcqxq1qxJUFBQmXwP+oQH06tlkLPCaR0fD5Iycnj9t7wBSbCfBxMHtMTPM3cnwBKtM96280w2fckY4xKsGHjPdo+z/eaYRDqFBWDX9DzXlp0xoqwlZsOeuFRMJiO/Rscy3fQprZLXoakmntWfJ7lOe97s1Ijvt5wgPjmLAO/KX6VUVH2VLvhwBB516tTBy8urRG9gmqaRnp6Ot7c3qiorTsUpi7HSdZ3MzExOnz4NkKcEf2kyqAqdwgKA3MJjY+cVLDyWkJLF6LnbeaRLY+d98+w9sZDDRNO3PG1ciKarvG+/C4CtxxJJOZ9TYFYl2M+D1/pdUyZfhxAAk7YbYfsmFDTeNH7FXcZ12HSVMVnjWKGFw/k07u/QiOHtGzoTTIWo6CpV8GG3252BR0BAQImfr2kaOTk5eHh4SPBxGWU1Vp6enkDueT916tQpsyUYKL7wWO7xWrAwOi7P/bPtfVHR+J/pO54x/YwdlZn2Ifz5z2neXfFvgWvFp2Qx/vtoprfPnR3p2LSOLM+IUuGYZRvR1M68wwpvGL5kmHEVmq7wrHUMK7SbUBV4997WAJJgKiqVShV8OHI8vLxk21hl5vj+Wa3WMg0+Lld4TAcSM6z41zCTmJHjvP9Lez8MaLxims9zph/RUJl1YlCx1VMBHpmzBX9vTyYOaCkFnMRVidwTz6Sl+zibdp7QGjpTjV9wj2ENdl3hv9bRLNFyz8rSdPAwyQcpUflUyp9aydeo3Mrr++dq4bHBresV2CHzmX0A06z3AfCC6XvGGX6By4YfF5dzZIuuKAm7prPx8DkWR8fx/h8HGT13e27grGs8kXUx8HjGOoZFWtc8z520dB92TTJNReVSqWY+hCgJVwuP9WoZRPtQ/wL5HLPsA1HQeMH0A/81/YSXks00230Udl6Gg2M5Z9LSffRqGSRLMOKyHLMc+WfpVDSmGT/hTsN6bLrKeOtYftU6FXi+4yRmR56TEJWBBB+iynIUHktIySp0zkIhdxuuo9x0r5ZBbDp8jrHztpN8PneJ72P7YLKw8JrpW0Ybl+JJNpNsD6IXM2moI28I4qLCdkg5gtKiTmI2Y+U900f0M2zGqht4yjqWZVrHIl9DTmIWlU2lXHapjB566CEURUFRFEwmE3Xr1qVXr1589dVXaJrm8nW+/vpratasWXYdrUJKUnjM0V5VFWfg4fCVvS8vWx9F0xUeMq5gqvELVC5+zzQ9dztkfvKGICL3xNN12kqGfb6JpxdEM+zzTc7KuUUlRHuSxRemt+ln2Ey2bmSc9aliAw+Qk5hF5VNtg49L11g3Hj5XLmumffr0IT4+nqNHj7Js2TJ69OjB008/Tf/+/bHZbGX++tWRo/BYkF/eX85FnexZVMAw396TZ62jsesKQ42r+cA0EzO5QYpVU3K3Q+YjbwjVm2NWI/9yiiMvaObKgwUe8yWdueYp3GLYTYZu4VHr8yzXbiryNS49NkCIyqRaLrtE7kng9d/+KbToVFnuUrBYLAQFBQEQEhJC27Zt6dixIz179uTrr7/mscce491332X27NkcOXIEf39/BgwYwPTp0/H29mb16tU8/PDDwMWkzYkTJxIREcHcuXOZMWMGBw4coEaNGtx6663MmDGDOnXqlNnXU1kUVnisqJM9iwsYFmldybKa+cD0If0Nm6hFGqOsz5COJ3U9L85+XLqcI6onV7Z5z15/NM/9tUnmG/NUrlWPk6J78VDOi+zQm+Z71kWFzd4JUVlUu5mPPw+cY+y8HUV+GinvXQq33norN9xwA7/88gsAqqrywQcfsGfPHubMmcPKlSt54YUXAOjcuTMzZszA19eX+Ph44uPjee655wDIycnh9ddfZ+fOnSxatIiYmBgeeuihcv1aKjJH4bFBrUPoFBZQ5C/ryx1QF6m152HrC6TrHnQx7GWB+Q3qkMyp8wqaLm8I1U1RM6iubPO+dHmvsRLPj+ZJXKse57Rek3tzXmOH3gxQuLeJvdCfx6Jm74SoDKrVzIdd05n+x5FiP424Y5dCixYt2LVrFwDjx4933h8aGsrrr7/O6NGj+fjjjzGbzfj5+aEoinMGxeGRRx5x/rtJkyZ88MEHtG/f3lmlVLjGkScyeu52FArfXLtBu46hOf/ja/M0wtWj/GSO4EHrSxzVgwj0NvNY11BybLrzHJkbG9Vi27Gky866iMqlsF0qjhnUbJtreVyKAm34ly/Mb+OvpHNcq80I68sc03P/fxvQ6VJXZ9FRqFXDzPAOjWgcWEN+jkSlV62Cjy1HEzmVllPk4+7apaDrunMZZdWqVUyePJl9+/aRmpqKzWYjKyuLjIwMatSoUeQ1duzYQUREBNHR0SQmJjqTWI8fP07Lli3L5euoKoo6oK6Oj4Usq53GgTXYFRvKXTmT+MY0lcbqKX4yT+KxnOeITm/K1MgDea6nKrlJqQ7lscQnylZRu1QcM6jjb2vm0nV6K5t53/QRHoqVnVoTHs15nrP4OR9/umcYZP7L7Idukuq5okqpVssup9MK2ZJQaLvy3aXwzz//EBoayrFjx7jjjjsIDw/n559/Ztu2bXz00UcAxZ7gm5GRQe/evfH29mbu3Lls2bKFhQsXArnLMaLk+oQHs+7FW5n/eEfev6818x/vyMaXe7Ll1dtYPLYLM4a25qQSxN05EezSQglUUvne/Dr91Y0FrpU/l1kKkVVul8vnAJi/+ThBvkUv3wE8YlzGLNP7eChW/rC34b6cV/MEHq/0bcHo7rk5HzLLIaqaahV81PGxuNiu/HYprFy5kt27d3PXXXexdetWbDYb77zzDh07dqR58+acPHkyT3uz2VzgiPv9+/dz9uxZpk6dys0330yLFi2ch7eJK1dYnojFaEBRFAa3CWHR2C6cxY/7cv5HlL0tFsXKTPOHPHWZaqiOR6QyZeXkSj5HQmo2w9o3BApu8zZiY5JxNq8Zv0VVdL613cYo67OcJ/f3TpPAGgR6WxjQul4ZfQVCuF+1Cj5uauxPXR9zkZ9GynrbWnZ2NgkJCcTFxbF9+3YmT57MoEGD6N+/Pw8++CBhYWHYbDY+/PBDjhw5wrfffssnn3yS5xqNGzcmPT2dP//8k7Nnz5KZmUnDhg0xm83O5y1ZsoTXX3+9TL4GUdB5xYNR1mf53NYXgGdNP/Ge6WMsuLbEJyoXV2dGGwd6FdjmXZM05pimMdIYBcBU6338z/YwdgwYFJgxtDV//rcb61/qQbCfZ5n0X4iKoFoFHwZV4YXbmgCuFZ0qbZGRkQQHB9O4cWP69OnDqlWr+OCDD1i8eDEGg4HWrVvz7rvvMm3aNMLDw/nuu++YMmVKnmt07tyZJ598kqFDh1K7dm2mT59O7dq1+frrr/nxxx9p2bIlU6dO5e233y6Tr0FcFOBtpra3hfB6vtzdROdN2whetj6KTVcZYljPPPOb1CGp2GtIIbLKx9WZ0bNp2fRqGcS6F2/l9lZBNFNiifKOoIthL+m6B4/nPMsn9oE4fvssHteVwW1C5HRaUS0ouq5XqHnf1NRU/Pz8SElJwdfXN89jWVlZxMTEEBoaiodHyZdGNE0jNTWVDccz3VLnozJxjJWvry+qWrox6tV+HyuSbJsdRbOzbNkynt1kwK4rdFF387HpffyUTE7rNRmT8xRb9RaFPn9cj6Z0aRpYbdb0rVYrv//+O3fccQcmk8nd3bkidk2n67SVRZbtv1Sgt5knbm7CvjU/8Lr9fXyU8ySZ63Ff2lMc0BvmaftK3xY80S0sz31VYbzKi4yV68pqrIp7/86vWu12cegTHsTt4cEuFZ0SojgWowGrNXdnUf2aHhxLyma9dh0Dc97gU9N7tFBPMN/8Jq/bHuAbe2/yz7nNXHWImasOEeznwf/6XUutGhb5mazgXNmO7ZCYnkV21P8xw7gIFNhob8mY1KdIouAv5snL9tMwwEs+AIlqoVoGH3AxmVCI0vJS3xaMnrcTgGN6EENyJjHN9DkDDRv5P9McblAPM8H6KFkUTHyOT8lizLwdee4rbDauuEPKRPkpajv2pQJI4QPTTLoY9gIwx9aL120jsBXxa1dOQxbVSbXK+RCiLN12bV2mDAnH8b5xHg+eso7jdesD2HSVuwzrWGJ+lebKCZeul39LbnGHlIny59iO/b9+1xZ4rJ2yn98sr9DFsJcM3cJTOeOYaHu4yMADJAlZVC8SfAhRioZ1aMSeSbcz77EOjOjYEIOi8KX9Dh6wvsJpvSbN1TiWmF/lPsNKip+wz7sl9/ddxR9SJgGIexhUhUzrxa3vKhpPGpawwPwGQUoSB7UQBua8wRKts8vXlCRkUR1I8CFEKfMyG+ncNJDXB1/H4nFdAdiktaRv9hTW2K/HQ7Ey1fQFM00f4kNmsddyfBp+dfGeYotaSc2QslfUOS574lIBCOYc88xv8pJpAUZFY5G9M4NyXuewHgLAHeFBRV77UnIasqgOqm3OhxDlSVHgnO7HQ9YXeFz7jeeNP9DfsIkblMM8bxvFJq34EviJGRXvWIDqpLBzXBw7WbYeTaSfuonJpi/wUzLJ0C1E2Ebyo70bjgTjIF8LM+5rw/bpK0lILbzSspyGLKoTmfkQogw5aoFcF+LH64PCURWVz+wDuCdnIse12jRQz7DA/AavGb/BA9fK/xdFpuvLhuMcl/xLXmfTc5i5bBuv5LzPR+YP8FMyidbCuCNnCj/au3PpzqaIga0wG1UiBrZCwT11hoSoSCT4EKIMBft5su6lHiwe24URnRoxc3gbAKL1pvTNmco8Ww8AHjFG8pv5FdooB6/4tWS6vvQVd47Lbeo2oizPc5fhL+y6wge2wdydM9F5Ii3kHio4ZUi4c8eSY5fMpVVPIXfGY9YDbWWbrag2ZNlFiDJ2abXKvtfV45W+55m8bD8ZePKK7XFWaDcx1fQ5YWo8P5kj+NJ+BzNsd5GJ68FETU8Tmq5j13T55HwFitrCXNg5LrVIJcL0DYMMGwA4ogXxgvUJZyG5NweH4+1hpLa3hdYNa+Jlzvtrtk94ML1aBsmWaVGtSfAhnL7++mvGjx9PcnKyu7tSpQ1oXY/P/orB18NIu8a1WLXfQu/0pkwyzWGIYT1PGH/jDsPfvGZ9iJVaW5eumXzeyv1f/C2Veq9AYfkcjnE8eDr9kpY6A9UNvGb6lkAlFbuu8Lm9P+/Z7iIbM4oCug43NKhJeIhfwRe6hNQZEtWdLLuUsxMnTvDoo49Sr149zGYzjRo14umnn+bcuXPONt27d2f8+PFFXmPVqlX06NEDf39/vLy8aNasGSNHjsRms1329VevXo2iKAX+vPrqqwwdOpR///3X2Xbq1Km0bevam59wXbCfJ+tf6sGf/+3G9LtvYPOEnnw9phfPWMfyUM7zxOqB1FfO8pX5bWaZ3qMurtd9kK23Re9KKUxR+RyOcfzzn1MANFdOsMD8Bh+YPyJQSWW/1oAhOf/HVNswsjHzaNdQrgvxo7a3hQBvc5l+fUJUBdV65mNXbDJTft/Py3e04Pr6Ncv89Y4cOUKnTp1o3rw58+fPJzQ0lL179/L888+zbNkyNm3ahL9/8Znue/fupW/fvjz11FN8+OGHeHp6cvDgQX766Sc0TXO5LwcOHMhTe9/b2xtPT088PeUkzfJw6VKMoiiYDbm312ht6JV9LU8bF/KY4Tf6GrZws7qbj20D+dJ+B9nkvrGpChT2nqpTvStlFjeLkX82qLh8Dsd9h2PjedX4Mw8ZlmNUNM7rZmbaBvOZvT/WC78+g3wtvHLHtagK5Ng1ORROCBdU65mPX7bHsfHIOX7ZHlcurzd27FjMZjMrVqygW7duNGzYkL59+/LHH38QFxfHhAkTLnuNqKgogoODmT59OuHh4YSFhdGnTx+++OILzGbXP3HVqVOHoKAg5x9vb2++/vpratasCeQuwUybNo2dO3c6Z0e+/vrrK/zKxeU4dsWEBtbgPB5MtQ2jf85ktmnN8FayeMH0A39anqO/uhHQCw08HK6kUmZJZgsqqsvNYlw6G2TXdL5eH1NkaXQDdoYZ/uRP83M8ZlyGUdGItN/Ebdlv8ZF9sDPwgNydLAZVkdNohSiBajfzcTIli2NpOgZVZenOkwAs3XmSu2+sj65DrRom6tfyKvXXTUxMZPny5bz55psFZheCgoK4//77+f777/n444+LvU5QUBDx8fGsXbuWW265pdT76TB06FB27NjBqlWr+OOPPwDw8yt+HVtcOceuGIOicPP0VcSnZLFfb8jdORMZqG7gRdMC6itnmWn+kJHacqZYh7Ndb17sNV3deluS2YKK6nKzGJfOBkXtSyjmTBadXuo2XjQuoKma+/shRqtLhO0h1mg35GmpKrnJpZVljISoSKpd8HHHrG3OfzsmpBMzcuj/4Trn/Uen9iv11z148CC6rnPttQXPgQC49tprSUpK4syZM8Ve55577mH58uV069aNoKAgOnbsSM+ePXnwwQcve4TxperXr5/n9rFjx/Lc9vT0pEaNGhiNRoKCXKvMKK6O41PzxAEteXLudgB0VBZrXVmefRNPGH7jSeNSblL/5RdLBCvtrXnXdjd79CaFXs+VrbeO2YL8b9qO2YLKsv2zsF0pl3LMBs1ceYgZf/xbaJDSTtnPi6YF3KTm5j0l6t58aBvCd/bbyCH32PFhN9WnY1hgkTtZhBCuqXbLLm8OaIbxwjq44xeQ42+jqjBjaGt3dAtdz+2FohS/Rm8wGJg9ezaxsbFMnz6devXq8eabb9KqVSvi411PMvzrr7+Ijo52/qlVq9ZV9V+Unj7hwXkOqAPIwsIH9jvpkf0O39u6Y9NVbjVE86vlVT41vUsL5Xiea9T0MvH+n/+yKza5yNdxJeehNMq2X7qkU1aHprk6yzN7fUy+r1enk7qX+aY3+Mnyf9yk/nshr2MQ3bJnMNve1xl4AMzfEovFqNK5aaAEHkJchWr3v6dfqzpc17A2Az/aUOCxRWO7XHaL3JVq2rQpiqKwb98+Bg8eXODx/fv3U6tWLQIDA126XkhICCNGjGDEiBG88cYbNG/enE8++YRJkya59PzQ0FBnfoeoeIZ1aMSgNiFEH0/mTHo2tb0tmI0qd3+ykRdtT/CxfSBPG39hsLqe2w1bud2wlT/tbfjM1p+/9RYkZ1rZdCSRX7bHFZpMfbmcByidsu35l3QsBp3p7eGPf07R9/r6l3m261wtsJZ83nrhXzrd1Z38x7iQG9Xcwm45uoGf7N1433Ynpyg68bu6JvMKUZqqXfBxKce+fMffZSkgIIBevXrx8ccf88wzz+TJ+0hISOC7777jwQcfvOzMR2Fq1apFcHAwGRkZpdllTCYTdrv98g1FmXAcUOewJy4FyP15PaYH8ax1DB8pg3jG+DN3qH/T07CDnoYd7NSaMEcZxKLsG535TP8mpDH372NEDGzFyeTzxeQ8FFTcrEJRxbmg6CUdgGe+j0ZRDaW2pNM+1J9gPw8SUrKKPSvYkyyGGNbzkCGS5mpuonm2bmK+vQef2gYQz+WDLDlHR4irVy2DD8fOguCaHgy9qQHfbzlBfHJWme/PnzlzJp07d+b222/njTfeyLPVNiQkhDfffNPZ9syZM0RHR+d5flBQEIsXLyY6OpohQ4YQFhZGVlYW33zzDXv37uXDDz8s1f42bNiQmJgYoqOjqV+/Pj4+PlgsllJ9DeG6/D+3s9cf5dDpEMZZn6KxEs9jht+527CWG9QjvMt7/NcSwPdZPXj0wwTnJ/n3ov5l9YEzxb5B53fwVDobD58rUIWzuETVXi2DilzScSjNGQSDqjBxQEtGz92OAgVet75yhgcMUdxnWEVNJTdIT9c9mG+/lc9s/ThDyZYd5RwdIa5OtQw+HDsLzAYVRVEY3r5huezPb9asGVu3biUiIoKhQ4dy7tw5goKCGDx4MBMnTsxT42PevHnMmzcvz/MnTpzIoEGDWLduHU8++SQnT57E29ubVq1asWjRIrp161aq/R04cCCRkZH06NGD5ORkZs+ezUMPPVSqryFcl//n9voQPwbMXI+iwFE9mFdtj/Ke7W4eNEYxwrCCEOUcz5p+4mnjz6zU2jLPfivr/70OvYT/7WeuOsTMVYfy7IC5XKLq+NuaubSk817Uv3RpGnhV5cUvrdcz64G2zoDIg2z6qpu527CWLoa9zvbHtDrMsd/Oj/ZupJF3Z5uXWSUz5/L1cuQcHSGuTrUMPqBgkafy2p/fqFEjZs+eXWyb1atXF/v4t99+e8Wv3717d2dya34PPfRQnuDCYrHw448/oqrVLi+5wrr05zTQx5JnJuSztUc4dg7es93Nx7aB9FE3M9y4kg7qfnoZttHLsI1zug+/2zuw1N6JLfo16CXIOXcEFh8Nb8Prv/1TbKLqZ38dcemahQU2xSmsMKCjXs83G44xsmMIc3tksjPyK3rpG/FRzjuf+5c9nDn221mptUEr4ut+++4b+L9f98mx90KUsWobfAhR2V06E6Lp8OGfF0/EzcbMYq0ri3O6EqbEMdywkkGG9QQqqYww/sEI4x/E6/5E2m9ildaav7VrndVTi+Kol/Hq4j0kZliLbZuRXbJcoaK29uYPNvIEGp0VFAVWRMdwm7qdTrtn0WDvdmoqGYSR29ljWh1+st/CL/abiaN2ka/vqNlxx/X1UFWF0c6tzhfJsfdClB4JPqqYvn378tdffxX62CuvvMIrr7xSzj0SZckxE7L5yLkiP60f1kN43TaCybbhdFH3MEDdyO2GLQQriTxsXM7DLCdTt7Bea8Ua7QY2ai05rNfj4tvtRTpcNvAoTlG7dh13PzV/B9+P8qBNw1rYNZ2PVh1m45FzTPl9Py/1bcHSnSdR0di/Yy2/7fyYLuoeVqoH8DBf7NNZ3ZcV9htZbO/K5svM7ozr0ZTOYQF5anY4jr3Pn88SVMkKrwlRkUnwUcV88cUXnD9/vtDHLndujKi8XEmAtGNgrXYDa7UbeNX2CLeou7hV3UEPQzRBShK9DNvpZcj9xJ+oe7NVu4Yt2jVEa03ZrzcskB9xJS5XMiTHrjMj6iCdmwbw+V9HOJueQy1SsRyNZuUnn/GWepg2lkPUUtLzPC9OD2C5/SYi7TexVb+myGWV/JrV9c6zo8hBjr0XomxJ8FHFhISEuLsLwg1KmgCZjZkorR1RWjuw6bRUjtFdjeYWw25aK4fwV9LpbdhGb8PFisCxeiD7tQYc0BtwTK9LrF6bOD2QeD0gTyGu4th12HIG7PlyOj3Joo6STD3lHA2OnMQcc5J3lHjCLCepr5wtcJ003ZNNWkvWaeGs08KLnKm5nOLGTY69F6LsSPAhRBXgap2Lwins0xuzz96Yj+2DMWGjlXKUduoB2qv7aakeo75yNveP4Sy3sSPPszVd4Rw+pOo1SMabFL0GKdQgRzdhR0FDxY6Kio6Xko3nsWxuM2bjpWQRQCp1lOQ8iaGFOawFE62HsUsLY6cWxm49FDtFJ4kbVQUvs4HULFsRX7EkjgrhThJ8CFEFXK7ORVHG9WhKsJ8HExbtcRbbs2EkWm9KtL0pX9hzzznyJYNrlBO0UI/TXImlvnLmQkByBk8lh9qkUltJvaqvIVO3EK/7E6MHcVivxxG9Hoe1YP7VG5BKjRJdy6bpPNq1Ce/98W+BxyRxVAj3k+BDiCqiqETJwjg++T/Tqzmn07Ko/UfeonuHT6eTkXNxx0oqNdiit2CLvUW+K+kEkEptJQU/MqippOOrZOBHBmZsqGgY0FAVDVDI1C1kYuE8FjJ1C0n4cEqvxWm9Jul4ciVLJ0VpHOjFJ5I4KkSFJMGHEFXIpYmSUfsS+Gr90QIzIfk/+RdWdO94YiZ3zdqIr4eRdo1rsfVoEokZOSSft+Y7jkAhET/O6X65r3OZKZfyOMrAoY6PB53CAiRxVIgKSIIPIaoYR6Jkp7AA2of6u/TJP3/RvUYBNVh/SUCi6zrHEzO5e9ZG5wzJtxuPceBUGi2CfHigYyO+33KCI2dyZ0zyBxghNT0Z0yOMBZuPc+RUChazmZpeZto1rsWmw+c4kXyeFkE+DG/fiP/7dS9W+5VHKPnzOSRxVIiKR4KPCkRRFBYuXFjoqbcAR48eJTQ0lB07dtC6dety7VthGjduzPjx4xk/fry7uyKKcDVbRgsLSPLPkKRn2/C2GPMcU2DX9Dyn8bas54ufpwlFUbinTTBLf1vG7X26UcPD7AxsUs5b2XcylTPp2Yzt3pQZfx4sUe6Ks58X/pZ8DiEqNgk+yslDDz3EnDlzADAYDNSrV49+/foxefJkatXKPdQqPj7e+e+K4tZbb2XNmjUF7rdarWzZsoUaNS4mAl4ueBLuUZqf/PMHJD4epjy3HY8XVjvD0caogsWoOk9wXr43ocDsTE2v3OsmZ5asoJnkcwhROUjwUY769OnD7Nmzsdls7Nu3j0ceeYTk5GTmz58P5J5aWxE9/vjj/N///V+e+4xGI7VrF12uWghXFHVAXUqmFR14umdT5mw4RvL5ooMQ/xom/te/FUG+ks8hRGUhJ4aVI4vFQlBQEPXr16d3794MHTqUFStWOB9XFIVFixY5b2/evJk2bdrg4eFBu3bt2LFjR4FrLlmyhGbNmuHp6UmPHj2YM2cOiqKQnJzsbLNhwwZuueUWPD09adCgAU899RQZGRku99vLy4ugoKA8fyB32WXGjBnOfwMMGTIERVGct4Uoil3TmbR0X5EH1CnAD1tjmTzkOhQK7oNx3Dd5yHUMaRNCp7AACTyEqCQqf/Ch65CT4fofa2bJ2hf35yrS9o8cOUJkZCQmU+GVITMyMujfvz/XXHMN27ZtIyIigueeey5Pm6NHj3L33XczePBgoqOjGTVqFBMmTMjTZvfu3dx+++3ceeed7Nq1i++//55169Yxbty4K+57YbZs2QLA7NmziY+Pd94WVYtd09l4+ByLo+PYePgc9svVSy/G5pjEYrcE60B8Sha1apiZ9UBbgvzyViMN8vMocBCdEKJyKLNll48//pi33nqL+Ph4WrVqxYwZM7j55ptL/4WsmTC5nktNVaBmab72KyfB7Hrxo19//RVvb2/sdjtZWbm/dN99991C23733XfY7Xa++uorvLy8aNWqFbGxsYwePdrZ5pNPPuGaa67hrbfeAuCaa65hz549vPnmm842b731FsOHD3cmhTZr1owPPviAbt26MWvWLDw8Ll+W++OPP+aLL75w3h41ahTvvPNOnjaOJZiaNWtW2OUjcXUi98QXyM0IvoocC1fOo3G0G9Q6RLbMClGFlEnw8f333zN+/Hg+/vhjunTpwqeffkrfvn3Zt28fDRs2LIuXrBR69OjBrFmzyMzM5IsvvuDff//lP//5T6Ft//nnH2644Qa8vC4e5tWpU6c8bQ4cOMBNN92U57727dvnub1t2zYOHTrEd99957xP13U0TSMmJoZrr732sv2+//7788yo1KxZ87LPEVVLUbkZCSlZjJ67/YpmIFw9j8bRTrbMClF1lEnw8e677/Loo4/y2GOPATBjxgyWL1/OrFmzmDJlSum+mMkrdwbCBZqmkZqWhq+PD6paCitOppKd8lmjRg2aNm0KwAcffECPHj2YNGkSr7/+eoG2ugtLOrquO3cMFPU8TdMYNWoUTz31VIHnuxoI+vn5Ofstqh9XcjMmLd1Hr5ZBJZqJuNx5NHL+ihBVV6kHHzk5OWzbto2XXnopz/29e/dmw4YNBdpnZ2eTnZ3tvJ2amns+hNVqxWrNm+FutVqdn9o17ZJjMY2eLvVN13Uw2dFNXmhKKUzX6rrLeR+6rjv77vC///2Pfv36MWrUKOrVy106cnxtLVq04NtvvyUjIwNPz9yvzzF+jjbXXHMNy5Yty3NNR66Fo02bNm3Yu3cvTZo0KbRfecYxX3+L6nf+do7HTCYTVqu1yLaXvqau61itVgyGog8HqywcP6f5f16ris0xiSSmn8dSzLcqMf08mw6dvmygcOlYmYDX+l3DM99HA4VXYX2t3zVodhuanWqpqv9slSYZK9eV1ViV5HqlHnycPXsWu91O3bp189xft25dEhISCrSfMmUKkyZNKnD/ihUr8iw5QO72zqCgINLT08nJybniPqalpV3xc6+U1WrFZrM5gyuAtm3b0qJFCyZNmuTM2zh//jypqan079+fV199lZEjR/Lcc89x/Phx3n77bSA3GTU1NZXhw4fz3nvv8cwzzzBixAh2797N7NmznV+jqqqMGTOG3r1788QTTzBy5Ei8vLw4cOAAq1evZvr06Zftt91uJycnJ0+/HTRNIysry/lYw4YNiYyM5Prrr8disRS5PJOTk8P58+dZu3YtNlvhp45WRlFRUe7uQpmZ3v7ybc7+s4nf/3HtepeO1bRirp0Ts43fY1y7ZlVWlX+2SpuMletKe6wyMzNdbltmCaeFLQfkvw/g5Zdf5tlnn3XeTk1NpUGDBvTu3RtfX988bbOysjhx4gTe3t4uJUrmp+s6aWlp+Pj4FNqXsmQymTAajQW+pv/+9788+uijvPrqqwB4enri6+uLr68vS5YsYcyYMXTr1o2WLVsybdo07rnnHmrUqIGvry/XXXcdP/zwA88//zyffvopnTp1YsKECYwdO5batWvj4eFB586dWbVqFa+++ip33HEHuq4TFhbGvffeW6Avl3KMlcFgwGw2F9pWVVU8PDycj73zzjs899xzfPPNN4SEhHDkyJFCr52VlYWnpye33HLLFX0fKxqr1UpUVBS9evUqcvdSZbY5JpFH5lx+99JXI29yaeajsLGyazrbjiVxNj2bQG8LNzaqJcmkVP2frdIkY+W6shqrwj6kFqXUg4/AwEAMBkOBWY7Tp08XmA2B3NoXFoulwP0mk6nAoNjtdhRFQVXVK8rZcCwHOK5RnhzVTfN74IEHeOCBB4CC+RqdO3cmOjo6z3352wwePDhPRdE333yT+vXr55k16tChQ4kjXMdYrVq1qsixOnr0aJ7bgwYNYtCgQZe9tqrmVrcs7HtcmVW1r8ehY9M6+Ht7XjY3o2PTOi4HDPnHygR0aV7w94PIVVV/tsqCjJXrSnusSnKtUn8HNpvN3HjjjQXe7KKioujcuXNpv1y19/HHH7NlyxaOHDnCt99+y1tvvcXIkSPd3S1RhRhUhYkDWgKFF/oCOUtFCFEyZbLs8uyzzzJixAjatWtHp06d+Oyzzzh+/DhPPvlkWbxctXbw4EHeeOMNEhMTadiwIf/97395+eWXXXruX3/9Rd++fYt8PDY2trS6KSq5PuHBzHqgrUsn5AohxOWUSfAxdOhQzp07x//93/8RHx9PeHg4v//+O40aNSqLl6vW3nvvPd57770rem67du0KLOs4XG7Hiqh+ruaEXCGEuFSZJZyOGTOGMWPGlNXlRSnw9PQssn6HpmklSh4S1YMU+hJClIbKf7aLEEIIISqVShl8yJJA5SbfPyGEqN7KbNmlLJjNZlRV5eTJk9SuXRuz2Vyieh2appGTk0NWVla5b7WtbMpirHRdJycnhzNnzqCqKmazuVSuK4QQonKpVMGHqqqEhoYSHx/PyZOunedyKV3XOX/+PJ6enuVeZKyyKcux8vLyomHDhhIACiFENVWpgg/Inf1o2LAhNpsNu71kBz5YrVbWrl3LLbfcIkVoLqOsxspgMGA0GiX4E0KIaqzSBR/AFVfHNBgM2Gw2PDw8JPi4DBkrIYQQZUXmvYUQQghRriT4EEIIIUS5kuBDCCGEEOWqwuV8OE5tLYvqmlarlczMTFJTUyWP4TJkrFwnY+U6GauSkfFynYyV68pqrBzv2/lPXy9MhQs+0tLSAGjQoIGbeyKEEEKIkkpLS8PPz6/YNoruSohSjjRN4+TJk/j4+JT6dszU1FQaNGjAiRMn8PX1LdVrVzUyVq6TsXKdjFXJyHi5TsbKdWU1Vrquk5aWRr169S5bx6nCzXyoqkr9+vXL9DV8fX3lh9NFMlauk7FynYxVych4uU7GynVlMVaXm/FwkIRTIYQQQpQrCT6EEEIIUa6qVfBhsViYOHEiFovF3V2p8GSsXCdj5ToZq5KR8XKdjJXrKsJYVbiEUyGEEEJUbdVq5kMIIYQQ7ifBhxBCCCHKlQQfQgghhChXEnwIIYQQolxV2+Bj4MCBNGzYEA8PD4KDgxkxYgQnT550d7cqnKNHj/Loo48SGhqKp6cnYWFhTJw4kZycHHd3rcJ688036dy5M15eXtSsWdPd3alQPv74Y0JDQ/Hw8ODGG2/kr7/+cneXKqS1a9cyYMAA6tWrh6IoLFq0yN1dqpCmTJnCTTfdhI+PD3Xq1GHw4MEcOHDA3d2qsGbNmsX111/vLC7WqVMnli1b5pa+VNvgo0ePHvzwww8cOHCAn3/+mcOHD3P33Xe7u1sVzv79+9E0jU8//ZS9e/fy3nvv8cknn/DKK6+4u2sVVk5ODvfccw+jR492d1cqlO+//57x48czYcIEduzYwc0330zfvn05fvy4u7tW4WRkZHDDDTcwc+ZMd3elQluzZg1jx45l06ZNREVFYbPZ6N27NxkZGe7uWoVUv359pk6dytatW9m6dSu33norgwYNYu/eveXfGV3ouq7rixcv1hVF0XNyctzdlQpv+vTpemhoqLu7UeHNnj1b9/Pzc3c3Koz27dvrTz75ZJ77WrRoob/00ktu6lHlAOgLFy50dzcqhdOnT+uAvmbNGnd3pdKoVauW/sUXX5T761bbmY9LJSYm8t1339G5c2c5itkFKSkp+Pv7u7sbohLJyclh27Zt9O7dO8/9vXv3ZsOGDW7qlahqUlJSAOT3kwvsdjsLFiwgIyODTp06lfvrV+vg48UXX6RGjRoEBARw/PhxFi9e7O4uVXiHDx/mww8/5Mknn3R3V0QlcvbsWex2O3Xr1s1zf926dUlISHBTr0RVous6zz77LF27diU8PNzd3amwdu/ejbe3NxaLhSeffJKFCxfSsmXLcu9HlQo+IiIiUBSl2D9bt251tn/++efZsWMHK1aswGAw8OCDD6JXk4KvJR0rgJMnT9KnTx/uueceHnvsMTf13D2uZLxEQYqi5Lmt63qB+4S4EuPGjWPXrl3Mnz/f3V2p0K655hqio6PZtGkTo0ePZuTIkezbt6/c+2Es91csQ+PGjeO+++4rtk3jxo2d/w4MDCQwMJDmzZtz7bXX0qBBAzZt2uSWKajyVtKxOnnyJD169KBTp0589tlnZdy7iqek4yXyCgwMxGAwFJjlOH36dIHZECFK6j//+Q9Llixh7dq11K9f393dqdDMZjNNmzYFoF27dmzZsoX333+fTz/9tFz7UaWCD0cwcSUcMx7Z2dml2aUKqyRjFRcXR48ePbjxxhuZPXs2qlqlJsxccjU/WyL3F96NN95IVFQUQ4YMcd4fFRXFoEGD3NgzUZnpus5//vMfFi5cyOrVqwkNDXV3lyodXdfd8r5XpYIPV23evJnNmzfTtWtXatWqxZEjR3jttdcICwurFrMeJXHy5Em6d+9Ow4YNefvttzlz5ozzsaCgIDf2rOI6fvw4iYmJHD9+HLvdTnR0NABNmzbF29vbvZ1zo2effZYRI0bQrl075wza8ePHJX+oEOnp6Rw6dMh5OyYmhujoaPz9/WnYsKEbe1axjB07lnnz5rF48WJ8fHycM2t+fn54enq6uXcVzyuvvELfvn1p0KABaWlpLFiwgNWrVxMZGVn+nSn3/TUVwK5du/QePXro/v7+usVi0Rs3bqw/+eSTemxsrLu7VuHMnj1bBwr9Iwo3cuTIQsdr1apV7u6a23300Ud6o0aNdLPZrLdt21a2RBZh1apVhf4MjRw50t1dq1CK+t00e/Zsd3etQnrkkUec//9q166t9+zZU1+xYoVb+qLoejXJsBRCCCFEhVD9Fu+FEEII4VYSfAghhBCiXEnwIYQQQohyJcGHEEIIIcqVBB9CCCGEKFcSfAghhBCiXEnwIYQQQohyJcGHEEIIIcqVBB9CCCGEKFcSfAghhBCiXEnwIYQQQohyJcGHEEIIIcrV/wP9YfhyKkFcJwAAAABJRU5ErkJggg==",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week37_208_1.png"
- }
- },
- "output_type": "display_data"
- }
- ],
- "source": [
- "\n",
- "np.random.seed(2018)\n",
- "n = 100\n",
- "# we do not include the intercept\n",
- "d = 2\n",
- "Lambda = 0.01\n",
- "\n",
- "# Make data set.\n",
- "x = np.linspace(-3, 3, n)\n",
- "y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)\n",
- "\n",
- "#Design matrix X does not include the intercept. \n",
- "X = np.zeros((len(x), d))\n",
- "for p in range(d): \n",
- " X[:, p] = x ** (p+1)\n",
- "\n",
- "\n",
- "#Split data in train and test\n",
- "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
- "\n",
- "# Scale data by subtracting mean value,own implementation\n",
- "#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable\n",
- "X_train_mean = np.mean(X_train,axis=0)\n",
- "#Center by removing mean from each feature\n",
- "X_train_scaled = X_train - X_train_mean\n",
- "X_test_scaled = X_test - X_train_mean\n",
- "#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered, note)\n",
- "y_scaler = np.mean(y_train)\n",
- "y_train_scaled = y_train - y_scaler\n",
- "\n",
- "\n",
- "#Calculate beta\n",
- "beta_OLS = OLS_fit_beta(X_train_scaled, y_train_scaled)\n",
- "beta_Ridge = Ridge_fit_beta(X_train_scaled, y_train_scaled,Lambda,d)\n",
- "print(beta_OLS)\n",
- "print(beta_Ridge)\n",
- "# calculate intercepts and print them\n",
- "interceptOLS = y_scaler - X_train_mean @ beta_OLS\n",
- "interceptRidge = y_scaler - X_train_mean @ beta_Ridge\n",
- "print(interceptOLS)\n",
- "print(interceptRidge)\n",
- "\n",
- "#predict value with intercept\n",
- "ytilde_test_OLS = X_test_scaled @ beta_OLS+y_scaler\n",
- "ytilde_test_Ridge = X_test_scaled @ beta_Ridge+y_scaler\n",
- "\n",
- "\n",
- "#Calculate MSE\n",
- "\n",
- "print(\" \")\n",
- "print(\"test MSE of OLS:\")\n",
- "print(MSE(y_test,ytilde_test_OLS))\n",
- "print(\" \")\n",
- "print(\"test MSE of Ridge\")\n",
- "print(MSE(y_test,ytilde_test_Ridge))\n",
- "\n",
- "plt.scatter(x,y,label='Data')\n",
- "plt.plot(x, X @ beta_OLS+interceptOLS,'*', label=\"OLS_Fit\")\n",
- "plt.plot(x, X @ beta_Ridge+interceptRidge, label=\"Ridge_Fit\")\n",
- "plt.grid()\n",
- "plt.legend()\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "522668c3",
- "metadata": {
- "editable": true
- },
- "source": [
- "Finally, instead of using our own function we repeat the same example\n",
- "using the **standardscaler** functionality of the library\n",
- "**Scikit-Learn**. Here we limit ourselves to Ridge regression only."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "id": "9954bcc1",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [
- {
- "ename": "NameError",
- "evalue": "name 'RegRidge' is not defined",
- "output_type": "error",
- "traceback": [
- "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
- "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
- "Cell \u001b[0;32mIn[11], line 34\u001b[0m\n\u001b[1;32m 32\u001b[0m ypredictOLS \u001b[38;5;241m=\u001b[39m OLS\u001b[38;5;241m.\u001b[39mpredict(X_test_scaled)\n\u001b[1;32m 33\u001b[0m linear_model\u001b[38;5;241m.\u001b[39mRidge(Lambda)\n\u001b[0;32m---> 34\u001b[0m \u001b[43mRegRidge\u001b[49m\u001b[38;5;241m.\u001b[39mfit(X_train_scaled,y_train_scaled)\n\u001b[1;32m 35\u001b[0m ypredictRidge \u001b[38;5;241m=\u001b[39m RegRidge\u001b[38;5;241m.\u001b[39mpredict(X_test_scaled)\n\u001b[1;32m 36\u001b[0m betaOLS \u001b[38;5;241m=\u001b[39m OLS\u001b[38;5;241m.\u001b[39mcoef_\n",
- "\u001b[0;31mNameError\u001b[0m: name 'RegRidge' is not defined"
- ]
- }
- ],
- "source": [
- "from sklearn import linear_model\n",
- "np.random.seed(2018)\n",
- "n = 10\n",
- "d = 2\n",
- "Lambda = 0.01\n",
- "\n",
- "# Make data set.\n",
- "x = np.linspace(-3, 3, n)\n",
- "y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)\n",
- "\n",
- "# Design matrix X does not include the intercept. \n",
- "X = np.zeros((n, d))\n",
- "for p in range(d): \n",
- " X[:, p] = x ** (p+1)\n",
- "\n",
- "#Split data in train and test\n",
- "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
- "# Scale data by subtracting mean value of the input using scikit-learn\n",
- "scaler = StandardScaler(with_std=False)\n",
- "scaler.fit(X_train)\n",
- "X_train_mean = np.mean(X_train,axis=0)\n",
- "X_train_scaled = scaler.transform(X_train)\n",
- "X_test_scaled = scaler.transform(X_test)\n",
- "# We scale also the output, here by our own code\n",
- "y_scaler = np.mean(y_train)\n",
- "y_train_scaled = y_train - y_scaler\n",
- "y_test_scaled = y_test- y_scaler\n",
- "\n",
- "#Calculate beta\n",
- "OLS = LinearRegression()\n",
- "betaOLS=OLS.fit(X_train_scaled,y_train_scaled)\n",
- "ypredictOLS = OLS.predict(X_test_scaled)\n",
- "linear_model.Ridge(Lambda)\n",
- "RegRidge.fit(X_train_scaled,y_train_scaled)\n",
- "ypredictRidge = RegRidge.predict(X_test_scaled)\n",
- "betaOLS = OLS.coef_\n",
- "betaRidge = RegRidge.coef_\n",
- "print(betaOLS)\n",
- "print(betaRidge)\n",
- "interceptOLS = np.mean(y_train) - X_train_mean @ betaOLS\n",
- "interceptRidge = y_scaler - X_train_mean @ betaRidge\n",
- "print(interceptOLS)\n",
- "print(interceptRidge)\n",
- "#predict value \n",
- "ytilde_test_Ridge = X_test_scaled @ betaRidge+y_scaler\n",
- "ytilde_test_OLS = X_test_scaled @ betaOLS+y_scaler\n",
- "\n",
- "#Calculate MSE\n",
- "print(\" \")\n",
- "print(\"test MSE of OLS\")\n",
- "print(MSE(y_test,ytilde_test_OLS))\n",
- "print(\" \")\n",
- "print(\"test MSE of Ridge\")\n",
- "print(MSE(y_test,ytilde_test_Ridge))\n",
- "plt.scatter(x,y,label='Data')\n",
- "plt.plot(x, X @ RegRidge.coef_ + RegRidge.intercept_ , label=\"Ridge_Fit\")\n",
- "plt.grid()\n",
- "plt.legend()\n",
- "plt.show()"
+ "For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended."
]
}
],
diff --git a/doc/LectureNotes/_build/jupyter_execute/week37.py b/doc/LectureNotes/_build/jupyter_execute/week37.py
index 8be12b1d5..b1ad40141 100644
--- a/doc/LectureNotes/_build/jupyter_execute/week37.py
+++ b/doc/LectureNotes/_build/jupyter_execute/week37.py
@@ -6,215 +6,49 @@
#
# # Week 37: Statistical interpretations and Resampling Methods
-# **Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University
+# **Morten Hjorth-Jensen**, Department of Physics, University of Oslo, Norway
#
# Date: **September 9, 2024**
#
#
-# ## Plans for week 37
+# ## Plans for week 37, lecture Monday
#
# **Material for the lecture on Monday September 9.**
#
#
#
-# * Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff
+# * Statistical interpretation of Ridge and Lasso regression, see also slides from last week
#
-# * Statistical interpretation of Ridge and Lasso regression
+# * Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff (this may partly be discussed during the exercise sessions as well.
#
# * Readings and Videos:
#
-# * Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap).
+# * Raschka et al, pages 175-192
+#
+# * Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap). See .
#
# * [Video on cross validation](https://www.youtube.com/watch?v=fSytzGwwBVw)
#
# * [Video on Bootstrapping](https://www.youtube.com/watch?v=Xz0x-8-cgaQ)
#
# * [Video on bias-variance tradeoff](https://www.youtube.com/watch?v=EuBBz3bI-aA)
+
+# ## Plans for week 37, lab sessions
#
-# **Material for the active learning sessions on Tuesday and Wednesday.**
+# **Material for the lab sessions on Tuesday and Wednesday.**
#
# * Calculations of expectation values
#
+# * Discussion of resampling techniques
+#
# * Exercise set for week 37
#
# * Work on project 1
#
-# * See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session. This note is added at the end of these slides.
-#
# * For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.
-# ## Material from last week and relevant for the weekly exercises
-
-# ## Linking the regression analysis with a statistical interpretation
-#
-# We will now couple the discussions of ordinary least squares, Ridge
-# and Lasso regression with a statistical interpretation, that is we
-# move from a linear algebra analysis to a statistical analysis. In
-# particular, we will focus on what the regularization terms can result
-# in. We will amongst other things show that the regularization
-# parameter can reduce considerably the variance of the parameters
-# $\beta$.
-#
-# The
-# advantage of doing linear regression is that we actually end up with
-# analytical expressions for several statistical quantities.
-# Standard least squares and Ridge regression allow us to
-# derive quantities like the variance and other expectation values in a
-# rather straightforward way.
-#
-# It is assumed that $\varepsilon_i
-# \sim \mathcal{N}(0, \sigma^2)$ and the $\varepsilon_{i}$ are
-# independent, i.e.:
-
-# $$
-# \begin{align*}
-# \mbox{Cov}(\varepsilon_{i_1},
-# \varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if}
-# & i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right.
-# \end{align*}
-# $$
-
-# The randomness of $\varepsilon_i$ implies that
-# $\mathbf{y}_i$ is also a random variable. In particular,
-# $\mathbf{y}_i$ is normally distributed, because $\varepsilon_i \sim
-# \mathcal{N}(0, \sigma^2)$ and $\mathbf{X}_{i,\ast} \, \boldsymbol{\beta}$ is a
-# non-random scalar. To specify the parameters of the distribution of
-# $\mathbf{y}_i$ we need to calculate its first two moments.
-#
-# Recall that $\boldsymbol{X}$ is a matrix of dimensionality $n\times p$. The
-# notation above $\mathbf{X}_{i,\ast}$ means that we are looking at the
-# row number $i$ and perform a sum over all values $p$.
-
-# ## Assumptions made
-#
-# The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)
-# that there exists a function $f(\boldsymbol{x})$ and a normal distributed error $\boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2)$
-# which describe our data
-
-# $$
-# \boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon}
-# $$
-
-# We approximate this function with our model from the solution of the linear regression equations, that is our
-# function $f$ is approximated by $\boldsymbol{\tilde{y}}$ where we want to minimize $(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2$, our MSE, with
-
-# $$
-# \boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
-# $$
-
-# ## Expectation value and variance
-#
-# We can calculate the expectation value of $\boldsymbol{y}$ for a given element $i$
-
-# $$
-# \begin{align*}
-# \mathbb{E}(y_i) & =
-# \mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i)
-# \, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
-# \end{align*}
-# $$
-
-# while
-# its variance is
-
-# $$
-# \begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i
-# - \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
-# [\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
-# \beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ &
-# = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i
-# \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
-# \ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2
-# \mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} +
-# \mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2
-# \\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
-# \mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
-# \end{align*}
-# $$
-
-# Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2)$, that is $\boldsymbol{y}$ follows a normal distribution with
-# mean value $\boldsymbol{X}\boldsymbol{\beta}$ and variance $\sigma^2$ (not be confused with the singular values of the SVD).
-
-# ## Expectation value and variance for $\boldsymbol{\beta}$
-#
-# With the OLS expressions for the optimal parameters $\boldsymbol{\hat{\beta}}$ we can evaluate the expectation value
-
-# $$
-# \mathbb{E}(\boldsymbol{\hat{\beta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
-# $$
-
-# This means that the estimator of the regression parameters is unbiased.
-#
-# We can also calculate the variance
-#
-# The variance of the optimal value $\boldsymbol{\hat{\beta}}$ is
-
-# $$
-# \begin{eqnarray*}
-# \mbox{Var}(\boldsymbol{\hat{\beta}}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \}
-# \\
-# & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y} - \boldsymbol{\beta}]^{T} \}
-# \\
-# % & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
-# % \\
-# % & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y} \, \mathbf{y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
-# % \\
-# & = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{y} \, \mathbf{y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
-# \\
-# & = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
-# % \\
-# % & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
-# % \\
-# % & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T
-# \\
-# & = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
-# \, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
-# \end{eqnarray*}
-# $$
-
-# where we have used that $\mathbb{E} (\mathbf{y} \mathbf{y}^{T}) =
-# \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
-# \sigma^2 \, \mathbf{I}_{nn}$. From $\mbox{Var}(\boldsymbol{\beta}) = \sigma^2
-# \, (\mathbf{X}^{T} \mathbf{X})^{-1}$, one obtains an estimate of the
-# variance of the estimate of the $j$-th regression coefficient:
-# $\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} $. This may be used to
-# construct a confidence interval for the estimates.
-#
-# In a similar way, we can obtain analytical expressions for say the
-# expectation values of the parameters $\boldsymbol{\beta}$ and their variance
-# when we employ Ridge regression, allowing us again to define a confidence interval.
-#
-# It is rather straightforward to show that
-
-# $$
-# \mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}.
-# $$
-
-# We see clearly that
-# $\mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big] \not= \hat{\boldsymbol{\beta}}^{\mathrm{OLS}}$ for any $\lambda > 0$.
-#
-# We can also compute the variance as
-
-# $$
-# \mbox{Var}[\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
-# $$
-
-# and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\boldsymbol{\beta}$ goes to zero.
-#
-# With this, we can compute the difference
-
-# $$
-# \mbox{Var}[\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}]-\mbox{Var}(\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
-# $$
-
-# The difference is non-negative definite since each component of the
-# matrix product is non-negative definite.
-# This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\boldsymbol{\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
-#
-# For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.
-
-# ## Material for lecture Thursday September 14
+# ## Material for lecture Monday September 9
# ## Deriving OLS from a probability distribution
#
@@ -459,7 +293,7 @@
# p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}).
# $$
-# We have a model for $p(\boldsymbol{D}\vert\boldsymbol{\beta})$ but need one for the **prior** $p(\boldsymbol{\beta}$!
+# We have a model for $p(\boldsymbol{D}\vert\boldsymbol{\beta})$ but need one for the **prior** $p(\boldsymbol{\beta})$!
# ## Ridge and Bayes
#
@@ -769,10 +603,10 @@
# With the OLS expressions for the parameters $\boldsymbol{\beta}$ we found
# $\mathbb{E}(\boldsymbol{\beta}) = \boldsymbol{\beta}$, which means that the estimator of the regression parameters is unbiased.
#
-# We found also that the variance of the estimate of the $j$-th regression coefficient is
+# In the exercises this week we show that the variance of the estimate of the $j$-th regression coefficient is
# $\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} $.
#
-# This quantity will be used to
+# This quantity can be used to
# construct a confidence interval for the estimates.
# ## Standard Approach based on the Normal Distribution
@@ -1515,336 +1349,172 @@ plt.legend()
plt.show()
-# ## Notes on scaling with examples
+# ## Material for the lab sessions
+
+# ## Linking the regression analysis with a statistical interpretation
#
-# The programs here use both ordinrary least squares (OLS) and Ridge
-# regression with one value only for the hyperparameter $\lambda$. The
-# first example has no scaling and includes the intercept as well and we
-# are trying to fit a second-order polynomial. The second code takes out
-# the intercept and subtracts the mean values of each column of the
-# design matrix and the mean value of the outputs.
+# We will now couple the discussions of ordinary least squares, Ridge
+# and Lasso regression with a statistical interpretation, that is we
+# move from a linear algebra analysis to a statistical analysis. In
+# particular, we will focus on what the regularization terms can result
+# in. We will amongst other things show that the regularization
+# parameter can reduce considerably the variance of the parameters
+# $\beta$.
#
-# The third and final code uses **Scikit-Learn** as library in order to
-# calculate the optimal parameters for OLS and Ridge regression. Note
-# that it is highly recommended to not include the intercept in Ridge
-# and Lasso regression, in order to avoid penalizing the optimization by
-# the intercept. The second and third codes do thus not include the
-# intercept. In the second code we do the scaling ourselves while the
-# last code uses the standard scaler option included in **Scikit-Learn**, known as centering (where
-# we subtract the mean values).
-
-# In[9]:
-
-
-import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.linear_model import LinearRegression
-from sklearn.preprocessing import PolynomialFeatures
-from sklearn.model_selection import train_test_split
-from sklearn.preprocessing import StandardScaler
-
-def MSE(y_data,y_model):
- n = np.size(y_model)
- return np.sum((y_data-y_model)**2)/n
-
-def OLS_fit_beta(X, y):
- return np.linalg.pinv(X.T @ X) @ X.T @ y
-
-def Ridge_fit_beta(X, y,L,d):
- I = np.eye(d,d)
- return np.linalg.pinv(X.T @ X + L*I) @ X.T @ y
-
-# Same random numbers for each test.
-np.random.seed(2018)
-n = 100
-d = 3
-# hyperparameter lambda
-Lambda = 0.01
-
-# Make data set, simple second-order polynomial
-x = np.linspace(-3, 3, n)
-y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)
-
-# The design matrix X includes the intercept and no scaling is made
-X = np.zeros((len(x), d))
-for p in range(d):
- X[:, p] = x ** (p)
-
-
-#Split data, no scaling is used and we include the intercept
-X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
-
-
-#Calculate beta, own code
-beta_OLS = OLS_fit_beta(X_train, y_train)
-beta_Ridge = Ridge_fit_beta(X_train, y_train,Lambda,d)
-print(beta_OLS)
-print(beta_Ridge)
-#predict value
-ytilde_test_OLS = X_test @ beta_OLS
-ytilde_test_Ridge = X_test @ beta_Ridge
-
-#Calculate MSE
-print(" ")
-print("test MSE of OLS:")
-print(MSE(y_test,ytilde_test_OLS))
-print(" ")
-print("test MSE of Ridge")
-print(MSE(y_test,ytilde_test_Ridge))
-
-plt.scatter(x,y,label='Data')
-plt.plot(x, X @ beta_OLS,'*', label="OLS_Fit")
-plt.plot(x, X @ beta_Ridge, label="Ridge_Fit")
-plt.grid()
-plt.legend()
-plt.show()
-
-
-# In this example we do not include the intercept and we scale the data by subtracting the mean values. This follows the discussion in the [lecture material](https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#more-on-rescaling-data).
-# see also the weekly slides [for week 36](https://compphysics.github.io/MachineLearning/doc/pub/week36/html/._week36-bs029.html).
-# It is recommended whrn we use Ridge and Lasso regression to not include the intercept in the optimization process.
+# The
+# advantage of doing linear regression is that we actually end up with
+# analytical expressions for several statistical quantities.
+# Standard least squares and Ridge regression allow us to
+# derive quantities like the variance and other expectation values in a
+# rather straightforward way.
#
-# Before we discuss the code, we repeat some of the basic math from the slides of week 36.
+# It is assumed that $\varepsilon_i
+# \sim \mathcal{N}(0, \sigma^2)$ and the $\varepsilon_{i}$ are
+# independent, i.e.:
+
+# $$
+# \begin{align*}
+# \mbox{Cov}(\varepsilon_{i_1},
+# \varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if}
+# & i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right.
+# \end{align*}
+# $$
+
+# The randomness of $\varepsilon_i$ implies that
+# $\mathbf{y}_i$ is also a random variable. In particular,
+# $\mathbf{y}_i$ is normally distributed, because $\varepsilon_i \sim
+# \mathcal{N}(0, \sigma^2)$ and $\mathbf{X}_{i,\ast} \, \boldsymbol{\beta}$ is a
+# non-random scalar. To specify the parameters of the distribution of
+# $\mathbf{y}_i$ we need to calculate its first two moments.
#
-# Let us try to understand what this may imply mathematically when we
-# subtract the mean values, also known as *zero centering* or simply *centering*. For
-# simplicity, we will focus on ordinary regression, as done in the above example.
+# Recall that $\boldsymbol{X}$ is a matrix of dimensionality $n\times p$. The
+# notation above $\mathbf{X}_{i,\ast}$ means that we are looking at the
+# row number $i$ and perform a sum over all values $p$.
+
+# ## Assumptions made
#
-# The cost/loss function for regression is
+# The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)
+# that there exists a function $f(\boldsymbol{x})$ and a normal distributed error $\boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2)$
+# which describe our data
# $$
-# C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,.
+# \boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon}
# $$
-# Recall also that we use the squared value. This expression can lead to an
-# increased penalty for higher differences between predicted and
-# output/target values.
+# We approximate this function with our model from the solution of the linear regression equations, that is our
+# function $f$ is approximated by $\boldsymbol{\tilde{y}}$ where we want to minimize $(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2$, our MSE, with
+
+# $$
+# \boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
+# $$
+
+# ## Expectation value and variance
#
-# What we have done is to single out the $\beta_0$ term in the
-# definition of the mean squared error (MSE). The design matrix $X$
-# does in this case not contain any intercept column. When we take the
-# derivative with respect to $\beta_0$, we want the derivative to obey
+# We can calculate the expectation value of $\boldsymbol{y}$ for a given element $i$
# $$
-# \frac{\partial C}{\partial \beta_j} = 0,
+# \begin{align*}
+# \mathbb{E}(y_i) & =
+# \mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i)
+# \, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
+# \end{align*}
# $$
-# for all $j$. For $\beta_0$ we have
+# while
+# its variance is
# $$
-# \frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right).
+# \begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i
+# - \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
+# [\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
+# \beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ &
+# = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i
+# \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
+# \ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2
+# \mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} +
+# \mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2
+# \\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
+# \mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
+# \end{align*}
# $$
-# Multiplying away the constant $2/n$, we obtain
+# Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2)$, that is $\boldsymbol{y}$ follows a normal distribution with
+# mean value $\boldsymbol{X}\boldsymbol{\beta}$ and variance $\sigma^2$ (not be confused with the singular values of the SVD).
-# $$
-# \sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j.
-# $$
-
-# Let us specialize first to the case where we have only two parameters $\beta_0$ and $\beta_1$.
-# Our result for $\beta_0$ simplifies then to
-
-# $$
-# n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.
-# $$
-
-# We obtain then
-
-# $$
-# \beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}.
-# $$
-
-# If we define
-
-# $$
-# \mu_{\boldsymbol{x}_1}=\frac{1}{n}\sum_{i=0}^{n-1} X_{i1},
-# $$
-
-# and the mean value of the outputs as
-
-# $$
-# \mu_y=\frac{1}{n}\sum_{i=0}^{n-1}y_i,
-# $$
-
-# we have
-
-# $$
-# \beta_0 = \mu_y - \beta_1\mu_{\boldsymbol{x}_1}.
-# $$
-
-# In the general case with more parameters than $\beta_0$ and $\beta_1$, we have
-
-# $$
-# \beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j.
-# $$
-
-# We can rewrite the latter equation as
-
-# $$
-# \beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\beta_j,
-# $$
-
-# where we have defined
-
-# $$
-# \mu_{\boldsymbol{x}_j}=\frac{1}{n}\sum_{i=0}^{n-1} X_{ij},
-# $$
-
-# the mean value for all elements of the column vector $\boldsymbol{x}_j$.
+# ## Expectation value and variance for $\boldsymbol{\beta}$
#
-# Replacing $y_i$ with $y_i - y_i - \overline{\boldsymbol{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)
+# With the OLS expressions for the optimal parameters $\boldsymbol{\hat{\beta}}$ we can evaluate the expectation value
# $$
-# C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}).
+# \mathbb{E}(\boldsymbol{\hat{\beta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
# $$
-# If we minimize with respect to $\boldsymbol{\beta}$ we have then
-
-# $$
-# \hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}},
-# $$
-
-# where $\boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\boldsymbol{y}}$
-# and $\tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj}$.
+# This means that the estimator of the regression parameters is unbiased.
#
-# For Ridge regression we need to add $\lambda \boldsymbol{\beta}^T\boldsymbol{\beta}$ to the cost function and get then
+# We can also calculate the variance
+#
+# The variance of the optimal value $\boldsymbol{\hat{\beta}}$ is
# $$
-# \hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}.
+# \begin{eqnarray*}
+# \mbox{Var}(\boldsymbol{\hat{\beta}}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \}
+# \\
+# & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y} - \boldsymbol{\beta}]^{T} \}
+# \\
+# % & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
+# % \\
+# % & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y} \, \mathbf{y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
+# % \\
+# & = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{y} \, \mathbf{y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
+# \\
+# & = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
+# % \\
+# % & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
+# % \\
+# % & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T
+# \\
+# & = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
+# \, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
+# \end{eqnarray*}
# $$
-# Now we try to implement this.
+# where we have used that $\mathbb{E} (\mathbf{y} \mathbf{y}^{T}) =
+# \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
+# \sigma^2 \, \mathbf{I}_{nn}$. From $\mbox{Var}(\boldsymbol{\beta}) = \sigma^2
+# \, (\mathbf{X}^{T} \mathbf{X})^{-1}$, one obtains an estimate of the
+# variance of the estimate of the $j$-th regression coefficient:
+# $\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} $. This may be used to
+# construct a confidence interval for the estimates.
+#
+# In a similar way, we can obtain analytical expressions for say the
+# expectation values of the parameters $\boldsymbol{\beta}$ and their variance
+# when we employ Ridge regression, allowing us again to define a confidence interval.
+#
+# It is rather straightforward to show that
-# In[10]:
+# $$
+# \mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}.
+# $$
+# We see clearly that
+# $\mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big] \not= \hat{\boldsymbol{\beta}}^{\mathrm{OLS}}$ for any $\lambda > 0$.
+#
+# We can also compute the variance as
-np.random.seed(2018)
-n = 100
-# we do not include the intercept
-d = 2
-Lambda = 0.01
+# $$
+# \mbox{Var}[\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
+# $$
-# Make data set.
-x = np.linspace(-3, 3, n)
-y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)
+# and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\boldsymbol{\beta}$ goes to zero.
+#
+# With this, we can compute the difference
-#Design matrix X does not include the intercept.
-X = np.zeros((len(x), d))
-for p in range(d):
- X[:, p] = x ** (p+1)
-
-
-#Split data in train and test
-X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
-
-# Scale data by subtracting mean value,own implementation
-#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable
-X_train_mean = np.mean(X_train,axis=0)
-#Center by removing mean from each feature
-X_train_scaled = X_train - X_train_mean
-X_test_scaled = X_test - X_train_mean
-#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered, note)
-y_scaler = np.mean(y_train)
-y_train_scaled = y_train - y_scaler
-
-
-#Calculate beta
-beta_OLS = OLS_fit_beta(X_train_scaled, y_train_scaled)
-beta_Ridge = Ridge_fit_beta(X_train_scaled, y_train_scaled,Lambda,d)
-print(beta_OLS)
-print(beta_Ridge)
-# calculate intercepts and print them
-interceptOLS = y_scaler - X_train_mean @ beta_OLS
-interceptRidge = y_scaler - X_train_mean @ beta_Ridge
-print(interceptOLS)
-print(interceptRidge)
-
-#predict value with intercept
-ytilde_test_OLS = X_test_scaled @ beta_OLS+y_scaler
-ytilde_test_Ridge = X_test_scaled @ beta_Ridge+y_scaler
-
-
-#Calculate MSE
-
-print(" ")
-print("test MSE of OLS:")
-print(MSE(y_test,ytilde_test_OLS))
-print(" ")
-print("test MSE of Ridge")
-print(MSE(y_test,ytilde_test_Ridge))
-
-plt.scatter(x,y,label='Data')
-plt.plot(x, X @ beta_OLS+interceptOLS,'*', label="OLS_Fit")
-plt.plot(x, X @ beta_Ridge+interceptRidge, label="Ridge_Fit")
-plt.grid()
-plt.legend()
-plt.show()
-
-
-# Finally, instead of using our own function we repeat the same example
-# using the **standardscaler** functionality of the library
-# **Scikit-Learn**. Here we limit ourselves to Ridge regression only.
-
-# In[11]:
-
-
-from sklearn import linear_model
-np.random.seed(2018)
-n = 10
-d = 2
-Lambda = 0.01
-
-# Make data set.
-x = np.linspace(-3, 3, n)
-y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)
-
-# Design matrix X does not include the intercept.
-X = np.zeros((n, d))
-for p in range(d):
- X[:, p] = x ** (p+1)
-
-#Split data in train and test
-X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
-# Scale data by subtracting mean value of the input using scikit-learn
-scaler = StandardScaler(with_std=False)
-scaler.fit(X_train)
-X_train_mean = np.mean(X_train,axis=0)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-# We scale also the output, here by our own code
-y_scaler = np.mean(y_train)
-y_train_scaled = y_train - y_scaler
-y_test_scaled = y_test- y_scaler
-
-#Calculate beta
-OLS = LinearRegression()
-betaOLS=OLS.fit(X_train_scaled,y_train_scaled)
-ypredictOLS = OLS.predict(X_test_scaled)
-linear_model.Ridge(Lambda)
-RegRidge.fit(X_train_scaled,y_train_scaled)
-ypredictRidge = RegRidge.predict(X_test_scaled)
-betaOLS = OLS.coef_
-betaRidge = RegRidge.coef_
-print(betaOLS)
-print(betaRidge)
-interceptOLS = np.mean(y_train) - X_train_mean @ betaOLS
-interceptRidge = y_scaler - X_train_mean @ betaRidge
-print(interceptOLS)
-print(interceptRidge)
-#predict value
-ytilde_test_Ridge = X_test_scaled @ betaRidge+y_scaler
-ytilde_test_OLS = X_test_scaled @ betaOLS+y_scaler
-
-#Calculate MSE
-print(" ")
-print("test MSE of OLS")
-print(MSE(y_test,ytilde_test_OLS))
-print(" ")
-print("test MSE of Ridge")
-print(MSE(y_test,ytilde_test_Ridge))
-plt.scatter(x,y,label='Data')
-plt.plot(x, X @ RegRidge.coef_ + RegRidge.intercept_ , label="Ridge_Fit")
-plt.grid()
-plt.legend()
-plt.show()
+# $$
+# \mbox{Var}[\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}]-\mbox{Var}(\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
+# $$
+# The difference is non-negative definite since each component of the
+# matrix product is non-negative definite.
+# This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\boldsymbol{\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
+#
+# For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.
diff --git a/doc/LectureNotes/_build/jupyter_execute/week37_121_0.png b/doc/LectureNotes/_build/jupyter_execute/week37_121_0.png
new file mode 100644
index 000000000..843d99fe9
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week37_121_0.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week37_137_1.png b/doc/LectureNotes/_build/jupyter_execute/week37_137_1.png
new file mode 100644
index 000000000..a1e369a41
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week37_137_1.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week37_139_2.png b/doc/LectureNotes/_build/jupyter_execute/week37_139_2.png
new file mode 100644
index 000000000..9ebeeb751
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week37_139_2.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week37_142_1.png b/doc/LectureNotes/_build/jupyter_execute/week37_142_1.png
new file mode 100644
index 000000000..a8501ad62
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week37_142_1.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week37_146_0.png b/doc/LectureNotes/_build/jupyter_execute/week37_146_0.png
new file mode 100644
index 000000000..fc9e0e560
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week37_146_0.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week37_148_9.png b/doc/LectureNotes/_build/jupyter_execute/week37_148_9.png
new file mode 100644
index 000000000..9fa6833f5
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week37_148_9.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week37_151_1.png b/doc/LectureNotes/_build/jupyter_execute/week37_151_1.png
new file mode 100644
index 000000000..03ff699cc
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week37_151_1.png differ
diff --git a/doc/LectureNotes/week37.ipynb b/doc/LectureNotes/week37.ipynb
index 94e3e70ef..c24beccbd 100644
--- a/doc/LectureNotes/week37.ipynb
+++ b/doc/LectureNotes/week37.ipynb
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
- "id": "672fea2a",
+ "id": "7c0ddab1",
"metadata": {
"editable": true
},
@@ -14,13 +14,13 @@
},
{
"cell_type": "markdown",
- "id": "479a6913",
+ "id": "6c033bf4",
"metadata": {
"editable": true
},
"source": [
"# Week 37: Statistical interpretations and Resampling Methods\n",
- "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n",
+ "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo, Norway\n",
"\n",
"Date: **September 9, 2024**\n",
"\n",
@@ -29,419 +29,69 @@
},
{
"cell_type": "markdown",
- "id": "10b8f456",
+ "id": "e4cd5fa1",
"metadata": {
"editable": true
},
"source": [
- "## Plans for week 37\n",
+ "## Plans for week 37, lecture Monday\n",
"\n",
"**Material for the lecture on Monday September 9.**\n",
"\n",
"\n",
"\n",
- " * Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff\n",
+ " * Statistical interpretation of Ridge and Lasso regression, see also slides from last week\n",
"\n",
- " * Statistical interpretation of Ridge and Lasso regression\n",
+ " * Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff (this may partly be discussed during the exercise sessions as well.\n",
"\n",
" * Readings and Videos:\n",
"\n",
- " * Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap). \n",
+ " * Raschka et al, pages 175-192\n",
+ "\n",
+ " * Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap). See .\n",
"\n",
" * [Video on cross validation](https://www.youtube.com/watch?v=fSytzGwwBVw)\n",
"\n",
" * [Video on Bootstrapping](https://www.youtube.com/watch?v=Xz0x-8-cgaQ)\n",
"\n",
- " * [Video on bias-variance tradeoff](https://www.youtube.com/watch?v=EuBBz3bI-aA)\n",
+ " * [Video on bias-variance tradeoff](https://www.youtube.com/watch?v=EuBBz3bI-aA)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bf80a276",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Plans for week 37, lab sessions\n",
"\n",
- "**Material for the active learning sessions on Tuesday and Wednesday.**\n",
+ "**Material for the lab sessions on Tuesday and Wednesday.**\n",
"\n",
" * Calculations of expectation values\n",
"\n",
+ " * Discussion of resampling techniques\n",
+ "\n",
" * Exercise set for week 37\n",
"\n",
" * Work on project 1\n",
"\n",
- " * See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session. This note is added at the end of these slides.\n",
- "\n",
" * For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended."
]
},
{
"cell_type": "markdown",
- "id": "f7e43b1b",
+ "id": "0e1f0a90",
"metadata": {
"editable": true
},
"source": [
- "## Material from last week and relevant for the weekly exercises"
+ "## Material for lecture Monday September 9"
]
},
{
"cell_type": "markdown",
- "id": "af6a447a",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Linking the regression analysis with a statistical interpretation\n",
- "\n",
- "We will now couple the discussions of ordinary least squares, Ridge\n",
- "and Lasso regression with a statistical interpretation, that is we\n",
- "move from a linear algebra analysis to a statistical analysis. In\n",
- "particular, we will focus on what the regularization terms can result\n",
- "in. We will amongst other things show that the regularization\n",
- "parameter can reduce considerably the variance of the parameters\n",
- "$\\beta$.\n",
- "\n",
- "The\n",
- "advantage of doing linear regression is that we actually end up with\n",
- "analytical expressions for several statistical quantities. \n",
- "Standard least squares and Ridge regression allow us to\n",
- "derive quantities like the variance and other expectation values in a\n",
- "rather straightforward way.\n",
- "\n",
- "It is assumed that $\\varepsilon_i\n",
- "\\sim \\mathcal{N}(0, \\sigma^2)$ and the $\\varepsilon_{i}$ are\n",
- "independent, i.e.:"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b6b7ea38",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\begin{align*} \n",
- "\\mbox{Cov}(\\varepsilon_{i_1},\n",
- "\\varepsilon_{i_2}) & = \\left\\{ \\begin{array}{lcc} \\sigma^2 & \\mbox{if}\n",
- "& i_1 = i_2, \\\\ 0 & \\mbox{if} & i_1 \\not= i_2. \\end{array} \\right.\n",
- "\\end{align*}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e53d8d41",
- "metadata": {
- "editable": true
- },
- "source": [
- "The randomness of $\\varepsilon_i$ implies that\n",
- "$\\mathbf{y}_i$ is also a random variable. In particular,\n",
- "$\\mathbf{y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n",
- "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\beta}$ is a\n",
- "non-random scalar. To specify the parameters of the distribution of\n",
- "$\\mathbf{y}_i$ we need to calculate its first two moments. \n",
- "\n",
- "Recall that $\\boldsymbol{X}$ is a matrix of dimensionality $n\\times p$. The\n",
- "notation above $\\mathbf{X}_{i,\\ast}$ means that we are looking at the\n",
- "row number $i$ and perform a sum over all values $p$."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "401644e7",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Assumptions made\n",
- "\n",
- "The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)\n",
- "that there exists a function $f(\\boldsymbol{x})$ and a normal distributed error $\\boldsymbol{\\varepsilon}\\sim \\mathcal{N}(0, \\sigma^2)$\n",
- "which describe our data"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0603387c",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "eb2b6edf",
- "metadata": {
- "editable": true
- },
- "source": [
- "We approximate this function with our model from the solution of the linear regression equations, that is our\n",
- "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we want to minimize $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, our MSE, with"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f1a08194",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ad40fa0b",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Expectation value and variance\n",
- "\n",
- "We can calculate the expectation value of $\\boldsymbol{y}$ for a given element $i$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d3a4907d",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\begin{align*} \n",
- "\\mathbb{E}(y_i) & =\n",
- "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}) + \\mathbb{E}(\\varepsilon_i)\n",
- "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n",
- "\\end{align*}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d7908c9c",
- "metadata": {
- "editable": true
- },
- "source": [
- "while\n",
- "its variance is"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2528f74e",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n",
- "- \\mathbb{E}(y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( y_i^2 ) -\n",
- "[\\mathbb{E}(y_i)]^2 \\\\ & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n",
- "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \\\\ &\n",
- "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2 \\varepsilon_i\n",
- "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
- "\\ast} \\, \\beta)^2 \\\\ & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2\n",
- "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} +\n",
- "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \n",
- "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n",
- "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2. \n",
- "\\end{align*}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "525f3a59",
- "metadata": {
- "editable": true
- },
- "source": [
- "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
- "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD)."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ead1d419",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Expectation value and variance for $\\boldsymbol{\\beta}$\n",
- "\n",
- "With the OLS expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}}$ we can evaluate the expectation value"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e471f7ab",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mathbb{E}(\\boldsymbol{\\hat{\\beta}}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7886e97f",
- "metadata": {
- "editable": true
- },
- "source": [
- "This means that the estimator of the regression parameters is unbiased.\n",
- "\n",
- "We can also calculate the variance\n",
- "\n",
- "The variance of the optimal value $\\boldsymbol{\\hat{\\beta}}$ is"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "49c32ca5",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\begin{eqnarray*}\n",
- "\\mbox{Var}(\\boldsymbol{\\hat{\\beta}}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n",
- "\\\\\n",
- "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}]^{T} \\}\n",
- "\\\\\n",
- "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "% \\\\\n",
- "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} \\, \\mathbf{y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "% \\\\\n",
- "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{y} \\, \\mathbf{y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "\\\\\n",
- "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "% \\\\\n",
- "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^T \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
- "% \\\\\n",
- "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\boldsymbol{\\beta}^T\n",
- "\\\\\n",
- "& = & \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
- "\\, \\, \\, = \\, \\, \\, \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1},\n",
- "\\end{eqnarray*}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9bd27980",
- "metadata": {
- "editable": true
- },
- "source": [
- "where we have used that $\\mathbb{E} (\\mathbf{y} \\mathbf{y}^{T}) =\n",
- "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n",
- "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n",
- "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n",
- "variance of the estimate of the $j$-th regression coefficient:\n",
- "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $. This may be used to\n",
- "construct a confidence interval for the estimates.\n",
- "\n",
- "In a similar way, we can obtain analytical expressions for say the\n",
- "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n",
- "when we employ Ridge regression, allowing us again to define a confidence interval. \n",
- "\n",
- "It is rather straightforward to show that"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "184f9c3b",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d0f46be1",
- "metadata": {
- "editable": true
- },
- "source": [
- "We see clearly that \n",
- "$\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}$ for any $\\lambda > 0$.\n",
- "\n",
- "We can also compute the variance as"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "03e3ac5f",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "13ec2524",
- "metadata": {
- "editable": true
- },
- "source": [
- "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n",
- "\n",
- "With this, we can compute the difference"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "58c3f236",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}]-\\mbox{Var}(\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "bfcfc2a0",
- "metadata": {
- "editable": true
- },
- "source": [
- "The difference is non-negative definite since each component of the\n",
- "matrix product is non-negative definite. \n",
- "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. \n",
- "\n",
- "For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f3a9f084",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Material for lecture Thursday September 14"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a4338936",
+ "id": "1b290d64",
"metadata": {
"editable": true
},
@@ -464,7 +114,7 @@
},
{
"cell_type": "markdown",
- "id": "d7566301",
+ "id": "ceed3a82",
"metadata": {
"editable": true
},
@@ -476,7 +126,7 @@
},
{
"cell_type": "markdown",
- "id": "c8af7fae",
+ "id": "1d7e8f98",
"metadata": {
"editable": true
},
@@ -489,7 +139,7 @@
},
{
"cell_type": "markdown",
- "id": "959e33e8",
+ "id": "58722d7d",
"metadata": {
"editable": true
},
@@ -501,7 +151,7 @@
},
{
"cell_type": "markdown",
- "id": "c847b78b",
+ "id": "45120edc",
"metadata": {
"editable": true
},
@@ -513,7 +163,7 @@
},
{
"cell_type": "markdown",
- "id": "75c188ba",
+ "id": "4f3b680a",
"metadata": {
"editable": true
},
@@ -525,7 +175,7 @@
},
{
"cell_type": "markdown",
- "id": "60b89ba3",
+ "id": "08489797",
"metadata": {
"editable": true
},
@@ -536,7 +186,7 @@
},
{
"cell_type": "markdown",
- "id": "fd5d5a54",
+ "id": "a1c81f06",
"metadata": {
"editable": true
},
@@ -548,7 +198,7 @@
},
{
"cell_type": "markdown",
- "id": "56f25582",
+ "id": "4e092776",
"metadata": {
"editable": true
},
@@ -559,7 +209,7 @@
},
{
"cell_type": "markdown",
- "id": "9974b303",
+ "id": "79e5358a",
"metadata": {
"editable": true
},
@@ -571,7 +221,7 @@
},
{
"cell_type": "markdown",
- "id": "4112e053",
+ "id": "e5a4f865",
"metadata": {
"editable": true
},
@@ -581,7 +231,7 @@
},
{
"cell_type": "markdown",
- "id": "7be4a14c",
+ "id": "10b81ba7",
"metadata": {
"editable": true
},
@@ -612,7 +262,7 @@
},
{
"cell_type": "markdown",
- "id": "35078f48",
+ "id": "3349cb27",
"metadata": {
"editable": true
},
@@ -624,7 +274,7 @@
},
{
"cell_type": "markdown",
- "id": "cb25ead1",
+ "id": "0fd1502e",
"metadata": {
"editable": true
},
@@ -636,7 +286,7 @@
},
{
"cell_type": "markdown",
- "id": "5cc80036",
+ "id": "43cba0e8",
"metadata": {
"editable": true
},
@@ -646,7 +296,7 @@
},
{
"cell_type": "markdown",
- "id": "8ebc6099",
+ "id": "d6f4e7dc",
"metadata": {
"editable": true
},
@@ -658,7 +308,7 @@
},
{
"cell_type": "markdown",
- "id": "3160dff4",
+ "id": "50c2d673",
"metadata": {
"editable": true
},
@@ -668,7 +318,7 @@
},
{
"cell_type": "markdown",
- "id": "8a3aaa69",
+ "id": "30315a10",
"metadata": {
"editable": true
},
@@ -680,7 +330,7 @@
},
{
"cell_type": "markdown",
- "id": "7af6708c",
+ "id": "fd8cdf51",
"metadata": {
"editable": true
},
@@ -690,7 +340,7 @@
},
{
"cell_type": "markdown",
- "id": "c82e095e",
+ "id": "7323296b",
"metadata": {
"editable": true
},
@@ -702,7 +352,7 @@
},
{
"cell_type": "markdown",
- "id": "9431eca4",
+ "id": "e21334ab",
"metadata": {
"editable": true
},
@@ -712,7 +362,7 @@
},
{
"cell_type": "markdown",
- "id": "1db6c222",
+ "id": "82974124",
"metadata": {
"editable": true
},
@@ -732,7 +382,7 @@
},
{
"cell_type": "markdown",
- "id": "98c457ce",
+ "id": "e053e43c",
"metadata": {
"editable": true
},
@@ -744,7 +394,7 @@
},
{
"cell_type": "markdown",
- "id": "b6a3893f",
+ "id": "94098e13",
"metadata": {
"editable": true
},
@@ -754,7 +404,7 @@
},
{
"cell_type": "markdown",
- "id": "17f0e52e",
+ "id": "27625b1f",
"metadata": {
"editable": true
},
@@ -766,7 +416,7 @@
},
{
"cell_type": "markdown",
- "id": "c2bfaea6",
+ "id": "8f3cc8b4",
"metadata": {
"editable": true
},
@@ -778,7 +428,7 @@
},
{
"cell_type": "markdown",
- "id": "377eefdf",
+ "id": "d7d5464d",
"metadata": {
"editable": true
},
@@ -790,7 +440,7 @@
},
{
"cell_type": "markdown",
- "id": "6da6253e",
+ "id": "c292afee",
"metadata": {
"editable": true
},
@@ -802,7 +452,7 @@
},
{
"cell_type": "markdown",
- "id": "e5906b3e",
+ "id": "cdbbd444",
"metadata": {
"editable": true
},
@@ -814,7 +464,7 @@
},
{
"cell_type": "markdown",
- "id": "792d292c",
+ "id": "100a8014",
"metadata": {
"editable": true
},
@@ -826,7 +476,7 @@
},
{
"cell_type": "markdown",
- "id": "c3bc0674",
+ "id": "d2e40c4d",
"metadata": {
"editable": true
},
@@ -838,7 +488,7 @@
},
{
"cell_type": "markdown",
- "id": "0faadf3f",
+ "id": "1e48456e",
"metadata": {
"editable": true
},
@@ -850,7 +500,7 @@
},
{
"cell_type": "markdown",
- "id": "a2d478a7",
+ "id": "38ced5b6",
"metadata": {
"editable": true
},
@@ -860,7 +510,7 @@
},
{
"cell_type": "markdown",
- "id": "d51ac6ce",
+ "id": "1163c07b",
"metadata": {
"editable": true
},
@@ -872,7 +522,7 @@
},
{
"cell_type": "markdown",
- "id": "91d489d9",
+ "id": "becc087f",
"metadata": {
"editable": true
},
@@ -882,7 +532,7 @@
},
{
"cell_type": "markdown",
- "id": "ebe26b44",
+ "id": "5ca2b09d",
"metadata": {
"editable": true
},
@@ -901,7 +551,7 @@
},
{
"cell_type": "markdown",
- "id": "a67660e2",
+ "id": "6572c77d",
"metadata": {
"editable": true
},
@@ -922,7 +572,7 @@
},
{
"cell_type": "markdown",
- "id": "40b7eac0",
+ "id": "2a3f65e5",
"metadata": {
"editable": true
},
@@ -934,7 +584,7 @@
},
{
"cell_type": "markdown",
- "id": "16a3b6ea",
+ "id": "2d264f77",
"metadata": {
"editable": true
},
@@ -945,7 +595,7 @@
},
{
"cell_type": "markdown",
- "id": "f41ccc6d",
+ "id": "06b8a13c",
"metadata": {
"editable": true
},
@@ -958,7 +608,7 @@
},
{
"cell_type": "markdown",
- "id": "ae81305b",
+ "id": "6d958ed3",
"metadata": {
"editable": true
},
@@ -970,7 +620,7 @@
},
{
"cell_type": "markdown",
- "id": "f4829b81",
+ "id": "6330f3f2",
"metadata": {
"editable": true
},
@@ -980,7 +630,7 @@
},
{
"cell_type": "markdown",
- "id": "812273af",
+ "id": "ed6a9f67",
"metadata": {
"editable": true
},
@@ -992,7 +642,7 @@
},
{
"cell_type": "markdown",
- "id": "31ecba0e",
+ "id": "de42ab8a",
"metadata": {
"editable": true
},
@@ -1002,7 +652,7 @@
},
{
"cell_type": "markdown",
- "id": "ca6682ec",
+ "id": "4f36b126",
"metadata": {
"editable": true
},
@@ -1014,7 +664,7 @@
},
{
"cell_type": "markdown",
- "id": "662eaffb",
+ "id": "0a7eef00",
"metadata": {
"editable": true
},
@@ -1024,7 +674,7 @@
},
{
"cell_type": "markdown",
- "id": "08d5551d",
+ "id": "29940f94",
"metadata": {
"editable": true
},
@@ -1038,7 +688,7 @@
},
{
"cell_type": "markdown",
- "id": "03f3de9c",
+ "id": "f2204679",
"metadata": {
"editable": true
},
@@ -1050,7 +700,7 @@
},
{
"cell_type": "markdown",
- "id": "c57ace49",
+ "id": "d3f4597f",
"metadata": {
"editable": true
},
@@ -1060,7 +710,7 @@
},
{
"cell_type": "markdown",
- "id": "cb014b2b",
+ "id": "c9e9ca01",
"metadata": {
"editable": true
},
@@ -1072,7 +722,7 @@
},
{
"cell_type": "markdown",
- "id": "1054d47e",
+ "id": "1c3062b6",
"metadata": {
"editable": true
},
@@ -1082,7 +732,7 @@
},
{
"cell_type": "markdown",
- "id": "1485d44c",
+ "id": "b455f707",
"metadata": {
"editable": true
},
@@ -1094,7 +744,7 @@
},
{
"cell_type": "markdown",
- "id": "f457f909",
+ "id": "5ac8fdd3",
"metadata": {
"editable": true
},
@@ -1104,7 +754,7 @@
},
{
"cell_type": "markdown",
- "id": "59447b2a",
+ "id": "827927db",
"metadata": {
"editable": true
},
@@ -1116,17 +766,17 @@
},
{
"cell_type": "markdown",
- "id": "4fe26100",
+ "id": "0c740a13",
"metadata": {
"editable": true
},
"source": [
- "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$!"
+ "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta})$!"
]
},
{
"cell_type": "markdown",
- "id": "d73c2d9c",
+ "id": "a855fc3c",
"metadata": {
"editable": true
},
@@ -1142,7 +792,7 @@
},
{
"cell_type": "markdown",
- "id": "8cdf2c97",
+ "id": "9dda8b9e",
"metadata": {
"editable": true
},
@@ -1154,7 +804,7 @@
},
{
"cell_type": "markdown",
- "id": "7272ef9f",
+ "id": "d160205c",
"metadata": {
"editable": true
},
@@ -1164,7 +814,7 @@
},
{
"cell_type": "markdown",
- "id": "0dc5b2b7",
+ "id": "1b7501a6",
"metadata": {
"editable": true
},
@@ -1176,7 +826,7 @@
},
{
"cell_type": "markdown",
- "id": "b6c8c684",
+ "id": "d75a8c65",
"metadata": {
"editable": true
},
@@ -1189,7 +839,7 @@
},
{
"cell_type": "markdown",
- "id": "f681ed50",
+ "id": "c9f718df",
"metadata": {
"editable": true
},
@@ -1201,7 +851,7 @@
},
{
"cell_type": "markdown",
- "id": "5c3bc898",
+ "id": "b07b6425",
"metadata": {
"editable": true
},
@@ -1211,7 +861,7 @@
},
{
"cell_type": "markdown",
- "id": "a3cea046",
+ "id": "215a8730",
"metadata": {
"editable": true
},
@@ -1223,7 +873,7 @@
},
{
"cell_type": "markdown",
- "id": "68276410",
+ "id": "7013d0f0",
"metadata": {
"editable": true
},
@@ -1233,7 +883,7 @@
},
{
"cell_type": "markdown",
- "id": "dfd55611",
+ "id": "d56b4ce0",
"metadata": {
"editable": true
},
@@ -1245,7 +895,7 @@
},
{
"cell_type": "markdown",
- "id": "83656dd8",
+ "id": "f9c6b14d",
"metadata": {
"editable": true
},
@@ -1257,7 +907,7 @@
},
{
"cell_type": "markdown",
- "id": "b62e29fd",
+ "id": "0bde16cb",
"metadata": {
"editable": true
},
@@ -1267,7 +917,7 @@
},
{
"cell_type": "markdown",
- "id": "1de8855c",
+ "id": "0a2869fc",
"metadata": {
"editable": true
},
@@ -1279,7 +929,7 @@
},
{
"cell_type": "markdown",
- "id": "212152f1",
+ "id": "75b83b9b",
"metadata": {
"editable": true
},
@@ -1291,7 +941,7 @@
},
{
"cell_type": "markdown",
- "id": "afb042c4",
+ "id": "1e76d916",
"metadata": {
"editable": true
},
@@ -1303,7 +953,7 @@
},
{
"cell_type": "markdown",
- "id": "ab1fd3c5",
+ "id": "09dfe26d",
"metadata": {
"editable": true
},
@@ -1313,7 +963,7 @@
},
{
"cell_type": "markdown",
- "id": "43979108",
+ "id": "4037962f",
"metadata": {
"editable": true
},
@@ -1325,7 +975,7 @@
},
{
"cell_type": "markdown",
- "id": "96622b66",
+ "id": "39a85ba0",
"metadata": {
"editable": true
},
@@ -1335,7 +985,7 @@
},
{
"cell_type": "markdown",
- "id": "e17a5b6d",
+ "id": "b263a1fe",
"metadata": {
"editable": true
},
@@ -1354,7 +1004,7 @@
},
{
"cell_type": "markdown",
- "id": "24e35401",
+ "id": "41730165",
"metadata": {
"editable": true
},
@@ -1382,7 +1032,7 @@
},
{
"cell_type": "markdown",
- "id": "a37f1d06",
+ "id": "4076cee3",
"metadata": {
"editable": true
},
@@ -1408,7 +1058,7 @@
},
{
"cell_type": "markdown",
- "id": "58f54a23",
+ "id": "11139cb3",
"metadata": {
"editable": true
},
@@ -1425,7 +1075,7 @@
},
{
"cell_type": "markdown",
- "id": "6ed7c809",
+ "id": "b2bead9b",
"metadata": {
"editable": true
},
@@ -1445,7 +1095,7 @@
},
{
"cell_type": "markdown",
- "id": "bed9bb2f",
+ "id": "8df8364e",
"metadata": {
"editable": true
},
@@ -1474,7 +1124,7 @@
},
{
"cell_type": "markdown",
- "id": "2cafa0a3",
+ "id": "6b5273f1",
"metadata": {
"editable": true
},
@@ -1499,7 +1149,7 @@
},
{
"cell_type": "markdown",
- "id": "fefbb980",
+ "id": "a260b371",
"metadata": {
"editable": true
},
@@ -1519,7 +1169,7 @@
},
{
"cell_type": "markdown",
- "id": "4560153d",
+ "id": "2ff1e63c",
"metadata": {
"editable": true
},
@@ -1531,7 +1181,7 @@
},
{
"cell_type": "markdown",
- "id": "ffcbdd5a",
+ "id": "ab4b2e8c",
"metadata": {
"editable": true
},
@@ -1541,7 +1191,7 @@
},
{
"cell_type": "markdown",
- "id": "d055ea55",
+ "id": "85b6fb62",
"metadata": {
"editable": true
},
@@ -1556,7 +1206,7 @@
},
{
"cell_type": "markdown",
- "id": "004f0161",
+ "id": "42449344",
"metadata": {
"editable": true
},
@@ -1569,7 +1219,7 @@
},
{
"cell_type": "markdown",
- "id": "039f4ad5",
+ "id": "35e4f511",
"metadata": {
"editable": true
},
@@ -1582,7 +1232,7 @@
},
{
"cell_type": "markdown",
- "id": "413459f1",
+ "id": "c787234f",
"metadata": {
"editable": true
},
@@ -1594,7 +1244,7 @@
},
{
"cell_type": "markdown",
- "id": "bef022c8",
+ "id": "183711dc",
"metadata": {
"editable": true
},
@@ -1607,7 +1257,7 @@
},
{
"cell_type": "markdown",
- "id": "3d53693d",
+ "id": "2351078d",
"metadata": {
"editable": true
},
@@ -1618,7 +1268,7 @@
},
{
"cell_type": "markdown",
- "id": "e682c32a",
+ "id": "408ba3b5",
"metadata": {
"editable": true
},
@@ -1632,7 +1282,7 @@
},
{
"cell_type": "markdown",
- "id": "f44053ea",
+ "id": "3ea79849",
"metadata": {
"editable": true
},
@@ -1642,7 +1292,7 @@
},
{
"cell_type": "markdown",
- "id": "f10de930",
+ "id": "69439aad",
"metadata": {
"editable": true
},
@@ -1656,7 +1306,7 @@
},
{
"cell_type": "markdown",
- "id": "1f91402a",
+ "id": "e6c29fd8",
"metadata": {
"editable": true
},
@@ -1669,7 +1319,7 @@
},
{
"cell_type": "markdown",
- "id": "2d7a8e36",
+ "id": "4bb52549",
"metadata": {
"editable": true
},
@@ -1682,7 +1332,7 @@
},
{
"cell_type": "markdown",
- "id": "4c756ca2",
+ "id": "3430d548",
"metadata": {
"editable": true
},
@@ -1692,7 +1342,7 @@
},
{
"cell_type": "markdown",
- "id": "908f782e",
+ "id": "9ba636b3",
"metadata": {
"editable": true
},
@@ -1705,7 +1355,7 @@
},
{
"cell_type": "markdown",
- "id": "f12152da",
+ "id": "cb19e98f",
"metadata": {
"editable": true
},
@@ -1715,7 +1365,7 @@
},
{
"cell_type": "markdown",
- "id": "85dd290d",
+ "id": "3cc840d7",
"metadata": {
"editable": true
},
@@ -1728,7 +1378,7 @@
},
{
"cell_type": "markdown",
- "id": "57631d10",
+ "id": "a5062213",
"metadata": {
"editable": true
},
@@ -1740,7 +1390,7 @@
},
{
"cell_type": "markdown",
- "id": "2d4d5ccb",
+ "id": "8d555244",
"metadata": {
"editable": true
},
@@ -1759,7 +1409,7 @@
},
{
"cell_type": "markdown",
- "id": "a29c8551",
+ "id": "c4356604",
"metadata": {
"editable": true
},
@@ -1772,7 +1422,7 @@
},
{
"cell_type": "markdown",
- "id": "6509ba36",
+ "id": "09dc08a1",
"metadata": {
"editable": true
},
@@ -1784,7 +1434,7 @@
},
{
"cell_type": "markdown",
- "id": "733307b1",
+ "id": "dab5056a",
"metadata": {
"editable": true
},
@@ -1797,7 +1447,7 @@
},
{
"cell_type": "markdown",
- "id": "a69b7331",
+ "id": "8a1f8583",
"metadata": {
"editable": true
},
@@ -1817,7 +1467,7 @@
},
{
"cell_type": "markdown",
- "id": "ced659da",
+ "id": "1be08ebb",
"metadata": {
"editable": true
},
@@ -1831,16 +1481,16 @@
"With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ we found \n",
"$\\mathbb{E}(\\boldsymbol{\\beta}) = \\boldsymbol{\\beta}$, which means that the estimator of the regression parameters is unbiased.\n",
"\n",
- "We found also that the variance of the estimate of the $j$-th regression coefficient is\n",
+ "In the exercises this week we show that the variance of the estimate of the $j$-th regression coefficient is\n",
"$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $.\n",
"\n",
- "This quantity will be used to\n",
+ "This quantity can be used to\n",
"construct a confidence interval for the estimates."
]
},
{
"cell_type": "markdown",
- "id": "2c0d0a71",
+ "id": "2a368353",
"metadata": {
"editable": true
},
@@ -1855,7 +1505,7 @@
},
{
"cell_type": "markdown",
- "id": "d3a4d693",
+ "id": "6fee4a1a",
"metadata": {
"editable": true
},
@@ -1867,7 +1517,7 @@
},
{
"cell_type": "markdown",
- "id": "3ed3d852",
+ "id": "1a97c9ef",
"metadata": {
"editable": true
},
@@ -1887,7 +1537,7 @@
},
{
"cell_type": "markdown",
- "id": "e826e70b",
+ "id": "0a70afb2",
"metadata": {
"editable": true
},
@@ -1907,7 +1557,7 @@
},
{
"cell_type": "markdown",
- "id": "10b5d4d5",
+ "id": "dc32eaea",
"metadata": {
"editable": true
},
@@ -1931,7 +1581,7 @@
},
{
"cell_type": "markdown",
- "id": "1645ce3c",
+ "id": "016d0ab3",
"metadata": {
"editable": true
},
@@ -1952,7 +1602,7 @@
},
{
"cell_type": "markdown",
- "id": "4986483a",
+ "id": "3f371f0d",
"metadata": {
"editable": true
},
@@ -1982,7 +1632,7 @@
},
{
"cell_type": "markdown",
- "id": "fe853f3b",
+ "id": "abc56fe7",
"metadata": {
"editable": true
},
@@ -2006,7 +1656,7 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "81ec18f8",
+ "id": "a3d2885b",
"metadata": {
"collapsed": false,
"editable": true
@@ -2045,7 +1695,7 @@
},
{
"cell_type": "markdown",
- "id": "f0b5b588",
+ "id": "5db67125",
"metadata": {
"editable": true
},
@@ -2055,7 +1705,7 @@
},
{
"cell_type": "markdown",
- "id": "262a95e8",
+ "id": "6f1b5417",
"metadata": {
"editable": true
},
@@ -2066,7 +1716,7 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "d235b516",
+ "id": "9d98136a",
"metadata": {
"collapsed": false,
"editable": true
@@ -2086,7 +1736,7 @@
},
{
"cell_type": "markdown",
- "id": "c1620bb7",
+ "id": "c87eac4e",
"metadata": {
"editable": true
},
@@ -2104,7 +1754,7 @@
},
{
"cell_type": "markdown",
- "id": "fd3bd4b5",
+ "id": "2f0f4313",
"metadata": {
"editable": true
},
@@ -2116,7 +1766,7 @@
},
{
"cell_type": "markdown",
- "id": "4a129d86",
+ "id": "7895ee8b",
"metadata": {
"editable": true
},
@@ -2133,7 +1783,7 @@
},
{
"cell_type": "markdown",
- "id": "ff57b405",
+ "id": "be454b28",
"metadata": {
"editable": true
},
@@ -2145,7 +1795,7 @@
},
{
"cell_type": "markdown",
- "id": "5a13919e",
+ "id": "7475f8f9",
"metadata": {
"editable": true
},
@@ -2155,7 +1805,7 @@
},
{
"cell_type": "markdown",
- "id": "45d4a7b6",
+ "id": "fd472029",
"metadata": {
"editable": true
},
@@ -2167,7 +1817,7 @@
},
{
"cell_type": "markdown",
- "id": "c837442b",
+ "id": "b43c8f67",
"metadata": {
"editable": true
},
@@ -2184,7 +1834,7 @@
},
{
"cell_type": "markdown",
- "id": "4b3c7b8e",
+ "id": "c47cfb6d",
"metadata": {
"editable": true
},
@@ -2196,7 +1846,7 @@
},
{
"cell_type": "markdown",
- "id": "7e6f977d",
+ "id": "593188cd",
"metadata": {
"editable": true
},
@@ -2206,7 +1856,7 @@
},
{
"cell_type": "markdown",
- "id": "adeab0fd",
+ "id": "76de4d81",
"metadata": {
"editable": true
},
@@ -2218,7 +1868,7 @@
},
{
"cell_type": "markdown",
- "id": "63b18db5",
+ "id": "afc38c97",
"metadata": {
"editable": true
},
@@ -2228,7 +1878,7 @@
},
{
"cell_type": "markdown",
- "id": "9c7eb00d",
+ "id": "b0f2971c",
"metadata": {
"editable": true
},
@@ -2240,7 +1890,7 @@
},
{
"cell_type": "markdown",
- "id": "c3f4530b",
+ "id": "5a65e495",
"metadata": {
"editable": true
},
@@ -2250,7 +1900,7 @@
},
{
"cell_type": "markdown",
- "id": "90792236",
+ "id": "31419298",
"metadata": {
"editable": true
},
@@ -2266,7 +1916,7 @@
},
{
"cell_type": "markdown",
- "id": "d73c7cd4",
+ "id": "217ddd53",
"metadata": {
"editable": true
},
@@ -2277,7 +1927,7 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "58124207",
+ "id": "ece05002",
"metadata": {
"collapsed": false,
"editable": true
@@ -2342,7 +1992,7 @@
},
{
"cell_type": "markdown",
- "id": "4b4e7bf5",
+ "id": "de6dc4e2",
"metadata": {
"editable": true
},
@@ -2353,7 +2003,7 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "47b138a9",
+ "id": "da218205",
"metadata": {
"collapsed": false,
"editable": true
@@ -2410,7 +2060,7 @@
},
{
"cell_type": "markdown",
- "id": "37b66761",
+ "id": "5ea5df06",
"metadata": {
"editable": true
},
@@ -2448,7 +2098,7 @@
},
{
"cell_type": "markdown",
- "id": "5b5c601e",
+ "id": "40f41d74",
"metadata": {
"editable": true
},
@@ -2459,7 +2109,7 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "9bc32416",
+ "id": "03bb50ae",
"metadata": {
"collapsed": false,
"editable": true
@@ -2541,7 +2191,7 @@
},
{
"cell_type": "markdown",
- "id": "eb934dbe",
+ "id": "cd11e9fd",
"metadata": {
"editable": true
},
@@ -2566,7 +2216,7 @@
},
{
"cell_type": "markdown",
- "id": "c65f802b",
+ "id": "6cf68ff2",
"metadata": {
"editable": true
},
@@ -2594,7 +2244,7 @@
},
{
"cell_type": "markdown",
- "id": "9accabd4",
+ "id": "c445bfb2",
"metadata": {
"editable": true
},
@@ -2607,7 +2257,7 @@
{
"cell_type": "code",
"execution_count": 6,
- "id": "d5ed3f5c",
+ "id": "8e26035a",
"metadata": {
"collapsed": false,
"editable": true
@@ -2707,7 +2357,7 @@
},
{
"cell_type": "markdown",
- "id": "db1976f4",
+ "id": "86a6b012",
"metadata": {
"editable": true
},
@@ -2718,7 +2368,7 @@
{
"cell_type": "code",
"execution_count": 7,
- "id": "3c35540c",
+ "id": "4a0e1afd",
"metadata": {
"collapsed": false,
"editable": true
@@ -2807,7 +2457,7 @@
},
{
"cell_type": "markdown",
- "id": "e60fabdf",
+ "id": "bb1e0f15",
"metadata": {
"editable": true
},
@@ -2817,7 +2467,7 @@
},
{
"cell_type": "markdown",
- "id": "9b6750d6",
+ "id": "83c87d33",
"metadata": {
"editable": true
},
@@ -2830,7 +2480,7 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "5e6b91e1",
+ "id": "74eaac59",
"metadata": {
"collapsed": false,
"editable": true
@@ -2908,622 +2558,364 @@
},
{
"cell_type": "markdown",
- "id": "e83d7b10",
+ "id": "aa703354",
"metadata": {
"editable": true
},
"source": [
- "## Notes on scaling with examples\n",
- "\n",
- "The programs here use both ordinrary least squares (OLS) and Ridge\n",
- "regression with one value only for the hyperparameter $\\lambda$. The\n",
- "first example has no scaling and includes the intercept as well and we\n",
- "are trying to fit a second-order polynomial. The second code takes out\n",
- "the intercept and subtracts the mean values of each column of the\n",
- "design matrix and the mean value of the outputs.\n",
- "\n",
- "The third and final code uses **Scikit-Learn** as library in order to\n",
- "calculate the optimal parameters for OLS and Ridge regression. Note\n",
- "that it is highly recommended to not include the intercept in Ridge\n",
- "and Lasso regression, in order to avoid penalizing the optimization by\n",
- "the intercept. The second and third codes do thus not include the\n",
- "intercept. In the second code we do the scaling ourselves while the\n",
- "last code uses the standard scaler option included in **Scikit-Learn**, known as centering (where\n",
- "we subtract the mean values)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "id": "73807393",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "import matplotlib.pyplot as plt\n",
- "import numpy as np\n",
- "from sklearn.linear_model import LinearRegression\n",
- "from sklearn.preprocessing import PolynomialFeatures\n",
- "from sklearn.model_selection import train_test_split\n",
- "from sklearn.preprocessing import StandardScaler\n",
- "\n",
- "def MSE(y_data,y_model):\n",
- " n = np.size(y_model)\n",
- " return np.sum((y_data-y_model)**2)/n\n",
- "\n",
- "def OLS_fit_beta(X, y):\n",
- " return np.linalg.pinv(X.T @ X) @ X.T @ y\n",
- "\n",
- "def Ridge_fit_beta(X, y,L,d):\n",
- " I = np.eye(d,d)\n",
- " return np.linalg.pinv(X.T @ X + L*I) @ X.T @ y\n",
- "\n",
- "# Same random numbers for each test.\n",
- "np.random.seed(2018)\n",
- "n = 100\n",
- "d = 3\n",
- "# hyperparameter lambda\n",
- "Lambda = 0.01\n",
- "\n",
- "# Make data set, simple second-order polynomial\n",
- "x = np.linspace(-3, 3, n)\n",
- "y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)\n",
- "\n",
- "# The design matrix X includes the intercept and no scaling is made\n",
- "X = np.zeros((len(x), d))\n",
- "for p in range(d): \n",
- " X[:, p] = x ** (p) \n",
- "\n",
- "\n",
- "#Split data, no scaling is used and we include the intercept\n",
- "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
- "\n",
- "\n",
- "#Calculate beta, own code\n",
- "beta_OLS = OLS_fit_beta(X_train, y_train)\n",
- "beta_Ridge = Ridge_fit_beta(X_train, y_train,Lambda,d)\n",
- "print(beta_OLS)\n",
- "print(beta_Ridge)\n",
- "#predict value\n",
- "ytilde_test_OLS = X_test @ beta_OLS\n",
- "ytilde_test_Ridge = X_test @ beta_Ridge\n",
- "\n",
- "#Calculate MSE\n",
- "print(\" \")\n",
- "print(\"test MSE of OLS:\")\n",
- "print(MSE(y_test,ytilde_test_OLS))\n",
- "print(\" \")\n",
- "print(\"test MSE of Ridge\")\n",
- "print(MSE(y_test,ytilde_test_Ridge))\n",
- "\n",
- "plt.scatter(x,y,label='Data')\n",
- "plt.plot(x, X @ beta_OLS,'*', label=\"OLS_Fit\")\n",
- "plt.plot(x, X @ beta_Ridge, label=\"Ridge_Fit\")\n",
- "plt.grid()\n",
- "plt.legend()\n",
- "plt.show()"
+ "## Material for the lab sessions"
]
},
{
"cell_type": "markdown",
- "id": "b09d20a2",
+ "id": "f8bffca6",
"metadata": {
"editable": true
},
"source": [
- "In this example we do not include the intercept and we scale the data by subtracting the mean values. This follows the discussion in the [lecture material](https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#more-on-rescaling-data).\n",
- "see also the weekly slides [for week 36](https://compphysics.github.io/MachineLearning/doc/pub/week36/html/._week36-bs029.html).\n",
- "It is recommended whrn we use Ridge and Lasso regression to not include the intercept in the optimization process.\n",
+ "## Linking the regression analysis with a statistical interpretation\n",
"\n",
- "Before we discuss the code, we repeat some of the basic math from the slides of week 36.\n",
+ "We will now couple the discussions of ordinary least squares, Ridge\n",
+ "and Lasso regression with a statistical interpretation, that is we\n",
+ "move from a linear algebra analysis to a statistical analysis. In\n",
+ "particular, we will focus on what the regularization terms can result\n",
+ "in. We will amongst other things show that the regularization\n",
+ "parameter can reduce considerably the variance of the parameters\n",
+ "$\\beta$.\n",
"\n",
- "Let us try to understand what this may imply mathematically when we\n",
- "subtract the mean values, also known as *zero centering* or simply *centering*. For\n",
- "simplicity, we will focus on ordinary regression, as done in the above example.\n",
+ "The\n",
+ "advantage of doing linear regression is that we actually end up with\n",
+ "analytical expressions for several statistical quantities. \n",
+ "Standard least squares and Ridge regression allow us to\n",
+ "derive quantities like the variance and other expectation values in a\n",
+ "rather straightforward way.\n",
"\n",
- "The cost/loss function for regression is"
+ "It is assumed that $\\varepsilon_i\n",
+ "\\sim \\mathcal{N}(0, \\sigma^2)$ and the $\\varepsilon_{i}$ are\n",
+ "independent, i.e.:"
]
},
{
"cell_type": "markdown",
- "id": "ef1cec97",
+ "id": "109c07d6",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n",
+ "\\begin{align*} \n",
+ "\\mbox{Cov}(\\varepsilon_{i_1},\n",
+ "\\varepsilon_{i_2}) & = \\left\\{ \\begin{array}{lcc} \\sigma^2 & \\mbox{if}\n",
+ "& i_1 = i_2, \\\\ 0 & \\mbox{if} & i_1 \\not= i_2. \\end{array} \\right.\n",
+ "\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "e43c422b",
+ "id": "1ea48f22",
"metadata": {
"editable": true
},
"source": [
- "Recall also that we use the squared value. This expression can lead to an\n",
- "increased penalty for higher differences between predicted and\n",
- "output/target values.\n",
+ "The randomness of $\\varepsilon_i$ implies that\n",
+ "$\\mathbf{y}_i$ is also a random variable. In particular,\n",
+ "$\\mathbf{y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n",
+ "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\beta}$ is a\n",
+ "non-random scalar. To specify the parameters of the distribution of\n",
+ "$\\mathbf{y}_i$ we need to calculate its first two moments. \n",
"\n",
- "What we have done is to single out the $\\beta_0$ term in the\n",
- "definition of the mean squared error (MSE). The design matrix $X$\n",
- "does in this case not contain any intercept column. When we take the\n",
- "derivative with respect to $\\beta_0$, we want the derivative to obey"
+ "Recall that $\\boldsymbol{X}$ is a matrix of dimensionality $n\\times p$. The\n",
+ "notation above $\\mathbf{X}_{i,\\ast}$ means that we are looking at the\n",
+ "row number $i$ and perform a sum over all values $p$."
]
},
{
"cell_type": "markdown",
- "id": "852577bc",
+ "id": "212f90ec",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Assumptions made\n",
+ "\n",
+ "The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)\n",
+ "that there exists a function $f(\\boldsymbol{x})$ and a normal distributed error $\\boldsymbol{\\varepsilon}\\sim \\mathcal{N}(0, \\sigma^2)$\n",
+ "which describe our data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a019e7cb",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C}{\\partial \\beta_j} = 0,\n",
+ "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "4edd229a",
+ "id": "2ca46c43",
"metadata": {
"editable": true
},
"source": [
- "for all $j$. For $\\beta_0$ we have"
+ "We approximate this function with our model from the solution of the linear regression equations, that is our\n",
+ "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we want to minimize $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, our MSE, with"
]
},
{
"cell_type": "markdown",
- "id": "bb93d608",
+ "id": "c61ed9c8",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n",
+ "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "5ef162e3",
+ "id": "96d84043",
"metadata": {
"editable": true
},
"source": [
- "Multiplying away the constant $2/n$, we obtain"
+ "## Expectation value and variance\n",
+ "\n",
+ "We can calculate the expectation value of $\\boldsymbol{y}$ for a given element $i$"
]
},
{
"cell_type": "markdown",
- "id": "b7c5467d",
+ "id": "e53b66df",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n",
+ "\\begin{align*} \n",
+ "\\mathbb{E}(y_i) & =\n",
+ "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}) + \\mathbb{E}(\\varepsilon_i)\n",
+ "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n",
+ "\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "493c75d6",
+ "id": "71d542a3",
"metadata": {
"editable": true
},
"source": [
- "Let us specialize first to the case where we have only two parameters $\\beta_0$ and $\\beta_1$.\n",
- "Our result for $\\beta_0$ simplifies then to"
+ "while\n",
+ "its variance is"
]
},
{
"cell_type": "markdown",
- "id": "862f2222",
+ "id": "01b18637",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n",
+ "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n",
+ "- \\mathbb{E}(y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( y_i^2 ) -\n",
+ "[\\mathbb{E}(y_i)]^2 \\\\ & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n",
+ "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \\\\ &\n",
+ "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2 \\varepsilon_i\n",
+ "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
+ "\\ast} \\, \\beta)^2 \\\\ & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2\n",
+ "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} +\n",
+ "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \n",
+ "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n",
+ "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2. \n",
+ "\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "e0d0a4e0",
+ "id": "e35bdcc1",
"metadata": {
"editable": true
},
"source": [
- "We obtain then"
+ "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
+ "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD)."
]
},
{
"cell_type": "markdown",
- "id": "73d327de",
+ "id": "15b7ea3c",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Expectation value and variance for $\\boldsymbol{\\beta}$\n",
+ "\n",
+ "With the OLS expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}}$ we can evaluate the expectation value"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a4c5f625",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n",
+ "\\mathbb{E}(\\boldsymbol{\\hat{\\beta}}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "8be27cbd",
+ "id": "450007e4",
"metadata": {
"editable": true
},
"source": [
- "If we define"
+ "This means that the estimator of the regression parameters is unbiased.\n",
+ "\n",
+ "We can also calculate the variance\n",
+ "\n",
+ "The variance of the optimal value $\\boldsymbol{\\hat{\\beta}}$ is"
]
},
{
"cell_type": "markdown",
- "id": "d49a9bac",
+ "id": "3bcf81c1",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\mu_{\\boldsymbol{x}_1}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1},\n",
+ "\\begin{eqnarray*}\n",
+ "\\mbox{Var}(\\boldsymbol{\\hat{\\beta}}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n",
+ "\\\\\n",
+ "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}]^{T} \\}\n",
+ "\\\\\n",
+ "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "% \\\\\n",
+ "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} \\, \\mathbf{y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "% \\\\\n",
+ "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{y} \\, \\mathbf{y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "\\\\\n",
+ "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "% \\\\\n",
+ "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^T \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
+ "% \\\\\n",
+ "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\boldsymbol{\\beta}^T\n",
+ "\\\\\n",
+ "& = & \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+ "\\, \\, \\, = \\, \\, \\, \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1},\n",
+ "\\end{eqnarray*}\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "dd964823",
+ "id": "f69a019e",
"metadata": {
"editable": true
},
"source": [
- "and the mean value of the outputs as"
+ "where we have used that $\\mathbb{E} (\\mathbf{y} \\mathbf{y}^{T}) =\n",
+ "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n",
+ "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n",
+ "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n",
+ "variance of the estimate of the $j$-th regression coefficient:\n",
+ "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $. This may be used to\n",
+ "construct a confidence interval for the estimates.\n",
+ "\n",
+ "In a similar way, we can obtain analytical expressions for say the\n",
+ "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n",
+ "when we employ Ridge regression, allowing us again to define a confidence interval. \n",
+ "\n",
+ "It is rather straightforward to show that"
]
},
{
"cell_type": "markdown",
- "id": "09a87e67",
+ "id": "95b37da0",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\mu_y=\\frac{1}{n}\\sum_{i=0}^{n-1}y_i,\n",
+ "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "9b8797a0",
+ "id": "241c2335",
"metadata": {
"editable": true
},
"source": [
- "we have"
+ "We see clearly that \n",
+ "$\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}$ for any $\\lambda > 0$.\n",
+ "\n",
+ "We can also compute the variance as"
]
},
{
"cell_type": "markdown",
- "id": "30aeafdb",
+ "id": "cf794384",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\beta_0 = \\mu_y - \\beta_1\\mu_{\\boldsymbol{x}_1}.\n",
+ "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "dc867cbb",
+ "id": "4cd5260a",
"metadata": {
"editable": true
},
"source": [
- "In the general case with more parameters than $\\beta_0$ and $\\beta_1$, we have"
+ "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n",
+ "\n",
+ "With this, we can compute the difference"
]
},
{
"cell_type": "markdown",
- "id": "f154321e",
+ "id": "7361e846",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n",
+ "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}]-\\mbox{Var}(\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "6015f267",
+ "id": "6a07837b",
"metadata": {
"editable": true
},
"source": [
- "We can rewrite the latter equation as"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f181e88f",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\beta_j,\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a6e6c0e7",
- "metadata": {
- "editable": true
- },
- "source": [
- "where we have defined"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "708652ea",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mu_{\\boldsymbol{x}_j}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{ij},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b360788c",
- "metadata": {
- "editable": true
- },
- "source": [
- "the mean value for all elements of the column vector $\\boldsymbol{x}_j$.\n",
+ "The difference is non-negative definite since each component of the\n",
+ "matrix product is non-negative definite. \n",
+ "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. \n",
"\n",
- "Replacing $y_i$ with $y_i - y_i - \\overline{\\boldsymbol{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8f473602",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "70f518a7",
- "metadata": {
- "editable": true
- },
- "source": [
- "If we minimize with respect to $\\boldsymbol{\\beta}$ we have then"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "674d02d9",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6715261c",
- "metadata": {
- "editable": true
- },
- "source": [
- "where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n",
- "and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n",
- "\n",
- "For Ridge regression we need to add $\\lambda \\boldsymbol{\\beta}^T\\boldsymbol{\\beta}$ to the cost function and get then"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "17600ee3",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a8c6f068",
- "metadata": {
- "editable": true
- },
- "source": [
- "Now we try to implement this."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "1d89f80d",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "\n",
- "np.random.seed(2018)\n",
- "n = 100\n",
- "# we do not include the intercept\n",
- "d = 2\n",
- "Lambda = 0.01\n",
- "\n",
- "# Make data set.\n",
- "x = np.linspace(-3, 3, n)\n",
- "y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)\n",
- "\n",
- "#Design matrix X does not include the intercept. \n",
- "X = np.zeros((len(x), d))\n",
- "for p in range(d): \n",
- " X[:, p] = x ** (p+1)\n",
- "\n",
- "\n",
- "#Split data in train and test\n",
- "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
- "\n",
- "# Scale data by subtracting mean value,own implementation\n",
- "#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable\n",
- "X_train_mean = np.mean(X_train,axis=0)\n",
- "#Center by removing mean from each feature\n",
- "X_train_scaled = X_train - X_train_mean\n",
- "X_test_scaled = X_test - X_train_mean\n",
- "#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered, note)\n",
- "y_scaler = np.mean(y_train)\n",
- "y_train_scaled = y_train - y_scaler\n",
- "\n",
- "\n",
- "#Calculate beta\n",
- "beta_OLS = OLS_fit_beta(X_train_scaled, y_train_scaled)\n",
- "beta_Ridge = Ridge_fit_beta(X_train_scaled, y_train_scaled,Lambda,d)\n",
- "print(beta_OLS)\n",
- "print(beta_Ridge)\n",
- "# calculate intercepts and print them\n",
- "interceptOLS = y_scaler - X_train_mean @ beta_OLS\n",
- "interceptRidge = y_scaler - X_train_mean @ beta_Ridge\n",
- "print(interceptOLS)\n",
- "print(interceptRidge)\n",
- "\n",
- "#predict value with intercept\n",
- "ytilde_test_OLS = X_test_scaled @ beta_OLS+y_scaler\n",
- "ytilde_test_Ridge = X_test_scaled @ beta_Ridge+y_scaler\n",
- "\n",
- "\n",
- "#Calculate MSE\n",
- "\n",
- "print(\" \")\n",
- "print(\"test MSE of OLS:\")\n",
- "print(MSE(y_test,ytilde_test_OLS))\n",
- "print(\" \")\n",
- "print(\"test MSE of Ridge\")\n",
- "print(MSE(y_test,ytilde_test_Ridge))\n",
- "\n",
- "plt.scatter(x,y,label='Data')\n",
- "plt.plot(x, X @ beta_OLS+interceptOLS,'*', label=\"OLS_Fit\")\n",
- "plt.plot(x, X @ beta_Ridge+interceptRidge, label=\"Ridge_Fit\")\n",
- "plt.grid()\n",
- "plt.legend()\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "522668c3",
- "metadata": {
- "editable": true
- },
- "source": [
- "Finally, instead of using our own function we repeat the same example\n",
- "using the **standardscaler** functionality of the library\n",
- "**Scikit-Learn**. Here we limit ourselves to Ridge regression only."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "id": "9954bcc1",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "from sklearn import linear_model\n",
- "np.random.seed(2018)\n",
- "n = 10\n",
- "d = 2\n",
- "Lambda = 0.01\n",
- "\n",
- "# Make data set.\n",
- "x = np.linspace(-3, 3, n)\n",
- "y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)\n",
- "\n",
- "# Design matrix X does not include the intercept. \n",
- "X = np.zeros((n, d))\n",
- "for p in range(d): \n",
- " X[:, p] = x ** (p+1)\n",
- "\n",
- "#Split data in train and test\n",
- "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
- "# Scale data by subtracting mean value of the input using scikit-learn\n",
- "scaler = StandardScaler(with_std=False)\n",
- "scaler.fit(X_train)\n",
- "X_train_mean = np.mean(X_train,axis=0)\n",
- "X_train_scaled = scaler.transform(X_train)\n",
- "X_test_scaled = scaler.transform(X_test)\n",
- "# We scale also the output, here by our own code\n",
- "y_scaler = np.mean(y_train)\n",
- "y_train_scaled = y_train - y_scaler\n",
- "y_test_scaled = y_test- y_scaler\n",
- "\n",
- "#Calculate beta\n",
- "OLS = LinearRegression()\n",
- "betaOLS=OLS.fit(X_train_scaled,y_train_scaled)\n",
- "ypredictOLS = OLS.predict(X_test_scaled)\n",
- "linear_model.Ridge(Lambda)\n",
- "RegRidge.fit(X_train_scaled,y_train_scaled)\n",
- "ypredictRidge = RegRidge.predict(X_test_scaled)\n",
- "betaOLS = OLS.coef_\n",
- "betaRidge = RegRidge.coef_\n",
- "print(betaOLS)\n",
- "print(betaRidge)\n",
- "interceptOLS = np.mean(y_train) - X_train_mean @ betaOLS\n",
- "interceptRidge = y_scaler - X_train_mean @ betaRidge\n",
- "print(interceptOLS)\n",
- "print(interceptRidge)\n",
- "#predict value \n",
- "ytilde_test_Ridge = X_test_scaled @ betaRidge+y_scaler\n",
- "ytilde_test_OLS = X_test_scaled @ betaOLS+y_scaler\n",
- "\n",
- "#Calculate MSE\n",
- "print(\" \")\n",
- "print(\"test MSE of OLS\")\n",
- "print(MSE(y_test,ytilde_test_OLS))\n",
- "print(\" \")\n",
- "print(\"test MSE of Ridge\")\n",
- "print(MSE(y_test,ytilde_test_Ridge))\n",
- "plt.scatter(x,y,label='Data')\n",
- "plt.plot(x, X @ RegRidge.coef_ + RegRidge.intercept_ , label=\"Ridge_Fit\")\n",
- "plt.grid()\n",
- "plt.legend()\n",
- "plt.show()"
+ "For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended."
]
}
],