+
Activation functions
Finally, before we look at the neural network, we will look at the
activation functions which can be specified between the hidden layers
and as the output function. Each function can be valued for any given
diff --git a/doc/LectureNotes/_build/jupyter_execute/week42.ipynb b/doc/LectureNotes/_build/jupyter_execute/week42.ipynb
index f5f67eb6c..27badbb0c 100644
--- a/doc/LectureNotes/_build/jupyter_execute/week42.ipynb
+++ b/doc/LectureNotes/_build/jupyter_execute/week42.ipynb
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
- "id": "474b773e",
+ "id": "71674611",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
- "id": "a8ac8249",
+ "id": "65b3502e",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
- "id": "c8760378",
+ "id": "1d840be4",
"metadata": {
"editable": true
},
@@ -56,7 +56,7 @@
},
{
"cell_type": "markdown",
- "id": "76c6895d",
+ "id": "8c43f62c",
"metadata": {
"editable": true
},
@@ -73,7 +73,7 @@
},
{
"cell_type": "markdown",
- "id": "ebb39354",
+ "id": "bb52c881",
"metadata": {
"editable": true
},
@@ -92,7 +92,7 @@
},
{
"cell_type": "markdown",
- "id": "cfb26b1b",
+ "id": "e53a998a",
"metadata": {
"editable": true
},
@@ -108,7 +108,7 @@
},
{
"cell_type": "markdown",
- "id": "28f60678",
+ "id": "2be1dbc1",
"metadata": {
"editable": true
},
@@ -119,7 +119,7 @@
},
{
"cell_type": "markdown",
- "id": "3c7f8f61",
+ "id": "d81e5954",
"metadata": {
"editable": true
},
@@ -133,7 +133,7 @@
},
{
"cell_type": "markdown",
- "id": "9ee5faa5",
+ "id": "bf67ca94",
"metadata": {
"editable": true
},
@@ -148,7 +148,7 @@
},
{
"cell_type": "markdown",
- "id": "e82fbb01",
+ "id": "ea5ccdfd",
"metadata": {
"editable": true
},
@@ -160,7 +160,7 @@
},
{
"cell_type": "markdown",
- "id": "e1b4862d",
+ "id": "a67526dc",
"metadata": {
"editable": true
},
@@ -174,7 +174,7 @@
},
{
"cell_type": "markdown",
- "id": "f7b41bcb",
+ "id": "004f244e",
"metadata": {
"editable": true
},
@@ -186,7 +186,7 @@
},
{
"cell_type": "markdown",
- "id": "2825a689",
+ "id": "d5019705",
"metadata": {
"editable": true
},
@@ -202,7 +202,7 @@
},
{
"cell_type": "markdown",
- "id": "1853d23c",
+ "id": "b28a1451",
"metadata": {
"editable": true
},
@@ -218,7 +218,7 @@
},
{
"cell_type": "markdown",
- "id": "1625a8c3",
+ "id": "bb9e817a",
"metadata": {
"editable": true
},
@@ -230,7 +230,7 @@
},
{
"cell_type": "markdown",
- "id": "7427a24a",
+ "id": "cb070b5e",
"metadata": {
"editable": true
},
@@ -240,7 +240,7 @@
},
{
"cell_type": "markdown",
- "id": "d4911a64",
+ "id": "f68d4801",
"metadata": {
"editable": true
},
@@ -252,7 +252,7 @@
},
{
"cell_type": "markdown",
- "id": "0f290988",
+ "id": "fffa97bd",
"metadata": {
"editable": true
},
@@ -262,7 +262,7 @@
},
{
"cell_type": "markdown",
- "id": "4cbbad95",
+ "id": "7f751c77",
"metadata": {
"editable": true
},
@@ -274,7 +274,7 @@
},
{
"cell_type": "markdown",
- "id": "cc8122a6",
+ "id": "e8f8479b",
"metadata": {
"editable": true
},
@@ -284,7 +284,7 @@
},
{
"cell_type": "markdown",
- "id": "39f03825",
+ "id": "9d52f786",
"metadata": {
"editable": true
},
@@ -296,7 +296,7 @@
},
{
"cell_type": "markdown",
- "id": "3eeb7a29",
+ "id": "cdb55ad0",
"metadata": {
"editable": true
},
@@ -312,7 +312,7 @@
},
{
"cell_type": "markdown",
- "id": "33e1d35f",
+ "id": "ece1a1cc",
"metadata": {
"editable": true
},
@@ -324,7 +324,7 @@
},
{
"cell_type": "markdown",
- "id": "e846a441",
+ "id": "e2af50fb",
"metadata": {
"editable": true
},
@@ -336,7 +336,7 @@
},
{
"cell_type": "markdown",
- "id": "fbb3f1ff",
+ "id": "c883f2ef",
"metadata": {
"editable": true
},
@@ -346,7 +346,7 @@
},
{
"cell_type": "markdown",
- "id": "40200643",
+ "id": "f1129306",
"metadata": {
"editable": true
},
@@ -358,7 +358,7 @@
},
{
"cell_type": "markdown",
- "id": "a2e96689",
+ "id": "1d14bedd",
"metadata": {
"editable": true
},
@@ -368,7 +368,7 @@
},
{
"cell_type": "markdown",
- "id": "1a014316",
+ "id": "84378f69",
"metadata": {
"editable": true
},
@@ -384,7 +384,7 @@
},
{
"cell_type": "markdown",
- "id": "06a0b720",
+ "id": "7f6f41e3",
"metadata": {
"editable": true
},
@@ -396,7 +396,7 @@
},
{
"cell_type": "markdown",
- "id": "48581ce4",
+ "id": "f38cb151",
"metadata": {
"editable": true
},
@@ -408,7 +408,7 @@
},
{
"cell_type": "markdown",
- "id": "58f46792",
+ "id": "d7f60566",
"metadata": {
"editable": true
},
@@ -420,7 +420,7 @@
},
{
"cell_type": "markdown",
- "id": "dc5aa924",
+ "id": "81219134",
"metadata": {
"editable": true
},
@@ -432,7 +432,7 @@
},
{
"cell_type": "markdown",
- "id": "407690af",
+ "id": "8f0f27a7",
"metadata": {
"editable": true
},
@@ -444,7 +444,7 @@
},
{
"cell_type": "markdown",
- "id": "8f5cacad",
+ "id": "aa9974f2",
"metadata": {
"editable": true
},
@@ -454,7 +454,7 @@
},
{
"cell_type": "markdown",
- "id": "79f592c1",
+ "id": "02021c85",
"metadata": {
"editable": true
},
@@ -470,7 +470,7 @@
},
{
"cell_type": "markdown",
- "id": "412247f9",
+ "id": "d5b4c3d8",
"metadata": {
"editable": true
},
@@ -482,7 +482,7 @@
},
{
"cell_type": "markdown",
- "id": "83c09361",
+ "id": "0c0f2d45",
"metadata": {
"editable": true
},
@@ -494,7 +494,7 @@
},
{
"cell_type": "markdown",
- "id": "d341a3c7",
+ "id": "9f8b567b",
"metadata": {
"editable": true
},
@@ -504,7 +504,7 @@
},
{
"cell_type": "markdown",
- "id": "2bb5b78f",
+ "id": "6afa5b0f",
"metadata": {
"editable": true
},
@@ -516,7 +516,7 @@
},
{
"cell_type": "markdown",
- "id": "3d6558a8",
+ "id": "a8447e5f",
"metadata": {
"editable": true
},
@@ -530,7 +530,7 @@
},
{
"cell_type": "markdown",
- "id": "f55c404e",
+ "id": "0262d1c4",
"metadata": {
"editable": true
},
@@ -548,7 +548,7 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "076950d8",
+ "id": "91932727",
"metadata": {
"collapsed": false,
"editable": true
@@ -678,7 +678,7 @@
},
{
"cell_type": "markdown",
- "id": "b5a2150f",
+ "id": "6695945c",
"metadata": {
"editable": true
},
@@ -688,7 +688,7 @@
},
{
"cell_type": "markdown",
- "id": "cb93f2d2",
+ "id": "30bd4411",
"metadata": {
"editable": true
},
@@ -700,51 +700,51 @@
"layer with two hidden nodes and one output layer with one output node/neuron only (see graph)..\n",
"\n",
"We need to define the following parameters and variables with the input layer (layer $(0)$) \n",
- "where we label the nodes $x_0$ and $x_1$"
+ "where we label the nodes $x_1$ and $x_2$"
]
},
{
"cell_type": "markdown",
- "id": "893b1b33",
+ "id": "03303707",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "x_0 = a_0^{(0)} \\wedge x_1 = a_1^{(0)}.\n",
+ "x_1 = a_1^{(0)} \\wedge x_2 = a_2^{(0)}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "f1554902",
+ "id": "24dc6874",
"metadata": {
"editable": true
},
"source": [
- "The hidden layer (layer $(1)$) has nodes which yield the outputs $a_0^{(1)}$ and $a_1^{(1)}$) with weight $\\boldsymbol{w}$ and bias $\\boldsymbol{b}$ parameters"
+ "The hidden layer (layer $(1)$) has nodes which yield the outputs $a_1^{(1)}$ and $a_2^{(1)}$) with weight $\\boldsymbol{w}$ and bias $\\boldsymbol{b}$ parameters"
]
},
{
"cell_type": "markdown",
- "id": "e01cd011",
+ "id": "9229190d",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "w_{ij}^{(1)}=\\left\\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)}\\right\\} \\wedge b^{(1)}=\\left\\{b_0^{(1)},b_1^{(1)}\\right\\}.\n",
+ "w_{ij}^{(1)}=\\left\\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)}\\right\\} \\wedge b^{(1)}=\\left\\{b_1^{(1)},b_2^{(1)}\\right\\}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "691e42c5",
+ "id": "db21c4eb",
"metadata": {
"editable": true
},
"source": [
- "## Layout of a simple neural network with two input nodes, one hidden layer and one output node\n",
+ "## Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node\n",
"\n",
"\n",
"\n",
@@ -755,31 +755,31 @@
},
{
"cell_type": "markdown",
- "id": "57573a8a",
+ "id": "cf9b69e8",
"metadata": {
"editable": true
},
"source": [
"## The ouput layer\n",
"\n",
- "Finally, we have the ouput layer given by layer label $(2)$ with output $a^{(2)}$ and weights and biases to be determined given by the variables"
+ "We have the ouput layer given by layer label $(2)$ with output $a^{(2)}$ and weights and biases to be determined given by the variables"
]
},
{
"cell_type": "markdown",
- "id": "91ff8dfb",
+ "id": "53130107",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "w_{i}^{(2)}=\\left\\{w_{0}^{(2)},w_{1}^{(2)}\\right\\} \\wedge b^{(2)}.\n",
+ "w_{i}^{(2)}=\\left\\{w_{1}^{(2)},w_{2}^{(2)}\\right\\} \\wedge b^{(2)}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "315f787f",
+ "id": "5835245c",
"metadata": {
"editable": true
},
@@ -790,19 +790,19 @@
},
{
"cell_type": "markdown",
- "id": "612ebd67",
+ "id": "dd31d181",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{\\Theta}=\\left\\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)},w_{0}^{(2)},w_{1}^{(2)},b_0^{(1)},b_1^{(1)},b^{(2)}\\right\\}.\n",
+ "\\boldsymbol{\\Theta}=\\left\\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)},w_{1}^{(2)},w_{2}^{(2)},b_1^{(1)},b_2^{(1)},b^{(2)}\\right\\}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "c28762dc",
+ "id": "36a9d52a",
"metadata": {
"editable": true
},
@@ -815,19 +815,19 @@
},
{
"cell_type": "markdown",
- "id": "b4d0309d",
+ "id": "3af3b240",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\begin{bmatrix}z_0^{(1)} \\\\ z_1^{(1)} \\end{bmatrix}=\\left(\\begin{bmatrix}w_{00}^{(1)} & w_{01}^{(1)}\\\\ w_{10}^{(1)} &w_{11}^{(1)} \\end{bmatrix}\\right)^{T}\\begin{bmatrix}a_0^{(0)} \\\\ a_1^{(0)} \\end{bmatrix}+\\begin{bmatrix}b_0^{(1)} \\\\ b_1^{(1)} \\end{bmatrix},\n",
+ "\\begin{bmatrix}z_1^{(1)} \\\\ z_2^{(1)} \\end{bmatrix}=\\left(\\begin{bmatrix}w_{11}^{(1)} & w_{12}^{(1)}\\\\ w_{21}^{(1)} &w_{22}^{(1)} \\end{bmatrix}\\right)^{T}\\begin{bmatrix}a_1^{(0)} \\\\ a_2^{(0)} \\end{bmatrix}+\\begin{bmatrix}b_1^{(1)} \\\\ b_2^{(1)} \\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "f3c715f1",
+ "id": "3ef7b15b",
"metadata": {
"editable": true
},
@@ -837,19 +837,19 @@
},
{
"cell_type": "markdown",
- "id": "5db82289",
+ "id": "31e47e2c",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\begin{bmatrix}a_0^{(1)} \\\\ a_1^{(1)} \\end{bmatrix}=\\begin{bmatrix}\\sigma^{(1)}(z_0^{(1)}) \\\\ \\sigma^{(1)}(z_1^{(1)}) \\end{bmatrix}.\n",
+ "\\begin{bmatrix}a_1^{(1)} \\\\ a_2^{(1)} \\end{bmatrix}=\\begin{bmatrix}\\sigma^{(1)}(z_1^{(1)}) \\\\ \\sigma^{(1)}(z_2^{(1)}) \\end{bmatrix}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "b6b2ad78",
+ "id": "c5690e76",
"metadata": {
"editable": true
},
@@ -861,19 +861,19 @@
},
{
"cell_type": "markdown",
- "id": "dcefdba1",
+ "id": "919ce153",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "z^{(2)} = w_{0}^{(2)}a_0^{(1)} +w_{1}^{(2)}a_1^{(1)}+b^{(2)},\n",
+ "z^{(2)} = w_{1}^{(2)}a_1^{(1)} +w_{2}^{(2)}a_2^{(1)}+b^{(2)},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "d554c6ce",
+ "id": "69df1f30",
"metadata": {
"editable": true
},
@@ -883,7 +883,7 @@
},
{
"cell_type": "markdown",
- "id": "651ff447",
+ "id": "42ea9246",
"metadata": {
"editable": true
},
@@ -895,7 +895,7 @@
},
{
"cell_type": "markdown",
- "id": "f171ceaa",
+ "id": "af203af1",
"metadata": {
"editable": true
},
@@ -911,7 +911,7 @@
},
{
"cell_type": "markdown",
- "id": "609bb2dd",
+ "id": "fd36e08b",
"metadata": {
"editable": true
},
@@ -923,7 +923,7 @@
},
{
"cell_type": "markdown",
- "id": "da1d696b",
+ "id": "997bddf7",
"metadata": {
"editable": true
},
@@ -933,7 +933,7 @@
},
{
"cell_type": "markdown",
- "id": "d21d62cc",
+ "id": "ba4380bd",
"metadata": {
"editable": true
},
@@ -945,7 +945,7 @@
},
{
"cell_type": "markdown",
- "id": "fb9dd90f",
+ "id": "13be072b",
"metadata": {
"editable": true
},
@@ -955,7 +955,7 @@
},
{
"cell_type": "markdown",
- "id": "9026fb2d",
+ "id": "2c6bcc22",
"metadata": {
"editable": true
},
@@ -967,7 +967,7 @@
},
{
"cell_type": "markdown",
- "id": "2d20d5d1",
+ "id": "a50dfdeb",
"metadata": {
"editable": true
},
@@ -979,20 +979,20 @@
},
{
"cell_type": "markdown",
- "id": "e6fbb9b5",
+ "id": "0d50cd56",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C}{\\partial w_{00}^{(1)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n",
- "\\frac{\\partial z^{(2)}}{\\partial z_0^{(1)}}\\frac{\\partial z_0^{(1)}}{\\partial w_{00}^{(1)}}= \\delta^{(2)}\\frac{\\partial z^{(2)}}{\\partial z_0^{(1)}}\\frac{\\partial z_0^{(1)}}{\\partial w_{00}^{(1)}},\n",
+ "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n",
+ "\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}}= \\delta^{(2)}\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "781a7ac8",
+ "id": "4fc9436b",
"metadata": {
"editable": true
},
@@ -1002,19 +1002,19 @@
},
{
"cell_type": "markdown",
- "id": "3ee09d8f",
+ "id": "7545f5c9",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "z^{(2)} =w_0^{(2)}a_0^{(1)}+w_1^{(2)}a_1^{(1)}+b^{(2)},\n",
+ "z^{(2)} =w_1^{(2)}a_1^{(1)}+w_2^{(2)}a_2^{(1)}+b^{(2)},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "65f44692",
+ "id": "bcbea03f",
"metadata": {
"editable": true
},
@@ -1024,19 +1024,19 @@
},
{
"cell_type": "markdown",
- "id": "0b84b7af",
+ "id": "ee05da6d",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial z^{(2)}}{\\partial z_0^{(1)}}\\frac{\\partial z_0^{(1)}}{\\partial w_{00}^{(1)}}=w_0^{(2)}\\frac{\\partial a_0^{(1)}}{\\partial z_0^{(1)}}a_0^{(1)}.\n",
+ "\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}a_1^{(1)}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "53c138bd",
+ "id": "1f9491ce",
"metadata": {
"editable": true
},
@@ -1047,19 +1047,19 @@
},
{
"cell_type": "markdown",
- "id": "3f81bb3f",
+ "id": "07772fef",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\delta_0^{(1)}=w_0^{(2)}\\frac{\\partial a_0^{(1)}}{\\partial z_0^{(1)}}\\delta^{(2)},\n",
+ "\\delta_1^{(1)}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}\\delta^{(2)},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "eb6783b2",
+ "id": "c432668f",
"metadata": {
"editable": true
},
@@ -1069,19 +1069,19 @@
},
{
"cell_type": "markdown",
- "id": "1c754dbb",
+ "id": "4274417c",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C}{\\partial w_{00}^{(1)}}=\\delta_0^{(1)}a_0^{(1)}.\n",
+ "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\delta_1^{(1)}a_1^{(1)}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "e0b9760c",
+ "id": "b615718d",
"metadata": {
"editable": true
},
@@ -1091,19 +1091,19 @@
},
{
"cell_type": "markdown",
- "id": "b1efb446",
+ "id": "c541b15f",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C}{\\partial w_{01}^{(1)}}=\\delta_0^{(1)}a_1^{(1)}.\n",
+ "\\frac{\\partial C}{\\partial w_{12}^{(1)}}=\\delta_1^{(1)}a_2^{(1)}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "2cf7b43f",
+ "id": "6b741552",
"metadata": {
"editable": true
},
@@ -1115,19 +1115,19 @@
},
{
"cell_type": "markdown",
- "id": "262b517f",
+ "id": "d564e7a9",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C}{\\partial w_{10}^{(1)}}=\\delta_1^{(1)}a_0^{(1)},\n",
+ "\\frac{\\partial C}{\\partial w_{21}^{(1)}}=\\delta_2^{(1)}a_1^{(1)},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "59285e3f",
+ "id": "927894b5",
"metadata": {
"editable": true
},
@@ -1137,19 +1137,19 @@
},
{
"cell_type": "markdown",
- "id": "2813790f",
+ "id": "75624550",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\delta_1^{(1)}a_1^{(1)},\n",
+ "\\frac{\\partial C}{\\partial w_{22}^{(1)}}=\\delta_2^{(1)}a_2^{(1)},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "a56b94b4",
+ "id": "9252c078",
"metadata": {
"editable": true
},
@@ -1159,19 +1159,19 @@
},
{
"cell_type": "markdown",
- "id": "a3ed9010",
+ "id": "10c7da6e",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\delta_1^{(1)}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}\\delta^{(2)}.\n",
+ "\\delta_2^{(1)}=w_2^{(2)}\\frac{\\partial a_2^{(1)}}{\\partial z_2^{(1)}}\\delta^{(2)}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "03a5562c",
+ "id": "0fe640a9",
"metadata": {
"editable": true
},
@@ -1183,19 +1183,19 @@
},
{
"cell_type": "markdown",
- "id": "025bd04f",
+ "id": "01ff9a38",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C}{\\partial b_{0}^{(1)}}=\\delta_0^{(1)},\n",
+ "\\frac{\\partial C}{\\partial b_{1}^{(1)}}=\\delta_1^{(1)},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "e1ef6c2f",
+ "id": "37fda9de",
"metadata": {
"editable": true
},
@@ -1205,19 +1205,19 @@
},
{
"cell_type": "markdown",
- "id": "157d7580",
+ "id": "861af2b9",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C}{\\partial b_{1}^{(1)}}=\\delta_1^{(1)}.\n",
+ "\\frac{\\partial C}{\\partial b_{2}^{(1)}}=\\delta_2^{(1)}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "389a7e2d",
+ "id": "f9cea8b7",
"metadata": {
"editable": true
},
@@ -1227,7 +1227,7 @@
},
{
"cell_type": "markdown",
- "id": "bccf9918",
+ "id": "12e3298b",
"metadata": {
"editable": true
},
@@ -1240,7 +1240,7 @@
},
{
"cell_type": "markdown",
- "id": "ab05945c",
+ "id": "a104df98",
"metadata": {
"editable": true
},
@@ -1252,7 +1252,7 @@
},
{
"cell_type": "markdown",
- "id": "cb2ef195",
+ "id": "9bc2f036",
"metadata": {
"editable": true
},
@@ -1262,7 +1262,7 @@
},
{
"cell_type": "markdown",
- "id": "da672f61",
+ "id": "568ced5c",
"metadata": {
"editable": true
},
@@ -1274,7 +1274,7 @@
},
{
"cell_type": "markdown",
- "id": "95dd3d2b",
+ "id": "906d2bd9",
"metadata": {
"editable": true
},
@@ -1284,7 +1284,7 @@
},
{
"cell_type": "markdown",
- "id": "532b99a2",
+ "id": "79992e6f",
"metadata": {
"editable": true
},
@@ -1296,7 +1296,7 @@
},
{
"cell_type": "markdown",
- "id": "5d5f45cc",
+ "id": "0745b6ba",
"metadata": {
"editable": true
},
@@ -1306,7 +1306,7 @@
},
{
"cell_type": "markdown",
- "id": "2333cfb1",
+ "id": "4fb2781f",
"metadata": {
"editable": true
},
@@ -1318,7 +1318,7 @@
},
{
"cell_type": "markdown",
- "id": "4b555f81",
+ "id": "57576b6e",
"metadata": {
"editable": true
},
@@ -1328,7 +1328,7 @@
},
{
"cell_type": "markdown",
- "id": "20c0952f",
+ "id": "d1f38053",
"metadata": {
"editable": true
},
@@ -1345,7 +1345,7 @@
},
{
"cell_type": "markdown",
- "id": "5a155f81",
+ "id": "6f6f31e8",
"metadata": {
"editable": true
},
@@ -1357,7 +1357,7 @@
},
{
"cell_type": "markdown",
- "id": "ff1fc071",
+ "id": "f206ae2b",
"metadata": {
"editable": true
},
@@ -1369,7 +1369,7 @@
},
{
"cell_type": "markdown",
- "id": "63576478",
+ "id": "5e7af877",
"metadata": {
"editable": true
},
@@ -1385,7 +1385,7 @@
},
{
"cell_type": "markdown",
- "id": "3410ad89",
+ "id": "96c13dab",
"metadata": {
"editable": true
},
@@ -1402,7 +1402,7 @@
},
{
"cell_type": "markdown",
- "id": "c71d887c",
+ "id": "a6781c7d",
"metadata": {
"editable": true
},
@@ -1414,7 +1414,7 @@
},
{
"cell_type": "markdown",
- "id": "ffeae21c",
+ "id": "4db58da4",
"metadata": {
"editable": true
},
@@ -1427,7 +1427,7 @@
},
{
"cell_type": "markdown",
- "id": "2fe917a4",
+ "id": "b4458c55",
"metadata": {
"editable": true
},
@@ -1439,7 +1439,7 @@
},
{
"cell_type": "markdown",
- "id": "65404bec",
+ "id": "b32e0714",
"metadata": {
"editable": true
},
@@ -1455,7 +1455,7 @@
},
{
"cell_type": "markdown",
- "id": "a6c0c8b6",
+ "id": "fffb7785",
"metadata": {
"editable": true
},
@@ -1467,7 +1467,7 @@
},
{
"cell_type": "markdown",
- "id": "2454faec",
+ "id": "08bff16c",
"metadata": {
"editable": true
},
@@ -1483,7 +1483,7 @@
},
{
"cell_type": "markdown",
- "id": "e5b67040",
+ "id": "fb907bb3",
"metadata": {
"editable": true
},
@@ -1495,7 +1495,7 @@
},
{
"cell_type": "markdown",
- "id": "aaec49a4",
+ "id": "f97ed7ef",
"metadata": {
"editable": true
},
@@ -1507,7 +1507,7 @@
},
{
"cell_type": "markdown",
- "id": "e8ad40d3",
+ "id": "136c2230",
"metadata": {
"editable": true
},
@@ -1517,7 +1517,7 @@
},
{
"cell_type": "markdown",
- "id": "e04a4d28",
+ "id": "4b2344b6",
"metadata": {
"editable": true
},
@@ -1529,7 +1529,7 @@
},
{
"cell_type": "markdown",
- "id": "9b121a33",
+ "id": "52a4e7a7",
"metadata": {
"editable": true
},
@@ -1539,7 +1539,7 @@
},
{
"cell_type": "markdown",
- "id": "3bf7d884",
+ "id": "163ee2e5",
"metadata": {
"editable": true
},
@@ -1551,7 +1551,7 @@
},
{
"cell_type": "markdown",
- "id": "7d58918c",
+ "id": "5aa607a5",
"metadata": {
"editable": true
},
@@ -1565,7 +1565,7 @@
},
{
"cell_type": "markdown",
- "id": "d21dce97",
+ "id": "da13c77b",
"metadata": {
"editable": true
},
@@ -1577,7 +1577,7 @@
},
{
"cell_type": "markdown",
- "id": "dc143bea",
+ "id": "7bf944d0",
"metadata": {
"editable": true
},
@@ -1587,7 +1587,7 @@
},
{
"cell_type": "markdown",
- "id": "49415a06",
+ "id": "ea130e95",
"metadata": {
"editable": true
},
@@ -1599,7 +1599,7 @@
},
{
"cell_type": "markdown",
- "id": "4398e11e",
+ "id": "2fba64b7",
"metadata": {
"editable": true
},
@@ -1609,7 +1609,7 @@
},
{
"cell_type": "markdown",
- "id": "da3fca7e",
+ "id": "5904a528",
"metadata": {
"editable": true
},
@@ -1621,7 +1621,7 @@
},
{
"cell_type": "markdown",
- "id": "9055bc57",
+ "id": "86f8199b",
"metadata": {
"editable": true
},
@@ -1633,7 +1633,7 @@
},
{
"cell_type": "markdown",
- "id": "c7995e73",
+ "id": "e4370f9f",
"metadata": {
"editable": true
},
@@ -1645,7 +1645,7 @@
},
{
"cell_type": "markdown",
- "id": "d142f966",
+ "id": "f60e1730",
"metadata": {
"editable": true
},
@@ -1655,7 +1655,7 @@
},
{
"cell_type": "markdown",
- "id": "d5dc5f9f",
+ "id": "e282d002",
"metadata": {
"editable": true
},
@@ -1667,7 +1667,7 @@
},
{
"cell_type": "markdown",
- "id": "5f598e3e",
+ "id": "5de6d59f",
"metadata": {
"editable": true
},
@@ -1677,7 +1677,7 @@
},
{
"cell_type": "markdown",
- "id": "9769b50e",
+ "id": "97c35e7d",
"metadata": {
"editable": true
},
@@ -1689,7 +1689,7 @@
},
{
"cell_type": "markdown",
- "id": "5a8851ae",
+ "id": "c4754c54",
"metadata": {
"editable": true
},
@@ -1707,7 +1707,7 @@
},
{
"cell_type": "markdown",
- "id": "b455efff",
+ "id": "0b03f12b",
"metadata": {
"editable": true
},
@@ -1725,7 +1725,7 @@
},
{
"cell_type": "markdown",
- "id": "8b430778",
+ "id": "ad079735",
"metadata": {
"editable": true
},
@@ -1737,7 +1737,7 @@
},
{
"cell_type": "markdown",
- "id": "7fb2cea5",
+ "id": "0bf757b6",
"metadata": {
"editable": true
},
@@ -1747,7 +1747,7 @@
},
{
"cell_type": "markdown",
- "id": "6c77ac74",
+ "id": "b6246783",
"metadata": {
"editable": true
},
@@ -1759,7 +1759,7 @@
},
{
"cell_type": "markdown",
- "id": "708f42bb",
+ "id": "b7575d50",
"metadata": {
"editable": true
},
@@ -1771,7 +1771,7 @@
},
{
"cell_type": "markdown",
- "id": "40f66bf5",
+ "id": "e6c93f95",
"metadata": {
"editable": true
},
@@ -1783,7 +1783,7 @@
},
{
"cell_type": "markdown",
- "id": "21b9ebbe",
+ "id": "b52b78ac",
"metadata": {
"editable": true
},
@@ -1793,7 +1793,7 @@
},
{
"cell_type": "markdown",
- "id": "665ad548",
+ "id": "a5fdfb9c",
"metadata": {
"editable": true
},
@@ -1805,7 +1805,7 @@
},
{
"cell_type": "markdown",
- "id": "c182abd9",
+ "id": "8bd7f846",
"metadata": {
"editable": true
},
@@ -1815,7 +1815,7 @@
},
{
"cell_type": "markdown",
- "id": "aa70c6cf",
+ "id": "550334c8",
"metadata": {
"editable": true
},
@@ -1827,7 +1827,7 @@
},
{
"cell_type": "markdown",
- "id": "db460a07",
+ "id": "ac298c05",
"metadata": {
"editable": true
},
@@ -1845,7 +1845,7 @@
},
{
"cell_type": "markdown",
- "id": "3af036d6",
+ "id": "6609cbf5",
"metadata": {
"editable": true
},
@@ -1855,7 +1855,7 @@
},
{
"cell_type": "markdown",
- "id": "5662288a",
+ "id": "64741262",
"metadata": {
"editable": true
},
@@ -1873,7 +1873,7 @@
},
{
"cell_type": "markdown",
- "id": "b6a8a1ac",
+ "id": "76c4610d",
"metadata": {
"editable": true
},
@@ -1883,7 +1883,7 @@
},
{
"cell_type": "markdown",
- "id": "dda8fce0",
+ "id": "efd0ba93",
"metadata": {
"editable": true
},
@@ -1901,7 +1901,7 @@
},
{
"cell_type": "markdown",
- "id": "f1957ab0",
+ "id": "98e88435",
"metadata": {
"editable": true
},
@@ -1913,7 +1913,7 @@
},
{
"cell_type": "markdown",
- "id": "31c8f2c9",
+ "id": "08bdbdff",
"metadata": {
"editable": true
},
@@ -1925,7 +1925,7 @@
},
{
"cell_type": "markdown",
- "id": "bba8d29e",
+ "id": "e234c8a6",
"metadata": {
"editable": true
},
@@ -1935,7 +1935,7 @@
},
{
"cell_type": "markdown",
- "id": "af4922ec",
+ "id": "6a4118be",
"metadata": {
"editable": true
},
@@ -1947,7 +1947,7 @@
},
{
"cell_type": "markdown",
- "id": "f43f0f87",
+ "id": "d211df4b",
"metadata": {
"editable": true
},
@@ -1959,7 +1959,7 @@
},
{
"cell_type": "markdown",
- "id": "eba489e4",
+ "id": "bf0c2817",
"metadata": {
"editable": true
},
@@ -1969,7 +1969,7 @@
},
{
"cell_type": "markdown",
- "id": "82610f92",
+ "id": "2de9333a",
"metadata": {
"editable": true
},
@@ -1981,7 +1981,7 @@
},
{
"cell_type": "markdown",
- "id": "62513d62",
+ "id": "00d82ece",
"metadata": {
"editable": true
},
@@ -1991,7 +1991,7 @@
},
{
"cell_type": "markdown",
- "id": "ca381ec8",
+ "id": "19f78643",
"metadata": {
"editable": true
},
@@ -2003,7 +2003,7 @@
},
{
"cell_type": "markdown",
- "id": "bdf32ad4",
+ "id": "1424f687",
"metadata": {
"editable": true
},
@@ -2015,14 +2015,40 @@
},
{
"cell_type": "markdown",
- "id": "764448ea",
+ "id": "9d7c23b2",
"metadata": {
"editable": true
},
"source": [
- "## Setting up the back propagation algorithm\n",
+ "## Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations\n",
"\n",
- "The four equations provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.\n",
+ "**The architecture (our model).**\n",
+ "\n",
+ "1. Set up your inputs and outputs (scalars, vectors, matrices or higher-order arrays)\n",
+ "\n",
+ "2. Define the number of hidden layers and hidden nodes\n",
+ "\n",
+ "3. Define activation functions for hidden layers and output layers\n",
+ "\n",
+ "4. Define optimizer (plan learning rate, momentum, ADAgrad, RMSprop, ADAM etc) and array of initial learning rates\n",
+ "\n",
+ "5. Define cost function and possible regularization terms with hyperparameters\n",
+ "\n",
+ "6. Initialize weights and biases\n",
+ "\n",
+ "7. Fix number of iterations for the feed forward part and back propagation part"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1decfbef",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Setting up the back propagation algorithm, part 1\n",
+ "\n",
+ "The four equations provide us with a way of computing the gradients of the cost function. Let us write this out in the form of an algorithm.\n",
"\n",
"**First**, we set up the input data $\\boldsymbol{x}$ and the activations\n",
"$\\boldsymbol{z}_1$ of the input layer and compute the activation function and\n",
@@ -2038,7 +2064,7 @@
},
{
"cell_type": "markdown",
- "id": "989c3082",
+ "id": "7f237d52",
"metadata": {
"editable": true
},
@@ -2050,7 +2076,7 @@
},
{
"cell_type": "markdown",
- "id": "9d14949a",
+ "id": "d37fa1b5",
"metadata": {
"editable": true
},
@@ -2062,7 +2088,7 @@
},
{
"cell_type": "markdown",
- "id": "4ee118b1",
+ "id": "213b757d",
"metadata": {
"editable": true
},
@@ -2072,7 +2098,7 @@
},
{
"cell_type": "markdown",
- "id": "ba89ad67",
+ "id": "3137751d",
"metadata": {
"editable": true
},
@@ -2084,7 +2110,7 @@
},
{
"cell_type": "markdown",
- "id": "e1f840c5",
+ "id": "da1cf61b",
"metadata": {
"editable": true
},
@@ -2098,7 +2124,7 @@
},
{
"cell_type": "markdown",
- "id": "9f06bcfc",
+ "id": "3b57ef97",
"metadata": {
"editable": true
},
@@ -2110,7 +2136,7 @@
},
{
"cell_type": "markdown",
- "id": "7063f2be",
+ "id": "bb0c4d59",
"metadata": {
"editable": true
},
@@ -2122,7 +2148,7 @@
},
{
"cell_type": "markdown",
- "id": "3e299445",
+ "id": "a9483fc9",
"metadata": {
"editable": true
},
@@ -2132,7 +2158,7 @@
},
{
"cell_type": "markdown",
- "id": "28921878",
+ "id": "49c7c2f3",
"metadata": {
"editable": true
},
@@ -2144,7 +2170,7 @@
},
{
"cell_type": "markdown",
- "id": "90f5404c",
+ "id": "734ef014",
"metadata": {
"editable": true
},
@@ -2156,7 +2182,7 @@
},
{
"cell_type": "markdown",
- "id": "53c2dfbd",
+ "id": "ac393c38",
"metadata": {
"editable": true
},
@@ -2166,7 +2192,7 @@
},
{
"cell_type": "markdown",
- "id": "03805799",
+ "id": "c38ed8eb",
"metadata": {
"editable": true
},
@@ -2178,7 +2204,7 @@
},
{
"cell_type": "markdown",
- "id": "4f8f9c02",
+ "id": "d097bbf6",
"metadata": {
"editable": true
},
@@ -2190,17 +2216,17 @@
},
{
"cell_type": "markdown",
- "id": "9bf86f9c",
+ "id": "56e28349",
"metadata": {
"editable": true
},
"source": [
- "### Activation functions\n",
+ "## Activation functions\n",
"\n",
"A property that characterizes a neural network, other than its\n",
- "connectivity, is the choice of activation function(s). As described\n",
- "in, the following restrictions are imposed on an activation function\n",
- "for a FFNN to fulfill the universal approximation theorem\n",
+ "connectivity, is the choice of activation function(s). The following\n",
+ "restrictions are imposed on an activation function for an FFNN to\n",
+ "fulfill the universal approximation theorem\n",
"\n",
" * Non-constant\n",
"\n",
@@ -2213,7 +2239,7 @@
},
{
"cell_type": "markdown",
- "id": "eeaed73c",
+ "id": "0f764f08",
"metadata": {
"editable": true
},
@@ -2232,7 +2258,7 @@
},
{
"cell_type": "markdown",
- "id": "0786aaf0",
+ "id": "697fbd9c",
"metadata": {
"editable": true
},
@@ -2244,7 +2270,7 @@
},
{
"cell_type": "markdown",
- "id": "aa76601b",
+ "id": "e9f79c9b",
"metadata": {
"editable": true
},
@@ -2254,7 +2280,7 @@
},
{
"cell_type": "markdown",
- "id": "e1bca915",
+ "id": "8285a58e",
"metadata": {
"editable": true
},
@@ -2266,12 +2292,12 @@
},
{
"cell_type": "markdown",
- "id": "1832a8c4",
+ "id": "eba13151",
"metadata": {
"editable": true
},
"source": [
- "### Relevance\n",
+ "## Relevance\n",
"\n",
"The *sigmoid* function are more biologically plausible because the\n",
"output of inactive neurons are zero. Such activation function are\n",
@@ -2283,7 +2309,7 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "5c689736",
+ "id": "d5693cd0",
"metadata": {
"collapsed": false,
"editable": true
@@ -2298,7 +2324,7 @@
},
"metadata": {
"filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_173_0.png"
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_174_0.png"
}
},
"output_type": "display_data"
@@ -2312,7 +2338,7 @@
},
"metadata": {
"filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_173_1.png"
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_174_1.png"
}
},
"output_type": "display_data"
@@ -2326,7 +2352,7 @@
},
"metadata": {
"filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_173_2.png"
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_174_2.png"
}
},
"output_type": "display_data"
@@ -2340,7 +2366,7 @@
},
"metadata": {
"filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_173_3.png"
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_174_3.png"
}
},
"output_type": "display_data"
@@ -2424,7 +2450,228 @@
},
{
"cell_type": "markdown",
- "id": "0b3a4c95",
+ "id": "b0e28a6b",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Vanishing gradients\n",
+ "\n",
+ "The Back propagation algorithm we derived above works by going from\n",
+ "the output layer to the input layer, propagating the error gradient on\n",
+ "the way. Once the algorithm has computed the gradient of the cost\n",
+ "function with regards to each parameter in the network, it uses these\n",
+ "gradients to update each parameter with a Gradient Descent (GD) step.\n",
+ "\n",
+ "Unfortunately for us, the gradients often get smaller and smaller as\n",
+ "the algorithm progresses down to the first hidden layers. As a result,\n",
+ "the GD update leaves the lower layer connection weights virtually\n",
+ "unchanged, and training never converges to a good solution. This is\n",
+ "known in the literature as **the vanishing gradients problem**."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "436fb27b",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Exploding gradients\n",
+ "\n",
+ "In other cases, the opposite can happen, namely the the gradients can\n",
+ "grow bigger and bigger. The result is that many of the layers get\n",
+ "large updates of the weights the algorithm diverges. This is the\n",
+ "**exploding gradients problem**, which is mostly encountered in\n",
+ "recurrent neural networks. More generally, deep neural networks suffer\n",
+ "from unstable gradients, different layers may learn at widely\n",
+ "different speeds"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9319de67",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Is the Logistic activation function (Sigmoid) our choice?\n",
+ "\n",
+ "Although this unfortunate behavior has been empirically observed for\n",
+ "quite a while (it was one of the reasons why deep neural networks were\n",
+ "mostly abandoned for a long time), it is only around 2010 that\n",
+ "significant progress was made in understanding it.\n",
+ "\n",
+ "A paper titled [Understanding the Difficulty of Training Deep\n",
+ "Feedforward Neural Networks by Xavier Glorot and Yoshua Bengio](http://proceedings.mlr.press/v9/glorot10a.html) found that\n",
+ "the problems with the popular logistic\n",
+ "sigmoid activation function and the weight initialization technique\n",
+ "that was most popular at the time, namely random initialization using\n",
+ "a normal distribution with a mean of 0 and a standard deviation of\n",
+ "1."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e90fbb0a",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Logistic function as the root of problems\n",
+ "\n",
+ "They showed that with this activation function and this\n",
+ "initialization scheme, the variance of the outputs of each layer is\n",
+ "much greater than the variance of its inputs. Going forward in the\n",
+ "network, the variance keeps increasing after each layer until the\n",
+ "activation function saturates at the top layers. This is actually made\n",
+ "worse by the fact that the logistic function has a mean of 0.5, not 0\n",
+ "(the hyperbolic tangent function has a mean of 0 and behaves slightly\n",
+ "better than the logistic function in deep networks)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7a18427f",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## The derivative of the Logistic funtion\n",
+ "\n",
+ "Looking at the logistic activation function, when inputs become large\n",
+ "(negative or positive), the function saturates at 0 or 1, with a\n",
+ "derivative extremely close to 0. Thus when backpropagation kicks in,\n",
+ "it has virtually no gradient to propagate back through the network,\n",
+ "and what little gradient exists keeps getting diluted as\n",
+ "backpropagation progresses down through the top layers, so there is\n",
+ "really nothing left for the lower layers.\n",
+ "\n",
+ "In their paper, Glorot and Bengio propose a way to significantly\n",
+ "alleviate this problem. We need the signal to flow properly in both\n",
+ "directions: in the forward direction when making predictions, and in\n",
+ "the reverse direction when backpropagating gradients. We don’t want\n",
+ "the signal to die out, nor do we want it to explode and saturate. For\n",
+ "the signal to flow properly, the authors argue that we need the\n",
+ "variance of the outputs of each layer to be equal to the variance of\n",
+ "its inputs, and we also need the gradients to have equal variance\n",
+ "before and after flowing through a layer in the reverse direction."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ac352aa1",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Insights from the paper by Glorot and Bengio\n",
+ "\n",
+ "One of the insights in the 2010 paper by Glorot and Bengio was that\n",
+ "the vanishing/exploding gradients problems were in part due to a poor\n",
+ "choice of activation function. Until then most people had assumed that\n",
+ "if Nature had chosen to use roughly sigmoid activation functions in\n",
+ "biological neurons, they must be an excellent choice. But it turns out\n",
+ "that other activation functions behave much better in deep neural\n",
+ "networks, in particular the ReLU activation function, mostly because\n",
+ "it does not saturate for positive values (and also because it is quite\n",
+ "fast to compute)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "67a8bef0",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## The RELU function family\n",
+ "\n",
+ "The ReLU activation function suffers from a problem known as the dying\n",
+ "ReLUs: during training, some neurons effectively die, meaning they\n",
+ "stop outputting anything other than 0.\n",
+ "\n",
+ "In some cases, you may find that half of your network’s neurons are\n",
+ "dead, especially if you used a large learning rate. During training,\n",
+ "if a neuron’s weights get updated such that the weighted sum of the\n",
+ "neuron’s inputs is negative, it will start outputting 0. When this\n",
+ "happen, the neuron is unlikely to come back to life since the gradient\n",
+ "of the ReLU function is 0 when its input is negative."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "76de2016",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## ELU function\n",
+ "\n",
+ "To solve this problem, nowadays practitioners use a variant of the\n",
+ "ReLU function, such as the leaky ReLU discussed above or the so-called\n",
+ "exponential linear unit (ELU) function"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e798fa5d",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "$$\n",
+ "ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z < 0,\\\\ z & z \\ge 0.\\end{array}\\right.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f33abb9",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Which activation function should we use?\n",
+ "\n",
+ "In general it seems that the ELU activation function is better than\n",
+ "the leaky ReLU function (and its variants), which is better than\n",
+ "ReLU. ReLU performs better than $\\tanh$ which in turn performs better\n",
+ "than the logistic function.\n",
+ "\n",
+ "If runtime performance is an issue, then you may opt for the leaky\n",
+ "ReLU function over the ELU function If you don’t want to tweak yet\n",
+ "another hyperparameter, you may just use the default $\\alpha$ of\n",
+ "$0.01$ for the leaky ReLU, and $1$ for ELU. If you have spare time and\n",
+ "computing power, you can use cross-validation or bootstrap to evaluate\n",
+ "other activation functions."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ad76ab9d",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## More on activation functions, output layers\n",
+ "\n",
+ "In most cases you can use the ReLU activation function in the hidden\n",
+ "layers (or one of its variants).\n",
+ "\n",
+ "It is a bit faster to compute than other activation functions, and the\n",
+ "gradient descent optimization does in general not get stuck.\n",
+ "\n",
+ "**For the output layer:**\n",
+ "\n",
+ "* For classification the softmax activation function is generally a good choice for classification tasks (when the classes are mutually exclusive).\n",
+ "\n",
+ "* For regression tasks, you can simply use no activation function at all."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4d0588bb",
"metadata": {
"editable": true
},
@@ -2453,7 +2700,7 @@
},
{
"cell_type": "markdown",
- "id": "9955a636",
+ "id": "cb5679f8",
"metadata": {
"editable": true
},
@@ -2461,7 +2708,7 @@
"## Hidden layers\n",
"\n",
"For many problems you can start with just one or two hidden layers and\n",
- "it will work just fine. For the MNIST data set you ca easily get a\n",
+ "it will work just fine. For the MNIST data set discussed below you can easily get a\n",
"high accuracy using just one hidden layer with a few hundred neurons.\n",
"You can reach for this data set above 98% accuracy using two hidden\n",
"layers with the same total amount of neurons, in roughly the same\n",
@@ -2479,228 +2726,7 @@
},
{
"cell_type": "markdown",
- "id": "f1aa77e9",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Vanishing gradients\n",
- "\n",
- "The Back propagation algorithm we derived above works by going from\n",
- "the output layer to the input layer, propagating the error gradient on\n",
- "the way. Once the algorithm has computed the gradient of the cost\n",
- "function with regards to each parameter in the network, it uses these\n",
- "gradients to update each parameter with a Gradient Descent (GD) step.\n",
- "\n",
- "Unfortunately for us, the gradients often get smaller and smaller as\n",
- "the algorithm progresses down to the first hidden layers. As a result,\n",
- "the GD update leaves the lower layer connection weights virtually\n",
- "unchanged, and training never converges to a good solution. This is\n",
- "known in the literature as **the vanishing gradients problem**."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "3703bb33",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Exploding gradients\n",
- "\n",
- "In other cases, the opposite can happen, namely the the gradients can\n",
- "grow bigger and bigger. The result is that many of the layers get\n",
- "large updates of the weights the algorithm diverges. This is the\n",
- "**exploding gradients problem**, which is mostly encountered in\n",
- "recurrent neural networks. More generally, deep neural networks suffer\n",
- "from unstable gradients, different layers may learn at widely\n",
- "different speeds"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "43b1aa26",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Is the Logistic activation function (Sigmoid) our choice?\n",
- "\n",
- "Although this unfortunate behavior has been empirically observed for\n",
- "quite a while (it was one of the reasons why deep neural networks were\n",
- "mostly abandoned for a long time), it is only around 2010 that\n",
- "significant progress was made in understanding it.\n",
- "\n",
- "A paper titled [Understanding the Difficulty of Training Deep\n",
- "Feedforward Neural Networks by Xavier Glorot and Yoshua Bengio](http://proceedings.mlr.press/v9/glorot10a.html) found that\n",
- "the problems with the popular logistic\n",
- "sigmoid activation function and the weight initialization technique\n",
- "that was most popular at the time, namely random initialization using\n",
- "a normal distribution with a mean of 0 and a standard deviation of\n",
- "1."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "463f4f64",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Logistic function as the root of problems\n",
- "\n",
- "They showed that with this activation function and this\n",
- "initialization scheme, the variance of the outputs of each layer is\n",
- "much greater than the variance of its inputs. Going forward in the\n",
- "network, the variance keeps increasing after each layer until the\n",
- "activation function saturates at the top layers. This is actually made\n",
- "worse by the fact that the logistic function has a mean of 0.5, not 0\n",
- "(the hyperbolic tangent function has a mean of 0 and behaves slightly\n",
- "better than the logistic function in deep networks)."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6c9ea582",
- "metadata": {
- "editable": true
- },
- "source": [
- "## The derivative of the Logistic funtion\n",
- "\n",
- "Looking at the logistic activation function, when inputs become large\n",
- "(negative or positive), the function saturates at 0 or 1, with a\n",
- "derivative extremely close to 0. Thus when backpropagation kicks in,\n",
- "it has virtually no gradient to propagate back through the network,\n",
- "and what little gradient exists keeps getting diluted as\n",
- "backpropagation progresses down through the top layers, so there is\n",
- "really nothing left for the lower layers.\n",
- "\n",
- "In their paper, Glorot and Bengio propose a way to significantly\n",
- "alleviate this problem. We need the signal to flow properly in both\n",
- "directions: in the forward direction when making predictions, and in\n",
- "the reverse direction when backpropagating gradients. We don’t want\n",
- "the signal to die out, nor do we want it to explode and saturate. For\n",
- "the signal to flow properly, the authors argue that we need the\n",
- "variance of the outputs of each layer to be equal to the variance of\n",
- "its inputs, and we also need the gradients to have equal variance\n",
- "before and after flowing through a layer in the reverse direction."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "80c83d2c",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Insights from the paper by Glorot and Bengio\n",
- "\n",
- "One of the insights in the 2010 paper by Glorot and Bengio was that\n",
- "the vanishing/exploding gradients problems were in part due to a poor\n",
- "choice of activation function. Until then most people had assumed that\n",
- "if Nature had chosen to use roughly sigmoid activation functions in\n",
- "biological neurons, they must be an excellent choice. But it turns out\n",
- "that other activation functions behave much better in deep neural\n",
- "networks, in particular the ReLU activation function, mostly because\n",
- "it does not saturate for positive values (and also because it is quite\n",
- "fast to compute)."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "55a1ad5d",
- "metadata": {
- "editable": true
- },
- "source": [
- "## The RELU function family\n",
- "\n",
- "The ReLU activation function suffers from a problem known as the dying\n",
- "ReLUs: during training, some neurons effectively die, meaning they\n",
- "stop outputting anything other than 0.\n",
- "\n",
- "In some cases, you may find that half of your network’s neurons are\n",
- "dead, especially if you used a large learning rate. During training,\n",
- "if a neuron’s weights get updated such that the weighted sum of the\n",
- "neuron’s inputs is negative, it will start outputting 0. When this\n",
- "happen, the neuron is unlikely to come back to life since the gradient\n",
- "of the ReLU function is 0 when its input is negative."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b7fafe4a",
- "metadata": {
- "editable": true
- },
- "source": [
- "## ELU function\n",
- "\n",
- "To solve this problem, nowadays practitioners use a variant of the\n",
- "ReLU function, such as the leaky ReLU discussed above or the so-called\n",
- "exponential linear unit (ELU) function"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "46e10153",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z < 0,\\\\ z & z \\ge 0.\\end{array}\\right.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0930ba50",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Which activation function should we use?\n",
- "\n",
- "In general it seems that the ELU activation function is better than\n",
- "the leaky ReLU function (and its variants), which is better than\n",
- "ReLU. ReLU performs better than $\\tanh$ which in turn performs better\n",
- "than the logistic function.\n",
- "\n",
- "If runtime performance is an issue, then you may opt for the leaky\n",
- "ReLU function over the ELU function If you don’t want to tweak yet\n",
- "another hyperparameter, you may just use the default $\\alpha$ of\n",
- "$0.01$ for the leaky ReLU, and $1$ for ELU. If you have spare time and\n",
- "computing power, you can use cross-validation or bootstrap to evaluate\n",
- "other activation functions."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "5c0c59af",
- "metadata": {
- "editable": true
- },
- "source": [
- "## More on activation functions, output layers\n",
- "\n",
- "In most cases you can use the ReLU activation function in the hidden\n",
- "layers (or one of its variants).\n",
- "\n",
- "It is a bit faster to compute than other activation functions, and the\n",
- "gradient descent optimization does in general not get stuck.\n",
- "\n",
- "**For the output layer:**\n",
- "\n",
- "* For classification the softmax activation function is generally a good choice for classification tasks (when the classes are mutually exclusive).\n",
- "\n",
- "* For regression tasks, you can simply use no activation function at all."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "582e4df4",
+ "id": "fa928c9b",
"metadata": {
"editable": true
},
@@ -2726,7 +2752,7 @@
},
{
"cell_type": "markdown",
- "id": "ea360013",
+ "id": "4a9462ce",
"metadata": {
"editable": true
},
@@ -2746,7 +2772,7 @@
},
{
"cell_type": "markdown",
- "id": "fee98b12",
+ "id": "e523997a",
"metadata": {
"editable": true
},
@@ -2766,7 +2792,7 @@
},
{
"cell_type": "markdown",
- "id": "c9f175b0",
+ "id": "19bba5e1",
"metadata": {
"editable": true
},
@@ -2792,7 +2818,7 @@
},
{
"cell_type": "markdown",
- "id": "f6f04fbb",
+ "id": "ff6b5f13",
"metadata": {
"editable": true
},
@@ -2821,7 +2847,7 @@
},
{
"cell_type": "markdown",
- "id": "d4184eee",
+ "id": "df905d9f",
"metadata": {
"editable": true
},
@@ -2839,7 +2865,7 @@
},
{
"cell_type": "markdown",
- "id": "82d314da",
+ "id": "233e93d6",
"metadata": {
"editable": true
},
@@ -2855,7 +2881,7 @@
},
{
"cell_type": "markdown",
- "id": "abc6640b",
+ "id": "0034168c",
"metadata": {
"editable": true
},
@@ -2867,7 +2893,7 @@
},
{
"cell_type": "markdown",
- "id": "bb7d1f89",
+ "id": "cd701f7d",
"metadata": {
"editable": true
},
@@ -2881,109 +2907,7 @@
},
{
"cell_type": "markdown",
- "id": "442b1b5e",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Setting up the back-propagation algorithm\n",
- "\n",
- "Let us write this out in the form of an algorithm.\n",
- "\n",
- "First, we set up the input data $\\boldsymbol{x}$ and the activations\n",
- "$\\boldsymbol{z}_1$ of the input layer and compute the activation function and\n",
- "the pertinent outputs $\\boldsymbol{a}^1$.\n",
- "\n",
- "Secondly, we perform then the feed forward till we reach the output\n",
- "layer and compute all $\\boldsymbol{z}_l$ of the input layer and compute the\n",
- "activation function and the pertinent outputs $\\boldsymbol{a}^l$ for\n",
- "$l=2,3,\\dots,L$.\n",
- "\n",
- "Thereafter we compute the ouput error $\\boldsymbol{\\delta}^L$ by computing all"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0005e340",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\delta_j^L = f'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2b27751b",
- "metadata": {
- "editable": true
- },
- "source": [
- "Then we compute the back propagate error for each $l=L-1,L-2,\\dots,2$ as"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "dd544b4b",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l).\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "57a7d213",
- "metadata": {
- "editable": true
- },
- "source": [
- "Finally, we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,2$ and update the weights and biases according to the rules"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "932510f4",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "w_{ij}^l\\leftarrow = w_{ij}^l- \\eta \\delta_j^la_i^{l-1},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "bf80ed99",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f682bf76",
- "metadata": {
- "editable": true
- },
- "source": [
- "The parameter $\\eta$ is the learning parameter discussed in connection with the gradient descent methods.\n",
- "Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ad2b26a4",
+ "id": "84048fb0",
"metadata": {
"editable": true
},
@@ -3011,7 +2935,7 @@
},
{
"cell_type": "markdown",
- "id": "a08331b2",
+ "id": "61ea03ed",
"metadata": {
"editable": true
},
@@ -3023,7 +2947,7 @@
},
{
"cell_type": "markdown",
- "id": "9ebfde52",
+ "id": "abd0c817",
"metadata": {
"editable": true
},
@@ -3033,7 +2957,7 @@
},
{
"cell_type": "markdown",
- "id": "531dfa25",
+ "id": "2e0379cc",
"metadata": {
"editable": true
},
@@ -3045,7 +2969,7 @@
},
{
"cell_type": "markdown",
- "id": "8941d38c",
+ "id": "59290cf7",
"metadata": {
"editable": true
},
@@ -3056,7 +2980,7 @@
},
{
"cell_type": "markdown",
- "id": "7ae7e222",
+ "id": "6bc256eb",
"metadata": {
"editable": true
},
@@ -3068,7 +2992,7 @@
},
{
"cell_type": "markdown",
- "id": "cbb157cc",
+ "id": "831eee08",
"metadata": {
"editable": true
},
@@ -3081,7 +3005,7 @@
},
{
"cell_type": "markdown",
- "id": "0178770c",
+ "id": "ba46bcc0",
"metadata": {
"editable": true
},
@@ -3106,7 +3030,7 @@
},
{
"cell_type": "markdown",
- "id": "cb69f3df",
+ "id": "a475ed33",
"metadata": {
"editable": true
},
@@ -3119,7 +3043,7 @@
},
{
"cell_type": "markdown",
- "id": "11aa1335",
+ "id": "378895ce",
"metadata": {
"editable": true
},
@@ -3131,7 +3055,7 @@
},
{
"cell_type": "markdown",
- "id": "a225e57a",
+ "id": "268989a6",
"metadata": {
"editable": true
},
@@ -3143,7 +3067,7 @@
},
{
"cell_type": "markdown",
- "id": "569dd3f2",
+ "id": "1ac5dfe0",
"metadata": {
"editable": true
},
@@ -3153,7 +3077,7 @@
},
{
"cell_type": "markdown",
- "id": "f8e84cae",
+ "id": "4d3d6bb7",
"metadata": {
"editable": true
},
@@ -3165,7 +3089,7 @@
},
{
"cell_type": "markdown",
- "id": "1dd303e8",
+ "id": "bc4e4a53",
"metadata": {
"editable": true
},
@@ -3177,7 +3101,7 @@
},
{
"cell_type": "markdown",
- "id": "33494270",
+ "id": "4cc000a1",
"metadata": {
"editable": true
},
@@ -3189,7 +3113,7 @@
},
{
"cell_type": "markdown",
- "id": "01bd6a05",
+ "id": "7f5ea691",
"metadata": {
"editable": true
},
@@ -3201,7 +3125,7 @@
},
{
"cell_type": "markdown",
- "id": "60f16a47",
+ "id": "a7f4fc8e",
"metadata": {
"editable": true
},
@@ -3211,7 +3135,7 @@
},
{
"cell_type": "markdown",
- "id": "ab0c42b3",
+ "id": "acdecb39",
"metadata": {
"editable": true
},
@@ -3223,7 +3147,7 @@
},
{
"cell_type": "markdown",
- "id": "dcfb2e28",
+ "id": "b0c3e8c7",
"metadata": {
"editable": true
},
@@ -3233,7 +3157,7 @@
},
{
"cell_type": "markdown",
- "id": "5076b392",
+ "id": "6e130deb",
"metadata": {
"editable": true
},
@@ -3245,7 +3169,7 @@
},
{
"cell_type": "markdown",
- "id": "9d6001f4",
+ "id": "23e83ce8",
"metadata": {
"editable": true
},
@@ -3258,7 +3182,7 @@
},
{
"cell_type": "markdown",
- "id": "d0245c18",
+ "id": "67c77893",
"metadata": {
"editable": true
},
@@ -3270,7 +3194,7 @@
},
{
"cell_type": "markdown",
- "id": "030ac014",
+ "id": "36ede636",
"metadata": {
"editable": true
},
@@ -3280,7 +3204,7 @@
},
{
"cell_type": "markdown",
- "id": "bdc01c83",
+ "id": "4008b26c",
"metadata": {
"editable": true
},
@@ -3292,7 +3216,7 @@
},
{
"cell_type": "markdown",
- "id": "1a77b25c",
+ "id": "258ae1d4",
"metadata": {
"editable": true
},
@@ -3303,7 +3227,7 @@
},
{
"cell_type": "markdown",
- "id": "1f169ed1",
+ "id": "b9c648be",
"metadata": {
"editable": true
},
@@ -3315,7 +3239,7 @@
},
{
"cell_type": "markdown",
- "id": "6bc460e2",
+ "id": "615f1fce",
"metadata": {
"editable": true
},
@@ -3325,7 +3249,7 @@
},
{
"cell_type": "markdown",
- "id": "fe1f92c0",
+ "id": "620b34cb",
"metadata": {
"editable": true
},
@@ -3337,7 +3261,7 @@
},
{
"cell_type": "markdown",
- "id": "78cf6843",
+ "id": "7d09dc1a",
"metadata": {
"editable": true
},
@@ -3347,7 +3271,7 @@
},
{
"cell_type": "markdown",
- "id": "81ed1ccc",
+ "id": "8589b8ee",
"metadata": {
"editable": true
},
@@ -3358,7 +3282,7 @@
},
{
"cell_type": "markdown",
- "id": "2a16cf53",
+ "id": "63160327",
"metadata": {
"editable": true
},
@@ -3371,7 +3295,7 @@
},
{
"cell_type": "markdown",
- "id": "21988141",
+ "id": "98614055",
"metadata": {
"editable": true
},
@@ -3381,7 +3305,7 @@
},
{
"cell_type": "markdown",
- "id": "b37cc0cd",
+ "id": "37435d7c",
"metadata": {
"editable": true
},
@@ -3393,7 +3317,7 @@
},
{
"cell_type": "markdown",
- "id": "259618c5",
+ "id": "64a4d79f",
"metadata": {
"editable": true
},
@@ -3403,7 +3327,7 @@
},
{
"cell_type": "markdown",
- "id": "af5a331e",
+ "id": "7e3891af",
"metadata": {
"editable": true
},
@@ -3415,7 +3339,7 @@
},
{
"cell_type": "markdown",
- "id": "208ab286",
+ "id": "7205e781",
"metadata": {
"editable": true
},
@@ -3425,7 +3349,7 @@
},
{
"cell_type": "markdown",
- "id": "8f30fbc3",
+ "id": "a5645092",
"metadata": {
"editable": true
},
@@ -3449,7 +3373,7 @@
},
{
"cell_type": "markdown",
- "id": "09538ef9",
+ "id": "09e9b12d",
"metadata": {
"editable": true
},
@@ -3499,7 +3423,7 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "ced37e42",
+ "id": "84fd1480",
"metadata": {
"collapsed": false,
"editable": true
@@ -3523,7 +3447,7 @@
},
"metadata": {
"filenames": {
- "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_242_1.png"
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_235_1.png"
}
},
"output_type": "display_data"
@@ -3576,7 +3500,7 @@
},
{
"cell_type": "markdown",
- "id": "ebabb274",
+ "id": "e671e2a8",
"metadata": {
"editable": true
},
@@ -3597,7 +3521,7 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "2c87b9f9",
+ "id": "5d1c4624",
"metadata": {
"collapsed": false,
"editable": true
@@ -3644,7 +3568,7 @@
},
{
"cell_type": "markdown",
- "id": "0c186fec",
+ "id": "b397d4ee",
"metadata": {
"editable": true
},
@@ -3688,7 +3612,7 @@
},
{
"cell_type": "markdown",
- "id": "2cbece98",
+ "id": "1fce534f",
"metadata": {
"editable": true
},
@@ -3728,7 +3652,7 @@
},
{
"cell_type": "markdown",
- "id": "442935af",
+ "id": "9c50a158",
"metadata": {
"editable": true
},
@@ -3749,7 +3673,7 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "07fbb18c",
+ "id": "4daeba9a",
"metadata": {
"collapsed": false,
"editable": true
@@ -3775,7 +3699,7 @@
},
{
"cell_type": "markdown",
- "id": "6a3ea326",
+ "id": "f26a835c",
"metadata": {
"editable": true
},
@@ -3803,7 +3727,7 @@
},
{
"cell_type": "markdown",
- "id": "525624c4",
+ "id": "09f1187b",
"metadata": {
"editable": true
},
@@ -3840,7 +3764,7 @@
{
"cell_type": "code",
"execution_count": 6,
- "id": "32d0a43f",
+ "id": "35d71f0e",
"metadata": {
"collapsed": false,
"editable": true
@@ -3903,7 +3827,7 @@
},
{
"cell_type": "markdown",
- "id": "da2ede88",
+ "id": "e318575f",
"metadata": {
"editable": true
},
@@ -3934,7 +3858,7 @@
},
{
"cell_type": "markdown",
- "id": "069a250d",
+ "id": "788f45d2",
"metadata": {
"editable": true
},
@@ -3972,7 +3896,7 @@
},
{
"cell_type": "markdown",
- "id": "29ab6ee7",
+ "id": "599a7b8f",
"metadata": {
"editable": true
},
@@ -4006,7 +3930,7 @@
},
{
"cell_type": "markdown",
- "id": "b7c4a178",
+ "id": "cc694849",
"metadata": {
"editable": true
},
@@ -4047,7 +3971,7 @@
{
"cell_type": "code",
"execution_count": 7,
- "id": "4142969e",
+ "id": "e3607c66",
"metadata": {
"collapsed": false,
"editable": true
@@ -4064,20 +3988,15 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_47048/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
" return 1/(1 + np.exp(-x))\n"
]
},
{
- "ename": "KeyboardInterrupt",
- "evalue": "",
- "output_type": "error",
- "traceback": [
- "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
- "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
- "Cell \u001b[0;32mIn[7], line 56\u001b[0m\n\u001b[1;32m 53\u001b[0m lmbd \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m0.01\u001b[39m\n\u001b[1;32m 54\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m i \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mrange\u001b[39m(\u001b[38;5;241m1000\u001b[39m):\n\u001b[1;32m 55\u001b[0m \u001b[38;5;66;03m# calculate gradients\u001b[39;00m\n\u001b[0;32m---> 56\u001b[0m dWo, dBo, dWh, dBh \u001b[38;5;241m=\u001b[39m \u001b[43mbackpropagation\u001b[49m\u001b[43m(\u001b[49m\u001b[43mX_train\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mY_train_onehot\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 58\u001b[0m \u001b[38;5;66;03m# regularization term gradients\u001b[39;00m\n\u001b[1;32m 59\u001b[0m dWo \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m lmbd \u001b[38;5;241m*\u001b[39m output_weights\n",
- "Cell \u001b[0;32mIn[7], line 38\u001b[0m, in \u001b[0;36mbackpropagation\u001b[0;34m(X, Y)\u001b[0m\n\u001b[1;32m 36\u001b[0m error_output \u001b[38;5;241m=\u001b[39m probabilities \u001b[38;5;241m-\u001b[39m Y\n\u001b[1;32m 37\u001b[0m \u001b[38;5;66;03m# error in the hidden layer\u001b[39;00m\n\u001b[0;32m---> 38\u001b[0m error_hidden \u001b[38;5;241m=\u001b[39m \u001b[43mnp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmatmul\u001b[49m\u001b[43m(\u001b[49m\u001b[43merror_output\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moutput_weights\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mT\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;241m*\u001b[39m a_h \u001b[38;5;241m*\u001b[39m (\u001b[38;5;241m1\u001b[39m \u001b[38;5;241m-\u001b[39m a_h)\n\u001b[1;32m 40\u001b[0m \u001b[38;5;66;03m# gradients for the output layer\u001b[39;00m\n\u001b[1;32m 41\u001b[0m output_weights_gradient \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39mmatmul(a_h\u001b[38;5;241m.\u001b[39mT, error_output)\n",
- "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "New accuracy on training data: 0.09951287404314545\n"
]
}
],
@@ -4154,7 +4073,7 @@
},
{
"cell_type": "markdown",
- "id": "4d79c3b3",
+ "id": "0a70cdcf",
"metadata": {
"editable": true
},
@@ -4175,7 +4094,7 @@
},
{
"cell_type": "markdown",
- "id": "58012d93",
+ "id": "b0600212",
"metadata": {
"editable": true
},
@@ -4189,7 +4108,7 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "ea4b7741",
+ "id": "4e0c1326",
"metadata": {
"collapsed": false,
"editable": true
@@ -4299,7 +4218,7 @@
},
{
"cell_type": "markdown",
- "id": "04860b2c",
+ "id": "125f8eca",
"metadata": {
"editable": true
},
@@ -4318,12 +4237,20 @@
{
"cell_type": "code",
"execution_count": 9,
- "id": "b52d4df3",
+ "id": "cef0e788",
"metadata": {
"collapsed": false,
"editable": true
},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Accuracy score on test set: 0.9444444444444444\n"
+ ]
+ }
+ ],
"source": [
"epochs = 100\n",
"batch_size = 100\n",
@@ -4345,7 +4272,7 @@
},
{
"cell_type": "markdown",
- "id": "6dce68ff",
+ "id": "b12d6f86",
"metadata": {
"editable": true
},
@@ -4359,12 +4286,723 @@
{
"cell_type": "code",
"execution_count": 10,
- "id": "65caeb78",
+ "id": "974faa4e",
"metadata": {
"collapsed": false,
"editable": true
},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.11666666666666667\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.20833333333333334\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.12222222222222222\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.14722222222222223\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.17777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.16111111111111112\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.20277777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.5305555555555556\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.5944444444444444\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.5888888888888889\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.6111111111111112\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.5222222222222223\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.5555555555555556\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.8055555555555556\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.85\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.85\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.875\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.8666666666666667\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.8638888888888889\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.9555555555555556\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.925\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.01\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.9472222222222222\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.01\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.9277777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.01\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.9472222222222222\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.01\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.9305555555555556\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.01\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.9555555555555556\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.01\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.7694444444444445\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.01\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.19166666666666668\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.1\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.10555555555555556\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.1\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.08611111111111111\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.1\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.10555555555555556\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.1\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.08888888888888889\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.1\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.08611111111111111\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.1\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.08888888888888889\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.1\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.09166666666666666\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:43: RuntimeWarning: overflow encountered in exp\n",
+ " exp_term = np.exp(self.z_o)\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide\n",
+ " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1.0\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.07777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:43: RuntimeWarning: overflow encountered in exp\n",
+ " exp_term = np.exp(self.z_o)\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide\n",
+ " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1.0\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.07777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:43: RuntimeWarning: overflow encountered in exp\n",
+ " exp_term = np.exp(self.z_o)\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide\n",
+ " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1.0\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.07777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:43: RuntimeWarning: overflow encountered in exp\n",
+ " exp_term = np.exp(self.z_o)\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide\n",
+ " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1.0\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.07777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:43: RuntimeWarning: overflow encountered in exp\n",
+ " exp_term = np.exp(self.z_o)\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide\n",
+ " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1.0\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.07777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1.0\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.10555555555555556\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:43: RuntimeWarning: overflow encountered in exp\n",
+ " exp_term = np.exp(self.z_o)\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide\n",
+ " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1.0\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.07777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:43: RuntimeWarning: overflow encountered in exp\n",
+ " exp_term = np.exp(self.z_o)\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide\n",
+ " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 10.0\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.07777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:43: RuntimeWarning: overflow encountered in exp\n",
+ " exp_term = np.exp(self.z_o)\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide\n",
+ " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 10.0\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.07777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:43: RuntimeWarning: overflow encountered in exp\n",
+ " exp_term = np.exp(self.z_o)\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide\n",
+ " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 10.0\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.07777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:43: RuntimeWarning: overflow encountered in exp\n",
+ " exp_term = np.exp(self.z_o)\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide\n",
+ " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 10.0\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.07777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:43: RuntimeWarning: overflow encountered in exp\n",
+ " exp_term = np.exp(self.z_o)\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide\n",
+ " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 10.0\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.07777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:43: RuntimeWarning: overflow encountered in exp\n",
+ " exp_term = np.exp(self.z_o)\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide\n",
+ " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 10.0\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.07777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:43: RuntimeWarning: overflow encountered in exp\n",
+ " exp_term = np.exp(self.z_o)\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide\n",
+ " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 10.0\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.07777777777777778\n",
+ "\n"
+ ]
+ }
+ ],
"source": [
"eta_vals = np.logspace(-5, 1, 7)\n",
"lmbd_vals = np.logspace(-5, 1, 7)\n",
@@ -4390,7 +5028,7 @@
},
{
"cell_type": "markdown",
- "id": "0c44c38e",
+ "id": "3bb19122",
"metadata": {
"editable": true
},
@@ -4401,12 +5039,57 @@
{
"cell_type": "code",
"execution_count": 11,
- "id": "fbdc64f1",
+ "id": "baeb1ea3",
"metadata": {
"collapsed": false,
"editable": true
},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n",
+ "/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_55841/953065564.py:4: RuntimeWarning: overflow encountered in exp\n",
+ " return 1/(1 + np.exp(-x))\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyAAAANbCAYAAAC6lftqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABdFUlEQVR4nO3deZxVdf0/8NcIDKiAiakoigsuIy5IYm4oGfLL0tKMSgrFUNQ0cMk901TMSpTUJDcoywxyLRVT1L4WLYhLpiJoLqjJ4gqyyCAzvz/IqQnQC17OhZnn8/G4j8fM555zP+97PF7mfd/v8zlV9fX19QEAACjAGpUOAAAAaD4kIAAAQGEkIAAAQGEkIAAAQGEkIAAAQGEkIAAAQGEkIAAAQGEkIAAAQGEkIAAAQGEkIMBq6Ywzzsi22277gY9Pf/rTH2mOW2+9Ndtuu21eeeWVlbrPR/Xyyy+npqYmn/zkJ/Puu+8WNi8ArIiq+vr6+koHAbC8Xnrppbz55psNv48YMSKTJk3KT37yk4ax6urqdO3adYXnePPNN/PSSy+la9euqa6uXmn7fFTDhw/PPffck1deeSUXXHBBvvjFLxYyLwCsCAkI0CScccYZeeihh/LAAw9UOpRC1dXV5dOf/nQOPvjgTJo0KbNmzcqYMWMqHRYALJMWLKBJmzBhQrbddtuMHj06++67b/bcc8+MHz8+SXLTTTflkEMOyc4775yddtopBx10UMaOHduw7/+2U51xxhk54ogjcsstt+Qzn/lMdthhh3zhC1/Igw8++JH2SZLHHnssX//617PzzjvnU5/6VK6//vocccQROeOMMz7w/Y0fPz7Tpk3Lvvvumy984Qv5+9//nsmTJy+x3RtvvJGzzjore+65Z7p3756vf/3reeSRRxqeX7hwYa688srst99+2WmnnXLAAQfklltuaXj+sMMOy2GHHbbUYzthwoSG9961a9fcdNNN6dmzZ/bZZ588++yzWbRoUa655poceOCB2WmnnbLzzjvn0EMPzV//+tdGr/fkk0/mqKOOyi677JLdd989J510UqZNm5b33nsvPXv2zLe//e0l3tdnP/vZnHnmmR94jABYtUhAgGZh+PDhOf3003P66adn5513zq9+9aucc8456d27d66++upcfPHFadWqVU499dS8+uqry3ydJ598MiNHjsyQIUNy5ZVXpmXLlhkyZEhmzZq1wvs899xzOeKII5Ikl156aQYPHpxrrrmmUYKwLLfccku22GKLdOvWLX369En79u3z61//utE28+bNy6GHHpq//OUv+fa3v52f/OQnWXvttXPUUUflueeeS5Kcfvrpueaaa9K3b99cffXV6dWrV84666zcfvvtHxrDf1u0aFGuuuqqDB06NCeeeGK22mqrDBs2LFdeeWW++tWv5rrrrsv555+ft956KyeccELmzZuXJJk8eXL69euX+fPn5wc/+EHOP//8TJo0KQMHDkx9fX0OPvjg3HfffZkzZ07DXI8//nief/75HHLIIcsVIwCV1bLSAQAU4dBDD83+++/f8PvLL7+cgQMH5vjjj28Y22STTXLIIYfk0UcfzcYbb7zU13nnnXdy6623pnPnzkmStdZaK/3798/f/va3fOYzn1mhfa6++uq0bds21113XdZcc80kyZZbbplDDz30A9/T22+/nQceeCCDBw9OkrRu3ToHHHBAfve73+W0007L2muvnSS57bbb8vLLL+f2229PTU1NkqRHjx45+OCDM3HixNTV1eWuu+7Kd77znRx++OFJkj322COvvvpqJkyYkIMPPvgD4/hfxx57bD71qU81/D5z5sycdNJJjSoobdq0yeDBgzNlypR07949I0aMyDrrrJNRo0aldevWSZKOHTvmxBNPzJQpU/KlL30p1157be6555586UtfanhfnTt3To8ePZYrPgAqSwICNAvbbrtto9/fb21655138uKLL+bFF19saAlauHDhMl+nQ4cODYlEsviP5CSZP3/+Cu/zt7/9Lb169WpIPpKke/fu6dSp0we+p9/97nd577338ulPfzqzZ89OknzmM5/Jr3/969xxxx0NCczDDz+cTTbZpCH5SBYnK3fffXeSNFRM+vTp0+j1f/zjH3/g/MuyzTbbNPr9kksuSbL4Av2pU6fmhRdeaLhW5/1j/cgjj6RXr14NyUeS7LTTTo2u6dlll13y29/+Nl/60pdSW1ubsWPHZsCAAamqqlqhOAGoDAkI0Cyst956jX5/6aWXcs455+Rvf/tbWrZsmS233LIhSfmgtTn+O0lI0vDHb11d3Qrv8+abby4RX5Ksv/76y3zNZPE1F3V1dTnggAOWeG706NENCcjbb7+91Nd/39tvv51kyWO0ov73dZ544omcd955eeKJJ9KmTZtstdVWDcnV+8f6w2JMkr59++ass87Kq6++mscffzyzZ8+24hfAakgCAjQ7dXV1Ofroo9OqVav85je/SdeuXdOyZcv885//zO9+97vC4+nYsWPeeOONJcbfeOONbLHFFkvdZ9KkSXn66afzrW99K5/85CcbPffAAw/k5z//eR5//PF069Yt7dq1W+p9SR577LG0bds27du3T7I4EXq/OpMkzz//fN58882GFqdFixY12v/96zc+yJw5c3LUUUdl2223zZ133pkuXbpkjTXWyIMPPph77rmnYbt27do1Wlb5fQ8++GBqamqy4YYbZv/998/QoUNzzz335LHHHssee+yxzFY5AFZdLkIHmp233norL7zwQvr27ZuddtopLVsu/i7mj3/8Y5IPrmasDLvuumv++Mc/ZsGCBQ1jTz/99AfezPDmm29OdXV1jjjiiOy2226NHkceeWRatGiR0aNHJ1l8vcfLL7+cKVOmNOxfW1ubwYMH5ze/+U122WWXJMl9993XaI7hw4fnggsuSJK0bds206dPb/T8o48++qHv7fnnn8/bb7+dww8/PFtvvXXWWGPxPzv/e6x79OiRP/3pT6mtrW3Yd8qUKTn66KPzxBNPJFl87cznPve53HnnnfnTn/6k+gGwmlIBAZqd9dZbL506dcqvfvWrdOzYMe3bt8/48eNz/fXXJ/ng6zlWhmOPPTZjx47NUUcdlYEDB2b27Nm57LLLUlVVtdTrG2pra3PXXXelV69eadeu3RLPb7DBBtlrr70yduzYnHnmmTnkkEPyy1/+Mt/85jdzwgknpEOHDvnVr36Vd999N4cddlg6d+6c/fffP8OGDcu7776b7bffPuPHj8+4ceMargPZd99988ADD+TCCy/Mfvvtl0ceeaSkFbK22GKLtG3bNldddVVatmyZli1b5p577snNN9+c5D/H+rjjjstXv/rVDBo0KAMGDEhtbW0uu+yybL/99tlnn30aXq9v37756le/mrZt2+b//b//twJHG4BKUwEBmqURI0Zkww03zBlnnJETTzwxf//73/PTn/40W265ZR5++OFCY9lss80ycuTILFiwIEOGDMnw4cMzaNCgrL/++g0rWf23++67L2+//XYOPPDAZb7mF7/4xbz77ru57bbb0rZt29xwww3p3r17LrzwwpxwwglZsGBBfvnLXzZcHH/xxRfn8MMPzy9/+cscc8wxGT9+fH784x83rBz2pS99KYMGDcrYsWMzaNCgPProo7nssss+9L21a9cuI0aMSH19fU444YScdtppefXVV3PDDTdk7bXXbjjWXbt2zS9/+cvU1dXlpJNOyvnnn5+dd9451157baM7yu+8885Zd911c8ABB6RNmzbLdZwBWDW4EzpAhf31r39Nq1atGi0nO2vWrOy111457bTTGpbGJfnHP/6RL3/5y7nllluyww47VDocAFaAFiyACnvqqady+eWX5+STT87222+ft956K6NGjUq7du0+sMrRnEyYMCETJkzI7bffnt13313yAbAak4AAVNjAgQNTW1ubX//615k2bVrWWmutfPKTn8wPf/jDdOjQodLhrRLeeuut/OxnP8tWW22Viy66qNLhAPARaMECAAAK4yJ0AACgMBIQAACgMBIQAACgMBIQAACgMM1qFaz923+j0iE0aVXtl7wjM+VV/86cSofQpFW1aV3pEOCja+08Xpnq1lun0iE0efc8dl6lQ1imuunbVDqEpVqj4zOVDmG5qIAAAACFkYAAAACFaVYtWAAAsKLqUlfpEJZqdasorG7xAgAAqzEJCAAAUBgtWAAAUIJF9atmC9bq9ge9CggAAFAYCQgAAFCY1a1iAwAAFVGX+kqH0CSogAAAAIWRgAAAAIXRggUAACVYVW9EuLpRAQEAAAojAQEAAAqjBQsAAEqwqN4qWOWgAgIAABRGAgIAABRGCxYAAJTAjQjLQwUEAAAojAQEAAAojBYsAAAowSItWGWhAgIAABRGAgIAABRGCxYAAJTAKljloQICAAAURgICAAAURgsWAACUYFG9FqxyUAEBAAAKIwEBAAAKowULAABKUFfpAJoIFRAAAKAwEhAAAKAwWrAAAKAEi9yIsCxUQAAAgMJIQAAAgMJowQIAgBIs0oFVFiogAABAYSQgAABAYbRgAQBACdyIsDxUQAAAgMJIQAAAgMJowQIAgBIsSlWlQ2gSVEAAAIDCSEAAAIDCaMECAIAS1LkRYVmogAAAAIWRgAAAAIXRglWQXfbbIQPOPiSdazbOrNffydhR/5cxl95V0r5b7bxZfnz/2Tmy+xmZ8dIbjZ7bf8A++eJx/y8dN18/r73yRu647g/57U/HrYy3sErb5VPb5fBTD0jnbTpm1htzMvaXf85vriztOGy146YZ/ruTc+TeF2TmK282em6TLhvkyO8clJ322DrvvbcoT074Z649//ZM/5//Ds3BLr13yIDvHpLO2260+Bz+2f9lzKVjS9p3q503y4/v+06O/MSZSz+Hv9nn3+fwm7njugfy26vuWxlvYZW2y75dc/gZX0jnbTbKrDfeydhf/Cm/ufyekvbdaqfOGT72tBy5xzmZ+XLjc/hXT/wgHTZYZ4l9vrbD6XnrtdlliX114RivXLv0qsnhp34unbf+9+fwr/6S31xZ2v/LW+24SYbfflKO7HVhw+fwBpt0yPV/OWeZ+9z7mwkZfsqvyxL76qLHnltlwPGfTuct1s+st+flrpsfzphRf1rm9i1btsiXDt8zfQ7slvU7rpPXZszOH+7+R8aMGp/33lvUsF3P3tvlK0f0zCabfzzz5izIYw89n5GXjcvbb84t4m2tVqyCVR4SkAJs98mt8r3RJ+SPtz6U64femh322CYDzjkkVWtUZfSwOz9w3y122DTn33RSWrZa8j/VAUfum8HDD8+YS+/KY394Ktv22DJHX/jVtFmrOmMuKS25aQq222WLnDtqUP54x2P5xcV3Zftdt8yA0w/IGmtUZfQV937gvltst3HOu/6YtGzVYonnPr7Rx3LJ7Sflledm5offuj7VbVplwGkH5MIbj8s39/tBat9duLLe0ipnu092yfdGD1l8Dl9wa3bYY+sM+O4hqVpjjdLO4d+cuIxz+FMZfOnhGTN8bONzeO3Wzesc7rFlzv3FN/PH3z6SX/zgd9l+t60y4MwvLD6Hf/z7D9x3i66dct6vjlvqOfyx9dulwwbr5Opzbsrkh19o9Nzst+aU9T2s6hzjlWu7XTbPuSOPyh/vfCy/uHjs4s/hUz+XNaqqMvonH/xl0BbbbZzzfnb0Esf3rZmzctJBw5fY/sABPbPPgd1z75i/lfU9rOq6dts03/txvzx4z1O5/soHsv3OnXPE8Z/OGlVV+fXIPy51n2NP3T99Pr9zfnXtg3nmqX9lq5qNc9ixn8oGG30sw8/7bZJk7/265uyLv5q7bp6Yn1/5QNZdb+0c/s1P50fXHJHjv3Z1Fta+V+TbpJmQgBSg/5kH5fknXsrFR1+bJHnkvifTomWLfOWkz+XWn9yz1D9kW7ZqkS8cs18OP/uLy/xD9ysnfS4P3vJQfva9m5Mkf3/w6WyyVcccdMx+zeqPt6+ftH+en/SvDDvhl0mSR/7v6bRs1SJfPn6/3HrtH5Z9fL+xTw479YBlHt/Dvv25zJ/zbs469CdZ8O9tZrz8Rs4ddXS23mnTPPXQ8yvvTa1i+p/x73P4mOuSJI/c/2RatGqRr5xYwjn8nYOXfQ6f+Lk8eOv/nMNdNsxBR/duXufwKQfk+adeybBv/TxJ8sgfJqVlyzXy5cGfya1X3b/s43vkp3LY6V9I7bu1S33dLjtsmiT5y11/X6K619w4xivX10/89+fwib9Kkjzy4OTFn8PH9c6t1/5fahcs4/gesXcOO+VzSz3+C2sXZfJjUxuNbb3TptnnwO65/kd35amJLyyxT1P29aM/leenTM/F3701SfLwX/6Zli1b5Cvf6JlbbvhLahc0ThTatV8zB/TtkZGX3Zebf/HnJMnfH1p8zAad9P8y6vJxmfXWvHxtUK9M+NMzufzC/3yZ9PILr+eKXx2T3fbZJuPvm1TQO6Q5cQ3IStaqumV27Llt/vy7RxqNj//tw1mr3ZrZYc9tlrrfrv9vp3z9jIMyetidGXXOTUvd5uxDLs3Ic37TaGxh7Xtp1br55JWtqltmpz22zp/vfrzR+Pi7/p612rbJDp/sstT9dv1013z9pM9mzOX3ZtT3f7fUbfb87E65Z/TfGpKPJHn2Hy+nf4/vNqvko+EcvuPRRuOLz+E2H3wOn/6FjL7krow6dxnn8JeGZ+T/nN8LFy5qfufwnlvnz3c91mh8/J2PLT6Hd9tqqfvtut8O+fopB2TMZXdn1NDbl7rNljtsknfentes/zBOHOOVrVV1i+y0+1b58+//0Wj8P5/DWy51v1337Zqvn7h/xlwxLqMuuqOkuY4f2jcv/3NGbrvu/z5q2KuVVq1aZKcem2f8A083Gv/TfU9lrbVbZ4fumy2xz1ptW+eumx/O3x6c3Gj8Xy+9niTp2GndVFVV5bEJz+fuWx9utM0r/26V3XiTDuV8G03ColStko/VjQRkJeu4+fqpbt0q//rnjEbjrz6/+PdOW3Vc6n7PPPpCBux4akYPuzOL/qtP87+9/My0zHx58YdE23XXzv6H75P9+u2VO659oIzvYNXWsfN6adW6Zf71/MxG46+++FqSpNOW6y91v2cefykD9vheRl9x71KP74abdkjbddbKjFfezHFDv5wxT1yU3/7zknzvZ0dn/Y3XLf8bWYX95xye3mj81X8f805dNlzqfs88+kIG7HTav8/huqVus+Q5vHf2O3TP3HHtH8r4DlZtHTf7eFq1bpV/Pfc/5/AL/z6Hu2yw1P2eeezFDOhxdkb/+PfL/IzossOmmTNrXs4edXRufvbS3Pr88Jxx9ZFZd4P25X0TqzjHeOXq2Pnj//4cfq3R+KtTF/+h+4Gfw3uen9E/GbfMz4j/9qmDPpFtd94sV33vttQ1s7VQO26ybqqrW+ZfUxtfQ/fqv69H2mSz9ZbYZ8arb+cnF92VV/5nn70+3TULF76Xf019I/X19bnm0nvy1/+b0mibnp/eLkny4v/87QLlUvGvGd97773ce++9efjhh/Pqq6+mtrY2a665Zjp27JgePXqkT58+admy4mGusLYfWytJMu+d+Y3G573zbpJkrXZtlrrfG9PeLnmOrrttlUvHfSfJ4j/6mtMFvGu3XzPJf47n++bNWZAkWavtMo7v9Fkf+LrrrNc2STLwzC9kyt+n5ofHX591Pt423zjj8/nhbwbnm31+kAXzl96S0dS0Xef9c/h/jvH75/C//xv8r+U+h+89K8niP/p+e3UzOofX+fc5POd/z+H3PyOWcXw/5BxOki233yQf3+hj+f0N43Pb1Q+k8zYd0/+0A/Oj20/Ot/b7fhbMax7nsGO8cjV8Di9xfD/kc3jGhx/f//alo/fNUxOfzxN/++cKRLl6a/vvvxXmzV3QaHzev8+vtdZuXdLr9Oy9XXofsFNuv3FC5vzPZ/r7Nu7cIUed+P/y7KRXM/HPze9YU4yK/mX/0ksvZdCgQZkxY0a6du2aDTbYIOuss04WLFiQp59+OrfcckuuuOKKXHfdddl4440rGeoKq6paXBarr1/6tzX1ZfgWZ/rU13PqZ3+Q9TZeN4eddXCuePDcDPnU+Xm7Gay+ssYa/y47LuMw1i3juH+YVv++YPrt19/J0EEjG/77TXvx9Qz/3cn59CE9cvev/rJCr726qVqjoHP4cz/Iehv9+xz+v3MyZN8Lmsc5XPX+Obz041hX9+HfDC/LpSf+IgvfXZjnnnwlSfLUhH9m6uRXc8mdp2a/L++eu65f+oWrTY1jvHL953N4Gcd3BT+H/1vXHltkqx03zXlHXveRX2t19GF/S5RyjHvu1zWnX/ilPPHI1Iy6fOkLA2y6xcdz0U8Pz8KFi3LBqWOWOV9zVle/+rU7rYoqmoCcd9552WSTTXLzzTenXbt2Szw/e/bsnHTSSTn//PNz1VVXVSDCj27urHlJlvyG7f3Kx9zZ85fYZ3m9Of3tvDn97STJlIefy8jHfpD9B+zzoasTNQVz/n38/reStFbbxd8GzZu99G94Psz73+RN/MOkRh/Akx99Me+8PS9dtt9khV53dfTh5/C8jzxHo3P4kecz8tGLmt853PZ/z+F/f+O5gudwkiVWZUqSSROfz5xZ87LF9p1W+HVXN47xyrXs4/vRPof/W8/Pdcs7b8/NxD80zwui575fcf6fSsdaa1Un+U+1aVkO6b9Hjjrx/+UfD7+Y75306yxcuGRLYbcem+e7lxya+XMX5IxvXp8Zr75dnuBhKSp6DcgjjzyS0047banJR5K0b98+p556aiZOnFhwZOXz6gszs+i9Rdl4y8Y9xhtvubhv/qXJr67Q667Ztk32/cru2eh/XnfaC69lztvzsn6n5nHh2LSpr2fRe4uy0eYfbzS+8eaLe45fenb60nYr7XUX1aVV9ZI5estWLRpdmN7ULfscXvz7RzqHv/xB53DzuNZm2ouvLT6Ht/if47vFv8/hZ6at0Ouu3X7N/L9+e6TzNkteZ9ayVcvMbkbr+zvGK9cyP4c3W/z7in4O/7dP9t4+f73nyZKuFWmKXn3lrcWfw5s2vtZj400X/1s/9X+ug/xvx53+uRzz7f0z/r5J+e7gG/LuUtqH991/x1w44rC8MXN2Tjpi5BLXjUC5VTQBad++fWbOXPb/NEny6quvpk2bpfePrg4WLngvT/z5mez1hV0ajfc8qEfeeWtupjyyYqsp1S2qy0k/GZgvn/DZRuPbfGKLtO/QNs8/+fIKx7w6WbjgvTwx4bns9dlujcZ7HrBz3nl7Xqb8feoy9vxg786rzVMPLX7d/05Cdt5rm6y5dus89dBzHynu1cnCBe/lib88k70+/4lG4z0P6pF33p6bKY+s2FKYi8/hbyzlHN68+Z3Df/tn9jpg50bjPQ/svvgcfuzFFXrd9xa+l+N/0C9fHvyZRuN7fLZb2qxVnX/85ZkVjHj14xivXA2fw/sv5XN41rxM+ftLH+n1266zVjptsX6eerj5rD74vxbWvpcnHp2avXpv12h87/22zzuz52fKk/9a6n7fGLxfDjp0t9xyw1/y/TNuWmrlY9eeW+fUC76YSY+/nJO+MTKvz2z6ra8fRaVXu2oqq2BVtAWrb9++OfPMMzNkyJDstttu2WijjVJdXZ3a2trMmDEjDz30UIYNG5a+fftWMsyP7NcX35GLfndKvnP9cbnnhj+l625bpe8J+2fUOTel9t2FWatdm3Su2TjTnn8ts954p6TXXDC/Nr/58dh87bTP55035+ax/3sqnbbqmP5nHpTn/vFS7r1h2XdGbWpGX3ZPvj/6+Jx11Tdy75i/ZbtdtsiXjv10Rn3/d4uPb9s26bxNx0x78fXMerP0G4P97KI78qObhuT8XxyTW65+IB/7eLsMPOsLmfzoi/nbvU+sxHe06vn1xXfkot+eku9c/83c88vxi8/hIftn1Lk3/+cc3nbjTHthRc/hOXns/yal01Ybpv8Z75/D41fyu1p1jB5+d75/05Ccde1RuffXf812u26ZLx3fJ6MuuP0/5/C2G2Xai69l1hulncML5i/MzVfem699+4C8/do7eeQPT2Xzrp3S/5QDM2HcE/n7Hyd/+Is0IY7xyjX6inH5/o3fzFk/PSL3jpmQ7XbZPF86Zt+MuuiO1C5YmLXatk7nrTtm2tTXM2s5K0Nb1GyUJHnp2ea9ItON1/0xP7jq8HznR1/JPb99NF27dU7fAXtm5GX3pXbBe1lr7dbpvOX6mfbKm5n11rxsuU3HfOWIvTLlqX/lj/c+lZodG7cOv/T8a1m4cFFOOucLmTevNqOv+2M6b9F4xbLXZ8yWkLBSVDQBGTx4cNZYY4388Ic/zLx5S/aRr7322vn617+eE044oQLRlc/jf3w6Q/tfmcPOOjjn3Dg4b7z6Vq47+ze59Sf3JEm26rZZfjT2jFxy7HUZd+OfS37dX13027w1Y1YOPOrTOfi4Pnnnrbn5420Tc/0Ft2bhguZz59LH//JsLjx6VPp/+7M557pBeX362xk59Le59ZrFS7l22XGT/OimIbnkpBty300Plfy6kx99Mad/5YoMOP3AfOeaI7Ngfm3+es8Tue6C25vdEpCP/3Fyhh42IoedeVDOufFbeWPa27nuuzc1PofvOj2XfHPk8p3DP/hd3po5Kwce+ekc/M3/OoeH3ta8zuHxU3LhwGvS/7QDc87Pj8nr02dl5Hm35tar7k+SdNlp0/zotpNzyZDrc99y3P35hovvyluvvZMDjtgnnx/YK++8NTdjf/Gn3HBx07+25n85xivX4395Nhce87P0P/mzOefaI/P6jLcz8sLf5dZr/y/J4uWKf/Sbb+WSk2/MfTeX/jmcLL7bfJLMmfXRrzdbnT0+8YVccMqYHHbsvjn30n55Y+bsXPfjcbnll4sXRNmqZqNcfN03Muyc2zLujr+nZ+/tssYaa2Tb7Tvlsl8MWuL1Tj3qZ6laoyrrrb94yeiLrhqwxDa/vOoPueHq/1up74vmqap+FVjiYOHChXn66aczY8aMzJ8/P23atEnHjh1TU1OT6urqss2zf/tvlO21WFJV+6Vfy0P51L9TegWH5VfVprSlLGGV1tp5vDLVrbdOpUNo8u557LxKh7BME1/avNIhLNWunV+sdAjLZZW4wUarVq2y0047VToMAABgJXMndAAAoDCrRAUEAABWdW5EWB4qIAAAQGEkIAAAQGG0YAEAQAlWx5v+rYpUQAAAgMJIQAAAgMJowQIAgBIsqvfdfTk4igAAQGEkIAAAQGG0YAEAQAnqfHdfFo4iAABQGAkIAABQGC1YAABQAjciLA8VEAAAoDASEAAAoDBasAAAoARuRFgejiIAADRTdXV1ufzyy7P33nunW7duGThwYKZOnbrM7V9++eUce+yx+eQnP5m99torQ4cOzfz585drTgkIAAA0UyNGjMjo0aMzdOjQjBkzJlVVVRk0aFBqa2uX2Padd95Jv379MmvWrFx33XW56qqr8uSTT+b4449frjklIAAAUIK6VK2SjxVVW1ubUaNGZfDgwenVq1dqamoyfPjwzJgxI+PGjVti+9tuuy1z5szJlVdemZ122ik77rhjhg8fnr/85S95+OGHS55XAgIAAM3Q5MmTM3fu3Oy+++4NY+3bt0/Xrl0zceLEJbZ/4YUXsuWWW6ZDhw4NYxtttFHWXXfdPPTQQyXP6yJ0AABYjfXu3fsDn7///vuXOj59+vQki5OI/7bBBhtk2rRpS2y//vrr57XXXsuiRYvSokWLJMmcOXMya9asvPHGGyXHqwICAAAlWJQ1VsnHinr/4vHq6upG461bt86CBQuW2P6AAw7IrFmz8v3vfz9z587N7Nmzc+6556aqqmqp14wsiwoIAACsxpZV4fgwbdq0SbL4WpD3f06SBQsWZM0111xi+8022yxXXHFFzjnnnPzqV79KmzZtcthhh2WHHXZI27ZtS55XAgIAAM3Q+61XM2fOTOfOnRvGZ86cmZqamqXu06tXrzz44IN57bXX0q5du7Rp0yZ77rlnDjnkkJLn1YIFAAAlWFS/xir5WFE1NTVp27ZtJkyY0DA2e/bsTJo0KT169Fhi+0ceeST9+/dPbW1t1l9//bRp0yYPPfRQ3nrrrey5554lz6sCAgAAzVB1dXX69++fYcOGpUOHDunUqVMuvvjidOzYMX369MmiRYvy5ptvNlQ6unTpkmeffTbf//73c+SRR+bll1/OaaedlkMPPTSbbrppyfNKQAAAoJkaMmRI3nvvvZx99tl59913s+uuu2bkyJGprq7OK6+8kt69e+eiiy7KIYccko997GO55pprctFFF+Xzn/981l133Rx66KH55je/uVxzVtXX19evpPezytm//TcqHUKTVtW+XaVDaPLq35lT6RCatKo2rSsdAnx0rZ3HK1PdeutUOoQm757Hzqt0CMv02+d3rnQIS3XQln+vdAjLxTUgAABAYSQgAABAYVwDAgAAJVhUX1XpEJoEFRAAAKAwEhAAAKAwWrAAAKAEi3x3XxaOIgAAUBgJCAAAUBgJCAAAUBjXgAAAQAnq6n13Xw6OIgAAUBgJCAAAUBgtWAAAUALL8JaHowgAABRGAgIAABRGCxYAAJRgUX1VpUNoElRAAACAwkhAAACAwmjBAgCAEtT57r4smlUCMvknNZUOoUmrXru20iE0ebVzOlU6hCatw/rvVDqEJm/Bwmb1z05FzH17zUqH0KQds+uDlQ4BVnvSOAAAoDC+igIAgBIsqvfdfTk4igAAQGEkIAAAQGG0YAEAQAnq4kaE5aACAgAAFEYCAgAAFEYLFgAAlMAqWOXhKAIAAIWRgAAAAIXRggUAACVY5Lv7snAUAQCAwkhAAACAwmjBAgCAEtTVuxFhOaiAAAAAhZGAAAAAhdGCBQAAJbAKVnk4igAAQGEkIAAAQGG0YAEAQAnq6n13Xw6OIgAAUBgJCAAAUBgtWAAAUIJFcSPCclABAQAACiMBAQAACqMFCwAASmAVrPJwFAEAgMJIQAAAgMJowQIAgBJYBas8VEAAAIDCSEAAAIDCaMECAIASWAWrPBxFAACgMBIQAACgMFqwAACgBIu0YJWFowgAABRGAgIAABRGCxYAAJSgzo0Iy0IFBAAAKIwEBAAAKIwWLAAAKIFVsMrDUQQAAAojAQEAAAqjBQsAAEpQV28VrHJQAQEAAAojAQEAAAqjBQsAAEqwyHf3ZeEoAgAAhZGAAAAAhdGCBQAAJbAKVnmogAAAAIWRgAAAAIXRggUAACWo8919WTiKAADQTNXV1eXyyy/P3nvvnW7dumXgwIGZOnXqMrd/7bXXcvLJJ2e33XbLbrvtlhNOOCHTp09frjklIAAA0EyNGDEio0ePztChQzNmzJhUVVVl0KBBqa2tXer2J510UqZNm5af/exn+dnPfpbp06fnuOOOW645K96Cddhhh6WqqrQVBX7xi1+s5GgAAGDpFjWxVbBqa2szatSonHrqqenVq1eSZPjw4dl7770zbty4HHDAAY22nz17diZOnJif/vSn6dq1a5Lk6KOPznHHHZe33nor6667bknzVrwCsscee2TixIl544030qlTpw98AAAA5TF58uTMnTs3u+++e8NY+/bt07Vr10ycOHGJ7Vu3bp211lort99+e+bMmZM5c+bkt7/9bTbffPOss846Jc9b8QrIcccdl7XWWiuXX355rr766myyySaVDgkAAJq896/d2GijjRqNb7DBBpk2bdoS27du3ToXXnhhzj///PTo0SNVVVVZf/31c8MNN2SNNUqva1Q8AUmSI444IuPHj8+Pf/zjDBs2rNLhAADAElbVGxH27t37A5+///77lzo+f/78JEl1dXWj8datW2fWrFlLbF9fX58pU6ake/fuOeqoo7Jo0aIMHz48xx9/fH7961+nbdu2JcW7SiQgSXLhhRdm0qRJlQ4DAACahTZt2iRZfC3I+z8nyYIFC7Lmmmsusf1dd92VG2+8MX/4wx8ako2rrroq++67b2655ZYMGDCgpHlXmQRkww03zIYbbljpMAAAYLWyrArHh3m/9WrmzJnp3Llzw/jMmTNTU1OzxPaPPPJItthii0aVjnXWWSdbbLFFXnzxxZLnrfhF6AAAsDqoq19jlXysqJqamrRt2zYTJkxoGJs9e3YmTZqUHj16LLH9RhttlKlTp2bBggUNY/Pnz88rr7ySzTbbrOR5JSAAANAMVVdXp3///hk2bFjuv//+TJ48OSeddFI6duyYPn36ZNGiRXnttdfy7rvvJkkOPvjgJMmJJ56YyZMnN2xfXV2dQw45pOR5JSAAANBMDRkyJH379s3ZZ5+dfv36pUWLFhk5cmSqq6szbdq09OzZM2PHjk2yeHWsG2+8MfX19RkwYEC+8Y1vpFWrVvn1r3+d9u3blzznKnMNCAAArMoWZdVcBeujaNGiRU499dSceuqpSzy3ySabZMqUKY3GunTpkquuuuojzakCAgAAFEYCAgAAFEYLFgAAlGBVvRHh6kYFBAAAKIwEBAAAKIwWLAAAKMFHuekf/+EoAgAAhZGAAAAAhdGCBQAAJahrgjcirAQVEAAAoDASEAAAoDBasAAAoASL3IiwLFRAAACAwkhAAACAwmjBAgCAErgRYXk4igAAQGEkIAAAQGG0YAEAQAnqrIJVFiogAABAYSQgAABAYbRgAQBACeqiBascVEAAAIDCSEAAAIDCaMECAIASWAWrPFRAAACAwkhAAACAwmjBAgCAEtTV++6+HBxFAACgMBIQAACgMFqwAACgBFbBKg8VEAAAoDASEAAAoDBasAAAoAR10YJVDiogAABAYSQgAABAYbRgAQBACayCVR4qIAAAQGEkIAAAQGG0YAEAQAm0YJWHCggAAFAYCQgAAFAYLVgAAFACLVjloQICAAAURgICAAAUplm1YP2p948rHUKTtqi+0hHAR7MwSusrW4v4oFjZNmrRptIhNGmvLFpQ6RCoIC1Y5aECAgAAFEYCAgAAFKZZtWABAMCKqtOqWxYqIAAAQGEkIAAAQGG0YAEAQAmsglUeKiAAAEBhJCAAAEBhtGABAEAJtGCVhwoIAABQGAkIAABQGC1YAABQAi1Y5aECAgAAFEYCAgAAFEYLFgAAlEALVnmogAAAAIWRgAAAAIXRggUAACWo14JVFiogAABAYSQgAABAYbRgAQBACeqiBascVEAAAIDCSEAAAIDCaMECAIASuBFheaiAAAAAhZGAAAAAhdGCBQAAJXAjwvJQAQEAAAojAQEAAAqjBQsAAErQFFfBqqury09+8pPcdNNNmT17dnbZZZece+652WyzzZbY9oorrshPfvKTpb7OIYcckosuuqikOVVAAACgmRoxYkRGjx6doUOHZsyYMamqqsqgQYNSW1u7xLYDBw7M+PHjGz1OPPHEtGnTJgMGDCh5TgkIAAA0Q7W1tRk1alQGDx6cXr16paamJsOHD8+MGTMybty4JbZfe+21s/766zc85s+fn6uvvjpnnHFGampqSp5XAgIAACWor69aJR8ravLkyZk7d2523333hrH27duna9eumThx4ofu/4Mf/CBbb711vvrVry7XvK4BAQCAZmj69OlJko022qjR+AYbbJBp06Z94L5PPPFE7r///lx//fVZY43lq2lIQAAAYDXWu3fvD3z+/vvvX+r4/PnzkyTV1dWNxlu3bp1Zs2Z94Gv+/Oc/T7du3RpVT0olAQEAgBI0tVWw2rRpk2TxtSDv/5wkCxYsyJprrrnM/ebNm5dx48bl3HPPXaF5JSAAALAaW1aF48O833o1c+bMdO7cuWF85syZH3hR+Z/+9KfU1dWlT58+KzSvi9ABAKAZqqmpSdu2bTNhwoSGsdmzZ2fSpEnp0aPHMvd75JFHsv3226d9+/YrNK8KCAAAlKC+vtIRlFd1dXX69++fYcOGpUOHDunUqVMuvvjidOzYMX369MmiRYvy5ptvpl27do1atCZPnpxtttlmhedVAQEAgGZqyJAh6du3b84+++z069cvLVq0yMiRI1NdXZ1p06alZ8+eGTt2bKN9Xn/99XzsYx9b4Tmr6uubWi63bC//a6MP34gVtqjZnEk0VQvTtC4uXBW1iA+KlW2jFm0+fCNW2CuLFlQ6hCZv601erXQIy7Tr3WdVOoSlmvjZ71c6hOWiBQsAAEpQ54uqstCCBQAAFEYCAgAAFEYLFgAAlKC+id2IsFJUQAAAgMJIQAAAgMJowQIAgBLUacEqCxUQAACgMBIQAACgMFqwAACgBPX1lY6gaVABAQAAClPxBOSFF17IFVdckaFDh+bBBx9c4vk5c+bkzDPPrEBkAABAuVU0AXnkkUfyxS9+MXfeeWf++Mc/5thjj83gwYNTW1vbsM27776b22+/vXJBAgBAFt+IcFV8rG4qmoBccskl6du3b+65557ce++9ufTSS/PnP/85xx57bBYuXFjJ0AAAgJWgognIlClT0r9//4bfP/vZz+baa6/NY489ltNOO62CkQEAACtDRROQtm3b5q233mo0tssuu+Tiiy/OPffck4suuqhCkQEAQGOVbrXSglUGvXr1yvnnn5/HH3+8UcvVfvvtl7POOivXX399zj///ApGCAAAlFNFE5Bvf/vbWXfddXPooYfmr3/9a6Pn+vfvn3POOScPPPBAhaIDAADKraI3IlxnnXUyatSovPTSS1l33XWXeP5rX/ta9thjj9x7770ViA4AAP6jbjVsd1oVrRJ3Qu/cufMyn9tiiy1yzDHHFBgNAACwslT8RoQAAEDzsUpUQAAAYFVXX1/pCJoGFRAAAKAwEhAAAKAwWrAAAKAEq+NN/1ZFKiAAAEBhJCAAAEBhtGABAEAJtGCVhwoIAABQGAkIAABQGC1YAABQAvchLA8VEAAAoDASEAAAoDBasAAAoARWwSoPFRAAAKAwEhAAAKAwWrAAAKAUlsEqCxUQAACgMBIQAACgMFqwAACgBFbBKg8VEAAAoDASEAAAoDBasAAAoAT1VsEqCxUQAACgMBIQAACgMFqwAACgBFbBKg8VEAAAoDASEAAAoDBasAAAoBRasMpCBQQAACiMBAQAACiMFiwAACiBGxGWhwoIAABQGAkIAABQGC1YAABQCi1YZaECAgAAFEYCAgAAFEYLFgAAlKDejQjLQgUEAAAojAQEAAAojBYsAAAohVWwykIFBAAAKIwEBAAAKIwWLAAAKIFVsMpDBQQAACiMBAQAACiMFiwAACiFVbDKQgUEAAAoTLOqgHRq0a7SIQDASnX+69tVOoQm7W9f27HSITR5v/9HpSNgZWtWCQgAAKw4q2CVgxYsAACgMBIQAACgMFqwAACgFFbBKgsVEAAAoDASEAAAaKbq6upy+eWXZ++99063bt0ycODATJ06dZnbL1y4MJdcckn23nvv7Lzzzunfv3+efvrp5ZpTAgIAAKWoX0UfH8GIESMyevToDB06NGPGjElVVVUGDRqU2trapW7/ve99LzfffHMuuOCC3HLLLfnYxz6WQYMG5Z133il5TgkIAAA0Q7W1tRk1alQGDx6cXr16paamJsOHD8+MGTMybty4JbZ/+eWXc/PNN+eiiy7Kpz71qXTp0iXf//73U11dnSeffLLkeSUgAADQDE2ePDlz587N7rvv3jDWvn37dO3aNRMnTlxi+/Hjx6d9+/bZZ599Gm3/wAMPZI899ih5XqtgAQBAKepXzRsR9u7d+wOfv//++5c6Pn369CTJRhtt1Gh8gw02yLRp05bY/sUXX8ymm26ae++9N9dcc01mzJiRrl275owzzkiXLl1KjlcFBAAAmqH58+cnSaqrqxuNt27dOgsWLFhi+zlz5uSll17KiBEjcvLJJ+enP/1pWrZsma997Wt54403Sp5XBQQAAFZjy6pwfJg2bdokWXwtyPs/J8mCBQuy5pprLrF9q1at8s4772T48OENFY/hw4enV69eue2223LUUUeVNK8KCAAAlKC+ftV8rKj3W69mzpzZaHzmzJnp2LHjEtt37NgxLVu2bNRu1aZNm2y66aZ55ZVXSp5XAgIAAM1QTU1N2rZtmwkTJjSMzZ49O5MmTUqPHj2W2L5Hjx5577338sQTTzSMvfvuu3n55Zez2WablTyvFiwAAGiGqqur079//wwbNiwdOnRIp06dcvHFF6djx47p06dPFi1alDfffDPt2rVLmzZt0qNHj+y55545/fTTc/755+djH/tYLr/88rRo0SIHHXRQyfOqgAAAQCkqfcPBlXAjwiFDhqRv3745++yz069fv7Ro0SIjR45MdXV1pk2blp49e2bs2LEN219xxRX55Cc/mW9961vp27dv5syZk1/84hfp0KFDyXNW1dd/lM6x1Uvd9G0qHQIArFTnv75dpUNo0v72tR0rHUKT9/t/DK10CMu02cgfVTqEpZp65GmVDmG5qIAAAACFcQ0IAACUYhW9EeHqRgUEAAAojAQEAAAojBYsAAAoQVWzWbpp5VIBAQAACiMBAQAACqMFCwAASqEFqyxUQAAAgMJIQAAAgMJowQIAgFK4EWFZqIAAAACFkYAAAACF0YIFAAClsApWWaiAAAAAhZGAAAAAhdGCBQAApdCCVRYqIAAAQGEkIAAAQGG0YAEAQCm0YJWFCggAAFAYCQgAAFAYLVgAAFCK+qpKR9AkqIAAAACFkYAAAACF0YIFAAAlqLIKVlmogAAAAIWRgAAAAIXRggUAAKXQglUWKiAAAEBhJCAAAEBhJCAAAEBhJCAAAEBhlusi9Ndeey1XXnllXn755Xz84x/Pdtttlx122CHbb7991lxzzZUVIwAA0EQsVwJy1llnZfz48dl6663zyiuv5I477kh9fX3WWGONbLnlltlhhx2y4447Zscdd0xNTU1atWq1suIGAIBCuRFheSxXAvLYY4/l1FNPzcCBA5Mk8+bNy1NPPZUnnngiTzzxRCZOnJjbbrstSVJdXZ1//OMfH/qaCxYsyLPPPputttoqbdq0ydNPP50bbrghM2bMyNZbb50BAwakY8eOK/DWAACAVc1yJSCtW7dO165dG35fa621suuuu2bXXXdtGHv77bfzj3/8I08++eSHvt5zzz2XI444Iq+99lo23njjDB06NMcdd1w22WSTdOnSJffdd19uvfXW3HjjjenSpcvyhAoAAKyClusi9P322y+TJk36wG0+9rGPZZ999slxxx33oa/3ox/9KN27d8/tt9+eXXbZJd/85jfz+c9/PnfccUcuu+yy3H333dlrr71y0UUXLU+YAABQfvVVq+ZjNbNcCciXvvSl3H333fnnP/9ZlskfeuihnHjiiampqcnpp5+eBQsWpF+/fqmqWnwgW7ZsmWOPPTaPPPJIWeYDAAAqa7lasL7yla+kqqoqX/7yl7P//vtn7733zvbbb5/NNttshSZv06ZN3n333STJxz/+8XzlK19J69atG20ze/bstGvXboVeHwAAWLUsVwIydOjQPP3003nqqady991357bbbktVVVXWXnvtdO3aNTvssENOO+20kl+vZ8+eueCCCzJ06NB06dIl559/fsNz9fX1eeihh3Leeedlv/32W54wAQCg/KyCVRbL1YLVt2/ffPe7383o0aPz6KOP5o477shFF12Ugw8+OO+9915Gjx69XJOfeeaZWbRoUUaMGLHEc2PHjs2AAQPSqVOnnHzyycv1ugAAwKppuSog/22NNdbI1ltvna233joHH3xwksVVi+XRoUOH/OY3v8nbb7+9xHN77LFHbr/99tTU1KxoiAAAwCpmhROQpXn/4vHl9bGPfWyJsQ4dOqRDhw4fMSIAACgTLVhlsVwtWAAAAB+FBAQAAChMWVuwAACgqarSglUWKiAAAEBhJCAAAEBhtGABAEAptGCVhQoIAABQGAkIAABQGC1YAABQCi1YZaECAgAAFEYCAgAAFEYLFgAAlMCNCMtDBQQAACiMBAQAACiMFiwAAChFfVWlI2gSVEAAAIDCSEAAAIDCaMECAIBSWAWrLFRAAACAwkhAAACAwmjBAgCAErgRYXmogAAAAIWRgAAAAIXRggUAAKXQglUWKiAAAEBhJCAAAEBhtGABAEAJrIJVHiogAABAYSQgAABAYbRgAQBAKbRglYUKCAAANFN1dXW5/PLLs/fee6dbt24ZOHBgpk6dusztb7vttmy77bZLPD5on/+lAgIAAM3UiBEjMnr06Fx00UXZcMMNc/HFF2fQoEG58847U11dvcT2U6ZMySc/+clceumljcY7dOhQ8pwqIAAAUIr6VfSxgmprazNq1KgMHjw4vXr1Sk1NTYYPH54ZM2Zk3LhxS93nmWeeSU1NTdZff/1GjxYtWpQ8rwQEAACaocmTJ2fu3LnZfffdG8bat2+frl27ZuLEiUvdZ8qUKdlqq60+0rxasAAAYDXWu3fvD3z+/vvvX+r49OnTkyQbbbRRo/ENNtgg06ZNW2L7N998M6+//nomTpyYX/7yl3n77bfTrVu3nHLKKdliiy1KjlcFBAAASlBVv2o+VtT8+fOTZIlrPVq3bp0FCxYssf0zzzyTJGnRokV++MMfZvjw4Zk3b16+9rWv5fXXXy95XhUQAABYjS2rwvFh2rRpk2TxtSDv/5wkCxYsyJprrrnE9rvvvnseeuihrLPOOg1jV155Zfbdd9/ceuutOfroo0uaVwUEAACaofdbr2bOnNlofObMmenYseNS9/nv5CNJ1lprrWyyySaZMWNGyfNKQAAAoBmqqalJ27ZtM2HChIax2bNnZ9KkSenRo8cS2994443Zbbfd8u677zaMzZkzJy+++OJyXZguAQEAgGaouro6/fv3z7Bhw3L//fdn8uTJOemkk9KxY8f06dMnixYtymuvvdaQcOy7776pr6/PaaedlmeffTZPPPFEBg8enA4dOuSLX/xiyfNKQAAAoJkaMmRI+vbtm7PPPjv9+vVLixYtMnLkyFRXV2fatGnp2bNnxo4dm2Rxy9b111+fuXPnpl+/fjniiCPSrl27/OIXv2h0DcmHqaqvr/8I186vXuqmb1PpEABgpTr/9e0qHUKT9rev7VjpEJq83/9jaKVDWKZtzx9e6RCWaso5J1U6hOWiAgIAABRGAgIAABTGfUAAAKAEH+Wmf/yHCggAAFAYCQgAAFAYLVgAAFAKLVhl0awSkM9s3K3SIQCrsipF4ZWuvq7SETR9zuOVq35KpSOA1Z5PKQAAoDDNqgICAAArTAtWWaiAAAAAhZGAAAAAhdGCBQAAJXAjwvJQAQEAAAojAQEAAAqjBQsAAEqhBassVEAAAIDCSEAAAIDCaMECAIASWAWrPFRAAACAwkhAAACAwmjBAgCAUmjBKgsVEAAAoDASEAAAoDBasAAAoBRasMpCBQQAACiMBAQAACiMFiwAACiBGxGWhwoIAABQGAkIAABQGC1YAABQCi1YZaECAgAAFEYCAgAAFEYLFgAAlEILVlmogAAAAIWRgAAAAIXRggUAACVwI8LyUAEBAAAKIwEBAAAKowULAABKoQWrLFRAAACAwkhAAACAwmjBAgCAElgFqzxUQAAAgMJIQAAAgMJowQIAgFJowSoLFRAAAKAwEhAAAKAwEhAAAKAwrgEBAIBSuAakLFRAAACAwkhAAACAwmjBAgCAElRVOoAmQgUEAAAojAQEAAAojBYsAAAohVWwykIFBAAAKIwEBAAAKIwWLAAAKEGVFqyyUAEBAAAKs8omIJ///Oczbdq0SocBAACUUUVbsG6//fZlPjd16tTcfffd6dChQ5Lk4IMPLiYoAABYGi1YZVHRBOS8887Lu+++mySpr1/yv+iPfvSjJElVVZUEBAAAmoCKJiC33nprTjnllLRr1y4//OEPs+GGGzY817179/zud7/LpptuWsEIAQCAcqroNSBbbLFFxowZk5122ikHHXRQxo4dW8lwAABg2epX0cdqpuIXobds2TInn3xyrrjiigwbNizf/va3884771Q6LAAAYCWoeALyvl133bXhovQDDzwwCxcurGxAAABA2a1SNyJs3759Lrnkktx+++259dZb07p160qHBAAASdyIsFxWqQTkfQcffLBVrwAAoAlaZVqwAACApm+VrIAAAMAqRwtWWaiAAAAAhZGAAAAAhdGCBQAAJbAKVnmogAAAQDNVV1eXyy+/PHvvvXe6deuWgQMHZurUqSXte8cdd2TbbbfNK6+8slxzSkAAAKCZGjFiREaPHp2hQ4dmzJgxqaqqyqBBg1JbW/uB+/3rX//Keeedt0JzSkAAAKAU9avoYwXV1tZm1KhRGTx4cHr16pWampoMHz48M2bMyLhx45a5X11dXU499dRsv/32KzSvBAQAAJqhyZMnZ+7cudl9990bxtq3b5+uXbtm4sSJy9zvqquuysKFC3PMMces0LwuQgcAgGZo+vTpSZKNNtqo0fgGG2yQadOmLXWff/zjHxk1alRuvvnmzJgxY4XmlYAAAEAJVtVVsHr37v2Bz99///1LHZ8/f36SpLq6utF469atM2vWrCW2nzdvXk455ZSccsop2XzzzVc4AdGCBQAAzVCbNm2SZIkLzhcsWJA111xzie2HDh2azTffPIceeuhHmlcFBAAAVmPLqnB8mPdbr2bOnJnOnTs3jM+cOTM1NTVLbH/LLbekuro63bt3T5IsWrQoSXLggQfmC1/4Qs4///yS5pWAAABAKVbRFqwVVVNTk7Zt22bChAkNCcjs2bMzadKk9O/ff4nt77333ka/P/744zn11FNzzTXXpEuXLiXPKwEBAIBmqLq6Ov3798+wYcPSoUOHdOrUKRdffHE6duyYPn36ZNGiRXnzzTfTrl27tGnTJptttlmj/d+/iH3jjTfOeuutV/K8rgEBAIBmasiQIenbt2/OPvvs9OvXLy1atMjIkSNTXV2dadOmpWfPnhk7dmxZ56yqr69vYsWkZeuzxpcrHQKwKqvyncxKV19X6QiaPufxyuUcXunG1d1U6RCW6RPHDK90CEv16NUnVTqE5eJTCgAAKIwEBAAAKIyL0AEAoASr6o0IVzcqIAAAQGEkIAAAQGG0YAEAQCm0YJWFCggAAFAYCQgAAFAYLVgAAFCCquZz/+6VSgUEAAAojAQEAAAojBYsAAAohQ6sslABAQAACiMBAQAACqMFCwAASlClBassVEAAAIDCSEAAAIDCaMECAIBSaMEqCxUQAACgMBIQAACgMFqwAACgBFbBKg8VEAAAoDASEAAAoDBasAAAoBRasMpCBQQAACiMBAQAACiMFiwAACiBVbDKQwUEAAAojAQEAAAojBYsAAAohRasslABAQAACtO8KiBV8i2AivI5vPLV11U6gqbNOQwfWfNKQAAAYAVZBas8pPEAAEBhJCAAAEBhtGABAEAp6vVglYMKCAAAUBgJCAAAUBgtWAAAUAKrYJWHCggAAFAYCQgAAFAYLVgAAFAKLVhloQICAAAURgICAAAURgsWAACUoKqu0hE0DSogAABAYSQgAABAYbRgAQBAKayCVRYqIAAAQGEkIAAAQGG0YAEAQAmqtGCVhQoIAABQGAkIAABQGC1YAABQino9WOWgAgIAABRGAgIAABRGCxYAAJTAKljloQICAAAURgICAAAURgsWAACUQgtWWaiAAAAAhZGAAAAAhdGCBQAAJbAKVnmogAAAAIWRgAAAAIXRggUAAKWo14NVDiogAABAYSQgAABAYbRgAQBACayCVR4qIAAAQGEkIAAAQGG0YAEAQCm0YJWFCggAAFAYCQgAAFAYCQgAAJSgqn7VfHwUdXV1ufzyy7P33nunW7duGThwYKZOnbrM7Z988skMGDAg3bt3z+67755zzjkns2fPXq45JSAAANBMjRgxIqNHj87QoUMzZsyYVFVVZdCgQamtrV1i25kzZ+Yb3/hGOnfunNtuuy0jRozIo48+mtNPP3255pSAAABAM1RbW5tRo0Zl8ODB6dWrV2pqajJ8+PDMmDEj48aNW2L7f/3rX9l7771z7rnnZvPNN88nPvGJfPnLX85f//rX5ZrXKlgAAFCKuqa1DNbkyZMzd+7c7L777g1j7du3T9euXTNx4sQccMABjbbv3r17unfv3vD7P//5z9x2223Za6+9lmteCQgAAKzGevfu/YHP33///Usdnz59epJko402ajS+wQYbZNq0aR/4mp/5zGfy4osvplOnThkxYsRyRKsFCwAAmqX58+cnSaqrqxuNt27dOgsWLPjAfYcNG5Ybbrgh66+/fg4//PDMnTu35HlVQAAAoBSraAfWsiocH6ZNmzZJFl8L8v7PSbJgwYKsueaaH7jvjjvumCS54oor0qtXr4wbNy4HH3xwSfOqgAAAQDP0fuvVzJkzG43PnDkzHTt2XGL75557Lg8++GCjsQ022CDrrLNOZsyYUfK8EhAAAGiGampq0rZt20yYMKFhbPbs2Zk0aVJ69OixxPZ/+tOfcsIJJ2TOnDkNYy+99FLeeuutdOnSpeR5JSAAAFCCSt9wsNw3Iqyurk7//v0zbNiw3H///Zk8eXJOOumkdOzYMX369MmiRYvy2muv5d13302SHHTQQWnXrl1OPfXUPPvss3n44YczZMiQ7LTTTtl3331LnreiCcjNN9+8xE1O/va3v+Xoo4/OF77whXz729/OP//5zwpFBwAATduQIUPSt2/fnH322enXr19atGiRkSNHprq6OtOmTUvPnj0zduzYJMm6666bX/ziF6mrq0u/fv1y/PHHp2vXrhk5cmRatGhR8pxV9fX1FbucZrvttsv48eOz3nrrJUnGjx+fQYMGZa+99so222yTJ598Mo8//nh+9rOf5ROf+MRHnq9Pi69+5NcAgFVafV2lI2jaqjSPrGzjFo2pdAjL9KnP/qjSISzV/919WqVDWC4VXQXrf3OfESNG5PDDD8+ZZ57ZMHbRRRdl2LBhufHGG4sODwAA/qNy39s3KatUGj916tQcdNBBjca++tWvZtKkSRWKCAAAKKeKJiBVVVWNft98880zb968RmNvvfVW2rVrV2RYAADASlLxFqzevXtniy22SJcuXVJdXZ2LL744N9xwQ1q1apVHH3005513Xnr16lXJMAEA4COtOMV/VDQBeeCBBzJlypQ888wzmTJlSl577bW8+OKLWbRoUVq1apUjjzwy2267bb797W9XMkwAAKBMKpqAbLzxxtl4440brRu8cOHCtGrVKkkyevTobLPNNku0agEAAKuniiYgS/N+8pEk2267bQUjAQCA/6IFqyxWqVWwAACApk0CAgAAFGaVa8ECAIBVUZUbEZaFCggAAFAYCQgAAFAYLVgAAFCKukoH0DSogAAAAIWRgAAAAIXRggUAACWwClZ5qIAAAACFkYAAAACF0YIFAACl0IFVFiogAABAYSQgAABAYbRgAQBAKayCVRYqIAAAQGEkIAAAQGG0YAEAQAmqdGCVhQoIAABQGAkIAABQGC1YAABQCqtglYUKCAAAUBgJCAAAUBgtWAAAUIKqukpH0DSogAAAAIWRgAAAAIXRggUAAKWwClZZqIAAAACFkYAAAACF0YIFAACl0IFVFiogAABAYSQgAABAYbRgAQBACaqsglUWKiAAAEBhJCAAAEBhtGABAEAptGCVhQoIAABQGAkIAABQGC1YAABQirpKB9A0qIAAAACFkYAAAACF0YIFAAAlcCPC8lABAQAACiMBAQAACqMFCwAASqEFqyxUQAAAgMJIQAAAgMJowQIAgFJowSoLCQgANCVVmhuAVZtPKQAAoDAqIAAAUIq6SgfQNKiAAAAAhZGAAAAAhdGCBQAAJaiyClZZqIAAAACFkYAAAACF0YIFAACl0IJVFiogAABAYSQgAABAYbRgAQBAKbRglYUKCAAAUBgJCAAAUBgtWAAAUAotWGWhAgIAABRGAgIAABRGCxYAAJSirtIBNA0qIAAAQGEkIAAA0EzV1dXl8ssvz957751u3bpl4MCBmTp16jK3f/bZZ3P00Udnt912yx577JEhQ4bk1VdfXa45JSAAAFCCqvr6VfLxUYwYMSKjR4/O0KFDM2bMmFRVVWXQoEGpra1dYtu33nor3/jGN7L22mvnhhtuyLXXXpu33norRx11VBYsWFDynBIQAABohmprazNq1KgMHjw4vXr1Sk1NTYYPH54ZM2Zk3LhxS2x/3333Zf78+fnBD36QrbfeOjvssEMuvvjiPPfcc3n00UdLnlcCAgAAzdDkyZMzd+7c7L777g1j7du3T9euXTNx4sQltt9jjz1y5ZVXpnXr1ks8N2vWrJLntQoWAACUYhW9EWHv3r0/8Pn7779/qePTp09Pkmy00UaNxjfYYINMmzZtie032WSTbLLJJo3Grr766rRu3Tq77rpryfGqgAAAQDM0f/78JEl1dXWj8datW5d0TccvfvGL3HjjjTn55JOz3nrrlTyvCggAAKzGllXh+DBt2rRJsvhakPd/TpIFCxZkzTXXXOZ+9fX1ueyyy/LTn/40xxxzTI444ojlmlcCAgAApahbNVuwVtT7rVczZ85M586dG8ZnzpyZmpqape6zcOHCnHnmmbnzzjtz2mmn5cgjj1zuebVgAQBAM1RTU5O2bdtmwoQJDWOzZ8/OpEmT0qNHj6Xuc9ppp+X3v/99LrnkkhVKPhIVEAAAaJaqq6vTv3//DBs2LB06dEinTp1y8cUXp2PHjunTp08WLVqUN998M+3atUubNm1y6623ZuzYsTnttNPyyU9+Mq+99lrDa72/TSlUQAAAoBT19avm4yMYMmRI+vbtm7PPPjv9+vVLixYtMnLkyFRXV2fatGnp2bNnxo4dmyS58847kyQ/+tGP0rNnz0aP97cpRVV9/Sq6nthK0KfFVysdAgAAH2DcojGVDmGZPrvN6ZUOYanufuaHlQ5huaiAAAAAhXENCAAAlKL5NA6tVCogAABAYSQgAABAYbRgAQBAKbRglYUKCAAAUBgJCAAAUBgtWAAAUIo6LVjloAICAAAURgICAAAURgsWAACUor6u0hE0CSogAABAYSQgAABAYbRgAQBAKdyIsCxUQAAAgMJIQAAAgMJowQIAgFK4EWFZqIAAAACFkYAAAACF0YIFAAClsApWWaiAAAAAhZGAAAAAhal4C9bjjz+eCRMm5Oijj06S/O1vf8vPf/7zvPLKK+ncuXMGDhyYHj16VDhKAACaPS1YZVHRCsjvf//79OvXLw899FCS5A9/+EO+8Y1vpL6+Pr169crChQszYMCA/OEPf6hkmAAAQJlUtALyk5/8JN/61rdy3HHHJUl++tOf5thjj80JJ5zQsM1Pf/rTXH755dl3330rFSYAAFAmFa2AvPTSS/n85z/f8Psrr7ySz3zmM422OfDAA/Pcc88VHRoAADRWX79qPlYzFU1ANt100zz44IMNv2+33XaZPHlyo23+8Y9/ZMMNNyw6NAAAYCWoaAvWoEGD8p3vfCfTp0/PgQcemOOOOy5nnHFGFixYkK233jqPP/54rrzyynzrW9+qZJgAAECZVDQBOfjgg1NVVZXLL7881113XaqqqlJfX59zzz03SbL22mvnqKOOyhFHHFHJMAEAIKmrq3QETULFl+E96KCDctBBB+X555/Piy++mDlz5qRVq1bp2LFjunbtmtatW1c6RAAAoEwqnoC8b8stt8yWW25Z6TAAAICVaJVJQAAAYJW2Gq44tSqq6CpYAABA8yIBAQAACqMFCwAASqEFqyxUQAAAgMJIQAAAgMJowQIAgFLUacEqBxUQAACgMBIQAACgMFqwAACgBPX1dZUOoUlQAQEAAAojAQEAAAqjBQsAAEphFayyUAEBAAAKIwEBAAAKowULAABKUa8FqxxUQAAAgMJIQAAAgMJowQIAgFLUuRFhOaiAAAAAhZGAAAAAhdGCBQAApbAKVlmogAAAAIWRgAAAAIXRggUAACWotwpWWaiAAAAAhZGAAAAAhdGCBQAApbAKVlmogAAAAIWRgAAAAIXRggUAAKWo04JVDiogAABAYSQgAABAYbRgAQBAKerdiLAcVEAAAIDCSEAAAIDCaMECAIAS1FsFqyxUQAAAgMJIQAAAgMJowQIAgFJYBassVEAAAIDCSEAAAIDCSEAAAKAE9XX1q+Tjo6irq8vll1+evffeO926dcvAgQMzderUkvY78sgjc8UVVyz3nBIQAABopkaMGJHRo0dn6NChGTNmTKqqqjJo0KDU1tYuc5933303p556asaPH79Cc0pAAACgGaqtrc2oUaMyePDg9OrVKzU1NRk+fHhmzJiRcePGLXWfRx99NF/84hfz+OOPp3379is0rwQEAABKUV+3aj5W0OTJkzN37tzsvvvuDWPt27dP165dM3HixKXu86c//Sl9+vTJ7bffnnbt2q3QvJbhBQCAZmj69OlJko022qjR+AYbbJBp06YtdZ8TTjjhI88rAQEAgNVY7969P/D5+++/f6nj8+fPT5JUV1c3Gm/dunVmzZpVnuCWolklIOMWjal0CAAArKbG1d1U6RCW6sMSkGVp06ZNksXXgrz/c5IsWLAga665ZlliW5pmlYAAAEBTs6wKx4d5v/Vq5syZ6dy5c8P4zJkzU1NTU5bYlsZF6AAA0AzV1NSkbdu2mTBhQsPY7NmzM2nSpPTo0WOlzasCAgAAzVB1dXX69++fYcOGpUOHDunUqVMuvvjidOzYMX369MmiRYvy5ptvpl27do1atD4qFRAAAGimhgwZkr59++bss89Ov3790qJFi4wcOTLV1dWZNm1aevbsmbFjx5Z1zqr6+vqPdv92AACAEqmAAAAAhZGAAAAAhZGAAAAAhZGAAAAAhZGAAAAAhZGAAAAAhZGAAAAAhZGAAAAAhZGArGLq6upy+eWXZ++99063bt0ycODATJ06tdJhNVkjRozIYYcdVukwmpS3334755xzTvbZZ5984hOfSL9+/fLwww9XOqwm5Y033sipp56a3XffPd27d8/RRx+df/7zn5UOq0l64YUX0r1799x6662VDqVJ+de//pVtt912icdNN91U6dCalNtvvz2f+9znsuOOO+aAAw7I3XffXemQIIkEZJUzYsSIjB49OkOHDs2YMWNSVVWVQYMGpba2ttKhNTk///nPc/nll1c6jCbn5JNPzuOPP55LL700N998c7bffvsceeSRee655yodWpPxzW9+My+//HKuvfba3HzzzWnTpk2OOOKIzJ8/v9KhNSkLFy7MKaecknnz5lU6lCZnypQpad26df70pz9l/PjxDY/Pf/7zlQ6tyfjtb3+bs846K1/96ldz55135nOf+1xOPvnkPPbYY5UODSQgq5La2tqMGjUqgwcPTq9evVJTU5Phw4dnxowZGTduXKXDazJmzJiRo446Kpdddlm22GKLSofTpEydOjV//vOfc+6556ZHjx7Zcsst853vfCcbbrhh7rzzzkqH1yS89dZb2WSTTXLBBRdkxx13TJcuXXLcccfltddey7PPPlvp8JqUK664ImuvvXalw2iSnnnmmWyxxRbZYIMNsv766zc82rRpU+nQmoT6+vpcdtllGTBgQAYMGJDNNtssxx9/fPbcc8889NBDlQ4PJCCrksmTJ2fu3LnZfffdG8bat2+frl27ZuLEiRWMrGl56qmnss466+R3v/tdunXrVulwmpR1110311xzTXbYYYeGsaqqqtTX12fWrFkVjKzpWHfddXPppZdm6623TpK8/vrrGTlyZDp27JitttqqwtE1HRMnTsyYMWPywx/+sNKhNElTpkxxvq5Ezz//fP71r38tUVEaOXJkjjnmmApFBf/RstIB8B/Tp09Pkmy00UaNxjfYYINMmzatEiE1SZ/+9Kfz6U9/utJhNEnt27dPr169Go3dfffdeemll9KzZ88KRdV0ffe7381vfvObVFdX56c//WnWWmutSofUJMyePTunnXZazj777CU+jymPZ555Juuvv36+9rWv5cUXX8xmm22W4447LnvvvXelQ2sSXnzxxSTJvHnzcuSRR2bSpEnZZJNN8s1vftO/f6wSVEBWIe/3b1dXVzcab926dRYsWFCJkOAjeeSRR3LWWWeld+/e/tFbCQYMGJBbbrklX/jCF3L88cfnqaeeqnRITcL3vve97Lzzzq5HWElqa2vz4osvZs6cOTnxxBNzzTXXZMcdd8ygQYPy17/+tdLhNQlz5sxJkpx++uk58MADM2rUqOy111457rjjHGNWCSogq5D3e19ra2sb9cEuWLAga665ZqXCghVy33335ZRTTkm3bt1y6aWXVjqcJun9FpYLLrggf//733PDDTfkoosuqnBUq7fbb789Dz/8cO64445Kh9JkVVdXZ+LEiWnZsmXDF2477LBDnnvuuYwcOTJ77LFHhSNc/bVq1SpJcuSRR+aLX/xikmS77bbLpEmT8rOf/cwxpuJUQFYh75f6Z86c2Wh85syZ6dixYyVCghVyww03ZPDgwdlnn31y7bXXurC0jN54443ceeedWbRoUcPYGmuskS5duizx2cHyu+WWW/LGG2/kU5/6VLp3757u3bsnSc4999wccMABFY6u6VhrrbWWqPZvs802mTFjRoUialre/5thm222aTS+1VZb5ZVXXqlESNCIBGQVUlNTk7Zt22bChAkNY7Nnz86kSZPSo0ePCkYGpbvxxhtzwQUX5Otf/3p+/OMfL/FHBh/NzJkz8+1vf7vRSjYLFy7MpEmT0qVLlwpG1jQMGzYsY8eOze23397wSJIhQ4bkmmuuqWxwTcTkyZPTvXv3Je4P9OSTT7owvUy6du2atddeO48//nij8WeeeSadO3euUFTwH1qwViHV1dXp379/hg0blg4dOqRTp065+OKL07Fjx/Tp06fS4cGHeuGFF/L9738/ffr0yTHHHJM33nij4bk2bdqkXbt2FYyuaaipqUnPnj1z3nnnZejQoWnfvn2uuuqqzJ49O0cccUSlw1vtbbjhhksdX2+99dKpU6eCo2mattlmm2y99dY577zzcu6552bdddfNb37zm/z973/PzTffXOnwmoQ2bdrkqKOOypVXXpkNN9wwO+20U+666678+c9/zs9//vNKhwcSkFXNkCFD8t577+Xss8/Ou+++m1133TUjR470LTKrhXvuuScLFy7MuHHjlrh3zRe/+MX84Ac/qFBkTUdVVVV+/OMf55JLLsmJJ56Yd955Jz169MivfvWrbLzxxpUODz7UGmuskauuuirDhg3LiSeemNmzZ6dr16752c9+lm233bbS4TUZxx13XNZcc82G+4l16dIlV1xxRXbbbbdKhwapqq+vr690EAAAQPPgGhAAAKAwEhAAAKAwEhAAAKAwEhAAAKAwEhAAAKAwEhAAAKAwEhAAAKAwEhAAAKAwEhCA1cyCBQvStWvXdO/ePRdccEGlwwGA5SIBAVjNVFVV5frrr89OO+2UG264IS+88EKlQwKAkklAAFYz1dXV2XXXXXPUUUclSZ566qkKRwQApZOAAKymttxyyyTJ008/XeFIAKB0EhCA1dS1116bJJk8eXKFIwGA0klAAFZD48ePz69//euss846mTRpUqXDAYCSSUAAVjOzZ8/OWWedld69e6dfv3558803M2PGjEqHBQAlkYAArGbOO++8vPfeexk6dGi6du2aRBsWAKsPCQjAauT3v/997rzzzlx44YXp0KFDQwLiQnQAVhcSEIDVxGuvvZZzzz03X/3qV7PvvvsmSTbddNO0b9/edSAArDYkIACrie9+97tZZ511csYZZzQa32677bRgAbDakIAArAZuuumm/PGPf8yPfvSjrLXWWo2e69q1a1566aXMmTOnQtEBQOmq6uvr6ysdBAAA0DyogAAAAIWRgAAAAIWRgAAAAIWRgAAAAIWRgAAAAIWRgAAAAIWRgAAAAIWRgAAAAIWRgAAAAIWRgAAAAIWRgAAAAIWRgAAAAIX5/9Dk+djdKdGNAAAAAElFTkSuQmCC",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "filenames": {
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_258_1.png"
+ }
+ },
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyAAAANbCAYAAAC6lftqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABfe0lEQVR4nO3deVxU9f7H8feIDqCISVfFfUETcSELt9TIjJtli5X3Fmm5a2mSe5tpmmWmZrllGrSZQbmWaUl2f6YtilpuhJopLiG4Iy6AzPz+ILkXAT3oeA4Mr+fjMY+HfOec+X44nWb4zOdzvsfmdDqdAgAAAAATlLI6AAAAAAAlBwkIAAAAANOQgAAAAAAwDQkIAAAAANOQgAAAAAAwDQkIAAAAANOQgAAAAAAwDQkIAAAAANOQgADAdcS9XgEAyK201QEAgJmef/55LVmy5LLbVK9eXd9///01z/XFF19oz549ev7556+47cKFC/XSSy+pffv2ev/99695bgAAiiqbk6/nAJQg+/fv1/Hjx3N+nj17tuLj4zVz5sycMbvdrqCgoGue684771TLli31xhtvXHHb8PBwpaWl6Y8//tCqVatUs2bNa54fAICiiAoIgBKlVq1aqlWrVs7Pfn5+stvtuvnmmy2Lae/evdq8ebPmzp2rkSNH6vPPP9fw4cMtiwcAgOuJa0AAIB+7du3SgAEDdMstt+iWW27RoEGDdODAgVzbfPLJJ+rUqZOaNm2q9u3b65VXXlFaWpqk7OrHoUOHtGTJEjVs2FAHDx4scK5FixapfPnyatOmjTp16qRFixYpIyMjz3bbt29X3759deutt6p169YaOnSokpKScp4/duyYXnzxRd12221q3ry5unXrpk2bNuU837BhQ82YMSPXa86YMUMNGzbM+fn5559Xjx49NHbsWIWEhOihhx7ShQsXdPz4cY0bN04dOnRQkyZN1LJlSw0aNCjP7/X111/r4YcfVnBwsO644w5NnjxZGRkZ2r17txo2bKiYmJhc2ycnJ6tRo0ZXbIsDALgPEhAAuMTevXv12GOP6dixY3rjjTf02muv6cCBAwoPD9exY8ckZf+hPWnSJHXr1k2RkZEaNGiQli1bpgkTJkiSZs6cqUqVKik0NFQxMTGqXLlyvnNlZWVp2bJluvfee2W32/Xwww/r2LFj+u6773Jtl5CQoPDwcJ07d05vvPGGxo8fr/j4ePXu3VuZmZk6e/asHnvsMf30008aPny4Zs6cqXLlyqlv377as2dPoX7/jRs3KjExUTNmzNCgQYPk4eGhAQMG6Mcff9Tw4cMVGRmpgQMH6qefftKYMWNy9ouOjtawYcPUqFEjzZw5UwMGDNCCBQv0yiuvqEGDBgoODtayZctyzbVs2TJ5eXnp7rvvLlSMAIDiixYsALjEzJkz5eXlpQ8//FA+Pj6SpDZt2uiuu+7S+++/r+eee07r169X9erV1a1bN5UqVUotW7ZU2bJldeLECUlSUFCQ7Ha7/Pz8Ltve9cMPPyglJUWPPPKIJOnmm29W/fr19dlnn+nee+/N2W727NmqUKGCoqKi5OnpKUny9/fXkCFDtHPnTm3ZskUHDhzQ0qVLFRgYKEkKCQlRly5dFBcXp4CAAMO//4ULFzRu3DjVrl1bUnaVwtvbW88995xCQkIkSa1atdLBgwcVHR0tSXI4HJoxY4bCwsL02muv5bxWenq6lixZooyMDD3yyCMaM2aMDhw4kHONy9KlS3XPPfeobNmyhuMDABRvVEAA4BK//PKLWrVqJS8vL124cEEXLlyQj4+PQkJC9NNPP0mSWrdurX379unhhx/OuZD9/vvvV48ePQo116JFi1S7dm3VrVtXqampSk1N1T333KMNGzbkqlxs2rRJt99+e07yIUnNmjXT999/ryZNmmjjxo2qUaNGTvIhSZ6enlq5cqUee+yxQsXk5eWV6zqZKlWq6OOPP1ZISIj++usv/fzzz5o/f742b96szMxMSdlVo6NHj+quu+7K9Vo9e/bUsmXLZLfb1blzZ3l7e+dUQbZu3ao9e/bo4YcfLlR8AIDijQoIAFzi5MmTWrFihVasWJHnOT8/P0nSvffeK4fDoQULFmjmzJl65513VL16dQ0fPlydO3c2NM/x48f1f//3f8rMzFSLFi3yPB8TE6MXX3wxJ6Ybb7zxsjFf7vnCuPHGG2Wz2XKNffnll3rrrbeUlJSkG264QYGBgfLy8so1/8V9C+Lj46NOnTrpyy+/1DPPPKMlS5aodu3aOVUVAEDJQAICAJcoX768brvtNvXq1SvPc6VL//dt87777tN9992n06dPa926dZo3b55GjhypkJAQValS5YrzLFu2TJmZmZo5c6Z8fX1zPTdr1iwtXbpUw4YNk5eXl8qXL59r+eCL1qxZo8DAQJUvXz7fC91//fVX+fj4qEGDBpKyrzn5X2fPnr1inBs3btRzzz2n7t27q0+fPvL395ckvfnmmzkXuV+M/9IYT548qR07dujmm29WuXLl9Mgjj2jJkiXaunWrvv32Wz3xxBNXnB8A4F5owQKAS7Rs2VJ//PGHGjVqpKZNm6pp06Zq0qSJPvzwQ8XGxkqShgwZomeeeUZSdsJyzz33aODAgcrKylJKSookqVSpy7/FLl68WDfffLPCwsLUqlWrXI/w8HCdOnVKK1eulJR9PcfatWtzrY61c+dO9e/fX9u2bVNISIgOHDignTt35jyfkZGhwYMH6/PPP5eUXYE4fPhwrhg2b958xePx66+/yuFwKCIiIif5yMrKymlHczgcqlevnipWrKjVq1fn2verr75Sv379lJ6eLklq0aKF6tSpo8mTJ+vEiRPq0qXLFecHALgXEhAAuMTAgQO1f/9+DRgwQN99953Wrl2rwYMH6+uvv865xqJ169aKjY3VpEmT9PPPP+vbb7/VO++8ozp16uRs4+vrq/j4eG3YsEHnz5/PNcfWrVu1a9euAtu1OnbsqAoVKuRc5D1w4ECdOHFC/fr10/fff69vvvlGQ4YMUePGjXX77bfr4YcfVs2aNfX0009r2bJlWrt2rSIiInT+/PmcKsMdd9yhr7/+WgsWLNDPP/+sUaNGKTEx8YrHo1mzZpKk8ePH65dfftGqVavUq1cvJSQkSMquonh4eGjw4MH69ttv9corr+jHH3/Up59+qrffflvh4eE5rWuS9Mgjj2jDhg1q06aNqlatWpj/NAAAN0ACAgCXCAwM1KeffiqbzaZRo0YpIiJCR44c0axZs/TPf/5TkvTYY49p9OjR+uGHH/TUU09pzJgxCggIUFRUlMqUKSNJ6t27t44ePao+ffpo+/btueZYtGiRPDw8cq109b/sdrvuuece/fbbb/r9998VFBSkTz75RA6HQ0OHDtX48eN18803a968ebLb7fLx8dH8+fPVvHlzvfbaa3r22WeVnp6uTz75JOeC8hdeeEF33nmnJk+erIiICHl7exu64WGrVq00ZswY/frrr+rXr58mTpyoatWq5dw9/mIbVrdu3fTGG29o48aNGjBggKKiotS7d289//zzuV7vjjvukCQuPgeAEsrmdDqdVgcBACg55s2bp/fff19r166V3W63OhwAgMm4CB0AYIolS5Zo165dWrBggfr370/yAQAlFAkIAMAUCQkJio6O1l133aV+/fpZHQ4AwCK0YAEAAAAwDRehAwAAADANCQgAAAAA05CAAAAAADANCQgAAAAA05SoVbA6+bHqyvXkDKhhdQhuz7b3kNUhuDVb2bJWh+D+fDjG111GptURuDVHBc7h6+3bTeOsDqFAjsM3WR1Cvkr577I6hEKhAgIAAADANCQgAAAAAExTolqwAAAAgKvlkMPqEPJV3CoKxS1eAAAAAMUYCQgAAAAA09CCBQAAABiQ5SyaLVjF7Q96KiAAAAAATEMCAgAAAMA0xa1iAwAAAFjCIafVIbgFKiAAAAAATEMCAgAAAMA0tGABAAAABhTVGxEWN1RAAAAAAJiGBAQAAACAaWjBAgAAAAzIcrIKlitQAQEAAABgGhIQAAAAAKahBQsAAAAwgBsRugYVEAAAAACmIQEBAAAAYBpasAAAAAADsmjBcgkqIAAAAABMQwICAAAAwDS0YAEAAAAGsAqWa1ABAQAAAGAaEhAAAAAApqEFCwAAADAgy0kLlitQAQEAAABgGhIQAAAAAKahBQsAAAAwwGF1AG6CCggAAAAA05CAAAAAADANLVgAAACAAVnciNAlqIAAAAAAMA0JCAAAAADT0IIFAAAAGJBFB5ZLUAEBAAAAYBoSEAAAAACmoQULAAAAMIAbEboGFRAAAAAApiEBAQAAAGAaWrAAAAAAA7JkszoEt0AFBAAAAIBpSEAAAACAEsrhcGj69Olq3769goOD1bt3byUmJha4/YEDB/TUU0+pZcuWatu2rSZMmKBz584Vak4SEAAAAMAAh7NoPq7F7NmzFR0drQkTJigmJkY2m039+vVTRkZGnm1Pnz6t8PBwnTp1Su+//77mzJmj7du3a9CgQYWakwQEAAAAKIEyMjIUFRWlwYMHKzQ0VIGBgZo2bZqSk5MVGxubZ/slS5YoLS1Ns2bNUrNmzdS0aVNNmzZNP/30kzZu3Gh4XhIQAAAAoARKSEjQmTNn1Lp165wxX19fBQUFKS4uLs/2e/fuVb169eTn55czVrVqVVWsWFEbNmwwPC+rYJnk1o6N1ePFLqrVsKpOHUvTig/WKObtlYb2rR9cS2+vekF9QkYr+cCxXM+1vf8W/Tuik2rc5K+zqef025oERY5bqJNHTl+PX6PICmkToB5P3ala9Srp1Ikz+nrxJsV8uK7A7UuXLqVHut+msM7BqlTFV0dSUvWfb7Yp5sN1unAh722Gypaz690FT2v+vP9T7PIt1/NXKbJuvbOxerz4oGrdVE2njp3Wig/XKOadbwztWz+4lt7+5gX1aZnPOXzfLfp3xN2q0eDvc/iHBEWOX1TizuFbQwP15Mh7VauBf/Z7xKc/6fNZ3xnat37TGpq2dKj6hL6mlIPHcz336cbx8qvsm2efx299WSdK2jFu10BPRoSpVkBlnTpxRitiNujzeWsM7Vs/qJqmffa0+twzVSl/ncz1XJuOjRT+1J2qUfcfOnE0Tau//FWfz1ujC5lZ1+G3KLpuvb2hnhx6t2rVr6JTx89oxWc/6/M5/zG0b/3G1TVt4WD1uWuSUg6dyPVcjXqV1Oe5zmrWKkAXMrO0PW6v5k38SocPHC/g1UqWkDb11WPgxc+/s/p60UbFfLC2wO1Ll/bQI0/cprD7glWpSgUdSU7Vf77ZqpgP1unChZJ1zl6NoroKVseOHS/7/OrVq/MdP3z4sKTsJOJ/Va5cWUlJSXm2r1Spko4cOaKsrCx5eHhIktLS0nTq1CkdO3Ysz/YFIQExQaOWAXrl02f0w5I4ffT6UjVp1UA9RneRrZRN0W+tuOy+dRvX0PjoCJUuk/c/VfsHb9VLHzylrz9Yo49eX6qKlXz1xAsPatKyEXqmw6vKTL9wvX6lIiWoWQ29MjVca2K366M536txcC31fPpOlbLZ9FkBb8JPDe+ksM7B+jTyB+2K/0v1G1bVE/1DVdn/Bk2b8GWubcv7eumVqeHyr3aDCb9N0dSoRT29Mn+Qfli6UR+9vkxNWtVXj5e6yFaqlKKnGTiHFwxW6TIeeZ5r/8CteilqgL7+cI0+en1Z9jn8/AOatGS4nuk4ocScw41uraOxkX31w/Jf9fHkFWrcop56jLxXpWw2Rc/MWwL/X3UbVdO4D/rne3xv+IeP/Cr76r1xS5SweV+u51JPnHHlr1DkNbq5lsbOekI/rNymj6fHqvEtddTj2TCVKmVT9Hv/d9l96zb017g5PfI9xs3b1Nfod7rph5Xb9MFb36rOTVXUc8g/VaFiOb372lfX6bcpeho1r62xc3rqhxVb9PG0b9X41jrqMaxT9jn87veX3bduYFWNe79Pvsf3H1UraGrMIB3ce0SThi6Q3auMegy9W6992E9P3ztVGSXkPaIgQc1q6pVp4Vqzaoc+mv29Gt9cSz0H/v35F/VDvvs8NaKTwu67WZ++v0a7dhxS/cBqemLAHdmff68uM/k3gNUuXjxut9tzjXt6eurUqVN5tu/cubPmzJmj119/XcOGDVNWVpbGjRsnm82W7zUjBSEBMUH3Uffrz20HNPnpKEnSptU75FHGQ/9+9h4tnh2rjPOZefYpXcZDD/S/U0++8GC+z0vS4yPu04ZVWzVj+PycsQO7D2v66pfU6u5mWvfl5uvzCxUx3fqG6s9dhzV57FJJ0saf96h0aQ/9u0dbLVrwc54PqPK+Xur8cIgiZ3ynhfN/kiT9FrdXktTv2TBFzfxOp06elSS1ub2hnh7RSd7euf/HLGm6j7xff24/oMkD/z6Hv794DnfS4ncvcw73u1NPPv+gMs7n/6b0+PDO2hC7TTNGfJozduCPJE2PfUmt/tlM674qIefwkE76M/6QpgzJPg6b1iSodBkP/WtgRy2e93/KSC/g+PZsrydG3Fvge0RA4xqSpJ++2ZrnW+WSptvAO/Xn70ma8vwXkqRN63ardOlS+lffUC3+cF2+f8iWLuOhB7q10ROD7yrwD92wh27VkaRTmvzc53I4nPr15z90g5+PHurRVnMnfa2sfCqq7qhbRJj+/P0vTRkRLUna9MNOlS7toX8N6KDFUT8UfHyfbKsnhtxd4Dn8RMQ/de5Mul58cq7S/94m+cBxjX2vpxo0rakdG/dev1+qGOjW/w79ufOwJo9ZLEna+PMf2Z9/Pdtp0ac/5fP5563Oj4Qocvp3WvjJj5L+5/NvyD8VNSM25/MPxUtBFY4r8fLykpR9LcjFf0tSenq6vL2982xfu3ZtzZgxQ2PGjNGnn34qLy8vPfHEE2rSpIl8fHwMz8s1INdZGXtpNW17k35cnvsPqXVfblLZ8l5q0qZBvvu1CGuqbqPuV/RbKxQ1blGe5202mzb/X7xWfJT7G46DfyRLkqrWqeyi36BoK1PGQ81uraN1//k91/ja7+NVtpynmtxcK88+ZX289PXijfpl7c5c44f2Z5cO/atXlCSV8/HUy2/+W1s37dOLEfPzvE5J8d9z+Ndc4+u+2qSyPl5q0rqAc/iupuo28j5FT1uhqPGL8zxvs9m0eU28Vnx86TmcIkmqWqeSi36Doq2M3UPNWtfXj99szTW+7uvfso9vy3r57teiQ5C6DemkmBmxipqY/zft9RpX1+lTZ0t88lGmjIeataynH7/bkWt83art2e8Tt9bJd78Wt9+kbgPvVMzc/1PU1PzbDcvYPXT+XIYc/7MMTerJMypjLy3vcp4u+x2KsjJ2DzVrFaAfV23PNb7um63Z53CLAs7h0EB1GxymmNmrFTU5/0rqbXc31bdfxOUkH5K0e/tBdW87ocQnHwV+/q3ekX1eN6+dZ5+yPp76etFG/fJDQq7xQ/uPSpL8a1S8fgG7iSzZiuTjal1svUpJSck1npKSIn9//3z3CQ0N1Zo1a7R27Vr98ssvGj58uA4cOKA6deoYnpcE5Drzr/MP2T3L6NCe5Fzjf/2Z/R+6ekCVfPfb9es+9Qh+XtFvrcj3GzSn06l5L3+hX1bmvh6h7f3NJUmJvx9yRfhFnn/1irLbS+ckDxf99XdvcI1aN+bZJ/mvk5o5aYUOJl5yLUKHQGVmZuW8Vvr5TPV/dLamjFum1BL8jZB/7YLO4SOSrnAON39B0dNWKCufvmKn06l5YxbmPYfv+/scTvjLFeEXef61/qEynqV16O/jedFfidl/EFSvl38itmvLfvW4bbyiZ8YW+C17QFB1pZ06q9Hv9dLC7RO1+PdJen7mk6qYzzUh7sy/pp/K2Evr0L5L3if+/n+9ep1/5Lvfrm2H1CNssqLf+z9lZeV/jL9a8Iuq1b5Rj/Rur3LlvRTYrKa6PNFWG9YkKO1U4dbFL678a96YfXz3XnoO/3186xZ0fA+oR+jrin73+3zP4So1KsrH11vJh45r4CsPKSbuFS3b8bpemdtLlare4PLfo7jJ+fxLLOTn3xtf5/38uzNImZkX8rwW3F9gYKB8fHy0fv36nLHU1FTFx8crJCQkz/abNm1S9+7dlZGRoUqVKsnLy0sbNmzQiRMndNtttxme1/IWrAsXLmjVqlXauHGj/vrrL2VkZMjb21v+/v4KCQlRWFiYSpe2PMyr5lOhrCTp7OnzucbPpmX/XLZ83vKWJB1LOlnouarVq6y+47pq92+Jivtu+5V3cAM+5bPLhWfPpOcaP3s2++eyBr+BbNehkTreG6yl0euV9vd/qwsXHHnepEuiK5/DXnn2kaRjh08Weq5q9Sqr7ytdtXtLyTmHy/lmvwdcPJ4XnU37+xz2KeD4Juftzb1UvaDq+of/Dfrms1+0JHKNatWvou7D7tGbnz+jZ+6ZovRzxvt1i7NyOe8TlxzjM9m/f1mf/N8njqWkXvG1t274Uwsj16rviHvUd8Q9kqQ/4g9p0siYawm5WPnvOXzJ+/CZK53Dlz++Ffyy2zl6j7xXO7ce0KShC1ThRh/1GnGPJn36lJ7uPFXp5/Jv3SoJCv78+/u8Nvr5d2cjdby3mZZ+9t/PP5Qcdrtd3bt315QpU+Tn56fq1atr8uTJ8vf3V1hYmLKysnT8+HGVL19eXl5eCggI0O7du/X666+rT58+OnDggEaNGqXHHntMNWvWNDyvpX/Z79+/X/369VNycrKCgoJUuXJlVahQQenp6fr999+1aNEizZgxQ++//76qVatmZahXzVYquyzmdOZ/lxinwzX9wTVv8tfri4cpM/2CJvScU+B87sZmu3h883/eYeA4tLuzkZ579WFt27xPUTONrTpUklz5HHbNuVazgb9eXzRUmRmZmtCr5JzDpf4+vgWdxEbO4YK8NeIzZaZnas+O7Irojg1/KnHXYU1d/KzueqSFvp7/41W/dnHy32Oc//OOaziHB7/SRWEP3aIF736v337ZI/8aFdX9mbs0YW4vvdA7MlfrkLsqZbvCOXyVx7fM3xelnzyWpgkDP855T0hKPKppCwfrzgdv1croX67qtd3Bld6bDX3+dQzScxMe0bbNiYqacfkFL5DN4Syaq2Bdi4iICF24cEGjR4/W+fPn1aJFC0VGRsput+vgwYPq2LGjJk6cqIcfflg33HCD5s6dq4kTJ+r+++9XxYoV9dhjj+npp58u1JyWJiDjxo1TjRo1tHDhQpUvXz7P86mpqRo6dKjGjx+vOXPmWBDhtTvzdwn+0krHxW+EzqRee4m+WbuGevnjp3Uu7bxeeHiakv/u5SwJzlz8Fv6Sb3rKls3++dJv5C718OOt1TciTFs379Mrw6OVWcKWzTTizKns9rNLKx055/BpF53DHz6lc2npeqHrNCXvLzmVp7S/3wMu/Zb44rfyZ1Ov/hvJS1e+kqT4jXuVduqc6gYVzy91rsbFb3XzvE+Uy15c4uzpy79PFOTGyr7q1DVEMXPX6JMZ2V9ebIvbq13bDmnOl8/qnw/fqq8WuP8fyGmnCziH/z7el1ZPjbr4zX7cmoRcf2Qn/LZfp0+dVUCjknMO5+fMxfPa59LPv7/P6yt9/nVro77P/lNbN+3TK8M+4/OvBPPw8NDIkSM1cuTIPM/VqFFDO3fmvmY2ODhY0dHR1zSnpQnIpk2bFBMTk2/yIWXfCGXkyJHq1q2byZG5zl97U5R1IUvVLunjrlYv+yLx/TvzrrFcGHc80lLDZ/XSoT3JGv2vd3T0r5J1selfB48r64JD1Wr45RqvVjP758RLepL/18AR9+jBR1tqTex2TR67lDffAvy178jf53DuhQ0untMuOYdn9Mw+h//9jo5eRfthcZaUeFRZF7JU9ZLrEKrVzv55/+7DV/W65Xy91bZTMyX8uk/7d+e+fqe03UOpx0vOMrxJ+49nH+PauXviq/3dI79/T0p+u11RpaoVVKpUKcX/mphrPPGPZJ06cUa16+d/fZS7SUo8lv/x/fvn/X8k57fblV93/zFlZTlUxp73T5XSpT2Uns/qcCXJXwdPZL8317jkuF/8/Puz4PN64Mh79eBjrbRm1XZNHrOYzz+YztKL0H19ffNcdX+pv/76K9eyYMVNZvoFbftpt9red0uu8XYP3KrTJ89o5+arX8WjxV1NNPLd3vp9wx4N6zSpxCUfkpSZkaVtvyaqbYfAXOPt7wzS6dRz2rkj/4vxew3qqAcfbalFn/6s119cxJvvZWSmX9C2n3erbefmucbb3e+ic3hWL/0et0fD7n2zxCUf0t/Hd/0ete0UnGu8XeebdfrUWe38bf9Vve6FjAsaNKGr/jXwrlzjbf7ZVF7edm395Y+rjrm4ycy4oG0b96ntXY1zjbf7ZxOdPnVOO7cduKrXvZjYNL5kFa3qdf6hChXL6fDBknGjvMyMC9oWt1dt726aa7xdp2bZ5/CWqzuHz5/N0I6Ne9X2n01Vxv7fe4Tc3Ka+vMt5akdcyV4FKzPjQvbn352Nco2379j48p9/z9ylBx9rpUXzf9LrL3zB518hWb3alatXwbKKpRWQrl276oUXXlBERIRatWqlqlWrym63KyMjQ8nJydqwYYOmTJmirl27WhnmNfts6teauGSoXvpggL799EcFtQxQ18H/VNS4Rco4n6my5b1Uq2E1Je1N0aljaYZes4xnaQ15p4fOpp3XZ299rVoNc9/B8uhfJ0pMQrIg6ge9MetJvTSxq7796jcFNauprk/cpsgZ3ykj/YLKlrOrVt1KSjp4QqdOnlW9m6ro30+21c74Q/rhux0KbFI91+vt33sk5+JUZPvsra81cdFQvRQ5QN8u+FFBLeqp6zP/VNT4xdnnsI+XajWsqqR9Rwp3Dr/95N/n8ArVuin3cn9H/zpRYhKS6Bmxen3B03rx3Z5aFbNejW6to0cGdFDUxK+UkZ6psj6eqtXAX0mJR3XKYOUi/XymFs75Xo8PuVsnj5zWpjW/q05gNXUf2knrV+/Qb+t2XeffqmiJfu8/ej2yt16cFq5Vizep0c219Ejv9op669u/3yc8VSugspIOHNcpgzdpPHXijJZ+8pO69movSfr1pz9UudoN6jbwTiX/dULfLIy7nr9SkRI96zu9/nF/vTiju1Z9EadGt9TRI/1CFfXmiuzj6+OpWvWrKGn/McPnsCR9MGWl3vz0KY1/v48Wvb9GN/yjvHqPulcJvyXql9U7rvwCbm5B5A96Y/aTemnSv/Xtss0KCq6lrk/epsjp3/33vK5bSUkHj//9+eevf/doq507DumH2B0KbFIj1+tlf/5dXUsiUBiWJiCDBw9WqVKlNGnSJJ09m3eZ03Llyqlbt2569tlnLYjOdbasTdCEHnP0xPMPaMwnA3Us6aTeH7tQi2dlX/BVv1ktvfnVSE0d9IFiP/vJ0GsGtayvG/9ehnDi4mF5np8/6UvNn1Qy7sK7ZeM+vfrc53qi/x0aO/lRHTtyWu9Pj9WiT3+WJNVvWFWT3+upKeOWKnb5FrXr0EilStnUMKi63vmgb57XGzngQ23dnJhnvCTbsnanJvScoyeee0BjPn46+xx+ZZEWz/77HA6upTeXjdDUZz5QbPTPhl4zqGWAbvS/QZI0cdHQPM/Pf/MrzX+zhJzDP+3WawM+UPdh92jMvD46mnxSka99qcXz/k+SFNCkpt78/BlNHbZA3y3cYPh150/7RieOnlbnJ9rq/p7tdPrEWa1Y8JPmF3BPC3e2Zf2feu3ZBer+TEeNmdFdR5NTFTnlGy3+cJ0kKSComt78qJ+mvrhQ3y01fgPM9yev1NHDp3Tvo630SM92On7ktDb/uFsfvROrtGu4fqe42fLLHr026BN1f/afGjOnp44ePqXISV9rcWT2fX4CGlfXm58+ramjYvTd4o2GXzfh10Q9122OegzvpJdmPan0c5n6+bvten/i8mtaPMBdbInbq1dHxeiJAR00dmq4jqWk6v13YrXo75vs1g+sqslze2nKK0sU+9VvandnI5UqVUoNG1fXOx/1y/N6I/t/oK2b9pn8W6AksjmLwFIzmZmZ+v3335WcnKxz587Jy8tL/v7+CgwMzHNr+GvRyS/v/2xwHWdAjStvhGti21sy7u9iFVvZslaH4P58OMbXXUbJvjbienNU4By+3r7dNM7qEAoUt7+O1SHkq0WtfVaHUChF4gYbZcqUUbNmzawOAwAAAMB1xp3QAQAAAJimSFRAAAAAgKLOHW9EaAUqIAAAAABMQwICAAAAwDS0YAEAAAAGFMeb/hVFVEAAAAAAmIYEBAAAAIBpaMECAAAADMhy8t29K3AUAQAAAJiGBAQAAACAaWjBAgAAAAxw8N29S3AUAQAAAJiGBAQAAACAaWjBAgAAAAzgRoSuQQUEAAAAgGlIQAAAAACYhhYsAAAAwABuROgaHEUAAAAApiEBAQAAAGAaWrAAAAAAAxysguUSVEAAAAAAmIYEBAAAAIBpaMECAAAADMjiu3uX4CgCAAAAMA0JCAAAAADT0IIFAAAAGMCNCF2DowgAAADANCQgAAAAAExDCxYAAABggIPv7l2CowgAAADANCQgAAAAAExDCxYAAABgQJbTZnUIboEKCAAAAADTkIAAAAAAMA0JCAAAAADTcA0IAAAAYEAW3927BEcRAAAAgGlIQAAAAACYhhYsAAAAwACHk+/uXYGjCAAAAMA0JCAAAAAATEMLFgAAAGAAq2C5BkcRAAAAgGlIQAAAAACYhhYsAAAAwIAsp83qENwCFRAAAAAApiEBAQAAAGAaWrAAAAAAAxx8d+8SJSoBSXg10OoQ3FqpiulWh+D2nM66Vofg1krbs6wOwe1dOOxtdQhuz+HtsDoEtzakbazVIQDFHmkcAAAAANOUqAoIAAAAcLWynHx37wocRQAAAACmIQEBAAAAYBpasAAAAAADHOJGhK5ABQQAAACAaUhAAAAAAJiGFiwAAADAAFbBcg2OIgAAAADTkIAAAAAAMA0tWAAAAIABWXx37xIcRQAAAACmIQEBAAAAYBpasAAAAAADHE5uROgKVEAAAAAAmIYEBAAAAIBpaMECAAAADGAVLNfgKAIAAAAwDQkIAAAAANPQggUAAAAY4HDy3b0rcBQBAACAEsrhcGj69Olq3769goOD1bt3byUmJha4/ZEjRzRs2DC1atVKrVq10rPPPqvDhw8Xak4SEAAAAKCEmj17tqKjozVhwgTFxMTIZrOpX79+ysjIyHf7oUOHKikpSR988IE++OADHT58WAMHDizUnCQgAAAAgAFZshXJx9XKyMhQVFSUBg8erNDQUAUGBmratGlKTk5WbGxsnu1TU1MVFxenfv36KSgoSEFBQerfv7927NihEydOGJ6XBAQAAAAogRISEnTmzBm1bt06Z8zX11dBQUGKi4vLs72np6fKli2rpUuXKi0tTWlpaVq2bJnq1KmjChUqGJ6Xi9ABAACAEujitRtVq1bNNV65cmUlJSXl2d7T01Ovvfaaxo8fr5CQENlsNlWqVEnz589XqVLG6xokIAAAAIABRXUVrI4dO172+dWrV+c7fu7cOUmS3W7PNe7p6alTp07l2d7pdGrnzp1q3ry5+vbtq6ysLE2bNk2DBg3SZ599Jh8fH0PxkoAAAAAAJZCXl5ek7GtBLv5bktLT0+Xt7Z1n+6+//loLFizQf/7zn5xkY86cOerQoYMWLVqkHj16GJqXBAQAAAAoxgqqcFzJxdarlJQU1apVK2c8JSVFgYGBebbftGmT6tatm6vSUaFCBdWtW1f79u0zPG/RrCMBAAAARYzVq125ehWswMBA+fj4aP369Tljqampio+PV0hISJ7tq1atqsTERKWnp+eMnTt3TgcPHlTt2rUNz0sCAgAAAJRAdrtd3bt315QpU7R69WolJCRo6NCh8vf3V1hYmLKysnTkyBGdP39ektSlSxdJ0pAhQ5SQkJCzvd1u18MPP2x4XhIQAAAAoISKiIhQ165dNXr0aIWHh8vDw0ORkZGy2+1KSkpSu3bttGLFCknZq2MtWLBATqdTPXr0UK9evVSmTBl99tln8vX1NTynzel0Oq/XL1TU1Jk11eoQ3FqpiulX3gjXxOm8+jIrrqy0PcvqENzehcN5L2qEazm8HVaH4NaGtM17cza41tBGq6wOoUCv7bjP6hDy9VLj5VaHUChUQAAAAACYhgQEAAAAgGlYhhcAAAAwIKuI3oiwuOEoAgAAADANCQgAAAAA09CCBQAAABjguIab/uG/qIAAAAAAMA0JCAAAAADT0IIFAAAAGMAqWK7BUQQAAABgGhIQAAAAAKahBQsAAAAwwOFkFSxXoAICAAAAwDQkIAAAAABMQwsWAAAAYEAW3927BEcRAAAAgGlIQAAAAACYhhYsAAAAwABWwXINKiAAAAAATEMCAgAAAMA0tGABAAAABjj47t4lOIoAAAAATEMCAgAAAMA0lrdgPfHEE7LZjK0o8PHHH1/naAAAAID8ZbEKlktYnoC0adNGM2bMUL169dSsWTOrwwEAAABwHVmegAwcOFBly5bV9OnT9d5776lGjRpWhwQAAADgOikS14D07NlTt9xyi95++22rQwEAAADy5XDaiuSjuLG8AnLRa6+9pvj4eKvDAAAAAHAdFZkEpEqVKqpSpYrVYQAAAAC4jopMAgIAAAAUZQ5nkbh6odjjKAIAAAAwDQkIAAAAANPQggUAAAAYkKXit+JUUUQFBAAAAIBpSEAAAAAAmIYWLAAAAMCA4njTv6KICggAAAAA05CAAAAAADANLVgAAACAAdyI0DU4igAAAABMQwICAAAAwDS0YAEAAAAGOLgRoUtQAQEAAABgGhIQAAAAAKahBQsAAAAwIIsbEboEFRAAAAAApiEBAQAAAGAaWrAAAAAAA7gRoWtwFAEAAACYhgQEAAAAgGlowQIAAAAMcLAKlktQAQEAAABgGhIQAAAAAKahBQsAAAAwwCFasFyBCggAAAAA05CAAAAAADANLVgAAACAAayC5RpUQAAAAACYhgQEAAAAgGlowQIAAAAMcDj57t4VOIoAAAAATEMCAgAAAMA0tGABAAAABrAKlmtQAQEAAABgGhIQAAAAAKahBQsAAAAwwCFasFyBCggAAAAA05CAAAAAADANLVgAAACAAayC5RpUQAAAAACYhgQEAAAAgGlowQIAAAAMoAXLNaiAAAAAADANCQgAAAAA09CCBQAAABhAC5ZrUAEBAAAAYBoSEAAAAKCEcjgcmj59utq3b6/g4GD17t1biYmJ+W47Y8YMNWzYMN/HCy+8YHhOm9PpdLrqFyjqlv15s9UhANfEr1Sa1SG4tUx5WB2C28ty8r3X9dbRO8vqENzaMccZq0Nwe5WqHbI6hAJ1/iHC6hDy9fXt069635kzZ2rBggWaOHGiqlSposmTJ+vAgQNavny57HZ7rm3PnDmjs2fP5hpbuHCh5syZo5iYGAUGBhqak08CAAAAoATKyMhQVFSUBg8erNDQUAUGBmratGlKTk5WbGxsnu3LlSunSpUq5TzOnTun9957T88//7zh5EMiAQEAAABKpISEBJ05c0atW7fOGfP19VVQUJDi4uKuuP8bb7yhBg0a6NFHHy3UvKyCBQAAABjgUNFcBatjx46XfX716tX5jh8+fFiSVLVq1VzjlStXVlJS0mVfc9u2bVq9erU++ugjlSpVuJoGFRAAAACgBDp37pwk5bnWw9PTU+np6Zfd98MPP1RwcHCu6olRVEAAAACAYqygCseVeHl5Scq+FuTivyUpPT1d3t7eBe539uxZxcbGauzYsVc1LwkIAAAAYIC73YjwYutVSkqKatWqlTOekpJy2YvK165dK4fDobCwsKualxYsAAAAoAQKDAyUj4+P1q9fnzOWmpqq+Ph4hYSEFLjfpk2b1LhxY/n6+l7VvFRAAAAAgBLIbrere/fumjJlivz8/FS9enVNnjxZ/v7+CgsLU1ZWlo4fP67y5cvnatFKSEjQTTfddNXzUgEBAAAADHA4bUXycS0iIiLUtWtXjR49WuHh4fLw8FBkZKTsdruSkpLUrl07rVixItc+R48e1Q033HDVc3IndKAY4U7o1xd3Qr/+uBP69ced0K8v7oR+/RXlO6GH/d9Qq0PIV+wd06wOoVD4JAAAAABgGq4BAQAAAAxwt1WwrEIFBAAAAIBpSEAAAAAAmIYWLAAAAMAAWrBcgwoIAAAAANOQgAAAAAAwDS1YAAAAgAFOWrBcggoIAAAAANOQgAAAAAAwDS1YAAAAgAEO0YLlClRAAAAAAJiGBAQAAACAaWjBAgAAAAzgRoSuQQUEAAAAgGlIQAAAAACYhhYsAAAAwABuROgaVEAAAAAAmIYEBAAAAIBpaMECAAAADGAVLNegAgIAAADANCQgAAAAAExDCxYAAABgAKtguQYVEAAAAACmIQEBAAAAYBpasAAAAAADWAXLNaiAAAAAADANCQgAAAAA09CCBQAAABjgdFodgXugAgIAAADANCQgAAAAAExDCxYAAABggEOsguUKVEAAAAAAmIYEBAAAAIBpaMECAAAADHByI0KXoAICAAAAwDQkIAAAAABMQwsWAAAAYICDFiyXoAICAAAAwDQkIAAAAABMQwsWAAAAYIDTaXUE7oEKCAAAAADTWJ6A7N27VzNmzNCECRO0Zs2aPM+npaXphRdesCAyAAAAAK5maQKyadMmPfTQQ1q+fLl++OEHPfXUUxo8eLAyMjJytjl//ryWLl1qXZAAAACAsm9EWBQfxY2lCcjUqVPVtWtXffvtt1q1apXeeust/fjjj3rqqaeUmZlpZWgAAAAArgNLE5CdO3eqe/fuOT/fc889mjdvnn799VeNGjXKwsgAAAAAXA+WJiA+Pj46ceJErrFbb71VkydP1rfffquJEydaFBkAAACQm9WtVrRguUBoaKjGjx+vLVu25Gq5uuuuu/Tiiy/qo48+0vjx4y2MEAAAAIArWZqADB8+XBUrVtRjjz2mn3/+Oddz3bt315gxY/T9999bFB0AAAAAV7P0RoQVKlRQVFSU9u/fr4oVK+Z5/vHHH1ebNm20atUqC6IDAAAA/stRDNudiqIicSf0WrVqFfhc3bp1NWDAABOjAQAAAHC9WH4jQgAAAAAlR5GogAAAAABFndNpdQTugQoIAAAAANOQgAAAAAAwDS1YAAAAgAHF8aZ/RREVEAAAAACmIQEBAAAAYBpasAAAAAADaMFyDSogAAAAAExDAgIAAADANLRgAQAAAAZwH0LXoAICAAAAwDQkIAAAAABMQwsWAAAAYACrYLkGFRAAAAAApiEBAQAAAGAaWrAAAAAAI1gGyyWogAAAAAAwDQkIAAAAANPQggUAAAAYwCpYrkEFBAAAAIBpSEAAAAAAmIYWLAAAAMAAJ6tguQQVEAAAAACmIQEBAAAAYBoSEAAAAMAAp9NWJB/XwuFwaPr06Wrfvr2Cg4PVu3dvJSYmFrh9Zmampk6dqvbt2+vmm29W9+7d9fvvvxdqThIQAAAAoISaPXu2oqOjNWHCBMXExMhms6lfv37KyMjId/tXXnlFCxcu1KuvvqpFixbphhtuUL9+/XT69GnDc5KAAAAAACVQRkaGoqKiNHjwYIWGhiowMFDTpk1TcnKyYmNj82x/4MABLVy4UBMnTtQdd9yhgIAAvf7667Lb7dq+fbvheVkFCwAAADDCzW5EmJCQoDNnzqh169Y5Y76+vgoKClJcXJw6d+6ca/t169bJ19dXt99+e67tv//++0LNSwICAAAAFGMdO3a87POrV6/Od/zw4cOSpKpVq+Yar1y5spKSkvJsv2/fPtWsWVOrVq3S3LlzlZycrKCgID3//PMKCAgwHC8tWAAAAEAJdO7cOUmS3W7PNe7p6an09PQ826elpWn//v2aPXu2hg0bpnfffVelS5fW448/rmPHjhmelwoIAAAAYEBRvRFhQRWOK/Hy8pKUfS3IxX9LUnp6ury9vfNsX6ZMGZ0+fVrTpk3LqXhMmzZNoaGhWrJkifr27WtoXiogAAAAQAl0sfUqJSUl13hKSor8/f3zbO/v76/SpUvnarfy8vJSzZo1dfDgQcPzkoAAAAAAJVBgYKB8fHy0fv36nLHU1FTFx8crJCQkz/YhISG6cOGCtm3bljN2/vx5HThwQLVr1zY8Ly1YAAAAgBFFtAXratntdnXv3l1TpkyRn5+fqlevrsmTJ8vf319hYWHKysrS8ePHVb58eXl5eSkkJES33XabnnvuOY0fP1433HCDpk+fLg8PDz344IOG56UCAgAAAJRQERER6tq1q0aPHq3w8HB5eHgoMjJSdrtdSUlJateunVasWJGz/YwZM9SyZUs988wz6tq1q9LS0vTxxx/Lz8/P8Jw2p7OoXk7jesv+vNnqEIBr4lcqzeoQ3FqmPKwOwe1lOfne63rr6J1ldQhu7ZjjjNUhuL1K1Q5ZHUKB6i143eoQ8vXn4y9aHUKh0IIFAAAAGOB0sxsRWoWvogAAAACYhgQEAAAAgGlowQIAAACMKDFXTl9fVEAAAAAAmIYEBAAAAIBpaMECAAAADGAVLNegAgIAAADANCQgAAAAAExDCxYAAABgBKtguQQVEAAAAACmKVEVkE7eqVaH4NZSnelWh+D2fG2eVofg1tKc560Owe1VLFXW6hDc3qjk5laH4Na2P3mT1SG4vW+2WB0BrrcSlYAAAAAAV49VsFyBFiwAAAAApiEBAQAAAGAaWrAAAAAAI1gFyyWogAAAAAAwDQkIAAAAANPQggUAAAAYQQuWS1ABAQAAAGAaEhAAAAAApqEFCwAAADDCyY0IXYEKCAAAAADTkIAAAAAAMA0tWAAAAIABTlbBcgkqIAAAAABMQwICAAAAwDS0YAEAAABG0ILlElRAAAAAAJiGBAQAAACAaWjBAgAAAIzgRoQuQQUEAAAAgGlIQAAAAACYhhYsAAAAwAAbq2C5BBUQAAAAAKYhAQEAAABgGlqwAAAAACNowXIJKiAAAAAATEMCAgAAAMA0tGABAAAARnAjQpegAgIAAADANCQgAAAAAExDCxYAAABgBKtguQQVEAAAAACmIQEBAAAAYBpasAAAAAAjaMFyCSogAAAAAExDAgIAAADANLRgAQAAAEbQguUSVEAAAAAAmIYEBAAAAIBpaMECAAAAjHDarI7ALVABAQAAAGAaEhAAAAAApqEFCwAAADDAxipYLkEFBAAAAIBpSEAAAAAAmIYWLAAAAMAIWrBcggoIAAAAANOQgAAAAAAwDQkIAAAAANOQgAAAAAAwTaEuQj9y5IhmzZqlAwcO6B//+IcaNWqkJk2aqHHjxvL29r5eMQIAAABwE4VKQF588UWtW7dODRo00MGDB/XVV1/J6XSqVKlSqlevnpo0aaKmTZuqadOmCgwMVJkyZa5X3AAAAICpuBGhaxQqAfn11181cuRI9e7dW5J09uxZ7dixQ9u2bdO2bdsUFxenJUuWSJLsdru2bt16xddMT0/X7t27Vb9+fXl5een333/X/PnzlZycrAYNGqhHjx7y9/e/il8NAAAAQFFTqATE09NTQUFBOT+XLVtWLVq0UIsWLXLGTp48qa1bt2r79u1XfL09e/aoZ8+eOnLkiKpVq6YJEyZo4MCBqlGjhgICAvTdd99p8eLFWrBggQICAgoTKgAAAIAiqFAXod91112Kj4+/7DY33HCDbr/9dg0cOPCKr/fmm2+qefPmWrp0qW699VY9/fTTuv/++/XVV1/pnXfe0cqVK9W2bVtNnDixMGECAAAArue0Fc1HMVOoBOSRRx7RypUr9ccff7hk8g0bNmjIkCEKDAzUc889p/T0dIWHh8tmyz6QpUuX1lNPPaVNmza5ZD4AAAAA1ipUC9a///1v2Ww2/etf/1KnTp3Uvn17NW7cWLVr176qyb28vHT+/HlJ0j/+8Q/9+9//lqenZ65tUlNTVb58+at6fQAAAABFS6ESkAkTJuj333/Xjh07tHLlSi1ZskQ2m03lypVTUFCQmjRpolGjRhl+vXbt2unVV1/VhAkTFBAQoPHjx+c853Q6tWHDBo0bN0533XVXYcIEAAAAXI9VsFyiUC1YXbt21csvv6zo6Ght3rxZX331lSZOnKguXbrowoULio6OLtTkL7zwgrKysjR79uw8z61YsUI9evRQ9erVNWzYsEK9LgAAAICiqVAVkP9VqlQpNWjQQA0aNFCXLl0kZVctCsPPz0+ff/65Tp48mee5Nm3aaOnSpQoMDLzaEAEAAAAUMVedgOTn4sXjhXXDDTfkGfPz85Ofn981RgQAAAC4CC1YLlGoFiwAAAAAuBYkIAAAAABM49IWLAAAAMBd2WjBcgkqIAAAAEAJ5XA4NH36dLVv317BwcHq3bu3EhMTC9x+yZIlatiwYZ7H5fa5FBUQAAAAoISaPXu2oqOjNXHiRFWpUkWTJ09Wv379tHz5ctnt9jzb79y5Uy1bttRbb72Va7wwi0dRAQEAAACMcBbRx1XKyMhQVFSUBg8erNDQUAUGBmratGlKTk5WbGxsvvvs2rVLgYGBqlSpUq6Hh4eH4XlJQAAAAIASKCEhQWfOnFHr1q1zxnx9fRUUFKS4uLh899m5c6fq169/TfPSggUAAAAUYx07drzs86tXr853/PDhw5KkqlWr5hqvXLmykpKS8mx//PhxHT16VHFxcfrkk0908uRJBQcHa8SIEapbt67heKmAAAAAAEZY3Wrl4hasc+fOSVKeaz08PT2Vnp6eZ/tdu3ZJkjw8PDRp0iRNmzZNZ8+e1eOPP66jR48anpcKCAAAAFCMFVThuBIvLy9J2deCXPy3JKWnp8vb2zvP9q1bt9aGDRtUoUKFnLFZs2apQ4cOWrx4sfr3729oXiogAAAAQAl0sfUqJSUl13hKSor8/f3z3ed/kw9JKlu2rGrUqKHk5GTD85KAAAAAAAbYnEXzcbUCAwPl4+Oj9evX54ylpqYqPj5eISEhebZfsGCBWrVqpfPnz+eMpaWlad++fYW6MJ0EBAAAACiB7Ha7unfvrilTpmj16tVKSEjQ0KFD5e/vr7CwMGVlZenIkSM5CUeHDh3kdDo1atQo7d69W9u2bdPgwYPl5+enhx56yPC8JCAAAABACRUREaGuXbtq9OjRCg8Pl4eHhyIjI2W325WUlKR27dppxYoVkrJbtj766COdOXNG4eHh6tmzp8qXL6+PP/441zUkV2JzOp3XULgpXtKT6lkdgltLdeZdLQGu5WvztDoEt5bmzLA6BLdXsVRZq0Nwe6OSm1sdglvb/uRNVofg9r7Z8qrVIRSo/pvTrA4hX3+MGmp1CIVCBQQAAACAaUhAAAAAAJiG+4AAAAAARpSYCxeuLyogAAAAAExDAgIAAADANLRgAQAAAAZcy03/8F9UQAAAAACYhgQEAAAAgGlowQIAAACMoAXLJaiAAAAAADANCQgAAAAA09CCBQAAABjAKliuQQUEAAAAgGlIQAAAAACYhhYsAAAAwAhasFyCCggAAAAA05CAAAAAADANLVgAAACAEbRguQQVEAAAAACmIQEBAAAAYBpasAAAAAADuBGha1ABAQAAAGAaEhAAAAAApiEBAQAAAGAaEhAAAAAApiEBAQAAAGAaVsECAAAAjGAVLJegAgIAAADANCQgAAAAAExDCxYAAABgADcidA0qIAAAAABMQwICAAAAwDS0YAEAAABG0ILlEiUqAXmwYajVIbg1x7lzVofg9kp5e1sdgltznE+3OgT353RYHYH7s9HccD3ZSv1pdQhAsce7FAAAAADTlKgKCAAAAHDVaMFyCSogAAAAAExDAgIAAADANLRgAQAAAAZwI0LXoAICAAAAwDQkIAAAAABMQwsWAAAAYAQtWC5BBQQAAACAaUhAAAAAAJiGFiwAAADAAFbBcg0qIAAAAABMQwICAAAAwDS0YAEAAABG0ILlElRAAAAAAJiGBAQAAACAaWjBAgAAAIygBcslqIAAAAAAMA0JCAAAAADT0IIFAAAAGMCNCF2DCggAAAAA05CAAAAAADANLVgAAACAEbRguQQVEAAAAACmIQEBAAAAYBpasAAAAAAjaMFyCSogAAAAAExDAgIAAADANLRgAQAAAAZwI0LXoAICAAAAwDQkIAAAAABMQwsWAAAAYAQtWC5BBQQAAACAaUhAAAAAAJiGFiwAAADAAFbBcg0qIAAAAABMQwICAAAAwDS0YAEAAABG0ILlElRAAAAAAJiGBAQAAACAaUhAAAAAAJiGBAQAAAAwwllEH9fA4XBo+vTpat++vYKDg9W7d28lJiYa2verr75Sw4YNdfDgwULNSQICAAAAlFCzZ89WdHS0JkyYoJiYGNlsNvXr108ZGRmX3e/QoUMaN27cVc1JAgIAAACUQBkZGYqKitLgwYMVGhqqwMBATZs2TcnJyYqNjS1wP4fDoZEjR6px48ZXNS8JCAAAAGCArYg+rlZCQoLOnDmj1q1b54z5+voqKChIcXFxBe43Z84cZWZmasCAAVc1L/cBAQAAAEqgw4cPS5KqVq2aa7xy5cpKSkrKd5+tW7cqKipKCxcuVHJy8lXNSwICAAAAFGMdO3a87POrV6/Od/zcuXOSJLvdnmvc09NTp06dyrP92bNnNWLECI0YMUJ16tQhAQEAAACuKze7E7qXl5ek7GtBLv5bktLT0+Xt7Z1n+wkTJqhOnTp67LHHrmleEhAAAACgGCuownElF1uvUlJSVKtWrZzxlJQUBQYG5tl+0aJFstvtat68uSQpKytLknTffffpgQce0Pjx4w3NSwICAAAAlECBgYHy8fHR+vXrcxKQ1NRUxcfHq3v37nm2X7VqVa6ft2zZopEjR2ru3LkKCAgwPC8JCAAAAGCAzc1asOx2u7p3764pU6bIz89P1atX1+TJk+Xv76+wsDBlZWXp+PHjKl++vLy8vFS7du1c+1+8iL1atWq68cYbDc/LMrwAAABACRUREaGuXbtq9OjRCg8Pl4eHhyIjI2W325WUlKR27dppxYoVLp3T5nQ6i2Qud//992vu3Ll5lgW7Fp18e7nstZCX4++VFHD9lMrngjC4juN8utUhuD+nw+oI3J+N7xavJ1upa7nrAoz49vynVodQoOAh06wOIV9b3h5qdQiFYmkL1tKlSwt8LjExUStXrpSfn58kqUuXLuYEBQAAAOSnSH5tX/xYmoCMGzdO58+flyTlV4h58803JUk2m40EBAAAAHADliYgixcv1ogRI1S+fHlNmjRJVapUyXmuefPm+vLLL1WzZk0LIwQAAADgSpY2itatW1cxMTFq1qyZHnzwQZdf4AIAAAC4jLOIPooZy69UK126tIYNG6YZM2ZoypQpGj58uE6fPm11WAAAAACuA8sTkItatGiRc1H6fffdp8zMTGsDAgAAAOByRepGhL6+vpo6daqWLl2qxYsXy9PT0+qQAAAAAEnudyNCqxSpBOSiLl26sOoVAAAA4IaKTAsWAAAAAPdXJCsgAAAAQJFDC5ZLUAEBAAAAYBoSEAAAAACmoQULAAAAMIBVsFyDCggAAAAA05CAAAAAADANLVgAAACAEbRguQQVEAAAAACmIQEBAAAAYBpasAAAAAADWAXLNaiAAAAAADANCQgAAAAA09CCBQAAABhBC5ZLUAEBAAAAYBoSEAAAAACmoQULAAAAMIIWLJegAgIAAADANCQgAAAAAExDCxYAAABgADcidA0qIAAAAABMQwICAAAAwDS0YAEAAABG0ILlElRAAAAAAJiGBAQAAACAaWjBAgAAAAywOenBcgUqIAAAAABMQwICAAAAwDS0YAEAAABG0IHlElRAAAAAAJiGBAQAAACAaWjBAgAAAAyw0YLlElRAAAAAAJiGBAQAAACAaWjBAgAAAIygBcslqIAAAAAAMA0JCAAAAADT0IIFAAAAGMAqWK5BBQQAAACAaUhAAAAAAJiGFiwAAADACFqwXIIKCAAAAADTkIAAAAAAMA0tWAAAAIABrILlGlRAAAAAAJiGBAQAAACAaWjBAgAAAIygBcslqIAAAAAAME2JqoBkpaVZHQJwTTiHAVyJh4+31SG4Nd6HgWtXohIQAAAA4GqxCpZr0IIFAAAAwDQkIAAAAABMQwsWAAAAYISTHixXoAICAAAAwDQkIAAAAABMQwsWAAAAYACrYLkGFRAAAAAApiEBAQAAAGAaWrAAAAAAI2jBcgkqIAAAAABMQwICAAAAwDS0YAEAAAAG2BxWR+AeqIAAAAAAMA0JCAAAAADT0IIFAAAAGMEqWC5BBQQAAACAaUhAAAAAAJiGFiwAAADAABstWC5BBQQAAAAooRwOh6ZPn6727dsrODhYvXv3VmJiYoHbb9++XT169FDz5s3VunVrjRkzRqmpqYWakwQEAAAAKKFmz56t6OhoTZgwQTExMbLZbOrXr58yMjLybJuSkqJevXqpVq1aWrJkiWbPnq3NmzfrueeeK9ScJCAAAACAEU5n0XxcpYyMDEVFRWnw4MEKDQ1VYGCgpk2bpuTkZMXGxubZ/tChQ2rfvr3Gjh2rOnXq6JZbbtG//vUv/fzzz4WalwQEAAAAKIESEhJ05swZtW7dOmfM19dXQUFBiouLy7N98+bN9dZbb6l06ezLyP/44w8tWbJEbdu2LdS8XIQOAAAAFGMdO3a87POrV6/Od/zw4cOSpKpVq+Yar1y5spKSki77mnfffbf27dun6tWra/bs2YWIlgoIAAAAYIjNWTQfV+vcuXOSJLvdnmvc09NT6enpl913ypQpmj9/vipVqqQnn3xSZ86cMTwvFRAAAACgGCuownElXl5ekrKvBbn4b0lKT0+Xt7f3Zfdt2rSpJGnGjBkKDQ1VbGysunTpYmheKiAAAABACXSx9SolJSXXeEpKivz9/fNsv2fPHq1ZsybXWOXKlVWhQgUlJycbnpcEBAAAADDCWUQfVykwMFA+Pj5av359zlhqaqri4+MVEhKSZ/u1a9fq2WefVVpaWs7Y/v37deLECQUEBBielwQEAAAAKIHsdru6d++uKVOmaPXq1UpISNDQoUPl7++vsLAwZWVl6ciRIzp//rwk6cEHH1T58uU1cuRI7d69Wxs3blRERISaNWumDh06GJ6XBAQAAAAooSIiItS1a1eNHj1a4eHh8vDwUGRkpOx2u5KSktSuXTutWLFCklSxYkV9/PHHcjgcCg8P16BBgxQUFKTIyEh5eHgYntPmdF7D3UuKmbBS/7I6BAAArisPHx+rQ3BrWf/TeoLrI9bxhdUhFKj9Q1OsDiFfa5eMsDqEQqECAgAAAMA0JCAAAAAATMN9QAAAAAAjSs6VC9cVFRAAAAAApiEBAQAAAGAaWrAAAAAAA2x0YLkEFRAAAAAApiEBAQAAAGAaWrAAAAAAI2jBcgkqIAAAAABMQwICAAAAwDS0YAEAAAAGsAqWa1ABAQAAAGAaEhAAAAAApqEFCwAAADDCQQ+WK1ABAQAAAGAaEhAAAAAApqEFCwAAADCCDiyXoAICAAAAwDQkIAAAAABMQwsWAAAAYAA3InQNSysgCxcuVEZGRq6xX375Rf3799cDDzyg4cOH648//rAoOgAAAACuZmkC8vLLL+v06dM5P69bt069evWSw+FQu3btdOTIET3yyCPavHmzhVECAAAAcBVLW7Ccztx1rNmzZ+vJJ5/UCy+8kDM2ceJETZkyRQsWLDA7PAAAAOC/nPRguUKRugg9MTFRDz74YK6xRx99VPHx8RZFBAAAAMCVLE1AbDZbrp/r1Kmjs2fP5ho7ceKEypcvb2ZYAAAAAK4Ty1uwOnbsqLp16yogIEB2u12TJ0/W/PnzVaZMGW3evFnjxo1TaGiolWECAAAArILlIpYmIN9//7127typXbt2aefOnTpy5Ij27dunrKwslSlTRn369FHDhg01fPhwK8MEAAAA4CKWJiDVqlVTtWrV1KFDh5yxzMxMlSlTRpIUHR2tm266KU+rFgAAAIDiqcjdiPBi8iFJDRs2tDASAAAA4H/QguUSRWoVLAAAAADujQQEAAAAgGmKXAsWAAAAUBTZuBGhS1ABAQAAAGAaEhAAAAAApqEFCwAAADDCYXUA7oEKCAAAAADTkIAAAAAAMA0tWAAAAIABrILlGlRAAAAAAJiGBAQAAACAaWjBAgAAAIygA8slqIAAAAAAMA0JCAAAAADT0IIFAAAAGMEqWC5BBQQAAACAaUhAAAAAAJiGFiwAAADAABsdWC5BBQQAAACAaUhAAAAAAJiGFiwAAADACFbBcgkqIAAAAABMQwICAAAAwDS0YAEAAAAG2BxWR+AeqIAAAAAAMA0JCAAAAADT0IIFAAAAGMEqWC5BBQQAAACAaUhAAAAAAJiGFiwAAADACDqwXIIKCAAAAADTkIAAAAAAMA0tWAAAAIABNlbBcgkqIAAAAABMQwICAAAAwDS0YAEAAABG0ILlElRAAAAAAJiGBAQAAACAaWjBAgAAAIxwWB2Ae6ACAgAAAMA0JCAAAAAATEMLFgAAAGAANyJ0DSogAAAAAExDAgIAAADANLRgAQAAAEbQguUSVEAAAAAAmIYEBAAAACihHA6Hpk+frvbt2ys4OFi9e/dWYmJigdvv3r1b/fv3V6tWrdSmTRtFRETor7/+KtScJCAAAACAEU5n0Xxcg9mzZys6OloTJkxQTEyMbDab+vXrp4yMjDzbnjhxQr169VK5cuU0f/58zZs3TydOnFDfvn2Vnp5ueE6uAQEAwI1kpaVZHQKAYiIjI0NRUVEaOXKkQkNDJUnTpk1T+/btFRsbq86dO+fa/rvvvtO5c+f0xhtvyNPTU5I0efJkhYaGavPmzWrTpo2heamAAAAAACVQQkKCzpw5o9atW+eM+fr6KigoSHFxcXm2b9OmjWbNmpWTfPyvU6dOGZ6XCggAAABghMPqAPLXsWPHyz6/evXqfMcPHz4sSapatWqu8cqVKyspKSnP9jVq1FCNGjVyjb333nvy9PRUixYtDMdLBQQAAAAogc6dOydJstvtucY9PT0NXdPx8ccfa8GCBRo2bJhuvPFGw/NSAQEAAACKsYIqHFfi5eUlKftakIv/lqT09HR5e3sXuJ/T6dQ777yjd999VwMGDFDPnj0LNS8JCAAAAGCAzc1uRHix9SolJUW1atXKGU9JSVFgYGC++2RmZuqFF17Q8uXLNWrUKPXp06fQ89KCBQAAAJRAgYGB8vHx0fr163PGUlNTFR8fr5CQkHz3GTVqlL755htNnTr1qpIPiQoIAAAAUCLZ7XZ1795dU6ZMkZ+fn6pXr67JkyfL399fYWFhysrK0vHjx1W+fHl5eXlp8eLFWrFihUaNGqWWLVvqyJEjOa91cRsjqIAAAAAARlh9w8HrcCPCiIgIde3aVaNHj1Z4eLg8PDwUGRkpu92upKQktWvXTitWrJAkLV++XJL05ptvql27drkeF7cxwuZ0ulkz22WElfqX1SEAAADgMmIdX1gdQoE6Bb9sdQj5+mbLq1aHUChUQAAAAACYhmtAAAAAACNKTuPQdUUFBAAAAIBpSEAAAAAAmIYWLAAAAMAIWrBcggoIAAAAANOQgAAAAAAwDS1YAAAAgBEOqwNwD1RAAAAAAJiGBAQAAACAaWjBAgAAAAywsQqWS1ABAQAAAGAaEhAAAAAApqEFCwAAADCCFiyXoAICAAAAwDQkIAAAAABMQwsWAAAAYISDFixXoAICAAAAwDQkIAAAAABMQwsWAAAAYASrYLkEFRAAAAAApiEBAQAAAGAaWrAAAAAAI2jBcgkqIAAAAABMQwICAAAAwDS0YAEAAABG0ILlElRAAAAAAJiGBAQAAACAaWjBAgAAAIxw0ILlClRAAAAAAJiGBAQAAACAaWjBAgAAAIxwOqyOwC1QAQEAAABgGhIQAAAAAKahBQsAAAAwghsRugQVEAAAAACmIQEBAAAAYBpasAAAAAAjuBGhS1ABAQAAAGAaEhAAAAAApqEFCwAAADCCVbBcggoIAAAAANOQgAAAAAAwjeUtWFu2bNH69evVv39/SdIvv/yiDz/8UAcPHlStWrXUu3dvhYSEWBwlAAAASjxasFzC0grIN998o/DwcG3YsEGS9J///Ee9evWS0+lUaGioMjMz1aNHD/3nP/+xMkwAAAAALmJpBWTmzJl65plnNHDgQEnSu+++q6eeekrPPvtszjbvvvuupk+frg4dOlgVJgAAAAAXsbQCsn//ft1///05Px88eFB33313rm3uu+8+7dmzx+zQAAAAgNyczqL5KGYsTUBq1qypNWvW5PzcqFEjJSQk5Npm69atqlKlitmhAQAAALgOLG3B6tevn1566SUdPnxY9913nwYOHKjnn39e6enpatCggbZs2aJZs2bpmWeesTJMAAAAAC5iaQLSpUsX2Ww2TZ8+Xe+//75sNpucTqfGjh0rSSpXrpz69u2rnj17WhkmAAAAIDkcVkfgFixfhvfBBx/Ugw8+qD///FP79u1TWlqaypQpI39/fwUFBcnT09PqEAEAAAC4iOUJyEX16tVTvXr1rA4DAAAAwHVUZBIQAAAAoEgrhitOFUWWroIFAAAAoGQhAQEAAABgGlqwAAAAACNowXIJKiAAAAAATEMCAgAAAMA0tGABAAAARjhowXIFKiAAAAAATEMCAgAAAMA0tGABAAAABjidDqtDcAtUQAAAAACYhgQEAAAAgGlowQIAAACMYBUsl6ACAgAAAMA0JCAAAAAATEMLFgAAAGCEkxYsV6ACAgAAAMA0JCAAAAAATEMLFgAAAGCEgxsRugIVEAAAAACmIQEBAAAAYBpasAAAAAAjWAXLJaiAAAAAADANCQgAAABQQjkcDk2fPl3t27dXcHCwevfurcTEREP79enTRzNmzCj0nCQgAAAAgAFOh6NIPq7F7NmzFR0drQkTJigmJkY2m039+vVTRkZGgfucP39eI0eO1Lp1665qThIQAAAAoATKyMhQVFSUBg8erNDQUAUGBmratGlKTk5WbGxsvvts3rxZDz30kLZs2SJfX9+rmpcEBAAAACiBEhISdObMGbVu3TpnzNfXV0FBQYqLi8t3n7Vr1yosLExLly5V+fLlr2peVsECAAAAjHCzVbAOHz4sSapatWqu8cqVKyspKSnffZ599tlrnpcEBAAAACjGOnbseNnnV69ene/4uXPnJEl2uz3XuKenp06dOuWa4PJBCxYAAABQAnl5eUlSngvO09PT5e3tfd3mpQICAAAAGOEomi1YBVU4ruRi61VKSopq1aqVM56SkqLAwECXxJYfKiAAAABACRQYGCgfHx+tX78+Zyw1NVXx8fEKCQm5bvNSAQEAAABKILvdru7du2vKlCny8/NT9erVNXnyZPn7+yssLExZWVk6fvy4ypcvn9Ou5QokIAAAAIARzmu76V9RFBERoQsXLmj06NE6f/68WrRoocjISNntdh08eFAdO3bUxIkT9fDDD7tsTpvT6WbriV1GWKl/WR0CAAAALiPW8YXVIRTobq9uVoeQr2/Pf2p1CIXCNSAAAAAATEMLFgAAAGCAs4iuglXcUAEBAAAAYBoSEAAAAACmoQULAAAAMMINV8GyAhUQAAAAAKYhAQEAAABgGlqwAAAAAANYBcs1qIAAAAAAMA0JCAAAAADT0IIFAAAAGMEqWC5BBQQAAACAaUhAAAAAAJjG5nQ6uZwfAAAAgCmogAAAAAAwDQkIAAAAANOQgAAAAAAwDQkIAAAAANOQgAAAAAAwDQkIAAAAANOQgAAAAAAwDQkIAAAAANOQgAAAAAAwDQkIAAAAANOQgAAAAAAwDQkIAAAAANOQgAAAAAAwDQlIEeNwODR9+nS1b99ewcHB6t27txITE60Oy23Nnj1bTzzxhNVhuJWTJ09qzJgxuv3223XLLbcoPDxcGzdutDost3Ls2DGNHDlSrVu3VvPmzdW/f3/98ccfVofllvbu3avmzZtr8eLFVofiVg4dOqSGDRvmeXzxxRdWh+ZWli5dqnvvvVdNmzZV586dtXLlSqtDAiSRgBQ5s2fPVnR0tCZMmKCYmBjZbDb169dPGRkZVofmdj788ENNnz7d6jDczrBhw7Rlyxa99dZbWrhwoRo3bqw+ffpoz549VofmNp5++mkdOHBA8+bN08KFC+Xl5aWePXvq3LlzVofmVjIzMzVixAidPXvW6lDczs6dO+Xp6am1a9dq3bp1OY/777/f6tDcxrJly/Tiiy/q0Ucf1fLly3Xvvfdq2LBh+vXXX60ODSABKUoyMjIUFRWlwYMHKzQ0VIGBgZo2bZqSk5MVGxtrdXhuIzk5WX379tU777yjunXrWh2OW0lMTNSPP/6osWPHKiQkRPXq1dNLL72kKlWqaPny5VaH5xZOnDihGjVq6NVXX1XTpk0VEBCggQMH6siRI9q9e7fV4bmVGTNmqFy5claH4ZZ27dqlunXrqnLlyqpUqVLOw8vLy+rQ3ILT6dQ777yjHj16qEePHqpdu7YGDRqk2267TRs2bLA6PIAEpChJSEjQmTNn1Lp165wxX19fBQUFKS4uzsLI3MuOHTtUoUIFffnllwoODrY6HLdSsWJFzZ07V02aNMkZs9lscjqdOnXqlIWRuY+KFSvqrbfeUoMGDSRJR48eVWRkpPz9/VW/fn2Lo3MfcXFxiomJ0aRJk6wOxS3t3LmT8/U6+vPPP3Xo0KE8FaXIyEgNGDDAoqiA/yptdQD4r8OHD0uSqlatmmu8cuXKSkpKsiIkt3TnnXfqzjvvtDoMt+Tr66vQ0NBcYytXrtT+/fvVrl07i6JyXy+//LI+//xz2e12vfvuuypbtqzVIbmF1NRUjRo1SqNHj87zfgzX2LVrlypVqqTHH39c+/btU+3atTVw4EC1b9/e6tDcwr59+yRJZ8+eVZ8+fRQfH68aNWro6aef5vMPRQIVkCLkYv+23W7PNe7p6an09HQrQgKuyaZNm/Tiiy+qY8eOfOhdBz169NCiRYv0wAMPaNCgQdqxY4fVIbmFV155RTfffDPXI1wnGRkZ2rdvn9LS0jRkyBDNnTtXTZs2Vb9+/fTzzz9bHZ5bSEtLkyQ999xzuu+++xQVFaW2bdtq4MCBHGMUCVRAipCLva8ZGRm5+mDT09Pl7e1tVVjAVfnuu+80YsQIBQcH66233rI6HLd0sYXl1Vdf1W+//ab58+dr4sSJFkdVvC1dulQbN27UV199ZXUobstutysuLk6lS5fO+cKtSZMm2rNnjyIjI9WmTRuLIyz+ypQpI0nq06ePHnroIUlSo0aNFB8frw8++IBjDMtRASlCLpb6U1JSco2npKTI39/fipCAqzJ//nwNHjxYt99+u+bNm8eFpS507NgxLV++XFlZWTljpUqVUkBAQJ73DhTeokWLdOzYMd1xxx1q3ry5mjdvLkkaO3asOnfubHF07qNs2bJ5qv033XSTkpOTLYrIvVz8m+Gmm27KNV6/fn0dPHjQipCAXEhAipDAwED5+Pho/fr1OWOpqamKj49XSEiIhZEBxi1YsECvvvqqunXrprfffjvPHxm4NikpKRo+fHiulWwyMzMVHx+vgIAACyNzD1OmTNGKFSu0dOnSnIckRUREaO7cudYG5yYSEhLUvHnzPPcH2r59Oxemu0hQUJDKlSunLVu25BrftWuXatWqZVFUwH/RglWE2O12de/eXVOmTJGfn5+qV6+uyZMny9/fX2FhYVaHB1zR3r179frrryssLEwDBgzQsWPHcp7z8vJS+fLlLYzOPQQGBqpdu3YaN26cJkyYIF9fX82ZM0epqanq2bOn1eEVe1WqVMl3/MYbb1T16tVNjsY93XTTTWrQoIHGjRunsWPHqmLFivr888/122+/aeHChVaH5xa8vLzUt29fzZo1S1WqVFGzZs309ddf68cff9SHH35odXgACUhRExERoQsXLmj06NE6f/68WrRoocjISL5FRrHw7bffKjMzU7GxsXnuXfPQQw/pjTfesCgy92Gz2fT2229r6tSpGjJkiE6fPq2QkBB9+umnqlatmtXhAVdUqlQpzZkzR1OmTNGQIUOUmpqqoKAgffDBB2rYsKHV4bmNgQMHytvbO+d+YgEBAZoxY4ZatWpldWiAbE6n02l1EAAAAABKBq4BAQAAAGAaEhAAAAAApiEBAQAAAGAaEhAAAAAApiEBAQAAAGAaEhAAAAAApiEBAQAAAGAaEhAAAAAApiEBAYBiJj09XUFBQWrevLleffVVq8MBAKBQSEAAoJix2Wz66KOP1KxZM82fP1979+61OiQAAAwjAQGAYsZut6tFixbq27evJGnHjh0WRwQAgHEkIABQTNWrV0+S9Pvvv1scCQAAxpGAAEAxNW/ePElSQkKCxZEAAGAcCQgAFEPr1q3TZ599pgoVKig+Pt7qcAAAMIwEBACKmdTUVL344ovq2LGjwsPDdfz4cSUnJ1sdFgAAhpCAAEAxM27cOF24cEETJkxQUFCQJNqwAADFBwkIABQj33zzjZYvX67XXntNfn5+OQkIF6IDAIoLEhAAKCaOHDmisWPH6tFHH1WHDh0kSTVr1pSvry/XgQAAig0SEAAoJl5++WVVqFBBzz//fK7xRo0a0YIFACg2SEAAoBj44osv9MMPP+jNN99U2bJlcz0XFBSk/fv3Ky0tzaLoAAAwzuZ0Op1WBwEAAACgZKACAgAAAMA0JCAAAAAATEMCAgAAAMA0JCAAAAAATEMCAgAAAMA0JCAAAAAATEMCAgAAAMA0JCAAAAAATEMCAgAAAMA0JCAAAAAATEMCAgAAAMA0JCAAAAAATPP/aLJtd5wFrX8AAAAASUVORK5CYII=",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "filenames": {
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_258_2.png"
+ }
+ },
+ "output_type": "display_data"
+ }
+ ],
"source": [
"# visual representation of grid search\n",
"# uses seaborn heatmap, you can also do this with matplotlib imshow\n",
@@ -4445,7 +5128,7 @@
},
{
"cell_type": "markdown",
- "id": "e07e69e3",
+ "id": "d75dad8e",
"metadata": {
"editable": true
},
@@ -4468,12 +5151,607 @@
{
"cell_type": "code",
"execution_count": 12,
- "id": "f48eaa77",
+ "id": "d55fae28",
"metadata": {
"collapsed": false,
"editable": true
},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.18333333333333332\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.18611111111111112\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.13055555555555556\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.24444444444444444\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.23333333333333334\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.12777777777777777\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1e-05\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.1527777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.9111111111111111\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.8888888888888888\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.8722222222222222\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.8305555555555556\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.8888888888888888\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.8805555555555555\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.0001\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.8944444444444445\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.975\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.9777777777777777\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.9805555555555555\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.9861111111111112\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.9805555555555555\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.9777777777777777\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.001\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.9444444444444444\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.01\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.9861111111111112\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.01\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.9888888888888889\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.01\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.9888888888888889\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.01\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.9861111111111112\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.01\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.9888888888888889\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.01\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.9722222222222222\n",
+ "\n",
+ "Learning rate = 0.01\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.9527777777777777\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.1\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.9027777777777778\n",
+ "\n",
+ "Learning rate = 0.1\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.8583333333333333\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.1\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.8722222222222222\n",
+ "\n",
+ "Learning rate = 0.1\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.9055555555555556\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.1\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.8805555555555555\n",
+ "\n",
+ "Learning rate = 0.1\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.8722222222222222\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 0.1\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.8666666666666667\n",
+ "\n",
+ "Learning rate = 1.0\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.08611111111111111\n",
+ "\n",
+ "Learning rate = 1.0\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.10555555555555556\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1.0\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.10555555555555556\n",
+ "\n",
+ "Learning rate = 1.0\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.17777777777777778\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1.0\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.08333333333333333\n",
+ "\n",
+ "Learning rate = 1.0\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.08888888888888889\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 1.0\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.09444444444444444\n",
+ "\n",
+ "Learning rate = 10.0\n",
+ "Lambda = 1e-05\n",
+ "Accuracy score on test set: 0.17222222222222222\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 10.0\n",
+ "Lambda = 0.0001\n",
+ "Accuracy score on test set: 0.11666666666666667\n",
+ "\n",
+ "Learning rate = 10.0\n",
+ "Lambda = 0.001\n",
+ "Accuracy score on test set: 0.10555555555555556\n",
+ "\n",
+ "Learning rate = 10.0\n",
+ "Lambda = 0.01\n",
+ "Accuracy score on test set: 0.1388888888888889\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 10.0\n",
+ "Lambda = 0.1\n",
+ "Accuracy score on test set: 0.11388888888888889\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Learning rate = 10.0\n",
+ "Lambda = 1.0\n",
+ "Accuracy score on test set: 0.10555555555555556\n",
+ "\n",
+ "Learning rate = 10.0\n",
+ "Lambda = 10.0\n",
+ "Accuracy score on test set: 0.09444444444444444\n",
+ "\n"
+ ]
+ }
+ ],
"source": [
"from sklearn.neural_network import MLPClassifier\n",
"# store models for later use\n",
@@ -4495,7 +5773,7 @@
},
{
"cell_type": "markdown",
- "id": "ac3c7bcb",
+ "id": "276badf8",
"metadata": {
"editable": true
},
@@ -4506,12 +5784,41 @@
{
"cell_type": "code",
"execution_count": 13,
- "id": "2d81262e",
+ "id": "7d6c714e",
"metadata": {
"collapsed": false,
"editable": true
},
- "outputs": [],
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyAAAANbCAYAAAC6lftqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABZHklEQVR4nO3deZxd8/kH8M9EMolstiIkYgkSQQixh1hLS1G0qBBbbK19V60iqggpaqkl2lJb7SV2LaWW2JcIrSURslgiEVkmyczvj8j0N5JwcXNuMnm/+5rXq3PmnHufe5remc99nvM9VXV1dXUBAAAoQJNKFwAAACw4BBAAAKAwAggAAFAYAQQAACiMAAIAABRGAAEAAAojgAAAAIURQAAAgMIIIAAAQGEEEGC+dNJJJ6Vz585f+bXlllt+p+e47bbb0rlz54wYMWKuHvNdvffee+nSpUvWX3/9TJ48ubDnBYBvo6qurq6u0kUAfFPDhw/PJ598Uv/9pZdemiFDhuQPf/hD/bbq6up07dr1Wz/HJ598kuHDh6dr166prq6ea8d8VwMGDMj999+fESNG5Mwzz8yPf/zjQp4XAL4NAQRoFE466aQ888wzeeSRRypdSqFqa2uz5ZZbZuedd86QIUMybty43HTTTZUuCwDmyAgW0Kg9/fTT6dy5c2688cZsscUW2XjjjfP4448nSf72t79ll112ydprr51u3bplp512yqBBg+qP/fI41UknnZR99903t956a7bddtusscYa2XHHHfPoo49+p2OS5IUXXshee+2VtddeO5tvvnn+/Oc/Z999981JJ530la/v8ccfz8iRI7PFFltkxx13zIsvvpihQ4fOst/HH3+cU045JRtvvHG6d++evfbaK88991z9z6dOnZpLLrkkW2+9dbp165btt98+t956a/3P99577+y9996zPbdPP/10/Wvv2rVr/va3v6Vnz57ZbLPN8p///CfTp0/PFVdckR122CHdunXL2muvnT322CNPPvlkg8d79dVXc+CBB2bdddfNhhtumKOPPjojR47MtGnT0rNnzxx77LGzvK4f/OAHOfnkk7/yHAEwbxFAgAXCgAEDcuKJJ+bEE0/M2muvnb/+9a/59a9/na222ip//OMfc95556VZs2Y5/vjj88EHH8zxcV599dVcffXVOeKII3LJJZekadOmOeKIIzJu3Lhvfcxbb72VfffdN0lywQUX5PDDD88VV1zRICDMya233poVV1wxa621VrbZZpu0bds2N9xwQ4N9Jk6cmD322CP//ve/c+yxx+YPf/hDWrVqlQMPPDBvvfVWkuTEE0/MFVdckd122y1//OMf06tXr5xyyim54447vraG/2/69Om5/PLL069fvxx11FFZeeWV079//1xyySXZfffdc9VVV+WMM87I2LFjc+SRR2bixIlJkqFDh2bPPffMpEmT8rvf/S5nnHFGhgwZkv333z91dXXZeeed89BDD2XChAn1z/XSSy/l7bffzi677PKNagSgsppWugCAIuyxxx7Zbrvt6r9/7733sv/+++fnP/95/bYOHTpkl112yfPPP59ll112to/z2Wef5bbbbkvHjh2TJC1btkzv3r3z1FNPZdttt/1Wx/zxj39M69atc9VVV2XhhRdOkqy00krZY489vvI1ffrpp3nkkUdy+OGHJ0maN2+e7bffPnfddVdOOOGEtGrVKkly++2357333ssdd9yRLl26JEl69OiRnXfeOYMHD05tbW3uueee/PKXv8w+++yTJNloo43ywQcf5Omnn87OO+/8lXV82SGHHJLNN9+8/vsxY8bk6KOPbtBBadGiRQ4//PC88cYb6d69ey699NIsssgiGThwYJo3b54kadeuXY466qi88cYb2XXXXXPllVfm/vvvz6677lr/ujp27JgePXp8o/oAqCwBBFggdO7cucH3M0ebPvvss7z77rt5991360eCpk6dOsfHWXzxxeuDRDLjj+QkmTRp0rc+5qmnnkqvXr3qw0eSdO/ePe3bt//K13TXXXdl2rRp2XLLLTN+/Pgkybbbbpsbbrghf//73+sDzLPPPpsOHTrUh49kRli59957k6S+Y7LNNts0ePzf//73X/n8c7Lqqqs2+P78889PMuMC/WHDhuWdd96pv1Zn5rl+7rnn0qtXr/rwkSTdunVrcE3PuuuumzvvvDO77rprampqMmjQoPTp0ydVVVXfqk4AKkMAARYISyyxRIPvhw8fnl//+td56qmn0rRp06y00kr1IeWr1ub4/yEhSf0fv7W1td/6mE8++WSW+pJkySWXnONjJjOuuaitrc32228/y89uvPHG+gDy6aefzvbxZ/r000+TzHqOvq0vP84rr7yS008/Pa+88kpatGiRlVdeuT5czTzXX1djkuy222455ZRT8sEHH+Sll17K+PHjrfgFMB8SQIAFTm1tbQ466KA0a9YsN998c7p27ZqmTZvmv//9b+66667C62nXrl0+/vjjWbZ//PHHWXHFFWd7zJAhQ/L666/nF7/4RdZff/0GP3vkkUfypz/9KS+99FLWWmuttGnTZrb3JXnhhRfSunXrtG3bNsmMIDSzO5Mkb7/9dj755JP6Eafp06c3OH7m9RtfZcKECTnwwAPTuXPn3H333enUqVOaNGmSRx99NPfff3/9fm3atGmwrPJMjz76aLp06ZKll1462223Xfr165f7778/L7zwQjbaaKM5jsoBMO9yETqwwBk7dmzeeeed7LbbbunWrVuaNp3xWcxjjz2W5Ku7GXPDeuutl8ceeyxTpkyp3/b6669/5c0Mb7nlllRXV2fffffNBhts0ODrgAMOyEILLZQbb7wxyYzrPd5777288cYb9cfX1NTk8MMPz80335x11103SfLQQw81eI4BAwbkzDPPTJK0bt06o0aNavDz559//mtf29tvv51PP/00++yzT1ZZZZU0aTLj186Xz3WPHj3yr3/9KzU1NfXHvvHGGznooIPyyiuvJJlx7cwPf/jD3H333fnXv/6l+wEwn9IBARY4SyyxRNq3b5+//vWvadeuXdq2bZvHH388f/7zn5N89fUcc8MhhxySQYMG5cADD8z++++f8ePH58ILL0xVVdVsr2+oqanJPffck169eqVNmzaz/HyppZbKJptskkGDBuXkk0/OLrvskmuvvTaHHnpojjzyyCy++OL561//msmTJ2fvvfdOx44ds91226V///6ZPHlyVl999Tz++ON58MEH668D2WKLLfLII4/krLPOytZbb53nnnuupBWyVlxxxbRu3TqXX355mjZtmqZNm+b+++/PLbfckuR/5/qwww7L7rvvnr59+6ZPnz6pqanJhRdemNVXXz2bbbZZ/ePttttu2X333dO6det8//vf/xZnG4BK0wEBFkiXXnppll566Zx00kk56qij8uKLL+ayyy7LSiutlGeffbbQWpZffvlcffXVmTJlSo444ogMGDAgffv2zZJLLlm/ktX/99BDD+XTTz/NDjvsMMfH/PGPf5zJkyfn9ttvT+vWrXPdddele/fuOeuss3LkkUdmypQpufbaa+svjj/vvPOyzz775Nprr83BBx+cxx9/PL///e/rVw7bdddd07dv3wwaNCh9+/bN888/nwsvvPBrX1ubNm1y6aWXpq6uLkceeWROOOGEfPDBB7nuuuvSqlWr+nPdtWvXXHvttamtrc3RRx+dM844I2uvvXauvPLKBneUX3vttbPYYotl++23T4sWLb7ReQZg3uBO6AAV9uSTT6ZZs2YNlpMdN25cNtlkk5xwwgn1S+OSvPzyy/nJT36SW2+9NWussUalywHgWzCCBVBhr732Wi666KIcc8wxWX311TN27NgMHDgwbdq0+coux4Lk6aefztNPP5077rgjG264ofABMB8TQAAqbP/9909NTU1uuOGGjBw5Mi1btsz666+fc845J4svvnily5snjB07Ntdcc01WXnnlnH322ZUuB4DvwAgWAACQSy+9NE8++WSuvfbaOe4zduzY9OvXr341w+222y4nn3xyWrZsWfLzuAgdAAAWcH/6059y0UUXfe1+RxxxRN577736/Z944omcfvrp3+i5jGABAMACavTo0fnlL3+Z5557bo43v53phRdeyDPPPJNBgwalU6dOSZIzzjgjBx54YI455pgsvfTSJT2nDggAACygXnvttSyyyCK56667stZaa33lvs8++2yWXHLJ+vCRJOuvv36qqqry3HPPlfycOiAAADAf22qrrb7y5w8//PAcf7bllltmyy23LOl5Ro8enWWWWabBturq6iy66KIZOXJkSY+RLGABZLu1f13pEhq1qs8nV7qExm/C55WuoFGr6dy+0iU0es0+9m94bpv+2puVLqFRa7r8cpUuodG7950LKl3CHNWOWrXSJcxBMf8uJ02a1ODmsDM1b948U6ZMKflxFqgAAgAAjc1XdTjKqUWLFqmpqZll+5QpU6yCBQAAlFe7du0yZsyYBttqamry6aeflnwBeiKAAABASWrn0f8UZb311suoUaMybNiw+m1PP/10kmSdddYp+XEEEAAAYBbTp0/Phx9+mMmTZ1znu9Zaa2WdddbJ0UcfnZdffjlPPfVUTjvttOy88846IAAAwHczcuTI9OzZM4MGDUqSVFVV5Q9/+EM6dOiQPn365Kijjspmm22W3/zmN9/ocavq6urq5kK98ySrYM1dVsEqgFWw5iqrYM19VsGa+6yCNXdZBWvum5dXwZoycqVKlzBbzZd5u9IlfCM6IAAAQGEEEAAAoDDuAwIAACWozQJz5cJcpQMCAAAURgABAAAKYwQLAABKUORN/xozHRAAAKAwAggAAFAYI1gAAFCC6QvO/bvnKh0QAACgMAIIAABQGCNYAABQAjciLA8dEAAAoDACCAAAUBgjWAAAUILpRrDKQgcEAAAojAACAAAUxggWAACUwCpY5aEDAgAAFEYAAQAACmMECwAASjC9zghWOeiAAAAAhRFAAACAwhjBAgCAEtRWuoBGQgcEAAAojAACAAAUxggWAACUYLobEZaFDggAAFAYAQQAACiMESwAACjBdBNYZaEDAgAAFEYAAQAACmMECwAASuBGhOWhAwIAABRGAAEAAApjBAsAAEowPVWVLqFR0AEBAAAKI4AAAACFMYIFAAAlqHUjwrLQAQEAAAojgAAAAIUxglWQdTdeOX1+vlU6rrRkxo2dmEG3DM5NA/81x/2bNl0ou+6zcbb+0dpZst0i+Wj0+Dwy6OXcPPBfmTZt+iz7t2zVPJf97bBcd/k/8uBdL87FVzJvWnfTVbPP0dul48pLZdwnn2fQDU/l5j/+o6RjV169fQb87Rc5YJtzM+b9sUmSpdovlj//8+Q5HvPArYMz4KS/laX2+cW6W3TNPifukI6rLpNxH0/IoL/8Kzdf/EBJx67cbbkMuPv4HLDxbzJmxCdz3O+g03fNjw/aMj9Y5uflKnu+sd76K2W/A3pl+eW/l3GfTszf73o+N1z/5Bz3b9q0SX7y0w3y/e26Zckl2+SjDz/Lww+9lhuu/3emTfvfrbKW67hEDjp4i6y19vKZPm16Xn75vVx+6cMZOfLTAl7VvG/dTVZJn8O3+eK9+fMMuvmZ3HT1Y3Pcv2nThbJrn02y9U7rZMmlF8lHo8flkUEv5earHpvte3Nj1mPbtbPfmXukY9cOGffh+Nz9xwdy4+/u+Mpjttpr0+xx0o+zzEpLZ8zwj/K3/nfm3qsfabDPxjutl71O3S3LdV42n4z6NA9d92huPPuOTJs6rX6fU64/Klvsscksj3/WngPyz5v+XZbXNy9ad7Mu2efYH6TjKktn3CcTMuivT+bmyx4u6diV1+iQAbcdmQO2+G3977qZOqy0VA44eYd022DlTJs2Pa8+83auPOvOjHpvzu/XCyqrYJWHAFKA1dZaLr+58Gd57P5X8+dLHs4a3ZdPn19slaomVbnxqtn/ojv4hB9kmx+tneuveDRvvvZ+Vl5tmfQ+ZIssvcwiGXD6nQ32bd124Zx+4c+y9LKLFfFy5jmrdV8+p12+bx4b9FL+MuC+rN5jxfQ5Zts0aVKVGy975CuPXbHLMjn9yv3TtNlCDbaP/XB8jv7JH2bZf4e9NspmP1wrD/xtcFlfw7xutR4r5rQ/HZzH7no+fznn7qy+/krpc9KPZpzjC+//ymNX7No+p1972Czn+MvW2HDl7HjA5mWsev7RdfX2OfOsn+Sf/xiSa65+NGusuVz2P3DzVDWpyvXXzf6PqcN+sU2+v+2aue7aJ/LG0JFZZZWls8++m2bppdum/3mDkiRLLtkmF128d95775P8tt+dqa5umv0P6JVz+u+RA/e7KjU102b72AuK1dbqmN9c3DuP3fdK/nzxg1ljneXT54htUtWkSW688p+zPebgE7fPNjt2z/VX/CNvvjoiK6+2bHofulWWXmbRDDjt9mJfQAV13WjVnHHniXn0pn/nml/dmDV6dsl+/fZMkyZNcv1vb5vtMZvttmFO+PMvcvtFg/LsfS9m453XzzFXHpopk2ryyPWPJ0nW2bpbTrv1uDx6079z9cl/zYprdsz+Z+2ZRZdcJH84/Or6x+q09gp56NrHctel9zV4jhH/GTn3XnSFrbbOCjntyv3z2D0v5i/n35vV11sxfY77wYz34Use+spjV1xt2Zw+8MDZvg9/b5lFc/4th2fE22NyzlHXpbpFs/Q59gc56y+H5NDtzkvNlKlz6yWxABNACtD74C3y9hujct6pM96Un/v3f7NQ0yb56X6b5rZr/52aKQ3/CGjdduFsv1uPDLzwwdzy5yeSJC8+83aS5MCjt83Aix7MuLETkyQbbt4lh57wwyzcsrrAVzRv2evwrfP26x+k//E3JUme+9ebMz4dPmiL3DbwsVnOb5I0bbZQdtx7k+x91PdTM3nWN9epNdMz9MXhDbatskaHbPbDtfLnC+7La8+9O1dey7xqr2N/mLdfG5H+h/85SfLcP4akadOF8pNffD+3/fGR2Z7Dps0Wyo77b569T9whNZNqvvLxmy9cnWMG9M4no8ZlyfYLXpDep8+meeu/o/O73/49STL4mbfTdKEm2fNnG+WWm5+ZJSi0adMiP9pxnVz5x0dy801PJ0leeP7dJMnBh26VK6/4Z8aNm5g++22WiZNqcvyx12fKF/8/GDXq05x51k/SufMyeeWV94p7kfOg3odumbeHjsp5p9ySJHnuif9koaYL5acHbJbb/vL47N+bf7peBg64P7f8acYfzC8+/cV787E/yMDf31//3tzY7f3rn+StF9/NOX0uTpI8e/+Ladpsoex+4s655YK7UzN51v/P73vmnvnXLU/l8mNmvI88+8BLabNY6+zzm93rA8i2+26RMcM/yu/2vji1tbV5/qGXs+hSi2SXo7bPZUf/KdOnTU/zhavTfpVlcuPvbs/rT/+nuBddYXsd+f0Zv+uOuT5J8txjQ2e8Dx+yZW676tHZBoWmzRbKjn16Zu9jfjDb9+kk2fuobTPp8yk5pfflmfLFPqPf+zinXXlAVunWIa8NfmfuvSgWWK4BmcuaNVsoa/ZYIU88/HqD7Y8/9FpatmqeNdZZfpZjWrVunntueTZP/XNog+0jhn2cJGnXfvEZ+7VpkV+dv0defu6d/PKwv8ylVzBva1a9ULpt0ClPPPBqg+2P3/dKWrZunjXWW3G2x63Xq0v2Onzr3HTZIxl43r0lPdfPf7Nz3ntrTG6/Zs6jc41Rs+qm6bbRKnli0IsNtj9+9wtp2bpF1tig02yPW2+r1bPXsT/ITRfel4Fn3TnbfWbqe9ou+WTM+Dx405xHjhqrZs0Wylprd8y//vVGg+2PPTo0LVs2z5rdlpvlmFatmufvdz2ff/+74R9fI74Yb1tm2UWTJJtu1jn33vNSffhIkjffGJXdd7t4gQ8fzZotlDXXWzFPPPxag+2PP/DqF+/NK8xyTKs2LXLPzYPn/N7cYfG5Vu+8pFl103TbfPU8fvvTDbY/dstTadlm4ay5aZdZjll6+SWzXOdlZznmX7c+mfYrt0v7VZaZ8djNm2by51NSW/u/McJxH41PdfNmadlm4STJSt2Wz0ILNclbL75b5lc275rxu27lPHHfyw22P37vSzPeh9efw++6zVfLXkdsm5sueSgDz7l7tvtsvF233H/z0/XhI0n+88qI9N7wdOFjNqanap78mt8IIHNZuw6Lpbq6ad4f9lGD7R8Mn/GHQvuOS8xyzOgPPs0lv727/pfaTJtstVqmTp1W/1hTJk3NwbtcnPN/dXvGf7pgfOr2Ze2WWyLNqpvm/Xe/dH6/OHftV1hytse9+cp76bP52bnxskcyffrXz21v/qO103mtjrm8312pXcDW4Gu3/BJp1rxZ3n97TIPtH7z7YZKk/UpLz/a4N18clj7r/zo3Xnh/pv+/axK+rPtmXbLVT9bPgKOvW+DObZIss8yiqa5umhFfmrV+/4sZ7Q7LzfpH7ahR43LR7++f5ZhNN+2cqVOnZ8R7n6Rdu0XSunWLjBo1LkccuW1uv/Oo3PvACen3259kqaXazr0XNJ9o12HxGe/NX37veG/me8f3Zjlm9Ptjc8lZd2XEl47ZZKuuDd6bG7tlVlo61c2b5f03P2iw/YP/jkqSdFh12VmO6bha+yTJiDcbjki9X3/MjABy5yX3pf0qy+Qnx+2YVou0zGobrJJdjtw+T9/zfD4bOyHJjPGrJNnhkO/npg+uzKDJN+SCR89Il/VXLt+LnMe0W26JNGveNO+/82GD7R988W+x/YpLzfa4N19+L3027ZcbL3lotu/DS3dYPK3bLpzRIz7JYWfskpuePzN3Dj0nv7nqgCz5xQcZMDdUfARr2rRpeeCBB/Lss8/mgw8+SE1NTRZeeOG0a9cuPXr0yDbbbJOmTSte5rfW+otPbCZ+PqXB9okTZ7SnW7ZuUdLjbLJV12y1/Vq54/qnMuGzyUmSadOmzxJSFjSt2sw4fxMnTG6wfeb5btm6+WyP+3j0+G/0PLse0CuvPftOXvliFG5B0qptyyTJxM++dI4nfHGO28z+3/DHo8Z97WO3bNMiR12wV649955ZAs6CovUX7wETJ37pPWLSjO9btZz9v+Ev23Szztn6+2vmtlsHZ8KEyWnfYcYoW9+Dt8gbQz9IvzPvzGKLtcwBfbfI+QP2St8DrsrkOYxkLAhat/3ivH/5vfnzL96bW5V23jfZevVs9aO1c8dfn8yE8ZO//oBGoNWirZIkn4+f1GD7xM9mfN+y7cKzHNP6i2MmfumYSV+8r7T84n3mpX++lpvPuzMHnbt3Djp37yTJf55/O7/d68L6Y2YGkOYtqnPWngPSdok22ePEnXPeI7/JERudkndeaTg+2xi0+uKcfvPfdV/9PrzIEq2TJPufuEPeeGl4zjny2iyyRJvsd/wPc871h+XQH/TPlK8ZoYVvo6J/2Q8fPjx9+/bN6NGj07Vr1yy11FJZZJFFMmXKlLz++uu59dZbc/HFF+eqq67KssvO+onK/KCqyYy2WN0cPtitK+ET355bd80Jv90trzw3LNdc+GA5y5vvNfni/GYOp7Ecn6h3XWf5rLx6+5x+yJ++82PNj77+HM+5u/F1Dj5jt3z0wae5/YqvXiygMfu694jaOf3g/9msV5ec/Msd8/JLw3PVFTNWf2v2xcWmY8d+ntN+dWv947///tj84dJ9s/U2a+Tuv7/w3V/AfKqqauZ5n/35ndP2/6/nNqvnhN/9NK88+26uGfDVizE0Jv97T5j9OZrd++7//p03/FnVzIf64n3kyMsPyrb7bpHr+t2SFx5+JcusuFT2+c3uOfveX+aErU/PlEk1uXXAPXnsb0/mhUf+N3r7wsOv5E9vXpyfnbJrztpzwHd9ifOcufW7bub7xKcffZZ+h/yp/n+fke9+lAG3H5ktd143996w4I3GfpXauvlv3GleVNEAcvrpp6dDhw655ZZb0qZNm1l+Pn78+Bx99NE544wzcvnll1egwu/u85mfCH3p07SWX1w0/vmEr/7EbJfeG+eAo7+fl599N6cfdX2mTl2wlnn8OjM/cfzypz8zz/eXPy36Nnpu1y2ffToxgx8d+vU7N0ITxn3xb/hLnY6Z53zit/zUd/2t10ivndbNEdudm6omValKVf0v2SYLNUldbV1JfwTO7yZ88W+05ZcWkmi58Izz+/mEKbMc8//t9pP1c9AhW+alF4fnV6f+rf49YmaXdfDTbzX4O/H1IR/ks88mpdPKsx+dW1B8PvOT91Zf+nfdqrrBz+dkl302yQHHbJeXB7+T04+4boF6b57w6edJ/te1mGnmNRqfj5t1JHjCF2PCX+6OtPiiA/j5uIlZYtnF88MDt8oNZ9+eP/96xqIiLz86JG8MfitXvnJBttt/y9x5yX0Z8eYHGfGl8a/Px03Ma08MzUprzXpdZWMw4YvO0ZenJup/133Nv9c5mdlBGfzPoQ3eb4e+OCyfjZuYTl3nzw9/mfdVNIA899xzuemmm2YbPpKkbdu2Of7447PXXnsVXFn5fPDe2EyfNj3Ldmw4xz3z++Fvfzi7w5Ikh574w+y054Z59P5X0v/U2xaoX3ClGjn840yfNj3LLN9wXnvZ5WdcWzP8v999rGf9LVbLkw+99pXXMTRmI4d9OOMcf+l6mmW/+H74t1z2sucO3dN84er88dFTZ/nZPSMuzoM3PZULjrr2Wz32/OSDD8Zm+vTatP/S6l8zvx/2FdcV/OKI7+fHu/TIPx4ZknPO/nuD94iZj9us2axv802bLrTAL635wXufzP69ebkv3ju+YiTw0JN3yE4/2yiP3vdy+p9yywL33vzBW6NnnLuV2zXYPvP74UNGzHLMiDdmBIb2K7drcPF4+y+OGTZkRJbq+L00adIkrz3R8MOed197L+M+Gp/lV5+xIMPmu2+c8R9PyPMPNbwgu/nC1Rn/0Wff7cXNo0YO++J33ZeuTVr2i++H/3f0t3zcj2a8TzSf/fvElAV4TJO5q6IXobdt2zZjxnz1H4gffPBBWrQo7TqJedHUmml55flh2WTLrg2299x69Xw2flLeeHXWN+ok2e/wrbPTnhvmtmv/nbNP/NsC9wuuVFNrpuWVwe9kk++v0WB7z+3WzGfjJuaNl77bLHDrRRZO+xW+t8Atu/v/TZ0yLa889d9s8sO1GmzvuUP3fPbpxLzxwrBv9bjX9b8nR2x3ToOve6+bsRTnEdudk+v63/Oda58fTK2ZnpdfGp5NN+vcYPtmvbrks88mZejrH8z2uAP6bp4f79Ijt9z8dPqdcccs7xGTJ03NK6+8l56bda4fs0iS7uuskIUXrs7LLy/Yq2BNrZmWV557N5tsvXqD7T2/v8aM9+ZX5vDefOT3s9PPNsptf3k8Zx9/0wL53jx1ytS8/Njr6fnjDRps32y3DfPZ2AkZ+sx/Zznmg7dG5YO3RmXTXTdqsH3TXTfKe298kDHDP8oH/x2V6dOmZ81NV2uwT4dVl80i32ubUe/M+CP7R4dumyMu7Zum/y9cL7Hs4ll9ky556dGGq5o1FlNrpuWVZ97OJtuu2WB7zx+sNeN33Yvf7nfd5Ik1eW3wjMdtVv2/94m1N14lC7dqbhWs2aj0aleNZRWsinZAdtttt5x88sk54ogjssEGG2SZZZZJdXV1ampqMnr06DzzzDPp379/dtttt0qW+Z3dcOWjOfuPffLL836a++94IV3XWi679dkkAy98MDVTpqVlq+bpuNKSGTnik4wbOzErdW6Xn+zXM2++9n4ee+DVdFmzQ4PHG/72h7NcOLkgu/HSh/PbP/fNKRf1zgO3DM5q6yyfXQ/slYHn3Tvj/LZuno4rL52Rwz/OuE8+/0aPvWLnGSuzfNtPlxqLG39/X3578+E55YoD8sCNT2a1Hitl18O2zsB+d6Zm8tS0bN0iHVdtl5HDPsq4jyeU9JhjRnwyy13R1996RpD8z3cMjvOb6659Iued/7P8+jc/zn2DXkrXNTrkp3tsmCv/+EhqaqalZcvqLL/C9/LB+59m3LiJ6bTyUtljz40ydOgH+ec/X89qXxqTGPbuR5k4sSZXX/HPnP/7vfLb3+2ev930VBZbvFX6HrRlhgx5P0/+e8G5f8Kc3HDFP3P2lfvll+fvkftvfz5d1+6Y3fbtmYED7v/fe3OnpTLyvY+/eG9eJj/Zf9O8+eqIPHb/q+nypSWSh781ZoF5b77+rFtzzoO/yq9uOib3XfNIum7cOT85bsdcddJfUzO5Ji3bLJzlu3bIB2+NzriPZiz68dd+t+b4a36e8Z98lifvejYb7dgjm+++cc7c/YIkM5bbve3Ce/KT43ZMkjz34MtZevkl0/vXP8noYR9m0JUz7vh93Zm35Oz7Ts1ptx6XOy+5L20Wb519TvtJPhs7IX/rf1dlTkgBbvzDg/ntdYfklEv2yQM3P5PV1l0hux60eQaec09qpkz94nddu4wc/tE3+l13zbn35Nwbfp4zBvbNrVf+M4t+r032P2mHDH1hWJ566NWvfwD4FqrqKjhkXVdXl0suuSTXXHNNJk6cdWa0VatW2WuvvXLkkUemSZPv3qzZbu1ff+fH+LY23mK17H3oFmm/wvfy8Zjx+ftNz+S2a2fc4bhbjxVy7lX75/xf35YH73oxex+6ZfY6ePM5PtYJBw7My8++22Db0ssumj8POqb+MSqh6vPKrQCz8Tarp/cR30+HlZbMR6PH5e7rnsxtA2fcZX7N9VfKuX89JOefeFMeuu25WY7depd1c+w5u6fP5mdnzBdLn8606Q+65ZSLeqfvtudlxFeMyxVmwjcLUOW08Q/WSu/jtk+HTkvlo1Hjcvc1j+W2P874g2DNjVbJubcdlfOPvDYP3fzULMdu/dMNc+yFe6fPer+aJXT8f3sd+8P0Pm77/GCZn8+11/FVajq3r8jzJskmPVfNvvttlg7LLZ6PPvosd93xXP528zNJkrXW7pgLft875/7u77n/vley736bZe8+Pef4WMccdV1e+uIT0a6rt88BB26eLqstmylTpuaJx9/M5Zc9/LXXlswtzT6u3L/h2dl4y67Z++db/e+9+YancttfZtwAtluPFXPuNQfm/FNvyYN3vpC9f75V9jpkyzk+1gn7XZWXn638J8bTX3uzkOfZZOf1s89vfpoOnZfNx+9/krsuvS+3XDDjXhPdenXN+f84Peftd0ke+PM/64/Z/qCt85Njd8ySyy2RkW+PyY2/uz0PXfdYg8f98ZE/zA4Hfz/tVlwqn4wcm+cefDnX/PKG+iCTzLhjeu9f7ZaVui2f2traPHv/S7nyxOvy4XtzfynkpsvPem+eomz8/TXT++ht02HFpWb8rrv28dx21aNJkjU36JRzb/x5zj/uhjx06+BZjt161/VybP8906fnmbP8rlttnRXS57gfpvPaHTNlUk2efPDVXHXWXV97LdTccu87F1TkeUvx/PCOlS5httbpOH99cFfRADLT1KlT8/rrr2f06NGZNGlSWrRokXbt2qVLly6pri7fHb4rGUAWBJUMIAuMCgaQBUElA8iCYl4LII1RUQFkQVXJALKgmJcDyODhK1S6hNlar+O7lS7hG5knbrDRrFmzdOvWrdJlAAAAc5k7oQMAAIWZJzogAAAwr3MjwvLQAQEAAAojgAAAAIUxggUAACWYH2/6Ny/SAQEAAAojgAAAAIUxggUAACWYXuez+3JwFgEAgMIIIAAAQGGMYAEAQAlqfXZfFs4iAABQGAEEAAAojBEsAAAogRsRlocOCAAAUBgBBAAAKIwRLAAAKIEbEZaHswgAABRGAAEAAApjBAsAAEpQaxWsstABAQAACiOAAAAAhTGCBQAAJZjus/uycBYBAIDCCCAAAEBhjGABAEAJ3IiwPJxFAACgMAIIAABQGCNYAABQglqf3ZeFswgAABRGAAEAAApjBAsAAEowva6q0iU0CjogAABAYQQQAACgMAIIAABQGNeAAABACab77L4snEUAAKAwAggAAFAYI1gAAFCC2jqf3ZeDswgAABRGAAEAAApjBAsAAEpgFazycBYBAIDCCCAAAEBhjGABAEAJptdVVbqERkEHBAAAKIwAAgAAFMYIFgAAlKDWZ/dlsUAFkJPvuLHSJTRqk+sWqH9OFTG5rrrSJTRqk2ubVbqERm+5Zh9XuoRG77PaFpUuoVGbntcrXQLM98Q4AACgMD6yBgCAEkyv89l9OTiLAABAYQQQAACgMEawAACgBLVxI8Jy0AEBAAAKI4AAAACFMYIFAAAlsApWeTiLAABAYQQQAACgMEawAACgBNN9dl8WziIAAFAYAQQAACiMESwAAChBbZ0bEZaDDggAAFAYAQQAACiMESwAACiBVbDKw1kEAAAKI4AAAACFMYIFAAAlqK3z2X05OIsAAEBhBBAAAKAwRrAAAKAE0+NGhOWgAwIAABRGAAEAAApjBAsAAEpgFazycBYBAIDCCCAAAEBhjGABAEAJrIJVHjogAABAYQQQAACgMEawAACgBFbBKg9nEQAAKIwAAgAAFMYIFgAAlGC6EayycBYBAIDCCCAAAEBhjGABAEAJat2IsCx0QAAAgMIIIAAAQGGMYAEAQAmsglUeziIAAFAYAQQAACiMESwAAChBbZ1VsMpBBwQAACiMAAIAABTGCBYAAJRgus/uy8JZBAAACiOAAAAAhTGCBQAAJbAKVnnogAAAAIURQAAAgMIIIAAAUILaNJknv77Ta6qtzUUXXZRNN900a621Vvbff/8MGzZsjvt/+OGHOeaYY7LBBhtkgw02yJFHHplRo0Z9o+cUQAAAYAF16aWX5sYbb0y/fv1y0003paqqKn379k1NTc1s9z/66KMzcuTIXHPNNbnmmmsyatSoHHbYYd/oOQUQAABYANXU1GTgwIE5/PDD06tXr3Tp0iUDBgzI6NGj8+CDD86y//jx4zN48OD07ds3Xbt2TdeuXXPQQQfltddey9ixY0t+3oqvgrX33nunqqq0FQX+8pe/zOVqAABg9qY3slWwhg4dms8//zwbbrhh/ba2bduma9euGTx4cLbffvsG+zdv3jwtW7bMHXfckfXXXz9Jcuedd2aFFVbIIossUvLzVjyAbLTRRrn44ouz0korpVu3bpUuBwAAFggzr91YZpllGmxfaqmlMnLkyFn2b968ec4666ycccYZ6dGjR6qqqrLkkkvmuuuuS5MmpQ9WVTyAHHbYYWnZsmUuuuii/PGPf0yHDh0qXRIAAMw3ttpqq6/8+cMPPzzb7ZMmTUqSVFdXN9jevHnzjBs3bpb96+rq8sYbb6R79+458MADM3369AwYMCA///nPc8MNN6R169Yl1TtPXAOy7777Zp111snvf//7SpcCAACzVVtXNU9+fVstWrRIklkuOJ8yZUoWXnjhWfa/5557cv311+e8887Luuuum/XXXz+XX3553n///dx6660lP2/FOyAznXXWWRkyZEilywAAgPnKnDocX2fm6NWYMWPSsWPH+u1jxoxJly5dZtn/ueeey4orrtig07HIIotkxRVXzLvvvlvy884THZAkWXrppbPFFltUugwAAFggdOnSJa1bt87TTz9dv238+PEZMmRIevToMcv+yyyzTIYNG5YpU6bUb5s0aVJGjBiR5ZdfvuTnnWcCCAAAzMtq65rMk1/fVnV1dXr37p3+/fvn4YcfztChQ3P00UenXbt22WabbTJ9+vR8+OGHmTx5cpJk5513TpIcddRRGTp0aP3+1dXV2WWXXUp+XgEEAAAWUEcccUR22223nHrqqdlzzz2z0EIL5eqrr051dXVGjhyZnj17ZtCgQUlmrI51/fXXp66uLn369Ml+++2XZs2a5YYbbkjbtm1Lfs6qurq6urn1guY1j767aqVLaNQm180zlxQ1WpPrqr9+J761ybXNKl1Co7dcs48rXUKj91lti0qX0KhN99ntXPfDFV+tdAlzdPjze1W6hNm6eJ2/VrqEb8RfjAAAUILpaVw3IqwUMR4AACiMAAIAABTGCBYAAJTgu9z0j//RAQEAAAojgAAAAIUxggUAACX4Ljf943+cRQAAoDACCAAAUBgjWAAAUIJaNyIsCx0QAACgMAIIAABQGCNYAABQguluRFgWOiAAAEBhBBAAAKAwRrAAAKAEbkRYHs4iAABQGAEEAAAojBEsAAAoQa1VsMpCBwQAACiMAAIAABTGCBYAAJSgNkawykEHBAAAKIwAAgAAFMYIFgAAlMAqWOWhAwIAABRGAAEAAApjBAsAAEpQW+ez+3JwFgEAgMIIIAAAQGGMYAEAQAmsglUeOiAAAEBhBBAAAKAwRrAAAKAEtTGCVQ46IAAAQGEEEAAAoDBGsAAAoARWwSoPHRAAAKAwAggAAFAYI1gAAFACI1jloQMCAAAURgABAAAKYwQLAABKYASrPHRAAACAwgggAABAYRaoEaxNWtRWuoRGrqbSBTR6TTKt0iXAd7RA/dqpiKl1kytdQqM2ttb5XZAZwSoPHRAAAKAwAggAAFAYvXAAAChBbYxglYMOCAAAUBgBBAAAKIwRLAAAKIFVsMpDBwQAACiMAAIAABTGCBYAAJTACFZ56IAAAACFEUAAAIDCGMECAIASGMEqDx0QAACgMAIIAABQGCNYAABQAiNY5aEDAgAAFEYAAQAACmMECwAASlBnBKssdEAAAIDCCCAAAEBhjGABAEAJamMEqxx0QAAAgMIIIAAAQGGMYAEAQAnciLA8dEAAAIDCCCAAAEBhjGABAEAJ3IiwPHRAAACAwgggAABAYYxgAQBACayCVR46IAAAQGEEEAAAoDBGsAAAoARWwSoPHRAAAKAwAggAAFAYI1gAAFACq2CVhw4IAABQGAEEAAAojBEsAAAoQV1dpStoHHRAAACAwgggAABAYYxgAQBACWpjFaxy0AEBAAAKI4AAAACFMYIFAAAlqHMjwrLQAQEAAAojgAAAAIUxggUAACWoNYJVFjogAABAYQQQAACgMEawAACgBHV1la6gcdABAQAAClPxAPLOO+/k4osvTr9+/fLoo4/O8vMJEybk5JNPrkBlAABAuVU0gDz33HP58Y9/nLvvvjuPPfZYDjnkkBx++OGpqamp32fy5Mm54447KlckAABkxo0I58Wv+U1FA8j555+f3XbbLffff38eeOCBXHDBBXniiSdyyCGHZOrUqZUsDQAAmAsqGkDeeOON9O7du/77H/zgB7nyyivzwgsv5IQTTqhgZQAAwNxQ0QDSunXrjB07tsG2ddddN+edd17uv//+nH322RWqDAAAGqr0qJURrDLo1atXzjjjjLz00ksNRq623nrrnHLKKfnzn/+cM844o4IVAgAA5VTRAHLsscdmscUWyx577JEnn3yywc969+6dX//613nkkUcqVB0AAFBuFb0R4SKLLJKBAwdm+PDhWWyxxWb5+c9+9rNstNFGeeCBBypQHQAA/E/tfDjuNC+aJ+6E3rFjxzn+bMUVV8zBBx9cYDUAAMDcUvEbEQIAAAuOeaIDAgAA87q6ukpX0DjogAAAAIURQAAAgMIYwQIAgBLMjzf9mxfpgAAAAIURQAAAgMIYwQIAgBIYwSoPHRAAAKAwAggAAFAYI1gAAFAC9yEsDx0QAACgMAIIAABQGCNYAABQAqtglYcOCAAAUBgBBAAAKIwRLAAAKIVlsMpCBwQAACiMAAIAABTGCBYAAJTAKljloQMCAAAURgABAAAKYwQLAABKUGcVrLLQAQEAAAojgAAAAIUxggUAACWwClZ56IAAAACFEUAAAIDCGMECAIBSGMEqCx0QAACgMAIIAABQGCNYAABQAjciLA8dEAAAoDACCAAAUBgjWAAAUAojWGWhAwIAABRGAAEAAApjBAsAAEpQ50aEZaEDAgAAFEYAAQAACmMECwAASmEVrLLQAQEAAAojgAAAAIUxggUAACWwClZ56IAAAACFEUAAAIDCGMECAIBSWAWrLHRAAACAwixQHZAm8hYAjdxCVX7XzU1TfQQO35l3KQAAKEnVPPr17dXW1uaiiy7KpptumrXWWiv7779/hg0bNsf9p06dmvPPPz+bbrpp1l577fTu3Tuvv/76N3pOAQQAABZQl156aW688cb069cvN910U6qqqtK3b9/U1NTMdv/f/OY3ueWWW3LmmWfm1ltvzaKLLpq+ffvms88+K/k5BRAAAFgA1dTUZODAgTn88MPTq1evdOnSJQMGDMjo0aPz4IMPzrL/e++9l1tuuSVnn312Nt9883Tq1Cm//e1vU11dnVdffbXk5xVAAACgFHXz6Ne3NHTo0Hz++efZcMMN67e1bds2Xbt2zeDBg2fZ//HHH0/btm2z2WabNdj/kUceyUYbbVTy8wogAACwABo1alSSZJlllmmwfamllsrIkSNn2f/dd9/NcsstlwceeCC77LJLNtlkk/Tt2zdvvfXWN3reBWoVLAAAaGy22mqrr/z5ww8/PNvtkyZNSpJUV1c32N68efOMGzdulv0nTJiQ4cOH59JLL80JJ5yQtm3b5rLLLsvPfvazDBo0KEsssURJ9eqAAABAKSo9alXmEawWLVokySwXnE+ZMiULL7zwLPs3a9Ysn332WQYMGJCePXumW7duGTBgQJLk9ttvL/l5dUAAAGA+NqcOx9eZOXo1ZsyYdOzYsX77mDFj0qVLl1n2b9euXZo2bZpOnTrVb2vRokWWW265jBgxouTn1QEBAIAFUJcuXdK6des8/fTT9dvGjx+fIUOGpEePHrPs36NHj0ybNi2vvPJK/bbJkyfnvffey/LLL1/y8+qAAABAKeq+203/5jXV1dXp3bt3+vfvn8UXXzzt27fPeeedl3bt2mWbbbbJ9OnT88knn6RNmzZp0aJFevTokY033jgnnnhizjjjjCy66KK56KKLstBCC2WnnXYq+Xl1QAAAYAF1xBFHZLfddsupp56aPffcMwsttFCuvvrqVFdXZ+TIkenZs2cGDRpUv//FF1+c9ddfP7/4xS+y2267ZcKECfnLX/6SxRdfvOTnrKqrq/sOl67MX2pHrVrpEgBgrqpNbaVLaNRGTv+80iU0esu1n3X513nFCtecW+kSZuvd/U6odAnfiBEsAAAowYLzsf3cZQQLAAAojAACAAAUxggWAACUwghWWeiAAAAAhRFAAACAwhjBAgCAUjSyGxFWig4IAABQGAEEAAAojBEsAAAoQZVVsMpCBwQAACiMAAIAABTGCBYAAJTCCFZZ6IAAAACFEUAAAIDCGMECAIBSuBFhWeiAAAAAhRFAAACAwhjBAgCAUlgFqyx0QAAAgMIIIAAAQGGMYAEAQCmMYJWFDggAAFAYAQQAACiMESwAACiFEayy0AEBAAAKI4AAAACFMYIFAAClqKuqdAWNgg4IAABQGAEEAAAojBEsAAAoQZVVsMpCBwQAACiMAAIAABTGCBYAAJTCCFZZ6IAAAACFEUAAAIDCCCAAAEBhBBAAAKAw3+gi9A8//DCXXHJJ3nvvvXzve9/LaqutljXWWCOrr756Fl544blVIwAA0Eh8owByyimn5PHHH88qq6ySESNG5O9//3vq6urSpEmTrLTSSlljjTWy5pprZs0110yXLl3SrFmzuVU3AAAUyo0Iy+MbBZAXXnghxx9/fPbff/8kycSJE/Paa6/llVdeySuvvJLBgwfn9ttvT5JUV1fn5Zdf/trHnDJlSv7zn/9k5ZVXTosWLfL666/nuuuuy+jRo7PKKqukT58+adeu3bd4aQAAwLzmGwWQ5s2bp2vXrvXft2zZMuutt17WW2+9+m2ffvppXn755bz66qtf+3hvvfVW9t1333z44YdZdtll069fvxx22GHp0KFDOnXqlIceeii33XZbrr/++nTq1OmblAoAAMyDvtFF6FtvvXWGDBnylfssuuii2WyzzXLYYYd97eOde+656d69e+64446su+66OfTQQ/OjH/0of//733PhhRfm3nvvzSabbJKzzz77m5QJAADlV1c1b37NZ75RANl1111z77335r///W9ZnvyZZ57JUUcdlS5duuTEE0/MlClTsueee6aqasaJbNq0aQ455JA899xzZXk+AACgsr7RCNZPf/rTVFVV5Sc/+Um22267bLrppll99dWz/PLLf6snb9GiRSZPnpwk+d73vpef/vSnad68eYN9xo8fnzZt2nyrxwcAAOYt3yiA9OvXL6+//npee+213Hvvvbn99ttTVVWVVq1apWvXrlljjTVywgknlPx4PXv2zJlnnpl+/fqlU6dOOeOMM+p/VldXl2eeeSann356tt56629SJgAAlJ9VsMqiqq6u7ludytra2rz11lt57bXX8uqrr2bIkCEZOnRonn/++ZIf45NPPskhhxyS5ZZbLueff36Dn91zzz059thjs+mmm2bAgAFp3br1tymzYc2jVv3OjwEA87La1Fa6hEZt5PTPK11Co7dc+5GVLmGOVvr9BZUuYbbePuqYSpfwjXzrADI7dXV19ddvfBOffvppFl100QbbPvnkk4wZMyZdunQpU3UCCACNnwAydwkgc58A8s3NbwHkG41gfZ1vEz6SzBI+kmTxxRfP4osv/h0rAgCAMjGCVRbfaBUsAACA70IAAQAAClPWESwAAGisqoxglYUOCAAAUBgBBAAAKIwRLAAAKIURrLLQAQEAAAojgAAAAIUxggUAAKUwglUWOiAAAEBhBBAAAKAwRrAAAKAEbkRYHjogAABAYQQQAACgMEawAACgFHVVla6gUdABAQAACiOAAAAAhTGCBQAApbAKVlnogAAAAIURQAAAgMIYwQIAgBK4EWF56IAAAACFEUAAAIDCGMECAIBSGMEqCx0QAACgMAIIAABQGCNYAABQAqtglYcOCAAAUBgBBAAAKIwRLAAAKIURrLLQAQEAAAojgAAAAIUxggUAAKUwglUWOiAAAEBhBBAAAKAwRrAAAKAEbkRYHjogAABAYQQQAACgMAIIAABQGAEEAAAojAACAAAUxipYAABQCqtglYUOCAAAUBgBBAAAKIwRLAAAKIEbEZaHDggAAFAYAQQAACiMESwAACiFEayyWKACyFNTplW6hEZt1LRFKl1Co9eqyZRKl9CotWkyudIlwHc2vc5ww9z0ypQulS6h0Tus0gUw13mXAgAACrNAdUAAAOBbM4JVFjogAABAYQQQAACgMEawAACgBG5EWB46IAAAQGEEEAAAoDBGsAAAoBRGsMpCBwQAACiMAAIAABTGCBYAAJTAKljloQMCAAAURgABAAAKYwQLAABKYQSrLHRAAACAwgggAABAYYxgAQBAKYxglYUOCAAAUBgBBAAAKIwRLAAAKIEbEZaHDggAAFAYAQQAACiMESwAACiFEayy0AEBAAAKI4AAAACFMYIFAAClMIJVFjogAABAYQQQAACgMEawAACgBG5EWB46IAAAQGEEEAAAoDBGsAAAoBRGsMpCBwQAACiMAAIAABTGCBYAAJTAKljloQMCAAAURgABAAAKYwQLAABKYQSrLHRAAACAwgggAABAYQQQAACgMK4BAQCAUrgGpCx0QAAAgMIIIAAAQGGMYAEAQAmqKl1AI6EDAgAAFEYAAQAACmMECwAASmEVrLLQAQEAAAojgAAAAIUxggUAACWoMoJVFjogAABAYebZAPKjH/0oI0eOrHQZAABAGVV0BOuOO+6Y48+GDRuWe++9N4svvniSZOeddy6mKAAAmB0jWGVR0QBy+umnZ/LkyUmSurpZ/xc999xzkyRVVVUCCAAANAIVDSC33XZbjjvuuLRp0ybnnHNOll566fqfde/ePXfddVeWW265ClYIAACUU0WvAVlxxRVz0003pVu3btlpp50yaNCgSpYDAABzVjePfs1nKn4RetOmTXPMMcfk4osvTv/+/XPsscfms88+q3RZAADAXFDxADLTeuutV39R+g477JCpU6dWtiAAAKDs5pkAkiRt27bN+eefn6OPPjrrrLNOmjdvXumSAAAgyYwbEc6LX99FbW1tLrroomy66aZZa621sv/++2fYsGElHfv3v/89nTt3zogRI77Rc85TAWSmnXfeOX/5y1+y1FJLVboUAABotC699NLceOON6devX2666aZUVVWlb9++qamp+crj3n///Zx++unf6jnnyQACAADMXTU1NRk4cGAOP/zw9OrVK126dMmAAQMyevToPPjgg3M8rra2Nscff3xWX331b/W8AggAAJSi0qtdlXkVrKFDh+bzzz/PhhtuWL+tbdu26dq1awYPHjzH4y6//PJMnTo1Bx988Ld63oreBwQAAPhuttpqq6/8+cMPPzzb7aNGjUqSLLPMMg22L7XUUhk5cuRsj3n55ZczcODA3HLLLRk9evS3qFYHBAAAFkiTJk1KklRXVzfY3rx580yZMmWW/SdOnJjjjjsuxx13XFZYYYVv/bw6IAAAUILvuuLU3DKnDsfXadGiRZIZ14LM/O9JMmXKlCy88MKz7N+vX7+ssMIK2WOPPb5doV8QQAAAYAE0c/RqzJgx6dixY/32MWPGpEuXLrPsf+utt6a6ujrdu3dPkkyfPj3JjHv47bjjjjnjjDNKel4BBAAAFkBdunRJ69at8/TTT9cHkPHjx2fIkCHp3bv3LPs/8MADDb5/6aWXcvzxx+eKK65Ip06dSn5eAQQAAEoxj45gfVvV1dXp3bt3+vfvn8UXXzzt27fPeeedl3bt2mWbbbbJ9OnT88knn6RNmzZp0aJFll9++QbHz7yIfdlll80SSyxR8vO6CB0AABZQRxxxRHbbbbeceuqp2XPPPbPQQgvl6quvTnV1dUaOHJmePXtm0KBBZX3Oqrq6ukaW5ebs38NWqnQJjdqoaYtUuoRGr1WTWVekoHzaNJlc6RLgO5te57PFuemVKctVuoRG77DO/6h0CXPU/bABlS5htl649OhKl/CNGMECAIASzKurYM1vfEwCAAAURgABAAAKYwQLAABKYQSrLHRAAACAwgggAABAYYxgAQBAKYxglYUOCAAAUBgBBAAAKIwRLAAAKIEbEZaHDggAAFAYAQQAACiMESwAACiFEayy0AEBAAAKI4AAAACFMYIFAAAlqKozg1UOOiAAAEBhBBAAAKAwRrAAAKAUJrDKQgcEAAAojAACAAAUxggWAACUoMoIVlnogAAAAIURQAAAgMIYwQIAgFIYwSoLHRAAAKAwAggAAFAYI1gAAFACq2CVhw4IAABQGAEEAAAojBEsAAAohRGsstABAQAACiOAAAAAhTGCBQAAJbAKVnnogAAAAIURQAAAgMIYwQIAgFIYwSoLHRAAAKAwC1QH5PTOG1W6BGAe1mSZpStdQqNXO3J0pUto/Kp8tjg3VTVboP50qojDxle6AuY2/y8CAIASWAWrPHxMAgAAFEYAAQAACmMECwAASlFnBqscdEAAAIDCCCAAAEBhjGABAEAJrIJVHjogAABAYQQQAACgMEawAACgFEawykIHBAAAKIwAAgAAFMYIFgAAlKCqttIVNA46IAAAQGEEEAAAoDBGsAAAoBRWwSoLHRAAAKAwAggAAFAYI1gAAFCCKiNYZaEDAgAAFEYAAQAACmMECwAASlFnBqscdEAAAIDCCCAAAEBhjGABAEAJrIJVHjogAABAYQQQAACgMEawAACgFEawykIHBAAAKIwAAgAAFMYIFgAAlMAqWOWhAwIAABRGAAEAAApjBAsAAEpRZwarHHRAAACAwgggAABAYYxgAQBACayCVR46IAAAQGEEEAAAoDBGsAAAoBRGsMpCBwQAACiMAAIAABTGCBYAAJTAKljloQMCAAAURgABAAAKYwQLAABKUWsGqxx0QAAAgMIIIAAAQGGMYAEAQClMYJWFDggAAFAYAQQAACiMESwAACiBGxGWR0U7ILfccktqamoabHvqqady0EEHZccdd8yxxx6b//73vxWqDgAAKLeKBpBf/epX+eyzz+q/f/zxx7PffvultrY2PXv2zIcffphdd901zz//fAWrBAAAyqWiI1h1dQ37WJdeemn22WefnHzyyfXbzj777PTv3z/XX3990eUBAMD/1JnBKod56iL0YcOGZaeddmqwbffdd8+QIUMqVBEAAFBOFQ0gVVVVDb5fYYUVMnHixAbbxo4dmzZt2hRZFgAAMJdUfARrq622yoorrphOnTqluro65513Xq677ro0a9Yszz//fE4//fT06tWrkmUCAIBVsMqkogHkkUceyRtvvJE333wzb7zxRj788MO8++67mT59epo1a5YDDjggnTt3zrHHHlvJMgEAgDKpaABZdtlls+yyy2aLLbao3zZ16tQ0a9YsSXLjjTdm1VVXnWVUCwAAmD/NczcinBk+kqRz584VrAQAAP4fI1hlMU+tggUAADRuAggAAFCYeW4ECwAA5kVVbkRYFjogAABAYQQQAACgMEawAACgFLWVLqBx0AEBAAAKI4AAAACFMYIFAAAlsApWeeiAAAAAhRFAAACAwhjBAgCAUpjAKgsdEAAAoDACCAAAUBgjWAAAUAqrYJWFDggAAFAYAQQAACiMESwAAChBlQmsstABAQAACiOAAAAAhTGCBQAApbAKVlnogAAAAIURQAAAgMIYwQIAgBJU1Va6gsZBBwQAACiMAAIAABTGCBYAAJTCKlhloQMCAAAURgABAAAKYwQLAABKYQKrLHRAAACAwgggAABAYYxgAQBACaqsglUWOiAAAEBhBBAAAKAwRrAAAKAURrDKQgcEAAAojAACAAAUxggWAACUorbSBTQOOiAAAEBhBBAAAKAwRrAAAKAEbkRYHjogAABAYQQQAACgMEawAACgFEawykIHBAAAKIwAAgAAFMYIFgAAlMIIVlksUAGkyYodK11Cozb9v+9UuoRGr2qhhSpdQuPm/M51TRZpW+kSGr3aceMrXUKjVjd1WqVLgPmeESwAAKAwC1QHBAAAvrXaShfQOOiAAAAAhRFAAACAwhjBAgCAElRZBassdEAAAIDCCCAAAEBhBBAAAChFXd28+fUd1NbW5qKLLsqmm26atdZaK/vvv3+GDRs2x/3/85//5KCDDsoGG2yQjTbaKEcccUQ++OCDb/ScAggAACygLr300tx4443p169fbrrpplRVVaVv376pqamZZd+xY8dmv/32S6tWrXLdddflyiuvzNixY3PggQdmypQpJT+nAAIAAAugmpqaDBw4MIcffnh69eqVLl26ZMCAARk9enQefPDBWfZ/6KGHMmnSpPzud7/LKquskjXWWCPnnXde3nrrrTz//PMlP68AAgAApaj0qFWZR7CGDh2azz//PBtuuGH9trZt26Zr164ZPHjwLPtvtNFGueSSS9K8efNZfjZu3LiSn9cyvAAAMB/baqutvvLnDz/88Gy3jxo1KkmyzDLLNNi+1FJLZeTIkbPs36FDh3To0KHBtj/+8Y9p3rx51ltvvZLr1QEBAIAF0KRJk5Ik1dXVDbY3b968pGs6/vKXv+T666/PMccckyWWWKLk59UBAQCAUsyjNyKcU4fj67Ro0SLJjGtBZv73JJkyZUoWXnjhOR5XV1eXCy+8MJdddlkOPvjg7Lvvvt/oeXVAAABgATRz9GrMmDENto8ZMybt2rWb7TFTp07N8ccfn8svvzwnnHBCjjnmmG/8vAIIAAAsgLp06ZLWrVvn6aefrt82fvz4DBkyJD169JjtMSeccELuu+++nH/++TnggAO+1fMawQIAgFLUVrqA8qqurk7v3r3Tv3//LL744mnfvn3OO++8tGvXLttss02mT5+eTz75JG3atEmLFi1y2223ZdCgQTnhhBOy/vrr58MPP6x/rJn7lEIHBAAAFlBHHHFEdtttt5x66qnZc889s9BCC+Xqq69OdXV1Ro4cmZ49e2bQoEFJkrvvvjtJcu6556Znz54NvmbuU4qqurp59GqaueAHq51c6RIaten/fafSJTR6VQstVOkSGrUmHZatdAmN32cTKl1Bo1c7bnylS2jcqnx2O7fdP+naSpcwR9ut/stKlzBb9712VqVL+EaMYAEAQAmqFpzP7ecqMR4AACiMAAIAABTGCBYAAJTCCFZZ6IAAAACFEUAAAIDCGMECAIBS1BrBKgcdEAAAoDACCAAAUBgjWAAAUAqrYJWFDggAAFAYAQQAACiMESwAACiFEayy0AEBAAAKI4AAAACFMYIFAAClMIJVFjogAABAYQQQAACgMEawAACgFLVGsMpBBwQAACiMAAIAABTGCBYAAJSirrbSFTQKOiAAAEBhBBAAAKAwRrAAAKAUbkRYFjogAABAYQQQAACgMEawAACgFG5EWBY6IAAAQGEEEAAAoDBGsAAAoBRWwSoLHRAAAKAwAggAAFCYio9gvfTSS3n66adz0EEHJUmeeuqp/OlPf8qIESPSsWPH7L///unRo0eFqwQAYIFnBKssKtoBue+++7LnnnvmmWeeSZL84x//yH777Ze6urr06tUrU6dOTZ8+ffKPf/yjkmUCAABlUtEOyB/+8If84he/yGGHHZYkueyyy3LIIYfkyCOPrN/nsssuy0UXXZQtttiiUmUCAABlUtEOyPDhw/OjH/2o/vsRI0Zk2223bbDPDjvskLfeeqvo0gAAoKG6unnzaz5T0QCy3HLL5dFHH63/frXVVsvQoUMb7PPyyy9n6aWXLro0AABgLqjoCFbfvn3zy1/+MqNGjcoOO+yQww47LCeddFKmTJmSVVZZJS+99FIuueSS/OIXv6hkmQAAQJlUNIDsvPPOqaqqykUXXZSrrroqVVVVqaury2mnnZYkadWqVQ488MDsu+++lSwTAACS2tpKV9AoVHwZ3p122ik77bRT3n777bz77ruZMGFCmjVrlnbt2qVr165p3rx5pUsEAADKpOIBZKaVVlopK620UqXLAAAA5qJ5JoAAAMA8bT5ccWpeVNFVsAAAgAWLAAIAABTGCBYAAJTCCFZZ6IAAAACFEUAAAIDCGMECAIBS1BrBKgcdEAAAoDACCAAAUBgjWAAAUIK6utpKl9Ao6IAAAACFEUAAAIDCGMECAIBSWAWrLHRAAACAwgggAABAYYxgAQBAKeqMYJWDDggAAFAYAQQAACiMESwAAChFrRsRloMOCAAAUBgBBAAAKIwRLAAAKIVVsMpCBwQAACiMAAIAABTGCBYAAJSgzipYZaEDAgAAFEYAAQAACmMECwAASmEVrLLQAQEAAAojgAAAAIUxggUAAKWoNYJVDjogAABAYQQQAACgMEawAACgFHVuRFgOOiAAAEBhBBAAAKAwRrAAAKAEdVbBKgsdEAAAoDACCAAAUBgjWAAAUAqrYJWFDggAAFAYAQQAACiMESwAACiBVbDKQwcEAAAojAACAAAUxggWAACUwipYZaEDAgAAFEYAAQAAClNVV1fncn4AAKAQOiAAAEBhBBAAAKAwAggAAFAYAQQAACiMAAIAABRGAAEAAAojgAAAAIURQAAAgMIIIAAAQGEEEAAAoDACCAAAUBgBBAAAKIwAAgAAFEYAmcfU1tbmoosuyqabbpq11lor+++/f4YNG1bpshqtSy+9NHvvvXely2hUPv300/z617/OZpttlnXWWSd77rlnnn322UqX1ah8/PHHOf7447Phhhume/fuOeigg/Lf//630mU1Su+88066d++e2267rdKlNCrvv/9+OnfuPMvX3/72t0qX1qjccccd+eEPf5g111wz22+/fe69995KlwRJBJB5zqWXXpobb7wx/fr1y0033ZSqqqr07ds3NTU1lS6t0fnTn/6Uiy66qNJlNDrHHHNMXnrppVxwwQW55ZZbsvrqq+eAAw7IW2+9VenSGo1DDz007733Xq688srccsstadGiRfbdd99MmjSp0qU1KlOnTs1xxx2XiRMnVrqURueNN95I8+bN869//SuPP/54/dePfvSjSpfWaNx555055ZRTsvvuu+fuu+/OD3/4wxxzzDF54YUXKl0aCCDzkpqamgwcODCHH354evXqlS5dumTAgAEZPXp0HnzwwUqX12iMHj06Bx54YC688MKsuOKKlS6nURk2bFieeOKJnHbaaenRo0dWWmml/PKXv8zSSy+du+++u9LlNQpjx45Nhw4dcuaZZ2bNNddMp06dcthhh+XDDz/Mf/7zn0qX16hcfPHFadWqVaXLaJTefPPNrLjiillqqaWy5JJL1n+1aNGi0qU1CnV1dbnwwgvTp0+f9OnTJ8svv3x+/vOfZ+ONN84zzzxT6fJAAJmXDB06NJ9//nk23HDD+m1t27ZN165dM3jw4ApW1ri89tprWWSRRXLXXXdlrbXWqnQ5jcpiiy2WK664ImussUb9tqqqqtTV1WXcuHEVrKzxWGyxxXLBBRdklVVWSZJ89NFHufrqq9OuXbusvPLKFa6u8Rg8eHBuuummnHPOOZUupVF64403/Hudi95+++28//77s3SUrr766hx88MEVqgr+p2mlC+B/Ro0alSRZZpllGmxfaqmlMnLkyEqU1ChtueWW2XLLLStdRqPUtm3b9OrVq8G2e++9N8OHD0/Pnj0rVFXj9atf/So333xzqqurc9lll6Vly5aVLqlRGD9+fE444YSceuqps7wfUx5vvvlmllxyyfzsZz/Lu+++m+WXXz6HHXZYNt1000qX1ii8++67SZKJEyfmgAMOyJAhQ9KhQ4cceuihfv8xT9ABmYfMnN+urq5usL158+aZMmVKJUqC7+S5557LKaeckq222sovvbmgT58+ufXWW7Pjjjvm5z//eV577bVKl9Qo/OY3v8naa6/teoS5pKamJu+++24mTJiQo446KldccUXWXHPN9O3bN08++WSly2sUJkyYkCQ58cQTs8MOO2TgwIHZZJNNcthhhznHzBN0QOYhM2dfa2pqGszBTpkyJQsvvHClyoJv5aGHHspxxx2XtdZaKxdccEGly2mUZo6wnHnmmXnxxRdz3XXX5eyzz65wVfO3O+64I88++2z+/ve/V7qURqu6ujqDBw9O06ZN6z9wW2ONNfLWW2/l6quvzkYbbVThCud/zZo1S5IccMAB+fGPf5wkWW211TJkyJBcc801zjEVpwMyD5nZ6h8zZkyD7WPGjEm7du0qURJ8K9ddd10OP/zwbLbZZrnyyitdWFpGH3/8ce6+++5Mnz69fluTJk3SqVOnWd47+OZuvfXWfPzxx9l8883TvXv3dO/ePUly2mmnZfvtt69wdY1Hy5YtZ+n2r7rqqhk9enSFKmpcZv7NsOqqqzbYvvLKK2fEiBGVKAkaEEDmIV26dEnr1q3z9NNP128bP358hgwZkh49elSwMijd9ddfnzPPPDN77bVXfv/738/yRwbfzZgxY3Lsscc2WMlm6tSpGTJkSDp16lTByhqH/v37Z9CgQbnjjjvqv5LkiCOOyBVXXFHZ4hqJoUOHpnv37rPcH+jVV191YXqZdO3aNa1atcpLL73UYPubb76Zjh07Vqgq+B8jWPOQ6urq9O7dO/3798/iiy+e9u3b57zzzku7du2yzTbbVLo8+FrvvPNOfvvb32abbbbJwQcfnI8//rj+Zy1atEibNm0qWF3j0KVLl/Ts2TOnn356+vXrl7Zt2+byyy/P+PHjs++++1a6vPne0ksvPdvtSyyxRNq3b19wNY3TqquumlVWWSWnn356TjvttCy22GK5+eab8+KLL+aWW26pdHmNQosWLXLggQfmkksuydJLL51u3brlnnvuyRNPPJE//elPlS4PBJB5zRFHHJFp06bl1FNPzeTJk7Peeuvl6quv9iky84X7778/U6dOzYMPPjjLvWt+/OMf53e/+12FKms8qqqq8vvf/z7nn39+jjrqqHz22Wfp0aNH/vrXv2bZZZetdHnwtZo0aZLLL788/fv3z1FHHZXx48ena9euueaaa9K5c+dKl9doHHbYYVl44YXr7yfWqVOnXHzxxdlggw0qXRqkqq6urq7SRQAAAAsG14AAAACFEUAAAIDCCCAAAEBhBBAAAKAwAggAAFAYAQQAACiMAAIAABRGAAEAAAojgADMZ6ZMmZKuXbume/fuOfPMMytdDgB8IwIIwHymqqoqf/7zn9OtW7dcd911eeeddypdEgCUTAABmM9UV1dnvfXWy4EHHpgkee211ypcEQCUTgABmE+ttNJKSZLXX3+9wpUAQOkEEID51JVXXpkkGTp0aIUrAYDSCSAA86HHH388N9xwQxZZZJEMGTKk0uUAQMkEEID5zPjx43PKKadkq622yp577plPPvkko0ePrnRZAFASAQRgPnP66adn2rRp6devX7p27ZrEGBYA8w8BBGA+ct999+Xuu+/OWWedlcUXX7w+gLgQHYD5hQACMJ/48MMPc9ppp2X33XfPFltskSRZbrnl0rZtW9eBADDfEEAA5hO/+tWvssgii+Skk05qsH211VYzggXAfEMAAZgP/O1vf8tjjz2Wc889Ny1btmzws65du2b48OGZMGFChaoDgNJV1dXV1VW6CAAAYMGgAwIAABRGAAEAAAojgAAAAIURQAAAgMIIIAAAQGEEEAAAoDACCAAAUBgBBAAAKIwAAgAAFEYAAQAACiOAAAAAhRFAAACAwvwf1EnqOseroygAAAAASUVORK5CYII=",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "filenames": {
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_262_0.png"
+ }
+ },
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyAAAANbCAYAAAC6lftqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABVn0lEQVR4nO3dd5hU9dkG4GdBlqWIilFREAsWxIqCLWI3sUWNmkQjBhtqsMQSbDFWjIm9a1TQGEUw1thrrDH22BCNRsGCgKAUQdru9wdCvhXQUYczsNz3de11Zc+cmfPucTPsM+97fqeqrq6uLgAAAAVoVOkCAACABYcAAgAAFEYAAQAACiOAAAAAhRFAAACAwgggAABAYQQQAACgMAIIAABQGAEEYC5yr1cAqG+hShcAUKTjjjsut91229fu07Zt2zzyyCPf+1h/+9vf8s477+S44477xn1vvvnm/O53v0u3bt1y9dVXf+9jA8C8qqrOx3PAAmTo0KEZPXr0zO8vu+yyDBo0KJdccsnMbdXV1enUqdP3PtaWW26Z9ddfP3/84x+/cd8999wz48ePz9tvv50HHnggyy677Pc+PgDMi3RAgAVK+/bt0759+5nft27dOtXV1VlnnXUqVtO7776bF198MVdeeWV69+6dm266KUcffXTF6gGAuck1IACz8dZbb+Wggw7Kuuuum3XXXTeHHHJI3n///Xr7/PWvf822226bNddcM926dcspp5yS8ePHJ5ne/fjwww9z2223ZdVVV80HH3wwx2PdcsstWXjhhbPRRhtl2223zS233JLJkyfPst9rr72WAw44IOutt1423HDDHHnkkRk2bNjMx0eNGpUTTjghG2+8cTp37py99torL7zwwszHV1111Vx88cX1XvPiiy/OqquuOvP74447Lj169MjJJ5+cLl265Kc//WmmTp2a0aNH59RTT80WW2yRNdZYI+uvv34OOeSQWX6uu+++O7vuumvWXnvtbL755jn77LMzefLk/Oc//8mqq66agQMH1tt/+PDhWW211b5xLA6AhkMAAfiKd999N3vssUdGjRqVP/7xjznjjDPy/vvvZ88998yoUaOSTP9D+09/+lP22muv9O3bN4ccckjuuOOO9OnTJ0lyySWXZIkllshmm22WgQMHZskll5ztsaZNm5Y77rgj22+/faqrq7Prrrtm1KhReeihh+rtN3jw4Oy5556ZOHFi/vjHP+a0007LoEGDst9++2XKlCmZMGFC9thjj/zzn//M0UcfnUsuuSQtWrTIAQcckHfeeedb/fzPP/98hgwZkosvvjiHHHJIGjdunIMOOihPPfVUjj766PTt2ze9evXKP//5z5x00kkznzdgwIAcddRRWW211XLJJZfkoIMOSv/+/XPKKadk5ZVXztprr5077rij3rHuuOOO1NTU5Mc//vG3qhGA+ZcRLICvuOSSS1JTU5Nrr702LVu2TJJstNFG2XrrrXP11Vfn2GOPzTPPPJO2bdtmr732SqNGjbL++uunefPm+fTTT5MknTp1SnV1dVq3bv21412PP/54RowYkd122y1Jss4662SllVbKjTfemO23337mfpdddlkWWWSR9OvXL02bNk2StGnTJkcccUTefPPNvPzyy3n//fdz++23p2PHjkmSLl26ZJdddslzzz2XDh06lPzzT506NaeeemqWW265JNO7FM2aNcuxxx6bLl26JEk22GCDfPDBBxkwYECSpLa2NhdffHG22WabnHHGGTNfa9KkSbntttsyefLk7LbbbjnppJPy/vvvz7zG5fbbb892222X5s2bl1wfAPM3HRCAr/jXv/6VDTbYIDU1NZk6dWqmTp2ali1bpkuXLvnnP/+ZJNlwww3z3nvvZdddd515IftPfvKT9OjR41sd65Zbbslyyy2XFVZYIWPHjs3YsWOz3Xbb5dlnn63XuXjhhRey6aabzgwfSbLWWmvlkUceyRprrJHnn38+7dq1mxk+kqRp06a59957s8cee3yrmmpqaupdJ7PUUkvluuuuS5cuXfLRRx/l6aefzvXXX58XX3wxU6ZMSTK9a/TJJ59k6623rvda++yzT+64445UV1dnhx12SLNmzWZ2QV555ZW888472XXXXb9VfQDM33RAAL7is88+yz333JN77rlnlsdat26dJNl+++1TW1ub/v3755JLLsmFF16Ytm3b5uijj84OO+xQ0nFGjx6dRx99NFOmTEnXrl1neXzgwIE54YQTZta0+OKLf23NX/f4t7H44ounqqqq3ra///3vOe+88zJs2LAsuuii6dixY2pqauodf8Zz56Rly5bZdttt8/e//z2HHnpobrvttiy33HIzuyoALBgEEICvWHjhhbPxxhtn3333neWxhRb639vmjjvumB133DHjxo3Lk08+mauuuiq9e/dOly5dstRSS33jce64445MmTIll1xySVq1alXvsUsvvTS33357jjrqqNTU1GThhReut3zwDI899lg6duyYhRdeeLYXur/00ktp2bJlVl555STTrzn5/yZMmPCNdT7//PM59thj07179+y///5p06ZNkuSss86aeZH7jPq/WuNnn32W119/Peuss05atGiR3XbbLbfddlteeeWV3H///dl7772/8fgANCxGsAC+Yv3118/bb7+d1VZbLWuuuWbWXHPNrLHGGrn22mvz4IMPJkmOOOKIHHrooUmmB5btttsuvXr1yrRp0zJixIgkSaNGX/8We+utt2adddbJNttskw022KDe15577pkxY8bk3nvvTTL9eo4nnnii3upYb775Zg488MC8+uqr6dKlS95///28+eabMx+fPHlyDjvssNx0001JpncgPv7443o1vPjii994Pl566aXU1tbm8MMPnxk+pk2bNnMcrba2NiuuuGIWW2yxPPzww/Wee+edd6Znz56ZNGlSkqRr165Zfvnlc/bZZ+fTTz/NLrvs8o3HB6BhEUAAvqJXr14ZOnRoDjrooDz00EN54okncthhh+Xuu++eeY3FhhtumAcffDB/+tOf8vTTT+f+++/PhRdemOWXX37mPq1atcqgQYPy7LPP5osvvqh3jFdeeSVvvfXWHMe1ttpqqyyyyCIzL/Lu1atXPv300/Ts2TOPPPJI7rvvvhxxxBFZffXVs+mmm2bXXXfNsssum1//+te544478sQTT+Twww/PF198MbPLsPnmm+fuu+9O//798/TTT+eYY47JkCFDvvF8rLXWWkmS0047Lf/617/ywAMPZN99983gwYOTTO+iNG7cOIcddljuv//+nHLKKXnqqadyww035IILLsiee+45c3QtSXbbbbc8++yz2WijjbL00kt/m/80ADQAAgjAV3Ts2DE33HBDqqqqcswxx+Twww/PyJEjc+mll+ZHP/pRkmSPPfbIiSeemMcffzwHH3xwTjrppHTo0CH9+vVLkyZNkiT77bdfPvnkk+y///557bXX6h3jlltuSePGjeutdPX/VVdXZ7vttsu///3vvPHGG+nUqVP++te/pra2NkceeWROO+20rLPOOrnqqqtSXV2dli1b5vrrr0/nzp1zxhln5De/+U0mTZqUv/71rzMvKD/++OOz5ZZb5uyzz87hhx+eZs2alXTDww022CAnnXRSXnrppfTs2TNnnnlmlllmmZl3j58xhrXXXnvlj3/8Y55//vkcdNBB6devX/bbb78cd9xx9V5v8803TxIXnwMsoKrq6urqKl0EAAuOq666KldffXWeeOKJVFdXV7ocAArmInQACnHbbbflrbfeSv/+/XPggQcKHwALKAEEgEIMHjw4AwYMyNZbb52ePXtWuhwAKsQIFgAAUBgXoQMAAIURQAAAgMIIIAAAQGEEEAAAoDAL1CpY2612fKVLaNiqqipdQcM3cnSlK2jQpqy2bKVLaPCqplr3ZG5r/NbQSpfQoFU1a1bpEhq8e9+/sNIlzFHtx6tUuoTZatTmrUqX8K3ogAAAAIURQAAAgMIsUCNYAADwXdWmttIlzNb81lGY3+oFAADmYwIIAABQGCNYAABQgml18+YI1vz2B70OCAAAUBgBBAAAKMz81rEBAICKqI2bqZaDDggAAFAYAQQAACiMESwAACjBvHojwvmNDggAAFAYAQQAACiMESwAACjBtDqrYJWDDggAAFAYAQQAACiMESwAACiBGxGWhw4IAABQGAEEAAAojBEsAAAowTQjWGWhAwIAABRGAAEAAApjBAsAAEpgFazy0AEBAAAKI4AAAACFMYIFAAAlmFZnBKscdEAAAIDCCCAAAEBhjGABAEAJaitdQAOhAwIAABRGAAEAAApjBAsAAEowzY0Iy0IHBAAAKIwAAgAAFMYIFgAAlGCaCayy0AEBAAAKI4AAAACFMYIFAAAlcCPC8tABAQAACiOAAAAAhTGCBQAAJZiWqkqX0CDogAAAAIURQAAAgMIYwQIAgBLUuhFhWeiAAAAAhRFAAACAwhjBKsh6m6ySX/1mm7TvsGTGfPp57hnwTG666rGSnrtSp2Vy/oBe2X/bczLio8/qPbbRVp2y56+3TLvlf5BPPxmfh//+Um666tFMnTJtLvwU8671Nlk5vzr8/53fgc9+u/N746+z/3bnznJ+t95l3ey2b7cs0751Ro8cl4f//lJuvOIfmTZ1wbsV0Xpbdsqvjt857VdZOmNGjcs9f3k8N114f0nPXWmt9jn/vmOz/wYnZcT7o+o9tm33TbLLQVumzXI/yMgPR+euax7PHVc+Mjd+hHla1/VXzD4HbJ7llv9Bxnw2IXfd8WJuvOGfc9x/oYUa5We/2DDbbLtmlliyVT4ZOS4PP/haBtzwz0ydw+/nrw/dOrv9fINsvekZc+vHmKd12WDF7Hvg5mm/whLTz/FtL2TAX7/+HO++54bZZvu1pp/jEePyyAOvZcBfn6p3jrfbqXN2/fn6Wbrtohnx8djcedvzue2m54r4keYp6225enqcsHPar7LM9PeIax/LwAvvK+m5K63dPhfcd3z2X//EDP/qe8Tem+SnB2098z3izn6PLpDvEUmy3uYd86veO6T9ym0yZtT43HP9U7np0odKeu5Ka7bL+Xcclf037ZMRH4yu99gNL5ye1ku2muU5v1z3xHw6clxZam8orIJVHgJIAVZbp31OvnTvPH7fq7nuwgez+nrLpccRP0qjRlUZ8OdHv/a5K6zaJqdesU8WatJ4lsc6b7xSTrxorzx+76u55rz7svzKbbLPkT/KIq1b5PI+f59LP828Z+b5vffVXHfRg1l93eXT4zfbfIvz22O253fn7hvn4BN2zBP3v5q+59ybVos2T/dDt84Kq7TJ6YffMJd+mnnTal1XzMl/7ZXHb38+1515R1bfYKX0OGHnNGrUKAPOv/drn7vC6m1z6o2HzPYc77DPpjn07F/mpovuy0uPvpFV11shPU/dLTXNqzPwgtL+cGkIOq3RNqed+fM8+sigXHv1o1ljrWWzb8/NU9WoKv3/+tRsn9PrsB9lm23XzA3XPZk33xiWlVdpk7337Zal2iySc/909yz7r7n2stllt65z+0eZZ3Vao11OO+sXeezhQbnmykezxtrts+9BW6RRo6r0/8vsz/Gvj/hRttlurdxw7ZN5a9BHWWnVNtl7/02zZJtFct6ZdyVJdvzpuvlN7+0z4K9P5cVn303H1dvmoEO3SU1NdW68bvav2xCt1nXFnHL9IXn89ufzlz/ckTU2WCk9frdLqho1yoDz7/na566weruc1v+w2b9H7LtZDjt7rwy88N6Z7xEHnvaz1DRvmoEXfP17T0Oz2nrL5+S+PfP4nS/lurPvzupdV0yPY3aY/m/dxQ9+7XNXWG2ZnHrtQbM9x4v+YOG0XrJV/nzqbRn8wrv1Hhv76edl/RlgBgGkAHsdslX+O3hYzjn2piTJC0++lYUWapyf9dw8t177ZCZPmjrLcxZq0jg77bVR9j58m9k+niTb/HS9jBw2JmcfMzC1tXV56Z9vZ9HFW+SnPTbJlX+8a4H5lH6vXlvmv28MyznH/S1J8sKT/5n+6fABm33z+T1s69k+3qhRVfY6ZMu8+NR/8ocjb5y5/e1BH+XPdx6RzhutlJeefnvu/VDzmL1675D/vvZBzjnk2iTJC48Mmv47fPiPc+vlD2XyF1Nmec5CTRpnpwO2yN7H7ZTJX0ye7ev+7PAf57Hbn881p9+eJPn3E2+mbYelstMBWyxQAeRX+3TLO28Pz5/OmP7BwXPP/jeNF2qUPfbaKDcPfCaTJ9f/HV144ZrsuPO6ufqKR3LTgH8lSV568b0kyYG9tsrVV/wjY8ZMmLl/TU2T9D7uJxk1anyWnM2nnAuCvffvlnf+83H+dNodSZLnn/lvFlqoUX7RfePcfOMczvEu6+Xqyx7O3/p/eY5feC9JcuChW6fv5Y9kzGcTssfeP8yjD72evpf/Y+Y+7ZZtnZ1377JABZDuvX+S/772fs7u1S9J8sIjr6dxk8b5+W+2za2XPzjn94ieW+ZXx+08x/eInx++bR67/blcc/ptSZJ/PzE47ToslZ17brnABZC9jtw2/x30Yc454vokyQuPDp7+Ptxr69x61aNzPsf7bpq9f7v9bB9Pkg5rtE2S/PPelzPiw0/n3g8A/49rQOayJk0aZ631V8xTD75eb/uT97+W5i2aZo31Vpjt87puumr2OmSrDPzzP9Lv3Nm/yTapXihfTJyc2v+3JMPYTyekSfVCadaiafl+iHnYzPP70FfO7wMzzu/ys31e101XyV69tszAKx9Nv3Nn/UN30cVbZuFFmueZRwfX2z70nREZM/rzrL95x7L9DPO6JtULZa2NV8lTd79Ub/uTd76Y5i1rssaGK832eV23XiN79d4hAy+4N/2+/OPhq37/i4vT79Rb622bOnlqmlQvOJ+NNGnSOGuts1yefLz+79oTjw5O8+ZNs+bay87ynBYtm+auO17MP596q972D74cq1h6mUXrbT+o11YZPXp87r/n5fIWP59o0qRx1uq8XJ587M162x//xxtp3qJp1lyn/SzPadGyJnfd/kKefvIr5/j9+uf4hCP756rL6o8DTZk6bcH6Ha5eKGv+cJU8dddX3yNe+PI9YuXZPq/r1mtmr947ZsD596TfabfOdp8Tf35h+p5yS71tU6YsWO8RSdKkunHW2nDlPHXvK/W2P3nPy9PP8forzvZ5XbfslL2O2DYDL34g/f4w+8mIFTu1y7jPJggfJZqWqnnya34jgMxlbZZtnSbVC+XDIZ/U2/7R0Onft13+B7N93luvfpAeW5+VAX9+dI6djDtveDrLtF88u+3XLS0WrknHtZfNLr/6YZ59bHDGj5lY3h9kHjXz/L5Xf2b4o6HTv5/z+f0wPbY5e/r5nTbr+f183BeZOmValmy7aL3tLVvVpGWrmrRpu1h5foD5QJvlfpAmTZvkw3eG19v+0bsjkyRtOyw12+e99dKQ9Fj3dxlw/r1z/B1+/z8fz5xFbrlo8/y4+w+z1c83zJ39Hi3fDzCPW3qZRVNdvdDMP2xn+PCD6X8MtFt28Vme8/GwMbno/Ptmec4mm66aKVOm1du+bpcVsvWP18zZZ96VuroFc/3IGef4w6FfeZ+YeY5bz/Kcj4d9lovPuS8fDP3KOd6sY71zPHTIqIz4eEyS6V2T7X6yTrbZdq38/Zbn58aPMk9qs9wPUj2794j/ftN7xHvp0fn4DDj/nkybOvvrFr/6HrFt902y9c83yp19/1HGn2De16b9D9Kk6UL58L8j6m3/6L0vz/EKS872eW+9PDQ9Nj41Ay5+cLb/1iVJh9XbZvyYCTnxyv1y8+t/zK2Dz8pxl/bIYgtot5RiVPwjhKlTp+aBBx7I888/n48++iiTJ09Os2bN0qZNm3Tp0iXbbLNNFlqo4mV+Zy1aNUuSTBj/Rb3tEz6f3m5u3nL2nYpRI8Z+42u/8ux/c3O/x3NA7+1zQO/tkyRvD/owf/rtgO9T8nylxcI1SZIJn5f3/E76Ykoev++V7PTLjTL07RH550OvZ5HWLXPw8Ttm6tTa1DSvLkP184cWi3z5OzzuK+f4y9/p5l/+N/iqUR9/VvIxOq3fIefe3TtJ8ta/h+TvVz/67QudT7VoOeN3eFK97RMmTv++eYm/a902WzVb/2jN3Hbzsxn/5X+bFi2a5rfH7pC/9Hs8H37lotMFyYz3ic8/rz/mM2HCl+e4xI7xJpt3zNbbrpnbbno247/y/4fV12yXC/68T5LkrTc+yh03LzgXobdcpHmSuf8ecd49xyZJ3vr3e7nj6gUrgMzxb4nxX/4Oz/Ecj/nG116xU9v8YOlFc9+NT+e2qx9N+5WWSvejt89Zfzssh257diZNnP14HHwfFe2ADB06NDvssENOOOGEDB48ODU1NVliiSXSpEmTvPHGGzn++OOz00475aOPPqpkmd9Lo6ov22Jz+OCx9nvc0eawU3bJ7vttmv6XP5JjfnVlzjvh5rRatEX6XLVfmtY0+c6vOz9p1Gjund+LT70jj9z57/zmtJ/mb/86KZfcfGjeeHlo3nrtg3wxYcF5Q27U6Mu3iTl8ev59zvEMHw/5JMfsfG7+dFDftGjVLBc9eHwWXWLh7/2684MZ7xFzak6U0rXotlnHHP/7XfLKv4fk6j//7w+zXx+2TUaOHJdbbnqmLLXOr/73PjyH3+FSzvEWHXP8KbvklZeGpO/ls67A9PGwz3J0r+vyh5NvS4uWNbm07/5ZdLEW36vu+UVVoxm/w7M/j3XleI8Y+kl673RO/njQ1WnRqlkufuh3C8x7RDJ3/6077+j+OXKn83LTpQ/l9Wf/m3v7P50zDuqXdisuma13X3AXrpiT2rqqefJrflPR1sKpp56adu3a5eabb87CC8/6RjJ27NgceeSROe2003LFFVdUoMLvb/y46aNQX/0kvnmL6Z9qfvXTjFItvmSrbPuzrhl45aP560XTV7949bl389ZrH+SKvx+RH+3aJXf2f/p7VD5/mPEp5Fc/wZx5fsdNmuU5pfpiwuRc8Ptbc8WZd2XJZRbN8A8/zaSJU/KjXdfLK+//97sXPZ8Z/+XFzM0XblZve/MZn9yP/f7jfqOHj8no4dM/qXvzxXdz9TOnZdvum3zjClsNwYxuxSy/w82mf//5+K//Hd7t5+vnwF9vlZf/PSQnnfC3TPlyCe4NNlopW2zVKb169ktVo6pUJan68g/xRo2rUldbN8fQ09DM8Rw3n3GOv/59eLc9NkjPQ7bKKy8NyUnH3jTzHP9/oz4Zn1GfjE+SDB70Ua4d2Cvb77TOHFfYakg+n/keUf9T+BnvEZ+PK8N7xMdjMvrLT/PffOHd9H329Gzbvds3rrDVUIz/8n121nM8/Xd4wvc4x4O/XMDi/xv0/LsZP2ZCVlit7Xd+Xfg6FQ0gL7zwQgYOHDjb8JEkrVq1Su/evbPXXnsVXFn5DBs6OtOmTsvS7evPcS/Tfvq1CUPfGTG7p32jJZZZNI0aNcqgF4fU2z7kP8Mz5tPPs9zKs58HbWhmnt/lvnp+p3//Xc9vkqy/2aoZP3ZiBr00NEPfnv46i7RukSXaLJK335h/u3Lf1rD3Rk4/xyssUW/7Ml9+P/StYd/pdZu1aJoNt107g198N8O+vJ5k+vE+yfjPJuQHyywY19l89NGnmTa1Nm2/cl1R23bTvx/y3ieze1qS5NDf/Ci77NY1jz4yKH864+/1/jDedPOOadq0Sfped9Asz3vgHyfk/ntfztlfLiXb0H304fRzvEy7+ud4xvdD353zOT7kyB9nl591zaMPvZ6zTq9/jps1r85Gm6ySwa9/mI/+3wW8wz78NOPHTcwSC8gM/Udfvkcss2L9f3eWWfHL94g3v+N7RMsv3yNe+Op7xMiM/2xClliArsUbNuSTL/+tq39d4zLLf3mO//Pxd3rdFq2a5YfbrZXBL76Xof+pfw3PQtULZeyn479bwfANKjqC1apVq4wY8fV/IH700UepqZn9bOP8YMrkqXn1+ffyw23WqLd9kx+vkXFjJubNV97/Tq87bMioTJs6Lat/ZZWntsv/IIss1iIff7BgrGYx8/xuvXq97Zv86Mvz++p3O79Jsv0vNph5bc0Mu/zqh6mdVpdnv7I6VkM2ZdLUvPr0f/LDHTrX277JT9bNuM8+z5uz+fSsFLW1tTnigr3zs0N/VG/7Kussl1atW+bd1z/4riXPV6ZMnpZXXhmaTTZdtd72bpt3zLhxEzN4DmF3/wM3zy67dc3NA59Jn1Num+VT+euueSK9evar93X336evUtSrZ79cd80Tc+cHmgdNmTwtr7w8NJt8ZfW6TbdYLePGTszgQbM/x/sdvEV2+VnX3DzgXznjpFnPce202hx9/I75efeN6m1fZbWl02qR5nnn7fp/0DVUc36PWO/L94h35/DMr1c7rS5HXtAjPzvsx/W2r9J5+nvEfxeQ94jky3P8zDv54XZr1du+yfZrZ9xnE/Lmv4d+p9edOnlqDunzs/ys19b1tm/04zVT06w6ryxAy82XqtKrXTWUVbAq2gHZfffdc/zxx+fwww/PBhtskKWXXjrV1dWZPHlyhg8fnmeffTbnnHNOdt9990qW+b0NuOKR/KHf/jnh/F/mgVufz2qdl8tu+3VLv3Pvy+RJU9O8RdO0X2nJDBs6OmNKvOnPmE8/z+3XPZXd99s0SfLSP9/Oksssmr0O2SrDP/o09/3t2bn5I81TBvz5H/lD3/1ywvl75oFbX8hq67Sffn7Pu/9/57fDkhn2funnN0n+fv0/c8bV++Wg43bIv/7xRtbeoEP2OHDzDLzq0QUm4M0w4Lx784dbfpMT+vbMA/3/mdW6rpjdDt0m/U67LZO/mJLmLWvSftWlM+y9kRkzqrRPzCZNnJK/XXx/9jx6+4z99PP8+7E30rbDUtmr945557X388CNc75DdUNzw3VP5qzz9srvT901993zclZfo11+vsdGufqKRzJ58tQ0b16d5Zb/QT768LOMGTMhHVZaKr/45cZ5842P8tg/BmW1TsvUe70h732S4R+PyfCvXIC64cbTl0x+6zt+Ij0/63/tk/nThXvl9312zX13vZxOa7bLz365Ua6+7OH/neMVlshHH36aMZ9NSIeVl8ovuk8/x48//EZWW73+KMqQd0dmwoTJGXj9P7PXvt0ydszEvPTcu2nbvnV+tf+meeetj3P/3QvOssc3nnd3zrzlyPyu70G5v/9T6dR1xex+6I/S77Rbv8d7xOTcdPF9+eXRO2Tc6M/z0mNvpO1KS6V775/knVffzwP9G/542/834KIH8ocbe+WEy/fJAwOfyWpdVshuB2+Zfn+488tz3DTtV26TYUM+yZjRpf1bN+mLKbn5iofzyyO2zWefjMsLjw7O8qstk+5HbptnHno9//7KMtRQLlV1FVyXsa6uLpdeemmuueaaTJgwYZbHW7Rokb322iu/+c1v/nch7Pew3WrHf+/X+K423rpTuh+6ddqtsEQ+GT42d/V/Orde+2SSZM2uK+Ss6w7Mucf/LQ/d/uIsz916l3Vz9Jk/S4+t/pQRH31W77FdfvXDbP+L9dOmXeuMHjkuLz71n/zlgge+1R/aZVNVuQS+8Vad0v3Qrf53fm/8V/3z+5eeOfeEm+d8fv+we3psfdYs53ez7dfKngdvkaXaLpYRH32Wuwc8k7/fUMFra0ZWbiWjjbdfJ92P2THtVloqnwz7LHf1eyy3Xv5QkmTNjVfJWXcclXMP+0seGjDr+dl6j41y9MU90mPd32XE+/9bCrWqqirb9+iWHfbdLMussETGfTYh/7z7pfzlD3fMsqJOEaasNus9N4ryw26rpsd+3dJu2cUz6pNxueO2F3LzwOkXj6+9Tvuce9HeOesPd+aB+15Jj/02zd77dJvjax19+F/z8mw+Ef3Vvt3yq303zdabnjHXfo5vUjW1chee/HDTVfOrAzZNu/aLZ9TIcfn7rc/n5hunn+O1Oi+Xcy/dO2f3+XseuOeV9Dhgs3Tf72vO8SF/zSsvDUlVVbLDLutmp127ZJl2i2Xc2Il58tE3c82Vj86ysllRGr/13T4N/7423n6d7H3sTmm70lIZNeyz3Nnv0dx62fRrFNf64So5647f5txDr8mDs3mP2GaPjXL0JfumR+fjM/yr7xH7bJod9938y/eIz/PUXTPeIyqz3HxVs2bfvNNcsvG2a6X7Udul3YpL5pOPP8td1z2ZW6+cvvDEmhuulLP+dljOPeqGPDSbDyG3/tn6Ofq8vdJjo1NnLm2cTD/HO/zqh9lh702y9HKLZ9ynE/KP21/I9efdO8ebF85t975/YUWOW4oXh85636B5wbrtK/P/+++qogFkhilTpuSNN97I8OHDM3HixNTU1KRNmzbp2LFjqqvLt9xpJQPIAqGCAWSBUcEAsiCoZABZUFQygCwoKhVAFhSVDCALink5gDw3dPlKlzBbXdu/V+kSvpV54gYbTZo0yVprrfXNOwIAAPM1d0IHAAAKM090QAAAYF43P970b16kAwIAABRGAAEAAApjBAsAAEowP970b16kAwIAABRGAAEAAApjBAsAAEowrc5n9+XgLAIAAIURQAAAgMIYwQIAgBLU+uy+LJxFAACgMAIIAABQGCNYAABQAjciLA8dEAAAoDACCAAAUBgjWAAAUAI3IiwPZxEAACiMAAIAABTGCBYAAJSg1ipYZaEDAgAAFEYAAQAACmMECwAASjDNZ/dl4SwCAACFEUAAAIDCGMECAIASuBFheTiLAABAYQQQAACgMEawAACgBLU+uy8LZxEAACiMAAIAABTGCBYAAJRgWl1VpUtoEHRAAACAwgggAABAYYxgAQBACab57L4snEUAAKAwAggAAFAYAQQAACiMa0AAAKAEtXU+uy8HZxEAACiMAAIAABTGCBYAAJTAMrzl4SwCAACFEUAAAIDCGMECAIASTKurqnQJDYIOCAAAUBgBBAAAKIwRLAAAKEGtz+7LYoEKIKfcN6DSJTRoo2pbVLqEBu/z2qaVLqFBGzL53UqX0OB1qvmw0iU0eFPqFqh/2gvXvGpSpUuA+Z4YBwAAFMbHJAAAUIJpdT67LwdnEQAAKIwAAgAAFMYIFgAAlKA2bkRYDjogAABAYQQQAACgMEawAACgBFbBKg9nEQAAKIwAAgAAFMYIFgAAlGCaz+7LwlkEAAAKI4AAAACFMYIFAAAlqK1zI8Jy0AEBAAAKI4AAAACFMYIFAAAlsApWeTiLAABAYQQQAACgMEawAACgBLV1PrsvB2cRAAAojAACAAAUxggWAACUYFrciLAcdEAAAIDCCCAAAEBhjGABAEAJrIJVHs4iAABQGAEEAAAojBEsAAAogVWwykMHBAAAKIwAAgAAFMYIFgAAlMAqWOXhLAIAAIURQAAAgMIYwQIAgBJMM4JVFs4iAABQGAEEAAAojBEsAAAoQa0bEZaFDggAAFAYAQQAACiMESwAACiBVbDKw1kEAAAKI4AAAACFMYIFAAAlqK2zClY56IAAAACFEUAAAGABVVtbm4suuijdunXL2muvnf322y9DhgyZ4/4jR47MUUcdlQ022CAbbLBBfvOb3+Tjjz/+VscUQAAAoATT0mie/Po+LrvssgwYMCB9+vTJwIEDU1VVlZ49e2by5Mmz3f/II4/MsGHDcs011+Saa67Jxx9/nF69en2rYwogAACwAJo8eXL69euXww47LJtttlk6duyY888/P8OHD8+DDz44y/5jx47Nc889l549e6ZTp07p1KlTDjzwwLz++uv59NNPSz6uAAIAAAugwYMH5/PPP8+GG244c1urVq3SqVOnPPfcc7Ps37Rp0zRv3jy33357xo8fn/Hjx+eOO+7I8ssvn0UWWaTk41oFCwAASjCvroK11VZbfe3jDz/88Gy3z7h2Y+mll663fckll8ywYcNm2b9p06Y544wzctppp6VLly6pqqrKEksskeuvvz6NGpXe19ABAQCABdDEiROTJNXV1fW2N23aNJMmTZpl/7q6urz55pvp3LlzbrjhhvzlL39J27Ztc8ghh2T8+PElH1cHBAAA5mNz6nB8k5qamiTTrwWZ8b+TZNKkSWnWrNks+999993p379//vGPf6Rly5ZJkiuuuCJbbLFFbrnllvTo0aOk4wogAABQgtoGNjw0Y/RqxIgRad++/cztI0aMSMeOHWfZ/4UXXsgKK6wwM3wkySKLLJIVVlgh7733XsnHbVhnEQAAKEnHjh3TsmXLPPPMMzO3jR07NoMGDUqXLl1m2X/ppZfOkCFD6o1nTZw4MR988EGWW265ko8rgAAAwAKouro63bt3zznnnJOHH344gwcPzpFHHpk2bdpkm222ybRp0zJy5Mh88cUXSZJddtklSXLEEUdk8ODBM/evrq7OrrvuWvJxKz6Ctffee6eqqrQVBa677rq5XA0AAMzetHl0Fazv4/DDD8/UqVNz4okn5osvvkjXrl3Tt2/fVFdX54MPPshWW22VM888M7vuumuWXHLJ9O/fP2effXZ69OiRRo0apUuXLrnxxhvTqlWrko9Z8QCy0UYb5eKLL86KK66YtdZaq9LlAADAAqNx48bp3bt3evfuPctj7dq1y5tvvllvW4cOHXLFFVd8r2NWPID06tUrzZs3z0UXXZQ///nPadeuXaVLAgAA5pJ54hqQffbZJ+uuu24uuOCCSpcCAACzVVtXNU9+zW8q3gGZ4YwzzsigQYMqXQYAADAXzTMBZKmllspSSy1V6TIAAIC5aJ4JIAAAMC+rrZsnrl6Y7zmLAABAYQQQAACgMEawAACgBNMy/604NS/SAQEAAAojgAAAAIUxggUAACWYH2/6Ny/SAQEAAAojgAAAAIUxggUAACVwI8LycBYBAIDCCCAAAEBhjGABAEAJat2IsCx0QAAAgMIIIAAAQGGMYAEAQAmmuRFhWeiAAAAAhRFAAACAwhjBAgCAErgRYXk4iwAAQGEEEAAAoDBGsAAAoAS1VsEqCx0QAACgMAIIAABQGCNYAABQgtoYwSoHHRAAAKAwAggAAFAYI1gAAFACq2CVhw4IAABQGAEEAAAojBEsAAAoQW2dz+7LwVkEAAAKI4AAAACFMYIFAAAlsApWeeiAAAAAhRFAAACAwhjBAgCAEtTGCFY56IAAAACFEUAAAIDCGMECAIASWAWrPHRAAACAwgggAABAYYxgAQBACYxglYcOCAAAUBgBBAAAKIwRLAAAKIERrPLQAQEAAAojgAAAAIVZoEaw1qqudAUN3eeVLqDBa5QJlS6hQWvSYlylS2jwptRNrXQJDV5t6ipdQoM2fNoXlS6BCjKCVR46IAAAQGEEEAAAoDAL1AgWAAB8V7UxglUOOiAAAEBhBBAAAKAwRrAAAKAEVsEqDx0QAACgMAIIAABQGCNYAABQAiNY5aEDAgAAFEYAAQAACmMECwAASmAEqzx0QAAAgMIIIAAAQGGMYAEAQAmMYJWHDggAAFAYAQQAACiMESwAAChBnRGsstABAQAACiOAAAAAhTGCBQAAJaiNEaxy0AEBAAAKI4AAAACFMYIFAAAlcCPC8tABAQAACiOAAAAAhTGCBQAAJXAjwvLQAQEAAAojgAAAAIUxggUAACWwClZ56IAAAACFEUAAAIDCGMECAIASWAWrPHRAAACAwgggAABAYYxgAQBACayCVR46IAAAQGEEEAAAoDBGsAAAoAR1dZWuoGHQAQEAAAojgAAAAIUxggUAACWojVWwykEHBAAAKIwAAgAAFMYIFgAAlKDOjQjLQgcEAAAojAACAAAUxggWAACUoNYIVlnogAAAAIURQAAAgMIYwQIAgBLU1VW6goZBBwQAAChMxQPIu+++m4svvjh9+vTJY489Nsvj48ePz/HHH1+BygAAgHKraAB54YUX8tOf/jR33XVXHn/88Rx88ME57LDDMnny5Jn7fPHFF7n99tsrVyQAAGT6jQjnxa/5TUUDyLnnnpvdd989999/fx544IGcd955eeqpp3LwwQdnypQplSwNAACYCyoaQN58881079595vfbbbddrrrqqrz00ks55phjKlgZAAAwN1Q0gLRs2TKffvppvW3rrbdezj777Nx///0588wzK1QZAADUV+lRKyNYZbDZZpvltNNOy8svv1xv5GrrrbfOCSeckL/85S857bTTKlghAABQThUNIEcffXQWW2yx7LHHHnn66afrPda9e/ecdNJJeeSRRypUHQAAUG4VvRHhIosskn79+mXo0KFZbLHFZnn8l7/8ZTbaaKM88MADFagOAAD+p3Y+HHeaF80Td0Jv3779HB9bYYUVctBBBxVYDQAAMLdU/EaEAADAgmOe6IAAAMC8rq6u0hU0DDogAABAYQQQAACgMEawAACgBPPjTf/mRTogAABAYQQQAACgMEawAACgBEawykMHBAAAKIwAAgAAFMYIFgAAlMB9CMtDBwQAACiMAAIAABTGCBYAAJTAKljloQMCAAAURgABAAAKYwQLAABKYRmsstABAQAACiOAAAAAhTGCBQAAJbAKVnnogAAAAIURQAAAgMIYwQIAgBLUWQWrLHRAAACAwgggAABAYYxgAQBACayCVR46IAAAQGEEEAAAoDBGsAAAoBRGsMpCBwQAACiMAAIAABTGCBYAAJTAjQjLQwcEAAAojAACAAAUxggWAACUwghWWeiAAAAAhRFAAACAwhjBAgCAEtS5EWFZ6IAAAACFEUAAAIDCGMECAIBSWAWrLHRAAACAwgggAABAYYxgAQBACayCVR46IAAAsICqra3NRRddlG7dumXttdfOfvvtlyFDhsxx/ylTpuTcc89Nt27dss4666R79+554403vtUxBRAAAFhAXXbZZRkwYED69OmTgQMHpqqqKj179szkyZNnu/8pp5ySm2++OaeffnpuueWWLLroounZs2fGjRtX8jEFEAAAKEXdPPr1HU2ePDn9+vXLYYcdls022ywdO3bM+eefn+HDh+fBBx+cZf/3338/N998c84888xsvvnm6dChQ/7whz+kuro6r732WsnHFUAAAGABNHjw4Hz++efZcMMNZ25r1apVOnXqlOeee26W/Z988sm0atUqm266ab39H3nkkWy00UYlH3eBugi9aVWTSpcAsEBrUrVA/bNTERPrJlW6hAatuspFyMx7ttpqq699/OGHH57t9o8//jhJsvTSS9fbvuSSS2bYsGGz7P/ee+9l2WWXzQMPPJArr7wyw4cPT6dOnXLcccelQ4cOJderAwIAACWpmke/vpuJEycmSaqrq+ttb9q0aSZNmvXDjPHjx2fo0KG57LLLctRRR+Xyyy/PQgstlF/+8pcZNWpUycf1URQAAMzH5tTh+CY1NTVJpl8LMuN/J8mkSZPSrFmzWfZv0qRJxo0bl/PPP39mx+P888/PZpttlttuuy0HHHBAScfVAQEAgAXQjNGrESNG1Ns+YsSItGnTZpb927Rpk4UWWqjeuFVNTU2WXXbZfPDBByUfVwABAIBSVHq1qzKvgtWxY8e0bNkyzzzzzMxtY8eOzaBBg9KlS5dZ9u/SpUumTp2aV199dea2L774Iu+//36WW265ko9rBAsAABZA1dXV6d69e84555y0bt06bdu2zdlnn502bdpkm222ybRp0zJ69OgsvPDCqampSZcuXbLxxhvn2GOPzWmnnZZFF100F110URo3bpydd9655OPqgAAAwALq8MMPz+67754TTzwxe+65Zxo3bpy+ffumuro6w4YNyyabbJJ77rln5v4XX3xx1l9//Rx66KHZfffdM378+Fx33XVp3bp1ycesqqur+x6Nm/lL7cerVLoEAJirLMM7d42pnVLpEhq8Zdp+VOkS5mj5a/9U6RJm6719jq10Cd+KDggAAFAYAQQAACiMi9ABAKAUdd/9pn/8jw4IAABQGAEEAAAojBEsAAAowYKzduzcpQMCAAAURgABAAAKYwQLAABKYQSrLHRAAACAwgggAABAYYxgAQBAKdyIsCx0QAAAgMIIIAAAQGGMYAEAQAmqrIJVFjogAABAYQQQAACgMEawAACgFEawykIHBAAAKIwAAgAAFMYIFgAAlMKNCMtCBwQAACiMAAIAABTGCBYAAJTCKlhloQMCAAAURgABAAAKYwQLAABKYQSrLHRAAACAwgggAABAYYxgAQBAKYxglYUOCAAAUBgBBAAAKIwRLAAAKEVdVaUraBB0QAAAgMIIIAAAQGGMYAEAQAmqrIJVFjogAABAYQQQAACgMEawAACgFEawykIHBAAAKIwAAgAAFEYAAQAACiOAAAAAhflWF6GPHDkyl156ad5///384Ac/yGqrrZY11lgjq6++epo1aza3agQAABqIbxVATjjhhDz55JNZeeWV88EHH+TOO+9MXV1dGjVqlBVXXDFrrLFG1lxzzay55prp2LFjmjRpMrfqBgCAQrkRYXl8qwDy0ksvpXfv3tlvv/2SJBMmTMjrr7+eV199Na+++mqee+653HbbbUmS6urqvPLKK9/4mpMmTcp//vOfrLTSSqmpqckbb7yR66+/PsOHD8/KK6+cHj16pE2bNt/hRwMAAOY13yqANG3aNJ06dZr5ffPmzdO1a9d07dp15rbPPvssr7zySl577bVvfL133nkn++yzT0aOHJllllkmffr0Sa9evdKuXbt06NAhDz30UG699db0798/HTp0+DalAgAA86BvdRH61ltvnUGDBn3tPosuumg23XTT9OrV6xtf76yzzkrnzp1z++23Z7311suvf/3r/OQnP8mdd96ZCy+8MPfee29++MMf5swzz/w2ZQIAQPnVVc2bX/OZbxVAdtttt9x77715++23y3LwZ599NkcccUQ6duyYY489NpMmTcqee+6ZqqrpJ3KhhRbKwQcfnBdeeKEsxwMAACrrW41g/fznP09VVVV+9rOfZdttt023bt2y+uqrZ7nllvtOB6+pqckXX3yRJPnBD36Qn//852natGm9fcaOHZuFF174O70+AAAwb/lWAaRPnz5544038vrrr+fee+/NbbfdlqqqqrRo0SKdOnXKGmuskWOOOabk19tkk01y+umnp0+fPunQoUNOO+20mY/V1dXl2Wefzamnnpqtt97625QJAADlZxWssqiqq6v7TqeytrY277zzTl5//fW89tprGTRoUAYPHpwXX3yx5NcYPXp0Dj744Cy77LI599xz6z1299135+ijj063bt1y/vnnp2XLlt+lzPo1f7zK934NAJiXTaybVOkSGrQxtVMqXUKDt0zbjypdwhyteMF5lS5htv57xFGVLuFb+c4BZHbq6upmXr/xbXz22WdZdNFF620bPXp0RowYkY4dO5apOgEEgIZPAJm7BJC5TwD59ua3APKtRrC+yXcJH0lmCR9J0rp167Ru3fp7VgQAAGViBKssvtUqWAAAAN+HAAIAABSmrCNYAADQUFUZwSoLHRAAAKAwAggAAFAYI1gAAFAKI1hloQMCAAAURgABAAAKYwQLAABKYQSrLHRAAACAwgggAABAYYxgAQBACdyIsDx0QAAAgMIIIAAAQGGMYAEAQCnqqipdQYOgAwIAABRGAAEAAApjBAsAAEphFayy0AEBAAAKI4AAAACFMYIFAAAlcCPC8tABAQAACiOAAAAAhTGCBQAApTCCVRY6IAAAQGEEEAAAoDBGsAAAoARWwSoPHRAAAKAwAggAAFAYI1gAAFAKI1hloQMCAAAURgABAAAKYwQLAABKYQSrLHRAAACAwgggAABAYYxgAQBACdyIsDx0QAAAgMIIIAAAQGEEEAAAoDACCAAAUBgBBAAAKIxVsAAAoBRWwSoLHRAAAKAwAggAAFAYI1gAAFACNyIsDx0QAACgMAIIAABQGCNYAABQCiNYZbFABZAnvqh0BQ3be1N+UOkSGrzmjSZVuoQGrU3jMZUuocEbXduy0iU0eJPrFq50CQ3alLoF6k+nivhlpQtgrjOCBQAAFEaMBwCAUhjBKgsdEAAAoDACCAAAUBgjWAAAUAI3IiwPHRAAAKAwAggAAFAYI1gAAFAKI1hloQMCAAAURgABAAAKYwQLAABKYBWs8tABAQAACiOAAAAAhTGCBQAApTCCVRY6IAAAQGEEEAAAoDBGsAAAoBRGsMpCBwQAACiMAAIAABTGCBYAAJTAjQjLQwcEAAAojAACAAAUxggWAACUwghWWeiAAAAAhRFAAACAwhjBAgCAUhjBKgsdEAAAoDACCAAAUBgjWAAAUAI3IiwPHRAAAKAwAggAAFAYI1gAAFAKI1hloQMCAAAURgABAAAKYwQLAABKYBWs8tABAQAACiOAAAAAhTGCBQAApTCCVRY6IAAAQGEEEAAAoDACCAAAUBjXgAAAQClcA1IWOiAAAEBhBBAAAKAwRrAAAKAEVZUuoIHQAQEAAAojgAAAAIUxggUAAKWwClZZ6IAAAACFEUAAAIDCGMECAIASVBnBKgsdEAAAoDDzbAD5yU9+kmHDhlW6DAAAoIwqOoJ1++23z/GxIUOG5N57703r1q2TJLvssksxRQEAwOwYwSqLigaQU089NV988UWSpK5u1v+iZ511VpKkqqpKAAEAgAagoiNYt956azp16pQNNtggjz32WAYPHjzzq1mzZnnwwQczePDgvPHGG5UsEwAAGqTa2tpcdNFF6datW9Zee+3st99+GTJkSEnPvfPOO7Pqqqvmgw8++FbHrGgAWWGFFTJw4MCstdZa2XnnnXPPPfdUshwAAJizunn063u47LLLMmDAgPTp0ycDBw5MVVVVevbsmcmTJ3/t8z788MOceuqp3+mYFb8IfaGFFspRRx2Viy++OOecc06OPvrojBs3rtJlAQBAgzZ58uT069cvhx12WDbbbLN07Ngx559/foYPH54HH3xwjs+rra1N7969s/rqq3+n41Y8gMzQtWvXmRel77jjjpkyZUplCwIAgAZs8ODB+fzzz7PhhhvO3NaqVat06tQpzz333Byfd8UVV2TKlCk56KCDvtNx56kbEbZq1Srnnntubr/99tx6661p2rRppUsCAIAk8+6NCLfaaquvffzhhx+e7faPP/44SbL00kvX277kkkvO8XYYr7zySvr165ebb745w4cP/w7VzmMBZIZddtnFqlcAADAXTZw4MUlSXV1db3vTpk0zZsyYWfafMGFCfvvb3+a3v/1tll9++YYVQAAAgNLMqcPxTWpqapJMvxZkxv9OkkmTJqVZs2az7N+nT58sv/zy2WOPPb5boV8SQAAAoBTz6AjWdzVj9GrEiBFp3779zO0jRoxIx44dZ9n/lltuSXV1dTp37pwkmTZtWpLp12/vtNNOOe2000o6rgACAAALoI4dO6Zly5Z55plnZgaQsWPHZtCgQenevfss+z/wwAP1vn/55ZfTu3fvXHnllenQoUPJxxVAAABgAVRdXZ3u3bvnnHPOSevWrdO2bducffbZadOmTbbZZptMmzYto0ePzsILL5yampost9xy9Z4/4yL2ZZZZJosvvnjJx51nluEFAIB5WVXdvPn1fRx++OHZfffdc+KJJ2bPPfdM48aN07dv31RXV2fYsGHZZJNNyn6z8Kq6uroGNs02Z4+9t0qlS2jQ3pvyg0qX0OA1bzSp0iU0aG0az7riB+U1urZlpUto8CbXNa50CQ3alDrDI3PbL1d6ptIlzFHnQ86vdAmz9dKlR1a6hG9FBwQAACiMGA8AAKVYYOaG5i4dEAAAoDACCAAAUBgjWAAAUILvu+IU0+mAAAAAhRFAAACAwhjBAgCAUhjBKgsdEAAAoDACCAAAUBgjWAAAUAojWGWhAwIAABRGAAEAAApjBAsAAErgRoTloQMCAAAURgABAAAKYwQLAABKYQSrLHRAAACAwgggAABAYYxgAQBACarqzGCVgw4IAABQGAEEAAAojBEsAAAohQmsstABAQAACiOAAAAAhTGCBQAAJagyglUWOiAAAEBhBBAAAKAwRrAAAKAURrDKQgcEAAAojAACAAAUxggWAACUwCpY5aEDAgAAFEYAAQAACmMECwAASmEEqyx0QAAAgMIIIAAAQGGMYAEAQAmsglUeOiAAAEBhBBAAAKAwRrAAAKAURrDKQgcEAAAozALVAemz4tqVLqFBa9SsWaVLgO+lUdulK11Cgzf17f9WuoSGr8pni3NTVePGlS6hwfvl5EpXwNy2QAUQAAD4rqyCVR4+JgEAAAojgAAAAIUxggUAAKWoM4NVDjogAABAYQQQAACgMEawAACgBFbBKg8dEAAAoDACCAAAUBgjWAAAUAojWGWhAwIAABRGAAEAAApjBAsAAEpQVVvpChoGHRAAAKAwAggAAFAYI1gAAFAKq2CVhQ4IAABQGAEEAAAojBEsAAAoQZURrLLQAQEAAAojgAAAAIUxggUAAKWoM4NVDjogAABAYQQQAACgMEawAACgBFbBKg8dEAAAoDACCAAAUBgjWAAAUAojWGWhAwIAABRGAAEAAApjBAsAAEpgFazy0AEBAAAKI4AAAACFMYIFAAClqDODVQ46IAAAQGEEEAAAoDBGsAAAoARWwSoPHRAAAKAwAggAAFAYI1gAAFAKI1hloQMCAAAURgABAAAKYwQLAABKYBWs8tABAQAACiOAAAAAhTGCBQAApag1g1UOOiAAAEBhBBAAAKAwRrAAAKAUJrDKQgcEAAAojAACAAAUxggWAACUwI0Iy6OiHZCbb745kydPrrftX//6Vw488MDstNNOOfroo/P2229XqDoAAKDcKhpAfv/732fcuHEzv3/yySez7777pra2NptssklGjhyZ3XbbLS+++GIFqwQAAMqloiNYdXX1+1iXXXZZfvWrX+X444+fue3MM8/MOeeck/79+xddHgAA/E+dGaxymKcuQh8yZEh23nnnett+8YtfZNCgQRWqCAAAKKeKBpCqqqp63y+//PKZMGFCvW2ffvppFl544SLLAgAA5pKKj2BttdVWWWGFFdKhQ4dUV1fn7LPPzvXXX58mTZrkxRdfzKmnnprNNtuskmUCAIBVsMqkogHkkUceyZtvvpm33norb775ZkaOHJn33nsv06ZNS5MmTbL//vtn1VVXzdFHH13JMgEAgDKpaABZZpllsswyy2SLLbaYuW3KlClp0qRJkmTAgAFZZZVVZhnVAgAA5k/z3I0IZ4SPJFl11VUrWAkAAPw/RrDKYp5aBQsAAGjYBBAAAKAw89wIFgAAzIuq3IiwLHRAAACAwgggAABAYYxgAQBAKWorXUDDoAMCAAAURgABAAAKYwQLAABKYBWs8tABAQAACiOAAAAAhTGCBQAApTCBVRY6IAAAQGEEEAAAoDBGsAAAoBRWwSoLHRAAAKAwAggAAFAYI1gAAFCCKhNYZaEDAgAAFEYAAQAACmMECwAASmEVrLLQAQEAAAojgAAAAIUxggUAACWoqq10BQ2DDggAAFAYAQQAACiMESwAACiFVbDKQgcEAAAojAACAAAUxggWAACUwgRWWeiAAAAAhRFAAACAwhjBAgCAElRZBassdEAAAIDCCCAAAEBhjGABAEApjGCVhQ4IAABQGAEEAAAojBEsAAAoRW2lC2gYdEAAAIDCCCAAAEBhjGABAEAJ3IiwPHRAAACAwgggAABAYYxgAQBAKYxglYUOCAAAUBgBBAAAKIwRLAAAKIURrLJYoALIQh1WqHQJDVrdiE8qXUKDVzd1aqVLaNjGjqt0BQ1e45YtK11Cg1c3bVqlS2jQ6qZ4H4bvywgWAABQmAWqAwIAAN9ZbaULaBh0QAAAgMIIIAAAQGGMYAEAQAmqrIJVFjogAACwgKqtrc1FF12Ubt26Ze21185+++2XIUOGzHH///znPznwwAOzwQYbZKONNsrhhx+ejz766FsdUwABAIAF1GWXXZYBAwakT58+GThwYKqqqtKzZ89Mnjx5ln0//fTT7LvvvmnRokWuv/76XHXVVfn0009zwAEHZNKkSSUfUwABAIBS1NXNm1/f0eTJk9OvX78cdthh2WyzzdKxY8ecf/75GT58eB588MFZ9n/ooYcyceLE/PGPf8zKK6+cNdZYI2effXbeeeedvPjiiyUfVwABAIAF0ODBg/P5559nww03nLmtVatW6dSpU5577rlZ9t9oo41y6aWXpmnTprM8NmbMmJKP6yJ0AACYj2211VZf+/jDDz882+0ff/xxkmTppZeut33JJZfMsGHDZtm/Xbt2adeuXb1tf/7zn9O0adN07dq15HoFEAAAKEUDWwVr4sSJSZLq6up625s2bVpSR+O6665L//79c/zxx2fxxRcv+bgCCAAAzMfm1OH4JjU1NUmmXwsy438nyaRJk9KsWbM5Pq+uri4XXnhhLr/88hx00EHZZ599vtVxXQMCAAALoBmjVyNGjKi3fcSIEWnTps1snzNlypT07t07V1xxRY455pgcddRR3/q4AggAAJSi0qtdlXkVrI4dO6Zly5Z55plnZm4bO3ZsBg0alC5dusz2Occcc0zuu+++nHvuudl///2/03GNYAEAwAKouro63bt3zznnnJPWrVunbdu2Ofvss9OmTZtss802mTZtWkaPHp2FF144NTU1ufXWW3PPPffkmGOOyfrrr5+RI0fOfK0Z+5RCBwQAABZQhx9+eHbfffeceOKJ2XPPPdO4ceP07ds31dXVGTZsWDbZZJPcc889SZK77rorSXLWWWdlk002qfc1Y59SVNXVNbDL+b/GdisfU+kSGrS6EZ9UuoQGr27q1EqX0KA1WrhlpUto8OomTKx0CQ1e3bRplS6hQaub4n14bntgcv9KlzBH267+u0qXMFv3vX5GpUv4VnRAAACAwgggAABAYVyEDgAAJahacK5cmKt0QAAAgMIIIAAAQGGMYAEAQCmMYJWFDggAAFAYAQQAACiMESwAAChFrRGsctABAQAACiOAAAAAhTGCBQAApbAKVlnogAAAAIURQAAAgMIYwQIAgFIYwSoLHRAAAKAwAggAAFAYI1gAAFAKI1hloQMCAAAURgABAAAKYwQLAABKUWsEqxx0QAAAgMIIIAAAQGGMYAEAQCnqaitdQYOgAwIAABRGAAEAAApjBAsAAErhRoRloQMCAAAURgABAAAKYwQLAABK4UaEZaEDAgAAFEYAAQAACmMECwAASmEVrLLQAQEAAAojgAAAAIWp+AjWyy+/nGeeeSYHHnhgkuRf//pXrr322nzwwQdp37599ttvv3Tp0qXCVQIAsMAzglUWFe2A3Hfffdlzzz3z7LPPJkn+8Y9/ZN99901dXV0222yzTJkyJT169Mg//vGPSpYJAACUSUU7IJdcckkOPfTQ9OrVK0ly+eWX5+CDD85vfvObmftcfvnlueiii7LFFltUqkwAAKBMKtoBGTp0aH7yk5/M/P6DDz7Ij3/843r77LjjjnnnnXeKLg0AAOqrq5s3v+YzFQ0gyy67bB577LGZ36+22moZPHhwvX1eeeWVLLXUUkWXBgAAzAUVHcHq2bNnfve73+Xjjz/OjjvumF69euW4447LpEmTsvLKK+fll1/OpZdemkMPPbSSZQIAAGVS0QCyyy67pKqqKhdddFGuvvrqVFVVpa6uLieffHKSpEWLFjnggAOyzz77VLJMAABIamsrXUGDUPFleHfeeefsvPPO+e9//5v33nsv48ePT5MmTdKmTZt06tQpTZs2rXSJAABAmVQ8gMyw4oorZsUVV6x0GQAAwFw0zwQQAACYp82HK07Niyq6ChYAALBgEUAAAIDCGMECAIBSGMEqCx0QAACgMAIIAABQGCNYAABQilojWOWgAwIAABRGAAEAAApjBAsAAEpQV1db6RIaBB0QAACgMAIIAABQGCNYAABQCqtglYUOCAAAUBgBBAAAKIwRLAAAKEWdEaxy0AEBAAAKI4AAAACFMYIFAAClqHUjwnLQAQEAAAojgAAAAIUxggUAAKWwClZZ6IAAAACFEUAAAIDCGMECAIAS1FkFqyx0QAAAgMIIIAAAQGGMYAEAQCmsglUWOiAAAEBhBBAAAKAwRrAAAKAUtUawykEHBAAAKIwAAgAAFMYIFgAAlKLOjQjLQQcEAAAojAACAAAUxggWAACUoM4qWGWhAwIAABRGAAEAAApjBAsAAEphFayy0AEBAAAKI4AAAACFMYIFAAAlsApWeeiAAAAAhRFAAACAwhjBAgCAUlgFqyx0QAAAgMIIIAAAQGGq6urqXM4PAAAUQgcEAAAojAACAAAURgABAAAKI4AAAACFEUAAAIDCCCAAAEBhBBAAAKAwAggAAFAYAQQAACiMAAIAABRGAAEAAAojgAAAAIURQAAAgMIIIPOY2traXHTRRenWrVvWXnvt7LfffhkyZEily2qwLrvssuy9996VLqNB+eyzz3LSSSdl0003zbrrrps999wzzz//fKXLalBGjRqV3r17Z8MNN0znzp1z4IEH5u233650WQ3Su+++m86dO+fWW2+tdCkNyocffphVV111lq+//e1vlS6tQbn99tuz/fbbZ80118wOO+yQe++9t9IlQRIBZJ5z2WWXZcCAAenTp08GDhyYqqqq9OzZM5MnT650aQ3Otddem4suuqjSZTQ4Rx11VF5++eWcd955ufnmm7P66qtn//33zzvvvFPp0hqMX//613n//fdz1VVX5eabb05NTU322WefTJw4sdKlNShTpkzJb3/720yYMKHSpTQ4b775Zpo2bZonnngiTz755Myvn/zkJ5UurcG44447csIJJ+QXv/hF7rrrrmy//fY56qij8tJLL1W6NBBA5iWTJ09Ov379cthhh2WzzTZLx44dc/7552f48OF58MEHK11egzF8+PAccMABufDCC7PCCitUupwGZciQIXnqqady8sknp0uXLllxxRXzu9/9LksttVTuuuuuSpfXIHz66adp165dTj/99Ky55prp0KFDevXqlZEjR+Y///lPpctrUC6++OK0aNGi0mU0SG+99VZWWGGFLLnkklliiSVmftXU1FS6tAahrq4uF154YXr06JEePXpkueWWyyGHHJKNN944zz77bKXLAwFkXjJ48OB8/vnn2XDDDWdua9WqVTp16pTnnnuugpU1LK+//noWWWSR/P3vf8/aa69d6XIalMUWWyxXXnll1lhjjZnbqqqqUldXlzFjxlSwsoZjscUWy3nnnZeVV145SfLJJ5+kb9++adOmTVZaaaUKV9dwPPfccxk4cGD+9Kc/VbqUBunNN9/0+zoX/fe//82HH344S0epb9++OeiggypUFfzPQpUugP/5+OOPkyRLL710ve1LLrlkhg0bVomSGqQtt9wyW265ZaXLaJBatWqVzTbbrN62e++9N0OHDs0mm2xSoaoart///ve56aabUl1dncsvvzzNmzevdEkNwtixY3PMMcfkxBNPnOX9mPJ46623ssQSS+SXv/xl3nvvvSy33HLp1atXunXrVunSGoT33nsvSTJhwoTsv//+GTRoUNq1a5df//rX/v1jnqADMg+ZMb9dXV1db3vTpk0zadKkSpQE38sLL7yQE044IVtttZV/9OaCHj165JZbbslOO+2UQw45JK+//nqlS2oQTjnllKyzzjquR5hLJk+enPfeey/jx4/PEUcckSuvvDJrrrlmevbsmaeffrrS5TUI48ePT5Ice+yx2XHHHdOvX7/88Ic/TK9evZxj5gk6IPOQGbOvkydPrjcHO2nSpDRr1qxSZcF38tBDD+W3v/1t1l577Zx33nmVLqdBmjHCcvrpp+ff//53rr/++px55pkVrmr+dvvtt+f555/PnXfeWelSGqzq6uo899xzWWihhWZ+4LbGGmvknXfeSd++fbPRRhtVuML5X5MmTZIk+++/f376058mSVZbbbUMGjQo11xzjXNMxemAzENmtPpHjBhRb/uIESPSpk2bSpQE38n111+fww47LJtuummuuuoqF5aW0ahRo3LXXXdl2rRpM7c1atQoHTp0mOW9g2/vlltuyahRo7L55punc+fO6dy5c5Lk5JNPzg477FDh6hqO5s2bz9LtX2WVVTJ8+PAKVdSwzPibYZVVVqm3faWVVsoHH3xQiZKgHgFkHtKxY8e0bNkyzzzzzMxtY8eOzaBBg9KlS5cKVgal69+/f04//fTstddeueCCC2b5I4PvZ8SIETn66KPrrWQzZcqUDBo0KB06dKhgZQ3DOeeck3vuuSe33377zK8kOfzww3PllVdWtrgGYvDgwencufMs9wd67bXXXJheJp06dUqLFi3y8ssv19v+1ltvpX379hWqCv7HCNY8pLq6Ot27d88555yT1q1bp23btjn77LPTpk2bbLPNNpUuD77Ru+++mz/84Q/ZZpttctBBB2XUqFEzH6upqcnCCy9cweoaho4dO2aTTTbJqaeemj59+qRVq1a54oorMnbs2Oyzzz6VLm++t9RSS812++KLL562bdsWXE3DtMoqq2TllVfOqaeempNPPjmLLbZYbrrppvz73//OzTffXOnyGoSampoccMABufTSS7PUUktlrbXWyt13352nnnoq1157baXLAwFkXnP44Ydn6tSpOfHEE/PFF1+ka9eu6du3r0+RmS/cf//9mTJlSh588MFZ7l3z05/+NH/84x8rVFnDUVVVlQsuuCDnnntujjjiiIwbNy5dunTJDTfckGWWWabS5cE3atSoUa644oqcc845OeKIIzJ27Nh06tQp11xzTVZdddVKl9dg9OrVK82aNZt5P7EOHTrk4osvzgYbbFDp0iBVdXV1dZUuAgAAWDC4BgQAACiMAAIAABRGAAEAAAojgAAAAIURQAAAgMIIIAAAQGEEEAAAoDACCAAAUBgBBGA+M2nSpHTq1CmdO3fO6aefXulyAOBbEUAA5jNVVVX5y1/+krXWWivXX3993n333UqXBAAlE0AA5jPV1dXp2rVrDjjggCTJ66+/XuGKAKB0AgjAfGrFFVdMkrzxxhsVrgQASieAAMynrrrqqiTJ4MGDK1wJAJROAAGYDz355JO58cYbs8gii2TQoEGVLgcASiaAAMxnxo4dmxNOOCFbbbVV9txzz4wePTrDhw+vdFkAUBIBBGA+c+qpp2bq1Knp06dPOnXqlMQYFgDzDwEEYD5y33335a677soZZ5yR1q1bzwwgLkQHYH4hgADMJ0aOHJmTTz45v/jFL7LFFlskSZZddtm0atXKdSAAzDcEEID5xO9///ssssgiOe644+ptX2211YxgATDfEEAA5gN/+9vf8vjjj+ess85K8+bN6z3WqVOnDB06NOPHj69QdQBQuqq6urq6ShcBAAAsGHRAAACAwgggAABAYQQQAACgMAIIAABQGAEEAAAojAACAAAURgABAAAKI4AAAACFEUAAAIDCCCAAAEBhBBAAAKAwAggAAFCY/wOanGnFWRH3rQAAAABJRU5ErkJggg==",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "filenames": {
+ "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week42_262_1.png"
+ }
+ },
+ "output_type": "display_data"
+ }
+ ],
"source": [
"# optional\n",
"# visual representation of grid search\n",
@@ -4551,7 +5858,7 @@
},
{
"cell_type": "markdown",
- "id": "8f5f8f4b",
+ "id": "0d45b429",
"metadata": {
"editable": true
},
@@ -4569,7 +5876,7 @@
},
{
"cell_type": "markdown",
- "id": "6ec638b9",
+ "id": "67aec670",
"metadata": {
"editable": true
},
@@ -4604,19 +5911,28 @@
{
"cell_type": "code",
"execution_count": 14,
- "id": "6443ca72",
+ "id": "0ab38a83",
"metadata": {
"collapsed": false,
"editable": true
},
- "outputs": [],
+ "outputs": [
+ {
+ "ename": "SyntaxError",
+ "evalue": "invalid syntax (2357089093.py, line 1)",
+ "output_type": "error",
+ "traceback": [
+ "\u001b[0;36m Cell \u001b[0;32mIn[14], line 1\u001b[0;36m\u001b[0m\n\u001b[0;31m pip3 install tensorflow\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
+ ]
+ }
+ ],
"source": [
"pip3 install tensorflow"
]
},
{
"cell_type": "markdown",
- "id": "a8918d58",
+ "id": "8f53f5b9",
"metadata": {
"editable": true
},
@@ -4628,7 +5944,7 @@
{
"cell_type": "code",
"execution_count": 15,
- "id": "20e9a0f5",
+ "id": "5190c7ff",
"metadata": {
"collapsed": false,
"editable": true
@@ -4641,7 +5957,7 @@
},
{
"cell_type": "markdown",
- "id": "1381a34b",
+ "id": "676ff9d7",
"metadata": {
"editable": true
},
@@ -4652,7 +5968,7 @@
{
"cell_type": "code",
"execution_count": 16,
- "id": "80e92af9",
+ "id": "479149f7",
"metadata": {
"collapsed": false,
"editable": true
@@ -4665,7 +5981,7 @@
},
{
"cell_type": "markdown",
- "id": "074ac69f",
+ "id": "62d1b789",
"metadata": {
"editable": true
},
@@ -4680,7 +5996,7 @@
{
"cell_type": "code",
"execution_count": 17,
- "id": "0d710bd3",
+ "id": "4a11035a",
"metadata": {
"collapsed": false,
"editable": true
@@ -4692,7 +6008,7 @@
},
{
"cell_type": "markdown",
- "id": "7adbcab8",
+ "id": "abf44b70",
"metadata": {
"editable": true
},
@@ -4704,7 +6020,7 @@
},
{
"cell_type": "markdown",
- "id": "e17c7253",
+ "id": "3f163559",
"metadata": {
"editable": true
},
@@ -4717,7 +6033,7 @@
{
"cell_type": "code",
"execution_count": 18,
- "id": "5ae741f5",
+ "id": "f7418c1e",
"metadata": {
"collapsed": false,
"editable": true
@@ -4772,7 +6088,7 @@
{
"cell_type": "code",
"execution_count": 19,
- "id": "4f7ef6bb",
+ "id": "49ec0156",
"metadata": {
"collapsed": false,
"editable": true
@@ -4801,7 +6117,7 @@
{
"cell_type": "code",
"execution_count": 20,
- "id": "7b74e049",
+ "id": "302ad127",
"metadata": {
"collapsed": false,
"editable": true
@@ -4831,7 +6147,7 @@
{
"cell_type": "code",
"execution_count": 21,
- "id": "eaa25983",
+ "id": "436a3e0a",
"metadata": {
"collapsed": false,
"editable": true
@@ -4858,7 +6174,7 @@
{
"cell_type": "code",
"execution_count": 22,
- "id": "949eca1f",
+ "id": "a26e83e0",
"metadata": {
"collapsed": false,
"editable": true
@@ -4900,7 +6216,7 @@
},
{
"cell_type": "markdown",
- "id": "bb8b8ac1",
+ "id": "8d69b494",
"metadata": {
"editable": true
},
@@ -4911,7 +6227,7 @@
{
"cell_type": "code",
"execution_count": 23,
- "id": "9d5ebbb2",
+ "id": "cad16bbe",
"metadata": {
"collapsed": false,
"editable": true
@@ -5088,7 +6404,7 @@
},
{
"cell_type": "markdown",
- "id": "f1baeb0b",
+ "id": "61624838",
"metadata": {
"editable": true
},
@@ -5107,7 +6423,7 @@
},
{
"cell_type": "markdown",
- "id": "6549fa13",
+ "id": "f825f2da",
"metadata": {
"editable": true
},
@@ -5129,7 +6445,7 @@
{
"cell_type": "code",
"execution_count": 24,
- "id": "21062ebc",
+ "id": "409b4250",
"metadata": {
"collapsed": false,
"editable": true
@@ -5270,7 +6586,7 @@
},
{
"cell_type": "markdown",
- "id": "f965b277",
+ "id": "c6830d86",
"metadata": {
"editable": true
},
@@ -5286,7 +6602,7 @@
{
"cell_type": "code",
"execution_count": 25,
- "id": "35ab935b",
+ "id": "041fc0bf",
"metadata": {
"collapsed": false,
"editable": true
@@ -5299,7 +6615,7 @@
},
{
"cell_type": "markdown",
- "id": "b26972b4",
+ "id": "0e18ef84",
"metadata": {
"editable": true
},
@@ -5311,7 +6627,7 @@
{
"cell_type": "code",
"execution_count": 26,
- "id": "3df9caac",
+ "id": "8be0e7fd",
"metadata": {
"collapsed": false,
"editable": true
@@ -5333,7 +6649,7 @@
},
{
"cell_type": "markdown",
- "id": "1c07f6da",
+ "id": "2d2bc7a5",
"metadata": {
"editable": true
},
@@ -5349,7 +6665,7 @@
{
"cell_type": "code",
"execution_count": 27,
- "id": "743aa0bd",
+ "id": "f5cb107e",
"metadata": {
"collapsed": false,
"editable": true
@@ -5387,7 +6703,7 @@
},
{
"cell_type": "markdown",
- "id": "c78bdc16",
+ "id": "beb4f622",
"metadata": {
"editable": true
},
@@ -5400,7 +6716,7 @@
{
"cell_type": "code",
"execution_count": 28,
- "id": "4ec3a28c",
+ "id": "1b508839",
"metadata": {
"collapsed": false,
"editable": true
@@ -5421,7 +6737,7 @@
},
{
"cell_type": "markdown",
- "id": "5c924e11",
+ "id": "afb2e0af",
"metadata": {
"editable": true
},
@@ -5437,7 +6753,7 @@
{
"cell_type": "code",
"execution_count": 29,
- "id": "87402ee9",
+ "id": "b96d6c89",
"metadata": {
"collapsed": false,
"editable": true
@@ -5495,7 +6811,7 @@
},
{
"cell_type": "markdown",
- "id": "45ddd0bc",
+ "id": "0be588f8",
"metadata": {
"editable": true
},
@@ -5510,7 +6826,7 @@
{
"cell_type": "code",
"execution_count": 30,
- "id": "77034679",
+ "id": "49b8531e",
"metadata": {
"collapsed": false,
"editable": true
@@ -5531,7 +6847,7 @@
},
{
"cell_type": "markdown",
- "id": "9fd8140f",
+ "id": "874d306a",
"metadata": {
"editable": true
},
@@ -5555,7 +6871,7 @@
{
"cell_type": "code",
"execution_count": 31,
- "id": "7540d755",
+ "id": "c25e9955",
"metadata": {
"collapsed": false,
"editable": true
@@ -6027,7 +7343,7 @@
},
{
"cell_type": "markdown",
- "id": "5ee996ef",
+ "id": "86746540",
"metadata": {
"editable": true
},
@@ -6039,7 +7355,7 @@
{
"cell_type": "code",
"execution_count": 32,
- "id": "183b180b",
+ "id": "6f232f0a",
"metadata": {
"collapsed": false,
"editable": true
@@ -6083,7 +7399,7 @@
},
{
"cell_type": "markdown",
- "id": "21a48508",
+ "id": "7227f21d",
"metadata": {
"editable": true
},
@@ -6099,7 +7415,7 @@
{
"cell_type": "code",
"execution_count": 33,
- "id": "37eba90c",
+ "id": "944ba89b",
"metadata": {
"collapsed": false,
"editable": true
@@ -6114,7 +7430,7 @@
},
{
"cell_type": "markdown",
- "id": "58a6b726",
+ "id": "3cef110a",
"metadata": {
"editable": true
},
@@ -6125,7 +7441,7 @@
{
"cell_type": "code",
"execution_count": 34,
- "id": "046d7076",
+ "id": "8eb39d5e",
"metadata": {
"collapsed": false,
"editable": true
@@ -6140,7 +7456,7 @@
},
{
"cell_type": "markdown",
- "id": "981e95ce",
+ "id": "4c0ab092",
"metadata": {
"editable": true
},
@@ -6156,7 +7472,7 @@
{
"cell_type": "code",
"execution_count": 35,
- "id": "6da23296",
+ "id": "af77a32b",
"metadata": {
"collapsed": false,
"editable": true
@@ -6170,7 +7486,7 @@
},
{
"cell_type": "markdown",
- "id": "ba0fea41",
+ "id": "dfdb722f",
"metadata": {
"editable": true
},
@@ -6185,7 +7501,7 @@
{
"cell_type": "code",
"execution_count": 36,
- "id": "1bd5c3e5",
+ "id": "d740dbad",
"metadata": {
"collapsed": false,
"editable": true
@@ -6211,7 +7527,7 @@
{
"cell_type": "code",
"execution_count": 37,
- "id": "9923ffbf",
+ "id": "bfb7c28b",
"metadata": {
"collapsed": false,
"editable": true
@@ -6226,7 +7542,7 @@
},
{
"cell_type": "markdown",
- "id": "95431510",
+ "id": "17a8dc93",
"metadata": {
"editable": true
},
@@ -6237,7 +7553,7 @@
{
"cell_type": "code",
"execution_count": 38,
- "id": "9d71cef9",
+ "id": "7efc0180",
"metadata": {
"collapsed": false,
"editable": true
@@ -6252,7 +7568,7 @@
},
{
"cell_type": "markdown",
- "id": "acfee190",
+ "id": "8255133c",
"metadata": {
"editable": true
},
@@ -6263,7 +7579,7 @@
{
"cell_type": "code",
"execution_count": 39,
- "id": "c834fb60",
+ "id": "4fa47196",
"metadata": {
"collapsed": false,
"editable": true
@@ -6283,7 +7599,7 @@
{
"cell_type": "code",
"execution_count": 40,
- "id": "0c557301",
+ "id": "b7b5ed9f",
"metadata": {
"collapsed": false,
"editable": true
@@ -6298,7 +7614,7 @@
},
{
"cell_type": "markdown",
- "id": "750490a3",
+ "id": "bc0fc41e",
"metadata": {
"editable": true
},
@@ -6313,7 +7629,7 @@
{
"cell_type": "code",
"execution_count": 41,
- "id": "5318d06b",
+ "id": "4ccf32f6",
"metadata": {
"collapsed": false,
"editable": true
@@ -6350,7 +7666,7 @@
},
{
"cell_type": "markdown",
- "id": "ff519831",
+ "id": "382301fa",
"metadata": {
"editable": true
},
@@ -6363,7 +7679,7 @@
{
"cell_type": "code",
"execution_count": 42,
- "id": "57e3fb33",
+ "id": "0feb8f2a",
"metadata": {
"collapsed": false,
"editable": true
@@ -6386,7 +7702,7 @@
},
{
"cell_type": "markdown",
- "id": "e4c08bad",
+ "id": "3f48285b",
"metadata": {
"editable": true
},
@@ -6406,7 +7722,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.9.18"
+ "version": "3.9.15"
}
},
"nbformat": 4,
diff --git a/doc/LectureNotes/_build/jupyter_execute/week42.py b/doc/LectureNotes/_build/jupyter_execute/week42.py
index 7fa538895..1c2f4c8ea 100644
--- a/doc/LectureNotes/_build/jupyter_execute/week42.py
+++ b/doc/LectureNotes/_build/jupyter_execute/week42.py
@@ -294,19 +294,19 @@ for i in range(50):
# layer with two hidden nodes and one output layer with one output node/neuron only (see graph)..
#
# We need to define the following parameters and variables with the input layer (layer $(0)$)
-# where we label the nodes $x_0$ and $x_1$
+# where we label the nodes $x_1$ and $x_2$
# $$
-# x_0 = a_0^{(0)} \wedge x_1 = a_1^{(0)}.
+# x_1 = a_1^{(0)} \wedge x_2 = a_2^{(0)}.
# $$
-# The hidden layer (layer $(1)$) has nodes which yield the outputs $a_0^{(1)}$ and $a_1^{(1)}$) with weight $\boldsymbol{w}$ and bias $\boldsymbol{b}$ parameters
+# The hidden layer (layer $(1)$) has nodes which yield the outputs $a_1^{(1)}$ and $a_2^{(1)}$) with weight $\boldsymbol{w}$ and bias $\boldsymbol{b}$ parameters
# $$
-# w_{ij}^{(1)}=\left\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)}\right\} \wedge b^{(1)}=\left\{b_0^{(1)},b_1^{(1)}\right\}.
+# w_{ij}^{(1)}=\left\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)}\right\} \wedge b^{(1)}=\left\{b_1^{(1)},b_2^{(1)}\right\}.
# $$
-# ## Layout of a simple neural network with two input nodes, one hidden layer and one output node
+# ## Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node
#
#
#
@@ -316,17 +316,17 @@ for i in range(50):
# ## The ouput layer
#
-# Finally, we have the ouput layer given by layer label $(2)$ with output $a^{(2)}$ and weights and biases to be determined given by the variables
+# We have the ouput layer given by layer label $(2)$ with output $a^{(2)}$ and weights and biases to be determined given by the variables
# $$
-# w_{i}^{(2)}=\left\{w_{0}^{(2)},w_{1}^{(2)}\right\} \wedge b^{(2)}.
+# w_{i}^{(2)}=\left\{w_{1}^{(2)},w_{2}^{(2)}\right\} \wedge b^{(2)}.
# $$
# Our output is $\tilde{y}=a^{(2)}$ and we define a generic cost function $C(a^{(2)},y;\boldsymbol{\Theta})$ where $y$ is the target value (a scalar here).
# The parameters we need to optimize are given by
# $$
-# \boldsymbol{\Theta}=\left\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)},w_{0}^{(2)},w_{1}^{(2)},b_0^{(1)},b_1^{(1)},b^{(2)}\right\}.
+# \boldsymbol{\Theta}=\left\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)},w_{1}^{(2)},w_{2}^{(2)},b_1^{(1)},b_2^{(1)},b^{(2)}\right\}.
# $$
# ## Compact expressions
@@ -335,13 +335,13 @@ for i in range(50):
# The inputs to the first hidden layer are
# $$
-# \begin{bmatrix}z_0^{(1)} \\ z_1^{(1)} \end{bmatrix}=\left(\begin{bmatrix}w_{00}^{(1)} & w_{01}^{(1)}\\ w_{10}^{(1)} &w_{11}^{(1)} \end{bmatrix}\right)^{T}\begin{bmatrix}a_0^{(0)} \\ a_1^{(0)} \end{bmatrix}+\begin{bmatrix}b_0^{(1)} \\ b_1^{(1)} \end{bmatrix},
+# \begin{bmatrix}z_1^{(1)} \\ z_2^{(1)} \end{bmatrix}=\left(\begin{bmatrix}w_{11}^{(1)} & w_{12}^{(1)}\\ w_{21}^{(1)} &w_{22}^{(1)} \end{bmatrix}\right)^{T}\begin{bmatrix}a_1^{(0)} \\ a_2^{(0)} \end{bmatrix}+\begin{bmatrix}b_1^{(1)} \\ b_2^{(1)} \end{bmatrix},
# $$
# with outputs
# $$
-# \begin{bmatrix}a_0^{(1)} \\ a_1^{(1)} \end{bmatrix}=\begin{bmatrix}\sigma^{(1)}(z_0^{(1)}) \\ \sigma^{(1)}(z_1^{(1)}) \end{bmatrix}.
+# \begin{bmatrix}a_1^{(1)} \\ a_2^{(1)} \end{bmatrix}=\begin{bmatrix}\sigma^{(1)}(z_1^{(1)}) \\ \sigma^{(1)}(z_2^{(1)}) \end{bmatrix}.
# $$
# ## Output layer
@@ -349,7 +349,7 @@ for i in range(50):
# For the final output layer we have the inputs to the final activation function
# $$
-# z^{(2)} = w_{0}^{(2)}a_0^{(1)} +w_{1}^{(2)}a_1^{(1)}+b^{(2)},
+# z^{(2)} = w_{1}^{(2)}a_1^{(1)} +w_{2}^{(2)}a_2^{(1)}+b^{(2)},
# $$
# resulting in the output
@@ -387,39 +387,39 @@ for i in range(50):
# Using the chain rule we have the following expressions for say one of the weight parameters (it is easy to generalize to the other weight parameters)
# $$
-# \frac{\partial C}{\partial w_{00}^{(1)}}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}
-# \frac{\partial z^{(2)}}{\partial z_0^{(1)}}\frac{\partial z_0^{(1)}}{\partial w_{00}^{(1)}}= \delta^{(2)}\frac{\partial z^{(2)}}{\partial z_0^{(1)}}\frac{\partial z_0^{(1)}}{\partial w_{00}^{(1)}},
+# \frac{\partial C}{\partial w_{11}^{(1)}}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}
+# \frac{\partial z^{(2)}}{\partial z_1^{(1)}}\frac{\partial z_1^{(1)}}{\partial w_{11}^{(1)}}= \delta^{(2)}\frac{\partial z^{(2)}}{\partial z_1^{(1)}}\frac{\partial z_1^{(1)}}{\partial w_{11}^{(1)}},
# $$
# which, noting that
# $$
-# z^{(2)} =w_0^{(2)}a_0^{(1)}+w_1^{(2)}a_1^{(1)}+b^{(2)},
+# z^{(2)} =w_1^{(2)}a_1^{(1)}+w_2^{(2)}a_2^{(1)}+b^{(2)},
# $$
# allows us to rewrite
# $$
-# \frac{\partial z^{(2)}}{\partial z_0^{(1)}}\frac{\partial z_0^{(1)}}{\partial w_{00}^{(1)}}=w_0^{(2)}\frac{\partial a_0^{(1)}}{\partial z_0^{(1)}}a_0^{(1)}.
+# \frac{\partial z^{(2)}}{\partial z_1^{(1)}}\frac{\partial z_1^{(1)}}{\partial w_{11}^{(1)}}=w_1^{(2)}\frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}a_1^{(1)}.
# $$
# ## Final expression
# Defining
# $$
-# \delta_0^{(1)}=w_0^{(2)}\frac{\partial a_0^{(1)}}{\partial z_0^{(1)}}\delta^{(2)},
+# \delta_1^{(1)}=w_1^{(2)}\frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}\delta^{(2)},
# $$
# we have
# $$
-# \frac{\partial C}{\partial w_{00}^{(1)}}=\delta_0^{(1)}a_0^{(1)}.
+# \frac{\partial C}{\partial w_{11}^{(1)}}=\delta_1^{(1)}a_1^{(1)}.
# $$
# Similarly, we obtain
# $$
-# \frac{\partial C}{\partial w_{01}^{(1)}}=\delta_0^{(1)}a_1^{(1)}.
+# \frac{\partial C}{\partial w_{12}^{(1)}}=\delta_1^{(1)}a_2^{(1)}.
# $$
# ## Completing the list
@@ -427,19 +427,19 @@ for i in range(50):
# Similarly, we find
# $$
-# \frac{\partial C}{\partial w_{10}^{(1)}}=\delta_1^{(1)}a_0^{(1)},
+# \frac{\partial C}{\partial w_{21}^{(1)}}=\delta_2^{(1)}a_1^{(1)},
# $$
# and
# $$
-# \frac{\partial C}{\partial w_{11}^{(1)}}=\delta_1^{(1)}a_1^{(1)},
+# \frac{\partial C}{\partial w_{22}^{(1)}}=\delta_2^{(1)}a_2^{(1)},
# $$
# where we have defined
# $$
-# \delta_1^{(1)}=w_1^{(2)}\frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}\delta^{(2)}.
+# \delta_2^{(1)}=w_2^{(2)}\frac{\partial a_2^{(1)}}{\partial z_2^{(1)}}\delta^{(2)}.
# $$
# ## Final expressions for the biases of the hidden layer
@@ -447,13 +447,13 @@ for i in range(50):
# For the sake of completeness, we list the derivatives of the biases, which are
# $$
-# \frac{\partial C}{\partial b_{0}^{(1)}}=\delta_0^{(1)},
+# \frac{\partial C}{\partial b_{1}^{(1)}}=\delta_1^{(1)},
# $$
# and
# $$
-# \frac{\partial C}{\partial b_{1}^{(1)}}=\delta_1^{(1)}.
+# \frac{\partial C}{\partial b_{2}^{(1)}}=\delta_2^{(1)}.
# $$
# As we will see below, these expressions can be generalized in a more compact form.
@@ -734,9 +734,27 @@ for i in range(50):
#
# We are now ready to set up the algorithm for back propagation and learning the weights and biases.
-# ## Setting up the back propagation algorithm
+# ## Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations
#
-# The four equations provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.
+# **The architecture (our model).**
+#
+# 1. Set up your inputs and outputs (scalars, vectors, matrices or higher-order arrays)
+#
+# 2. Define the number of hidden layers and hidden nodes
+#
+# 3. Define activation functions for hidden layers and output layers
+#
+# 4. Define optimizer (plan learning rate, momentum, ADAgrad, RMSprop, ADAM etc) and array of initial learning rates
+#
+# 5. Define cost function and possible regularization terms with hyperparameters
+#
+# 6. Initialize weights and biases
+#
+# 7. Fix number of iterations for the feed forward part and back propagation part
+
+# ## Setting up the back propagation algorithm, part 1
+#
+# The four equations provide us with a way of computing the gradients of the cost function. Let us write this out in the form of an algorithm.
#
# **First**, we set up the input data $\boldsymbol{x}$ and the activations
# $\boldsymbol{z}_1$ of the input layer and compute the activation function and
@@ -797,12 +815,12 @@ for i in range(50):
# b_j^l \leftarrow b_j^l-\eta \frac{\partial {\cal C}}{\partial b_j^l}=b_j^l-\eta \delta_j^l,
# $$
-# ### Activation functions
+# ## Activation functions
#
# A property that characterizes a neural network, other than its
-# connectivity, is the choice of activation function(s). As described
-# in, the following restrictions are imposed on an activation function
-# for a FFNN to fulfill the universal approximation theorem
+# connectivity, is the choice of activation function(s). The following
+# restrictions are imposed on an activation function for an FFNN to
+# fulfill the universal approximation theorem
#
# * Non-constant
#
@@ -833,7 +851,7 @@ for i in range(50):
# \sigma(x) = \tanh(x)
# $$
-# ### Relevance
+# ## Relevance
#
# The *sigmoid* function are more biologically plausible because the
# output of inactive neurons are zero. Such activation function are
@@ -919,45 +937,6 @@ ax.set_title('Rectified linear unit')
plt.show()
-# ## Fine-tuning neural network hyperparameters
-#
-# The flexibility of neural networks is also one of their main
-# drawbacks: there are many hyperparameters to tweak. Not only can you
-# use any imaginable network topology (how neurons/nodes are
-# interconnected), but even in a simple FFNN you can change the number
-# of layers, the number of neurons per layer, the type of activation
-# function to use in each layer, the weight initialization logic, the
-# stochastic gradient optmized and much more. How do you know what
-# combination of hyperparameters is the best for your task?
-#
-# * You can use grid search with cross-validation to find the right hyperparameters.
-#
-# However,since there are many hyperparameters to tune, and since
-# training a neural network on a large dataset takes a lot of time, you
-# will only be able to explore a tiny part of the hyperparameter space.
-#
-# * You can use randomized search.
-#
-# * Or use tools like [Oscar](http://oscar.calldesk.ai/), which implements more complex algorithms to help you find a good set of hyperparameters quickly.
-
-# ## Hidden layers
-#
-# For many problems you can start with just one or two hidden layers and
-# it will work just fine. For the MNIST data set you ca easily get a
-# high accuracy using just one hidden layer with a few hundred neurons.
-# You can reach for this data set above 98% accuracy using two hidden
-# layers with the same total amount of neurons, in roughly the same
-# amount of training time.
-#
-# For more complex problems, you can gradually ramp up the number of
-# hidden layers, until you start overfitting the training set. Very
-# complex tasks, such as large image classification or speech
-# recognition, typically require networks with dozens of layers and they
-# need a huge amount of training data. However, you will rarely have to
-# train such networks from scratch: it is much more common to reuse
-# parts of a pretrained state-of-the-art network that performs a similar
-# task.
-
# ## Vanishing gradients
#
# The Back propagation algorithm we derived above works by going from
@@ -1091,6 +1070,45 @@ plt.show()
#
# * For regression tasks, you can simply use no activation function at all.
+# ## Fine-tuning neural network hyperparameters
+#
+# The flexibility of neural networks is also one of their main
+# drawbacks: there are many hyperparameters to tweak. Not only can you
+# use any imaginable network topology (how neurons/nodes are
+# interconnected), but even in a simple FFNN you can change the number
+# of layers, the number of neurons per layer, the type of activation
+# function to use in each layer, the weight initialization logic, the
+# stochastic gradient optmized and much more. How do you know what
+# combination of hyperparameters is the best for your task?
+#
+# * You can use grid search with cross-validation to find the right hyperparameters.
+#
+# However,since there are many hyperparameters to tune, and since
+# training a neural network on a large dataset takes a lot of time, you
+# will only be able to explore a tiny part of the hyperparameter space.
+#
+# * You can use randomized search.
+#
+# * Or use tools like [Oscar](http://oscar.calldesk.ai/), which implements more complex algorithms to help you find a good set of hyperparameters quickly.
+
+# ## Hidden layers
+#
+# For many problems you can start with just one or two hidden layers and
+# it will work just fine. For the MNIST data set discussed below you can easily get a
+# high accuracy using just one hidden layer with a few hundred neurons.
+# You can reach for this data set above 98% accuracy using two hidden
+# layers with the same total amount of neurons, in roughly the same
+# amount of training time.
+#
+# For more complex problems, you can gradually ramp up the number of
+# hidden layers, until you start overfitting the training set. Very
+# complex tasks, such as large image classification or speech
+# recognition, typically require networks with dozens of layers and they
+# need a huge amount of training data. However, you will rarely have to
+# train such networks from scratch: it is much more common to reuse
+# parts of a pretrained state-of-the-art network that performs a similar
+# task.
+
# ## Batch Normalization
#
# Batch Normalization aims to address the vanishing/exploding gradients
@@ -1200,44 +1218,6 @@ plt.show()
#
# Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumvent these problems.
-# ## Setting up the back-propagation algorithm
-#
-# Let us write this out in the form of an algorithm.
-#
-# First, we set up the input data $\boldsymbol{x}$ and the activations
-# $\boldsymbol{z}_1$ of the input layer and compute the activation function and
-# the pertinent outputs $\boldsymbol{a}^1$.
-#
-# Secondly, we perform then the feed forward till we reach the output
-# layer and compute all $\boldsymbol{z}_l$ of the input layer and compute the
-# activation function and the pertinent outputs $\boldsymbol{a}^l$ for
-# $l=2,3,\dots,L$.
-#
-# Thereafter we compute the ouput error $\boldsymbol{\delta}^L$ by computing all
-
-# $$
-# \delta_j^L = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}.
-# $$
-
-# Then we compute the back propagate error for each $l=L-1,L-2,\dots,2$ as
-
-# $$
-# \delta_j^l = \sum_k \delta_k^{l+1}w_{kj}^{l+1}\sigma'(z_j^l).
-# $$
-
-# Finally, we update the weights and the biases using gradient descent for each $l=L-1,L-2,\dots,2$ and update the weights and biases according to the rules
-
-# $$
-# w_{ij}^l\leftarrow = w_{ij}^l- \eta \delta_j^la_i^{l-1},
-# $$
-
-# $$
-# b_j^l \leftarrow b_j^l-\eta \frac{\partial {\cal C}}{\partial b_j^l}=b_j^l-\eta \delta_j^l,
-# $$
-
-# The parameter $\eta$ is the learning parameter discussed in connection with the gradient descent methods.
-# Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training.
-
# ## Setting up a Multi-layer perceptron model for classification
#
# We are now gong to develop an example based on the MNIST data
diff --git a/doc/LectureNotes/_build/jupyter_execute/week42_174_0.png b/doc/LectureNotes/_build/jupyter_execute/week42_174_0.png
new file mode 100644
index 000000000..492df86d6
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week42_174_0.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week42_174_1.png b/doc/LectureNotes/_build/jupyter_execute/week42_174_1.png
new file mode 100644
index 000000000..90948aee6
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week42_174_1.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week42_174_2.png b/doc/LectureNotes/_build/jupyter_execute/week42_174_2.png
new file mode 100644
index 000000000..a76574b3e
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week42_174_2.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week42_174_3.png b/doc/LectureNotes/_build/jupyter_execute/week42_174_3.png
new file mode 100644
index 000000000..fc2c8902f
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week42_174_3.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week42_235_1.png b/doc/LectureNotes/_build/jupyter_execute/week42_235_1.png
new file mode 100644
index 000000000..72515c54f
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week42_235_1.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week42_258_1.png b/doc/LectureNotes/_build/jupyter_execute/week42_258_1.png
new file mode 100644
index 000000000..f1288a3be
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week42_258_1.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week42_258_2.png b/doc/LectureNotes/_build/jupyter_execute/week42_258_2.png
new file mode 100644
index 000000000..635de21f9
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week42_258_2.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week42_262_0.png b/doc/LectureNotes/_build/jupyter_execute/week42_262_0.png
new file mode 100644
index 000000000..cab8e99c4
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week42_262_0.png differ
diff --git a/doc/LectureNotes/_build/jupyter_execute/week42_262_1.png b/doc/LectureNotes/_build/jupyter_execute/week42_262_1.png
new file mode 100644
index 000000000..bc6721e0c
Binary files /dev/null and b/doc/LectureNotes/_build/jupyter_execute/week42_262_1.png differ
-
-
-
+
+
+
+
+
@@ -3397,28 +3378,11 @@ the Hadamard product, meaning element-wise multiplication.
+
+
+
+