diff --git a/doc/pub/week41/html/._week41-bs001.html b/doc/pub/week41/html/._week41-bs001.html
index 9c331f568..a8725795f 100644
--- a/doc/pub/week41/html/._week41-bs001.html
+++ b/doc/pub/week41/html/._week41-bs001.html
@@ -348,6 +348,8 @@ MathJax.Hub.Config({
- Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model.
- Building our own Feed-forward Neural Network
+ - Video of lecture notes
+ - Whiteboard notes
- Readings and Videos:
- These lecture notes
diff --git a/doc/pub/week41/html/week41-reveal.html b/doc/pub/week41/html/week41-reveal.html
index f74e19303..f26c14ca6 100644
--- a/doc/pub/week41/html/week41-reveal.html
+++ b/doc/pub/week41/html/week41-reveal.html
@@ -222,6 +222,10 @@ MathJax.Hub.Config({
- Building our own Feed-forward Neural Network
+- Video of lecture notes
+
+- Whiteboard notes
+
- Readings and Videos:
diff --git a/doc/pub/week41/html/week41-solarized.html b/doc/pub/week41/html/week41-solarized.html
index 72813e333..3ce38ba04 100644
--- a/doc/pub/week41/html/week41-solarized.html
+++ b/doc/pub/week41/html/week41-solarized.html
@@ -305,6 +305,8 @@ MathJax.Hub.Config({
- Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model.
- Building our own Feed-forward Neural Network
+ - Video of lecture notes
+ - Whiteboard notes
- Readings and Videos:
- These lecture notes
diff --git a/doc/pub/week41/html/week41.html b/doc/pub/week41/html/week41.html
index 3a8aa790c..8d8482788 100644
--- a/doc/pub/week41/html/week41.html
+++ b/doc/pub/week41/html/week41.html
@@ -382,6 +382,8 @@ MathJax.Hub.Config({
- Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model.
- Building our own Feed-forward Neural Network
+ - Video of lecture notes
+ - Whiteboard notes
- Readings and Videos:
- These lecture notes
diff --git a/doc/pub/week41/ipynb/ipynb-week41-src.tar.gz b/doc/pub/week41/ipynb/ipynb-week41-src.tar.gz
index b6a7c17e4..fc7a0209f 100644
Binary files a/doc/pub/week41/ipynb/ipynb-week41-src.tar.gz and b/doc/pub/week41/ipynb/ipynb-week41-src.tar.gz differ
diff --git a/doc/pub/week41/ipynb/week41.ipynb b/doc/pub/week41/ipynb/week41.ipynb
index 00f15cff8..e7bdb5237 100644
--- a/doc/pub/week41/ipynb/week41.ipynb
+++ b/doc/pub/week41/ipynb/week41.ipynb
@@ -2,8 +2,10 @@
"cells": [
{
"cell_type": "markdown",
- "id": "2ab4355a",
- "metadata": {},
+ "id": "9a4eccc2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
@@ -12,8 +14,10 @@
},
{
"cell_type": "markdown",
- "id": "bb11e4db",
- "metadata": {},
+ "id": "243c5d47",
+ "metadata": {
+ "editable": true
+ },
"source": [
"# Week 41 Neural networks and constructing a neural network code\n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n",
@@ -23,8 +27,10 @@
},
{
"cell_type": "markdown",
- "id": "1d2fb3fd",
- "metadata": {},
+ "id": "0cae636e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Plan for week 41\n",
"\n",
@@ -46,6 +52,10 @@
"\n",
" * Building our own Feed-forward Neural Network\n",
"\n",
+ " * [Video of lecture notes](https://youtu.be/5-RRTO9uDvI)\n",
+ "\n",
+ " * [Whiteboard notes](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesOct12.pdf)\n",
+ "\n",
" * Readings and Videos:\n",
"\n",
" * These lecture notes\n",
@@ -65,16 +75,20 @@
},
{
"cell_type": "markdown",
- "id": "9dcb51d0",
- "metadata": {},
+ "id": "51ff64f7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Lecture Thursday October 12"
]
},
{
"cell_type": "markdown",
- "id": "b2dde1a5",
- "metadata": {},
+ "id": "07ff4601",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Introduction to Neural networks\n",
"\n",
@@ -89,8 +103,10 @@
},
{
"cell_type": "markdown",
- "id": "9dbb7580",
- "metadata": {},
+ "id": "4806cddf",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Artificial neurons\n",
"\n",
@@ -111,8 +127,10 @@
},
{
"cell_type": "markdown",
- "id": "9b37b818",
- "metadata": {},
+ "id": "ccfdcf2b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -127,8 +145,10 @@
},
{
"cell_type": "markdown",
- "id": "f0c18c43",
- "metadata": {},
+ "id": "900fd38b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Here, the output $y$ of the neuron is the value of its activation function, which have as input\n",
"a weighted sum of signals $x_i, \\dots ,x_n$ received by $n$ other neurons.\n",
@@ -165,8 +185,10 @@
},
{
"cell_type": "markdown",
- "id": "4d476078",
- "metadata": {},
+ "id": "15e0098f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Neural network types\n",
"\n",
@@ -192,8 +214,10 @@
},
{
"cell_type": "markdown",
- "id": "6577603f",
- "metadata": {},
+ "id": "70d3fe5d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Feed-forward neural networks\n",
"\n",
@@ -211,8 +235,10 @@
},
{
"cell_type": "markdown",
- "id": "3f765006",
- "metadata": {},
+ "id": "838210ba",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Convolutional Neural Network\n",
"\n",
@@ -238,8 +264,10 @@
},
{
"cell_type": "markdown",
- "id": "20de12ca",
- "metadata": {},
+ "id": "8a78d22d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Recurrent neural networks\n",
"\n",
@@ -257,8 +285,10 @@
},
{
"cell_type": "markdown",
- "id": "48751fb1",
- "metadata": {},
+ "id": "4ca0b479",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Other types of networks\n",
"\n",
@@ -276,8 +306,10 @@
},
{
"cell_type": "markdown",
- "id": "4bc57cc0",
- "metadata": {},
+ "id": "b80ebf1a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Multilayer perceptrons\n",
"\n",
@@ -291,8 +323,10 @@
},
{
"cell_type": "markdown",
- "id": "f194e3be",
- "metadata": {},
+ "id": "749545b4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Why multilayer perceptrons?\n",
"\n",
@@ -310,8 +344,10 @@
},
{
"cell_type": "markdown",
- "id": "11dceaa5",
- "metadata": {},
+ "id": "f524768e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Illustration of a single perceptron model and a multi-perceptron model\n",
"\n",
@@ -324,8 +360,10 @@
},
{
"cell_type": "markdown",
- "id": "9564ced4",
- "metadata": {},
+ "id": "6c2f8b44",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Examples of XOR, OR and AND gates\n",
"\n",
@@ -339,8 +377,11 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "605a35fc",
- "metadata": {},
+ "id": "8dfb9c06",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"\"\"\"\n",
@@ -377,16 +418,20 @@
},
{
"cell_type": "markdown",
- "id": "7682eeab",
- "metadata": {},
+ "id": "b0033599",
+ "metadata": {
+ "editable": true
+ },
"source": [
"What is happening here?"
]
},
{
"cell_type": "markdown",
- "id": "8bbee5fe",
- "metadata": {},
+ "id": "4663d22e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Does Logistic Regression do a better Job?"
]
@@ -394,8 +439,11 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "9ede7953",
- "metadata": {},
+ "id": "846ae942",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -453,16 +501,20 @@
},
{
"cell_type": "markdown",
- "id": "110b8ae5",
- "metadata": {},
+ "id": "22786e56",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Not exactly impressive, but somewhat better."
]
},
{
"cell_type": "markdown",
- "id": "7e5f94af",
- "metadata": {},
+ "id": "b6bdebe1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Adding Neural Networks"
]
@@ -470,8 +522,11 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "f5aee400",
- "metadata": {},
+ "id": "fdfb8244",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"\n",
@@ -487,8 +542,10 @@
},
{
"cell_type": "markdown",
- "id": "812e072e",
- "metadata": {},
+ "id": "13c5155d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Mathematical model\n",
"\n",
@@ -497,8 +554,10 @@
},
{
"cell_type": "markdown",
- "id": "64b04fac",
- "metadata": {},
+ "id": "47b4c719",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"y = f\\left(\\sum_{i=1}^n w_ix_i + b_i\\right) = f(z),\n",
@@ -507,8 +566,10 @@
},
{
"cell_type": "markdown",
- "id": "7f8be757",
- "metadata": {},
+ "id": "d65d78c7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This function receives $x_i$ as inputs.\n",
"Here the activation $z=(\\sum_{i=1}^n w_ix_i+b_i)$. \n",
@@ -520,8 +581,10 @@
},
{
"cell_type": "markdown",
- "id": "a3a920b2",
- "metadata": {},
+ "id": "b8605d0d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Mathematical model\n",
"\n",
@@ -530,8 +593,10 @@
},
{
"cell_type": "markdown",
- "id": "dacd264b",
- "metadata": {},
+ "id": "be0a2d56",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -545,8 +610,10 @@
},
{
"cell_type": "markdown",
- "id": "5cb2ca57",
- "metadata": {},
+ "id": "8a8ec932",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Here $b_i$ is the so-called bias which is normally needed in\n",
"case of zero activation weights or inputs. How to fix the biases and\n",
@@ -558,8 +625,10 @@
},
{
"cell_type": "markdown",
- "id": "c33f37be",
- "metadata": {},
+ "id": "30b5df75",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -574,8 +643,10 @@
},
{
"cell_type": "markdown",
- "id": "8747dd79",
- "metadata": {},
+ "id": "9906f339",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where we assume that all nodes in the same layer have identical\n",
"activation functions, hence the notation $f$. In general, we could assume in the more general case that different layers have different activation functions.\n",
@@ -584,8 +655,10 @@
},
{
"cell_type": "markdown",
- "id": "238b66fc",
- "metadata": {},
+ "id": "705c18d2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -600,8 +673,10 @@
},
{
"cell_type": "markdown",
- "id": "ec349a1b",
- "metadata": {},
+ "id": "c0779fc6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $N_l$ is the number of nodes in layer $l$. When the output of\n",
"all the nodes in the first hidden layer are computed, the values of\n",
@@ -611,8 +686,10 @@
},
{
"cell_type": "markdown",
- "id": "ab5abd1e",
- "metadata": {},
+ "id": "128a41e3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Mathematical model\n",
"\n",
@@ -621,8 +698,10 @@
},
{
"cell_type": "markdown",
- "id": "f4cebe1e",
- "metadata": {},
+ "id": "e8efb8a6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -637,8 +716,10 @@
},
{
"cell_type": "markdown",
- "id": "dd700242",
- "metadata": {},
+ "id": "f975dcc6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -653,16 +734,20 @@
},
{
"cell_type": "markdown",
- "id": "6e309498",
- "metadata": {},
+ "id": "7c3843f7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where we have substituted $y_k^1$ with the inputs $x_k$. Finally, the ANN output reads"
]
},
{
"cell_type": "markdown",
- "id": "aad2882a",
- "metadata": {},
+ "id": "0008d41a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -677,8 +762,10 @@
},
{
"cell_type": "markdown",
- "id": "dc802cd2",
- "metadata": {},
+ "id": "e8dc6d51",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -694,8 +781,10 @@
},
{
"cell_type": "markdown",
- "id": "57e22c56",
- "metadata": {},
+ "id": "1455a093",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Mathematical model\n",
"\n",
@@ -705,8 +794,10 @@
},
{
"cell_type": "markdown",
- "id": "6e91389e",
- "metadata": {},
+ "id": "c2affab6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -721,8 +812,10 @@
},
{
"cell_type": "markdown",
- "id": "8755afdf",
- "metadata": {},
+ "id": "c7910d23",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which illustrates a basic property of MLPs: The only independent\n",
"variables are the input values $x_n$."
@@ -730,8 +823,10 @@
},
{
"cell_type": "markdown",
- "id": "f1dc0077",
- "metadata": {},
+ "id": "ec9e660d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Mathematical model\n",
"\n",
@@ -747,8 +842,10 @@
},
{
"cell_type": "markdown",
- "id": "71fb8777",
- "metadata": {},
+ "id": "29c77377",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -763,8 +860,10 @@
},
{
"cell_type": "markdown",
- "id": "96280b34",
- "metadata": {},
+ "id": "f52146ef",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where the parameters $c_i$ are weights and biases. By adjusting these\n",
"parameters, the activation functions can be shifted up and down or\n",
@@ -774,8 +873,10 @@
},
{
"cell_type": "markdown",
- "id": "6be3363c",
- "metadata": {},
+ "id": "7f9f65ce",
+ "metadata": {
+ "editable": true
+ },
"source": [
"### Matrix-vector notation\n",
"\n",
@@ -792,8 +893,10 @@
},
{
"cell_type": "markdown",
- "id": "d1eb96a8",
- "metadata": {},
+ "id": "7aec05b7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -823,8 +926,10 @@
},
{
"cell_type": "markdown",
- "id": "044d5e68",
- "metadata": {},
+ "id": "524f3145",
+ "metadata": {
+ "editable": true
+ },
"source": [
"### Matrix-vector notation and activation\n",
"\n",
@@ -833,8 +938,10 @@
},
{
"cell_type": "markdown",
- "id": "23a2c440",
- "metadata": {},
+ "id": "67ea322c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -850,8 +957,10 @@
},
{
"cell_type": "markdown",
- "id": "5f6921f1",
- "metadata": {},
+ "id": "fb2a1836",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This is not just a convenient and compact notation, but also a useful\n",
"and intuitive way to think about MLPs: The output is calculated by a\n",
@@ -862,8 +971,10 @@
},
{
"cell_type": "markdown",
- "id": "bc3e047f",
- "metadata": {},
+ "id": "c6e37074",
+ "metadata": {
+ "editable": true
+ },
"source": [
"### Activation functions\n",
"\n",
@@ -883,8 +994,10 @@
},
{
"cell_type": "markdown",
- "id": "f2ff2ee2",
- "metadata": {},
+ "id": "516427eb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"### Activation functions, Logistic and Hyperbolic ones\n",
"\n",
@@ -900,8 +1013,10 @@
},
{
"cell_type": "markdown",
- "id": "a336177e",
- "metadata": {},
+ "id": "607e0e1f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"f(x) = \\frac{1}{1 + e^{-x}},\n",
@@ -910,16 +1025,20 @@
},
{
"cell_type": "markdown",
- "id": "c9a263c9",
- "metadata": {},
+ "id": "4147fa7c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and the *hyperbolic tangent* function"
]
},
{
"cell_type": "markdown",
- "id": "f102c33c",
- "metadata": {},
+ "id": "4c880e2a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"f(x) = \\tanh(x)\n",
@@ -928,8 +1047,10 @@
},
{
"cell_type": "markdown",
- "id": "92eeded3",
- "metadata": {},
+ "id": "56c9cd1b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"### Relevance\n",
"\n",
@@ -943,8 +1064,11 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "17339021",
- "metadata": {},
+ "id": "187aa1c5",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"\"\"\"The sigmoid function (or the logistic curve) is a \n",
@@ -1022,8 +1146,10 @@
},
{
"cell_type": "markdown",
- "id": "8c915f6d",
- "metadata": {},
+ "id": "e3195fd0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The multilayer perceptron (MLP)\n",
"\n",
@@ -1058,8 +1184,10 @@
},
{
"cell_type": "markdown",
- "id": "cdcb2b32",
- "metadata": {},
+ "id": "f15de9c4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## From one to many layers, the universal approximation theorem\n",
"\n",
@@ -1086,8 +1214,10 @@
},
{
"cell_type": "markdown",
- "id": "566b0dcb",
- "metadata": {},
+ "id": "9d79d680",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Deriving the back propagation code for a multilayer perceptron model\n",
"\n",
@@ -1105,8 +1235,10 @@
},
{
"cell_type": "markdown",
- "id": "66c76764",
- "metadata": {},
+ "id": "31e1a9b1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"{\\cal C}(\\hat{W}) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - t_i\\right)^2,\n",
@@ -1115,8 +1247,10 @@
},
{
"cell_type": "markdown",
- "id": "e3d0bfa4",
- "metadata": {},
+ "id": "890921b9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where the $t_i$s are our $n$ targets (the values we want to\n",
"reproduce), while the outputs of the network after having propagated\n",
@@ -1128,8 +1262,10 @@
},
{
"cell_type": "markdown",
- "id": "01cf3c8d",
- "metadata": {},
+ "id": "fe55dffb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Definitions\n",
"\n",
@@ -1143,8 +1279,10 @@
},
{
"cell_type": "markdown",
- "id": "1417eea3",
- "metadata": {},
+ "id": "61548fd5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"z_j^l = \\sum_{i=1}^{M_{l-1}}w_{ij}^la_i^{l-1}+b_j^l,\n",
@@ -1153,8 +1291,10 @@
},
{
"cell_type": "markdown",
- "id": "3c4e10cf",
- "metadata": {},
+ "id": "7b437827",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $b_k^l$ are the biases from layer $l$. Here $M_{l-1}$\n",
"represents the total number of nodes/neurons/units of layer $l-1$. The\n",
@@ -1164,8 +1304,10 @@
},
{
"cell_type": "markdown",
- "id": "d9dc403d",
- "metadata": {},
+ "id": "88aeaf0f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{z}^l = \\left(\\hat{W}^l\\right)^T\\hat{a}^{l-1}+\\hat{b}^l.\n",
@@ -1174,8 +1316,10 @@
},
{
"cell_type": "markdown",
- "id": "83b73b28",
- "metadata": {},
+ "id": "d81878b1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"With the activation values $\\hat{z}^l$ we can in turn define the\n",
"output of layer $l$ as $\\hat{a}^l = f(\\hat{z}^l)$ where $f$ is our\n",
@@ -1186,8 +1330,10 @@
},
{
"cell_type": "markdown",
- "id": "0516dfcf",
- "metadata": {},
+ "id": "aa3e6b55",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"a_j^l = f(z_j^l) = \\frac{1}{1+\\exp{-(z_j^l)}}.\n",
@@ -1196,8 +1342,10 @@
},
{
"cell_type": "markdown",
- "id": "7cebeb6f",
- "metadata": {},
+ "id": "282fc0f1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Derivatives and the chain rule\n",
"\n",
@@ -1206,8 +1354,10 @@
},
{
"cell_type": "markdown",
- "id": "e94a91e9",
- "metadata": {},
+ "id": "5007c640",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial z_j^l}{\\partial w_{ij}^l} = a_i^{l-1},\n",
@@ -1216,16 +1366,20 @@
},
{
"cell_type": "markdown",
- "id": "343a68be",
- "metadata": {},
+ "id": "ed8b0010",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and"
]
},
{
"cell_type": "markdown",
- "id": "1eb93ade",
- "metadata": {},
+ "id": "3174619f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial z_j^l}{\\partial a_i^{l-1}} = w_{ji}^l.\n",
@@ -1234,16 +1388,20 @@
},
{
"cell_type": "markdown",
- "id": "b4294825",
- "metadata": {},
+ "id": "448a714d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"With our definition of the activation function we have that (note that this function depends only on $z_j^l$)"
]
},
{
"cell_type": "markdown",
- "id": "3c84d05d",
- "metadata": {},
+ "id": "862da704",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial a_j^l}{\\partial z_j^{l}} = a_j^l(1-a_j^l)=f(z_j^l)(1-f(z_j^l)).\n",
@@ -1252,8 +1410,10 @@
},
{
"cell_type": "markdown",
- "id": "acd76b39",
- "metadata": {},
+ "id": "9958b437",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Derivative of the cost function\n",
"\n",
@@ -1264,8 +1424,10 @@
},
{
"cell_type": "markdown",
- "id": "f0383586",
- "metadata": {},
+ "id": "be3a329b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"{\\cal C}(\\hat{W^L}) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - t_i\\right)^2=\\frac{1}{2}\\sum_{i=1}^n\\left(a_i^L - t_i\\right)^2,\n",
@@ -1274,16 +1436,20 @@
},
{
"cell_type": "markdown",
- "id": "81c5b644",
- "metadata": {},
+ "id": "2237bf2b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The derivative of this function with respect to the weights is"
]
},
{
"cell_type": "markdown",
- "id": "31fde909",
- "metadata": {},
+ "id": "74e6185e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial{\\cal C}(\\hat{W^L})}{\\partial w_{jk}^L} = \\left(a_j^L - t_j\\right)\\frac{\\partial a_j^L}{\\partial w_{jk}^{L}},\n",
@@ -1292,16 +1458,20 @@
},
{
"cell_type": "markdown",
- "id": "4e4e2612",
- "metadata": {},
+ "id": "462a7694",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The last partial derivative can easily be computed and reads (by applying the chain rule)"
]
},
{
"cell_type": "markdown",
- "id": "9c9e5698",
- "metadata": {},
+ "id": "cde99728",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial a_j^L}{\\partial w_{jk}^{L}} = \\frac{\\partial a_j^L}{\\partial z_{j}^{L}}\\frac{\\partial z_j^L}{\\partial w_{jk}^{L}}=a_j^L(1-a_j^L)a_k^{L-1},\n",
@@ -1310,8 +1480,10 @@
},
{
"cell_type": "markdown",
- "id": "421c9de0",
- "metadata": {},
+ "id": "11c5d932",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Bringing it together, first back propagation equation\n",
"\n",
@@ -1320,8 +1492,10 @@
},
{
"cell_type": "markdown",
- "id": "10597fa3",
- "metadata": {},
+ "id": "d46a1cc5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial{\\cal C}(\\hat{W^L})}{\\partial w_{jk}^L} = \\left(a_j^L - t_j\\right)a_j^L(1-a_j^L)a_k^{L-1},\n",
@@ -1330,16 +1504,20 @@
},
{
"cell_type": "markdown",
- "id": "0cf3157b",
- "metadata": {},
+ "id": "7f634831",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Defining"
]
},
{
"cell_type": "markdown",
- "id": "01128c99",
- "metadata": {},
+ "id": "c2245dab",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\delta_j^L = a_j^L(1-a_j^L)\\left(a_j^L - t_j\\right) = f'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)},\n",
@@ -1348,16 +1526,20 @@
},
{
"cell_type": "markdown",
- "id": "097ee928",
- "metadata": {},
+ "id": "fbda1939",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and using the Hadamard product of two vectors we can write this as"
]
},
{
"cell_type": "markdown",
- "id": "cb2ff4ec",
- "metadata": {},
+ "id": "54e8fb49",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\delta}^L = f'(\\hat{z}^L)\\circ\\frac{\\partial {\\cal C}}{\\partial (\\hat{a}^L)}.\n",
@@ -1366,8 +1548,10 @@
},
{
"cell_type": "markdown",
- "id": "ee3423ea",
- "metadata": {},
+ "id": "0fa8b43d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This is an important expression. The second term on the right handside\n",
"measures how fast the cost function is changing as a function of the $j$th\n",
@@ -1388,8 +1572,10 @@
},
{
"cell_type": "markdown",
- "id": "49a7e0e3",
- "metadata": {},
+ "id": "00455d1e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}\n",
@@ -1398,16 +1584,20 @@
},
{
"cell_type": "markdown",
- "id": "8efd3f58",
- "metadata": {},
+ "id": "8a503b44",
+ "metadata": {
+ "editable": true
+ },
"source": [
"With the definition of $\\delta_j^L$ we have a more compact definition of the derivative of the cost function in terms of the weights, namely"
]
},
{
"cell_type": "markdown",
- "id": "dd61c9f6",
- "metadata": {},
+ "id": "28aaa847",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial{\\cal C}(\\hat{W^L})}{\\partial w_{jk}^L} = \\delta_j^La_k^{L-1}.\n",
@@ -1416,8 +1606,10 @@
},
{
"cell_type": "markdown",
- "id": "7574ecb2",
- "metadata": {},
+ "id": "ca7f309e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Derivatives in terms of $z_j^L$\n",
"\n",
@@ -1426,8 +1618,10 @@
},
{
"cell_type": "markdown",
- "id": "103823da",
- "metadata": {},
+ "id": "79b5d957",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\delta_j^L =\\frac{\\partial {\\cal C}}{\\partial z_j^L}= \\frac{\\partial {\\cal C}}{\\partial a_j^L}\\frac{\\partial a_j^L}{\\partial z_j^L},\n",
@@ -1436,16 +1630,20 @@
},
{
"cell_type": "markdown",
- "id": "8fffaff6",
- "metadata": {},
+ "id": "ba24b0ba",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which can also be interpreted as the partial derivative of the cost function with respect to the biases $b_j^L$, namely"
]
},
{
"cell_type": "markdown",
- "id": "ae94d046",
- "metadata": {},
+ "id": "2f98d5ae",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\delta_j^L = \\frac{\\partial {\\cal C}}{\\partial b_j^L}\\frac{\\partial b_j^L}{\\partial z_j^L}=\\frac{\\partial {\\cal C}}{\\partial b_j^L},\n",
@@ -1454,16 +1652,20 @@
},
{
"cell_type": "markdown",
- "id": "3874588e",
- "metadata": {},
+ "id": "596bd7eb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"That is, the error $\\delta_j^L$ is exactly equal to the rate of change of the cost function as a function of the bias."
]
},
{
"cell_type": "markdown",
- "id": "296c98ed",
- "metadata": {},
+ "id": "9144dcb7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Bringing it together\n",
"\n",
@@ -1474,8 +1676,10 @@
},
{
"cell_type": "markdown",
- "id": "19e9a098",
- "metadata": {},
+ "id": "4f2e52cb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -1490,16 +1694,20 @@
},
{
"cell_type": "markdown",
- "id": "edb4be27",
- "metadata": {},
+ "id": "3b94d154",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and"
]
},
{
"cell_type": "markdown",
- "id": "f85640ae",
- "metadata": {},
+ "id": "18f353fd",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -1514,16 +1722,20 @@
},
{
"cell_type": "markdown",
- "id": "ef718563",
- "metadata": {},
+ "id": "631f4feb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and"
]
},
{
"cell_type": "markdown",
- "id": "69a17671",
- "metadata": {},
+ "id": "1793bcd7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -1538,8 +1750,10 @@
},
{
"cell_type": "markdown",
- "id": "84ef0653",
- "metadata": {},
+ "id": "0cc3885e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"An interesting consequence of the above equations is that when the\n",
"activation $a_k^{L-1}$ is small, the gradient term, that is the\n",
@@ -1563,8 +1777,10 @@
},
{
"cell_type": "markdown",
- "id": "fb567e6f",
- "metadata": {},
+ "id": "7f697fec",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Final back propagating equation\n",
"\n",
@@ -1573,8 +1789,10 @@
},
{
"cell_type": "markdown",
- "id": "42773e4f",
- "metadata": {},
+ "id": "42a9acc8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\delta_j^l =\\frac{\\partial {\\cal C}}{\\partial z_j^l}.\n",
@@ -1583,16 +1801,20 @@
},
{
"cell_type": "markdown",
- "id": "c01f5515",
- "metadata": {},
+ "id": "ac01943b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We want to express this in terms of the equations for layer $l+1$. Using the chain rule and summing over all $k$ entries we have"
]
},
{
"cell_type": "markdown",
- "id": "b23962dc",
- "metadata": {},
+ "id": "a6ec5df0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\delta_j^l =\\sum_k \\frac{\\partial {\\cal C}}{\\partial z_k^{l+1}}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}}=\\sum_k \\delta_k^{l+1}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}},\n",
@@ -1601,16 +1823,20 @@
},
{
"cell_type": "markdown",
- "id": "c84de332",
- "metadata": {},
+ "id": "8b0f324a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and recalling that"
]
},
{
"cell_type": "markdown",
- "id": "ed77e307",
- "metadata": {},
+ "id": "4ae4c890",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"z_j^{l+1} = \\sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_i^{l}+b_j^{l+1},\n",
@@ -1619,16 +1845,20 @@
},
{
"cell_type": "markdown",
- "id": "ab04fbdc",
- "metadata": {},
+ "id": "b8af6a3b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"with $M_l$ being the number of nodes in layer $l$, we obtain"
]
},
{
"cell_type": "markdown",
- "id": "d0ce3d50",
- "metadata": {},
+ "id": "c5a36652",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\delta_j^l =\\sum_k \\delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l),\n",
@@ -1637,8 +1867,10 @@
},
{
"cell_type": "markdown",
- "id": "a217439f",
- "metadata": {},
+ "id": "eb9f0a42",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This is our final equation.\n",
"\n",
@@ -1647,8 +1879,10 @@
},
{
"cell_type": "markdown",
- "id": "71731044",
- "metadata": {},
+ "id": "cd78653c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Setting up the Back propagation algorithm\n",
"\n",
@@ -1668,8 +1902,10 @@
},
{
"cell_type": "markdown",
- "id": "a2afecfd",
- "metadata": {},
+ "id": "adab4f13",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\delta_j^L = f'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n",
@@ -1678,16 +1914,20 @@
},
{
"cell_type": "markdown",
- "id": "0ffa0ea9",
- "metadata": {},
+ "id": "9091074a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Then we compute the back propagate error for each $l=L-1,L-2,\\dots,2$ as"
]
},
{
"cell_type": "markdown",
- "id": "025daba8",
- "metadata": {},
+ "id": "da04b8a8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l).\n",
@@ -1696,16 +1936,20 @@
},
{
"cell_type": "markdown",
- "id": "935b27ef",
- "metadata": {},
+ "id": "d707b6e0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Finally, we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,2$ and update the weights and biases according to the rules"
]
},
{
"cell_type": "markdown",
- "id": "50179be2",
- "metadata": {},
+ "id": "f26ac396",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"w_{jk}^l\\leftarrow = w_{jk}^l- \\eta \\delta_j^la_k^{l-1},\n",
@@ -1714,8 +1958,10 @@
},
{
"cell_type": "markdown",
- "id": "90daf72c",
- "metadata": {},
+ "id": "1d5d1ed0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n",
@@ -1724,8 +1970,10 @@
},
{
"cell_type": "markdown",
- "id": "63e9545c",
- "metadata": {},
+ "id": "d686cbb0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The parameter $\\eta$ is the learning parameter discussed in connection with the gradient descent methods.\n",
"Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training."
@@ -1733,8 +1981,10 @@
},
{
"cell_type": "markdown",
- "id": "e8fbaffb",
- "metadata": {},
+ "id": "31554688",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Setting up the Back propagation algorithm\n",
"\n",
@@ -1754,8 +2004,10 @@
},
{
"cell_type": "markdown",
- "id": "59b2fcba",
- "metadata": {},
+ "id": "a34466dc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\delta_j^L = f'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n",
@@ -1764,16 +2016,20 @@
},
{
"cell_type": "markdown",
- "id": "db7fd0df",
- "metadata": {},
+ "id": "742b0899",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Then we compute the back propagate error for each $l=L-1,L-2,\\dots,2$ as"
]
},
{
"cell_type": "markdown",
- "id": "fef9229b",
- "metadata": {},
+ "id": "813c7f07",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l).\n",
@@ -1782,16 +2038,20 @@
},
{
"cell_type": "markdown",
- "id": "720ee2fe",
- "metadata": {},
+ "id": "88fccb02",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Finally, we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,2$ and update the weights and biases according to the rules"
]
},
{
"cell_type": "markdown",
- "id": "df965f5a",
- "metadata": {},
+ "id": "a5d5593c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"w_{jk}^l\\leftarrow = w_{jk}^l- \\eta \\delta_j^la_k^{l-1},\n",
@@ -1800,8 +2060,10 @@
},
{
"cell_type": "markdown",
- "id": "29bef69b",
- "metadata": {},
+ "id": "1b5aeaa8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n",
@@ -1810,8 +2072,10 @@
},
{
"cell_type": "markdown",
- "id": "8f8670bc",
- "metadata": {},
+ "id": "29156297",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The parameter $\\eta$ is the learning parameter discussed in connection with the gradient descent methods.\n",
"Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training."
@@ -1819,8 +2083,10 @@
},
{
"cell_type": "markdown",
- "id": "b37be623",
- "metadata": {},
+ "id": "752b2b7a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Setting up the Back propagation algorithm\n",
"\n",
@@ -1840,8 +2106,10 @@
},
{
"cell_type": "markdown",
- "id": "6ebda602",
- "metadata": {},
+ "id": "29bb348b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\delta_j^L = f'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n",
@@ -1850,16 +2118,20 @@
},
{
"cell_type": "markdown",
- "id": "6f2371ba",
- "metadata": {},
+ "id": "2cbb4812",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Then we compute the back propagate error for each $l=L-1,L-2,\\dots,2$ as"
]
},
{
"cell_type": "markdown",
- "id": "b3458652",
- "metadata": {},
+ "id": "73fa8d95",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l).\n",
@@ -1868,16 +2140,20 @@
},
{
"cell_type": "markdown",
- "id": "4a46987f",
- "metadata": {},
+ "id": "d815b24a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Finally, we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,2$ and update the weights and biases according to the rules"
]
},
{
"cell_type": "markdown",
- "id": "2b898b0d",
- "metadata": {},
+ "id": "43d2f617",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"w_{jk}^l\\leftarrow = w_{jk}^l- \\eta \\delta_j^la_k^{l-1},\n",
@@ -1886,8 +2162,10 @@
},
{
"cell_type": "markdown",
- "id": "a1af5bcf",
- "metadata": {},
+ "id": "bf7af7f5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n",
@@ -1896,8 +2174,10 @@
},
{
"cell_type": "markdown",
- "id": "8d7a0d4d",
- "metadata": {},
+ "id": "9a5e5eaf",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The parameter $\\eta$ is the learning parameter discussed in connection with the gradient descent methods.\n",
"Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training."
@@ -1905,8 +2185,10 @@
},
{
"cell_type": "markdown",
- "id": "37f6a409",
- "metadata": {},
+ "id": "e62f23f4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Setting up a Multi-layer perceptron model for classification\n",
"\n",
@@ -1931,8 +2213,10 @@
},
{
"cell_type": "markdown",
- "id": "e7939992",
- "metadata": {},
+ "id": "42d7a94e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = \\frac{1}{1 + \\exp{(- \\boldsymbol{x}})} ,\n",
@@ -1941,16 +2225,20 @@
},
{
"cell_type": "markdown",
- "id": "d754aefd",
- "metadata": {},
+ "id": "5bfea505",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and"
]
},
{
"cell_type": "markdown",
- "id": "7711b16e",
- "metadata": {},
+ "id": "9f6b8143",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"P(y = 1 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = 1 - P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) ,\n",
@@ -1959,8 +2247,10 @@
},
{
"cell_type": "markdown",
- "id": "e831f3b5",
- "metadata": {},
+ "id": "194f2cbe",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $y \\in \\{0, 1\\}$ and $\\boldsymbol{\\theta}$ represents the weights and biases\n",
"of our network."
@@ -1968,8 +2258,10 @@
},
{
"cell_type": "markdown",
- "id": "428cb018",
- "metadata": {},
+ "id": "20cd18e6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Defining the cost function\n",
"\n",
@@ -1978,8 +2270,10 @@
},
{
"cell_type": "markdown",
- "id": "303e8c9e",
- "metadata": {},
+ "id": "b3e52796",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\theta}) = - \\ln P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = - \\sum_{i=1}^n\n",
@@ -1989,8 +2283,10 @@
},
{
"cell_type": "markdown",
- "id": "42c3514b",
- "metadata": {},
+ "id": "7cb68d89",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This last equality means that we can interpret our *cost* function as a sum over the *loss* function\n",
"for each point in the dataset $\\mathcal{L}_i(\\boldsymbol{\\theta})$. \n",
@@ -2012,8 +2308,10 @@
},
{
"cell_type": "markdown",
- "id": "6675c087",
- "metadata": {},
+ "id": "0a98f80a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"P(y_{ic} = 1 \\mid \\boldsymbol{x}_i, \\boldsymbol{\\theta}) = \\frac{\\exp{((\\boldsymbol{a}_i^{hidden})^T \\boldsymbol{w}_c)}}\n",
@@ -2023,8 +2321,10 @@
},
{
"cell_type": "markdown",
- "id": "73cfb814",
- "metadata": {},
+ "id": "b86447a6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which reduces to the logistic function in the binary case. \n",
"The likelihood of this $C$-class classifier\n",
@@ -2033,8 +2333,10 @@
},
{
"cell_type": "markdown",
- "id": "c32b2b0e",
- "metadata": {},
+ "id": "942e0060",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = \\prod_{i=1}^n \\prod_{c=0}^{C-1} [P(y_{ic} = 1)]^{y_{ic}} .\n",
@@ -2043,16 +2345,20 @@
},
{
"cell_type": "markdown",
- "id": "15c86254",
- "metadata": {},
+ "id": "99eae6f4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Again we take the negative log-likelihood to define our cost function:"
]
},
{
"cell_type": "markdown",
- "id": "9da11ce8",
- "metadata": {},
+ "id": "872017e1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\theta}) = - \\log{P(\\mathcal{D} \\mid \\boldsymbol{\\theta})}.\n",
@@ -2061,8 +2367,10 @@
},
{
"cell_type": "markdown",
- "id": "47275fce",
- "metadata": {},
+ "id": "f55259c0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"See the logistic regression lectures for a full definition of the cost function.\n",
"\n",
@@ -2071,8 +2379,10 @@
},
{
"cell_type": "markdown",
- "id": "22860f6c",
- "metadata": {},
+ "id": "db983bc4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Example: binary classification problem\n",
"\n",
@@ -2081,8 +2391,10 @@
},
{
"cell_type": "markdown",
- "id": "2c90d0da",
- "metadata": {},
+ "id": "165881cf",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = - \\sum_{i=1}^n \\left(y_i\\log{p(y_i \\vert x_i,\\boldsymbol{\\beta})}+(1-y_i)\\log{1-p(y_i \\vert x_i,\\boldsymbol{\\beta})}\\right),\n",
@@ -2091,16 +2403,20 @@
},
{
"cell_type": "markdown",
- "id": "12ae44b0",
- "metadata": {},
+ "id": "50ecaa83",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where we had defined the logistic (sigmoid) function"
]
},
{
"cell_type": "markdown",
- "id": "7d921bf4",
- "metadata": {},
+ "id": "b16d30f7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(y_i =1\\vert x_i,\\boldsymbol{\\beta})=\\frac{\\exp{(\\beta_0+\\beta_1 x_i)}}{1+\\exp{(\\beta_0+\\beta_1 x_i)}},\n",
@@ -2109,16 +2425,20 @@
},
{
"cell_type": "markdown",
- "id": "5e48ee91",
- "metadata": {},
+ "id": "2d4c1829",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and"
]
},
{
"cell_type": "markdown",
- "id": "6add3f84",
- "metadata": {},
+ "id": "4d7a8c79",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(y_i =0\\vert x_i,\\boldsymbol{\\beta})=1-p(y_i =1\\vert x_i,\\boldsymbol{\\beta}).\n",
@@ -2127,8 +2447,10 @@
},
{
"cell_type": "markdown",
- "id": "22759cfd",
- "metadata": {},
+ "id": "7c4a0d89",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The parameters $\\boldsymbol{\\beta}$ were defined using a minimization method like gradient descent or Newton-Raphson's method. \n",
"\n",
@@ -2138,8 +2460,10 @@
},
{
"cell_type": "markdown",
- "id": "d18311a9",
- "metadata": {},
+ "id": "96bccb9e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"a_i^l = y_i = \\frac{\\exp{(z_i^l)}}{1+\\exp{(z_i^l)}},\n",
@@ -2148,16 +2472,20 @@
},
{
"cell_type": "markdown",
- "id": "9761e98e",
- "metadata": {},
+ "id": "01f9ae68",
+ "metadata": {
+ "editable": true
+ },
"source": [
"with"
]
},
{
"cell_type": "markdown",
- "id": "01a4059b",
- "metadata": {},
+ "id": "590fc8bd",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"z_i^l = \\sum_{j}w_{ij}^l a_j^{l-1}+b_i^l,\n",
@@ -2166,8 +2494,10 @@
},
{
"cell_type": "markdown",
- "id": "56bf706a",
- "metadata": {},
+ "id": "f3dc3a43",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where the superscript $l-1$ indicates that these are the outputs from layer $l-1$.\n",
"Our cost function at the final layer $l=L$ is now"
@@ -2175,8 +2505,10 @@
},
{
"cell_type": "markdown",
- "id": "60613b46",
- "metadata": {},
+ "id": "7dd1bbaa",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{W}) = - \\sum_{i=1}^n \\left(t_i\\log{a_i^L}+(1-t_i)\\log{(1-a_i^L)}\\right),\n",
@@ -2185,16 +2517,20 @@
},
{
"cell_type": "markdown",
- "id": "e97cc842",
- "metadata": {},
+ "id": "dc472c94",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where we have defined the targets $t_i$. The derivatives of the cost function with respect to the output $a_i^L$ are then easily calculated and we get"
]
},
{
"cell_type": "markdown",
- "id": "9c431661",
- "metadata": {},
+ "id": "372c5019",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{W})}{\\partial a_i^L} = \\frac{a_i^L-t_i}{a_i^L(1-a_i^L)}.\n",
@@ -2203,16 +2539,20 @@
},
{
"cell_type": "markdown",
- "id": "1903bf14",
- "metadata": {},
+ "id": "783cc2c1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"In case we use another activation function than the logistic one, we need to evaluate other derivatives."
]
},
{
"cell_type": "markdown",
- "id": "dab43f84",
- "metadata": {},
+ "id": "d37a48b8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The Softmax function\n",
"In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation $z_i^l$, that is we need"
@@ -2220,8 +2560,10 @@
},
{
"cell_type": "markdown",
- "id": "03d7a2cf",
- "metadata": {},
+ "id": "e6790133",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial f(z_i^l)}{\\partial w_{jk}^l} =\n",
@@ -2231,16 +2573,20 @@
},
{
"cell_type": "markdown",
- "id": "1813356c",
- "metadata": {},
+ "id": "97578009",
+ "metadata": {
+ "editable": true
+ },
"source": [
"For the Softmax function we have"
]
},
{
"cell_type": "markdown",
- "id": "debe1289",
- "metadata": {},
+ "id": "13822f62",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"f(z_i^l) = \\frac{\\exp{(z_i^l)}}{\\sum_{m=1}^K\\exp{(z_m^l)}}.\n",
@@ -2249,16 +2595,20 @@
},
{
"cell_type": "markdown",
- "id": "6fc65621",
- "metadata": {},
+ "id": "10bf9fb7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Its derivative with respect to $z_j^l$ gives"
]
},
{
"cell_type": "markdown",
- "id": "8c86ad26",
- "metadata": {},
+ "id": "cf59a594",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial f(z_i^l)}{\\partial z_j^l}= f(z_i^l)\\left(\\delta_{ij}-f(z_j^l)\\right),\n",
@@ -2267,16 +2617,20 @@
},
{
"cell_type": "markdown",
- "id": "86ecd366",
- "metadata": {},
+ "id": "80a4d3d7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which in case of the simply binary model reduces to having $i=j$."
]
},
{
"cell_type": "markdown",
- "id": "d68a68c2",
- "metadata": {},
+ "id": "91baac68",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Developing a code for doing neural networks with back propagation\n",
"\n",
@@ -2297,8 +2651,10 @@
},
{
"cell_type": "markdown",
- "id": "c9ec79c6",
- "metadata": {},
+ "id": "b4d8a71e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Collect and pre-process data\n",
"\n",
@@ -2345,8 +2701,11 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "b07f9bb8",
- "metadata": {},
+ "id": "ad941e6f",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# import necessary packages\n",
@@ -2395,8 +2754,10 @@
},
{
"cell_type": "markdown",
- "id": "6b776a41",
- "metadata": {},
+ "id": "c2f821dc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Train and test datasets\n",
"\n",
@@ -2414,8 +2775,11 @@
{
"cell_type": "code",
"execution_count": 6,
- "id": "3e399ed6",
- "metadata": {},
+ "id": "d368bcd4",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
@@ -2449,8 +2813,10 @@
},
{
"cell_type": "markdown",
- "id": "2c165e0e",
- "metadata": {},
+ "id": "c0e21c2d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Define model and architecture\n",
"\n",
@@ -2491,8 +2857,10 @@
},
{
"cell_type": "markdown",
- "id": "94075506",
- "metadata": {},
+ "id": "ba8404a4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Layers\n",
"\n",
@@ -2529,8 +2897,10 @@
},
{
"cell_type": "markdown",
- "id": "0bb70540",
- "metadata": {},
+ "id": "5ea9e948",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Weights and biases\n",
"\n",
@@ -2548,8 +2918,11 @@
{
"cell_type": "code",
"execution_count": 7,
- "id": "f3894282",
- "metadata": {},
+ "id": "890bc8b7",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# building our neural network\n",
@@ -2571,8 +2944,10 @@
},
{
"cell_type": "markdown",
- "id": "25c08547",
- "metadata": {},
+ "id": "1acc9cfe",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Feed-forward pass\n",
"\n",
@@ -2597,8 +2972,10 @@
},
{
"cell_type": "markdown",
- "id": "96a5c9b5",
- "metadata": {},
+ "id": "e13f966a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Matrix multiplications\n",
"\n",
@@ -2632,8 +3009,11 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "948f1d5c",
- "metadata": {},
+ "id": "8df5db3a",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# setup the feed-forward pass, subscript h = hidden layer\n",
@@ -2675,8 +3055,10 @@
},
{
"cell_type": "markdown",
- "id": "6a48b541",
- "metadata": {},
+ "id": "aa73fc1e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Choose cost function and optimizer\n",
"\n",
@@ -2704,8 +3086,10 @@
},
{
"cell_type": "markdown",
- "id": "92ae85de",
- "metadata": {},
+ "id": "74081e44",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Optimizing the cost function\n",
"\n",
@@ -2740,8 +3124,10 @@
},
{
"cell_type": "markdown",
- "id": "50af9f08",
- "metadata": {},
+ "id": "1e1c4fe3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Regularization\n",
"\n",
@@ -2772,8 +3158,10 @@
},
{
"cell_type": "markdown",
- "id": "947d899e",
- "metadata": {},
+ "id": "6f66a9ad",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Matrix multiplication\n",
"\n",
@@ -2811,8 +3199,11 @@
{
"cell_type": "code",
"execution_count": 9,
- "id": "9ac16dfb",
- "metadata": {},
+ "id": "b4ba72fe",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# to categorical turns our integer vector into a onehot representation\n",
@@ -2887,8 +3278,10 @@
},
{
"cell_type": "markdown",
- "id": "15fa07c3",
- "metadata": {},
+ "id": "1bf824b2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Improving performance\n",
"\n",
@@ -2906,8 +3299,10 @@
},
{
"cell_type": "markdown",
- "id": "e8e7304b",
- "metadata": {},
+ "id": "c98cec8d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Full object-oriented implementation\n",
"\n",
@@ -2918,8 +3313,11 @@
{
"cell_type": "code",
"execution_count": 10,
- "id": "da41ccbe",
- "metadata": {},
+ "id": "26d4691e",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"class NeuralNetwork:\n",
@@ -3025,8 +3423,10 @@
},
{
"cell_type": "markdown",
- "id": "6ed32b94",
- "metadata": {},
+ "id": "24f75613",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Evaluate model performance on test data\n",
"\n",
@@ -3042,8 +3442,11 @@
{
"cell_type": "code",
"execution_count": 11,
- "id": "a2566347",
- "metadata": {},
+ "id": "52a987ae",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"epochs = 100\n",
@@ -3066,8 +3469,10 @@
},
{
"cell_type": "markdown",
- "id": "779ebe17",
- "metadata": {},
+ "id": "1605eb14",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Adjust hyperparameters\n",
"\n",
@@ -3078,8 +3483,11 @@
{
"cell_type": "code",
"execution_count": 12,
- "id": "f55e622e",
- "metadata": {},
+ "id": "952131d2",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"eta_vals = np.logspace(-5, 1, 7)\n",
@@ -3106,8 +3514,10 @@
},
{
"cell_type": "markdown",
- "id": "4a2ea09c",
- "metadata": {},
+ "id": "182a8a49",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Visualization"
]
@@ -3115,8 +3525,11 @@
{
"cell_type": "code",
"execution_count": 13,
- "id": "d1dc32a5",
- "metadata": {},
+ "id": "d3d1b7d5",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# visual representation of grid search\n",
@@ -3156,8 +3569,10 @@
},
{
"cell_type": "markdown",
- "id": "fb02adc3",
- "metadata": {},
+ "id": "aa705900",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## scikit-learn implementation\n",
"\n",
@@ -3177,8 +3592,11 @@
{
"cell_type": "code",
"execution_count": 14,
- "id": "94531c6e",
- "metadata": {},
+ "id": "09c32314",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"from sklearn.neural_network import MLPClassifier\n",
@@ -3201,8 +3619,10 @@
},
{
"cell_type": "markdown",
- "id": "1373fc77",
- "metadata": {},
+ "id": "fde4721e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Visualization"
]
@@ -3210,8 +3630,11 @@
{
"cell_type": "code",
"execution_count": 15,
- "id": "2693a59e",
- "metadata": {},
+ "id": "e855729e",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# optional\n",
@@ -3252,8 +3675,10 @@
},
{
"cell_type": "markdown",
- "id": "a7e93079",
- "metadata": {},
+ "id": "765ac227",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Testing our code for the XOR, OR and AND gates\n",
"\n",
@@ -3277,8 +3702,10 @@
},
{
"cell_type": "markdown",
- "id": "5ed2f79a",
- "metadata": {},
+ "id": "02d395cd",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The AND and XOR Gates\n",
"\n",
@@ -3313,8 +3740,10 @@
},
{
"cell_type": "markdown",
- "id": "cbf09e4f",
- "metadata": {},
+ "id": "0fba1e7d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Representing the Data Sets\n",
"\n",
@@ -3323,8 +3752,10 @@
},
{
"cell_type": "markdown",
- "id": "fbb06edc",
- "metadata": {},
+ "id": "9a3c4e68",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -3336,16 +3767,20 @@
},
{
"cell_type": "markdown",
- "id": "9859c6a0",
- "metadata": {},
+ "id": "545fccde",
+ "metadata": {
+ "editable": true
+ },
"source": [
"while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate."
]
},
{
"cell_type": "markdown",
- "id": "d71c8649",
- "metadata": {},
+ "id": "678395d0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Setting up the Neural Network\n",
"\n",
@@ -3355,8 +3790,11 @@
{
"cell_type": "code",
"execution_count": 16,
- "id": "e8dc02bd",
- "metadata": {},
+ "id": "7c64daef",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"\"\"\"\n",
@@ -3428,16 +3866,20 @@
},
{
"cell_type": "markdown",
- "id": "87cfac77",
- "metadata": {},
+ "id": "c35ff8f2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above."
]
},
{
"cell_type": "markdown",
- "id": "fa50c112",
- "metadata": {},
+ "id": "3b5396c0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The Code using Scikit-Learn"
]
@@ -3445,8 +3887,11 @@
{
"cell_type": "code",
"execution_count": 17,
- "id": "6673546e",
- "metadata": {},
+ "id": "3d4e508b",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# import necessary packages\n",
@@ -3509,25 +3954,7 @@
]
}
],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.16"
- }
- },
+ "metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}
diff --git a/doc/src/week41/exercisesweek41.ipynb b/doc/src/week41/exercisesweek41.ipynb
deleted file mode 100644
index b828548e8..000000000
--- a/doc/src/week41/exercisesweek41.ipynb
+++ /dev/null
@@ -1,1332 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "e3354581",
- "metadata": {},
- "source": [
- "\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8d5fc499",
- "metadata": {},
- "source": [
- "# Exercises week 41\n",
- "**October 9-13, 2023**\n",
- "\n",
- "Date: **Deadline is Sunday October 15 at midnight**"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d3ef3596",
- "metadata": {},
- "source": [
- "# Overarching aims of the exercises this week\n",
- "\n",
- "The aim of the exercises this week is to get started with implementing\n",
- "gradient methods of relevance for project 2. This exercise will also\n",
- "be continued next week with the addition of automatic differentation.\n",
- "Everything you develop here will be used in project 2. \n",
- "\n",
- "In order to get started, we will now replace in our standard ordinary\n",
- "least squares (OLS) and Ridge regression codes (from project 1) the\n",
- "matrix inversion algorithm with our own gradient descent (GD) and SGD\n",
- "codes. You can use the Franke function or the terrain data from\n",
- "project 1. **However, we recommend using a simpler function like**\n",
- "$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials.\n",
- "You can obviously test your final codes against for example the Franke\n",
- "function. Automatic differentiation will be discussed next week.\n",
- "\n",
- "You should include in your analysis of the GD and SGD codes the following elements\n",
- "1. A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression of the gradients\n",
- "\n",
- "2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate), again using the analytical expression of the gradients.\n",
- "\n",
- "3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)\n",
- "\n",
- "4. Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.\n",
- "\n",
- "5. Add RMSprop and Adam to your library of methods for tuning the learning rate.\n",
- "\n",
- "The lecture notes from weeks 39 and 40 contain more information and code examples. Feel free to use these examples.\n",
- "\n",
- "In summary, you should \n",
- "perform an analysis of the results for OLS and Ridge regression as\n",
- "function of the chosen learning rates, the number of mini-batches and\n",
- "epochs as well as algorithm for scaling the learning rate. You can\n",
- "also compare your own results with those that can be obtained using\n",
- "for example **Scikit-Learn**'s various SGD options. Discuss your\n",
- "results. For Ridge regression you need now to study the results as functions of the hyper-parameter $\\lambda$ and \n",
- "the learning rate $\\eta$. Discuss your results.\n",
- "\n",
- "You will need your SGD code for the setup of the Neural Network and\n",
- "Logistic Regression codes. You will find the Python [Seaborn\n",
- "package](https://seaborn.pydata.org/generated/seaborn.heatmap.html)\n",
- "useful when plotting the results as function of the learning rate\n",
- "$\\eta$ and the hyper-parameter $\\lambda$ when you use Ridge\n",
- "regression.\n",
- "\n",
- "We recommend reading chapter 8 on optimization from the textbook of [Goodfellow, Bengio and Courville](https://www.deeplearningbook.org/). This chapter contains many useful insights and discussions on the optimization part of machine learning."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9940e21b",
- "metadata": {},
- "source": [
- "# Code examples from week 39 and 40"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6559d6d2",
- "metadata": {},
- "source": [
- "## Code with a Number of Minibatches which varies, analytical gradient\n",
- "\n",
- "In the code here we vary the number of mini-batches."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "021b66d2",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Own inversion\n",
- "[[4.28776768]\n",
- " [2.88377084]]\n",
- "Eigenvalues of Hessian Matrix:[0.30773954 4.59131906]\n",
- "theta from own gd\n",
- "[[4.28776768]\n",
- " [2.88377084]]\n",
- "theta from own sdg\n",
- "[[4.2916904 ]\n",
- " [2.90327755]]\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAApNElEQVR4nO3deZwU9Z3/8deHmwFELhWUAQ8ElXiSuJt4TGJcj3hkk99uYkiMosGYTWKMG2PCJmqUzea+Nll/ZD3DxGjMsTn3FxNnvEEHBMQDT1BR5FSQcxg+vz+qBpu2u6e6u7qqq3k/H49+0F1dXfWpoqc+/f1+6ltl7o6IiEhPeqUdgIiIZIMShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhDcvMrjKz2WnHESczazGzl9KOQ3ZPShiSKDNbamabzewNM1thZjeZ2eC04xKRnilhSBrOdPfBwJHAUcCX0g1n92RmfdKOQbJFCUNS4+4rgP9HkDgAMLMrzOxZM9tgZo+b2T/mvHeemd1nZt82s3Vm9ryZnZbz/v5mdnf42TuBkbnrM7OzzOwxM3vNzNrN7JCc95aa2RfMbJGZbTSz681sbzP7c7i8v5rZsELb0d1NZGaXmdlKM3vFzM7Peb/dzC7M346c125mnzKzp8N1XWNmB5rZA2a23sxuN7N+eev8spmtDuOemjO9f7h/XjCzV83sOjMbmBfnF81sBXCjmY00sz+E+2Stmd1rZjouSEH6YkhqzGw/4DTgmZzJzwLHA0OBq4HZZjY65/1jgSUEyeCbwPVmZuF7Pwfmhe9dA3w8Z10HA7cCnwNGAX8Cfp93IP4gcDJwMHAm8Gfgy+H8vYDPlticfcKY9wUuAH5cLMEUcQpwDPB3wOXALOCjwFhgMnBO3rpGhuv6ODDLzCaG7/1HGP+RwEHhPF/N++xwYBwwHbgMeCncxr3D7dX1gqQgJQxJw2/NbAPwIrASuLL7DXf/pbu/7O473P024GngHTmfXebuP3X3LuBmYDSwt5k1A28HvuLuW939HuD3OZ/7EPBHd7/T3TuBbwMDgXfmzPMjd3/V3ZcD9wJz3f0Rd98C/Iag+6yYTuBr7t7p7n8C3gAmlpg/3zfdfb27PwYsBv7i7s+5++sEiSt/3d3beTfwR+Cfw8Q5HbjU3de6+wbg34EP53xuB3Bl+NnNYdyjgXFh7Pe6LjAnRShhSBre7+5DgBZgEjldR2Z2rpktCLtIXiP4dZ3btbSi+4m7bwqfDgbGAOvcfWPOvMtyno/Jfe3uOwgS1r4587ya83xzgdelivNr3H17zutNPcyfr5x1F9rOMQSthCZgXs7++99werdVYQLs9i2CFt5fzOw5M7uijJhlN6OEIakJfx3fRPBrHzMbB/wU+DQwwt33JPi1bUUWkesVYJiZDcqZ1pzz/GWCbhjCdRlBd8/yyrcgso0EB/Ju+1S5vELb+TKwmiC5HObue4aPoeEJBt12aT24+wZ3v8zdDwDOAj5vZidVGZ80KCUMSdv3gZPN7AhgEMEBbRVAWDieHGUh7r4M6ACuNrN+ZnYcQR2i2+3A+8zsJDPrS9B3vxV4IK4NKWEB8AEzazKzgwhqHNXq3s7jgTOAX4atpp8C3zOzvQDMbF8zO6XYQszsDDM7KEygrwNdBN1WIm+hhCGpcvdVwC3AV939ceA7wIMEXTJvA+4vY3EfISiKryWoi9ySs54lBEXkHxH8Ej+T4PTebTFsRk++B2wj2KabgdYql7cCWEfQqmgFPunuT4bvfZGgi2mOma0H/krpWsqEcJ43CPb7T9y9rcr4pEGZ6lsiIhKFWhgiIhKJEoaIiESihCEiIpEoYYiISCSZuvjYyJEjffz48WmHISKSKfPmzVvt7qN6nrO0TCWM8ePH09HRkXYYIiKZYmbLep6rZ+qSEhGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUiUMEREJJKaJwwzu8HMVprZ4gLvXWZmbmYjax2HiIhUJ4kWxk3AqfkTzWws8A/ACwnEICIiVap5wnD3ewjusZzve8DlgO4RKyKSAanUMMzsbGC5uy+MMO90M+sws45Vq1YlEJ2IiBSSeMIwsybgy8BXo8zv7rPcfYq7Txk1qurLuYuISIXSaGEcCOwPLDSzpcB+wHwz2yeFWEREJKLEb6Dk7o8Ce3W/DpPGFHdfnXQsIiISXRKn1d4KPAhMNLOXzOyCWq9TRETiV/MWhruf08P742sdg4iIVE8jvUVEJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUiUMEREJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIap4wzOwGM1tpZotzpn3LzJ40s0Vm9hsz27PWcYiISHWSaGHcBJyaN+1OYLK7Hw48BXwpgThERKQKNU8Y7n4PsDZv2l/cfXv4cg6wX63jEBGR6tRDDWMa8Odib5rZdDPrMLOOVatWJRiWiIjkSjVhmNkMYDvQWmwed5/l7lPcfcqoUaOSC05ERHbRJ60Vm9l5wBnASe7uacUhIiLRpJIwzOxU4HLgRHfflEYMIiJSniROq70VeBCYaGYvmdkFwH8CQ4A7zWyBmV1X6zhERKQ6NW9huPs5BSZfX+v1iohIvOrhLCkREckAJQwREYlECUNERCJRwhARkUiUMEREJBIlDBGRetLaCuPHQ69ewb+tRS+EkbjURnqLiEie1laYPh02heOZly0LXgNMnZpeXCG1MERE6sWMGW8mi26bNgXT64AShohIvVi2rLzpCVPCEBGplXLrEb17lzc9YaphiIjUQmsrnH8+dHYGr5ctC15D8XpEV1d50xOmFoaISC1ccsmbyaJbZ2cwvZhx48qbnjAlDBGRuOR2Qa1ZU3ieYtMBZs6EpqZdpzU1BdPrgBKGiEgcuk+JXbYMKr0n3NSpMGtW0KIwC/6dNasuTqkFJQwRkXgUOiW2kBEjylvu/ffXzUA+Fb1FROLwwgs9z9OvH/zgB8XfLzRw77/+6833Ux7IpxaGiAhUf0mO5ubC03v3DrqXRoyAIUPgYx8rvvworZQUB/IpYYhIbdXxtZF2yq8/dP+SLyfWYgXrm2+Gn/0MNm8OCt6llh+llVLOfHFz98w8jjnmGBeRDJk9272pyT04TAaPpqZgej0ZN27XGLsf48aVt5zZs4PPmAX/dm9n1OUXmy9KXMXW7e5Ah8dwDDavtJqfgilTpnhHR0faYYhIVOPHF76sxbhxsHRp0tEU16tX4TObzGDHjuSW39oK06bBtm3Fl9W3L9x44641jPzaBwStm/AMKzOb5+5Tqt6MahfQEzO7wcxWmtninGnDzexOM3s6/HdYreMQkRQU6zpJq0ulmGL1h2LTa7X8qVODOkcpe+zx1oJ3QhctTKKGcRNwat60K4C/ufsE4G/haxFpNLU+EMel1gPmyln+2rWll1Xo/SIJ2Je9wAf3nRMxyJ7VPGG4+z1A/haeDdwcPr8ZeH+t4xCRFNT5yOWdaj1grpzl95RMC7zvY8cWnHUZzcx7db9KIi4sjkJITw9gPLA45/VrOc8t93WBz04HOoCO5ubmEtUmEalLJYqxUkChEwXyThjo6uzyRXcs8R9+sN0/MOZB/wTX+Rvs+pnO3v195Vd+6O4ZK3qb2XjgD+4+OXz9mrvvmfP+OnfvsY6horeI7BZaW4P6w7Jl0Ls33tVF58gx3DX+fGYtP4O7VxzMWh8OwPg+L9Iyfinn73cn73ziBvqsfDlohcycubMFk5midxGvmtlogPDflSnFISKSniJjVHZ86BwWffsv/PAD7Xxgr/sYaWvov3o5p3VcyyOr9uWsAx/jpgvv4/l7X+L5zrHc+PTxnND2NfqseCkY8wGlBwhWKo5mSk8P3tol9S3givD5FcA3oyxH4zBEpKQsdX8V6Hra1ru/f3foVT7c1uycvH+fZX7+hHv85k/c60vve7HsZXpTk4+E5zyOY3kcCym5ArgVeAXoBF4CLgBGEJwd9TTwV2B4lGUpYYhIUfU6SLBAEuvq7PKtI8cUrFO8yL7RE0S+IgP/DoetnpUaRlxUwxCRoupxkGCBAXVbGMBn+QHX8Ul6EfNgwSIDBKcAHe5W2UJzFl/tAkRE6kIdDRLcsX0HC29fwvpPXPaWAXUD2MK/9/kqmwbvVfjD1YxRKfLZTigxdDw6JQwRaQwpDhLsThA/+MDd/OOYOYzs9zpHfmgigzcXPp9nZNdKBl/3nehjVKJewLHIuJeXYXnZG1VIHP1aST1UwxCpU7UuNkdZfoI1jK7OLn/kF0/6997f7mfvM8eH2dqdqzygz1KfNuEev+Wie71z730L1hR2XjywFttVYJnENA4j9SRQzkMJQ6QO1eJAnXvQGzHCvV+/aMuvUeKKmiCWPfDSW+Opdt/EcCXduBKGit4iUp24i82FrrxaSA2L2Tu272DRr56m/RcraJ8zgHtePZh14djiA/sso+WAF2h5Ty9OPG9/xh47pvTCugfhvfDCWwbURRLDlXTjGrinhCFSjWoPBo0g7kuDF0tAcS2/gFgTRNxiSMhxJQzd01ukUoXuv5zi/ZZT09xc+IBWabE56llNVRSzu7Z1BQnithW0zx3IPa9O5DWfCEzkwD7L+MCExTkJYhwwruJ1VW3mzML3ukjjAo5x9Gsl9VANQ+pKXHdpy7qo/fRR6wtR7jrXr19ZdYDtW7f7/J8/4d89u83P2meO72nrdi7qoL7P+4UT7/bZF9/nLz70cqV7obaqrM0QUw1DLQyRStXRef+p6m5NFeuaa22Fiy6CjRvf/Eyp1lihX9T5euhKL9yCmARM4qC+S/k/By+i5T29OfH8A9jv7eMJrl5Ux6ZOrYtWq2oYIpWqx5HFtVBNnaa1Fc4/Hzo7C79fbF/lrrNXL+jqKvnZrm1dLPzlU7Tf/mqYICbxOkMBOKjv0rAG0Z0gRkeLvYHEVcNIvZupnIe6pKSu1Ou1i+JU7TZG6V7qqYvFrODndmD+nbPa/My95/hQXtuli2nWPv/mGwbv7TuycBHCBKBxGCJ1IEtXR61EtXWaIgf7tzxKJaEiMTzPOAf3CX2f809MuttbP3Wfv/Twy9lM5DX+HsWVMNQlJSLFVXvKbNRTZKFoF9OO9rv51Ppv0MTmnbNutf50nPRFxn/9k+w7Ja+LKWtdhYXGnTQ1xXqLWI3DEJHaq/bg21MNI4djfPeMu2h/qIl7V07cWYOY0Pd5Lh95A+ds+ClNG1diPdVR4h4XUmsJJLis33FPRNIQ9SJ23YpczC7yGICpU+HGG2HQoB5nXUYz//qHFp5aN4oPTVrIzz/9AMvnreCpbftz4cvXMGjDCizK3eRSvAhhRbJ0tl0c/VpJPVTDkMxJqsZRy4vzVbMNs2e7jxjhO95SsN719Rbr7/effKUvn/dK6WVFHe+RpRpGAuN5UNFbpM4ldeCKup4EBxp2bu70Zy/6hm/r3b/wOsG7rFd5ZzGVE38lSS6tExgS+J4oYYjUu6QO0FHXU+yMJbOqQ+jc3OkP3/yYf+t9bf6+veb6Hrzmz1MkrkrXW8P4Cx60IbhSbhKJQ2dJxU9Fb8mUpIqvUdcTY3F1+5btLLj9KdpvX0n7w03svXIRX+FamnmBVxjN/46ZxrSXr6XkPUHLXW8ti8OlzuaK+YylNKjoLVLvkiq+Rl1PFQXs7Vu203HL43z7jHbO2PshRgzcyNs/fihf+GMLE9fcz0/s04xnGb1w9uVlLlj9zdLJopKL51VbgC9V8C9VYN60KRh1Xq1yTzioR3E0Uyp9AJcCjwGLgVuBAaXmV5eUVCxqUTjOboF6q2F0zxthGzs3d/pDNz3m3zy9zU8f9ZAP4fWdi57Y71m/6JC7/dbP3O8vP7Ii2mjuuLp5Kv0/6mkf9bQN1XZ7pVyIJ+s1DGBf4HlgYPj6duC8Up9RwpCKRPljrdUfdD2dJVVCqQQxqd8z/slD7/ZffDZMEPmijubufqShpzpPsRpGXHWnlK9sHFfCSK2GYWb7AnOAI4D1wG+BH7r7X4p9RjUMqUiUvu+sjQ6u0vYt23nktu4axCDuXTWRDewBwKR+z9Jy0HJa3tuHEy84iH0O36v0wioczZ2oKHWe1la45BJYs2bXeeKoYaQ8mLAhLj4IXAK8AawCWovMMx3oADqam5vjS7my+4hydk0tz8CpA52bO33uDYv9G6e1+WklWhCvLHy1/IUX+nXer597377xt9gqVetTcuNcfw3QAF1Sw4C7gFFAX4IWxkdLfSZzXVKNfmE692xsY5Q/1ga7GVJNE0Qhhb4H9fTdSHswn2oYVSeMfwKuz3l9LvCTUp/JVMJI+wuahKxsY5o1jITkJ4jBrN+5GYf0e8YvPuxuv+1zMSaILEo7gaW4/kZIGMcSnCHVBBhwM/CZUp/JVMJosF+sBWVpG9M4S6qGoiaIFY+uTDtUqQNxJYxUB+6Z2dXAh4DtwCPAhe6+tdj8mSp6Z+2KmZXYHbaxTnRu6mT+rUtov2M17R2DuG/1JN5gCACH9HuWlgnLaTm5DydeMIG9J49KOVqpN3EVvVO9p7e7XwlcmWYMNdPcXPjMkXq9YmYldodtTEnhBDEZgEP7P8O5k+fTcnI/Tph2EHtPPhA4MN2AZbfQ40hvM7vTzI5IIpiGUu2o1CxotG1McSRu56ZO5l6/mG+c1s5pox5m+KAt/N2Fk7nif1t4YcMwzp08n9svfZAVj67isS0H8eNHT+Sfvvv3lbcmGmHUsSSvpz4r4GigDbgRGB1HP1ilj0zVMNwz1SdesUbZxoSL3ts2bvMHf/qof/2UNj9lxMM+iA07V3to/6f9U5Pb/fZLH3hrDaKakc7dnxsxIjjtNc5tbZTvQYMi6aI38EFgIUEX0sA4Vl7uI3MJoxD9YdWnGhfwK04Quaq5n0WpUczVbmvGzzDbHcSVMCIVvc3MgMOA44BrgS3Al9z9Z/G3eYrLVNG7kATu3SsVirmA37mpk3k/765BDOa+NZPYyGAADuv/NC0Hv7yzBrHXYRG7lSodjR51JHalJyvsZqPksyixe3qb2f3A/gSnwM4B5gJPEozS7u/u06sNIqrMJwz9YdWvKv9vapIg8lWa1Ip9Ll+l30OdLVf3Ers0CEHLwoq890QczZyoj8x3STX45SdiUasuu56WW2a3yraN2/yB/7vI//0f2vwf8rqYDuv/lP/L29r9l59/wF9dHOM4iEq7zaJcTbaaLqQsjcfZTVEPA/eAA+IIIuoj8wkjjT+sLNVMannF2Kj3gs4tDI8YsXO/dd5wc9EEMbn/U/7pt7X7Hf/6gK98fFV1scaxHVE+17fvLttXdcFbNYy6VhcJI+lH5hNG0n9YWftDrlVCLXe5s2f7joG77rc3GOjnMDvZBFEktqrPkqrFD4cs/TBpBGXubyWMrEryDytrXQW16rKLsNytG7b6/dct8pknt/mKXvsUnP+NPfZOPkGI5Kvgh2BcCUP39G5kWStG1uqkgCLL3TJ8NN895ue0zxvC/WsnsYlBAHTRi15kaL/J7qWCvxPd01t6ltQ9pYspdzRxrUaOz5yJD9x1uRsZyLS132LGnS2s2DSECw7v4FdfmMOqJ9fQa1wM+00jqaVWit1/vNR9yeMSRzMlqUdDdEklKc0aRrnr7u6qA/fevd/sOqsw1twuppOHd/h5XO/PM867MF/OGL9p7Az/1Rce9FVPrq4+9rg/L1JKBV3NqIYhkaRVjCz3DmdVHmC3btjq9/1koV/73jZ/7/AOH8jGnYt624Al/pnD24sniEKq2W9Zqx1JtqiGEY1qGBlSTv2kgj7ZbW9s4+GfPUn7r9fSPn8I9689hM0E3U6HD1hCy8QVtJzSn+OnTWDkxBHR425thRkzguZ9c3PQHVbuKPys1Y4ke8r8niY20rueKGFkSDlJIMIBtmYJIldcl27RiH6pM4mN9K6nR2a7pLJ4jno5MRe7n3PUZnORLpzNw0b7NSe1+UnD5u3SxXT4gCf9s0e0+68vf9BXP7Umvm2OqytJNQypM6iGkRFZPHiUE3OpeaMmnR4GytUsQeSLcxxIFn8kSMOKK2GoS6rWstg9UU7MFW7f1vVbeXj2krCLaQ/2W7eQq7iaZl5gBaP5a/P57PGRMzl+2gRGTBhexcaUIYv/VyIRNMQtWncLaZ4zXalyYo4479b1W3nolidp/8067n5kDx5YN4nNHA7AEQOWMPnIA1hw6m0MmTaBMROGc2418Vdq5szCNYys3kFQJGYauFdr1Q6ei2MAWLnLKCfmIvP6fmO59z8Xcs1J7Zw0fD57Dt3BCZ85givvOoHVmwcx/ciH+c0Vc1n91FoWbJ7I9x85kfd//djkWhOFTJ0aFLjHjQsK7uPGlVfwTmuwngYJSlLi6Neq9AHsCdxBcH+NJ4C/LzX/blfDiOMqo5Wsv8oaxmYG+LncEHT/0+VHDnzCP3dUu//2S3N8zTNre97uUnHVa12gmivJVrNNWayRSeJohKI3cDNwYfi8H7BnqfkzmTDcKz8oxHEfg0rP/Okh5i2vb/F7frTAv/aeNr+26Vpfyljvwvx5mv2Kvt+MniDKKIzX9YGxkv0cxzZpkKBEEFfCSK3obWZDgQUE99SIFEQmi97ViONOaTENItu6fitzbw5qEO2PDOXB1yaxhYEYOzhi4FO0THqVllMHcPwFBzP8wGHRFlrOuId6L0hXsp/j2CYNEpQIMj9wz8yOBGYBjwNHAPOAS9x9Y95804HpAM3Nzccsi3Jv4kYRx72YqziLKfYEUU1spZKnWeWjsuNSyX6OY5vqPZFKXcj8wD1gCrAdODZ8/QPgmlKfyWyXVCVmzw5qFT11ScXU7bHl9S1+9w8X+NXvbvN37znfB7BpZw3iqIGP+6VHt/n/fHmOr31uXXzbWM64h1rfZrRalXQvxbFN9d5VJ3WBrNcwgH2ApTmvjwf+WOozu03CKHQQKPbo16+iwurmdZu9/fuP+FUtCSaIfMUOmCNGFN6GKPskzb77cmtVcW1TPZ8MIHUh8wkj2AbuBSaGz68CvlVq/t0mYURtWRQ7uBaQmyBa9pzv/dns5zDbn6fZuzBf03dv7zjrqtomiHyzZwcJL3+b+vYtfkZW94Gx2P6o9u58SWvEbZK60ygJ40igA1gE/BYYVmr+1BJGXL/goixn9uzoyaLEwWTrrBt987DRvgPzV2y0n8uNO1sQRw983H82foZ39h2w67LS6Moolhx7+lXdiGcHVbNNamVICQ2RMMp9pJIw4uojjrqcKP3aBQ4muS2IawZe62+w67q2Wn/veP/Vb7Yg6uWAW+n1mxqx776asRyNti8kVkoYSSl2YO3du7xfc1EP0KW6JvK6b7b3HeB3TJrhJw59xPuzeWcLYrmN7nldcV5orxr6Vb2rSrapXpK/1C0ljKSUOoDnHmQvvriy5eQfoIv88XcNG+6PTZ3p6/rv7V2YL2Wsn8Ns78V2P6bpMf/8MW3+u3+b6+uWvhZtXfVykNGv4+rVS/KXuqWEkZSoXURm8Yy4LnqpjRsdfGeCuOyYNv/9V8IEUcm66ulA3YgthSTVS/KXuqWEkZRyTnGtYjzEpjWb/K7vzPevntDmXxs4c+elNpbS7F/u+43SCaLMde0ynw7U2VdPyV/qkhJGIbU6AOYut3fv4gkjSqE2XE7XfmN98Ueu9a+e0OYnDH3E+7Elegui3JiVDBqf/r+lhLgSRuPcDyP/ukTLlgWvofrLRUyd+uYyWlvhYx8LUkS+Epcs37x2M3NePZT2cTfR/tqezHnpELb9vD+96OLopiV8dsqDtLxvEMdNO5ihzYcCh1YXcyXKvLG81JHc76hIrcSRdZJ6lGxhJNmPe/HFby005nUB5HYx5bcgpjQ95v86pc3/cOVD/tqyClsQpcyeHQx+y42v2GC43M+oW0OkIZH1q9VWouTVapO+amfer/FtX76S+zYcSfv/vE77wmHMXT+JbbzZgmg5dGVOC2Jo/PHkGjkS1qx56/QRI2D16sKf0UXsRBpW5q9WW4mSCaPYAW/ECBg8uOduljK7Yzav3cyDNz5ZMEEcM+hJTjxkVXIJIp9Z8feK/X/rMtkiDSvzV6ut5FGyS6rY3enyr1VU7GyhHrpjNq3Z5H/79nz/yvFtfvweC3bpYnr7oMX+hbe3+R+vqlEXU7lKnclVTC279FSQFUkVOkuqgPwDU9TrFBU5WG7acx//t+NKJ4jXX3y9dExJK3UtqlIXKqxVDUO1EZHUxZUwGqdLqpCo3SxF5tuB0ZdOjhn0JC2Hhl1MF0xkj/32qCD6hJS66dLs2aXPpKnFWVKqjYikTjWMKEocrDZ1PB7UIH63nn+5/yPs46+8ZbbNQ/ehc/GS+k4Q+UrdxS2N/2vVRkRSF1fC6BVHMHVr5szgHtE5tvXqz9WvXsSeo/rw3suP5uv3HceP+17Ctl79d/1sUxMDf/ztbCULKD4WZNy4ZOPoViyeEmNWRKQ+NWzC2LR6E39dfgi373spK2w0OzCWMo5pO37Kn3qfyaXveIA/Xf0wa1/cxDVbv0i/W64PDqpmwb+zZmVzIFSBJElTUzBd8YhINeIohCT1KFX03rhqo9/5jXk+411t/q4hC70vWx3ce9Ppxw561C9/R5v/6eoaFKnr8Qygeoup3uIR2c2wuxe9N67cyIM3LaH9d+tpXzSchzZMopN+9GY7UwY9Scvk1bScMZh3TZvIkDFDahNQ/uVIIPj1nNXWSVboEiYiZdkti94H73eo//P4n6SXIPLpDKDkKUmLlG23TBhmU7w3c9JLEPl0BlDylKRFyhZXwsjU1WonjNnEvIc3M2TM5LRDCTQ3Fz546Qyg2nnhhfKmi0hsUj9Lysx6m9kjZvaHnubdY3RTbVoTra3BL9devYJ/W1ujfU5nACVPp+mKpCb1hAFcAjyR2tq7+8SXLQu6l7rvoxElaUydGvSdN8LpuFmhJC2SmlQThpntB7wP+O/UgpgxY9cCKgSvZ8yI9vmpU4O+8x07gn/rLVlU2nqqV0rSIqlJu4XxfeByoGiF2Mymm1mHmXWsWrUq/giy1Cde7sG/mtZTPav3JC3SoFJLGGZ2BrDS3eeVms/dZ7n7FHefMmrUqPgDyUqfeCUH/2pbTyIiOdJsYbwLOMvMlgK/AN5jZrMTjyIrfeKVHPyz1HoSkbqXWsJw9y+5+37uPh74MHCXu3808UCy0ideycE/K60nEcmEtGsY9SELfeKVHPyz0noSkUyoi4Th7u3ufkbacdS1Sg7+WWk9iUgmZGqk926t+yBf7kX3pk5VghCRWNRFCyM2jTbmIF8Wus5EpGE1Tgsj/yqm3aedgg6sIiIxaJwWhsYciIjUVOMkDI05EBGpqcZJGBpzICJSU42TMDTmQESkphonYWjMgYhITTXOWVKgMQciIjWU/RZGo4+9EBGpE9luYWjshYhIYrLdwtDYCxGRxGQ7YWjshYhIYrKdMDT2QkQkMdlOGBp7ISKSmGwnDI29EBFJTLbPkgKNvRARSUi2WxgiIpIYJQwREYlECUNERCJJLWGY2VgzazOzx83sMTO7JK1YRESkZ2kWvbcDl7n7fDMbAswzszvd/fEUYxIRkSJSa2G4+yvuPj98vgF4Atg3rXhERKS0uqhhmNl44ChgboH3pptZh5l1rFq1KvHYREQkkHrCMLPBwK+Az7n7+vz33X2Wu09x9ymjRo1KPkAREQFSThhm1pcgWbS6+6/TjEVEREpL8ywpA64HnnD376YVh4iIRJNmC+NdwMeA95jZgvBxeorxiIhICamdVuvu9wGW1vpFRKQ8qRe9RUQkG5QwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUiUMEREJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJJNWGY2almtsTMnjGzK9KMRURESkstYZhZb+DHwGnAocA5ZnZoWvGIiEhpabYw3gE84+7Pufs24BfA2SnGIyIiJfRJcd37Ai/mvH4JODZ/JjObDkwPX241s8UJxFatkcDqtIOIQHHGJwsxguKMW1binBjHQtJMGJG4+yxgFoCZdbj7lJRD6pHijFcW4sxCjKA445alOONYTppdUsuBsTmv9wuniYhIHUozYTwMTDCz/c2sH/Bh4HcpxiMiIiWk1iXl7tvN7NPA/wN6Aze4+2M9fGxW7SOLheKMVxbizEKMoDjjtlvFae4ex3JERKTBaaS3iIhEooQhIiKR1E3C6OkyIWbW38xuC9+fa2bjc977Ujh9iZmdkmKMnzezx81skZn9zczG5bzXZWYLwkdNi/sR4jzPzFblxHNhznsfN7Onw8fHU47zezkxPmVmr+W8l8j+NLMbzGxlsfE/FvhhuA2LzOzonPeS3Jc9xTk1jO9RM3vAzI7IeW9pOH1BXKdfVhFni5m9nvN/+9Wc9xK7lFCEOL+QE+Pi8Ps4PHwvkf1pZmPNrC085jxmZpcUmCfe76e7p/4gKHo/CxwA9AMWAofmzfMp4Lrw+YeB28Lnh4bz9wf2D5fTO6UY3w00hc8v7o4xfP1GHe3L84D/LPDZ4cBz4b/DwufD0oozb/7PEJwYkfT+PAE4Glhc5P3TgT8DBvwdMDfpfRkxznd2r5/gcjxzc95bCoysk/3ZAvyh2u9LrePMm/dM4K6k9ycwGjg6fD4EeKrA33qs3896aWFEuUzI2cDN4fM7gJPMzMLpv3D3re7+PPBMuLzEY3T3NnffFL6cQzC2JGnVXHLlFOBOd1/r7uuAO4FT6yTOc4BbaxRLUe5+D7C2xCxnA7d4YA6wp5mNJtl92WOc7v5AGAek992Msj+LSfRSQmXGmdZ38xV3nx8+3wA8QXAFjVyxfj/rJWEUukxI/obvnMfdtwOvAyMifjapGHNdQJDZuw0wsw4zm2Nm769BfN2ixvnBsIl6h5l1D6BMal+Wta6wa29/4K6cyUntz54U244k92W58r+bDvzFzOZZcCmetP29mS00sz+b2WHhtLrcn2bWRHCg/VXO5MT3pwVd9EcBc/PeivX7WfeXBskiM/soMAU4MWfyOHdfbmYHAHeZ2aPu/mw6EfJ74FZ332pmFxG03N6TUixRfBi4w927cqbV0/7MDDN7N0HCOC5n8nHhvtwLuNPMngx/YadhPsH/7RtmdjrwW2BCSrFEcSZwv7vntkYS3Z9mNpggYX3O3dfXaj1QPy2MKJcJ2TmPmfUBhgJrIn42qRgxs/cCM4Cz3H1r93R3Xx7++xzQTvBroBZ6jNPd1+TE9t/AMVE/m2ScOT5MXpM/wf3Zk2LbUXeXvjGzwwn+v8929zXd03P25UrgN9SmSzcSd1/v7m+Ez/8E9DWzkdTh/gyV+m7WfH+aWV+CZNHq7r8uMEu8389aF2YiFm/6EBRd9ufNgtZhefP8C7sWvW8Pnx/GrkXv56hN0TtKjEcRFOYm5E0fBvQPn48EnqZGBbuIcY7Oef6PwBx/sxD2fBjvsPD58LTiDOebRFBEtDT2Z7iO8RQv0r6PXYuKDyW9LyPG2UxQ33tn3vRBwJCc5w8Ap6YY5z7d/9cEB9oXwn0b6fuSVJzh+0MJ6hyD0tif4X65Bfh+iXli/X7WbGdXsPGnE1T5nwVmhNO+RvBLHWAA8MvwS/8QcEDOZ2eEn1sCnJZijH8FXgUWhI/fhdPfCTwafskfBS5IeV9+HXgsjKcNmJTz2WnhPn4GOD/NOMPXVwH/kfe5xPYnwa/HV4BOgn7eC4BPAp8M3zeCG4E9G8YyJaV92VOc/w2sy/ludoTTDwj348LwOzEj5Tg/nfPdnENOgiv0fUkrznCe8whOuMn9XGL7k6Bb0YFFOf+vp9fy+6lLg4iISCT1UsMQEZE6p4QhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCJVCO9HcHL4/Foz+1HaMYnUii4+KFKdK4GvhReaOwo4K+V4RGpGI71FqmRmdwODgRYP7ksg0pDUJSVSBTN7G8Gdz7YpWUijU8IQqVB457JWgruavWFmNbujnkg9UMIQqUB4p7VfA5e5+xPANQT1DJGGpRqGiIhEohaGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCT/H6Bcv3g0WmVZAAAAAElFTkSuQmCC\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- }
- ],
- "source": [
- "%matplotlib inline\n",
- "\n",
- "# Importing various packages\n",
- "from math import exp, sqrt\n",
- "from random import random, seed\n",
- "import numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "\n",
- "n = 100\n",
- "x = 2*np.random.rand(n,1)\n",
- "y = 4+3*x+np.random.randn(n,1)\n",
- "\n",
- "X = np.c_[np.ones((n,1)), x]\n",
- "XT_X = X.T @ X\n",
- "theta_linreg = np.linalg.inv(X.T @ X) @ (X.T @ y)\n",
- "print(\"Own inversion\")\n",
- "print(theta_linreg)\n",
- "# Hessian matrix\n",
- "H = (2.0/n)* XT_X\n",
- "EigValues, EigVectors = np.linalg.eig(H)\n",
- "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
- "\n",
- "theta = np.random.randn(2,1)\n",
- "eta = 1.0/np.max(EigValues)\n",
- "Niterations = 1000\n",
- "\n",
- "# need convergence tests\n",
- "# stop if reached max number of iterations or | new value - old value | =< given tolerance\n",
- "for iter in range(Niterations):\n",
- " gradients = 2.0/n*X.T @ ((X @ theta)-y)\n",
- " theta -= eta*gradients\n",
- "print(\"theta from own gd\")\n",
- "print(theta)\n",
- "\n",
- "xnew = np.array([[0],[2]])\n",
- "Xnew = np.c_[np.ones((2,1)), xnew]\n",
- "ypredict = Xnew.dot(theta)\n",
- "ypredict2 = Xnew.dot(theta_linreg)\n",
- "\n",
- "n_epochs = 50\n",
- "M = 5 #size of each minibatch\n",
- "m = int(n/M) #number of minibatches\n",
- "t0, t1 = 5, 50\n",
- "\n",
- "def learning_schedule(t):\n",
- " return t0/(t+t1)\n",
- "\n",
- "theta = np.random.randn(2,1)\n",
- "\n",
- "for epoch in range(n_epochs):\n",
- "# Can you figure out a better way of setting up the contributions to each batch?\n",
- " for i in range(m):\n",
- " random_index = M*np.random.randint(m)\n",
- " xi = X[random_index:random_index+M]\n",
- " yi = y[random_index:random_index+M]\n",
- " gradients = (2.0/M)* xi.T @ ((xi @ theta)-yi)\n",
- " eta = learning_schedule(epoch*m+i)\n",
- " theta = theta - eta*gradients\n",
- "print(\"theta from own sdg\")\n",
- "print(theta)\n",
- "\n",
- "plt.plot(xnew, ypredict, \"r-\")\n",
- "plt.plot(xnew, ypredict2, \"b-\")\n",
- "plt.plot(x, y ,'ro')\n",
- "plt.axis([0,2.0,0, 15.0])\n",
- "plt.xlabel(r'$x$')\n",
- "plt.ylabel(r'$y$')\n",
- "plt.title(r'Random numbers ')\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d8e7110b",
- "metadata": {},
- "source": [
- "In the above code, we have use replacement in setting up the\n",
- "mini-batches. The discussion\n",
- "[here](https://sebastianraschka.com/faq/docs/sgd-methods.html) may be\n",
- "useful."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "74a7859a",
- "metadata": {},
- "source": [
- "## Momentum based GD\n",
- "\n",
- "The stochastic gradient descent (SGD) is almost always used with a\n",
- "*momentum* or inertia term that serves as a memory of the direction we\n",
- "are moving in parameter space. This is typically implemented as\n",
- "follows"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "619df8e8",
- "metadata": {},
- "source": [
- "$$\n",
- "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c759cfc4",
- "metadata": {},
- "source": [
- "\n",
- "\n",
- "\n",
- "$$\n",
- "\\begin{equation} \n",
- "\\boldsymbol{\\theta}_{t+1}= \\boldsymbol{\\theta}_t -\\mathbf{v}_{t},\n",
- "\\label{_auto1} \\tag{1}\n",
- "\\end{equation}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0a4571f9",
- "metadata": {},
- "source": [
- "where we have introduced a momentum parameter $\\gamma$, with\n",
- "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n",
- "indicate the gradient is to be taken over a different mini-batch at\n",
- "each step. We call this algorithm gradient descent with momentum\n",
- "(GDM). From these equations, it is clear that $\\mathbf{v}_t$ is a\n",
- "running average of recently encountered gradients and\n",
- "$(1-\\gamma)^{-1}$ sets the characteristic time scale for the memory\n",
- "used in the averaging procedure. Consistent with this, when\n",
- "$\\gamma=0$, this just reduces down to ordinary SGD as discussed\n",
- "earlier. An equivalent way of writing the updates is"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9bebfdd9",
- "metadata": {},
- "source": [
- "$$\n",
- "\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c9e635e8",
- "metadata": {},
- "source": [
- "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "eb544f8b",
- "metadata": {},
- "source": [
- "## Algorithms and codes for Adagrad, RMSprop and Adam\n",
- "\n",
- "The algorithms we have implemented are well described in the text by [Goodfellow, Bengio and Courville, chapter 8](https://www.deeplearningbook.org/contents/optimization.html).\n",
- "\n",
- "The codes which implement these algorithms are discussed after our presentation of automatic differentiation."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d088ad85",
- "metadata": {},
- "source": [
- "## Practical tips\n",
- "\n",
- "* **Randomize the data when making mini-batches**. It is always important to randomly shuffle the data when forming mini-batches. Otherwise, the gradient descent method can fit spurious correlations resulting from the order in which data is presented.\n",
- "\n",
- "* **Transform your inputs**. Learning becomes difficult when our landscape has a mixture of steep and flat directions. One simple trick for minimizing these situations is to standardize the data by subtracting the mean and normalizing the variance of input variables. Whenever possible, also decorrelate the inputs. To understand why this is helpful, consider the case of linear regression. It is easy to show that for the squared error cost function, the Hessian of the cost function is just the correlation matrix between the inputs. Thus, by standardizing the inputs, we are ensuring that the landscape looks homogeneous in all directions in parameter space. Since most deep networks can be viewed as linear transformations followed by a non-linearity at each layer, we expect this intuition to hold beyond the linear case.\n",
- "\n",
- "* **Monitor the out-of-sample performance.** Always monitor the performance of your model on a validation set (a small portion of the training data that is held out of the training process to serve as a proxy for the test set. If the validation error starts increasing, then the model is beginning to overfit. Terminate the learning process. This *early stopping* significantly improves performance in many settings.\n",
- "\n",
- "* **Adaptive optimization methods don't always have good generalization.** Recent studies have shown that adaptive methods such as ADAM, RMSPorp, and AdaGrad tend to have poor generalization compared to SGD or SGD with momentum, particularly in the high-dimensional limit (i.e. the number of parameters exceeds the number of data points). Although it is not clear at this stage why these methods perform so well in training deep neural networks, simpler procedures like properly-tuned SGD may work as well or better in these applications.\n",
- "\n",
- "Geron's text, see chapter 11, has several interesting discussions."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "937063d3",
- "metadata": {},
- "source": [
- "## Using Automatic differentation with OLS\n",
- "\n",
- "We conclude the part on optmization by showing how we can make codes\n",
- "for linear regression and logistic regression using **autograd**. The\n",
- "first example shows results with ordinary leats squares."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "e08215a6",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Own inversion\n",
- "[[4.01498551]\n",
- " [3.02821089]]\n",
- "Eigenvalues of Hessian Matrix:[0.29611739 4.55853898]\n",
- "theta from own gd\n",
- "[[4.01498551]\n",
- " [3.02821089]]\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHFCAYAAADxOP3DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABXX0lEQVR4nO3deXgT1foH8G/a0pa10ALdoWURaMu+CEKBqoAIuWAvoqAIuFwVEAp6r3C9KqjIlZ+yKG5cveIVi6gUJKggQgvIoqyyrxYoSylrCoVuyfn9EROaNmn2ZCbz/TxPn5KZyeRMZoZ5e857zlEJIQSIiIiIFCrA1wUgIiIi8iUGQ0RERKRoDIaIiIhI0RgMERERkaIxGCIiIiJFYzBEREREisZgiIiIiBSNwRAREREpGoMhIiIiUjQGQ0QStGjRIqhUKtNPUFAQoqOj8fDDD+PYsWM+K9f06dOhUql89vlycPLkSahUKrz99tu+LgoR2SnI1wUgIus+++wztG7dGsXFxdi8eTNmzpyJ7OxsHD58GA0aNPB18YiI/AKDISIJS0lJQZcuXQAAffv2hU6nw6uvvooVK1Zg7NixPi4d+VJZWZmp1pCIXMNmMiIZMQZGFy5cMC0rLi7G888/jw4dOiAsLAzh4eHo0aMHvvvuuyrvV6lUmDBhAr744gu0adMGtWrVQvv27bFq1aoq237//ffo0KEDQkJCkJiYaLXZp7i4GNOmTUNiYiKCg4MRGxuL8ePH49q1a2bbJSQkYPDgwVi1ahU6duyImjVrok2bNqbPXrRoEdq0aYPatWujW7du2LFjh83vw9icmJ2djWeffRYNGzZEREQE0tPTce7cuSrHPn369Cr7SEhIwJgxY6rsc/369XjqqacQERGBevXq4bHHHkNRURHy8/MxfPhw1K9fH9HR0XjhhRdQVlZWZb96vR4zZ85EkyZNEBoaii5dumDdunVVtjt27BhGjhyJxo0bIyQkBG3atMH7779vtk1OTg5UKhW++OILPP/884iNjUVISAiOHz+Omzdv4oUXXkBiYiJCQ0MRHh6OLl26YMmSJTa/PyIy4J8URDKSm5sLALjjjjtMy0pKSnDlyhW88MILiI2NRWlpKX7++Wekp6fjs88+w2OPPWa2j++//x7bt2/Ha6+9hjp16mD27Nl44IEHcOTIETRr1gwAsG7dOgwZMgQ9evTAV199BZ1Oh9mzZ5sFYQAghMDQoUOxbt06TJs2Dampqdi7dy9effVVbN26FVu3bkVISIhp+99//x3Tpk3DSy+9hLCwMMyYMQPp6emYNm0a1q1bhzfffBMqlQovvvgiBg8ejNzcXNSsWdPm9/Lkk09i0KBByMzMRF5eHv7+97/j0Ucfxfr1653+rp988kmkp6fjq6++wu7du/HPf/4T5eXlOHLkCNLT0/G3v/0NP//8M9566y3ExMRgypQpZu9fsGABmjZtinnz5kGv12P27NkYOHAgNmzYgB49egAADh48iLvuugtNmjTBO++8g6ioKKxZswYTJ07EpUuX8Oqrr5rtc9q0aejRowc++ugjBAQEoHHjxpgyZQq++OILvPHGG+jYsSOKioqwf/9+XL582eljJ1IcQUSS89lnnwkAYtu2baKsrExcv35drF69WkRFRYnevXuLsrIyq+8tLy8XZWVl4oknnhAdO3Y0WwdAREZGisLCQtOy/Px8ERAQIGbNmmVaduedd4qYmBhx69Yt07LCwkIRHh4uKv63sXr1agFAzJ492+xzli5dKgCIhQsXmpY1bdpU1KxZU5w5c8a0bM+ePQKAiI6OFkVFRablK1asEADEypUr7fqexo0bZ7Z89uzZAoA4f/682bG/+uqrVfbRtGlTMXr06Cr7fO6558y2Gzp0qAAg5syZY7a8Q4cOolOnTqbXubm5AoDV7+/ee+81LRswYICIi4sTWq3WbJ8TJkwQoaGh4sqVK0IIIbKzswUA0bt37yrlT0lJEUOHDq2ynIjsx2YyIgnr3r07atSogbp16+K+++5DgwYN8N1331XJE/nmm2/Qs2dP1KlTB0FBQahRowY+/fRTHDp0qMo+09LSULduXdPryMhING7cGKdOnQIAFBUVYfv27UhPT0doaKhpu7p160KtVpvty1jzUrGZCQAefPBB1K5du0qzUIcOHRAbG2t63aZNGwCGfKhatWpVWW4sky1/+ctfzF63a9fOofdbMnjwYLPXxjINGjSoynJLn2Pt+9u4cSN0Oh2Ki4uxbt06PPDAA6hVqxbKy8tNP/fffz+Ki4uxbds2s33+9a9/rfI53bp1w48//oipU6ciJycHt27dcvqYiZSKwRCRhP3vf//D9u3bsX79ejz99NM4dOgQRowYYbZNVlYWhg8fjtjYWCxevBhbt27F9u3b8fjjj6O4uLjKPiMiIqosCwkJMT1Er169Cr1ej6ioqCrbVV52+fJlBAUFoVGjRmbLVSoVoqKiqjTVhIeHm70ODg6udrml8ltS+ZiMTXOuBAaOlNVSOa19f6Wlpbhx4wYuX76M8vJyvPfee6hRo4bZz/333w8AuHTpktn7o6Ojq+zz3XffxYsvvogVK1YgLS0N4eHhGDp0qE+HYCCSG+YMEUlYmzZtTEnTaWlp0Ol0+OSTT/Dtt99i2LBhAIDFixcjMTERS5cuNRsDqKSkxKnPbNCgAVQqFfLz86usq7wsIiIC5eXluHjxollAJIRAfn4+unbt6lQZPCEkJMTid+Kp3Bpr319wcDDq1KmDGjVqIDAwEKNGjcL48eMt7iMxMdHstaUxnmrXro0ZM2ZgxowZuHDhgqmWSK1W4/Dhw+45GCI/x5ohIhmZPXs2GjRogFdeeQV6vR6A4QEZHBxs9qDMz8+32JvMHsbeXFlZWWY1HtevX4dGozHb9p577gFgCMgqWrZsGYqKikzrpSAhIQF79+41W7Z+/XrcuHHDI59n7ftLTU1FYGAgatWqhbS0NOzevRvt2rVDly5dqvxYqsWrTmRkJMaMGYMRI0bgyJEjuHnzprsPi8gvsWaISEYaNGiAadOm4R//+AcyMzPx6KOPYvDgwcjKysK4ceMwbNgw5OXl4fXXX0d0dLTTTSWvv/467rvvPvTr1w/PP/88dDod3nrrLdSuXRtXrlwxbdevXz8MGDAAL774IgoLC9GzZ09Tb7KOHTti1KhR7jp0l40aNQovv/wyXnnlFfTp0wcHDx7EggULEBYW5pHPCwwMRL9+/TBlyhTo9Xq89dZbKCwsxIwZM0zbzJ8/H7169UJqaiqeffZZJCQk4Pr16zh+/Dg0Go1dveHuvPNODB48GO3atUODBg1w6NAhfPHFF+jRo4dZHhYRWcdgiEhmnnvuOSxYsACvvfYaRowYgbFjx6KgoAAfffQR/vvf/6JZs2aYOnUqzpw5Y/bgdUS/fv2wYsUK/Otf/8JDDz2EqKgojBs3Drdu3TLbp0qlwooVKzB9+nR89tlnmDlzJho2bIhRo0bhzTffNOtW72t///vfUVhYiEWLFuHtt99Gt27d8PXXX2PIkCEe+bwJEyaguLgYEydOREFBAZKTk/H999+jZ8+epm2SkpKwa9cuvP766/jXv/6FgoIC1K9fHy1btjTlDdly9913Y+XKlZg7dy5u3ryJ2NhYPPbYY3jppZc8clxE/kglhBC+LgQRERGRrzBniIiIiBSNwRAREREpGoMhIiIiUjQGQ0RERKRoDIaIiIhI0RgMERERkaIpYpwhvV6Pc+fOoW7duhaHsyciIiLpEULg+vXriImJQUCA5+pvFBEMnTt3DvHx8b4uBhERETkhLy8PcXFxHtu/IoKhunXrAjB8mfXq1fNxaYiIiMgehYWFiI+PNz3HPUURwZCxaaxevXoMhoiIiGTG0ykuTKAmIiIiRWMwRERERIrGYIiIiIgUjcEQERERKRqDISIiIlI0BkNERESkaAyGiIiISNEYDBEREZGiMRgiIiIiRWMwRERERIrGYIiIiIgUjcEQERERKRqDISIiIlI0BkNERESkaAyGiIiISNEYDBEREZGiMRgiIiIiRfN5MLRx40ao1WrExMRApVJhxYoVVrd9+umnoVKpMG/ePK+Vj4iIiPybz4OhoqIitG/fHgsWLKh2uxUrVuDXX39FTEyMl0pGREREShDk6wIMHDgQAwcOrHabs2fPYsKECVizZg0GDRrkpZIRERGREvi8ZsgWvV6PUaNG4e9//zuSk5N9XRwiIiLyMz6vGbLlrbfeQlBQECZOnGj3e0pKSlBSUmJ6XVhY6ImiERERkR+QdM3Qzp07MX/+fCxatAgqlcru982aNQthYWGmn/j4eA+WkoiIiORM0sHQpk2bUFBQgCZNmiAoKAhBQUE4deoUnn/+eSQkJFh937Rp06DVak0/eXl53is0ERERyYqkm8lGjRqFe++912zZgAEDMGrUKIwdO9bq+0JCQhASEuLp4hEREZEf8HkwdOPGDRw/ftz0Ojc3F3v27EF4eDiaNGmCiIgIs+1r1KiBqKgotGrVyttFJSIiIj/k82Box44dSEtLM72eMmUKAGD06NFYtGiRj0pFRERESuHzYKhv374QQti9/cmTJz1XGCIiIlIcSSdQExEREXkagyEiIiJSNAZDREREpGgMhoiIiEjRGAwRERGRojEYIiIiIkVjMERERESKxmCIiIiIFI3BEBERESkagyEiIiJSNAZDREREpGgMhoiIiEjRGAwRERGRojEYIiIiIkVjMERERESKxmCIiIiIFI3BEBERESkagyEiIiJSNAZDREREpGgMhoiIiEjRGAwRERGRojEYIiIiIkVjMERERESKxmCIiIiIFI3BEBERESkagyEiIiJSNAZDREREpGgMhoiIiEjRGAwRERGRojEYIiIiIkVjMERERESKxmCIiIiIFI3BEBERESkagyEiIiJSNAZDREREpGgMhoiIiEjRGAwRERGRojEYIiIiIkVjMERERESK5vNgaOPGjVCr1YiJiYFKpcKKFStM68rKyvDiiy+ibdu2qF27NmJiYvDYY4/h3LlzviswERER+RWfB0NFRUVo3749FixYUGXdzZs3sWvXLrz88svYtWsXsrKycPToUfzlL3/xQUmJiIjIH6mEEMLXhTBSqVRYvnw5hg4danWb7du3o1u3bjh16hSaNGli134LCwsRFhYGrVaLevXquam0RERE5Eneen4HeWzPHqLVaqFSqVC/fn2r25SUlKCkpMT0urCw0AslIyIiIjnyeTOZI4qLizF16lSMHDmy2ghx1qxZCAsLM/3Ex8d7sZREREQkJ7IJhsrKyvDwww9Dr9fjgw8+qHbbadOmQavVmn7y8vK8VEoiIiKSG1k0k5WVlWH48OHIzc3F+vXrbbYbhoSEICQkxEulIyIiIjmTfDBkDISOHTuG7OxsRERE+LpIRERE5Ed8HgzduHEDx48fN73Ozc3Fnj17EB4ejpiYGAwbNgy7du3CqlWroNPpkJ+fDwAIDw9HcHCwr4pNREREfsLnXetzcnKQlpZWZfno0aMxffp0JCYmWnxfdnY2+vbta9dnsGs9ERGR/Cima33fvn1RXTwmoWGQiIiIyA/JpjcZERERkScwGCIiIiJFYzBEREREisZgiIiIiBTN5wnUREREfkOnAzZtAs6fB6KjgdRUIDDQ16UiGxgMERERuUNWFjBpEnDmzO1lcXHA/PlAerrvykU2sZmMiIjIVVlZwLBh5oEQAJw9a1ieleWbcpFdGAwRERG5Qqcz1AhZGhfPuCwjw7Cdkul0QE4OsGSJ4beEvg8GQ0RERK7YtKlqjVBFQgB5eYbtlCorC0hIANLSgJEjDb8TEiRTY8ZgiIiIyBXnz7t3O38jgyZEBkNERESuiI5273b+RCZNiAyGiIiIXJGaaug1plJZXq9SAfHxhu2URiZNiAyGiIiIXBEYaOg+D1QNiIyv581T5nhDMmlCZDBERETkqvR04NtvgdhY8+VxcYblSh1nSCZNiCohLDXk+ZfCwkKEhYVBq9WiXr16vi4OERH5K45AbU6nM/QaO3vWct4QYAgYT560+D156/nNmiEiIiJ3CQwE+vYFRoww/FZyIARU34RodOsW8N133iuTBQyGiIiIyHOMTYjh4ZbXX7ni8y72DIaIiIjIs4YMAUJDLa+TQBd7BkNERETkWZs2GfKGrPFxF3sGQ0RERORZEu9iH+STTyUiIiL7yb2XmsS72LNmiIiISMokPsmpXSQ+SjeDISIiIqmyNsnpmTM+74HlEImP0s1giIiISIqqm+QUMCz/2998PsmpXXQ6Q9f6SZOAiAjzdRIYpZs5Q0RERFJka5JTALh8GZg5E3jlFe+UyRlZWYYgqOKxNGoEPPKIocu9BPKfWDNEREQkRfb2rJo/X7q1Q9aa+S5dMpT7yhWfB0IAgyEiIiJpsrdn1ZUrPhufp1rVNfMJYfh55hmgtNT7ZauEwRAREZEUpaZan8KiMh+Nz1Mte5r5Ll405Az5OBGcwRAREZEUBQYaalbs4cr4PDodkJMDLFli+O2uJjd7A7SLF33eM47BEBERkVS99FLV3lcVuTo+jyfHMHI0QOPcZERERFRFYCCwcKHlda6Oz2MtufnsWffU1NgaaLEizk1GREREVqWnA8uWGQKLilwZn8dWcjPgek1NxYEW7eWj3CcGQ0RERFKXng6cPAlkZwOZmYbfubnOD1RoK7nZXTU16emGgK1hQ/u299HcZBx0kYiISA4CA4G+fd2zL2/OIp+eDgwebKjJunjR8jYqlWE95yYjIiIir/D2LPLBwcBHHxmCHs5NRkRERD7ni1nkjU1msbHmyzk3GREREXmdMbl52DBD4FMxkdqTNTXp6Yb5yDZtMjTBRUdLYm4yBkNERERyotO5J5gw1tRUnkQ1Ls4QCHmqpsaduU9u4vNmso0bN0KtViMmJgYqlQorVqwwWy+EwPTp0xETE4OaNWuib9++OHDggG8KS0RE5EvuHiTR3b3UZMrnwVBRURHat2+PBQsWWFw/e/ZszJkzBwsWLMD27dsRFRWFfv364fr1614uKRERkQ/ZO0iio9NrGGtqRoww/JbALPIAcPHQJSyZuNUrn6USwtKIS76hUqmwfPlyDB06FIChVigmJgYZGRl48cUXAQAlJSWIjIzEW2+9haefftqu/RYWFiIsLAxarRb16tXzVPGJiIg8Q6cz1ABZGxvI2DV9zhxg8uSqzV7z59uu7XFX85uThF7goOYENB+egWZLOLZeT4HADQCef35LOmcoNzcX+fn56N+/v2lZSEgI+vTpgy1btlgNhkpKSlBSUmJ6XVhY6PGyEhEReYy9gyQ++GDVdcaao+p6bGVlWc4dsieIckHpjVJs+nA/NJnXodmfgD/KWwBoYVrfNvQo9hV77ONNfN5MVp38/HwAQGRkpNnyyMhI0zpLZs2ahbCwMNNPfHy8R8tJRETkUa4Mfmhreg1Pz1FWyeVjV7D42c14qMkWNKp7C/f+oxPm7+mDP8qbIgTFGNhoOz54eCNObzuHXy7c4dbPtkbSNUNGqkrjIAghqiyraNq0aZgyZYrpdWFhIQMiIiKyj4+biyxydfDDitNrVOzJZWuOMpXKEEQNGeL0dyD0AkdW50Lz/mlofmmAzYUp0KOnaX1j1UUMbnEY6r8G495JyagT1dW0zlstO5IOhqKiogAYaoiiK1wIBQUFVWqLKgoJCUFISIjHy0dERH7GR81FNhkHSTx71nLgYq/KNUyOzFHmQHf4sptl+OWj/dB8qYVmXwKOlzUD0My0vm3oUag7nYN6bEN0G5OEgCDfTMNhJOlgKDExEVFRUVi7di06duwIACgtLcWGDRvw1ltv+bh0RETkV4zNRZWDDXtybjzN1iCJ9gZIlWuY7G1+W7bM8LuaWrKrudfw49sHoFmlwo+nk6FFR9O6YJSgb8Q+qNOKoM5ojqY97wDgnSYwe/g8Z+jGjRvYs2cP9uzZA8CQNL1nzx6cPn0aKpUKGRkZePPNN7F8+XLs378fY8aMQa1atTBy5EjfFpyIiPyHreYiwHrOjbdUN53F1187N72Gvc1vCxZYHNPo6JpcvKPOQd/6e9CoWR088kFPfHX6LmgRhoaqSxjd/Bd8+8JWXDpbijWXumDCN33QtGec/cfsJT7vWp+Tk4O0tLQqy0ePHo1FixZBCIEZM2bg448/xtWrV3HnnXfi/fffR0pKit2fwa71RH5Ainkc5D9ycgwPe1uys30/erK1e8FYswVYnl7DUs2Wscu+nc1vQqUCBPB5s+mYlTcKR8sSzdYnhxyDuuNZqMdE4M6xSQgMdu0e9dbz2+fBkDcwGCKSOanmcZD/WLLEMKKzLZmZhsEJpcrSvRIfX/30GtaCKCv0UOEM4pCIXARChz7h+6DuewPqSc2Q2Nu9nZUYDLkRgyEiGbOWx1HdX7tEjpJTzZAtztSiWgqibMgZtgCd5o5CvTjPPVcZDLkRgyEimbJ31N3cXDaZuUrpzZC2mov8/ForLy7H1oV7ceCDjahzdBceFV/YfpOlWjI3X0feen5LujcZESmch7r9UiVshrTdWwswNDVJORByMBDRntZizdwD0Hynxw8nk3BFdALQCX2Qg0dhRzBUOflaxteRz3uTERFZZW+3X1dG51U6L48+LGnV9daSenOsnbPZ/5FzGvPTN+De8F1o2LQWHpp3Fxbn9sIVEY4Gqqt4JGEzxk0IhD46xrGeaTK/jthMRkTS5U95HFLEZkjL5NZkWE1enQBw5G/vYNGejtDsicPBkhZmm7QK/gPqdqehfqwB7noqGUGhQeb7BGz3TPPgdcScITdiMEQkUwrP4/A4BpvyZyMQqdjzS49ABKIcqfX3Qd1bC/WEBLTsl2B93/b2TPPgdcScISIif8jjkDI2Q8qfjby6AAg0QR6mR36AFg92xn1TktAgsaPV7c2kpxvmJLNVS+YH1xGDISKSNmMeh6XEzOrGTiHb7B19+OBBw1//Um8uUhB9uR6/LTqIU7PX4SE7tn95bkNgxF2Of1BgoO3aHHuvI1cnm/UgNpMRkTzILY9DDhwcfVguPYP81Y38G1g7dz80y8vw/fHWKBCN0Ac5yIGPmzo92JzNZjIioors+QuVHFNdM6QlUpiwVGFObz2LVfOPQ7OuFtZfaodSdDetqwctouICUXS1EWrdvARVdYFI5TnJ3MkPmrPZtZ6ISMmsdSe3RCoTlvoxfbkev312AC+n5qBDzSNoelcsxi/tg9WXuqIUIWgWdAqTOmzAz7N34eL1mvgqLxW1//cRVEDVrvDeDETkPCwB2ExGRETA7WbIdeuAN96wvb0Ue5jJtCm1qKAIP8/bD82yUnx//A7k6yNN6wKgQ4+6B6DueQXqZ+PQZnBzqAIsjP/jzJxknmDpHABOnxc2kxERkfcYmyHl2jNIZqMfn9l+HqvmHoPm55pYd7EtSnCnaV1dFGJA7AGo79fh/ufboGGrdrZ3aG/PL0+r3Jwtk/PCYIiIiG6TY88ga4MOSijHSV+ux67Mw9B8WgDN9kjsvtUGwO3vMCEoD+qkP6AeUQd9JrRFcJ0ejn+I1PLqZHBejNhMRkREt8ltoEsJj6J989JNrJu/H5pvi7Hq6B04r4+6XSzo0b3OAajvugz1M7FIHtLCcvOXXLnpvLCZjIiIvE9uPYMkNpnvuV35WDXnKDRrQ7GuIAW30M20rg6uo3/MAagHluP+ya3QOLmtx8vjMxI7L7YwGCIiInNyGujSxzlOQi+we8lhaD65AM1vjbHzZhKA2zVATQLPQJ10AuqHaqPvc20RUq+79Z35E5nlnjEYIiKiqqSSkGuLD3Kcbl25hfXv7ofm61tYdaQlzurbAGgDwND81a32Qah7XIL66Ri0TW8JVUCc2z5bNmSWe8acISIiki8v5Tjl7y3AqneOQPNTMH7OT8FN1Datq4Ui9I/eB/WAMgx6vjUiUxo59yEyHRrAIjedF+YMERGRb8nh4exojpOdxyT0Ar9/cxSaheeh+bURthclA2hsWh8XeA7q1sehHl4TaRPbIrS+i81fMumCbje55Z4JBdBqtQKA0Gq1vi4KEZE8LFsmRFycEIbHmOEnLs6w3JryciGys4XIzDT8Li/3Vmktlzc+3ry8No7p1tVb4sfXt4txKTkiPvCM2WaAEF1r7xev3Z0tdn91WOh1eveWXaUSVT5QpTL8VPedS50956Ua3np+s5mMiIjMWRsfxvgXvaXxYaRQs1FdrY+VYxJ/1lrMrj8Lr1+bgCLUMa2riZvoF7UP6v6lGDT5DkR3iITbSXhoALdxoYbRW89vBkNERO4ih2YlW5x5ODsTPHmTjWPSQ4UziEMichEVUIDBdxyD+sFQ3JPRFjXDa3q2bDk5QJqPZ50HJHvteuv5zYlaiYjcISvL8MBNSwNGjjT8TkgwLJcTR8aHAQwP0UmTLCfJSmRi19LV66o9pgAINEEeDv5zMc6UReHjQ70x+LVung+EAGl0QfeXa9cFDIaIiFxlrBmp/MA1Tjsgp4eKow9nR4MnL7l46BI+f+oXDIvbiqcHV1O+ClqlBHt/FGhfd0H3p2vXBQyGiIhcIYOaEYc4+nCWQs0GDL2/Dnx3HP++Lwc96+1FZFI4xnzSC8vO9kAumtm3E1+MeZOaCkREVL9NeLjh+lmyxNCs5q5ryd+uXRewaz0RkStkNu2ATamphpwgW+PDpKYaXvuwZqP0Rik2frAfmszr0BxIRG55CwAtTOs71DwMdZd8qMdGQLwSB5W9xyQ1164B9957+7W7EtP97dp1AYMhIiJXSKRmxG0cHR/G0eDJRZePXcEP7xyE5vtArDmThEJ0Mq0LQTHubrQP6ntuYXBGC8Tf2RpAa8PKMImOebNpE3D5cvXb6PXmr90167u/XbsuYDBEROSoij1vLlyw7z0SmXbALo7MTebhwfWEXuDwD39g1fsncXpjLq7crImziMUm9IAegWisuojBLQ5D/ddg3DspGXWiurp+TK5ypGeWM4GGEIbvNiPDMGWKs0Gcr/OVJIRd64nkSKLdYBXB0ng6gYHW8yq8NU6MJ64JR/Zp6XuJj3cq0Ci7WYZNH+6H5stCaPYloF35TszHJMTj9r61wY1w4Yl/osW7ExEQ5ED6q6fvHUfHW7K3a701rnS599JUJq7w2vPbo0M6SgRHoCa/4szIwOQe1kYKtvbjrRGEpXJNuDAC9eXjV8TiZ38RD8VvFmG4ZjqMB7BM6KASejmMzuzMSNLl5YZz5ch1VfEnM9M9Za78+RL5fr31/GYwRCQn/jxsv9QZH1rVPZgCA52edsBpMr4mDv9wQvzfoGzRO2y3CESZWfEbqQrE2ObZoqhO46qBUMVjjI/37rQf1ti6Pqorq7WAxJ6f7GzXy+7ilBmexOk43IjNZOQXlDBsv5TZ25wxdy4QGWneBOOpphmZXRPlxeX45aP90Cy+Bs3epjhWlmi2PiXkGNSdzkI9JgLdxiQhcMsmaYzObA9XR5L2dfOrRJveOWs9EZljN1jfsjfRNTISGDHi9mtPztklg2vi2iktfnz7ADQagR9PJ+Oa6GBaVwOl6Bu+D+q0Gxg8sRkSe7cE0PL2m+XU28nVsqanG5KhKwYkly4Bw4cb1nu6B1xgoKL/32AwRCQXcnow+CNnet5Ym7PLz7tGH1t7EpoFJ6HZGIZN19pCh7tM6yJUl3F/4iGohwRiwJRk1IvrbH1Hcurt5I6yWgpIvNUDTuHYTEYkF1KZ0FGpHO15Y6sJCwAaNTKsDw52rkwSuSbKi8ux9ZMD0PzvKjS/x+NwaXOz9W2CT0DdIQ/q0eHo8WQyAoPtrM2QQW8nE0+WVaJNWN7AWevdiMEQ+QX+Z+t7xpoewHKzRcWaHnsDlYYNgY8/du6vfFeuCRfPufa0FmvmHoDmOz1+OJmEKyLctC4IZejdYB/Ufa5D/VwCmt/d1PFjM3LkO/c1OZVVJti13o3Ym4z8hie6wUqlW7Zc2NvzJjPTsS74zn7fzlwTTp7zE9mnxLwHcsQ9DXaKIJSavb2B6op4JOEX8dXEzeLaqWvOHUt1xyjR3k5VyKmsMsCu9X8qKysTL730kkhISBChoaEiMTFRzJgxQ+h0Orv3wWCI/Io7/7OVcbdsl7kwJo5d783OdiwYcqWLuCPXhAPnvLykXPzywe/ixTuzRVLIsSpvaRV8QrzQJVtseHePKLtV5lzZ7eXK+fI2OZVV4ti1/k8zZ87E3Llz8fnnnyM5ORk7duzA2LFj8cYbb2DSpEl27YPNZOR33NGsJbNu2W7lyR5eRraasCxxdTRhW9eEHedcHx2DrOFfQbMS+CG3NS6JhqbVgShHav19UPfWQj0hAS37JThXViI7sWv9n7Zu3YohQ4Zg0KBBAICEhAQsWbIEO3bs8HHJiHzIHd1gZdAt2yM83cPLqOKcXfZypdeXPdeEHec84NxZLJhXjg0w7Ku+6hoGNjkAtVqF+6YkoUFiR+fLSCRRDkzo4hu9evXCunXrcPToUQDA77//jl9++QX333+/j0tGJHMS7ZbtUTqdoUbIUk2NcVlGhvWB7hxlnBy0USP7tvd0F3E7z2XngN14/455ODBqFi7+sAOZJ7pjxHt3oUFifc+Wj5RNpzN0PFiyxPDbXfehHSRfM/Tiiy9Cq9WidevWCAwMhE6nw8yZMzGi4qBmlZSUlKCkpMT0urCw0BtFJZIXOY3h4i6+qA1LTwcGDwZiYw2D6FlibJJMTXXPZ1pwI/8Gdn13Fb3t2Pad8DeBo5eAowC+gPubEIkqs9Z0PWuWVz5e8jVDS5cuxeLFi5GZmYldu3bh888/x9tvv43PP//c6ntmzZqFsLAw0098fLwXS0wkE6mphv9sjN1+K1OpDLOOe/AB7XW+qg0LDjZ0n1epqn7fnhhN+E+nt57F+8M34L6GOxARXQNpS59GHuKgh5VzblQ5aDM2IWZlubV8RABuN11X/kPl7Flg1CivFEHyCdTx8fGYOnUqxo8fb1r2xhtvYPHixTh8+LDF91iqGYqPj2cCNXmWHMfqUdq4KL4epNDSX7/x8W4bTVhfrsf2zw9C89klaHZGY29xK7P1zYNO4Z/x/8PY3FcBFaCqfM6rexxILaFejvcbVWUjqb8QQBjABOqbN28iIMC8AiswMBB6vd7qe0JCQhASEuLpohHd5o3eSZ5gzGlRynD/xtowW4MUeqo2zNL8Uy4+xIsKirB27n5olpXi+xOtcEGfYloXAB3uqrcf6p5XoR4Xj9b3N4Mq4GUgK7nqOW/YELh40foHSSmhXq73G1Vlq+naSyQfDKnVasycORNNmjRBcnIydu/ejTlz5uDxxx/3ddGIDLzVO8lTPPCA9gh31ARU7OFVuSbEg81VVcrgYjBxZvt5rJp7DJqfa2LdxbYowZ2mdXVRiPviDkA9SIeBk9ugYav2VXdg6ZyfPQs8+qjtD/d1Qr3c7zcy5+vr6U+Sbya7fv06Xn75ZSxfvhwFBQWIiYnBiBEj8MorryDYzvl8OM4QeYySx+rxJnfXBHi4ucrd9OV67Mo8DM2nBdBsj8TuW20AGGp+UrEJ7QP2oUWiDm2e7IneE9ojuI4Tc535ugnRHrzf/I+N685bzWSSD4bcgcEQeYwcHiByZ60mwNW8JonnnNy8dBPr5u+H5ttirDp6B87ro0zrVNDjxdD5mCZmoV5JhaYtVwJEOUyKyvvN/9i47pgzRCQHShyrxxpPBBe2xgVSqQzjAg0Z4lyTmcQemOd25WPVnKPQrA3FzwVtUYxupnV1cB39Yw5APbAcD6QcQ9iU593bVCSFJkRbeL/5H1vXnZfqayTftZ5I0pQ4Vo8lWVmGv+7S0oCRIw2/ExJc74rtyLhAMiT0Aru+PIQZaTnoUvsgYjtH4ekve2NVQTcUoyaaBJ7B+LYbsPqNHbikDcays90x5uMeCHvnFc8MHGlMqI+NNV8eFyeNXBzeb/6puuvuiy+8UgSHmsny8vJkOWYPm8nIY+TQtOBpnmrGAgwj0Y4caXu7zEygmoFYvcpGDdmtK7ew/t390Hx9C6uOtMRZ/e0Htwp6dKt9EOoel6B+OgZt01tCFVBpTCBvNBVJtQmR95t/s3DdFRYVSW9ustatW2PKlCmYOnUqateu7akyEcmHHJoWPMmTzViA/GoCrCR6X538Gpb93gKan4Lxc34KbqKraXUtFKF/9D6oB5Rh0POtEdmmzZ8PhJ3AxnNVAxFvNBVJsAkRAO83f+fL686RKe43b94sunXrJqKjo8V///tfR97qU1qtVgAQWq3W10Uhf7VsmRBxcUIY/ns2/MTHG5b7s+xs82O29pOd7dz+y8sN36tKZXm/KpXhey4vd+dROWfZMovl1EEldFCJB7DMtDgu8Kx4NnmD+GHGb+LW1Vvm+6h8HcXFmV9Hnv7O5cDS99SokRBff+3rkpG7lJcLkZ0ttJ984pXnt0PBkNHnn38u4uLiRIcOHUS2DG44BkPkFX/evCIz0/BbCg9oT8vMtO/BnJnp/GcYg4zKgYZxmRQCzvJyoY+JFXorx6+DSpxXRYnX034Wu786LPQ6fdV9WAmmqhynnAJET/rmG0MAVF3gSPJUIdjVAtINhoQQ4ubNm+Lll18WtWrVEkOHDhXHjh1zZ7ncisEQkYd4q5ZCojVv+fsKxKdjNoqXwt937XswBjjW3lc5wJFDgOhJ9gaOJD+Vzq23giGne5MJIdC/f3/87W9/w8qVK5GSkoLnn38e169fd0/7HRFJn7cme01PB06eNCQFZ2Yafufmer13k9AL7P32KGb2y0H3OvsR3TYCTyxKxYkrDezbgbU8Hkd7zUm915cn2cpTA5zvTUe+Vd259TCHEqg/+ugjbN++Hdu3b8ehQ4cQGBiIdu3aYfz48ejQoQO+/PJLJCUlYfny5ejSpYunykxEUuHNhFYfJVeWFJYg57190CwtwqqDzXFKdweAO0zrO9c6iAEtLwC/27Eza4ne9iY7r1t3u6fNkCHymEbF3RwJHKWYBE7W+XCeMoeCoZkzZ6J79+4YPXo0unfvji5duphNiPr444/jzTffxJgxY7B//363F5aIJMgPJ3u9eOgSvn/nMDQ/BuGnc8m4gdt/3IXiFu5tvA/qfsUYlNESsV2SAF0rIOEd5yeAtbc33Btv3P63Uicm5cCL/suH58yhYCgvL8/mNk888QRefvllpwtERDIkl8lerRB6gYOaE1j5wRlotkRg241kCPQyrY8OyMfgO45CPSwU90xKQa2G3cx34GoN2cWLhnWONO0odWJSuQ23QPbz4Tlz+9xkQghs3LgRffr0ceduXcJBF4mostIbpdj4wX5oMq9DcyARueVNzNZ3rHkI6q4XoH6iMTqNbI2AIDtSLJ2ZANbaoJX2UOIggxx40X9ZOLecqNWNGAwREQBcPnYFP7xzEJrvA7HmTBIKEWZaF4Ji3NNoH9T33sLgyS0R19XJv1IdGb3Z1izs9lLaxKTGABKwXAuntNoyf1Lp3HKiViJHSHX6APIpoRc4/MMf0HyQB83mBthSmAJ9heavyIACDG5xBOq/BuPejBTUbty1mr3ZyZFEb3cljCotP8YP89ToT9bOrYcxGCL5szIFgiKTSwllN8uw6cP90HxZCM2+BJwobw6guWl9+9AjUHc5D/XjjdBlVBsEBDX2XWHdFcQoMT9G5nlqVI2K5/bECeDJJz3+kWwmI3nz5CShJBtXTlzFj+8chGaVCqvzkqGt0PwVjBKkReyD+u4iDJ7UHE17xvmwpJXYO+mqNcyPIT/nrec3gyGSL1v5FnJ8ULC5z25HfvwDmvdPQ/NLfWzWpkBXoaK7keoiBjU/DPUDNdAvIxl1Y+r6sKTVsCcZODwcuHLF8Jr5MaQw3np+s5mM5MvfBl9jc1+1yovL8ctH+6FZfA2avU1xrKwZgGam9Skhx6DudBbqMRHoNiYJgcEujnrtDfZ0yV+40PCb+TFEHsNgiOTLnwZfs9bcJ5exZDxUo3U19xpWzzkIjUbgx9PJuCY6mNbVQCn6hu+DOu0G1BnNkdCrJYCWLn+m19mbDMz8GCKPYTMZyZe9+RZS73ZsT/fqRo0M64ODvVYsu7m5RuvY2pPQLDgJzcYwbLrW1qz5K0J1GYMSD0E9NBD9JyejXpwf3c9sIiWqgjlDbsRgyE/5y+Br9gZ1jRoBH30krRoiNySwlxeXY8t/DkDzv6vQ7G2CI6XNzNYnhRyHusMZqEeHo/sTyQgMlvC5JCK3Ys4QkS3V5VsAhtdz5kg7EALsb8a7eFFaTWa2Zg9XqQyzhw8ZUuUcaE9rsfqdA9CsFPjxVBtcEe1N64JQhj4N9kHd9zrUExPRrG8LAC08eyxEpGgMhkjebA3QNXkyEBAgjeDBGkfHiLESYHidgwnsJ9afgua9k9BsqIuNV9uiHHeZNg1XXcH9CQehHhKAAZOTEdakkxcOgIjIgMEQyV96uqGWYvjwquvkkICcmmpozrPW3FeRlHrI2VmjlfnkOryRF49Dpc0BNDUtbx18Aur2eVA/1gA9nkxGUGgv6zshIvIgBkMkfzodMGWK5XU2mmuc/jx3JrpWbO6zlxR6yNlZo7XwxD04hOYIRDl6198HdR8t1M8losU95iNDO0TqycZSLx8RmbFjGmYiiXOkucZVWVmGpO20NGDkSMPvhATDclcYm/saNrRveylMv/BnjZaAyuJqPVQ4g1jENQnAkue24NLJIqy/2hGTV/RFi3uaWnyPXTx1DtxF6uUjoirYm4zkb8kSw0PHlsxMYMQI5z/HG1N/lJYamswuXrS8XgI95HSlOvy26CA0iy7j5vb9mFM+EQAQgNvfi+FfKuiXLEXgww+678OlPv2K1MvnLqz5Ii9h13o3YjDk57wx3pA3p/4wPlAByUy/cCP/Bn6asx+a5WX4/kRrXBSNTOv+im/wgeo5NBYXbr8hPt79oyO7+xy4+4Huj9PDWMKR0smLvPb8Fgqg1WoFAKHVan1dFOkrLxciO1uIzEzD7/JyX5fItvJyIeLihFCphDCED+Y/KpUQ8fGuHUt2tuV9V/7JznbPMS1bZjimivuOjzcs95JTW86IBQ/miAER20Uwis2KEoZr4qH4zWLxs7+Iy8evOH7dOHOdufMcWPp+4+Jc+369fY34wrJllu8zlcrw48Xrk5TBW89vJlDTbXL9i8+e+Z3mzXPtr3FvT/2Rnu716Rf05Xps//wgNJ9dgmZnNPYWtwIQa1rfPOgU1G1PQv1IPaQ+m4Iate4y34G9tW7OXmfuOgeemvrEn6aHqchYg3b2rGGoCifGlSKSPI+GWhLBmiE7+MNffJ6sTfHTv/pvXLghlk/dJh5vuVFEBlwwO5QAlIte9faItwZmi4Oa40Kv07v+ga5cZ+44B8ZaRGvvdaUW0R+vEUv3lD8dH0met57fzBmSG08kLvpTroOnEjv9ZeoPAHm/nsOqecehWVcT6y+2RQlCTevqohD3xR2AepAO9z+fhIiW4a5/YMWahYwM4NIly9vZ+g7dcQ48mV/mR9cIAOs1aLa42lGBqAJOx0FVeaoZy8GRhCUtMNAzZfRGU5wz7Aj+9OV67PzyMDSfFkCzIwp7brUGEGNanxh0GurkXKhH1kXvcSkIrtPDfeWzdM1aY+s6c8c58GRTllSvEWdUN9WKLVIY9oHIUR6td5IIv2gm82QzVmamfdXfmZnuOx65kkBic7Vl+TMJuOhikfjun9vEk602iKiAfPNLBjpxV93fxawB2WL/imPuaf6yVj5rSe2uXGeunANvNGVJ6Rpxlr3fk7uaGImsYDOZG8m+mczTzVieaDrw53FIpHBsVpowjK9GIBNLcbupog6uY0DsfqgH6nD/lNZo1MbOwR2dZeuarY4915mz58BbTVlSuEZcYe/YXUb+No4SSQbHGXIj2QdDnh5Hx90PCLn2SpOLP8+XOHPG4tjPhpGf49A3YBMGJZ+EekQd9BmfgpB6Id4ro73XbEXeyqmR4DhOkuPo+YuLA556CmjZUp7BH0mWt57fnI5DDjzdZdeY6wDcfiAYOZrrYHzQVK4RMHZb5pQELrl15Ra2jl0IWAmEAMNI0E2QhxNr/8B7e/ug/7TOzgdCOp3hwbhkieG3Tmff+xy9Fr2ZU2Oc+iQ21nx5XBwDISPj5MGV/z+oqFEjYPFiYMYMQ1D56qucfoRki8GQHNibkOhK4qI7HhDVJV0al2Vk2P9A9QfOBhMVnN9zAf95bCP+EvUrIiIE3v2ivl3vU13Id/izzLgyx5aj16K3A5H0dODkSUNtamam4XduLgMhI1t/IKlUwEcfATVrAtOnG/7YqYh//JDceDQjSSJkn0DtjRGWK36WsyNQ++M4K65wcpRjvU4vdmUeEjPSskWXWgeqfH0PBnztnSRgVxL27blmGzUSYvFi+Yx0rkTVJYN7cswmoj956/kti2DozJkz4pFHHhHh4eGiZs2aon379mLHjh12v1/2wZAQtx9OlR8uUhoUkb3SbnMwmLh19Zb4YcZv4pmkDSIu8GyVt3WtvV+8dne22J15QOh/WitEeLjnHkK2HnKAIZApKbHvO5DyNUu2WfsDiX/8kBdwOo4/Xb16FT179kRaWhp+/PFHNG7cGCdOnED9+vV9XTTvMjZjWUpMdveEmM7yRnOeHNhqLvxz2oILze/C9/OOQbOmBtaeT0ERupo2q4mb6Be1D+r+pRg0+Q5Ed0j+MzF9VPU9tNyRe2Nr3CkAuHjR0KT68cfWrz05XLNkm7Wxu/x1+hFSJMn3Jps6dSo2b96MTZs2Ob0P2fcmq0jKXXb9bQReZ9nZEycN65CDu02vYwPOY3CrY1APr4m7J6agZnjN2xvbOxqwO2aLd7Rb9bJl1X9eaSnwwQfAiRNA8+bAuHFAcLDz5SNp8HQvVyKwa71JUlISBgwYgDNnzmDDhg2IjY3FuHHj8NRTT1l9T0lJCUpKSkyvCwsLER8f7x/BkNSx27LdwcQIZOJYrfZQdyuA+slIdBzRGqoAC7137BmzJzwc+Pprw0PH1UDT0W7VERHAhQuWP5fDLPgv/vFDXsCu9X/6448/8OGHH6Jly5ZYs2YNnnnmGUycOBH/+9//rL5n1qxZCAsLM/3Ex8d7scQKx27LuFZay67t3vs4BDuKkvBqdl90eqSN5UAIsK/Z6soVwwPHHQ8de7pVV3T5siGAqozDLPg3dw7JQeRjkq8ZCg4ORpcuXbBlyxbTsokTJ2L79u3YunWrxfewZkgCpNyc56pKxyZ69sKBVbnQfHQWmi0R+O1Ga+SiGWJxFgFww1/M9jZbuXOCTEcn6fzXv4DXX7/92p8m/6XqWar9c0dzLRE4UatJdHQ0kpKSzJa1adMGy5Yts/qekJAQhIR4cbRdpaou4PHUhKm+ZuE//vOIxitYgOW4/R///OC/4/9KJ0FABRUsNBc68hezLxLTjTV8Y8YA1687/n5/mvyXqpeeDgwZ4r9//JAiSD4Y6tmzJ44cOWK27OjRo2jatKmPSkQAFJkLUvje56g7cQwAmI3+HIV8fIthmFXv34gY1B2DJ7dEXNeJQFace3pSGZutbOVmpKY6c1jWpacDdeoAAwbY3rZyQOPOnkb+XMvoL/z1jx9SDo923HeD3377TQQFBYmZM2eKY8eOiS+//FLUqlVLLF682O59+MU4Q1Li6oB8MqHX6cWB746Jf9+XLVLr7BKnESd0VsZS0Vsb28eVQSwr8tWYPeXlQkREVD+OTERE1eNy1xg0Tg5cSUT+gYMuVqDRaERKSooICQkRrVu3FgsXLnTo/QyG3MjPR50tLSoVP8/eKSZ1yBHNgk6aDqsPsn0/wFx1owF70rJl1R+zpc93x6jpCgm6icg6bz2/JZ9A7Q5+Nc6Qr/nh2CJXTlzFD28fhOZ7FVbnJaMQYaZ1wSjB3Q33YnILDfpve72avfzJnUnMlviqyeibb4BnnzX0HDOy1SzqyjALTMAmIjCBmqTKT0adPfLjH1i54DQ0vzTA5sIU6NHTtK6x6iIGtTgM9QM10G9yCupEdQVyioA0O4IhT4+u7YvcjKwsYMoU80CoUSNg7tzqc59cGYGaCdhE5EUMhsgxMp1yo+xmGTYvPADN4mvQ7G2KY2XNADQzrW8behTqTuegHtsQ3cYkISCoUjKyr5KYfc1aF/tLl4Dhw22PHeVsTyM/CbqJSB4YDHmLv/SIkVFQcDX3GlbPOQiNRuDH08m4JjqY1tVAKdIi9kKdVoTBk5ojodcdAO6wvjPjAHPDhhmO0VKzj78NMGfnHGsYMqT643amNkumQTcRyRNzhrzB37qhS3jKjWNrT0Kz4CQ0G8Ow6Vpb6CrE+w1VlzCo2WGoHwhE/8kpqBtT1/EPUNIAc77MD+NUD0QE5gz5D2vNDMYpCeQ4RYWEZiMvLy7Hlv8cgOZ/V6HZ2wRHSpsBSDCtTwo5jr90PAP1mAjcOTYJgcG97N+5pdo8JQ0w58umKiXWxBGRz7BmyJP8vUeMj5r+tKe1WP3OAWhWCvxwKglXRQPTuiCUoU+DfVD3vQ71xEQ069vEuQ/xt9o8Z0ih56CSauKIqArOWu9GPguGpPAw8RMn1p+C5r2T0Gyoi41X26IcNUzrwlVXcH/CQaiHBGDA5GSENQmrZk92sFabJ4FmQK+SSlOVv+TbEZHD2EzmD9gjxmm6Uh22fnIAms+vQLMnHodKmwO4PQVL6+ATULfPg/qxBujxZDKCQh1o/qr2g92UNOwPpNJUxakeiMjDGAx5EnvEOKTwTCHWzDkAzXc6/JDbBpdFO9O6IJQhtf5+qPtooX4uES3uaQ6gufsLIYXxbaRUEyKh/DAiIk9hMORJMuqG7lUVHvbnCwLx7YaGWLkhDBuutEUZepg2a6C6ioFNDkL9FxXuez4Z9Zt29HzZfF2bJ8VcJSUljRORIjEY8qSKzQyWCAE8/LCiHiq6r5ZC9+SzCC66CgCIBjAUccjGfJShM+6okQt1u1NQj6qPnk+nICi0Z/U7dDdf1uZJuechm6qIyI8xgdob/vEP4P/+z/I6lcryQ05KTSUuun7uOn6aux81//MeBmqXQFVpvR6ACsD5f32AmNef9UEJK/BV0rC/9zwkInKCt57fAR7bMxnodMCSJdVvk5Fh2M4oK8vwYExLA0aONPxOSDAsl4lTm89gwYMbMKDhDjSMDcaSt89ioNby9xAAQzAU8/ks8+/BF4y1ecDtJGEjTyYN5+TYn6tERERuxWDI0xxJyAVuN5VUfo+xqUSiAZG+XI9tn+zHSz1z0K7mUST0isNz3/bBT5e7oBxB+BhPQwVUqRUyI5WHvTFpODbWfHlcnGeaqrKyDPN82YM9D4mI3I45Q57mSEKuXLp1/9mEV3w4Fzs23sJn25Ow6kQyCkSKaZMA6NCz3n6oe13FQ6nnEDHtin37lsrD3ltJw9byhKxhz0MiIrdjMORpjiTkSqFbtw2X/v0JQl/7J+rcuohQAL0ANEUcrmI+1uEe3Bd/AOpBAgOntEFEy/aGN9lqJqxISg97TycNVxf8VqbUnodERF7AYMjTHOle//XX9u3Ti7Un+nI9dn55GJpPC1D06178X2kGAPPjiMMZLMNfUb54KWo8YqG5x94Ap1EjZT3sbQW/lXEuLiIij2DOkKc5kpArkUEab166iZUv/YqnWm9EbMhFdBuThJmbUpFR+n8ARJWLRgVApVKhxrQXLCdAGwNCW95/X1kPe3uD2vBw5UwBQkTkAwyGvMHehFxj0FA5aDJSqQyTVHqg9uTsjvP4+JGNGBz5GyIaqTDkzTvxyZHeyNdHoi4K8a+GHyEeZ6xfMNX1djIGhNaOCwD+/nfgwQfdcSjyYW9Q+/XXDISIiDyIzWTeYk9CrhfnghJ6gV2Zh6H55AI0v0Vi1602MAyBaNA08Az+knwC6hF10HtcCkK+DwdG2rFja7Ud1qZ1aNQI+OAD6wNT+jN7m1A52CERkUcxGPImexJyPTgX1K0rt7Bu3j5ovinGqqMtcU7fBkAbAIAKetxZ5wDUPS5D/XQMUh5oCVVAhaYtdzThcVoHc1KZCJWISOE4ArVUuWkE6vN7LmDVnCPQ/BSCny+0xS3UMq2rjRvoH70f6vvKMOj51mic3Kj68vhiZGYlsDQfWXw8J0IlIsXz1vObwZCfEXqBPUuPQPOffGh+bYwdN5PM1scHnoW6zXGoH6qNvhNSEFo/1P6dG8fEASzXYvg6yVfOU5jIuexERB7CYMiN/D0YKr5WjPXz90Hz9S2sOtICZ3QxZuu71d4PdfdLUP8tGu2G3QFVQDWJzLYeylKtxZDibO9EROQSBkNu5I/BUP7eAnw/5wg0a4KxNj8FN1HbtK4WitAvaj/UA0oxaEorRLVrbN9O7Q0opFaLYW0UZ6nUWBERkVMYDLmRPwRDQi+w99uj0Cw8D822hvitKMVsfWzAeahbH4N6eE2kPZeCmuE1HfsAuQYUnO2diMhvMRhyI7kGQyWFJch+dx80S4uw6lBznNaZD1zYpdZBqO8sgPqpKHR4qFX1zV/VkXNAkZMDpKXZ3i47m13UiYhkxlvPb3att5eXmoYKDlzE9+8chmZ1Dfx0PgW30BGp2IS7sAmt0QA1G9fDoAHlGJRxB2I6JQFIsrlPm2QwJ5pVjkyES0REZAGDIXt4MDlX6AX2Lz8GzcfnoNkagV9vJEPAMML0A8jCAkxADCo8yIPjgKHzgU69XfpcM3IOKCQyhQkREckXgyFbrOXSnD1rWO5ELk1JYQk2vL8fmiU3sOpQM5wsvwPAHab1nWoewj8Sv8Hwg9NReVJUVz7XKjkHFI5MhOtNUksyJyIiq5gzVB035tJcOnIZP7xzCJofArHmbDKu43Y5QnEL9zTeB/W9xRg8uSViOzb2bg6P3AdUlNr4R+zmT0TkFkygdiOnv0wXknOFXuDQqhPQfHgGms3h2Ho9GXrcDiQiAwowuMURqP8ajHszUlC78e2u8W5PCranlkJqAYWjpDL+kVx75RERSRATqKXAwVya0hul2PThfmgyr0OzPwF/lLcA0MK0WfvQI1B3OQ/1443QZVQbBARZGf/HnTk89tZSeHBONK+QwrxnOp3h+7P094UQhoAoI8NQTinWsBERKRSDoerYmSOzdsklfPLiFqzOS0YhOpmWB6MEdzfcC/U9NzF4Ugs06dEKQCu3fa7N7RzNd5JCQOEKeybC9SQ598ojIlIwNpNVx0YujQBwBnFIwElTE1hj1UUManEY6gdqoN/kFNSJquN4gd2RwyPnsYPkaskSYORI29tlZgIjRni+PEREMuetZrIAj+3ZHwQGAvPnQwAQqDqgoQpATdzCxBof4p935WDrf/bjfGkE/ns0FQ+81d25QKjC5xo+pNLnGl/Pm1d9EONILQW5h5x75RERKRiDISuu5l5D5vjNGDE5Co+Jz3EZ4Ra3i8AVzC2fiJnPX0H3J1MQEOSmr9SYwxMba748Ls6+JFw5jx0kV8Zu/pUDWCOVypDU7e1u/kREVC02k1VwdE0uNAtOQbOpPn7RpkD3Z0pVAHQ4jXjE4LyF+iF4tsnJ2fFqOE2Fb8i9Vx4RkYSwmcyKWbNmQaVSISMjw+V9lReXY8P8PXihSw5aBeei1X2JeGFVX2zQdoAOQUgOOYap3XPw++TPEWstEAI82+RkTAoeMcLw295gi7UUvuFqjR4REXmdrHqTbd++HQsXLkS7du2c3se1U1qsfucANCsFfjydhKuig2ldDZSiT/g+/KWPFn/tfRExkXpDbczZEPt2LqUmJ2Pe0bBhhsDHUi2Frbwjco7ce+URESmMbIKhGzdu4JFHHsF//vMfvPHGG07tY3CT37FV2x3luMu0LEJ1GfcnHoJ6SCAGTElGvd9OGcaKWV4h+bhhQ/s+QGqJsXIfO0jOfN3Nn4iI7CabnKHRo0cjPDwcc+fORd++fdGhQwfMmzfP4rYlJSUoKSkxvS4sLER8fDwALYB6aBN8AuoOeVCPDkePJ5MRGPznX+zWxuWxRerd1DlPFhERyRBHoK7gq6++wq5du7B9+3a7tp81axZmzJhRdfmgX/DglGQ0v7s5gObmK6sbPbgiOTY5sZaCiIjIKsknUOfl5WHSpElYvHgxQkND7XrPtGnToNVqTT95eXkAgHGZvdD87qaW32RrXB6jyk1mTIwlIiKSNcnXDO3cuRMFBQXo3LmzaZlOp8PGjRuxYMEClJSUILBSjUxISAhCQuxMejayN/l57lxDTyE2OREREfkFyQdD99xzD/bt22e2bOzYsWjdujVefPHFKoGQ0+xNfo6NZZMTERGRH5F8MFS3bl2kpKSYLatduzYiIiKqLHeJcVweW/OBcVweIiIivyL5nCGvccd8YERERCQ7sula7wqHuuZlZVUdlyc+nuPyEBEReRm71vsKRw8mIiJSFAZDlnBcHiIiIsVgzhAREREpGoMhIiIiUjQ2k8kR5xojIiJyGwZDcmOpt1tcnGFYAPZ2IyIichibyeQkKwsYNqzqHGpnzxqWZ2X5plxEREQyxmBILnQ6Q42QpWGhjMsyMgzbERERkd0YDMnFpk1Va4QqEgLIyzNsR0RERHZjMCQX58+7dzsiIiICwGBIPqKj3bsdERERAWAwJB+pqYZeY5UnkTVSqQxzqKWmerdcREREMsdgSC4CAw3d54GqAZHx9bx5HG+IiIjIQQyG5CQ9Hfj2WyA21nx5XJxhOccZIiIichgHXZSb9HRgyBCOQE1EROQmDIbkKDAQ6NvX16UgIiLyC2wmIyIiIkVjMERERESKxmCIiIiIFI3BEBERESkagyEiIiJSNAZDREREpGjK7Vqv03GsHiIiIlJoMJSVBUyaBJw5c3tZXJxhuguO4kxERKQoymsmy8oChg0zD4QA4OxZw/KsLN+Ui4iIiHxCWcGQTmeoERKi6jrjsowMw3ZERESkCMoKhrZsqVojVJEQQF6eIZfInXQ6ICcHWLLE8JvBFhERkWQoK2coP9++7c6fd99nMj+JiIhI0pRVMxQVZd920dHu+TzmJxEREUmesoKhu+4y1MqoVJbXq1RAfLyhm72rmJ9EREQkC8oKhgIDDc1TQNWAyPh63jz3jDe0aZNv8pOIiIjIIcoKhgBDns633wKxsebL4+IMy92Vx2Nv3pE785OIiIjIYcpKoDZKTweGDPHsCNT25h25Kz+JiIiInKLMYAgwBD59+3pu/6mphtqms2ct5w2pVIb17shPIiIiIqcpr5nMW7yZn0REREROYzDkSd7KTyIiIiKnKbeZzFu8kZ9ERERETmMw5A2ezk8iIiIip0m+mWzWrFno2rUr6tati8aNG2Po0KE4cuSIr4slTZwDjYiIyGGSD4Y2bNiA8ePHY9u2bVi7di3Ky8vRv39/FBUV+bpo0pKVBSQkAGlpwMiRht8JCZzyg4iIyAaVEJb6fUvXxYsX0bhxY2zYsAG9e/e26z2FhYUICwuDVqtFvXr1PFxCHzDOgVb5VBp7rTFZm4iIZMhbz2/J1wxVptVqAQDh4eFWtykpKUFhYaHZj9/iHGhEREQukVUwJITAlClT0KtXL6SkpFjdbtasWQgLCzP9xMfHe7GUXsY50IiIiFwiq2BowoQJ2Lt3L5YsWVLtdtOmTYNWqzX95OXleamEPsA50IiIiFwim671zz33HFauXImNGzciLi6u2m1DQkIQEhLipZL5GOdAIyIiconka4aEEJgwYQKysrKwfv16JCYm+rpI0mKcA63ylB9GKhUQH8850IiIiKyQfDA0fvx4LF68GJmZmahbty7y8/ORn5+PW7du+bpo0sA50IiIiFwi+WDoww8/hFarRd++fREdHW36Wbp0qa+LJh2cA42IiMhpks8ZktkwSL7DOdCIiIicIvlgiBzAOdCIiIgcJvlmMiIiIiJPYjBEREREisZgiIiIiBSNwRAREREpGoMhIiIiUjQGQ0RERKRoDIaIiIhI0RgMERERkaIxGCIiIiJFYzBEREREisZgiIiIiBSNwRAREREpGoMhIiIiUjQGQ0RERKRoDIaIiIhI0RgMERERkaIxGCIiIiJFYzBEREREisZgiIiIiBSNwRAREREpGoMhIiIiUjQGQ0RERKRoDIaIiIhI0RgMERERkaIxGCIiIiJFYzBEREREisZgiIiIiBSNwRAREREpGoMhIiIiUjQGQ0RERKRoDIaIiIhI0RgMERERkaIxGCIiIiJFYzBEREREisZgiIiIiBSNwRAREREpGoMhIiIiUjTZBEMffPABEhMTERoais6dO2PTpk2+LhIRERH5AVkEQ0uXLkVGRgZeeukl7N69G6mpqRg4cCBOnz7t66IRERGRzKmEEMLXhbDlzjvvRKdOnfDhhx+alrVp0wZDhw7FrFmzbL6/sLAQYWFh0Gq1qFevnieLSkRERG7iree35GuGSktLsXPnTvTv399sef/+/bFlyxYflYqIiIj8RZCvC2DLpUuXoNPpEBkZabY8MjIS+fn5Ft9TUlKCkpIS02utVgvAEGESERGRPBif255uxJJ8MGSkUqnMXgshqiwzmjVrFmbMmFFleXx8vEfKRkRERJ5z+fJlhIWFeWz/kg+GGjZsiMDAwCq1QAUFBVVqi4ymTZuGKVOmmF5fu3YNTZs2xenTpz36ZUpNYWEh4uPjkZeXp6hcKR43j1sJeNw8biXQarVo0qQJwsPDPfo5kg+GgoOD0blzZ6xduxYPPPCAafnatWsxZMgQi+8JCQlBSEhIleVhYWGKuoiM6tWrx+NWEB63svC4lUWpxx0Q4NkUZ8kHQwAwZcoUjBo1Cl26dEGPHj2wcOFCnD59Gs8884yvi0ZEREQyJ4tg6KGHHsLly5fx2muv4fz580hJScEPP/yApk2b+rpoREREJHOyCIYAYNy4cRg3bpxT7w0JCcGrr75qsenMn/G4edxKwOPmcSsBj9uzxy2LQReJiIiIPEXygy4SEREReRKDISIiIlI0BkNERESkaAyGiIiISNFkGQx98MEHSExMRGhoKDp37oxNmzZVu/2GDRvQuXNnhIaGolmzZvjoo4+qbLNs2TIkJSUhJCQESUlJWL58uaeK7zRHjjsrKwv9+vVDo0aNUK9ePfTo0QNr1qwx22bRokVQqVRVfoqLiz19KA5x5LhzcnIsHtPhw4fNtvO38z1mzBiLx52cnGzaRg7ne+PGjVCr1YiJiYFKpcKKFStsvscf7m9Hj9tf7m9Hj9tf7m9Hj9tf7u9Zs2aha9euqFu3Lho3boyhQ4fiyJEjNt/njXtcdsHQ0qVLkZGRgZdeegm7d+9GamoqBg4ciNOnT1vcPjc3F/fffz9SU1Oxe/du/POf/8TEiROxbNky0zZbt27FQw89hFGjRuH333/HqFGjMHz4cPz666/eOiybHD3ujRs3ol+/fvjhhx+wc+dOpKWlQa1WY/fu3Wbb1atXD+fPnzf7CQ0N9cYh2cXR4zY6cuSI2TG1bNnStM4fz/f8+fPNjjcvLw/h4eF48MEHzbaT+vkuKipC+/btsWDBAru295f729Hj9pf729HjNpL7/e3ocfvL/b1hwwaMHz8e27Ztw9q1a1FeXo7+/fujqKjI6nu8do8LmenWrZt45plnzJa1bt1aTJ061eL2//jHP0Tr1q3Nlj399NOie/fuptfDhw8X9913n9k2AwYMEA8//LCbSu06R4/bkqSkJDFjxgzT688++0yEhYW5q4ge4ehxZ2dnCwDi6tWrVvephPO9fPlyoVKpxMmTJ03L5HC+KwIgli9fXu02/nJ/V2TPcVsix/u7InuO21/u74qcOd/+cH8LIURBQYEAIDZs2GB1G2/d47KqGSotLcXOnTvRv39/s+X9+/fHli1bLL5n69atVbYfMGAAduzYgbKysmq3sbZPb3PmuCvT6/W4fv16lcnubty4gaZNmyIuLg6DBw+u8pelL7ly3B07dkR0dDTuueceZGdnm61Twvn+9NNPce+991YZpV3K59sZ/nB/u4Mc729XyPn+dgd/ub+1Wi0AVDsJq7fucVkFQ5cuXYJOp6syW31kZGSVWe2N8vPzLW5fXl6OS5cuVbuNtX16mzPHXdk777yDoqIiDB8+3LSsdevWWLRoEVauXIklS5YgNDQUPXv2xLFjx9xafmc5c9zR0dFYuHAhli1bhqysLLRq1Qr33HMPNm7caNrG38/3+fPn8eOPP+LJJ580Wy718+0Mf7i/3UGO97cz/OH+dpW/3N9CCEyZMgW9evVCSkqK1e28dY/LZjqOilQqldlrIUSVZba2r7zc0X36grNlXLJkCaZPn47vvvsOjRs3Ni3v3r07unfvbnrds2dPdOrUCe+99x7effdd9xXcRY4cd6tWrdCqVSvT6x49eiAvLw9vv/02evfu7dQ+fcXZMi5atAj169fH0KFDzZbL5Xw7yl/ub2fJ/f52hD/d387yl/t7woQJ2Lt3L3755Reb23rjHpdVzVDDhg0RGBhYJdorKCioEhUaRUVFWdw+KCgIERER1W5jbZ/e5sxxGy1duhRPPPEEvv76a9x7773VbhsQEICuXbtK5i8JV467ou7du5sdkz+fbyEE/vvf/2LUqFEIDg6udlupnW9n+MP97Qo539/uIrf72xX+cn8/99xzWLlyJbKzsxEXF1fttt66x2UVDAUHB6Nz585Yu3at2fK1a9firrvusvieHj16VNn+p59+QpcuXVCjRo1qt7G2T29z5rgBw1+MY8aMQWZmJgYNGmTzc4QQ2LNnD6Kjo10uszs4e9yV7d692+yY/PV8A4beGsePH8cTTzxh83Okdr6d4Q/3t7Pkfn+7i9zub1fI/f4WQmDChAnIysrC+vXrkZiYaPM9XrvH7U61loivvvpK1KhRQ3z66afi4MGDIiMjQ9SuXduUVT916lQxatQo0/Z//PGHqFWrlpg8ebI4ePCg+PTTT0WNGjXEt99+a9pm8+bNIjAwUPz73/8Whw4dEv/+979FUFCQ2LZtm9ePzxpHjzszM1MEBQWJ999/X5w/f970c+3aNdM206dPF6tXrxYnTpwQu3fvFmPHjhVBQUHi119/9frxWePocc+dO1csX75cHD16VOzfv19MnTpVABDLli0zbeOP59vo0UcfFXfeeafFfcrhfF+/fl3s3r1b7N69WwAQc+bMEbt37xanTp0SQvjv/e3ocfvL/e3ocfvL/e3ocRvJ/f5+9tlnRVhYmMjJyTG7bm/evGnaxlf3uOyCISGEeP/990XTpk1FcHCw6NSpk1m3vNGjR4s+ffqYbZ+TkyM6duwogoODRUJCgvjwww+r7PObb74RrVq1EjVq1BCtW7c2u7mkwpHj7tOnjwBQ5Wf06NGmbTIyMkSTJk1EcHCwaNSokejfv7/YsmWLF4/IPo4c91tvvSWaN28uQkNDRYMGDUSvXr3E999/X2Wf/na+hRDi2rVrombNmmLhwoUW9yeH823sOm3tuvXX+9vR4/aX+9vR4/aX+9uZ69wf7m9LxwxAfPbZZ6ZtfHWPq/4sIBEREZEiySpniIiIiMjdGAwRERGRojEYIiIiIkVjMERERESKxmCIiIiIFI3BEBERESkagyEiIiJSNAZDREREpGgMhoiIiEjRGAwRERGRojEYIiJZWrJkCUJDQ3H27FnTsieffBLt2rWDVqv1YcmISG44NxkRyZIQAh06dEBqaioWLFiAGTNm4JNPPsG2bdsQGxvr6+IRkYwE+boARETOUKlUmDlzJoYNG4aYmBjMnz8fmzZtYiBERA5jzRARyVqnTp1w4MAB/PTTT+jTp4+vi0NEMsScISKSrTVr1uDw4cPQ6XSIjIz0dXGISKZYM0REsrRr1y707dsX77//Pr766ivUqlUL33zzja+LRUQyxJwhIpKdkydPYtCgQZg6dSpGjRqFpKQkdO3aFTt37kTnzp19XTwikhnWDBGRrFy5cgU9e/ZE79698fHHH5uWDxkyBCUlJVi9erUPS0dEcsRgiIiIiBSNCdRERESkaAyGiIiISNEYDBEREZGiMRgiIiIiRWMwRERERIrGYIiIiIgUjcEQERERKRqDISIiIlI0BkNERESkaAyGiIiISNEYDBEREZGiMRgiIiIiRft/2Wqj00ldaBYAAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "# Using Autograd to calculate gradients for OLS\n",
- "from random import random, seed\n",
- "import numpy as np\n",
- "import autograd.numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "from autograd import grad\n",
- "\n",
- "def CostOLS(beta):\n",
- " return (1.0/n)*np.sum((y-X @ beta)**2)\n",
- "\n",
- "n = 100\n",
- "x = 2*np.random.rand(n,1)\n",
- "y = 4+3*x+np.random.randn(n,1)\n",
- "\n",
- "X = np.c_[np.ones((n,1)), x]\n",
- "XT_X = X.T @ X\n",
- "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
- "print(\"Own inversion\")\n",
- "print(theta_linreg)\n",
- "# Hessian matrix\n",
- "H = (2.0/n)* XT_X\n",
- "EigValues, EigVectors = np.linalg.eig(H)\n",
- "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
- "\n",
- "theta = np.random.randn(2,1)\n",
- "eta = 1.0/np.max(EigValues)\n",
- "Niterations = 1000\n",
- "# define the gradient\n",
- "training_gradient = grad(CostOLS)\n",
- "\n",
- "for iter in range(Niterations):\n",
- " gradients = training_gradient(theta)\n",
- " theta -= eta*gradients\n",
- "print(\"theta from own gd\")\n",
- "print(theta)\n",
- "\n",
- "xnew = np.array([[0],[2]])\n",
- "Xnew = np.c_[np.ones((2,1)), xnew]\n",
- "ypredict = Xnew.dot(theta)\n",
- "ypredict2 = Xnew.dot(theta_linreg)\n",
- "\n",
- "plt.plot(xnew, ypredict, \"r-\")\n",
- "plt.plot(xnew, ypredict2, \"b-\")\n",
- "plt.plot(x, y ,'ro')\n",
- "plt.axis([0,2.0,0, 15.0])\n",
- "plt.xlabel(r'$x$')\n",
- "plt.ylabel(r'$y$')\n",
- "plt.title(r'Random numbers ')\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "623ad923",
- "metadata": {},
- "source": [
- "## Same code but now with momentum gradient descent"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "b7fe53c1",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Own inversion\n",
- "[[4.]\n",
- " [3.]]\n",
- "Eigenvalues of Hessian Matrix:[0.28076442 4.37589157]\n",
- "0 [-10.1056039] [-11.51537034]\n",
- "1 [-0.16830135] [0.14316669]\n",
- "2 [-0.15750286] [0.13398088]\n",
- "3 [-0.14739722] [0.12538445]\n",
- "4 [-0.13793997] [0.11733958]\n",
- "5 [-0.12908951] [0.10981087]\n",
- "6 [-0.12080692] [0.10276523]\n",
- "7 [-0.11305575] [0.09617164]\n",
- "8 [-0.1058019] [0.09000111]\n",
- "9 [-0.09901348] [0.08422649]\n",
- "10 [-0.09266061] [0.07882238]\n",
- "11 [-0.08671535] [0.07376501]\n",
- "12 [-0.08115155] [0.06903212]\n",
- "13 [-0.07594474] [0.06460291]\n",
- "14 [-0.071072] [0.06045788]\n",
- "15 [-0.0665119] [0.0565788]\n",
- "16 [-0.06224439] [0.05294861]\n",
- "17 [-0.05825068] [0.04955134]\n",
- "18 [-0.05451322] [0.04637205]\n",
- "19 [-0.05101557] [0.04339674]\n",
- "20 [-0.04774232] [0.04061233]\n",
- "21 [-0.0446791] [0.03800658]\n",
- "22 [-0.04181241] [0.03556801]\n",
- "23 [-0.03912966] [0.03328591]\n",
- "24 [-0.03661904] [0.03115023]\n",
- "25 [-0.0342695] [0.02915158]\n",
- "26 [-0.03207071] [0.02728117]\n",
- "27 [-0.030013] [0.02553076]\n",
- "28 [-0.02808732] [0.02389267]\n",
- "29 [-0.02628519] [0.02235967]\n",
- "theta from own gd\n",
- "[[3.91238674]\n",
- " [3.07452882]]\n",
- "0 [-0.02459869] [0.02092504]\n",
- "1 [-0.0230204] [0.01958246]\n",
- "2 [-0.02106988] [0.01792324]\n",
- "3 [-0.01913285] [0.01627549]\n",
- "4 [-0.01732414] [0.0147369]\n",
- "5 [-0.01566999] [0.01332978]\n",
- "6 [-0.01416833] [0.01205238]\n",
- "7 [-0.01280877] [0.01089586]\n",
- "8 [-0.01157907] [0.00984981]\n",
- "9 [-0.01046722] [0.00890402]\n",
- "10 [-0.00946208] [0.00804898]\n",
- "11 [-0.00855343] [0.00727603]\n",
- "12 [-0.00773203] [0.00657731]\n",
- "13 [-0.00698952] [0.00594568]\n",
- "14 [-0.0063183] [0.00537471]\n",
- "15 [-0.00571154] [0.00485856]\n",
- "16 [-0.00516305] [0.00439199]\n",
- "17 [-0.00466724] [0.00397022]\n",
- "18 [-0.00421904] [0.00358895]\n",
- "19 [-0.00381387] [0.0032443]\n",
- "20 [-0.00344762] [0.00293274]\n",
- "21 [-0.00311654] [0.00265111]\n",
- "22 [-0.00281725] [0.00239652]\n",
- "23 [-0.00254671] [0.00216637]\n",
- "24 [-0.00230214] [0.00195833]\n",
- "25 [-0.00208106] [0.00177027]\n",
- "26 [-0.00188122] [0.00160027]\n",
- "27 [-0.00170056] [0.00144659]\n",
- "28 [-0.00153725] [0.00130767]\n",
- "29 [-0.00138963] [0.0011821]\n",
- "theta from own gd wth momentum\n",
- "[[3.99552586]\n",
- " [3.00380596]]\n"
- ]
- }
- ],
- "source": [
- "# Using Autograd to calculate gradients for OLS\n",
- "from random import random, seed\n",
- "import numpy as np\n",
- "import autograd.numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "from autograd import grad\n",
- "\n",
- "def CostOLS(beta):\n",
- " return (1.0/n)*np.sum((y-X @ beta)**2)\n",
- "\n",
- "n = 100\n",
- "x = 2*np.random.rand(n,1)\n",
- "y = 4+3*x#+np.random.randn(n,1)\n",
- "\n",
- "X = np.c_[np.ones((n,1)), x]\n",
- "XT_X = X.T @ X\n",
- "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
- "print(\"Own inversion\")\n",
- "print(theta_linreg)\n",
- "# Hessian matrix\n",
- "H = (2.0/n)* XT_X\n",
- "EigValues, EigVectors = np.linalg.eig(H)\n",
- "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
- "\n",
- "theta = np.random.randn(2,1)\n",
- "eta = 1.0/np.max(EigValues)\n",
- "Niterations = 30\n",
- "\n",
- "# define the gradient\n",
- "training_gradient = grad(CostOLS)\n",
- "\n",
- "for iter in range(Niterations):\n",
- " gradients = training_gradient(theta)\n",
- " theta -= eta*gradients\n",
- " print(iter,gradients[0],gradients[1])\n",
- "print(\"theta from own gd\")\n",
- "print(theta)\n",
- "\n",
- "# Now improve with momentum gradient descent\n",
- "change = 0.0\n",
- "delta_momentum = 0.3\n",
- "for iter in range(Niterations):\n",
- " # calculate gradient\n",
- " gradients = training_gradient(theta)\n",
- " # calculate update\n",
- " new_change = eta*gradients+delta_momentum*change\n",
- " # take a step\n",
- " theta -= new_change\n",
- " # save the change\n",
- " change = new_change\n",
- " print(iter,gradients[0],gradients[1])\n",
- "print(\"theta from own gd wth momentum\")\n",
- "print(theta)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "749000b3",
- "metadata": {},
- "source": [
- "## But noen of these can compete with Newton's method"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "e7b4abe1",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Using Newton's method\n",
- "from random import random, seed\n",
- "import numpy as np\n",
- "import autograd.numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "from autograd import grad\n",
- "\n",
- "def CostOLS(beta):\n",
- " return (1.0/n)*np.sum((y-X @ beta)**2)\n",
- "\n",
- "n = 100\n",
- "x = 2*np.random.rand(n,1)\n",
- "y = 4+3*x+np.random.randn(n,1)\n",
- "\n",
- "X = np.c_[np.ones((n,1)), x]\n",
- "XT_X = X.T @ X\n",
- "beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
- "print(\"Own inversion\")\n",
- "print(beta_linreg)\n",
- "# Hessian matrix\n",
- "H = (2.0/n)* XT_X\n",
- "# Note that here the Hessian does not depend on the parameters beta\n",
- "invH = np.linalg.pinv(H)\n",
- "EigValues, EigVectors = np.linalg.eig(H)\n",
- "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
- "\n",
- "beta = np.random.randn(2,1)\n",
- "Niterations = 5\n",
- "\n",
- "# define the gradient\n",
- "training_gradient = grad(CostOLS)\n",
- "\n",
- "for iter in range(Niterations):\n",
- " gradients = training_gradient(beta)\n",
- " beta -= invH @ gradients\n",
- " print(iter,gradients[0],gradients[1])\n",
- "print(\"beta from own Newton code\")\n",
- "print(beta)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d47c2b1c",
- "metadata": {},
- "source": [
- "## Including Stochastic Gradient Descent with Autograd\n",
- "In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "e2b04ba9",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Own inversion\n",
- "[[3.6259946 ]\n",
- " [3.13054971]]\n",
- "Eigenvalues of Hessian Matrix:[0.31026823 4.41529307]\n",
- "theta from own gd\n",
- "[[3.6259946 ]\n",
- " [3.13054971]]\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsnklEQVR4nO3deXxcdb3/8dcn6ZoWSpsW6EJSCqW0CAItq4BsKQgoXvlxBSOCgmVxQeUnolVckHsRvSJXL2JBBW9zUQSVK6I2QFootJQUCt0oBdq0dKf7viSf+8eZlGk6y0lmOTOT9/PxmEdmOXPO55yZfD9zvtsxd0dERCSdsqgDEBGR4qCEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGFIyTKz75nZxKjjyCYzO9vM3o06DumclDAkr8xssZltN7MtZrbSzB4ys95RxyUi6SlhSBQ+6u69geOBE4BvRhtO52RmXaKOQYqLEoZExt1XAv8kSBwAmNltZva2mW02s3lm9i9xr11jZlPN7Cdmtt7MFpnZR+JeP9zMpsTeWw/0j9+emX3MzOaa2QYzm2xmI+NeW2xmXzez181sq5n92swOMbO/x9b3tJn1TbQfrdVEZnaLma02sxVm9tm41yeb2XVt9yPusZvZTWa2MLatO8zsCDN70cw2mdmjZtatzTa/ZWbvxeKujXu+e+z4LDGzVWZ2v5n1bBPnN8xsJfBbM+tvZk/Gjsk6M3vezFQuSEL6YkhkzGwI8BHgrbin3wbOBPoA3wcmmtnAuNdPARYQJIO7gV+bmcVe+x9gZuy1O4Cr47Z1FPAI8BVgAPAU8Nc2BfFlQA1wFPBR4O/At2LLlwFfTrE7h8ZiHgxcC/xXsgSTxAXAaOBU4FZgAvBp4DDgA8CVbbbVP7atq4EJZjYi9tpdsfiPB46MLXN7m/f2A6qBccAtwLuxfTwktr+aL0gSUsKQKPzFzDYDS4HVwHdbX3D3P7r7cndvcfc/AAuBk+Pe2+TuD7h7M/AwMBA4xMyqgJOA77j7Tnd/Dvhr3Ps+CfzN3evdfTfwE6AncHrcMj9391Xuvgx4HnjJ3V919x3Anwmqz5LZDfzA3Xe7+1PAFmBEiuXbutvdN7n7XGAOMMnd33H3jQSJq+22W/dzCvA34F9jiXMc8FV3X+fum4F/A66Ie18L8N3Ye7fH4h4IVMdif941wZwkoYQhUfi4ux8AnA0cTVzVkZl9xsxmxapINhD8uo6vWlrZesfdt8Xu9gYGAevdfWvcsk1x9wfFP3b3FoKENThumVVx97cneJyqcX6tu++Je7wtzfJttWfbifZzEMFZQgUwM+74/SP2fKs1sQTY6scEZ3iTzOwdM7utHTFLJ6OEIZGJ/Tp+iODXPmZWDTwAfBGodPeDCH5tW5JVxFsB9DWzXnHPVcXdX05QDUNsW0ZQ3bOs43sQ2laCgrzVoRmuL9F+LgfeI0gux7j7QbFbn1gHg1b7nD24+2Z3v8XdhwEfA75mZudlGJ+UKCUMidrPgBoz+yDQi6BAWwMQazj+QJiVuHsT0Ah838y6mdkZBO0QrR4FLjaz88ysK0Hd/U7gxWztSAqzgE+YWYWZHUnQxpGp1v08E7gE+GPsrOkB4B4zOxjAzAab2QXJVmJml5jZkbEEuhFoJqi2EtmPEoZEyt3XAL8Dbnf3ecB/ANMIqmSOBV5ox+o+RdAovo6gXeR3cdtZQNCI/HOCX+IfJejeuysLu5HOPcAugn16GKjLcH0rgfUEZxV1wA3u/kbstW8QVDFNN7NNwNOkbksZHltmC8Fxv8/dGzKMT0qUqX1LRETC0BmGiIiEooQhIiKhKGGIiEgoShgiIhJKUU0+1r9/fx86dGjUYYiIFJWZM2e+5+4D0i+ZWlEljKFDh9LY2Bh1GCIiRcXMmtIvlZ6qpEREJBQlDBERCUUJQ0REQlHCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwREQkl5wnDzH5jZqvNbE6C124xMzez/rmOQ0REMpOPM4yHgAvbPmlmhwFjgSV5iEFERDKU84Th7s8RXGO5rXuAWwFdI1ZEpAhE0oZhZpcCy9z9tRDLjjOzRjNrXLNmTR6iExGRRPKeMMysAvgWcHuY5d19gruPcfcxAwZkPJ27iIh0UBRnGEcAhwOvmdliYAjwipkdGkEsIiISUt4voOTus4GDWx/HksYYd38v37GIiEh4+ehW+wgwDRhhZu+a2bW53qaIiGRfzs8w3P3KNK8PzXUMIiKSOY30FhGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwREQlFCUNEREJRwhARkVCUMEREJBQlDBERCUUJQ0REQlHCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwREQlFCUNERELJecIws9+Y2WozmxP33I/N7A0ze93M/mxmB+U6DhERyUw+zjAeAi5s81w98AF3Pw54E/hmHuIQEZEM5DxhuPtzwLo2z01y9z2xh9OBIbmOQ0REMlMIbRifA/6e7EUzG2dmjWbWuGbNmjyGJSIi8SJNGGY2HtgD1CVbxt0nuPsYdx8zYMCA/AUnIiL76BLVhs3sGuAS4Dx396jiEBGRcCJJGGZ2IXAr8GF33xZFDCIi0j756Fb7CDANGGFm75rZtcAvgAOAejObZWb35zoOERHJTM7PMNz9ygRP/zrX2xURkewqhF5SIiKBujoYOhTKyoK/dUn7w0gEImv0FhHZR10djBsH22LNmk1NwWOA2tro4pK9dIYhIoVh/Pj3k0WrbduC56UgKGGISGFYsqR9z0veKWGISGGoqmrf85J3ShgiUhjuvBMqKvZ9rqIieF4KghKGiBSG2lqYMAGqq8Es+Dthghq8C4h6SYlI4aitVYIoYDrDEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwRkWQ0e+4+lDBEpPjlomBvnT23qQnc3589txMnDSUMESluHSnYwyQYzZ67H3P3qGMIbcyYMd7Y2Bh1GCJSSIYODZJEW9XVsHjx/s+3ve4GBHNWtZ2GpKwsSEBtmUFLS6ZR55WZzXT3MZmuR2cYIlLc2jstetgzh0KaPbdA2lKUMESkfQqk8NqrvQV72ATT0dlzs318Cqktxd1zegN+A6wG5sQ91w+oBxbG/vYNs67Ro0e7iERo4kT3igr3oOgKbhUVwfPFElN19b7Ltt6qqxOvu7ra3Sz4m24/c3F82hNvEkCjZ6M8z8ZKUm4AzgJObJMw7gZui92/DfhRmHUpYYhELAuFV060p2DPZdLL9Pgk2g+zxOs0C72u42CnF0PC8CApDG2TMBYAA2P3BwILwqxHCUMkYh0tvApNe88cwgp7fBJtP1kiq6xsfxJqs67R4J6FsjwvvaTMbCjwpLt/IPZ4g7sfFLtvwPrWxwneOw4YB1BVVTW6KVFvCBHJj/b2SOpswhyfZL20evaEtWv3f29lJWzfnr5XV4o4xgCN7tbOvdlP5I3eHmSspFnL3Se4+xh3HzNgwIA8RiYi+9FlVFMLc3yS9dJKlCwA1q0LfSVCb3GW33oPnqMf1lFdcW+VmQ109xVmNpCgUVxECl1rITV+fNCrqKoqKAx1lbxAmOOTrJdWMlVVKa9EuHbhOp657w0mPdVMxcJZ/Lt/m4xPJZKIqkrqx8Bad7/LzG4D+rn7renWo4F7IlL0klVbhax62rlpJy8+OI/6xzYy6bWDeWXb0Thl9GEjb5YfzcHNK/dbddFUSZnZI8A0YISZvWtm1wJ3ATVmthA4P/ZYREpVoY3diFKyaqt7701Y9eRXfoq5T7zFPR+fzEUHv0y/Pns495YT+PG0D9Gzy26+d85zTHtgDu9t78XBLatyG3s2Ws7zdVMvKZEiVIhjN3ItXS+sNK+vnL3aJ9441T8z7HkfVLZ872Eb0e1t/+Kxk/2Jb033jUs37r/dJN16s9WtVnNJiUhudbaeVWHnqoqzfd12pk6Yx6THN1M/ZyCv7RgBQD9bx/lD3mDsuc3UXD+MqtMGp972TTfB/ffvOwdWRQUDtm1btMZ9WKa7poQhIrlVQpP47aeubv8G7vHj0ybIlj0tvP74Qup/t4JJ0w/g+XXHsJMedGUXZxw0l5qTN1JTezAnXDGC8m7l4WNpm6jM4IYbsF/+MiuTDyphiEhuleoZRrIzibZdZmPcjIc/9xz1Txv1S45ijQfDBI7pvpCxxyyj5tJenHXDKHod3Ktj8aQ4ztbUlJWEEVW3WhHpLO68M3HBWuxjN5KNpygvh+bm/RZv9jIm/bqJZ2wsY6vfpOb8BZx//REMHjMcGJ55PO2dtbcDIh+4JyIFJBe9mWprQw8865CoemAlKYi9uZkd9Njv+S40M7Hbdax4+J9MXPQhrn7gDAaPGZi9ePIxHXs2Ws7zdVMvKZEcKsbeTLmMOV1PpyQ9khZR7bd1vdv3UJbw9ZxN1JjiWJCl2WrVhiEigf79E09PUchtDblqH0nSPrHtrnupX3Es9U/uoNvcV7mjZTy9eH+ZPV26s/WHP6PPN26IprE/USN8bW3WrrinhCEiQUHz6U8nfq2QezPlqlBOkoiaqGIoTVSwlbMHzOOLVU9w3pLf0vW9FVjbaUAKqLFfl2gVkexpe3nSeFFckjSsHNTbv/1sE96UuH2iiqU03DOLdRu78LfVJ/GRxh/SbfUyrKUlSALx7TIlOFGjEoaIpO5JE1UB17Yx+6ab9m/czkKhvH7RBh7/+jSuH/kcw7ou4cjzqmkiccKx6irO/srxdD+we/oV56KxP+opVrLREJKvmxq9RXIk2ZXiKiujiSdRA27bW2vjdjsvhrRz806f8p+z/NtnNPjJvWZ7GXsc3A9go1966HT/xeWTffk37vGWQusAkEEDP8VyidZs3pQwRHKk0HpIJUtgHehx1NLc4vP/9rbf+4nJfsnBL3lvNjm4l7HHT+v9ut9+VoNPve8137V1175vzNVV+Toqg8u/ZithqNFbpNgk6QlTsOvtiGSN2W0ladxeM/89nvnlm9T/o5lJ7xzBu82DADiiSxNjRzRRc3E3zrlpJAdV98l25LmTQQN/thq9Iz9raM9NZxjS6RXamUAqYX+hJ1qunWcYOzbu8Gd+8op/45QGP7HnvL0vH2Tr/bLBL/qvaqf4O1OW5GW3c6YAzjAiTwLtuSlhSKeXQaGRV2ETW7LlbrwxbRtGc7ce/uSJ4/2Cype9J1sd3Luwy8/q86rfcV6DT39wtu/ZuSe7+xRlFZXaMJQwRNrFLHEBahZ1ZIF0ZwdtE1uqBNimgN5y+TW+ufch3oL5Eob4lUx0cD+621v+peMm+1+/85JvWrYpd/tVCGd2HUxa2UoYasMQKSYFNBhsP4lGR7fVtr49Rb38ttVbeP5X86j/8xYmzR3E7B1HAVBpa6k5bAE15zZTc8MRHHbKoCzvSAKFfNxD0MA9kVITpo99IQ8GSzR7a1ttB9QlGWC3yg6h34AyLvz2GH4+8zQG9NzCXRdOZubE+aze1ZdHmk7nc789M3GyyMVYhUxmgo167EQ2ZeM0JV83VUlJyWpPlUfUdenJJKsuS7E/7/3gF767vPs+y22hwm/t8hP/2ugG//sdL/vWNVvDx5CrqqOOth0VSFUWasMQKSGF0Jjd0etQh+nVFFt+84rN/tfvvORf/uBkH9ntLQf3K5noSxjiLZhv7n2Ir7vzvzq+D7k6jh0t+Avhc/USSRjAV4G5wBzgEaBHquWVMKRkRd2Yna5A7GBvppaKCl/4+bv8jvMa/Kw+r3pXdjq492SrX1D5sv/kkgZ//bEF3tLcknn8qZJW63EMc3aWLjG258wu6s81pugTBjAYWAT0jD1+FLgm1XuUMKRkRf1LNN32k71eXp44UYCvLj/Er+NXe58+oec8/8YpDf703TN9+/rt2Ys9zDQirQV8urOEbFchRf25xpRKwlgK9CO4VOyTwNhU71HCkJIVdV13ul/C6don2tyaMR9Svsw/O/w5f+RLL/jqeWtyF3u66rDW4xim8M52AR/15xpT9Akj2AduBrYAa4C6JMuMAxqBxqqqqqweRJGCEmVjdgfPMHYnuarczgGDMq9mCitVMisvf/84hqkeykUVUgF0UshWwoisW62Z9QUuBQ4HBgG9zGy/K7i4+wR3H+PuYwYMGJDvMEXCyUbXydraoE9/omsr5FqK7rre4qy44ivsLt93Su+tVPDn8svZVdZ9v/d1u+durMxyHHRMqmtftLS8fxzDXDsj3TId+Zyj/FyzLRtZpyM34HLg13GPPwPcl+o9qpKSglQg1Q4Zi/slvGfQEJ924Xf980dP8erypQ5Bb6alDPYWzLcccIhv/umv9ntfZFNmJDsziK9KyrQNo4g/Z4q9Sgo4haCHVAVgwMPAl1K9RwlDClKBNGxmYufmnT75Z6/6t05v8JN6zXGj2cH9QDb4xwdO8/uumOILn16cv2qm9rrxxv2TRrK5qzraS6qIP+dsJYxIpwYxs+8DnwT2AK8C17n7zmTLa2oQKUi5uq50DnmLM//Jt6l/aBmTplYwZc1IttKbcvZw6gHzqBm9jporKjn56pF06dEl6nDDyfX07EX4ObfK1tQgmktKJJUwhVCRzDO0eu4anr7vTeontVD/zpEsaxkIwPCui6gZsYSxH+3B2TccTZ+qPhFHWqCK5HNOJG/XwwDqgQ9m43Qm05uqpCSvMp2iO+K67e3rt3v9j2b6rSc3+PE95+8Nra+t88uHvOgTrprii55fmt2NRt2ekUsF+jmHQb7aMIATgQbgt8DAbGy0ozclDMlIewuz9tRZF0BB2dLc4q/9cYH/+OIGH1v5svdgm4N7V3b6h/u86nfWNPiMh+Zm9xoR8Yq4QA2tAD7njshWwghdJWVmlwG3A38C7nb37Rmf3rSTqqSkQ+rq4OabYe3afZ+vqIAJE5LXcxdBnfXyV1by9P1vManeeHrJcFa1HAzAqO5vUTNyGTWXVvDhG0bS+9DeuQ+miKtsSl1e2zDMzIBjgDOAHwI7gG+6+39nGkB7KGFIu6W7RkOqwqwAC8Ctq7fy3P3zqH9iK/VzBzNn53AABtgaaqrepOZ85/zrj2DISQPzH1wRJNjOKlsJI233BzN7gWBw3VxgOnAN8AZws5md6e7jMg1CJGfSXaMh1fUM7rxz/2ST52tPtOxp4dXfL6B+4iomvdSHFzaMYhcn0Z0dnNlvLledO5mxVw/kuMuGU9blQ3mLK6GqqsQJNtXAOikqYfrLjQPm+f6nIl8ys/k5iEkke9Jd4CZVYdZaVZXLrpoJLJm2jPpfvUP9s+U8/e4I1vpIYCTH9VjAl8dMo+YTB3Dm9aPo2W90TuNotwJIsJJbaROGu89N8fLFWYxFJPuS/eqFcIVZbW3OE8Tm5ZuZ/Mv5THpiO/ULDmPBrmHAYAaWreTiw+cz9gLj/JuO4pAPjABG5DSWjESUYCV/NA6jkOV6IFJnkKwNo7IS7r03kuO5Z8ceGie+Qf0j7zGpsS/TN41iD13pyTY+3H8uYz+0lZqrB3PMpUfmbz4mKWl5G4dRSLdO1a22M3RRzJcC6Ar5dkOT//LKKf6JQdP8IFsfTIBKs4+umOu3ndrgz/7HK77rgYcijzOvCuBz6Swo9rmkOnLrVAmjiOetEff1izf441+f5jeMmuLDuize+/EdVv6uX3vUc/7iBd/1PYOGvF9YJrpyXbZ/IBRSAa0fRHmVrYShKqlCpS6KRWX3tt289NB8Jv1+HfWvVjJjyyhaKKc3mznnkHnUnLGDsdcexlEXHI498j/7V5OZJf68s9WFN1HVXLpxKLlUgF2WS5nmkip1+ocqaN7ivPnPRdT/ZimTnu/B5FUj2cyBlNHMyb3nUXPCWmr+tS+nfm4UXSu67vvmZJ9tItn6gVBo3yf9IMqrvI3DkIgUexfFEmywX7twHc/c9waTnmqm/u3DWdI8DBjGsC5NfGrkLMZe0o1zbjyavocfm3pF6br6xsvWGIZk22xPLNmkMRvFKRv1Wvm6dao2DPfCqnNujxKpn96xcYc/+x+v+DdPa/DRFXP3XiOiDxv8E4Om+S+vnOJvPbO4/StO1j4V5noOHVVobWIl8h0pFqjRu0QVa5KIV2iFU0gtzS0+5y8L/aeXNvhHBszwCrY4uHdhl59x4Cz/wbkNPu2B2b57++72rzz+c62sdO/adf/C8sYbc/fZF2IBXQrf9SKhhFGKCvGfuiOSXS7TLOrI9rNy9mqfeONU/8yw531Q2fK9oY7o9rZ/8djJ/sS3pvvGpRvDrzBRIZjoc+3WLUgc+SwsVUB3WtlKGGr0LiSF1jDZUQW8H9vXbef5X82j/k+bqZ8zkNd2BCOnK20t5w1ZwNhzm6m5fhhVpw1u/8qT9UTq2XP/mXIhd8ejBNuPJDMauFeKCvWXeXt/mRbQmVLz7mZ/9fdv+N0XNfj5/Rq9O9v3XiPinINe8X8b2+CN/z3Pm3c3Z76xZFVxyW65+FwL6NgXnRI+A0NVUiWoEOv+O1oARfjP9+7Ly/231z7vn6qe6gNs9d6wj+n+pn/1xAZ/6vszfMuqLdmPN1nCT3bLxedaiN+hYlDiiVYJoxQV4pc2kwIoT0ljy6ot/rfvzfCbj5/so7ov3Bviwbbaa4dO9Yeue97ffXl56jizcdyTHavKyvx9roV6llroSjzRlkTCAA4CHiO4vsZ84LRUy5d8wnAvvNPijhZAOUx+e3bu8RkPzfU7axr87INe8a7sdHDvwTav6dfoP764wWf94Y3w1UzZKixS7XO+PtcSL/hypsQTbakkjIeB62L3uwEHpVq+UySMjshlYdTRAqi970uzD4unLvUJV03xy4e86P1s7d7VHd9zvn/9pAav/9FM37Z2W8f2MZuFRdQJvxDPUotBiSfaok8YQB9gEbHpScLclDASyHUB0dH1t6cQTrCNlp49/eWPfc+/cOxkH971nb0vDSpb7tcc+ZzX3TTVV85enZ0CutQKi6iTVjEq8URbCgnjeGAG8BDwKvAg0CvBcuOARqCxqqoqy4exBCQr7MrLs1dgdKQAak8hnGTZRVR7BVv8ogEz/J6PT/a5Tyz0luaWfePKxj95iRcWElIJJ9pSSBhjgD3AKbHH9wJ3pHqPzjASCNMzJ4rCL00h3NLc4gufXuz3XTHFm0m8Dy2Y79i4I/k2snlmUMKFhUi2EkZZxgM5Ou5d4F13fyn2+DHgxAjjKU5hJmvbti0YyJUrdXXBYL2ysuBvXV0wUGzChGBwmhlUV7Plzp/x+KxhXD/yOY7ovpTh51dz0+/PYjmDEq7WqqvofmD35NvN5oR6tbXBILqWluCvBrqJ7Cey2WrdfaWZLTWzEe6+ADgPmBdVPEXrzjvhs5+F3btTL5erWUnbjm5uagoeA7suvZzp6z5A/aPrmfRqfxq/OpIWyjmATZx76HxuOXMRYz9fzeBVP4LrOzAzb7IZT/v1CxKXRjqLZFc2TlM6eiNox2gEXgf+AvRNtXxeqqRSVU0UYrXFxInBvETpqqVyVU2TpFpoVdmh3ptNQXMKu/203q/77Wc1+NT7XvNdW3d1bFttl6+s3H/b3bolntgv159VIX43RGIo9jaMjtxynjDS9aPPVgNrNguWMNNR5LAhuCVJG0oz5jceM8X/dOs0X794Q2b7GCY2CBJIoiSS6x5PajSXAqeEkQupGlGz0cCai4IlVaN3DrqatoDvrOjjm8sP9GbMd1Me7rhkM1Gm+iyiGIBVat1ypeQoYeRCqsImGwVRLgqWXBVW7ZgXqSXdGU3YRBk2qaT6LKIovEt8lLAUv2wljCh7SRWeZD2OqqpSvxZWLi6TeeedQQNxvAwv5bry9dVs6XVw6OUNoLx8b28oJkzYt5F5/Ph9G7Rh/55brY3nTU1BcdvaeF5Xt/8GU30WOTgeaWXjuyFSDLKRdfJ1K/o2jFz9+m1PdU+CZbeu2er/+OHL/rXRDX5sjwUO7lcy0bfSM/RZRspf02F+gbfn2CT6LFq3UV2d2yvXJaI2DClwqEoqR3LZSyrqgiXB9rfTw6/iIQf3buzwc/vO9LsubPCZE+d58+/+O/w1HlIlvTDJoL3VOq2fRXyyiLKwVi8pKWBKGJmI8p8719tOsv6lM5b75l6HJCyU13Y7xP/xw5d965qtydeZqFdS2AI6TKLM1ySHIp2QEkZHRf0rP5cSTeIHPtFqHTzpFByhGmfjE1Fr99X2JL10iTIfkxyKdFJKGB1Vor9I9+zc4zsqBybct2bwJ48f7zv7D8rfvrcdWFdZGW4wXrqzr7bLRDHuQqTIKGF0VAn9In1nyhL/Ve0U/3+DX/S+ti75GURrAZqvs6uJE/cfbQ3BKOxMtpUo/qhGdosUESWMjiriM4wNTRv8z7dN9xuPmeJHdl20N/Qh5cv8s8Of8y0HJG6j2CchZrMNJdm6UjWU52LMSWWlGpxFUshWwrBgXcVhzJgx3tjYmNlK2k6WB0E//bZjBwrA7m27mfHwfOr/sI5JMyuZsWUkzXShF1s45+B51JyxnbHXHsaICw/HyizYt6uuCorRtqqrg1lYsyXVcUwWAwRjNVpaOrbNsrLE681knSKdgJnNdPcxma4nstlqI9OaFMaPL7jZTL3FeeuZJiY90ET98z1oWHk0mziOMpoZ02s+t31oKmM/2ZdTPzuSbr1P3n8FtbXwwgtw//37Fqy5GLiWajBesllkIbPBbMnWqwFyIvmRjdOUfN1K8QJKa99a549+9UW/bsQUry5fureWZWiXJT7u6Cn+x6+96GvfWpd8BYmqhfLRbThVW1A+2zDUXiGSFmrDKE47N+/0yT971b91eoOf1GuOG80O7geywT8+cJrfd8UUX/j04n0vRZpMlAVouragjvSSCkMD5ETaLVsJo/O1YeSZtzjzn3yb+oeWMWlqBVPWjGQrvSlnD6ceMI+a0euouaKSk68eSZce7awhHDo0cRVNttsrEimitiCRzk5tGNlQV5eTtozVc9fw9H1vUj+phfp3jmRZy5HAkQzvuoirPzCTsR/twdk3HE2fquMy21AuJjMMq4DbgkQkNzrvGUYWfyHv2LCDqRPmUf/4JibNPpRZ248GoK+t5/zBb1Bzzm5qxg1j6BlDshN7qyjPMESkaGTrDKM0pzevqwsK07Ky4G+iKbLDTLmdhLc4rz26gJ9cMpkL+jfSt69T840TuWfG6fTptoM7ayYz46F5rNlxII8uPY3P/+6s7CcLiGYqbxHpvLLREJKvW6hG77ANwe0c8b1s5gp/+PPPe+3QqX5I2aq9i4/qvtBvPn6y/+17M3zzis37xxLfQJuLabfVCCwiaaBG7yTCVtOkWW7r6q08d/886p/YSv3cwczZORyAAbaGmqo3qTnfOf/6Ixhy0sD33xvfJtKvH2zaBLt3J49VjcQikgfZqpKKPGGYWTnQCCxz90tSLRsqYYQdDZygDaO5Ww+ePOoW7n33Ml7YMIpddKc7Oziz31xqTtnM2KsHctxlwynrkqAmL1GbSBhqbxCRHCulNoybgflZW1vYy2XW1vLe13/E5l6H0IKxlCFctetBPj7nh6zdUcGXx0zjn/82k/Vrnfq1o7n1qbM5/pMjEicLSNwmEkY+ejRJ+4RpAxPphCJNGGY2BLgYeDBrK03RELx5+Wb++p0ZfOm4KRzd/R0GfP+LHLh1JUPKlvPtYY9w8Y1DWTl7Da9tH8GPXz6bsd8cTc9+PcNtt6MFfybTWqhgy772XFtcpLPJRkNIR2/AY8Bo4GzgySTLjCOosmqsqqoK18ITawhuMfMd/Qb646PG+xkHzvIu7HJw78lW/8iAGf7TSxt89p/eDDeqOp2wlzLN1qhsTZORG0U8m7FIMhR7o7eZXQJc5O43mdnZwP/3LLRhvDN5CZMmLKZ+SjeeXXE0G/wgjBZOrHiDmuNWM/byPpx+3Si6H9g9a/sCJG7D6NYNDjgA1q0LziQuugieeio7A900BiM3NCOulKBSGOn9IeBjZnYR0AM40Mwmuvun27OSrfc+CLd/h4pNq1jOIL7Nj3iEWg4rX8Zlw2dTc2E55900gv4jRgGjcrEfgXyPfI5ylHcp04y4IklF3ksKoD1nGNOem8ZLD81n0u/XcdyLv+QTzX+kjPf3YXd5d9Z87d8ZeNdXgmtElCqdYWRfXR3cfDOsXbvv8+r+LEWulHpJhfbWa1up7LWdM79wHG8/v5RPND+6T7IA6Nq8k0GP3ps+WRR7g7FGeWdXa5Vi22RRWalkIdIqGw0h+bp141i/fuQUf/zr03zPoCHJG5PTXZ87Gw3G8SOsKyuDW75HW2uUd/aosVtKGMXe6N0R+zR6W4oziHTVMplW56QbpKcqjOKjxm4pYZ2ySmqvurrkCcMsfbVMpg3G6QbphZzEUApI2AGfIp1YcSaM8eMT/xoEuOGG9L/sMy0cwiQW9VYqLmoTEkmrOBNGqsL4vvvSvz/TwiFMYtEv0+JSWxtUI1ZXB2ep1dWqVhRpozgTRrLCuLo63PszLRwSJZx4+mVanGprgzaslpbgr5KFyD6KM2Fko/ogk8KhbcKprAxu+mUqIiWsOK/pXQjXk66tVVIQkU6lOBMGqMAWEcmz4qySEhGRvFPCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJYy2iv3CSiIiOVK8A/dyoe11LpqagsegQYIi0unpDCNeoutc6NoWIiKAEsa+Mr2wkohICVPCiKerromIJBVZwjCzw8yswczmmdlcM7s5qlj20lXXRESSivIMYw9wi7uPAk4FvmBmoyKMR1ddExFJIbJeUu6+AlgRu7/ZzOYDg4F5UcUEaNp0EZEkCqINw8yGAicALyV4bZyZNZpZ45o1a/Iem4iIBCJPGGbWG3gc+Iq7b2r7urtPcPcx7j5mwIAB+Q9QRESAiBOGmXUlSBZ17v6nKGMREZHUouwlZcCvgfnu/tOo4hARkXCiPMP4EHAVcK6ZzYrdLoowHhERSSHKXlJTAYtq+yIi0j6RN3qLiEhxUMIQEZFQlDBERCQUJQwREQlFCUNEREJRwhARkVCUMEREJBQlDBERCUUJQ0REQlHCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwREQlFCUNEREJRwhARkVCUMEREJBQlDBERCSXShGFmF5rZAjN7y8xuizIWERFJLbKEYWblwH8BHwFGAVea2aio4hERkdSiPMM4GXjL3d9x913A74FLI4xHRERS6BLhtgcDS+Mevwuc0nYhMxsHjIs93Glmc/IQW6b6A+9FHUQIijN7iiFGUJzZVixxjsjGSqJMGKG4+wRgAoCZNbr7mIhDSktxZlcxxFkMMYLizLZiijMb64mySmoZcFjc4yGx50REpABFmTBeBoab2eFm1g24AvjfCOMREZEUIquScvc9ZvZF4J9AOfAbd5+b5m0Tch9ZVijO7CqGOIshRlCc2dap4jR3z8Z6RESkxGmkt4iIhKKEISIioRRMwkg3TYiZdTezP8Ref8nMhsa99s3Y8wvM7IIIY/yamc0zs9fN7Bkzq457rdnMZsVuOW3cDxHnNWa2Ji6e6+Jeu9rMFsZuV0cc5z1xMb5pZhviXsvL8TSz35jZ6mTjfyzwn7F9eN3MTox7LZ/HMl2ctbH4ZpvZi2b2wbjXFseen5Wt7pcZxHm2mW2M+2xvj3stb1MJhYjz63Exzol9H/vFXsvL8TSzw8ysIVbmzDWzmxMsk93vp7tHfiNo9H4bGAZ0A14DRrVZ5ibg/tj9K4A/xO6Pii3fHTg8tp7yiGI8B6iI3b+xNcbY4y0FdCyvAX6R4L39gHdif/vG7veNKs42y3+JoGNEvo/nWcCJwJwkr18E/B0w4FTgpXwfy5Bxnt66fYLpeF6Ke20x0L9AjufZwJOZfl9yHWebZT8KPJvv4wkMBE6M3T8AeDPB/3pWv5+FcoYRZpqQS4GHY/cfA84zM4s9/3t33+nui4C3YuvLe4zu3uDu22IPpxOMLcm3TKZcuQCod/d17r4eqAcuLJA4rwQeyVEsSbn7c8C6FItcCvzOA9OBg8xsIPk9lmnjdPcXY3FAdN/NMMczmbxOJdTOOKP6bq5w91di9zcD8wlm0IiX1e9noSSMRNOEtN3xvcu4+x5gI1AZ8r35ijHetQSZvVUPM2s0s+lm9vEcxNcqbJyXxU5RHzOz1gGU+TqW7dpWrGrvcODZuKfzdTzTSbYf+TyW7dX2u+nAJDObacFUPFE7zcxeM7O/m9kxsecK8niaWQVBQft43NN5P54WVNGfALzU5qWsfj8LfmqQYmRmnwbGAB+Oe7ra3ZeZ2TDgWTOb7e5vRxMhfwUecfedZnY9wZnbuRHFEsYVwGPu3hz3XCEdz6JhZucQJIwz4p4+I3YsDwbqzeyN2C/sKLxC8NluMbOLgL8AwyOKJYyPAi+4e/zZSF6Pp5n1JkhYX3H3TbnaDhTOGUaYaUL2LmNmXYA+wNqQ781XjJjZ+cB44GPuvrP1eXdfFvv7DjCZ4NdALqSN093XxsX2IDA67HvzGWecK2hzyp/H45lOsv0ouKlvzOw4gs/7Undf2/p83LFcDfyZ3FTphuLum9x9S+z+U0BXM+tPAR7PmFTfzZwfTzPrSpAs6tz9TwkWye73M9cNMyEbb7oQNLoczvsNWse0WeYL7Nvo/Wjs/jHs2+j9Drlp9A4T4wkEDXPD2zzfF+geu98fWEiOGuxCxjkw7v6/ANP9/YawRbF4+8bu94sqzthyRxM0IloUxzO2jaEkb6S9mH0bFWfk+1iGjLOKoH3v9DbP9wIOiLv/InBhhHEe2vpZExS0S2LHNtT3JV9xxl7vQ9DO0SuK4xk7Lr8DfpZimax+P3N2sDuw8xcRtPK/DYyPPfcDgl/qAD2AP8a+9DOAYXHvHR973wLgIxHG+DSwCpgVu/1v7PnTgdmxL/ls4NqIj+W/A3Nj8TQAR8e993OxY/wW8Nko44w9/h5wV5v35e14Evx6XAHsJqjnvRa4Abgh9roRXAjs7VgsYyI6lunifBBYH/fdbIw9Pyx2HF+LfSfGRxznF+O+m9OJS3CJvi9RxRlb5hqCDjfx78vb8SSoVnTg9bjP9aJcfj81NYiIiIRSKG0YIiJS4JQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwREQlFCUMkA7HrEdTE7v/QzH4edUwiuaLJB0Uy813gB7GJ5k4APhZxPCI5o5HeIhkysylAb+BsD65LIFKSVCUlkgEzO5bgyme7lCyk1ClhiHRQ7MpldQRXNdtiZjm7op5IIVDCEOmA2JXW/gTc4u7zgTsI2jNESpbaMEREJBSdYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiE8n8yEe7q1U3x0wAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "theta from own sdg\n",
- "[[3.54608489]\n",
- " [3.1531006 ]]\n"
- ]
- }
- ],
- "source": [
- "# Using Autograd to calculate gradients using SGD\n",
- "# OLS example\n",
- "from random import random, seed\n",
- "import numpy as np\n",
- "import autograd.numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "from autograd import grad\n",
- "\n",
- "# Note change from previous example\n",
- "def CostOLS(y,X,theta):\n",
- " return np.sum((y-X @ theta)**2)\n",
- "\n",
- "n = 100\n",
- "x = 2*np.random.rand(n,1)\n",
- "y = 4+3*x+np.random.randn(n,1)\n",
- "\n",
- "X = np.c_[np.ones((n,1)), x]\n",
- "XT_X = X.T @ X\n",
- "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
- "print(\"Own inversion\")\n",
- "print(theta_linreg)\n",
- "# Hessian matrix\n",
- "H = (2.0/n)* XT_X\n",
- "EigValues, EigVectors = np.linalg.eig(H)\n",
- "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
- "\n",
- "theta = np.random.randn(2,1)\n",
- "eta = 1.0/np.max(EigValues)\n",
- "Niterations = 1000\n",
- "\n",
- "# Note that we request the derivative wrt third argument (theta, 2 here)\n",
- "training_gradient = grad(CostOLS,2)\n",
- "\n",
- "for iter in range(Niterations):\n",
- " gradients = (1.0/n)*training_gradient(y, X, theta)\n",
- " theta -= eta*gradients\n",
- "print(\"theta from own gd\")\n",
- "print(theta)\n",
- "\n",
- "xnew = np.array([[0],[2]])\n",
- "Xnew = np.c_[np.ones((2,1)), xnew]\n",
- "ypredict = Xnew.dot(theta)\n",
- "ypredict2 = Xnew.dot(theta_linreg)\n",
- "\n",
- "plt.plot(xnew, ypredict, \"r-\")\n",
- "plt.plot(xnew, ypredict2, \"b-\")\n",
- "plt.plot(x, y ,'ro')\n",
- "plt.axis([0,2.0,0, 15.0])\n",
- "plt.xlabel(r'$x$')\n",
- "plt.ylabel(r'$y$')\n",
- "plt.title(r'Random numbers ')\n",
- "plt.show()\n",
- "\n",
- "n_epochs = 50\n",
- "M = 5 #size of each minibatch\n",
- "m = int(n/M) #number of minibatches\n",
- "t0, t1 = 5, 50\n",
- "def learning_schedule(t):\n",
- " return t0/(t+t1)\n",
- "\n",
- "theta = np.random.randn(2,1)\n",
- "\n",
- "for epoch in range(n_epochs):\n",
- "# Can you figure out a better way of setting up the contributions to each batch?\n",
- " for i in range(m):\n",
- " random_index = M*np.random.randint(m)\n",
- " xi = X[random_index:random_index+M]\n",
- " yi = y[random_index:random_index+M]\n",
- " gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
- " eta = learning_schedule(epoch*m+i)\n",
- " theta = theta - eta*gradients\n",
- "print(\"theta from own sdg\")\n",
- "print(theta)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f6d7455a",
- "metadata": {},
- "source": [
- "## Same code but now with momentum gradient descent"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "40a68e4f",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Using Autograd to calculate gradients using SGD\n",
- "# OLS example\n",
- "from random import random, seed\n",
- "import numpy as np\n",
- "import autograd.numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "from autograd import grad\n",
- "\n",
- "# Note change from previous example\n",
- "def CostOLS(y,X,theta):\n",
- " return np.sum((y-X @ theta)**2)\n",
- "\n",
- "n = 100\n",
- "x = 2*np.random.rand(n,1)\n",
- "y = 4+3*x+np.random.randn(n,1)\n",
- "\n",
- "X = np.c_[np.ones((n,1)), x]\n",
- "XT_X = X.T @ X\n",
- "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
- "print(\"Own inversion\")\n",
- "print(theta_linreg)\n",
- "# Hessian matrix\n",
- "H = (2.0/n)* XT_X\n",
- "EigValues, EigVectors = np.linalg.eig(H)\n",
- "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
- "\n",
- "theta = np.random.randn(2,1)\n",
- "eta = 1.0/np.max(EigValues)\n",
- "Niterations = 100\n",
- "\n",
- "# Note that we request the derivative wrt third argument (theta, 2 here)\n",
- "training_gradient = grad(CostOLS,2)\n",
- "\n",
- "for iter in range(Niterations):\n",
- " gradients = (1.0/n)*training_gradient(y, X, theta)\n",
- " theta -= eta*gradients\n",
- "print(\"theta from own gd\")\n",
- "print(theta)\n",
- "\n",
- "\n",
- "n_epochs = 50\n",
- "M = 5 #size of each minibatch\n",
- "m = int(n/M) #number of minibatches\n",
- "t0, t1 = 5, 50\n",
- "def learning_schedule(t):\n",
- " return t0/(t+t1)\n",
- "\n",
- "theta = np.random.randn(2,1)\n",
- "\n",
- "change = 0.0\n",
- "delta_momentum = 0.3\n",
- "\n",
- "for epoch in range(n_epochs):\n",
- " for i in range(m):\n",
- " random_index = M*np.random.randint(m)\n",
- " xi = X[random_index:random_index+M]\n",
- " yi = y[random_index:random_index+M]\n",
- " gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
- " eta = learning_schedule(epoch*m+i)\n",
- " # calculate update\n",
- " new_change = eta*gradients+delta_momentum*change\n",
- " # take a step\n",
- " theta -= new_change\n",
- " # save the change\n",
- " change = new_change\n",
- "print(\"theta from own sdg with momentum\")\n",
- "print(theta)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d1f7f38d",
- "metadata": {},
- "source": [
- "## AdaGrad algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "![](\"figures/adagrad.png\")
Figure 1:
\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b85cc96c",
- "metadata": {},
- "source": [
- "## Similar (second order function now) problem but now with AdaGrad"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "3ced9e13",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Own inversion\n",
- "[[2.]\n",
- " [3.]\n",
- " [4.]]\n",
- "theta from own AdaGrad\n",
- "[[2.00059687]\n",
- " [2.99708164]\n",
- " [4.00299455]]\n"
- ]
- }
- ],
- "source": [
- "# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent\n",
- "# OLS example\n",
- "from random import random, seed\n",
- "import numpy as np\n",
- "import autograd.numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "from autograd import grad\n",
- "\n",
- "# Note change from previous example\n",
- "def CostOLS(y,X,theta):\n",
- " return np.sum((y-X @ theta)**2)\n",
- "\n",
- "n = 1000\n",
- "x = np.random.rand(n,1)\n",
- "y = 2.0+3*x +4*x*x\n",
- "\n",
- "X = np.c_[np.ones((n,1)), x, x*x]\n",
- "XT_X = X.T @ X\n",
- "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
- "print(\"Own inversion\")\n",
- "print(theta_linreg)\n",
- "\n",
- "\n",
- "# Note that we request the derivative wrt third argument (theta, 2 here)\n",
- "training_gradient = grad(CostOLS,2)\n",
- "# Define parameters for Stochastic Gradient Descent\n",
- "n_epochs = 50\n",
- "M = 5 #size of each minibatch\n",
- "m = int(n/M) #number of minibatches\n",
- "# Guess for unknown parameters theta\n",
- "theta = np.random.randn(3,1)\n",
- "\n",
- "# Value for learning rate\n",
- "eta = 0.01\n",
- "# Including AdaGrad parameter to avoid possible division by zero\n",
- "delta = 1e-8\n",
- "for epoch in range(n_epochs):\n",
- " Giter = 0.0\n",
- " for i in range(m):\n",
- " random_index = M*np.random.randint(m)\n",
- " xi = X[random_index:random_index+M]\n",
- " yi = y[random_index:random_index+M]\n",
- " gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
- " Giter += gradients*gradients\n",
- " update = gradients*eta/(delta+np.sqrt(Giter))\n",
- " theta -= update\n",
- "print(\"theta from own AdaGrad\")\n",
- "print(theta)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ed1fa9e4",
- "metadata": {},
- "source": [
- "Running this code we note an almost perfect agreement with the results from matrix inversion."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "787b2b5e",
- "metadata": {},
- "source": [
- "## RMSProp algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "![](\"figures/rmsprop.png\")
Figure 1:
\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ac61b6e0",
- "metadata": {},
- "source": [
- "## RMSprop for adaptive learning rate with Stochastic Gradient Descent"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "id": "5febac4f",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent\n",
- "# OLS example\n",
- "from random import random, seed\n",
- "import numpy as np\n",
- "import autograd.numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "from autograd import grad\n",
- "\n",
- "# Note change from previous example\n",
- "def CostOLS(y,X,theta):\n",
- " return np.sum((y-X @ theta)**2)\n",
- "\n",
- "n = 1000\n",
- "x = np.random.rand(n,1)\n",
- "y = 2.0+3*x +4*x*x# +np.random.randn(n,1)\n",
- "\n",
- "X = np.c_[np.ones((n,1)), x, x*x]\n",
- "XT_X = X.T @ X\n",
- "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
- "print(\"Own inversion\")\n",
- "print(theta_linreg)\n",
- "\n",
- "\n",
- "# Note that we request the derivative wrt third argument (theta, 2 here)\n",
- "training_gradient = grad(CostOLS,2)\n",
- "# Define parameters for Stochastic Gradient Descent\n",
- "n_epochs = 50\n",
- "M = 5 #size of each minibatch\n",
- "m = int(n/M) #number of minibatches\n",
- "# Guess for unknown parameters theta\n",
- "theta = np.random.randn(3,1)\n",
- "\n",
- "# Value for learning rate\n",
- "eta = 0.01\n",
- "# Value for parameter rho\n",
- "rho = 0.99\n",
- "# Including AdaGrad parameter to avoid possible division by zero\n",
- "delta = 1e-8\n",
- "for epoch in range(n_epochs):\n",
- " Giter = 0.0\n",
- " for i in range(m):\n",
- " random_index = M*np.random.randint(m)\n",
- " xi = X[random_index:random_index+M]\n",
- " yi = y[random_index:random_index+M]\n",
- " gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
- "\t# Accumulated gradient\n",
- "\t# Scaling with rho the new and the previous results\n",
- " Giter = (rho*Giter+(1-rho)*gradients*gradients)\n",
- "\t# Taking the diagonal only and inverting\n",
- " update = gradients*eta/(delta+np.sqrt(Giter))\n",
- "\t# Hadamard product\n",
- " theta -= update\n",
- "print(\"theta from own RMSprop\")\n",
- "print(theta)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ee8a6301",
- "metadata": {},
- "source": [
- "## ADAM algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "![](\"figures/adam.png\")
Figure 1:
\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b2d0f16b",
- "metadata": {},
- "source": [
- "## And finally [ADAM](https://arxiv.org/pdf/1412.6980.pdf)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "59349e6f",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Own inversion\n",
- "[[2.]\n",
- " [3.]\n",
- " [4.]]\n",
- "theta from own ADAM\n",
- "[[2.00002174]\n",
- " [2.99992395]\n",
- " [4.00006323]]\n"
- ]
- }
- ],
- "source": [
- "# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent\n",
- "# OLS example\n",
- "from random import random, seed\n",
- "import numpy as np\n",
- "import autograd.numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "from autograd import grad\n",
- "\n",
- "# Note change from previous example\n",
- "def CostOLS(y,X,theta):\n",
- " return np.sum((y-X @ theta)**2)\n",
- "\n",
- "n = 1000\n",
- "x = np.random.rand(n,1)\n",
- "y = 2.0+3*x +4*x*x# +np.random.randn(n,1)\n",
- "\n",
- "X = np.c_[np.ones((n,1)), x, x*x]\n",
- "XT_X = X.T @ X\n",
- "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
- "print(\"Own inversion\")\n",
- "print(theta_linreg)\n",
- "\n",
- "\n",
- "# Note that we request the derivative wrt third argument (theta, 2 here)\n",
- "training_gradient = grad(CostOLS,2)\n",
- "# Define parameters for Stochastic Gradient Descent\n",
- "n_epochs = 50\n",
- "M = 5 #size of each minibatch\n",
- "m = int(n/M) #number of minibatches\n",
- "# Guess for unknown parameters theta\n",
- "theta = np.random.randn(3,1)\n",
- "\n",
- "# Value for learning rate\n",
- "eta = 0.01\n",
- "# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980\n",
- "beta1 = 0.9\n",
- "beta2 = 0.999\n",
- "# Including AdaGrad parameter to avoid possible division by zero\n",
- "delta = 1e-7\n",
- "iter = 0\n",
- "for epoch in range(n_epochs):\n",
- " first_moment = 0.0\n",
- " second_moment = 0.0\n",
- " iter += 1\n",
- " for i in range(m):\n",
- " random_index = M*np.random.randint(m)\n",
- " xi = X[random_index:random_index+M]\n",
- " yi = y[random_index:random_index+M]\n",
- " gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
- " # Computing moments first\n",
- " first_moment = beta1*first_moment + (1-beta1)*gradients\n",
- " second_moment = beta2*second_moment+(1-beta2)*gradients*gradients\n",
- " first_term = first_moment/(1.0-beta1**iter)\n",
- " second_term = second_moment/(1.0-beta2**iter)\n",
- "\t# Scaling with rho the new and the previous results\n",
- " update = eta*first_term/(np.sqrt(second_term)+delta)\n",
- " theta -= update\n",
- "print(\"theta from own ADAM\")\n",
- "print(theta)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "1d490115",
- "metadata": {},
- "source": [
- "## Introducing [JAX](https://jax.readthedocs.io/en/latest/)\n",
- "\n",
- "Presently, instead of using **autograd**, we recommend using [JAX](https://jax.readthedocs.io/en/latest/)\n",
- "\n",
- "**JAX** is Autograd and [XLA (Accelerated Linear Algebra))](https://www.tensorflow.org/xla),\n",
- "brought together for high-performance numerical computing and machine learning research.\n",
- "It provides composable transformations of Python+NumPy programs: differentiate, vectorize, parallelize, Just-In-Time compile to GPU/TPU, and more."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "3bc7785a",
- "metadata": {},
- "source": [
- "### Getting started with Jax, note the way we import numpy"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "f1090d5e",
- "metadata": {},
- "outputs": [],
- "source": [
- "import jax\n",
- "import jax.numpy as jnp\n",
- "import numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "\n",
- "from jax import grad as jax_grad"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ac6bf976",
- "metadata": {},
- "source": [
- "### A warm-up example"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "ed55121d",
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/opt/homebrew/lib/python3.9/site-packages/jax/_src/numpy/lax_numpy.py:164: UserWarning: Explicitly requested dtype float64 requested in asarray is not available, and will be truncated to dtype float32. To enable more dtypes, set the jax_enable_x64 configuration option or the JAX_ENABLE_X64 shell environment variable. See https://github.com/google/jax#current-gotchas for more.\n",
- " return asarray(x, dtype=self.dtype)\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "[]"
- ]
- },
- "execution_count": 6,
- "metadata": {},
- "output_type": "execute_result"
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAArY0lEQVR4nO3dd3hUVf7H8feZmfQeUoCEEJAaQoehiYCgC4prlyYgFtxVflZ21+66rmLBvnZBYQUUFNcCKmJdioQiJfQeQkmBMKkzyWTO749EFxEkJDO5U76v55mHyWSS+xl9/Hi499xzlNYaIYQQvsdkdAAhhBD1IwUuhBA+SgpcCCF8lBS4EEL4KClwIYTwUZbGPFhCQoJOT09vzEMKIYTPW7t2baHWOvHk1xu1wNPT01mzZk1jHlIIIXyeUmr/qV6XUyhCCOGjpMCFEMJHSYELIYSPkgIXQggfJQUuhBA+6oyzUJRSLYDZQDKggTe01i8opf4O3AQU1L71Pq31Yk8FFUIIXzMzeyaZtkKsWbPBlgsxqWRZJ5Adk8D1mdc3+PfXZQTuBO7WWmcAfYFblVIZtd97TmvdrfYh5S2EECfItBUydfssshz5gCbLkc/U7bPItBW65fefscC11oe11utqn5cAW4EUtxy9jv67s4BXvtvVmIcUQogGs2bN5p8FhdySnMjj8bFMTUpgen5BzYjcDc7qHLhSKh3oDqyqfWmKUmqjUmqmUiruND8zWSm1Rim1pqCg4FRvOaNlOwt5dskO8kvs9fp5IYQwhC2XPIsZh8nEvJhorikuxWp31JxOcYM6F7hSKhL4ELhDa10MvAqcA3QDDgPPnOrntNZvaK17aa17JSb+5k7QOrmmdwucLs3CdQfr9fNCCGEEHZPKWzHRmLXm5iIb86MjyQoNgZhUt/z+OhW4UiqImvKeo7VeCKC1ztNaV2utXcCbgNUtiU7hnMRIeqfHMX/1AWQHISGEr3iteSaHgoIYbytmynEb0/MLmZqUSJZ1glt+/xkLXCmlgBnAVq31sye83uyEt10OZLsl0Wlc06sFewrLWL2vyJOHEUIIt/m0dAcR1S7+5IwEFNaQJKa3n0h2TIJbfn9dFrMaAIwHNiml1te+dh8wRinVjZqphfuAm92S6DQu7tKMRz7dwvurD2BtFe/JQwkhRIPtz8nioLmSQfY0Iu7+/JfXrbjvdMUZC1xrvQxQp/hWo04bDA+2cEnX5nz0Uy4P/zGD6NCgxjy8EEKclZnfP4oCLug61WPH8Kk7MUf3boG9ysUn6w8ZHUUIIU6rovwYX1XtIbM8jBHWIR47jk8VeJfUGDo0jeL91QeMjiKEEKf1wXf/oMRsokv8VVjMnqtZnypwpRSjerdg00Eb2QdtRscRQojf0C4XHxz6hpYOGDX0Vo8ey6cKHODy7imEWEy8tzrH6ChCCPEbWevfZk+Qpk11D1omRnr0WD5X4LHhwVzUuRkf/3SI8kqn0XGEEOJXZm14i+hqF+db7/X4sXyuwAHGWNMocTj5bONho6MIIcQvDh9ay3JVQvvSZgzv2s7jx/PJAu+dHsc5iRG8lyWnUYQQ3uPdZY8B0C71TwRbPF+vPlngSinGWNNYl3Oc7UdKjI4jhBDYK4r4uHQ7GWWhjDpvRKMc0ycLHOCKHqkEm03Mk1G4EMILLFr+ODaziSZqBK0SIhrlmD5b4PERwQzPbMrCdbnYq6qNjiOECGDa5WJOzhLSHJoL+nl0VZFf8dkCh5qLmcV2J4vkYqYQwkDrNv2bnWYXiSWdGd652Zl/wE18usD7to6ndWIEc1btNzqKECKA/XvDm0RVu2jf9nZCLOZGO65PF7hSinF9WrIu5zhbDhUbHUcIEYCOHP6J71zHaWNLZkz/zo16bJ8ucIAre9TcmTk3S0bhQojG996Kx9BAWPgE0hvp4uXPfL7AY8ODubhLMz5ad5BSh9yZKYRoPPaKIj6wbSWjLITL+v+h0Y/v8wUOMK5PS8oqq2WZWSFEo/p8xTRsZhO6bCjDMpIb/fh+UeA90mLp0DSKOav2y56ZQohGoV0u5uz/khYOTc/Okwjy4LKxp+MXBa6UYlzflmw+VMz6A8eNjiOECAA/bXqX7WYXccczGN2npSEZ/KLAAS7r1pyIYDPv/ih3ZgohPG/OpreIqnYRkXATLeLDDcngNwUeFRrE5T1S+HTjIYrKKo2OI4TwY0eOrOdr5zHaFCcxZkBXw3L4TYEDXNu3JZVOFwvWypZrQgjPmb/icVxASdUoBrVNNCyHXxV4h6bRWNPjeffHHFwuuZgphHA/h93GguNbyCgLYUSfCzCZlGFZ/KrAAa7t15KcY+X8sLPA6ChCCD/0xYppHDcrHLbzuLpnC0Oz+F2BD+/UlITIYN79Ue7MFEK4V83UwS9oUalp1XoCcRHBhubxuwIPtpgY3TuNr7flk1tUbnQcIYQf2bB5HltN1cQVdWTCgFZGx/G/AgcY2ycNBTKlUAjhVnM2vE5ktQt76ES6psYYHcc/C7x5bBgXZjTl/dU5stmDEMIt8vI2stR5jLbFiYwZ0A2ljLt4+TO/LHCAif3TKSqv4pMNsj6KEKLh5q94jGogv/wqRnZpvE0bfo/fFnjf1vG0S45k1op9sj6KEKJBKh0lfFC0mYzyEM7vMYzQoMbbtOH3+G2BK6WY0C+dzYeKWZdTZHQcIYQP+3LFExwzK+zHzmVcX2PWPTkVvy1wgMu7pxAVamHWCplSKISoH+1yMWffIlIrNcktriUlNszoSL84Y4ErpVoopb5VSm1RSm1WSt1e+3q8UuorpdTO2j/jPB/37ESEWLi6ZwsWbzpMfrHd6DhCCB+0cct8NpuqiS/qwMRzjZ86eKK6jMCdwN1a6wygL3CrUioDuAf4WmvdFvi69muvM6FfS6q15t1VMqVQCHH25qx/lQiXC5tlPP1aNzE6zq+cscC11oe11utqn5cAW4EU4FJgVu3bZgGXeShjg6QnRDCkfRJzV+3H4ZQphUKIusvPy+Yr51Ha2RIYO6C7V0wdPNFZnQNXSqUD3YFVQLLW+nDtt44Ap9xPSCk1WSm1Rim1pqDAmPVJJg1Ip7C0ks82HD7zm4UQotaCn6cOll3O5d1TjI7zG3UucKVUJPAhcIfWuvjE7+maeXqnnKuntX5Da91La90rMdGYZRfPbZNAm6RI3l6xV6YUCiHqpNJRwoKiTXQsD2Zwj+GEBXvH1MET1anAlVJB1JT3HK31wtqX85RSzWq/3wzI90zEhlNKcV3/dLIPFrNmv0wpFEKc2ZcrnuSoWVFZNIDx/bxn6uCJ6jILRQEzgK1a62dP+NYnwMTa5xOBj90fz32u6JFCdKiFt5fvNTqKEMIHzN23iJRKTbM075o6eKK6jMAHAOOB85VS62sfFwFPABcopXYCw2q/9lrhwRbGWNP4cnMeB49XGB1HCOHFNm5+n2yTkybH2zFpYFuj45xWXWahLNNaK611F611t9rHYq31Ua31UK11W631MK31scYI3BDj+7VEa83slfuMjiKE8GJz179GuMtFRchEerX0ultcfuHXd2KeLDUunBGZzZi7Kocyh9PoOEIIL1RYsJUvqwpob2vC+IE9vW7q4IkCqsABbhjYihK7kwVrZONjIcRvLVj+KE6lOO64kos7Nzc6zu8KuALvkRZH97RY3l6xj2rZ+FgIcYIqRxnzj20ko8zCH6wXE2zx7or07nQecuO5rdl/tJylW/OMjiKE8CJLVj5JoVnhsg1kXJ80o+OcUUAW+B86JZMSG8aMZTKlUAjxP+/u/ZTmlZr27ScRG27shsV1EZAFbjGbuK5/Oll7j7Ep12Z0HCGEF9i0eQHZJieJXj518EQBWeAAo6wtiAyx8NayPUZHEUJ4gTk/vUK4y0VEk5s4JzHS6Dh1ErAFHh0axOjeLfhs42Fyi8qNjiOEMFBhwVa+dBbQrrgJkwZbjY5TZwFb4ACTahdnf3v5PmODCCEMNX/5P3Eqhck0mj6t4o2OU2cBXeApsWFc0qUZ72XlYKuoMjqOEMIAVY4yFhzbQMcyC1cPusyrb9w5WUAXOMCNA1tTVlnNvCzZsUeIQLT0x6cpNCvCKs5jeKemRsc5KwFf4JkpMQxo04S3l++l0ukyOo4QopHN3v0xzSpdDLJOwWL2rUr0rbQectPA1uQVO/hkwyGjowghGtHmrR+SbXaSUtKeUVbv2rC4LqTAgUHtEunQNIo3ftiNS26vFyJgzFrzEmEuFxlt7yAixGJ0nLMmBU7Njj03D2rNjrxSvt3utRsLCSHc6GjhDpZWF9KxJJ7rBvcxOk69SIHXGtmlOSmxYbz2/W6jowghGsHc7/9OlVKkJ0wiMSrE6Dj1IgVeK8hs4saBrVi9r4i1+71+bwohRANUVZWz0LaRjuUWJl04yug49SYFfoJRvVsQFx7Eq9/J7fVC+LPFy2pWHWxluYD0hAij49SbFPgJwoMtTOiXztKteezMKzE6jhDCQ+bt/oRmVZqxf/ib0VEaRAr8JBP7pxMWZOa172UULoQ/2rB5IZuDnHSo7EjXtCZGx2kQKfCTxEcEM9rago/XH5Td64XwQzNXvUCYy8Vl5z5odJQGkwI/hZsGtkYpePMHGYUL4U/yC3awTB2la0UCQzp3NjpOg0mBn0Lz2DCu6J7KvKwcCksdRscRQjTAzOyZZC1/Cp7L5D/vDqXSpOgR35q3N79tdLQGkwI/jT8NPoeqahczZds1IXxapq2QqdtnscKRz/vRkXR0OJhXnkWmrdDoaA0mBX4arRIiGNG5Gf9euV+WmhXCh1mzZjM9v4C7khPJt1g4EBTE9PwCrFmzjY7WYFLgv+OWwedQ4nDy75X7jI4ihKgvWy697A7Catc5GmsrwWp3gC3X4GANJwX+Ozo1j+H8DknMWLaXMofT6DhCiPqISeWV2BgKLWaGlJWzIDqSrNAQiEk1OlmDSYGfwZTz21BUXsWcVfuNjiKEqIcVvcbwZmw0zauqeC6/kOn5hUxNSiTLOsHoaA0mBX4GPdLiOLdNAm/8sBd7VbXRcYQQZ2lm7kpcSnG3rQozCmtIEtPbTyQ7JsHoaA3mewvgGuD/zm/DqDd+ZF5WDpMG+N6i70IEqvLyQnaW76Wd08LAKdsgJAgAa+3D151xBK6UmqmUyldKZZ/w2t+VUgeVUutrHxd5Nqax+rRugrVVPK99v1tG4UL4kFc/vY1jFhODEycRVlve/qQup1DeAYaf4vXntNbdah+L3RvL+9x2flvyih0sWOv7V66FCATHi/ayoHwDncuCuWHkFKPjeMQZC1xr/QMQ8AtkD2jThB5psbz67S4cThmFC+HtXlx8G+VK0bf5bYQH++fZ4oZcxJyilNpYe4ol7nRvUkpNVkqtUUqtKSgoaMDhjKWU4vZh7Thks7NgjYzChfBmhw+t5ePKvXQrjeL6EeOMjuMx9S3wV4FzgG7AYeCZ071Ra/2G1rqX1rpXYmJiPQ/nHc5rm0D3tFhekVG4EF7tuaV/QSvonf43In1ws+K6qleBa63ztNbVWmsX8Cb+cUH3jJRS3CmjcCG82u7dX/GlK5+uxQlMumCk0XE8ql4FrpRqdsKXlwPZp3uvvxnYNoEeabG8LKNwIbzS9P8+RJjW9Gr3kF+PvqFu0wjnASuB9kqpXKXUDcBTSqlNSqmNwBDgTg/n9BpKKe4Y1o7DNjvzZRQuhFdZnz2XZaqUDFtLJg45z+g4HnfG/z1prcec4uUZHsjiMwa2TaBnyzhe/mYXV/dMJTTIbHQkIQKedrl4Jms6cdpFz06P+P3oG+RW+npRSnH3Be04UmxnXlaO0XGEEMB/V7/EenMVbWwZTBrU3eg4jUIKvJ76t0mgb+t4Xv52N+WVslKhEEZyVTt5bvMMmla56NfjYb+d930yKfAGuPvC9hSWOpi9UlYqFMJIi354mF1mTaqtD+MHdDQ6TqORAm+A3unxDGqXyGvf76bELrv2CGGESkcJL+39mHQHDOl3f0Bdk5ICb6C7L2zH8fIq3l6+z+goQgSk+d/8lcNmRVzxBYy2BtZqoVLgDdQlNZYLM5J584c9FJVVGh1HiIBSVnqENw7/QPtyMxcPvJ1gS2BVWmB9Wg+5+8L2lFY6ee373UZHESKgzFp6F0VmE2EVV3BlrxZGx2l0UuBu0L5pFJd3T+GdFfs4YrMbHUeIgHC0cAezijaQWRrC2AtuwmIOvDoLvE/sIXcOa4dLa174eqfRUYQICK9/fRd2pQiunsjwzKZGxzGEFLibtIgPZ6w1jflrDrCnoNToOEL4tdzcH1lQsY8uxdHcOGI0SimjIxlCCtyNppzflhCLiWe+2mF0FCH82kvf3YNZQ0jwrZzbxvc3J64vKXA3SowK4YZzW7Fo42E25h43Oo4Qfmn7zkV87iwk43gSt150UcCOvkEK3O0mn9ea+Ihgpi3ehtba6DhC+J3nVzxKhEsTGz+V7mmn3QwsIEiBu1lUaBC3nd+GlXuO8t0O391CTghvtHbDLJZRRtuiVtx58SCj4xhOCtwDxvZpScsm4Tz5+TaqXTIKF8IdtMvFM2ufp4nTRatW99IqIcLoSIaTAveAYIuJv/yhPduOlPDRTweNjiOEX/j2x+lsMjtJO9aZ/7uwl9FxvIIUuIdc3LkZXVNjeGbJduxVsvWaEA1R7azk+W3v0rxS06PrAyREhhgdyStIgXuIUor7LurIYZudGcv2Gh1HCJ/2yXcPsNesaWrrz02DOhgdx2tIgXtQn9ZNuDAjmVe+3UV+idxiL0R9OOw2Xs5ZTGs7XDTwfiICYKu0upIC97B7L+qIw+niua/kFnsh6mPu0rvJMysS7SO5qnea0XG8ihS4h7VKiGB8v5a8vzqH7UdKjI4jhE8pKT7IjPyVdCy3MOniuzGbAvemnVORAm8Etw9tS2SIhccWbzU6ihA+5c0ld2Azm2huHs+AAL5l/nSkwBtBbHgwtw1tyw87Cvh2W77RcYTwCQX5m5lXspUuJWFMuWyy0XG8khR4I5nQL53WCRE8umgLlU6X0XGE8HovLrkLp4IOibfTJinS6DheSQq8kQRbTDwwsiN7CsqYvXKf0XGE8Gr7c5bxmfMgXUviuO2Sa4yO47WkwBvRkPZJDGqXyAtf7+RoqcPoOEJ4rae/vgeLhkEZfycmPMjoOF5LCrwRKaV4cGRHKiqrZc1wIU5jw+aFfG+y0aMslfGDhxgdx6tJgTeyNklRTOiXzrysHLIP2oyOI4TXeXbl48RUuxh93vSA3OfybMg/HQPcPqwt8eHBPPzJZlkzXIgTfLniFdYFOehT1YkhXTKNjuP1pMANEBMWxN9GdGDt/iIWrpPVCkVgm5k9k6zlT6Gfy+Sd7Odp6nQyrFUHZmbPNDqa15MCN8hVPVLp1iKWaZ9vo9heZXQcIQyTaStk6vZZvIqN7JAQhpeWMS33QzJthUZH83pnLHCl1EylVL5SKvuE1+KVUl8ppXbW/hnY+xrVg8mkePTSTI6WOXhhqayTIgKXNWs20/ILeD0uhrjqaj6OimR6fgHWrNlGR/N6dRmBvwMMP+m1e4CvtdZtga9rvxZnqXNqDGOsabyzYh/bjhQbHUcIY9hyWRsWikspisxmrikuxWp3gC3X6GRe74wFrrX+ATh20suXArNqn88CLnNvrMDxlwvbExMWxIP/ycYl26+JALQxOokZMdEEuzQ3F9mYHx1JVmgIxKQaHc3r1fcceLLW+nDt8yNA8uneqJSarJRao5RaU1Agm/yeLC4imHtGdGD1viI+WCsjDhFYKh0l3BUdhAam5xcw5biN6fmFTE1KJMs6weh4Xq/BFzF1zTy40w4dtdZvaK17aa17JSYmNvRwfumqHqn0To9j2udbOVZWaXQcIRrN4wvGkRdk4ipXa4YEJwIKa0gS09tPJDtGVh88k/pubZGnlGqmtT6slGoGyBJ7DWAyKf55WWcufvG/PPH5Vp66qqvRkYTwuLUb5/GRcw+9y2O49+b/wAk37VhrH+L31XcE/gkwsfb5ROBj98QJXO2bRnHDwFbMX5NL1t6TLzkI4V8cdhsPZz1OfLXmukGvEyR3XNZLXaYRzgNWAu2VUrlKqRuAJ4ALlFI7gWG1X4sGun1oW1Jiw7h34UYcTtnJXvivZz6ayP4gGGC6lPM6yR2X9VWXWShjtNbNtNZBWutUrfUMrfVRrfVQrXVbrfUwrbUMGd0gPNjCY5dnsrugjFe/2210HCE8Yv2m93jfsYseJVFMHfV3o+P4NPl7i5cZ3D6JS7s155Vvd7MrX/bQFP7FYbfxQNZjxFdr/tj7X8SGBxsdyadJgXuhB0dmEB5i5t6Fm2RuuPArz308kf0W6O68hCus3Y2O4/OkwL1QQmQI91/UkdX7ipiTlWN0HCHcYn32POZV7KJ7cTR3XfMQSskO8w0lBe6lruqZyrltEnhi8VYOHq8wOo4QDeKw23hg1eM0qdac1+lZUuPCjY7kF6TAvZRSimlXdEYD9y7cJOuGC5/286mTjhUjuX6wzPB2FylwL9YiPpx7RnTghx0FLJDb7IWP+tWpk1EPYjLJqRN3kQL3ctf2aYk1PZ5HP9vCEZvd6DhCnBV7RRH3r3qcBKemb4fpnJMYaXQkvyIF7uVMJsWTV3Wh0univo/kVIrwLc9/ch05FmhbMZLJ5/cxOo7fkQL3Aa0SIvjb8A58sy2f+WsOGB1HiDpZv2kucyt2090Wzd3XPCAbFHuA/BP1Edf1T6dPq3j+8ekWDhwrNzqOEL/LXlHE/VnTSHRqBmQ8Q9vkKKMj+SUpcB9hMimmX12zSuFfP9goN/gIr/bsxxPJsUBHxx+5UU6deIwUuA9pER/OgyMzWLnnKLNW7jM6jhCntG7ju7xn30OP4hj+MvoBzDLrxGOkwH3MqN4tOL9DEtM+38aOPFkrRXiXivJj3L/6SRKdmqHdXqBlkwijI/k1KXAfo5TiySu7EBVi4fb31suys8KrPP2fCeRaoJvrCsaf28PoOH5PCtwHJUaF8NRVXdh6uJhnluwwOo4QAKz6aRYfVO6jV3Es9459QNY6aQRS4D5qaMdkxvVJ483/7mHFrkKj44gAV1F+jIfXPU2SUzNm4OskRIYYHSkgSIH7sPsv7kirhAjunL9eNkMWhnr8w2s5aFEMDB7Nhd0yjI4TMKTAfVh4sIWXxnSnqKyKqQs2yF2awhDfrXqLj6tz6FMaz9/G3Gd0nIBS313phZfo1DyG+y7qwN8/3cLM5fu44dxWRkcSfm5m9kwybYVYs2ZTXnyQp1KaEqcUrbtcQmiQ2eh4AUVG4H5gYv90hnVM5onPt7Ip12Z0HOHnMm2FTN0+iyxHPi/FRXMgyEKlSTHM5DI6WsCRAvcDSimevqoLCZEhTJm3jmJ7ldGRhB+zZs1men4Btycl8m50FCEuFy/kFWDNmm10tIAjBe4n4iKCeWlMd3KLKviLnA8XnmTLJcXpxKkApRhXXILV7gCbrFnf2KTA/Uiv9HjuGd6BLzfnMWPZXqPjCD9VGt2c65sm4VCKUcUlfBQVSVZoCMSkGh0t4EiB+5kbB7biwoxknvh8G2v3HzM6jvAz1c5Kro+ycMhi4c5jx3ngaBHT8wuZmpRIlnWC0fECjhS4n1FK8fTVXWkeG8Ytc9ZRUOIwOpLwI/947wq2BmuG2uOZpGIBhTUkientJ5Idk2B0vIAj0wj9UExYEK9d25MrXl3OrXPWMeemPgTJYvqigf696C4WVu+nb1k80278BoL/N2XQWvsQjUv+q/ZTGc2jefLKLmTtO8Zji7YaHUf4uBVr3+bZgiV0rLBw/1UfERYs8729gYzA/dil3VLYcMDGzOV76ZIawxU95CKTOHsHDqzkrxumk1ytmNz3HdKT4o2OJGrJCNzP3XtRB/q2jueehZtYf+C40XGEjyktOcwtSybjAi5p+iDDunU1OpI4gRS4nwsym3hlXE+SokK4afYaDtsqjI4kfES1s5I7FlxKrlkzmKu5ZeQ1RkcSJ2lQgSul9imlNiml1iul1rgrlHCv+IhgZkzsTbnDyeTZa6molE0gxJk9Pv8KVpkr6F/alYfGPyjre3shd4zAh2itu2mte7nhdwkPad80ihfHdCf7kI2pH2yQTZHF75qz+E7mV+2nd3ETHp4wQxap8lJyCiWADO2YzD3DO7Bo42GmL9ludBzhpZaveZPp+V+RUW7htkveJyk61OhI4jQaWuAaWKKUWquUmnyqNyilJiul1iil1hQUFDTwcKKhJp/XmrF90njlu928l5VjdBzhZfbtX85fNz5P0yqYaH2HbunJRkcSv6OhBX6u1roHMAK4VSl13slv0Fq/obXupbXulZiY2MDDiYZSSvGPP3ZiULtE7v9PNj/skP+piholxQeZ8tWfALg89REu6ikzTrxdgwpca32w9s984CPkZiyfYDGbeHlcD9olR3HLnHVkH5Q1xAOds8rO/y24lIMWzfDgcUwecaXRkUQd1LvAlVIRSqmon58DFwLZ7gomPCsyxMLb1/UmJiyI695ezf6jZUZHEgZ6cO5lrLU4GOboxX1j7zU6jqijhozAk4FlSqkNQBawSGv9hXtiicbQNCaUWddbqXa5GD8jSxa+CiAzs2eStfwpeC6T+c+k8BkH6VURTNu+52I2yXRBX1HvAtda79Fad619dNJaP+bOYKJxtEmKZOZ1vSkocXDd21mym0+A+HlbtNm6iGlN4uhsd7Ar2E63UlmC2JfINEJB97Q4Xr22BzvySpj09mrKK51GRxIeZs2azZRjRUyPjyPK5eJAkIVn8mVbNF8jBS4AGNw+iRdHd+ennCJumr0Ge5XcrenPfrLn81yTOCJcmiKzmVHFpbItmg+SAhe/GNG5GU9f1ZXlu44yZe46Kp2yy7g/+nLZy9zcNJEIlwszmpuLbMyPlm3RfJEUuPiVK3um8uhlmSzdmi8l7oc+/W469+18lehqjV2ZeDa/kCnHbbItmo+SAhe/Mb5vSx75YyeWbMnjVilxv/HhV4/w0L53SK1SDGw9jmfbT8QakoRsi+a7ZEMHcUoT+6ejFDz08WZunbuOf43tTohFFjTyVfO++BtPHllEa4eZ+4e+T882HWq+MeCvv7xHtkXzPTICF6c1oV86j17aia+25HGTLEPrs2Z8OoVpRxbR1m7hwWEL/1fewudJgYvfNb5fOk9d2YX/7ixg4swsSmSeuE95ZeENPH/sezpVBPPQiE/pfs45RkcSbiQFLs7omt4teHF0d9blFDH2zVUcK6s0OpKog+nvj+HVkiy6loXyj8u+oHPLFkZHEm4mBS7q5JKuzXljQk925JVw5asryDlabnQkcRra5eKRdy9jlj2bnmWRTLtmCW2bJRkdS3iA0rrxdmbp1auXXrNGdl7zZWv2HeOGWWsIMpt4Z1JvMlNijI4U8GZmzyTTVog1azbalsu0JgnMiw4jrTKMWaO/IyEq3OiIooGUUmtPteuZjMDFWemVHs+Hf+5HiMXEqNdX8u22fKMjBbyf1zX50ZHPI03imBcdRohLc1+7y6W8/ZwUuDhrbZKiWHhLf1o2ieCGWauZsWwvjfk3OfFr1qzZPFJQyC1Nk/gwOpJQl4uX8/IZsG6O0dGEh0mBi3pJjg7lgz/3Y1jHZB79bAv3fZRNVbXc8GOELRV5PNUkjp+XIJtoK6GPrGsSEKTARb2FB1t47dqe/HnwOczLymHcm6vIL7EbHStgaJeL1z6awvhmyZSZTES6ZF2TQCMFLhrEZFL8bXgHnh/VjY0Hj3PJS8tYu7/I6Fh+r7z8KLe9M5iXi7+nRZXCheL5/AJZ1yTASIELt7isewoL/zyAEIuZ0W+s5O3lcl7cU7bs/JpRcwfzvekY51e05IJet8q6JgFKphEKt7KVV3Hn/PV8sy2fCzKSefqqLsSGBxsdyyedOD0QWy7EpPJUUiveq9xDuIbLI67hzqsfwiRboPk9mUYoGkVMeBAzJvbigYs78t32fC564b+s2nPU6Fg+6efpgVmOfKrQ3Blczr+d+2jqhH90f4W7Rz0s5R3gpMCF2ymluHFgaxb+eQDBFhOj3/yRf362RXb5OUvWrNlMzy/grqQEhrdoztLICIaVlrGwWHN+90FGxxNeQApceEzn1BgW3TaQcX3SeGvZXka+tIwNB44bHctnVNpy2RQSTKnJRL7FwrDSMp4rOEpoySGjowkvIQUuPCoixMI/L+vM7OutlNqdXP7Kch75dDOlDtk4+fd8sfx1rk5pyvPxcZiAsbZi1oaFyvRA8SuyoYNoFOe1S2TJXefx9BfbeWfFPr7IPsLf/9iJCzOSUSowz+Oe6iLll5kX8/KBb9hrthNnMhHhcvFiXgFWu4Oh5RVMTUpkevsJsvGCAGQELhpRdGgQj16WyQd/6k90aBA3/3st185YxfYjJUZHM8SJFyldaJ4yF/OXg4vYb6rgfHsql3a6iRfbyfRAcXoyjVAYoqraxZwf9/PsVzsodTgZY03j9qFtSYoONTpa43kukyxHPnckJRKmXeRbLLRzVPJYsabD1B1GpxNe5HTTCOUUijBEkNnEdQNacWm3FJ5buoO5q3L4cF0u1/VvxZ8GtfarueOnOlWSZZ3AF5YyDoRFU2I2UYKJIWXlvJBfiCIwTymJsyenUISh4iKC+celmSy9axDDOzXl9R92c+6T3zLt861+s67KiadKNJp3dBE375zNguhItgQHE+Zycf1xG+tDQ1gtFynFWZBTKMKrbDtSzL++2cXiTYexmE1c2SOV6/qn075plNHR6uSUo23sLLY4+TIyggiXizyLhdjqai4sc7AkIpRn8msuUmaFhtRepJyI9YTd4oWQOzGFT+jQNJp/je3BN3cP5soeqXy4Lpc/PP8Do15fyWcbD3n9zUAnjrZBs9hZyK2xIXwTEU6pyUSexcK55RUsPXCQFGclz8gaJqIBZAQuvNqxskrmrznAuz/uJ7eoguhQCyO7NueK7in0SIsz/Fby34y4lYm3osN5JTaG5Opqci01l5k6OSrZFxzEWFsJH0RHMj2/sKa478w2NL/wDXIRU/ik+Ihg/jToHG4a2Jpluwr5aF0uC9flMndVDklRIVyQkcwFGcn0bd2E0CBzo+f7ecT9UOVRiiPD+TAqko0hwaAUuSYTPSvsXFNSwhNN4n+Zz93Xbpf53MItGjQCV0oNB14AzMBbWusnfu/9MgIX7lDqcLJ0Sx5Lthzhu+0FlFdWE2wx0TMtjgFtmtAjLY7M1BiiQ4M8cnytNYdtdjbmHqfvJ4P50WLjL0kJoBQmrelmt7MjOISxxSUsiI5kaFk5I5wWrIT+ahZKdkwC12de75GMwr+cbgRe7wJXSpmBHcAFQC6wGhijtd5yup+RAhfuZq+qZuWeo6zYVcjyXUfZcrgYAKWgdUIE7ZKjaJMUSevECFJiw2kaHUpSdMgZR+taa0ocTvJsdo4U28k5Vs7u/DJ2F5Sy5XAxBSUOAPaEjMOkNJOTE1kZHsbIklKWh4fVnCKRC5PCTTxxCsUK7NJa76k9wHvApcBpC1wIdwsNMjOkfRJD2icBcLy8ko25NjYcOM6GXBvbjpSwZEse1a5fD1TCgsxEhFiICrVgqT2PrgGHs5pSu5MyRzWVJ+3xGRZk5pykCAa2SaBri1i6tohFfZBClqOAbSHB3FxkY3ZMFFOKjmN1OPnfhcma0bacLhHu1pACTwEOnPB1LtDn5DcppSYDkwHS0tIacDghziw2PJjz2iVyXrvEX15zOKs5cKycQ8drRtN5NjvF9ipKHdWUOpxUu/5X1CEWM5EhFiJCLMRHBJEcHUrT6FBS4sJoHhP2m4umWdaJTN0+i+m1UwGttee3O/S5/ZcRt7X2IYS7efwiptb6DeANqDmF4unjCXGyEIuZNklRtEly/1zy7JiEmtMjttlgz5URt2hUDSnwg0CLE75OrX1NiIDxy0XIE85vy4hbNJaG3MizGmirlGqllAoGRgOfuCeWEEKIM6n3CFxr7VRKTQG+pGYa4Uyt9Wa3JRNCCPG7GnQOXGu9GFjspixCCCHOgqyFIoQQPkoKXAghfJQUuBBC+CgpcCGE8FGNupysUqoA2N9oB3SfBKDQ6BCNKNA+L8hnDhS++plbaq0TT36xUQvcVyml1pxqIRl/FWifF+QzBwp/+8xyCkUIIXyUFLgQQvgoKfC6ecPoAI0s0D4vyGcOFH71meUcuBBC+CgZgQshhI+SAhdCCB8lBX4WlFJ3K6W0UirB6CyeppR6Wim1TSm1USn1kVIq1uhMnqKUGq6U2q6U2qWUusfoPJ6mlGqhlPpWKbVFKbVZKXW70Zkag1LKrJT6SSn1mdFZ3EUKvI6UUi2AC4Eco7M0kq+ATK11F2o2r77X4DweUbs598vACCADGKOUyjA2lcc5gbu11hlAX+DWAPjMALcDW40O4U5S4HX3HPBXava+9Xta6yVaa2ftlz9Ss+OSP/plc26tdSXw8+bcfktrfVhrva72eQk1pZZibCrPUkqlAhcDbxmdxZ2kwOtAKXUpcFBrvcHoLAa5Hvjc6BAecqrNuf26zE6klEoHugOrDI7iac9TMwBzneF9PsXjmxr7CqXUUqDpKb51P3AfNadP/MrvfWat9ce177mfmr9yz2nMbMLzlFKRwIfAHVrrYqPzeIpSaiSQr7Veq5QabHAct5ICr6W1Hnaq15VSnYFWwAalFNScSlinlLJqrY80YkS3O91n/plS6jpgJDBU++8NAwG5ObdSKoia8p6jtV5odB4PGwD8USl1ERAKRCul3tVaX2twrgaTG3nOklJqH9BLa+2LK5rVmVJqOPAsMEhrXWB0Hk9RSlmouUg7lJriXg2M9ef9XVXNSGQWcExrfYfBcRpV7Qh8qtZ6pMFR3ELOgYvT+RcQBXyllFqvlHrN6ECeUHuh9ufNubcC8/25vGsNAMYD59f+u11fOzoVPkZG4EII4aNkBC6EED5KClwIIXyUFLgQQvgoKXAhhPBRUuBCCOGjpMCFEMJHSYELIYSP+n/ar/W40xxDvQAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- }
- ],
- "source": [
- "def function(x):\n",
- " return x**2\n",
- "\n",
- "def analytical_gradient(x):\n",
- " return 2*x\n",
- "\n",
- "def gradient_descent(starting_point, learning_rate, num_iterations, solver=\"analytical\"):\n",
- " x = starting_point\n",
- " trajectory_x = [x]\n",
- " trajectory_y = [function(x)]\n",
- "\n",
- " if solver == \"analytical\":\n",
- " grad = analytical_gradient \n",
- " elif solver == \"jax\":\n",
- " grad = jax_grad(function)\n",
- " x = jnp.float64(x)\n",
- " learning_rate = jnp.float64(learning_rate)\n",
- "\n",
- " for _ in range(num_iterations):\n",
- " \n",
- " x = x - learning_rate * grad(x)\n",
- " trajectory_x.append(x)\n",
- " trajectory_y.append(function(x))\n",
- "\n",
- " return trajectory_x, trajectory_y\n",
- "\n",
- "x = np.linspace(-5, 5, 100)\n",
- "plt.plot(x, function(x), label=\"f(x)\")\n",
- "\n",
- "descent_x, descent_y = gradient_descent(5, 0.1, 10, solver=\"analytical\")\n",
- "jax_descend_x, jax_descend_y = gradient_descent(5, 0.1, 10, solver=\"jax\")\n",
- "\n",
- "plt.plot(descent_x, descent_y, label=\"Gradient descent\", marker=\"o\")\n",
- "plt.plot(jax_descend_x, jax_descend_y, label=\"JAX\", marker=\"x\")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4a994430",
- "metadata": {},
- "source": [
- "### A more advanced example"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "863db98d",
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/opt/homebrew/lib/python3.9/site-packages/jax/_src/numpy/lax_numpy.py:164: UserWarning: Explicitly requested dtype float64 requested in asarray is not available, and will be truncated to dtype float32. To enable more dtypes, set the jax_enable_x64 configuration option or the JAX_ENABLE_X64 shell environment variable. See https://github.com/google/jax#current-gotchas for more.\n",
- " return asarray(x, dtype=self.dtype)\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- ""
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABLr0lEQVR4nO29eZhcZ3Xn/31rX7qq12rtsvbFlrzKOyBL9oABx4YJ+z6EeFBIxoATgwnJZAEmhIwg/FgUA5mEiQ1DMGDMboyMjcFY8ibZ1r52S93qtbauqlt3eX9/3HpvVXfXcpf3VnV1v5/n4cFutW69Xe4699zv+Z5zCKUUAoFAIGhfPK0+gEAgEAicIQK5QCAQtDkikAsEAkGbIwK5QCAQtDkikAsEAkGb42vFi/b19dFVq1a14qUFAoGgbXnmmWfGKKWJmV9vSSBftWoV9u/f34qXFggEgraFEHKm2teFtCIQCARtjgjkAoFA0OaIQC4QCARtjgjkAoFA0OaIQC4QCARtjgjkAoFA0OaIQC4QCARtjgjkAoFAUIcjwxk8dXK81ceoy7wJ5Kom5qoLBAL+fP6Ro7jnuwdafYy6zItA/tODQ7jqU4/g2IVMq48iEAjmGRO5IoZS+TmdLM6LQH5qfArJnIw/uf9Z5IpKq48jEAjmEamcDFmlGMtKrT5KTeZFIC/IGgDg+GgW//Ohl1p8GoFAMJ9I5WUAwPlkvsUnqc08CeQqQn4P/mzHOvznM4P47jODrT6SQCCYJ7BAPpQqtPgkteEWyAkhXkLIc4SQH/G6pln0QO7FXbdswHVrevBXP3gRqZzc7GMIBIJ5hqSoyMsqgIWTkd8F4BDH65kmX1QR9nvh9RC8+7pVyMsqLmTm7t1TIBC0BywbB4DzybkbU7gEckLIcgCvB/B1HtezSkHREPJ7AQDhgP4j5YtqK44iEAjmEelpgXz+Z+RfAHAPAI3T9SyRL6pGIGf/zx6HBAKBwC4sI/d5CIZS8ziQE0JuAzBCKX2mwffdSQjZTwjZPzo66vRlpyEperETAMIikAsEAk6wQL6uvwPn53mx80YAtxNCTgP4NoCdhJD/mPlNlNL7KKXbKKXbEolZK+ccwTRyoJyRF4S0IhAIHJIsmSY2LY5hNCNBUuZmXHEcyCml91JKl1NKVwF4G4BfUUrf5fhkFigoZWmFBfTCHH3DBQJB+8Ay8s1L4gCACyn7TUGUUlDqTnfovPCRV2bk4UBJWim2RK4XCATzCBbINy6OAQDOO9DJnx9IYvW9P8HeIyNczlYJ10BOKX2MUnobz2uaoSBrCJY0creKnWNZSbT/CwQLjGRORkfQhxU9EQDOnCtFRU8ug17++fO8yMgLckVGzqQVToH85fNp/I9vPYdrPv1L/P2PXuZyTYFA0B6k8zI6w34s7QwDcNbdKbFA7ucfdn3cr9gCWGcnAPi9BF4P4eIj/+uHXsQ3f3cGHUEfOoI+DE7OXfuRQCDgT6oUyMMBL7ojfpxzkJGzQB7wenkdz6DtM3JKKfIVGTkhBGG/l4u08r1nz2HHxgSe/PhOXL6yG+mCkFYEgoUEC+QAsKQzjCFHgVyPSW5k5G0fyGWVQqMwfOSArpM7DeSaRjFVVLB1eRc6w37EQz5k8mJ+i0CwkEhWBPKlXWFn0kppSmvQJwL5LFjAZtKK/s8exz7ynKyCUiAW1NWneNiPdEEEcoFgIZGaFshDjqSVosoCuZBWZlGoEsjDfq9jH3m2JKNEWSAP+ZHOK675QAUCwdwjlZfRFSlLK5mCgqxkT2KVSrFKZORVYIE8XBnIA17Hxc6spGffHSGWkftQVDWjYCEQCFrPxFQRf/6fL+DM+BT3axdkFUVFQ7wiIwdgWyc3ip0ikM+murTiXCPPSvrfj1Vk5MD0aWgCgaC17D89ge8+M4i33/cUzo7nuF6btedXauQAbMsrhv1QBPLZsDVvbHwtgJJrxVnmzKSVckZeCuRCJxcI5gzjU0UAwGROxtu/9hQGJvgFc9bVWZZWShm5zYKnpKjwegh8oiFoNkxCCflmaOS8pBUjI9f/P5UXFkSBYK4wltFnn9z/x9ciKyl4231PYZzTkmQWyFlGvigegofY7+4sKpor2TgwDwI5K2qGAjM0cofSSoZl5EGRkQsEc5XxqSJiIR+uXNmNr7zzSpxL5vHbE+Ncrj0zkPu9HvTHQrY3BUkikNemUCUj56ORzwjkQiMXCOYco1kJfR1BAMD6RR0AdO83D5I5XbZhgRwAlnSFbC+YkGTNlUInMB8CeSkjDwf4+sinpOn2Q/Yf081APldnHQsEc5XxrIS+jgAA/p9RQyMPB4yvLe0K25ZWJEV1xUMOzINAzsbVhmf4yB1LK5KCoM9j3EFjJY3crTb9w8NpbPmfP8cLA0lXri8QzEfGskX0RvWMPOjzIuz3Gpm0U9J5GYSUP/sA0BsNYDJn70YhpJU6lBuCprtWFI1CVu07V7IFxZBV9Ot7EfR5XMvInz2ThKxS/PjgkCvXFwjmI+NZCX2xcsbcGfYbmbRTUnkZsaAPHg8xvhbye21PVi0qmitzVoB5EMir+ciZzOJklG1WUgzrIcPNNv1jIxkAwKOHLrhyfYFgviGrGiZzspGRA7pVMGkzY55JMi+jM+Kf9rWQzwNJ0Wx1eOsZuZBWqiLJKgiZbrLnsVxiSpqekQO6BTHtkv3w+EgWAHBidAqnx/h3qQkE843Jkoe8L1YO5J1hP7diZyovT9PHASBYii12OrwlRUXABQ85MA8CeV5WEfJ5QUj58cdYLuFg3VumUCWQu5mRX8hi20XdAIBfHea/CkogmG+MlvzifdHp0grPYmelYwVwtrhGEtJKbQqyNk0fByr2djqUVmIzpZUQv1+SSlJ5GcPpAm7evAjr+jtEIBcITDCenZ2R85RWqgVyJ0/7oiGoDpVLJRhhDtJKVlIM6yFDz8j5SytMVlnf34GbN/Xj96fGbU9YEwgWCmOljLw36lKxMycbjYAMljQWbIwAERp5HSrXvDHY44uTCYgzXSsA08j5Z+THS4XO9Ys6sHNTP2SV4jfHRrm/jkAwn6iekQeQl1XHO3sppdNG2DJCTqQVWRUZeS2qBXIeC5jruVZ4zyQ/diGLkN+D5d0RXHVRN+IhHx49JOQVgaAeY1kJAa/HmFAKVIzScJhw5YoqFI1WkVZYRm5PIxednTWop5E78XtKijbtFwTQNXJZpbYeq+pxbCSLtYkOYzLaTRv7sffICDRNLLEQCGoxli2iryMwzejQVQq8TuWVmXNWGGwUiJBWOJOX1Wnt+YBzjXxmez4jHmbdnXzlleMjWazv7zD+/ebN/RjLFnHgXIrr6wgE84mxrITejuC0rzEpxKkFceYscgYbzmdnA5loCKpDoWQ/rMRpIJ85MIvhxuCsrKTgXDKP9YtixteuXKnbEI8OZ7i9jkAw3xifKs9ZYbDAm3LoXCnPWamRkVusv2kaRVEVrpWa5GV12ghboHzXtFvsZIF8lv3QhVG2J0qOlXUVGXmiVLwZydjf2C0QzHfGMkVj8iGDNfA4zchZIK/pWrGYkbPFy0Ijr4EkazUzcrsaeTkjn/4fsTxdjZ818OiFkmOlIpCH/F50hv0YyfAZkC8QNItUTsZjR0a4LXeoBaUU41OzpZVOThp5upZG7renkUsyW/Pmjkbua/wtcxtdI59+P/J7PfB5iH1pZcaaN0Y8xF8jPz6SRcDrwcqeyLSv98eCGEmLQC5oD77y2HH8+MAQXh5Kg1LgndeuxKffuNW110vnFcgqnSWtxEI+EAKkHE5ATOZLs8g52Q/ZiGohrdSgmkYOlJZL2GzRzxgZ+fTr8rI2VXJsJIs1ieisPX798aCQVgRtwWhGwj/+7Ag0Ctx183pcvCSOF10u1I9NldrzZ2TkHg/hMm8llZfh9ZBZzrWw3YzcxcXLQJtn5JTSqq4VwNmWICMjnyGtuDGT/NhIBpct75r19f5YCPtOT3B7HYHALQ6eSwIA/vb2S3DN6h6k8woeePoMVI3CWzEClidsV+fMQA7w6e5M5WXEQ75p1kagHIitxhYjkPuF/XAWRVUDpZjVEAQA4YDHtkbO7IczpZWgz4uQ38OtBThXVDA4mcf6/tisP+uPBTGSkbg3HwkEvDkwmAIhwCVL4wCAzUtiKMgaTrk4xXO8NPmwd4a0AuhOE6fzVlJ5ZZY+DugZf8DngWRTWhHTD6vAphtWDeQOBsBnJAWEAJEq1+U5OOvk6BQoLe8arCQRC6KoaK6NzRUIePHiuRTWJTqMvovNS/SAfmgo7dprsjkr1TLyOKeMvFogB/SZ5NY1cpaRi0A+C2YBmtnZCThb95YtKIgGpm8GYfAcZXtmPAcAWN0XnfVn/fEQAGFBFMx9DgymsHV5p/Hv6xd1wOchLgfyIggBuiOzg21XJOA4kOeqjOhg6FuC7LpWRCCfBfOJz5x+CLBip137oTyrGYjBc7nEZKmy3hOd/XjYb3jJhXNFMHe5kC5gJCPh0mXlQB70ebE20eF6Rt4TCcwyCQBMWnHmWqk2VZURDnht+8hFi34Vyhl5NY3cvrQyJak178Y8M/Ja8xyAykAuMnLB3OXAoO5O2TqjYL95SQyHhtzrTB7PSlX1caBc7HQyqygvqzULkyGf9djCNHWRkVehXkbuRFrJVFnzxuCpkSdzRYT8nqo3IkNaEV5ywRzm4GASHgJcXNLFGZuXxDGcLhjr2HijD8yarY8D+rwVjQLZov0n50KxdkYe8nuQn2P2w7YO5EynqlZAcGY/lGe15zPiYR83+2EyN3snIKMj6EMk4BXSimBOc+BcChsWxWZZgN0ueI5XGZjFiHOYt1JPWgnaMFKUA/kclVYIISsIIXsJIS8TQl4ihNzF42BmYG9mbY3cXkNQVtKLndVgGTkPW2C1wfWVMAuiQDAXoZTi4GAKWyv0cQYL5C+7FMjZCNtq8BhlW6s/BdBji1374Vx2rSgA7qaUXgzgOgAfIoRczOG6DWGBnLf9sJFGrmjU0Ro5RrKOxQnQm4JG0kIjF8xNzqcKGJ8q4tLlswN5IhZEX0fQFZ28IKvISkodaUUP8HYDuabR0p6DWhq5x7JrpTjXpRVK6RCl9NnSP2cAHAKwzOl1zZCvk5E7aQjKFOq5VvgNzkrl6mfkiXgQoyIjF8xRDg4mAcwudDL0gif/jLzsIa9d7ARguymIySA8XStzXlqphBCyCsAVAH5f5c/uJITsJ4TsHx3ls4+S3RVrZeSKRiGr1u6clFJkJaWmRt7JcZRtMl+sqZEDrZdWziXz+IefHsaESwUrQXtzYDAFn4dg0+LZncmAXgA9PpK1/BlsBNvV2RutXewEyoOvrFJOEKuHx5DPurWZ+cjn/BhbQkgHgAcBfJhSOus2TCm9j1K6jVK6LZFIcHnNehk5C+5WJZC8rEKjs7cDMYwtQRycK8kGGXl/LISspCDnoPpuF02j+Mi3n8eeX5/AH371tzgz7l67taA9OXguhY2LYzUliM1L4iiqGk6O8v3dMTLyWPVA7nSUrRFXamrkdjo7Vfg8xLXZM1wCOSHEDz2I308p/R6Pa5qBvZnVCgjG3k6Ld87ywKwG0orDjLwgq5AUbdaYzEoML3kLLIjf/N1pPH16An/0itVI5op441d+i2fPTjb9HIK5CaUUBwZTVfVxhlvOlcmSZFKtqxPQk7igz2PbtcKy7Zoaud+LgmLdfuiWPg7wca0QAN8AcIhSutv5kcxTkFUQUr2AYHfdW63tQIw4p+USTL+rK63EW9PdeXY8h8/+7Ahu2pjAJ1+/GQ/uugEdQR/eft9TODmabepZBHOT4XQBqbw8yz9eyZpEFAGvh3sgz5SSqFiodhLkZAJiPTccoNsPi4pmqeFI39fpjj4O8MnIbwTwbgA7CSHPl/73Og7XbQibRT5z1CRgX1qpta+TwWu5BNPvGkkrQHO7OzWN4p4HX4DPQ/CZN24FIQRrEh34zn+/HqpG8a2nzzbtLIK5y9nSnKCLemfPCWL4vR6s6otwn4KYKdRPtgD9c2W32NlIWmEBXrKQlUuKOrczckrpbyilhFJ6KaX08tL/fsLjcI2o5/U0MnLO0kqM0wLmWlu6K2mFtPKjg0N46uQEPnnbZiztChtfX9wZwi2bF+F7z54zrFSChcvZCT2Qz9xsNZP+WIj7E2WmICPk98BfZyRsVzhgv9hZp2McqNjbaSFJlBTNtUIn0OadnfmiVufNtpeRs+1AtYqdAZ8HYb/XcXenmUDeFfEj4PU0VVp57MgIeqMBvGXbill/9parl2N8qohfHb7QtPMI5iYDEzl4CKbd7KvRH+Nvoc0UFKNWVQt9lK29z2i+Tn9K5detxBZJnuMaeSspKGrNTimj2GkxkE810MiBUpu+43nHjaUVQggSseaufNt3egLbVnVXlatetT6BRfEgvrN/sGnnEcxNBibzWNIZbphlJkqBnOeClEyhtj2Y0RXx297bWTDhWqn8PjPo0src1shbRr3BNnZ36zXSyIFSm75TjZwVOyO1i51A+YPQDIZTBQxM5HH1qp6qf+7zevCmq5bjsSMjGE6JjtOFzNmJXENZBSgtSFH5LkhJF+S6hU7A3WIn2xFsJbYUVZGR16SgqDUff+xq5KyQUqtFHyiNsnXqWsnL8HkIojXu+oz+WLBpGvnTpR2h16yuHsgB4M1XrYBGgQefFVn5QubsRA4reurLKoAeyAG+Bfu0mYw87MdUUbVVzzFjPwRgqbtTkjXX5qwAbR7I8/VGTQbsLUnNSgr8XlL3MSgS8GLKYZMOG5hVTcKopD/ePGll/+kJRAPeupayVX1RXLu6B9/ZP+Bo3rOgfckXVYxmJFMZedl5xS8ZyRTkhho5kyztZOVsRG2j+pvlYqdL+zqBNg/k+mCbGhq5jTcb0F0r9WQVAIgGfMhJzoZmpXL1B2Yx+mMhTObkpjhFnj41gSsv6q66daWSt169AmfGc9h/RjQILUQGJnXHygqT0goArvKgGY087qC7kyV/taQQoZFzpiDXllZCNqWVqTq7+hiRoPOMPJkvNtTHgbIFcTTrrrySyss4ciGDbRfVllUYr75kMXwegl8dHnH1TIK5yYBJ6yFQ2dTG76kyU2dfAKM8AdF6wVOPK56qO3uByozcio9cSCs1qRfI/V4PfB5iy37YEayfKUcDPuRs7gNl6EslTGTk7IPg8jjbZ85MgFLg6tXdDb+3I+jDtlXd+PVRPsPPBO2FWQ85AMSCPoT8Hm4ZuaxqKMhaQ2nFybyVepItYE9aKc71Fv1WUm+LB2Bv3Vu2oCDWQFqJBL2GTdEuyZxcd84Kww2NsRpPn5qE30twxYrGgRwAtm/ox6GhNC6IeekLjrMTOUQD3qpLw2dSttDy+f0109UJlJdL2OnubBRXytKKxYxcSCvVqaeRA0DIxgLmrKQgGqz/hkcDPkiKBsXBeM5Uvvaat0rKS5jdDeT7T09gy7LOmt7ZmWzfoE+wFFn5wmNgIocVPZGGhXqGviCFVyBvPGcFKLvOsjYSrrysIlTnc2Cn/ibJqujsrAal1FRGbtVHrmvk9X9JIqX/yDmbiytkVUNWUuo2AzG6S1mPW0tsAf0X8sBgCtfU8I9XY/OSGPpjQRHIFyBnS4HcLImOILcaD7P9NsrI2apGOxJovf4UwKb9UEgr1TE2bjSSVqz6yCUTrpXSn9t1rjDdzkwg93s9iIV8ri53eGEgiaKq1WwEqgYhBNs3JPDE0VFHTyaC9oJSioGJvCl9nNEfD3Kr8ZjNyEN+DwgBcjYz8nqBnAVksyOyVY1C0aiQVqrRqPsK0KUVWxp5I9dKKSO361wxM2elkt5owNVA/kxpzvi2Veb0ccb2jQmkCwpeKK38Esx/xrJF5GXVWiCPBZEuKLZXL1aSNqmRE0IQDfgwZSMjrzeMj1076POYnklu7OsUrpXZ1Fvzxgj5PJYCuaY1lmuAisc22xk5m7PSWCMHdHll0ubcCDMcv5DFks6Q6fMwXrkuAQ8Bfn1EyCsLBSuOFQZPLznLyM0kQZGA19Z2rXyxthuOEbKw3F0qSTCiIagK5ZnBtX+EsMViJ9O8Ig0KfpEgn4zcjP0Q0DNytqfQDU6MTWFNovZc6Vp0Rvy4YmU3HhM6+YKBecitaOTMecVDJzfrWgF0CXTKRrJVMJHMhS0FcpGR14S9iaE6upNVjdyYQ9wgkJcLKQ4DuQmNHAC6I+5l5JRSnBzNYk1fh62/v31DAgcGU8YeRcH8hmXky7sbz1lhJDjO1c802BdQie2M3EQg1/d2mpNW2OJloZFXwZgZ3MAmZEVaaTSHmMHsiXbu9oA+MAuov+atkp6OAManilxHgTLGp4rIFBRbGTlQtiH+5tgYz2MtLG6/HcOEQCEEw4QAt9/e6hPV5OxEDoviwYafkUqM7mQO3Z2ZgoxIwNtwjASgJ1z2MnKtYTJnRVopqvVb/nnQtoHcTEZu1UdupoAKABGHGXkqVwQh5h4PAaAnEkBR0Rx3k1aDbThf3WcvkG9Z1omuiB+/OS4CuWl270bW58NRQvAYIfiLn/8SyyKdCEa6sCwUw/DDD7f6hDUxO762kp5oAITw0cjTJtrzGZGg/Yy80Y0qaCFJLBgZuXvh1tw7MgdpNPwdsCOt1J96xmDSit2MPJXXp7fVmuUwE+Yln5gq1txcZBe2THltwp604vUQ3LC2F08eHwOl1HSTyIJk714U3/MenBkcxOXEA7r+OsQuvxWh1VdiBSl/yD90+jl8aihtbKGfSwxO5HDd2l5Lf8fn9aA3yqe7Ux+YZU6SjAZ8GJzMW7q+qlEUldqbxxghn8eQTBrBNHI3G4LaOJAz10qdYqffi4KimQ4wjZauGtdlDUF2NfLSCFuz9FYEcitFJjOcGptCwOdpuLKrHjes7cNPDg7j1NgU1ti8Icx7du/GxL334rZiES9sfhW6d7wfvlgflMw40k99F0pmHACFN9KFozvfidd98Qm86crl+MTrNhs38lYjKSqG0gXLGTnAb+WbmcmHjEjAa9lHbjzpNyhMhvxe03Ur5lpxUyNv20DeaEEqoAdcVaOQVYqAr3EgL5jUyAM+DwJejy2PKmB+YBbDyMhdKHieGJ3Cqt4IvCafDqrxinV9AIAnj4+JQF6N3bsx/LGPYXMoDv/rdyGx4XpI549i4hdfRf7EPoBOz+x+teev8ZPTKv79d6dxenwK93/gOlezObOcm8yDUmBFt/VAzmveSqYgm7bJRoPWfeSmkzm/F0NzyLXStoGcWQXrBV2mSeVNzjnIm9TIgZL+ZnNwVjIvo9OCZ9vIyF2wIJ4cy2JDf8zRNS7qjWBZVxi/OT6Gd1+/is/B5gu7duHsnj24av316HjdXSBePyb3fgPpfQ8ZAfyTADYBoACue/pprNuyEVduAbYu78Rd334en/rxy/i7O7a08qcAAEOmsPNU2B8L4vBw2vEZMgXF9Ovbca002g7EsOJaMRqChLQyGzNveOUCZjMNBGZ0d4bdrjFAL3ZeZOHDYMxb4ZyRK6qGs+M53HrJYkfXIYTgxnW9+NmLw1A16ii7nzfs3YviBz+Ik8eO4/pXvQed178F0vmjGPvRP0GZPA8C4IXVq7HsiSfQs2xZ1UvccfkyvHguha89cQpblnXiLdtWNPdnmAEL5Fash4xELIixbBGaRk3XhqqRtqKRB32QVV3zNvtEY9bwELJQ7DQycmE/nA17cxpp5ID55RLlm0Pjt8WuRxWwrpHHgj74vQTjnNv0BybzUDRq27FSyY3r+pAuKHjxXIrDydqc3bsxcdtt2Hn2PLa/+W/Qef1bkHn+Zxh+4B4ok+fRBeD04cPYevJkzSDO+Nitm3Djul588vsv4kCLRyEMTubg8xAsiocs/93+WBCqRh3Lg/qaN/MaOWCtlmX2qdxSZ2eDjUM8aNtAXpBVEFK/7dUI5CbfcGvSij2PqqbR0ghb84GcEKI3BXEO5MyxwkPXvmGtrpMveBvi7t0Y/sQnsD7Ug1Pv3o3Qiq0Y/9n/h4mffwlQFWyPRHBiaAgrN240dTmf14Mvvf1KdEf9+NSPDrl8+PoMTuaxtCts64mrvxT8nRQ8JUWFpGimi52Gu8yFpsCg37prRQTyKkiKhpDPW9eNwpqFrAZyM80OUZsZeaaggFJY0sgB3YvLe3AW85CvtdkMVEkiFsSmxTE8uZAD+a5dOHv33di6ZDOi7/4neIIRXPj2vci+8HN4ABy95x78YmwMPYutSVnd0QB2bV+Lp09P4KmT4+6c3QSDkzlbsgpQ0d3pIJCX2/PNJUFslIaVWpbZGBD2e1FUNagmFpALaaUOZga1s2Yhs49AhaKe5Zu5c0Zsdo0l2cAsCxk54FIgH5tCTzRgeVhWLW5c14f9Zya5TLlrO3btwrE9e7D1itcj8ua/hZIexdA3PwLp3GF0Azh1+DDWf/azCITtBcK3XbMSfR1BfPHRY3zPbYHBybztQG4sSHEwztbKnBXAXkZuRSMHytbCeojph3UwM6idad1mH4HYjAUznvOoza4xq3NWGN3RAHf74cnRLBd9nPGKdX0oKhr2n57kds05z969KG7ciJfv+zque/WH0P3qXcif2I/h+++Bmh7FTeEwjluQUmoR8nvxwe1r8NsT43jmzASnw5unIKsYyUhYbsN6CFRMQHQwk4dNPmy0r5NhaOQ2MvKGLfpsJrmJ2CKmH9bBzFbqkA2N3Iw+DpQychuulaSFpRKVuDGT/OTYFNZwDOTXrO6B30vw+LEFMg2RFTUHhrHzrX+P2BWvRep3/4nR738atJjH9kQCDw8NWZZSavGOa1eiJxrAFx89zuV6VjiXZNZDexl5JOBDR9DnaHCW5Yw8aEcjN9fdbSW2SIoGv5c4cus0om0DedHEMlOr267zRc30MKCoja4xAEiWsupOkwOzGN2RAFJ5mds2nkxBxmhG4trAEw36cM3qHjx2ZITbNecsJT18fXwZTr/3Cwgu3Yixh/8Jycf/HR6q6Xr4mTPo6Ozk9pKRgA8feOVq/ProKJ4fSHK7rhnK1kP7ncWJmLOVb2a3AzHcdq0A5mKLJLu7eBlo40AuKWpDaaW8JNVc8Cs02AxSSSToQ05WoZkodlSSLmXk8bA1C39PNABKyxm9U06NORuWVYsdG/tx9EIWg5M5rtedM5SklKN79mDLZbei453/CIBi+D/+AlMvPwYPgCMHDjjSw+vxnutXoTPsxzd+c4r7tevB/nva1ciB0u5OB8VOs9uBGEZGbqGWZXR319lzAFgM5CZilVPaOJA3Nvkzjdx0Rm5BWokGvKDU2gJWoPzLaFbnY/RwXsLM07FSyU0b9bG2j83HrUG7d2Pi1lux4/QAbnj9R9Fz65+icPYAhv7twyheOGEUNddt3eraETqCPvzBZUvwyMvDtjbE22VwMg+/lxhLIuzQHfU7+v01kiAXM/KCrMLTwNYMVMaWxkli0eXFy0A7B3LZTLHT2rbrfIPt2ZVEbNztAV3nC3g9luY5A+VAzksnPzk2BQ8BVvbyHcK1NtGB5d3h+RXIS1n4mbvvxrqei3D6vf+M6MXbkfzN/Rj57t9CK2SwPRLhUtQ0wx2XL0NB1vDIy8OuvxbDiYec0RMNOupONpZKmG4Isv4ZZTGgkeHBcK2Y1MjrLYnnQfsGckVtqDsFLVSWgdIcYpPSStTmBMSMhXnKlfAO5GfHp7CkM8xduyOEYMfGfvz2xJgpa9acp5SF7zx2HJde+ybE3vU5EJ8PF771CaSe/BYI08Nt+MPtctXKbizrCuOh58835fUAZx5yRk/Uj8mcbFmOZGQKCqIBr+mbiddDEPJ7LGvkZuRVa8VO1VXHCtDWgbxxRm5suzY9AF5F2KTX087dHmCzIhwEck4WxMHJvG0HQiN2bEogV1Sx71Qb2xArsvA1HQmceNc/ovum9yF37HcY+tc/gzT4kt7k46IeXguPh+D2y5fiiWNjTVuxNziZx/IuZ09vPVG9TT9dsFfnyRRkxC32X+gzkawFcjPJjRVpxYzDziltG8iLJh9XrCxgNrN0lcHWvdnLyK39MgK6awXgNwFRb+7gK6swrl/Th4DPg73t6l7ZtQvDO3dix7Hj2Hr1GxF/3xfh616K0R/+I8Ye+iw0aQrbvV6cO3nSVT28HndcvhSqRvGTg0Ouv1ZBVjGakbhk5ID9p0ors8gZ+pRSa8VOUxm5hWZDMzKwU7hcnRByKyHkCCHkOCHk4zyu2QhJ0Uw9roR85gO52ccqoCIjt+glzxQUy44VQJ+BHgv6uGTkkqLiQqbg+INZi3DAi+vW9LZfIN+7F8UVK/DSnj1Y2b8aJ9/9v9Gz849QOPUszn/jT5A79DgISq32mQwWr17dsqNuWhzHpsUx/OC5c66/FvOQL3f4BGckI3YDuWQ9CbKckZuskxmTVc10dqptYD8khHgBfBnAawFcDODthJCLnV63EZKimnpcsTI3OF9svKuPYadrDChl5EHrGTlQ6u7koJEPJQug1JknuBE7NiZwcnQKZ8fbxIZYysJfNTyKV2x/L5a89wvwxfow+oP/hdHvfxraVNKYWthsKaUWt1++FM+eTbr+HvPwkANAb1Tv7mxqRh7wWtp1a9a5Vs7IzXV2tkNGfg2A45TSk5TSIoBvA7iDw3XrYvZxxcq4yYJspSGILWC2npHb0cgBfvNWnMyVNsuOjf0AMPez8oosfPXqKzHwR19G53VvxtSLj+L8N3Yhd+RJEABHdu7EwPh4U1wpZrn9sqUAgB++4G5WzsNDDuj2Q8D+XP103kZGHvRhylKLvmbK8BC0YG2WZPPz0O3C4+rLAAxU/Ptg6WvTIITcSQjZTwjZPzrq3JommXxcMbvtWlE1FNXGS1cZEdsaufnB+DPpiQa4LJfg9cGsx6q+KNb3d+DHTdBwbVPKwl85kcFNf/DnWPSWvwNUGcMPfBzjP/0itEIWN3o8OH/yJDY8+ig6enpafeJpLO+O4MqVXfj5SxdcfR0eHnKgnJHbnavfjIy8UDRneAj6PCDEbENQm2jkZqCU3kcp3UYp3ZZIJJxey7TJ3uy264JibsYCw85kNVWjyEoOM3IOxc7ByTx8HoLFNhYEWOGOy5fi6VMThsY6Z9i9G9lwGAf37MG6LTtx/gNfRWTDjUj+5gGc/z9/BmngRRAAL37kI/hVNttSLbwROzf14+C5FJfFxrUYnMxjmUMPOaDryiG/x3ZTUKagWG6ks+NaMRMDrDjizFilncIjkJ8DULmDannpa67B5vuaeVwJB7ymChLGdiCTxc6QX78jW9HIsxZbjGfSw2kC4uBkDku6QvC57G29/TL9wezhF5rnd25IaUbKikAMt7zl79D3+o9CHh/A+X/7M6SefABQFSMLv2T37jmhhdfjppKE9euj7jVg6R5yPvWU3mjQVkZekFUUVfNLJRhWXStWDA9hv9d8Z2cb2A/3AVhPCFlNCAkAeBuAH3K4bk2sbNww61oxO4eYQQixvLczbXEM50x6ogEUZM32ijkGD0+wGVb2RnDFyq6mNq7UpCILv+Sy1yD2/i8juHQTxn/xFVy4/+NQxgfhQXtk4ZVcsjSO/ljQ1VqEkznkM7Hbpp8xRltYlVasZeQFC4YHs/W3ZkgrjpcvU0oVQsifAvg5AC+Af6WUvuT4ZHVgHYNmfORmXStW1rwxrO7tNH4ZbdgPAaCnwr7F7I92GJzM45Xr+2z/fSvccdlS/M3DL+PohQw2LIo15TVnUdpkf1m0G4E3fxy9a7ahcOYFjP3kn6Gm9QB4YM0aLHv88YY7NOcahBBs35DAz18ahqJq3J+yeHnIGd0RewV7q5MPGZGAnjWbXQpuZd5SyO81JNl6mJkL5RQuV6eU/oRSuoFSupZS+mke16xH0UpGbvKuWd7VZ/4tiVrc22n3l5HRzaFNv+whdz8jB4DXX7oUXg/BQ8+773eeRak789iePdi4Zhs63v8lBFdswfgvvooL3/4k1PQIugAce+45bD1xou2COGPHpn6kCwqec2G0reEh5yat2JMHrc4iZ7BalhnDg6xqUDRqOpAHfZ6Gi92V0jq4dtDIm44lacWka8XKvk6G3YzciUYOOAvkZQ95c7TfRCyIG9f14aHnz4NSezM2bFFa+nDL8ZO4ducHsOjNfwM1O4Ghf/8wss/9GADFwZtuwsD4ONZdfnnzzuUCr1jfB6+HYO9h/vLKmXF9SiavcQ7d0QAmp6y36KftZuQW9naa3Q7ECPm9DecJNWPxMtCugVy2FsjNuFbsSCtRi3s7M5KzjNwYZeug4DnQBOvhTO64bCkGJ/N49myTZq/s3o3hj30Ma71hHHvnPyB+9RuQ3v9DDH3zo1DGB/FKAGdeeglb9u6dc5ZCO8RDflx1UbcrEyfZuOM1fXwWkPRGA8hKiuWBak4zcjO1rILFZE6Xbetf14p64IT2DORMIzc53MbMtutC0drdGChVxC1k5Ok8n4x83IEF0WgG6mmOtAIAr75kEYI+D76zb9D9Fyu5UrYsvRgd7/1n+HtXYvT7n8Hko/cBqoxX9fXhJ8kkVl7sevNxU9mxsR8vD6UxnLK/3LgaJ8em0B3xG7KeU7qNufrWsvKyLGndRw7AVFNQweSaN4YZ14qRkYsxtrOxKq3of6f+ndN2Rm7BtWL3l5ERD/ng9xKMOQrkOfg8BItKy3CbQSzkx5uuWo7vPTeI8256ytkm+6vfgOhb/x5qPo2hb34UuaO/RSeA0/fei0fOnuW6fm2usGOT3pvx66N85RXeC7p7WTIyZc33zpKgTqvTD4PmO7DtSCuNMvJmLF4G2jSQG48rZmatmJxJbtu1YsFHnikoCPg8tgsfhBDH67IGJ/NN8ZDPZNdNa0Ep8C+/PsH/4qWi5ktf+zquf91d6N75AeSOPYXhb34UysQgugAcPnMGF33mM3PeF26XjYtiWBwPcZdXTo5Ocd3rygZnWc3Ik/kivB6CjqDNjNzEk3O5Tmbus6G7Vkxq5G3gI2865Yyc3wB4FujNNgQBJdeKJR+59c60mSTiIYxk7D8+N8tDPpPl3RH84ZXL8a19AxhJc3z8370bE294A24ZHMHNb/sMOrb+FySffABjP/gHULmA7ZEITgwNYfHKlfxecw7CbIhPHh9rKCOaJVOQMZKRsIbjOsAemxl5MiejK+xvuLlnJkZGbqKWZTQFWtDI88X6CWLRQqxyQpsG8tLjisnOTqDxTASrDUGAHdeKbLmhYSb9MacZufNNL3b5kx1roWoU//L4ST4XZJvsgz04+Z7dCCxag9Ef/C+kfvMAPKBN39zTam5Y14t0QcGL51Jcrnd6TC+Mr+EordjdPZvMy+iMWE+CrGTkVmNA0OdtuOqtXM8TGfksrLhWgiYHwOeLKnweAr8FySEa9EFWqXHXbYSTyYeMhINALikqLqSlpnnIZ3JRbxR3XLYU9//+jPPNNrt24fiePdi87hpE3/U5wOPFhfs/htyRJ+GFu5vs5yo3rNWbvJ48McbleifHsgDAVVrpigRACDCRsyatpEoZuVWMKaUu2A/NjP9gsaotGoKajTVpxbxGbnUhstUt3Xa3A1XSH9NnVciquZtHJeeTuqTRqowcAD60cx0kRcOX9x63d4GSHn50zx5su+7N6Puvn4Q8PoDhb34ExQsnsMPrxWALN/e0kkQsiE2LY3jyOJ9AfmJ0CoQAF3Fc0O31EHSF/ZiwKq3ki+iKWHfOMB+5GQnUaAq04FqRVQqlzmdR+MjrULTwuGJ227WdQG51AqLdfZ2VJEpuEzsWxGaMr23E2kQH3nntSvyfJ0/jZy9aHHFbavK5+fQAXvGGT6Br+3uRO/QELjxwL9TsBLYnEvjh+HjbzElxg1es68O+05OmZ/DX49TYFJZ381/QbacpKGkzIw94PfB5iKlky6rhgX1fvTZ9q950u7RlILdSCTZd7CyqltrzAWtdYwDLyJ1q5ProWTsFz1Z4yKvxV7ddjMtXdOHu77yAYxcy5v5SSQ9fG0ng5Ht2I7z+Wkw8+jWMPfw5EEXS9fAzZ+altdAKN67vQ1HRsP+08+ark6NZbo1AlfRGA5aLnamcPY2cEIJIwGuqcc8IuhYmoAKo26Zvxw1nh7YO5Ga8mcZd04S0YvXNtpqRO1kqwWAZuR2dvBUe8moEfV7seddViAR9uPP/PoNUvk52dvvtGCYEz9z3dVx649sRf98X4Al14MK3P4nM/ofgwcLUw2txzaoe+DzEsU5OKcWpsSmujhVGd8RaRi6rGjKSgq6wvaakaNBnLiO3KK2E/I3rb4YbTgTy2UiKCq+HmPJCh0yuZLITyK3s7VRUDbmi6th+2F8KwiM2AvnARB6LO5vvIa/G4s4QvvrOKzEwkcO7vv57PHNmovyHu3ZhghA8Qwh++PCPsHrVFfiD934eXa94J3KHn8T5b3wI0sBBdAE4dfjwgtTDaxEN+nDlym7HOvmFtIRcUeXqWGH0RAOWZpKnSzf6LhsZOaB/Tk1p5LI1wwMritZ72m9WRu54jG0rMLsdCKi4azbq7LQwh5jBPKpmfkmykrP2fEZfh/2MfGAyh5UtllUq2baqB196xxX464dewh9+9Xd47ZbFePtf/THu0lQcvugyBBavR8dlr8Gi7iVQMuMYefDvkD/+NACgC8ChM2fmvT/cDjeu68MXHj2KZM5egRDQZRWAr2OFwVYWUkpN+cKTDgN5NOgz7VqxEnDZ99aTVlgC6XZDUFsGciuD2s1uuy7IquV5ElZcK04nHzICPg+6I35bGvnARA63bF7k6PV5c+uWJXjVhgS+9vgp/MvjJ/DTt/49AICdsnD2IJJP/F/kjv4WUPX38OBNN2HVgw/Oi4FXbvCK9b34/C+B350Yx2u3LrF1jRNj+rAsnu35jJ5oAKpGkS4oplrumfRmtT2fYTYjL8iqpYbAsIn6W0FWQYj7rpX2DOQWtlKb3Xadl1UstZuRmyik2B3DWY3+WAgjaWsZ+ZSkYCxbxIo5lJEzIgEf7rplPd513Uo8/N5d+OC+fQClULPjUJLDxvddC+A7L7007wZe8ebS5V2IBrx48sSY7UB+cjSLsN/ryl7XynHMpgJ5jmXkNjXygA8XTCQ++aK1jDxkQlopyCpCPq/ljlSrtF4stYGVZaZmt1070shNZORs6I/Tzk6g1BRksaGGOVbmkrQyk96OIP519DikgRchDb5kBPF/A3D6nnvweC4ngrgJ/F4PrlvTiyeO2dfJT41NYXVfFB6HC5erYXVBSjKvf58d+yEARII+Uy36BVmzJa3UszZb2QHqhDYN5OalFUKIqb2d+aJm6bEKgLFuzUxG7nQ7UCX9saDljPzshO4hn8uBHAAef/xxvHLRInwNwA0AnnvkEbyXUlwkXCmW2L4xgTPjOZwuSSRWOTk6hdUuOFaA6SsLzZDMOZNWogGv6aFZ/KUVzRjc5yZtKa1Y3UptZm9nwUZG7vUQhPyepmrkAJCI6xm52WIRUA7kc1Famcnjw3om/oEWn6Od2b6BjbUdxSqLOrekqBiczOENly9142iW562wQB63rZGby8j1p3IrcYUVO2vHFqs3B7u0cUZuQctqMDeYUmpLWgHYTHIzgdzZL2MliY4giopmyDVmGJjIoSPoQ7fNyr+gvbioN4pVvRH8+qj1sbZnx3PQqDuOFaBCIze56SqV14fNmVmeXI1oUM/IG60aLFjs7jaTkUsljdxt2jSQq5YGtTfadi2rFKpGbWlZkaDX1N2eZ0beH7fe3Xl2IocVPRHXiy6CucP2DQn87sS45Xb9I6VuWzeagQC9thT0eSxIK/ZtlPrr+aDRciNhLaYkxah7mSEUaGykEBp5HSSL0kqjbdfMY26n+8p0Ri4pCPk9lqYr1iJhw0t+diKHlZwW6Arag+0bE8jLquV2/f2nJxHye7BpcdyVcxFC0BMNWCh2yrY95ICekQON171lJQWxoPnXCXg98JBGPnLN9KIKJ7RnIJfNFzsBvQOr3qq3gsXW3Er0meQm7Id555MPGf1xa92dlFIMTMytZiCB+1y3phcBnwePHbG2/m3f6QlcsaLb1dGrlgJ5TrZd6ATKpoRGn9NsQTEsxWYghCDs99bv7LRoabRLWwbyompRI2/gWrG64qmSaNBnarErj1nkjH6L81ZGMxIkRROBfIERCfhw7eoeSzp5piDj0FAaV692t9nKSpt+Ki87klaiJpZLaBrFVFFFh8XPaDhQP5AXFNX1xctAmwZySVYtZeSNXCtO5iFEAz6j/b4eaQ6zyBkdQR9Cfo9pjZw5Vlo99VDQfLZvSODYSBbnTC69fubMJDSqD99yk0RHEGMmE5FkrmjbQw7oPnKgvk2YBfmYxZ2gIb/XeKKvRkFk5LWRFPOdnUBj14qxq89GUaIz7K8/va9EpqBwaQYC9Ee6/ljIdEbeLh5yAX+YDfFxk1n5vtMT8HoIrljZ5eKpShbajNTQSaJptJSRO9DITTTusWTMckbeYAFzQREaeU1s2Q/rvNlOMvLOiNlALjuefFhJIhY0rZGfnciBEGBZlyh2LjTW9XdgWVfYtE6+79QktiyNW9KK7dAfC6GoaoZHvBYZSYFG7TcDAeYa97IlV1mHxZ87HPDWn0cuMvLaSIpqsSHIW9e0b2fxMqMz7EdB1hpavHhq5ECpu9NkIB+YyGNxPOT6TGTB3IMQgps2JvDEsbGGEqCkqHh+MImrXZZVAPPjmMsjbB1o5MHGGXlGshfIQ77aGjmlFAXF+lRVO7RdINc0ClmlljXyuvMQSkHejt+TPfI1ysp5B3IrS5gHSh5ywcLkv165DLmiip8cqL9a78BgCkVFw7amBvL6dR6WsTvSyE0sgJmyKa2EAl7ka9TfJEUDpe4vlQDaMJAXVfOLlxmuSivhxoFcVjXkZZVbsRPQPwipvGyq2eOssB4uaK5c2Y21iSj+3/6But+377S+3OPqVd2un8loamswM8gYmOWyj9y2tOL31Cx2Sk3aDgS0YSBnb46lYqfPa3RvVqNsP7SRkZfWT9XT+nh2dTLY7s6xBlMQC7KK4XQBK7pFIF+oEELw1qtX4Jkzkzg+UntH6r5TE1ibiKK3w/1VgGalFSMjdxDIw34vvB5iBOtq2JVW6vnIm7UdCGjHQF7KrK1KK0DtVlqjIcimawWon5HznHzISJj8IBjja3tFoXMh88YrlsPnIfjP/YNV/1zVKPafmcQ1LvvHGdGgD9GAt7G0knc+o4gQgnjIZ+wEqEbWZrJVz0decNCfYpU2DORMWrHW2QnUHm5jZOQ2OtlYppCsMwDIjYzcCOQNHk0HhPVQAP33Zeemfjz47CBkdbame2Q4g0xBaUqhk9EfDzVMRFKlz5UT1wqg3wjSdZItJrtYdevU85GLjLwORiC3ssnDWPdW+w0PeD22lhJ3mih2skyAp/3Q6O5sIK0MTLbP+FqBu7z16hUYyxbxq8OzrYj3//4MCAGuXdPbtPMkYkGMNtLIc3JpyJazYBgP1bcJZyUFQZ/1WUj1fOQFB5KtVdowkFuXVsrr3qpXl/XFy/beio6ADx7SSFrhn5H3dgThIcBouv6j6dnxHEJ+jzFoS7Bw2b4hgf5YEN/ZN73o+dvjY7j/92fxRzeubmqvgW6hbSytOHGsMOJhH9INNHI7n8+wX6+/VXvKcVJ7s0obBnIbxU5//Yy84GDUpMdD0Bn2myp28szIvR6CpV1hnGywAebMRA4rusX4WgHg83rwtmtW4tHDI/jczw+DUoopScE9Dx7Aqt4I7n71xqaepz/WWFpJ5mR0OvCQM+Kh+tJKtqBYLnQC9WNLMzVyRykiIeRzAP4AQBHACQD/jVKa5HCumjDXirViZ2m3Xp1HICc6VqM2ffYLxDMjB4DNS+I4NJSu+z2HhtK4bEUX19cVtC//Y+c6jGYK+PLeExiczCMa9OFcMo//d+f1TZmbXUl/PIhcUUVWqh1EU3lnc1YY8ZC/frFTUix7yIHpC5hnmhmYAtAO88gfAbCFUnopgKMA7nV+pPqUpRUrGrn+Y9bq7sxb3Awyk85IwKiuVyOVl0GIO4H81NhUzSeNiakiBifzuHRZJ9fXFbQvPq8Hn3njVvzFazbioefP44Hfn8X7bljVNLdKJYYFsY48mMw5m7PCiId9dTdqZSUF0YA9aQUAClViS97BeGyrOArklNJfUErZu/MUgOXOj1SfogPXSu1ip+bortkV9hvV9WqMZSV0RwK2iqn1uHhJDBrVHQfVOHguBQDYulwEckEZQgg+tGMdvvj2K3DbpUvwF69prqTCYL0Q9eQVp0slGPGQH3lZNeLHTLI2O6/rrXtzsrDGKjwjy/sB/LTWHxJC7iSE7CeE7B8dtb5HkME0ciu6k6Fj1ZJWis726jWSVsazRfRGnet8M9m8RN/gUkteOTiYBABsERm5oAq3X7YUX3rHlUYLe7NptCCFUn3yYWeYg0ZekmcyNeSVevJOPcKldW/VArkxVXUuBHJCyC8JIS9W+d8dFd/zlwAUAPfXug6l9D5K6TZK6bZEImH7wEax02vHflhbWnGUkUf8daWVsayEPhdcIyu6I4gGvDUD+YHBFNb0RbkWWQUCXjSSVgqyhqKicZNWANR0rtjWyFlGXsVLbifptEvDk1NKb6n354SQ9wG4DcDNtNFwYQ4YGrmljLx+Z2eeQ7EznZehaRSeKpu+x6eKuGQp//2HHg/BpiVxHBqqLa20QvsUCMzQGfYj4PPUHP5mzFnhUOxkDUW1nCtZydqaN0a4jmslX1ThIbC0KN4ujl6BEHIrgHsA3E4pzfE5Un3suFaCDeyHuo/cWSDXaHlew0zGMu5k5ACweUkMh4bTswb0j2QKGEoVsFXIKoI5CiEEiY7a45h5zFlhsKfSas4VSdG1c6vbgYD69bdCyUTRDOuv01vFlwDEADxCCHmeELKHw5nqYmf6Yb27Jvs607rsUO9uX5BVZCQFfR38NXIA2LQ4jkxBMWaqMF4sFTovXd7lyusKBDzoj9duCmKBnKdGXs25whZO2NLI6xQ7nT7pW8FRlYNSuo7XQcxiZ/qh30vgIQ00cgdvOBt6n8zJWDFDyWALZt3LyMsFz8o2/AODKRACVyQdgYAX/bEgTo5Wb2pL5fnMWQHqZ+TGCFsbtaRQPdeKrDVtmUsbdnaq8HkIvFW06FoQQmru7aSUcgjkpcFZ+dkWxPHSLBS3RoNuWhwDIZilkx8cTGFdosP1lV0CgRPqdXdylVZYsbPKU3NG0r/mpLOzWrFTl1aaE2LbMJBrlvRxRq3lEsYWDweulXqjbNm8cLeklWjQh4t6ItOcK5RSHDiXEv5xwZyn3oKUSUNacR7Iw34vfB5SNSPnIa3U08ibQdsF8qKiWZp8yAj5PFWllVzpThpxkpGH2SjbaoHcXWkFKLXqD5cD+YW0hNGMJDo6BXMe5iWv5lw5OzGFnmiAy1MlIQTxGv0eWZaR27Af+r26OtBqjbztArmkqPYy8hoD4Nl/2E4Hj29xExl5r0sZOaAH8jPjOWO57oFSI9BWUegUzHHqdXceu5DFuv4Obq8VD1Vv08/YXPMG6DeIcI3l7iIjr4NtacXnrbqA2QjkDh7fQn4vQn5P1UA+ni0iEvC62j3HCp5HSln5wXMpeD0EFy8RhU7B3IYtSBmd4VyhlOLYSBbreQbycPXBWSwBsjsLKVRj3Vu+icXOtquESbJmybHCCPmrSys8Ajmg7+6stiXIra7OSjYviQEAHj00gpfPp/HwC+exvr+j6dPsBAKr1GrTH81KSOVlvoG8xijbKZv7OhnhgKeqRi41sdjZfoFcUW1tC6nlWkkaq6ScSR+15q2MZ4uuyioAsKwrjHjIh688dgIAsLQzhA9uX+vqawoEPOiN6gtSZq4sPHYhCwDYsCjG7bXiYR+Gq4wDyBYUEAJEbCY+4RqxpW185K2gqNp3rVR7rEpzysg7I9WXS4xlJdfXrBFC8M9vuwIX0gVcv7YXK3vEIglBe+D1EPR1zG4KOnZBt9OuW+R+Rp6RFHQEfLY/M7WklWZq5G0XyCVZszRnhaHfNd2TVjrDfmPRcSVj2SKuWNnl6Npm2LGp3/XXEAjcQO/unJGRj2TRGfZzXVFYUyMv2BuYxQj5vVV95E6H8VmhLYuddobQBP2eqm92Micj7Pfa0t0r6aoiragaxcSU+xq5QNDO9MdCs6WVUqGT55NlPORDQdZmbQqzO8KWUU1aoZTqnZ0O44pZ2jCQ29fIq616S3EaXN9VRVpJ5orQqLsecoGg3VndF8Xx0SxyxbI18PhIFus5yipA5Uzy6RZEuyNsGeEq0ooxwlZk5NWRFHvSSshXW1rh0TnWGdY3kFTeLFgzkNvFToGgndmxsR9FRcNvj48D0MdaTEwVsa6fX6ETqJi3MuPJ2XFGXqVHxVi87GBhjRXaLpAXbbfoV7cIpfKycad2Atv0XSmvjBvt+SIjFwhqcc3qHkQDXjx6eASALqsA4Go9BGovl8gWnAXyUJWGIBbYhUZeA70hyPqbE/Z7oWgUsjr9DeeZkQPT7/ajLs9ZEQjmAwGfB69cn8CvDl/QG4FKjhXu0opbGXkVjZw9/YuhWTWQZNVmQ1D14TapvMxlA0m1eSvjTZizIhDMB3Zu7seFtISXzqdxbCSLWNCHxfEQ19cwkq1ClUDuRCMPeGZJK8xYIWat1MD+9EO27s3djLxSWhnLSvB5iNiZKRA0YMdG3T679/CIPmNlEV/HClB9uQSlFFlJsbUdiBH2e6HOeNovGCspRSCfhapRKBq1Ja1UW/dWVDTkiiqXQG7MJJ+Rkfd2BKru8RQIBGUSsSAuW9GFRw+PcJ+xwmAJVWWylSuqoBSOJixWWy5REBl5bYolS48t10rpDa10lfCYfMjoKrX4J2dk5L1RIasIBGa4eVM/nh9IYiwrYT1nxwqgP5X7vdNnkrOBWU4bgoBy8AbKGbmYflgFFoTtTT+cLa3w6uoE9MlphMyQVqaK6IuJQC4QmGFnRXcyz9Z8BiFkVpu+kxG2jGp7O5mLRWTkVWAmezvFTmYDqnyzeQZyT0kLT1VMQBzLSOiLCseKQGCGS5bGsag0DdENaQVgbfpljXzK4QhboHpsMXzkwrUyG7Z42W5nJzBdI+c1MItROQGRUorxKUlk5AKBSQgheM0li9ETDWBpZ9iV19CXS8yWVqIO9gUYGXmFtGL4yMXQrNk4kVZYoaOyGJnkuKUbKLXpl35JpooqCrKGXpGRCwSm+fhrN+HOV61xzSAwc3CWIa1w0MirZeTNcq20WSBnGbn1QL6kS/ekDqXyxtdSxpZuPsG2M1yetyK6OgUC60QCPle3acVDfpxPlmOAsR0oaD+ZY9JKoUogFxp5FYxAbuPNiYf86Aj6cD5ZnnucKvlJ4w7uxpV0hv0YzUhQVK0puzoFAoE14mFfVY3c6dAsANPa9AuyBg/RlzM3gzYL5Ppdzs4YWwBY2hWadjdO5WV0BH3w2bzeTG7a2I9zyTw++p0XcCEtMnKBYK4x07ViaORB+5lzudlwukYe9nubtuClPaUVm5XgJZ1hDKXKGXkyX+SmjwPAm65ajtGMhM/+7DCePjUBQARygWAuEQ/7ISmasb0nU1AQ8HpsGSgYVe2HTdwOBLRbRi7b18gBYGlXeJpGnubUnl/JrpvW4i9es9HYDdgjip0CwZyByaisyJmVZEeyClCeOT5TI29mIG+rjLyo2rcfAvpS4rFs0XiTec1ZmcmHdqyD30vwzJlJx5uHBAIBP+IVg7MSsaDjEbZAdfuhHmOa99lvq0AuyfbthwCwpEv3pg6nCljVF0UyJ2Ntwp3GgztfJbbYCwRzjZmjbLOS6jiQ+70e+Dxkhv1Qa9oscqDdpBUH9kNAL3YCMAqebmXkAoFgbjJzuQQPaQWYve4tX1Sbth0IaNtAblda0TPy86WCJ699nQKBoD2YOQHR6VIJRigwfblEQVFFRl4Lo7PTpva0uLPUFJTMoyCrkBSNy5o3gUDQHsQrNnllCjLGs0UugTzs905v0S/aWxJvl7bSyNkYW7s+8pDfi76OAM6n8lwHZgkEgvaAZeTf3ncWn/3ZYWQKCq5e3eP4uvq6t3JDkKQ0VyNvq0AuKRoCXo+jOQxLOsM4nyyIQC4QLEBCfg86gj68dD6N121Zgg9uX4utyzu5XHemRh4WrpXqSLLm2M63tCuEk6NTRiAXGrlAsHAghOCBP74W8ZAfq/qi3K4bmlHsLCiiIagmkqLadqwwWHcnG5glMnKBYGFx6fIurkEc0AdnFWZl5G0WyAkhdxNCKCGkj8f1avGR/7IB3911g6NrLO0KISspGJjMARCBXCAQOKey2KlpVF8S306dnYSQFQBeDeCs8+PUp68j6Hh2ydJSU9DhoQyA8q5NgUAgsEulj5zZpNstI/88gHsAUA7Xcp0lJS/5oeE0CHG24kkgEAiA6T7yZq95AxwGckLIHQDOUUpf4HQe12HdnUeGM4gFfa5tIhEIBAuHSmml2WveABPSCiHklwAWV/mjvwTwCeiySkMIIXcCuBMAVq5caeGIfOmPheD1EEiKhv64GDErEAicw6QVSmlFRj6HAjml9JZqXyeEbAWwGsALpeHpywE8Swi5hlI6XOU69wG4DwC2bdvWMhnG6yFYHA/hXDIv9HGBQMCFkN8DjQKySo2MvC3sh5TSg5TSfkrpKkrpKgCDAK6sFsTnGkxeEY4VgUDAg/64HlN+9tKw0eHZNhp5u8IKniKQCwQCHrzximW4YmUXPvG9gzg8nAbQfq4VAEApMx/jdT03WVLKyMXALIFAwAO/14MvveNKeD0En/rRIQBtIq20M8tKXnLRni8QCHixrCuM//3my8quFTHG1l2EtCIQCNzglosX4c5XrQEhQHekeWaKBdkNI4qdAoHALe597Sa85/qLkIg1z968IDPyjYti2HXTWty8ub/VRxEIBPMMQgiWd0ea+poLMiP3eT342K2bWn0MgUAg4MKCzMgFAoFgPiECuUAgELQ5IpALBAJBmyMCuUAgELQ5IpALBAJBmyMCuUAgELQ5IpALBAJBmyMCuUAgELQ5hNLm73gghIwCONP0F3ZOH4C2mPDIiYX28wLiZ14otOvPfBGlNDHziy0J5O0KIWQ/pXRbq8/RLBbazwuIn3mhMN9+ZiGtCAQCQZsjArlAIBC0OSKQW+O+Vh+gySy0nxcQP/NCYV79zEIjFwgEgjZHZOQCgUDQ5ohALhAIBG2OCOQ2IITcTQihhJC+Vp/FbQghnyOEHCaEHCCEfJ8Q0tXqM7kFIeRWQsgRQshxQsjHW30etyGErCCE7CWEvEwIeYkQclerz9QMCCFeQshzhJAftfosvBCB3CKEkBUAXg3gbKvP0iQeAbCFUnopgKMA7m3xeVyBEOIF8GUArwVwMYC3E0Iubu2pXEcBcDel9GIA1wH40AL4mQHgLgCHWn0InohAbp3PA7gHwIKoElNKf0EpVUr/+hSA5a08j4tcA+A4pfQkpbQI4NsA7mjxmVyFUjpEKX229M8Z6MFtWWtP5S6EkOUAXg/g660+C09EILcAIeQOAOcopS+0+iwt4v0AftrqQ7jEMgADFf8+iHke1CohhKwCcAWA37f4KG7zBeiJmNbic3BlQS5frgch5JcAFlf5o78E8Anossq8ot7PTCl9qPQ9fwn9Ufz+Zp5N4D6EkA4ADwL4MKU03erzuAUh5DYAI5TSZwghN7X4OFwRgXwGlNJbqn2dELIVwGoALxBCAF1ieJYQcg2ldLiJR+ROrZ+ZQQh5H4DbANxM52/jwTkAKyr+fXnpa/MaQogfehC/n1L6vVafx2VuBHA7IeR1AEIA4oSQ/6CUvqvF53KMaAiyCSHkNIBtlNJ2nKBmGkLIrQB2A9hOKR1t9XncghDig17MvRl6AN8H4B2U0pdaejAXIXpG8u8AJiilH27xcZpKKSP/c0rpbS0+CheERi5oxJcAxAA8Qgh5nhCyp9UHcoNSQfdPAfwcetHvO/M5iJe4EcC7Aews/bd9vpStCtoMkZELBAJBmyMycoFAIGhzRCAXCASCNkcEcoFAIGhzRCAXCASCNkcEcoFAIGhzRCAXCASCNkcEcoFAIGhz/n/8cCgV9pkAMQAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- }
- ],
- "source": [
- "backend = np\n",
- "\n",
- "def function(x):\n",
- " return x*backend.sin(x**2 + 1)\n",
- "\n",
- "def analytical_gradient(x):\n",
- " return backend.sin(x**2 + 1) + 2*x**2*backend.cos(x**2 + 1)\n",
- "\n",
- "\n",
- "x = np.linspace(-5, 5, 100)\n",
- "plt.plot(x, function(x), label=\"f(x)\")\n",
- "\n",
- "descent_x, descent_y = gradient_descent(1, 0.01, 300, solver=\"analytical\")\n",
- "\n",
- "# Change the backend to JAX\n",
- "backend = jnp\n",
- "jax_descend_x, jax_descend_y = gradient_descent(1, 0.01, 300, solver=\"jax\")\n",
- "\n",
- "plt.scatter(descent_x, descent_y, label=\"Gradient descent\", marker=\"v\", s=10, color=\"red\") \n",
- "plt.scatter(jax_descend_x, jax_descend_y, label=\"JAX\", marker=\"x\", s=5, color=\"black\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "817e0b3c",
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/doc/src/week41/ipynb-exercisesweek41-src.tar.gz b/doc/src/week41/ipynb-exercisesweek41-src.tar.gz
deleted file mode 100644
index 93f94f613..000000000
Binary files a/doc/src/week41/ipynb-exercisesweek41-src.tar.gz and /dev/null differ
diff --git a/doc/src/week41/week41.do.txt b/doc/src/week41/week41.do.txt
index c56dd3232..b412b1c46 100644
--- a/doc/src/week41/week41.do.txt
+++ b/doc/src/week41/week41.do.txt
@@ -17,6 +17,8 @@ DATE: Week 41
!bblock Material for the lecture on Thursday October 12, 2023
* Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model.
* Building our own Feed-forward Neural Network
+ * "Video of lecture notes":"https://youtu.be/5-RRTO9uDvI"
+ * "Whiteboard notes":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesOct12.pdf"
* Readings and Videos:
* These lecture notes
* For neural networks we recommend Goodfellow et al chapter 6.