[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-
Nov 3, 2018
+
Nov 4, 2018
@@ -260,10 +266,79 @@ $$
f(x) = \beta_0+\beta_1x = 0,
$$
-as the function that determines the line that separates two classes (our two features).
+as the function that determines the line \( L \) that separates two classes (our two features), see the figur here.
+
+
+Define a vector \( \hat{\beta}:\left\{\beta_0,\beta_1\right\} \). Let us label the values of \( \hat{\beta} \) that satisfy this constraint as \( \overline{\beta} \).
+
+
+Any two points \( x_1 \) and \( x_2 \) on the line \( L \) will satisfy \( \hat{\beta}(x_1-x_2)=0 \). We normalize the solution and define
+$$
+\overline{\beta} = \frac{\hat{\beta}}{\vert\vert \hat{\beta}\vert\vert},
+$$
+
+which is vector normal to the line \( L \).
+
+
+The signed distance from a point \( x_0 \) on \( L \) to any point \( x \) is then
+$$
+\overline{\beta}(x-x_0) = \frac{\beta_1 x + \beta_0}{\vert\vert \hat{\beta}\vert\vert}.
+$$
+
+
-
Examples with kernels
+
First attempt at a minimization approach
+
+
+How do we find the parameters \( \beta_0 \) and \( \beta_0 \)? What we could
+do is to define a cost function which now contains the set of all
+misclassified points \( M \) and attempt to minimize this function
+
+$$
+C(\beta_0,\beta_1) = -\sum_{i\in M} y_i(\beta_1x_1+\beta_0).
+$$
+
+
+We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \beta_1x_i+\beta_0 < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
+$$
+\frac{\partial C}{\partial \beta_0} = -\sum_{i\in M} y_i,
+$$
+
+and
+$$
+\frac{\partial C}{\partial \beta_1} = -\sum_{i\in M} y_ix_i.
+$$
+
+
+
+
+
Solving the equations
+
+
+We can now use the Newton-Raphson method or gradient descent to solve the equations
+$$
+\beta_0 \leftarrow \beta_0 +\eta \frac{\partial C}{\partial \beta_0},
+$$
+
+and
+$$
+\beta_1 \leftarrow \beta_1 +\eta \frac{\partial C}{\partial \beta_1},
+$$
+
+where \( \eta \) is our by now well-known learning rate.
+There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.
+
+
+
+
+
A better approach
+A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
+
+
-as the function that determines the line that separates two classes (our two features).
+as the function that determines the line \( L \) that separates two classes (our two features), see the figur here.
+
+
+Define a vector \( \hat{\beta}:\left\{\beta_0,\beta_1\right\} \). Let us label the values of \( \hat{\beta} \) that satisfy this constraint as \( \overline{\beta} \).
+
+
+Any two points \( x_1 \) and \( x_2 \) on the line \( L \) will satisfy \( \hat{\beta}(x_1-x_2)=0 \). We normalize the solution and define
+
+
+which is vector normal to the line \( L \).
+
+
+The signed distance from a point \( x_0 \) on \( L \) to any point \( x \) is then
+
+$$
+\overline{\beta}(x-x_0) = \frac{\beta_1 x + \beta_0}{\vert\vert \hat{\beta}\vert\vert}.
+$$
+
-
Examples with kernels
+
First attempt at a minimization approach
+
+
+How do we find the parameters \( \beta_0 \) and \( \beta_0 \)? What we could
+do is to define a cost function which now contains the set of all
+misclassified points \( M \) and attempt to minimize this function
+
+
+We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \beta_1x_i+\beta_0 < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
+
+
+where \( \eta \) is our by now well-known learning rate.
+There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.
+
+
+
+
+
A better approach
+A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
+
+
+
+
+
Examples with kernels
diff --git a/doc/pub/svm/html/svm-solarized.html b/doc/pub/svm/html/svm-solarized.html
index 467fd73a0..69804bbc9 100644
--- a/doc/pub/svm/html/svm-solarized.html
+++ b/doc/pub/svm/html/svm-solarized.html
@@ -40,7 +40,10 @@ div { text-align: justify; text-justify: inter-word; }
('What is a hyperplane', 2, None, '___sec3'),
('The two-dimensional case', 2, None, '___sec4'),
('Getting into the details', 2, None, '___sec5'),
- ('Examples with kernels', 2, None, '___sec6')]}
+ ('First attempt at a minimization approach', 2, None, '___sec6'),
+ ('Solving the equations', 2, None, '___sec7'),
+ ('A better approach', 2, None, '___sec8'),
+ ('Examples with kernels', 2, None, '___sec9')]}
end of tocinfo -->
@@ -82,7 +85,7 @@ MathJax.Hub.Config({
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-
Nov 3, 2018
+
Nov 4, 2018
@@ -215,10 +218,79 @@ $$
f(x) = \beta_0+\beta_1x = 0,
$$
-as the function that determines the line that separates two classes (our two features).
+as the function that determines the line \( L \) that separates two classes (our two features), see the figur here.
+
+
+Define a vector \( \hat{\beta}:\left\{\beta_0,\beta_1\right\} \). Let us label the values of \( \hat{\beta} \) that satisfy this constraint as \( \overline{\beta} \).
+
+
+Any two points \( x_1 \) and \( x_2 \) on the line \( L \) will satisfy \( \hat{\beta}(x_1-x_2)=0 \). We normalize the solution and define
+$$
+\overline{\beta} = \frac{\hat{\beta}}{\vert\vert \hat{\beta}\vert\vert},
+$$
+
+which is vector normal to the line \( L \).
+
+
+The signed distance from a point \( x_0 \) on \( L \) to any point \( x \) is then
+$$
+\overline{\beta}(x-x_0) = \frac{\beta_1 x + \beta_0}{\vert\vert \hat{\beta}\vert\vert}.
+$$
+
+
-
Examples with kernels
+
First attempt at a minimization approach
+
+
+How do we find the parameters \( \beta_0 \) and \( \beta_0 \)? What we could
+do is to define a cost function which now contains the set of all
+misclassified points \( M \) and attempt to minimize this function
+
+$$
+C(\beta_0,\beta_1) = -\sum_{i\in M} y_i(\beta_1x_1+\beta_0).
+$$
+
+
+We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \beta_1x_i+\beta_0 < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
+$$
+\frac{\partial C}{\partial \beta_0} = -\sum_{i\in M} y_i,
+$$
+
+and
+$$
+\frac{\partial C}{\partial \beta_1} = -\sum_{i\in M} y_ix_i.
+$$
+
+
+
+
+
Solving the equations
+
+
+We can now use the Newton-Raphson method or gradient descent to solve the equations
+$$
+\beta_0 \leftarrow \beta_0 +\eta \frac{\partial C}{\partial \beta_0},
+$$
+
+and
+$$
+\beta_1 \leftarrow \beta_1 +\eta \frac{\partial C}{\partial \beta_1},
+$$
+
+where \( \eta \) is our by now well-known learning rate.
+There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.
+
+
+
+
+
A better approach
+A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
+
+
+
+
+
Examples with kernels
diff --git a/doc/pub/svm/html/svm.html b/doc/pub/svm/html/svm.html
index 15f85129a..f5fb48306 100644
--- a/doc/pub/svm/html/svm.html
+++ b/doc/pub/svm/html/svm.html
@@ -45,7 +45,10 @@ div { text-align: justify; text-justify: inter-word; }
('What is a hyperplane', 2, None, '___sec3'),
('The two-dimensional case', 2, None, '___sec4'),
('Getting into the details', 2, None, '___sec5'),
- ('Examples with kernels', 2, None, '___sec6')]}
+ ('First attempt at a minimization approach', 2, None, '___sec6'),
+ ('Solving the equations', 2, None, '___sec7'),
+ ('A better approach', 2, None, '___sec8'),
+ ('Examples with kernels', 2, None, '___sec9')]}
end of tocinfo -->
@@ -87,7 +90,7 @@ MathJax.Hub.Config({
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-
Nov 3, 2018
+
Nov 4, 2018
@@ -220,10 +223,79 @@ $$
f(x) = \beta_0+\beta_1x = 0,
$$
-as the function that determines the line that separates two classes (our two features).
+as the function that determines the line \( L \) that separates two classes (our two features), see the figur here.
+
+
+Define a vector \( \hat{\beta}:\left\{\beta_0,\beta_1\right\} \). Let us label the values of \( \hat{\beta} \) that satisfy this constraint as \( \overline{\beta} \).
+
+
+Any two points \( x_1 \) and \( x_2 \) on the line \( L \) will satisfy \( \hat{\beta}(x_1-x_2)=0 \). We normalize the solution and define
+$$
+\overline{\beta} = \frac{\hat{\beta}}{\vert\vert \hat{\beta}\vert\vert},
+$$
+
+which is vector normal to the line \( L \).
+
+
+The signed distance from a point \( x_0 \) on \( L \) to any point \( x \) is then
+$$
+\overline{\beta}(x-x_0) = \frac{\beta_1 x + \beta_0}{\vert\vert \hat{\beta}\vert\vert}.
+$$
+
+
-
Examples with kernels
+
First attempt at a minimization approach
+
+
+How do we find the parameters \( \beta_0 \) and \( \beta_0 \)? What we could
+do is to define a cost function which now contains the set of all
+misclassified points \( M \) and attempt to minimize this function
+
+$$
+C(\beta_0,\beta_1) = -\sum_{i\in M} y_i(\beta_1x_1+\beta_0).
+$$
+
+
+We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \beta_1x_i+\beta_0 < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
+$$
+\frac{\partial C}{\partial \beta_0} = -\sum_{i\in M} y_i,
+$$
+
+and
+$$
+\frac{\partial C}{\partial \beta_1} = -\sum_{i\in M} y_ix_i.
+$$
+
+
+
+
+
Solving the equations
+
+
+We can now use the Newton-Raphson method or gradient descent to solve the equations
+$$
+\beta_0 \leftarrow \beta_0 +\eta \frac{\partial C}{\partial \beta_0},
+$$
+
+and
+$$
+\beta_1 \leftarrow \beta_1 +\eta \frac{\partial C}{\partial \beta_1},
+$$
+
+where \( \eta \) is our by now well-known learning rate.
+There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.
+
+
+
+
+
A better approach
+A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
+
+
+
+
+
Examples with kernels
diff --git a/doc/pub/svm/ipynb/.ipynb_checkpoints/svm-checkpoint.ipynb b/doc/pub/svm/ipynb/.ipynb_checkpoints/svm-checkpoint.ipynb
new file mode 100644
index 000000000..35c59c606
--- /dev/null
+++ b/doc/pub/svm/ipynb/.ipynb_checkpoints/svm-checkpoint.ipynb
@@ -0,0 +1,124 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "# Data Analysis and Machine Learning: Support Vector Machines\n",
+ "\n",
+ " \n",
+ "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
+ "\n",
+ "Date: **May 30, 2018**\n",
+ "\n",
+ "Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Support Vector Machines, overarching aims"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/usr/local/lib/python3.7/site-packages/sklearn/svm/base.py:196: FutureWarning: The default value of gamma will change from 'auto' to 'scale' in version 0.22 to account better for unscaled features. Set gamma explicitly to 'auto' or 'scale' to avoid this warning.\n",
+ " \"avoid this warning.\", FutureWarning)\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAFgdJREFUeJzt3X2MXXWdx/HPZ1sequgWZZaybd220mAqq0UmLITN4vKwViGU3WBSBLZGTDGg211JtA0hZDNhg3HjA1nc3a5tqdkGMDwEoqKWSmwIBby1w0MpSK0rTHNLx+VhB6WMnX73jzlTb9t5Pufec+783i8zmXvOPfee76/qZ373d3/ndxwRAgBMfX9UdgEAgNYg8AEgEQQ+ACSCwAeARBD4AJAIAh8AEkHgA0AiCHwASASBDwCJmF52AY1OOumkmDdvXtllAEBb2bZt228iomOs4yoV+PPmzVOtViu7DABoK7Z/PZ7jGNIBgEQQ+ACQCAIfABJB4ANAIsYd+LbX2d5n+9mGfV+1/bztp23fb3tmw3Orbe+y/YLtjxVdOABgYibSw79D0pIj9m2SdHpEfEjSLyStliTbiyQtk/TB7DXfsj0td7UAMAXV++o6747ztPfNvU09z7gDPyK2SHr1iH0/jogD2ebjkuZkj5dKuisi3o6IX0naJemsAuoFgCmna0uXHn3pUXX9tKup5ylyDP8zkh7KHs+W9HLDcz3ZPgBAg3pfXeu71+tgHNT67vVN7eUXEvi2b5R0QNLGSbx2he2a7Vpvb28R5QBA2+ja0qWDcVCSNBADTe3l5w5825+WdImkK+MPd0TfI2luw2Fzsn1HiYg1EdEZEZ0dHWNeGQwAU8ZQ775/oF+S1D/Q39Refq7At71E0pckXRoRv2t46kFJy2wfZ3u+pIWSnsxzLgCYahp790Oa2csf91o6tu+U9FFJJ9nukXSzBmflHCdpk21JejwiPhcRO2x/V9JzGhzquT4iBoouHgDaVb2vru889Z1Dvfsh/QP9eqznsaacc9yBHxFXDLN77SjH3yLplskUBQBTXdeWLr114C1d13mdbr/49packyttAaDFWjkzpxGBD6A0rbrgqErqfXWduebMls3MaUTgAyhNqy44qpJVD69S/c16y2bmNCLwAZSirGGNMtX76tr4zNGXK7Wql0/gAyhFKy84qoquLV0aGGbCYjNn5jQi8AG0XKsvOKqCoTY3mjF9huo31BU3h7Zfu73pNRD4AFqu1RccVUEV2kzgA2i5rT1bW3rBURVUoc3jvvAKAIrSiuGLqqlCm+nhA0AiCHwASASBDwCJIPABIBEEPgAkgsAHgEQQ+ABQoCqvAErgA0CBqrwCKIEPAAWp+gqgBD4AFKTqK4COO/Btr7O9z/azDfveY3uT7Rez3ydm+237Ntu7bD9t+yPNKB4AqqIdVgCdSA//DklLjti3StLmiFgoaXO2LUkfl7Qw+1kh6d/zlQkA1VaF1TDHMu7Aj4gtkl49YvdSSRuyxxskXdaw/zsx6HFJM22fkrdYAKiqKqyGOZa8q2WeHBH17PFeSSdnj2dLernhuJ5sX10AMMXU++p693HvVv2GumadMKvsckZU2Je2ERGSYqKvs73Cds12rbe3t6hyAKBlqjwVs1HewH9laKgm+70v279H0tyG4+Zk+44SEWsiojMiOjs6OnKWAwCtVfWpmI3yBv6DkpZnj5dLeqBh/99ns3XOlvRGw9APgCmoyleYNlPVp2I2msi0zDslbZV0mu0e29dIulXSRbZflHRhti1JP5C0W9IuSf8l6bpCqwZQOe0yrFGkdpiK2Wgis3SuiIhTIuKYiJgTEWsj4n8j4oKIWBgRF0bEq9mxERHXR8T7I+LPI6LWvCYAKFs7DWsUqR2mYjbiSlsAubXTsEaR2mEqZiMPTq6phs7OzqjV+DAAtJN6X10Lblug/Qf2H9o3Y/oM7V65u9JTFKcS29sionOs4+jhA8il3YY1UkbgA8il3YY1Upb3SlsAidt+7fayS8A40cMHgEQQ+ACQCAIfABJB4ANAi5S9/ASBDwAtUvbyEwQ+ALRAFZafIPABoAWqsPwEgQ8ATVaVVTUJfABosqosP0HgA0CTVWX5CZZWAIAmq8ryE/TwASARBD4AJILABwCVfxVsKxQS+Lb/yfYO28/avtP28bbn237C9i7bd9s+tohzAUAzlH0VbCvkDnzbsyX9g6TOiDhd0jRJyyR9RdLXI+JUSa9JuibvuQCgGapwFWwrFDWkM13SDNvTJb1DUl3S+ZLuyZ7fIOmygs4FAIWqwlWwrZA78CNij6R/lfSSBoP+DUnbJL0eEQeyw3okzc57LgAoWlWugm2FIoZ0TpS0VNJ8SX8q6Z2Slkzg9Sts12zXent785YDABNSlatgW6GIIZ0LJf0qInoj4veS7pN0rqSZ2RCPJM2RtGe4F0fEmojojIjOjo6OAsoBgNE1zsipylWwrVDElbYvSTrb9jskvSXpAkk1SY9IulzSXZKWS3qggHMBKFi9r65l9y7T3ZffrVknzCq7nJZonJFTlatgW6GIMfwnNPjl7M8lPZO95xpJX5b0Rdu7JL1X0tq85wJQvBSmIzZKZUbOcAqZpRMRN0fEByLi9Ii4OiLejojdEXFWRJwaEZ+MiLeLOBeA4qQYfqnMyBkOV9oCCUst/FKakTMcAh9IVIrhl9KMnOEQ+ECiqhx+zVrXJqUZOcNhPXwgUVUOv8Yvkm+/+PbC3jelGTnDcUSUXcMhnZ2dUavVyi4DQInqfXUtuG2B9h/YrxnTZ2j3yt3JTBedLNvbIqJzrOMY0gFQKal9kdxKBD6Aykjxi+RWIvABVEaVv0ieCgh8AJVR5S+SpwJm6QCojDJm0aS0lhA9fABJS2ktIQIfQLJSW0uIwAeQrNSmgBL4AJKU4hRQAh9AklKcAkrgA0hSilNAmZYJoG3lmVKZ4kJq9PABtK2UplQWgcAH0JZSm1JZhEIC3/ZM2/fYft72Ttvn2H6P7U22X8x+n1jEuQBASm9KZRGK6uF/U9IPI+IDkj4saaekVZI2R8RCSZuzbQDILcUplUXIHfi2/1jSX0laK0kR0R8Rr0taKmlDdtgGSZflPReAtA3d+nD15tXJTaksQhGzdOZL6pW03vaHJW2TtFLSyRFRz47ZK+nk4V5se4WkFZL0vve9r4ByAExVQ1/SPrfvueSmVBahiMCfLukjkr4QEU/Y/qaOGL6JiLA97L0UI2KNpDXS4C0OC6gHwBTU+CXtb3//W9VvqE/51S2LVsQYfo+knoh4Itu+R4N/AF6xfYokZb/3FXAuAIniS9r8cgd+ROyV9LLt07JdF0h6TtKDkpZn+5ZLeiDvuQCkiS9pi1HULJ0vSNpo+2lJiyX9i6RbJV1k+0VJF2bbADBhKa570wyFLK0QEd2SOod56oIi3h9A2lJc96YZWEsHQOWluO5NM7C0AgAkgsAHgEQQ+ACQCAIfABJB4ANAIgh8AEgEgQ9UwNAqkFw5imYi8IEK4FZ9aAUCHygZt+pDqxD4QMlYBRKtQuADJWIVSLQSgQ+UiFUg0UoEPlAiVoFEK7FaJlAiVoFEK9HDB4BEEPgAkAgCH0DhuuvdmnnrTD39ytNll4IGBD6Awl11/1V64+039Kl7P1V2KWhQWODbnmZ7u+3vZdvzbT9he5ftu20fW9S5AFRXd71bO3p3SJJ29O6gl18hRfbwV0ra2bD9FUlfj4hTJb0m6ZoCzwWgoq66/6rDtunlV0chgW97jqSLJX0727ak8yXdkx2yQdJlRZwLQHU19u6H0MuvjqJ6+N+Q9CVJQ5cMvlfS6xFxINvukTR7uBfaXmG7ZrvW29tbUDkAynBk734IvfxqyB34ti+RtC8itk3m9RGxJiI6I6Kzo6MjbzkASvTL1345of1orSKutD1X0qW2PyHpeEnvlvRNSTNtT896+XMk7SngXAAq7K0b3yq7BIwidw8/IlZHxJyImCdpmaSfRMSVkh6RdHl22HJJD+Q9FzAc7hYFjE8z5+F/WdIXbe/S4Jj+2iaeCwnjblHA+Dgiyq7hkM7OzqjVamWXgTZS76trwW0LtP/Afs2YPkO7V+7WrBNmlV0W0FK2t0VE51jHcaUt2hp3iwLGj8BH2+JuUcDEEPhoW9wtCpgYAh9ti7tFARPDHa/QtrhbFDAx9PABIBEEPgAkgsAHgEQQ+ACQCAIfABJB4AOJYbG5dBH4QGJYbC5dBD6QkKHlKA7GQZahSBCBDySExebSRuADiWCxORD4QCJYbA4EPpAIFpsDi6cBiWCxOdDDB4BE5A5823NtP2L7Ods7bK/M9r/H9ibbL2a/T8xfLgBgsoro4R+QdENELJJ0tqTrbS+StErS5ohYKGlztg0AKEnuwI+IekT8PHvcJ2mnpNmSlkrakB22QdJlec8FpI5lEZBHoWP4tudJOkPSE5JOjoh69tReSSeP8JoVtmu2a729vUWWA0wZQ0G/evNqlkXApDkiinkj+wRJP5V0S0TcZ/v1iJjZ8PxrETHqOH5nZ2fUarVC6gGminpfXWeuOVP1N+ua5mkaiAHNmD5Du1fu1qwTZpVdHirA9raI6BzruEJ6+LaPkXSvpI0RcV+2+xXbp2TPnyJpXxHnAlKz6uFVqr85+GF5IAYO/aaXj4kqYpaOJa2VtDMivtbw1IOSlmePl0t6IO+5gNTU++ra+MzGo/azLAImo4ge/rmSrpZ0vu3u7OcTkm6VdJHtFyVdmG0DmIBVD6861Ks/Er18TFTuK20j4lFJHuHpC/K+P5CqkXr3Q1gWARPFlbaoBKYbHq1rS9eIvfvFsxYrbg6WS8CEEPioBO7CdLStPVuH3b941mKCHpNS2LTMIjAtM031vroW3LZA+w/sZ7ohMAktnZYJ5MFdmIDWIPBRKu7CBLQOgY9ScRcmoHUIfJSKuzABrcMdr1AqZpsArUMPHwASQeADLcCFZagCAh9oAS4sQxUQ+ECTDU09PRgHmXKKUhH4QJNxYRmqgsDHlFfm+DkXlqFKCHxMCaOFepnj51xYhioh8DEljBTqZY+fc2EZqoTAb0MpTvEbrc2jhXrZ4+fbr92uuDmO+uGCM5SBwC9JntBu1yl+I7V5PP8Wo7V5pFBn/Bw4HIFfoImE+GRDu+whiuHqGWrzWO0fqc1j/VuM1ubRQp3xc+BwTQ9820tsv2B7l+1VzT5fmcYb4nlCe7JDFEUMAw33Ho1tHq39I7V5PP8Wo7V5tFBn/Bw4XFMD3/Y0SbdL+rikRZKusL2omecsy0RCPE9oT3aIoohhoCPfo7HN67av0/rtI7d/pDaP9W8xVptHC3XGz4HDNbuHf5akXRGxOyL6Jd0laWmTz1mK8YZ43tCezBBFEcNAw71HYz39A/2H2nRkTSO1+am9Tx21f932dTpn7TmHahyrzYQ6MH7NDvzZkl5u2O7J9h1ie4Xtmu1ab29vk8tpjomEeJ5x5ckOURQxU+XI91i1adVhbT6Y/Weopsb2j9TmK++78qj9/QP9erzn8UM1MiwDFKepNzG3fbmkJRHx2Wz7akl/ERGfH+74dr2J+XXfv05rt689LJiOnXasPnvGZ3X7xbcfduwZ/3mGuvd2H/Uei2ctbkqvtPEG4UMmeqPw4d5jmqdp2h9NOyqMhzS2f6Q2Hz/9+MPesxE3MwfGb7w3MW/2DVD2SJrbsD0n2zelTKQX2uqhhtE+URz5x2ii7zEwMDDiaxrbP942N/7hnGiNAMbW7MD/maSFtudrMOiXSfpUk8/ZclUeLy5iSGS495D+8KlkpE8RD1350LjPMdKw2E3n3UQvHyhIUwM/Ig7Y/rykH0maJmldROxo5jlxuCL+GI31Hs38FEEvHyhO0+9pGxE/kPSDZp8H5WnWpwi+nAWKxU3MkVsrPkUAyI+lFQAgEQQ+ACSCwAeARBD4AJAIAh8AEkHgA0AiCHwASASBDwCJIPABIBEEPgAkgsAHgEQQ+ACQCAIfABJB4ANAIgh8AEgEgQ8AiSDwASARBD4AJCJX4Nv+qu3nbT9t+37bMxueW217l+0XbH8sf6kAgDzy9vA3STo9Ij4k6ReSVkuS7UWSlkn6oKQlkr5le1rOcwEAcsgV+BHx44g4kG0+LmlO9nippLsi4u2I+JWkXZLOynMuAEA+RY7hf0bSQ9nj2ZJebniuJ9t3FNsrbNds13p7eyd14npfXefdcZ72vrl3Uq8HgBSMGfi2H7b97DA/SxuOuVHSAUkbJ1pARKyJiM6I6Ozo6JjoyyVJXVu69OhLj6rrp12Tej0ApGD6WAdExIWjPW/705IukXRBRES2e4+kuQ2Hzcn2Fa7eV9f67vU6GAe1vnu9bjrvJs06YVYzTgUAbS3vLJ0lkr4k6dKI+F3DUw9KWmb7ONvzJS2U9GSec42ka0uXDsZBSdJADNDLB4AR5B3D/zdJ75K0yXa37f+QpIjYIem7kp6T9ENJ10fEQM5zHWWod98/0C9J6h/o1/ru9YzlA8Aw8s7SOTUi5kbE4uzncw3P3RIR74+I0yLiodHeZ7Iae/dD6OUDwPDa+krbrT1bD/Xuh/QP9OuxnsdKqggAqmvML22rbPu128suAQDaRlv38AEA40fgA0AiCHwASASBDwCJIPABIBH+w2oI5bPdK+nXE3jJSZJ+06RyykB7qo32VFvK7fmziBhzMbJKBf5E2a5FRGfZdRSF9lQb7ak22jM2hnQAIBEEPgAkot0Df03ZBRSM9lQb7ak22jOGth7DBwCMX7v38AEA49TWgW/7k7Z32D5ou22/nbe9xPYLtnfZXlV2PXnZXmd7n+1ny64lL9tzbT9i+7nsf2sry64pD9vH237S9lNZe/657JqKYHua7e22v1d2LXnZ/h/bz2T3GKkV+d5tHfiSnpX0d5K2lF3IZNmeJul2SR+XtEjSFbYXlVtVbndIWlJ2EQU5IOmGiFgk6WxJ17f5fz9vSzo/Ij4sabGkJbbPLrmmIqyUtLPsIgr019k9RpiWOSQidkbEC2XXkdNZknZFxO6I6Jd0l6SlY7ym0iJii6RXy66jCBFRj4ifZ4/7NBgqs8utavJi0JvZ5jHZT1t/kWd7jqSLJX277Fqqrq0Df4qYLenlhu0etXGgTGW250k6Q9IT5VaSTzb80S1pn6RNEdHW7ZH0DQ3eW/vgWAe2iZD0Y9vbbK8o8o0rfwMU2w9LmjXMUzdGxAOtrgdpsn2CpHsl/WNE/F/Z9eSR3V96se2Zku63fXpEtOX3LbYvkbQvIrbZ/mjZ9RTkLyNij+0/0eD9wp/PPjXnVvnAj4gLy66hyfZImtuwPSfbh4qwfYwGw35jRNxXdj1FiYjXbT+iwe9b2jLwJZ0r6VLbn5B0vKR32/7viLiq5LomLSL2ZL/32b5fg8O+hQQ+Qzrl+5mkhbbn2z5W0jJJD5ZcEzK2LWmtpJ0R8bWy68nLdkfWs5ftGZIukvR8uVVNXkSsjog5ETFPg//f+Uk7h73td9p+19BjSX+jAv8Yt3Xg2/5b2z2SzpH0fds/KrumiYqIA5I+L+lHGvxC8LsRsaPcqvKxfaekrZJOs91j+5qya8rhXElXSzo/mybXnfUm29Upkh6x/bQGOxubIqLtpzJOISdLetT2U5KelPT9iPhhUW/OlbYAkIi27uEDAMaPwAeARBD4AJAIAh8AEkHgA0AiCHwASASBDwCJIPABIBH/DwOr7JiClR7jAAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xd4VNXWwOHfSiOUSBOQXhQVSGhGikgVERAVFLGgguLFgt0PRcWO99rBgiJYQERUUAQL0kSKijSDIAHpvQaCQCgp6/vjTIaUyaROZpKs93nyZM4+bZ2UWXP23mdvUVWMMcaYrAT5OwBjjDGBzRKFMcYYryxRGGOM8coShTHGGK8sURhjjPHKEoUxxhivLFEYnxCRrSLSNYt17UVkfWHH5Asi0l9EZvvx/DNFZEABHSvd78Xb7zCPx/9bRDoV1PFM4bFEYbIkIjeLyHIROSYie1xvSpfm97iqukhVL8hjTA1F5AsROSAi/4rIBhF5R0Rq5TeuvFDVSarazRfHFhEVkeOun3+ciMwTkRsynL+Hqk7I4bHO87ZNfn4vHs43XkRGZDh+E1X9pSCObwqXJQrjkYg8AowC/gtUA+oA7wHX+DGm84A/gN1AC1U9C2gHbALyncACVDNVLQdcAIwH3hWRZwv6JCISUtDHNMWIqtqXfaX7AsoDx4DrvWwzHhiRZrkTsDPN8lbgCWAtcBj4BAjPYtvawDfAASAOeDeLc34GfJdN7BWB713HOux6XStDXF3TLD8HfOZ6He46RxwQDywDqrnWDQQ2A0eBLUD/NOWL0xzvLWAH8C+wAmif4VxfAZ+6jvM3EO3lWhQ4L0NZX+AkUNm1/Atwp+v1ecAC4AhwEPjSVb7Qdazjrt/rDam/A+BxYC8wMZe/w3TXnTZeYDCQCJx2ne+7jD97oBTOB5Hdrq9RQKm0fx/Ao8B+YA9wu7//L0ryl91RGE/a4rxpTsvncfoDVwDnAucDwzNuICLBOG/m24B6QE3giyyO1xX4OptzBuG8odXFuQs6Abybw3gH4CTJ2kBl4G7ghIiUBd4GeqhqBHAJEJPFMZYBzYFKwOfAFBEJT7P+apzrqwDMyEVsqaYDIUArD+teBGbjJMtawDsAqtrBtb6ZqpZT1S9dy+e44qyL8+buSba/w4xUdSwwCXjVdb6rPGz2FNAG52fVzHU9aY99Ds7voiYwCBgtIhWzO7fxDUsUxpPKwEFVTcrncd5V1R2qegh4CbjJwzatgBrAUFU9rqonVXVxFsc7G+fTLwAicp+IxLvq8McBqGqcqn6tqgmqetR13o45jDcR59rPU9VkVV2hqv+61qUAkSJSWlX3qOrfng6gqp+5YkhS1TdwPjmnrfdfrKo/qmoyzqf4ZjmMLfX4iTh3C5WyiL8uUCObn2OqFOBZVT2lqiey2CYnv8O86A+8oKr7VfUA8Dxwa5r1ia71iar6I86dSYG0n5jcs0RhPIkDzi6AeusdaV5vw0kIGdUGtuUwKcUB1VMXVPVdVa2AU20RCiAiZUTkAxHZJiL/4lS7VHDduWRnIjAL+EJEdovIqyISqqrHcapr7gb2iMgPInKhpwOIyP+JSKyIHBGReJxPxWen2WRvmtcJQHhufs4iEgpUAQ55WP0YIMBSVw+jO7I53AFVPZnNNjn5HeZFDdfxsjp2XIa/iQSgXAGd2+SSJQrjye/AKaC3l22OA2XSLJ/jYZvaaV7XwamLzmgHUCeHb5bzgGuz2eZRnE+erdVp7E6tdhHX9yzjdn16fV5VG+NUL/UCbnOtm6Wql+MkqnXAuIwnFpH2OG/W/YCKriR2JM25C8I1QBKwNOMKVd2rqv9R1RrAXcB72fR0ysnQ0Vn9DtP9HEUk4+8/u2Pvxrn78XRsE2AsUZhMVPUI8AxOvXBv16f0UBHpISKvujaLAXqKSCXXm8RDHg41RERqiUglnDrpLz1ssxSnsfJlESkrIuEi0i6L0J4D2ovImyJSE0BEzgYapdkmAqddIt513ow9hGKAG13XE43TOIzrWJ1FJMp19/EvTvVHiohUE5FrXG0Vp3CqQVI8xBeB8yZ+AAgRkWeAs7K4llxx/Zz7A6OBV1Q1zsM216fpJnwY5806Nc59QIM8nDqr3+EqoImINHe1wTyXYb/szjcZGC4iVVy/w2dwOhKYAGSJwnjkql9/BKeB8QDOJ//7gG9dm0zEebPYitOA6ikJfO5atxmnC+uIjBu46uqvwuktsx2nt8sNGbdzbfsP0BqnoXaViBwFfsX5JPq0a7NRQGmcevwlwE8ZDvM0TsPsYZx68c/TrDsHmIqTJGJxehBNxPk/ecR1nkM4bR73eAhxlut8/+BUpZwkfdVNXqwSkWPARuBO4GFVfSaLbS8G/nBtPwN4UFU3u9Y9B0xwten0y8X5Pf4OXb+LF4C5wAYgY3vIR0Bj1/m+JbMRwHLgL2A1sBIPfx8mMIiqTVxkjDEma3ZHYYwxxitLFMYYY7yyRGGMMcYrSxTGGGO8KhYDgZ199tlar149f4dhjDFFyooVKw6qapXstisWiaJevXosX77c32EYY0yRIiLbst/Kqp6MMcZkwxKFMcYYryxRGGOM8apYtFF4kpiYyM6dOzl5MrvBMY0pGsLDw6lVqxahoaH+DsWUMMU2UezcuZOIiAjq1auHSEEO3mlM4VNV4uLi2LlzJ/Xr1/d3OKaEKbZVTydPnqRy5cqWJEyxICJUrlzZ7pCNXxTbRAFYkjDFiv09G3/xeaIQkY9FZL+IrElT9pqIrBORv0RkmohUSLPuCRHZKCLrReQKX8dnjDHGu8K4oxgPdM9QNgeIVNWmOGP3PwEgIo2BG4Emrn3ey+EUlgFp79693HjjjZx77rlcdNFF9OzZk3/++cfrPvXq1ePgwYOZyseMGcOnn36a43MHBwfTvHlz99fWrVtzG36W4uPjee+999zLu3fvpm/fvl72KDjPPPMMc+fOBWDUqFEkJCS415Url/1MmePHjycoKIi//vrLXRYZGZmrn0+nTp2Ijo52Ly9fvpxOnTp53Wf58uU88MADOT6HMR7FTiJuTG32vSYwth7ETiqU0/o8UajqQjLM76uqs9PMh7sEZyIacKZ5/MI12fsWnMlaWvk6Rl9QVfr06UOnTp3YtGkTK1as4H//+x/79u3L0/HuvvtubrvtthxvX7p0aWJiYtxfBTnEScZEUaNGDaZOnVpgx/fmhRdeoGvXrkDmRJFTtWrV4qWXXspXHPv372fmzJk53j46Opq33347X+c0JVzsJJg9mBcO7aRBAkw6tA1mDy6UZBEIbRR3AKn/cTVJPyPYTldZJiIyWESWi8jyAwcO+DjE3Js/fz6hoaHcfffd7rJmzZrRvn17fvnlF3r16uUuv++++xg/frx7+dVXXyUqKopWrVqxceNGAJ577jlef/11ADZu3EjXrl1p1qwZLVu2ZNOmTTmKafz48dx3333u5V69evHLL78Azqfxp556imbNmtGmTRt3Qtu3bx99+vShWbNmNGvWjN9++41hw4axadMmmjdvztChQ9m6dSuRkZGA04ng9ttvJyoqihYtWjB//nz3ua+99lq6d+9Ow4YNeeyxxzLFt2zZMq691pkSe/r06ZQuXZrTp09z8uRJGjRwZtUcOHAgU6dO5e2332b37t107tyZzp07u4/h6Roy6tWrF3///Tfr16/PtG7y5MlERUURGRnJ448/nuXPcujQoR6TTVbXn/Z3vmDBAvedXosWLTh69CgAr732GhdffDFNmzbl2WczzuBqSrxFT7H9dAJjEp25fqOCgKQEWPSUz0/t10QhIk/hzDGc65SoqmNVNVpVo6tUyXZMK+jUKfNX6qfihATP61PfvA8ezLwuG2vWrOGiiy7KzSW5lS9fntWrV3Pffffx0EOZp6Lu378/Q4YMYdWqVfz2229Ur1490zYnTpxwvxn16dMn23MeP36cNm3asGrVKjp06MC4ceMAeOCBB+jYsSOrVq1i5cqVNGnShJdffplzzz2XmJgYXnvttXTHGT16NCLC6tWrmTx5MgMGDHD31ImJieHLL79k9erVfPnll+zYkX6W0BYtWhATEwPAokWLiIyMZNmyZfzxxx+0bt063bYPPPAANWrUYP78+e4346yuIaOgoCAee+wx/vvf/6Yr3717N48//jg///wzMTExLFu2jG+/9TSLJ7Rt25awsDD3uXNy/alef/11Ro8eTUxMDIsWLaJ06dLMnj2bDRs2sHTpUmJiYlixYgULFy70eG5TQh3dzoun4TRwQwg0DT5T7mt+SxQiMhDoBfTXM/Ox7gJqp9mslqusRLnpppvc33///fd0644ePcquXbvcb/7h4eGUKVMm0zHSVj1NmzYt23OGhYW5P/FedNFF7jr7n3/+mXvucaaHDg4Opnz58l6Ps3jxYm655RYALrzwQurWretul7nssssoX7484eHhNG7cmG3b0o9HFhISwrnnnktsbCxLly7lkUceYeHChSxatIj27dvn+Ro8ufnmm1myZAlbtmxxly1btoxOnTpRpUoVQkJC6N+/v9c36+HDhzNiRPppnr1df6p27drxyCOP8PbbbxMfH09ISAizZ89m9uzZtGjRgpYtW7Ju3To2bNiQ7TWbkmND6ep8kgTBwPNhaVZE1PH5uf3ywJ2IdAceAzqqatpK5hnA5yLyJlADaAgsLZCTuqpYPCpTxvv6s8/2vt6DJk2aZFlvHxISQkpKins54yfOtN0gC7JLpLfzhoaGus8VHBxMUlJSpv3zq1SpUu7XWZ2jQ4cOzJw5k9DQULp27crAgQNJTk7OdOfiSW6uISQkhEcffZRXXnklD1fi6NKlC8OHD2fJkiW52m/YsGFceeWV/Pjjj7Rr145Zs2ahqjzxxBPcddddeY7HFFOxk2DRUzx3YDfJwB3/wvmp/TZCykD7/LW35URhdI+dDPwOXCAiO0VkEPAuEAHMEZEYERkDoKp/A18Ba4GfgCGqmuzrGH2hS5cunDp1irFjx7rL/vrrLxYtWkTdunVZu3Ytp06dIj4+nnnz5qXb98svv3R/b9u2bbp1ERER1KpVy10lcurUqRw36NarV4+YmBhSUlLYsWMHS5dmn4Mvu+wy3n//fQCSk5M5cuQIERER7nr1jNq3b8+kSU5N4j///MP27du54IILchRf6v6jRo2ibdu2VKlShbi4ONavX+9uA0nLWxw5MXDgQObOnUtqG1erVq1YsGABBw8eJDk5mcmTJ9OxY0evxxg+fDivvvpquvizu/5NmzYRFRXF448/zsUXX8y6deu44oor+Pjjjzl27BgAu3btYv/+/Xm+NlNMuBqw18RvY7JCaDI8k/quHVEXuo2FRv19HobP7yhU9SYPxR952f4lwPcp0sdEhGnTpvHQQw/xyiuvEB4eTr169Rg1ahS1a9emX79+REZGUr9+fVq0aJFu38OHD9O0aVNKlSrF5MmTMx174sSJ3HXXXTzzzDOEhoYyZcoUd2OvN+3ataN+/fo0btyYRo0a0bJly2z3eeuttxg8eDAfffQRwcHBvP/++7Rt25Z27doRGRlJjx49GDJkiHv7e++9l3vuuYeoqChCQkIYP358ujuJ7LRu3Zp9+/bRoUMHAJo2bcrevXs93lkNHjyY7t27u9sqcissLIwHHniABx98EIDq1avz8ssv07lzZ1SVK6+8kmuuucbrMXr27EnaNrKcXP+oUaOYP38+QUFBNGnShB49elCqVCliY2PdHwzKlSvHZ599RtWqVXN9XaYYWfQUJCUw/DSowF2loG55nCQxeGuhhSFnmgeKrujoaM04cVFsbCyNGjXyU0TG+EaJ/7t2VcNwdLtTN9/+pUL5RO03bwhLkqHtCSgDbCoD5wQBCDyaks3O2RORFaoand12gdA91hhjsueqhuHoNkCd74X0HIFfxE5CFYY5tZE8tC01SVAoDdhpWaIwxhQNrmqYdArpOQK/WPQUHyTBgmColABD3blBCqUBOy1LFMaYoiGr5wUK4TkCf9h0ZBv3nHJev78dKkSkrtFCr26zRGGMKRqyqm4p5GqYwpCcksx5aW6e+qXt9BdRt9DjsURhjCka2r/kPDeQViE9R1DY7vzuTvfrHkeAcNeCn67XEoUxpmho1N95biCiLiCF+hxBYVq1dxXjY8a7l79rWBt/X68lCh9KHeo7MjKS66+/PtsH43IyTLYv5GQI7IwDGfpC2sEF87ONKcYa9XeeH3g0xflezJLE6eTTNP+guXt50rWTCL5nu9+v1xKFD6WOt7RmzRrCwsIYM2aMv0PyyIbANiYwXD/l+nTLN0V6el658FmiKCTt27d3Dxn+5ptvEhkZSWRkJKNGjcq07W233ZZu1NL+/fszffp0r0N1ZzU8drly5Rg6dChNmjSha9euLF26lE6dOtGgQQNmzJgBpL9bWLp0KW3btqVFixZccsklHofiTmv8+PH07t2byy+/nHr16vHuu+/y5ptv0qJFC9q0acOhQ85UJDExMbRp04amTZvSp08fDh8+DMCKFSvcQ5iPHj3afdzk5GSGDh3qHnb7gw8+yNXP25iiZtG2RcxYP8O9HNP8g4CZ/tYvgwIWNnneNz9sfTZnT7UnJSUxc+ZMunfvzooVK/jkk0/4448/UFVat25Nx44d0w3jMWjQIEaOHEnv3r05cuQIv/32GxMmTOCzzz4jJiaGP//8k1KlSnHBBRdw//33ExwczOOPP86KFSuoWLEi3bp149tvv6V3794cP36cLl268Nprr9GnTx+GDx/OnDlzWLt2LQMGDODqq69OF+uFF17IokWLCAkJYe7cuTz55JN8/fXXXq9vzZo1/Pnnn5w8eZLzzjuPV155hT///JOHH36YTz/9lIceeojbbruNd955h44dO/LMM8/w/PPPM2rUKG6//XbeffddOnTowNChQ93H/OijjyhfvjzLli3j1KlTtGvXjm7dugXMP44xBSkhMYEO4zu4l/seOodm1wz2Y0Tp2R2FD6XOCREdHU2dOnUYNGgQixcvpk+fPpQtW5Zy5cpx7bXXsmjRonT7dezYkQ0bNnDgwAEmT57MddddR0iIk9M9DdXtbXjssLAwund3ZqKNioqiY8eOhIaGEhUV5XEY7iNHjnD99dcTGRnJww8/zN9//53tdXbu3JmIiAiqVKlC+fLlueqqq9zn27p1K0eOHCE+Pt49wN6AAQNYuHAh8fHxxMfHu8d1uvXWW93HnD17Np9++inNmzendevWxMXF2bDbptjq8EmHdMsjb/nMT5F4ViLuKHL6yb+gpbZR5MVtt93GZ599xhdffMEnn3ziLs/JUN1ppR16OygoyL1/UFCQx32ffvppOnfuzLRp09i6dWu2c0FnjCkn58gJVeWdd97hiiuuSFdekHN/GxMIZqyfwYo9K9zLTx5rSa2LL/NjRJnZHUUha9++Pd9++y0JCQkcP36cadOmeZyUZ+DAge72i8aNG3s9Zl6Gx87KkSNHqFnTmX027fSs+VG+fHkqVqzovnOaOHEiHTt2pEKFClSoUIHFixcDuIfnBrjiiit4//33SUxMBJwhu48fP14g8RgTKI6cPMI1X5wZobh8qfI8/uLPfozIsxJxRxFIWrZsycCBA2nVqhUAd955Z6ZhxgGqVatGo0aN6N27d7bHzMvw2Fl57LHHGDBgACNGjODKK6/M0zE8mTBhAnfffTcJCQk0aNDAfZf0ySefcMcddyAidOvWzb39nXfeydatW2nZsiWqSpUqVbKcltSYour8UemfKv9vk96cFe59Fkl/sGHGA1RCQgJRUVGsXLky2+lHTclR1P+uzRmf/DiYO5admdf9/EOwpnY4oVd+WGjPS9gw40XY3LlzadSoEffff78lCWOKoQPHD6RLEgCvHIdQORmQo+Fa1VMA6tq1K9u2bfN3GMYYH1BVqr6efubCjnvgmvNdCwE4Gm5hzJn9sYjsF5E1acoqicgcEdng+l7RVS4i8raIbBSRv0Qk+7k6jTGmCHnl11fSLYvCqHCQYFdBAI6GWxhVT+OB7hnKhgHzVLUhMM+1DNADaOj6Ggy8XwjxGWNModhxZAdPzHsiXdl/dkLz2q6FAB0N1+eJQlUXAocyFF8DTHC9ngD0TlP+qTqWABVEpLqvYzTGGF9TVepk6OV0VmhpXryoFv4eHTY7/mqjqKaqe1yv9wLVXK9rAjvSbLfTVbaHDERkMM5dB3XqBN6tmjHGpPXwrIczlT3buA9Vewf+nN9+7/WkTv/cXPfRVdWxqhqtqtFVqlTxQWT5t2/fPm6++WYaNGjARRddRNu2bZk2bZrPz5uTYcNzKjExkWHDhtGwYUNatmxJ27ZtmTlzptd9Bg4cyNSpU/MdV6dOnYiOPtNzb/ny5dk+KV6Q125MQVl3cB1v/fFWurLzD8N9/3wDsYGfKPx1R7FPRKqr6h5X1dJ+V/kuoHaa7Wq5yoocVaV3794MGDCAzz//HIBt27a5R2z1pejo6HRvsPnx9NNPs2fPHtasWUOpUqXYt28fCxYsKLS49u/fz8yZM+nRo4fPzmH8JHaS0xX06HanAbf9SwFZ7ZJfySnJNBqd+dmXUSchLNjVHTbAr9tfdxQzgAGu1wOA6WnKb3P1fmoDHElTReVbsZNgbD14I8j5ns8s//PPPxMWFsbdd9/tLqtbty73338/4IxZ1L59e1q2bEnLli357bffgMwTBN13333uoTSGDRtG48aNadq0Kf/3f/8HwJQpU4iMjKRZs2buwfVyMmy4tyHLUyUkJDBu3Djeeecd9/hN1apVo1+/fkD6iZamTp3KwIED3ctz584lOjqa888/n++//z5TXMeOHeP2228nKiqKpk2bZjlC7dChQ3nppcyNeydPnnTv36JFC+bPn5/pHAsWLKB58+Y0b96cFi1acPToUQBee+019/Dlzz77rMfzGh+LnQSzB8PRbYA632cPLhKfrnPrpq8zzynx8FrocZ5rIQC7w2bk8zsKEZkMdALOFpGdwLPAy8BXIjII2Ab0c23+I9AT2AgkALf7Oj7gzB9tkmsGutQ/Wshzpv/7779p2TLr3r1Vq1Zlzpw5hIeHs2HDBm666SYyPl2eVlxcHNOmTWPdunWICPHx8QC88MILzJo1i5o1a7rL0vI2bLinIctr1z5zQ7dx40bq1KnDWWedlevr37p1K0uXLmXTpk107tzZPRdHqhdffJHy5cuzevVqAPf8FBmlVtfNnz+fiIgId/no0aMREVavXs26devo1q0b//zzT7p9X3/9dUaPHk27du04duwY4eHhzJ49mw0bNrB06VJUlauvvpqFCxe6k6wpJIueOvP/liopoUh8us6NpbuWMmXtlHRll2yHV2oCqSPmB2B32IwKo9fTTapaXVVDVbWWqn6kqnGqepmqNlTVrqp6yLWtquoQVT1XVaNUNet3zoLk7Y+2gAwZMoRmzZpx8cUXA07d/3/+8x+ioqK4/vrrWbt2rdf9U4cWHzRoEN988w1lyjiTzLdr146BAwcybtw4kpOTM+3nbdhwT0OWF5R+/foRFBREw4YNadCgAevWrUu3fu7cuQwZMsS9XLFixSyPNXz4cEaMGJGubPHixdxyyy2Akwzr1q2bKVG0a9eORx55hLfffpv4+HhCQkKYPXs2s2fPpkWLFrRs2ZJ169bZ8OX+kNWn6CLw6TqnTiefpvWHrdOVVUmGrw5AaE1XQYB2h83I743ZAcEHf7RNmjRh5cqV7uXRo0czb948Dhw4AMDIkSOpVq0aq1atYvny5Zw+fRqAkJAQUlJS3PudPHnSXb506VL69u3L999/755jYsyYMYwYMYIdO3Zw0UUXERcXly6O1GHD16xZw3fffec+HmQ/ZPl5553H9u3b+ffffz1eY9pJhNIeN+M6T8u50aVLF06cOMGSJUtytd+wYcP48MMPOXHiBO3atWPdunWoKk888QQxMTHExMSwceNGBg0alOfYTB5l9Sm6CHy6zqlO4zulWxaEyV2GUbNXXQK9O2xGlijAJ3+0Xbp04eTJk7z//plnBhMSzty1HDlyhOrVqxMUFMTEiRPddwN169Zl7dq1nDp1ivj4eObNmwc4dfpHjhyhZ8+ejBw5klWrVgGwadMmWrduzQsvvECVKlXYsSNt7+L8DRtepkwZBg0axIMPPuhOZAcOHGDKFOdWulq1asTGxpKSkpKpN9eUKVNISUlh06ZNbN68mQsuuCDd+ssvvzzd1KdZVT2lGj58OK+++qp7uX379u5hyf/55x+2b9+e6RybNm0iKiqKxx9/nIsvvph169ZxxRVX8PHHH3Ps2DEAdu3axf79+zGFrP1LzqfptIrIp+ucmLVxFr/v/D1d2QtBl3HZpS/C4K3waIrzvQgkCbBE4fDBH62I8O2337JgwQLq169Pq1atGDBgAK+84jy+f++99zJhwgSaNWvGunXrKFu2LAC1a9emX79+REZG0q9fP/cQ5EePHqVXr140bdqUSy+9lDfffBNwGntT58q+5JJLaNasWbo4HnvsMZ544glatGiRp0mERowYQZUqVWjcuDGRkZH06tXL3Wbx8ssv06tXLy655BKqV0//XGSdOnVo1aoVPXr0YMyYMYSHh6dbP3z4cA4fPuxuiE9tjM5Kz549SdsN+t577yUlJYWoqChuuOEGxo8fn+4OCWDUqFFERkbStGlTQkND6dGjB926dePmm2+mbdu2REVF0bdvX3cjtylEjfo7n6Yjit6n6+wcP32c7pPSD0bRc0soT847DcHBWewV2GyYcfcOJaOrninabJjxwFd7ZG12/rvTvVw3qRwrR52g0pJV0KSJHyPLLKfDjNvosaka9bfEYIzJl89Xf54uSYRJKFM/Pkalu/8v4JJEbliiMMaYAhB/Mp7+36T/sDlqVXWiSYJnnvFTVAWjWCcKVc1XbxtjAklxqCYuziq+kr6Ld/+o/tzd/X74919I8wxQUVRsE0V4eDhxcXFUrlzZkoUp8lSVuLi4TJ0CTGAYtWRUuuUmVZrwQa8PkLCyeT9oALWbFttEUatWLXbu3Ol+bsGYoi48PJxatWr5OwyTwd5je9ONDFsurBxTf6lK2Y1Pg6t3Yq75YLSI/Ci2iSI0NJT69ev7OwxjTDE7kMJpAAAgAElEQVSmqlR/I33X8I+qDebCKW/C61fm/cABNsRJsU0Uxhjjaxlnq3ug+V30u2cytGgBDz6Y9wMH2BAnliiMMSYPNh3alG7+6za12vDaTymwbx/MmAEh+Xh7jajjGlnXQ7kf2JPZxhiTSymawnnvnOdeLhVciq86v0fYZ5OdO4n8zokSYEOc2B2FMcbk0m3Tbku3/N1N31G7QQtYswYqV87/CVLbIazXkzHGFD0xe2OYtPrMBEvPd3qey49XA1WoW7fgThRAo0VY1ZMxxuRQUkoSLT5o4V7uVK8Tw8tfBRddBG+84cfIfMvuKIwxJocun3h5uuUpvT8nqPOVUKkS3F44E3L6gyUKY4zJgV+2/sIvW39xLy+pfA5n31gD/gTeerBg2iYClF+rnkTkYRH5W0TWiMhkEQkXkfoi8oeIbBSRL0UkzJ8xGmPMqaRTdJ7Q2b08qnQoJ3bs5eBioBmg45ynqYspvyUKEakJPABEq2okEAzcCLwCjFTV84DDgM1TaYzxqzqjzjy/0DgklDdOJtK5DDxxJdCHM09NF1P+bswOAUqLSAhQBtgDdAGmutZPAHr7KTZjjGHq2qnsP35mutx+QYnscA3kW7EpUM61wk9PTRcGvyUKVd0FvA5sx0kQR4AVQLyqps7ZuROo6Z8IjTEl3bHTx7h+yvXu5Yl9JvLc6TPrX007+66fnpouDP6seqoIXAPUB2oAZYHuXndKv/9gEVkuIstthFhjjC9E/O/MPBIPtn6QW6fd6l4+kHZ6ED8+NV0Y/Fn11BXYoqoHVDUR+AZoB1RwVUUB1AJ2edpZVceqarSqRlepUqVwIjbGlBgvLUz/xv/pH2+5X39Vpgdn16gLCETUhW5jA+bhOF/wZ/fY7UAbESkDnAAuA5YD84G+wBfAAGC63yI0xmQWQBPq+MqB4wcYPn+4e/m6YPg62XndYz9cf1/gPDVdGPzZRvEHTqP1SmC1K5axwOPAIyKyEagMfOSvGI0xGaROqHN0G6BnJtQpZl1Dq75e1f36hpAzSQJg8jkU6x5Onvj1gTtVfRZ4NkPxZqCVH8IxxmQnwCbU8YVLP7403fKXSWde/xwP5WtRrHs4eeLv7rHGmKIkwCbUKWi/7/idX3f86nHdxFLQOXUm2mLcw8kTSxTGmJzL6g2yGLxxJiYncsnHl3hcN2or3BLqWijmPZw8sURhjMm5AJtQxy12EoytB28EOd/z0GYSNsLzaEFDfxMerHkOJaWHkyc2KKAxJucCbEId4EwDe2rbSWoDO+Q4rud/ed5jeZ9YeKXTS/DEEx7XlxSiqtlvFeCio6N1+fLl/g7DGOMPY+tlMb90XRi8Ndvd1x9cz4WjL8xU3vCQsO7vzgTNngNBxbPyRURWqGq287baHYUxpmjLRwN7QmKCxyQBsOb7OgQtmlhsk0Ru2E/AGFO05bGBXVVp+n5Tj+uOP3mcsNh/oEaN/EZXLFiiMMYUbXlsYH9/+ftsOrwpU3lcjVGUCSkNYTYVTipLFMaYoq1Rf6cnUkTOx15asXsFQ34ckql8w6cVqPTeeEhM9F28RZC1URhjir5GOR976fCJw0SPy9x+Oz+mGeft3QA/TM76bqIEjHPliSUKY0yJkaIp9JvaL1P5mOQedPp2JnzyCVzouXG7ILrhFlVW9WSMKTFe/fVV5m6em67srnNv4K4RP8GgQTBwYNY7exvnqpizOwpjTIkwf8t8npiX/sG5qKpRjLnlC6h6B7Rv7/0AxXycK2/sjsIYU+ztPrqbHpN6ZCr/s5VrFoNu3aB0ae8HKcbjXGXHEoUxplhLTE7khqk3cCr5VLryI4fuIrhtO9iUuYusR4E6zlUhsERhjCnWnpz3JIu3L05XtmlfRc56+wO4914499ycHSgP3XCLC2ujMMYUW9Nip/H676+nK5t/Ahp8dBjqBsHtLXJ3wFx0wy1O7I7CGFMsbTy0kWu/ujZT+QPHYX8F4LYU+CPjBJvGE0sUxphi50TiCS4ed7HHdavPhuM3AOUpET2WCoJfE4WIVBCRqSKyTkRiRaStiFQSkTkissH1vaI/YzTGFD3XT7me+JPxHtd1C4b6dV0LJaDHUkHw9x3FW8BPqnoh0AyIBYYB81S1ITDPtWyMMTky8veR/LDhB4/rqqTAT+GuhRLSY6kg+C1RiEh5oAPwEYCqnlbVeOAaYIJrswlAb/9EaIwpan7f8TuPzH4ky/U7KlREpGT1WCoI/uz1VB84AHwiIs2AFcCDQDVV3ePaZi9QzdPOIjIYGAxQp47dPhpT0h1MOMglH1+S5fpdyVCq7Fkw+FAhRlU8+LPqKQRoCbyvqi2A42SoZlJnnlaPc7Wq6lhVjVbV6CpVqvg8WGNM4EpOSabKa1m/D/weDzWs8TrPsk0UItIuJ2V5sBPYqap/uJan4iSOfSJS3XWe6sD+AjiXMaaYUlVCXsy6cmTSHmhTy7Vgjdd5kpM7indyWJYrqroX2CEiF7iKLgPWAjOAAa6yAcD0/J7LGFN8XfHZFVmueyYUbm7oWrDG6zzLMg2LSFvgEqCKiKRtHToLCC6g898PTBKRMGAzcDtO8vpKRAYB24DMg8cbYwzw7PxnmbN5jsd1N1ZtyXNBB+HYjhI1yZAveGvMDgPKubaJSFP+L9C3IE6uqjFA5qmmnLsLY4zJ0md/fcYLC1/wuK7N4XJ88shcJMIewyoIWSYKVV0ALBCR8aq6TUTKqGpCVtsbY0xh+WnjT9w67VaP6+oeDebbB34j3JJEgclJG0UNEVkLrAMQkWYi8p5vwzLGGM9+2/Gbx7klACJOwfe9v6Rag6hCjqp4y0miGAVcAcQBqOoqnAfljDGmUK3et5p2H2fd6XJG9BtEXnpdIUZUMuTogTtV3SEiaYuSfROOMcZ4tvnwZjqMz/oz6ub7NlK/cg7nljC5kpNEsUNELgFUREJxnp6O9W1Yxhhzxt5je7l84uVZDvR3fGg8ZcqUL+SoSo6cVD3dDQwBagK7gOauZWOM8bn4k/Fc8dkVbD682eP6o/H3Uqb0WYUcVcmS7R2Fqh4ErPOxMYEmdhIsesoZlqKYPieQkJhAr8978de+vzyun7SzNeU+eAfSV42bApZtohCRtz0UHwGWq6o9NW2MP8ROgtmDIcnVY/3oNmcZik2ySExOpO9Xffl1x68e11dODOXmd36BIH/PllD85eQnHI5T3bTB9dUUqAUMEpFRPozNGJOVRU+dSRKpkhKc8mIgRVMYOH0gMzfOzHKb3UP3Qnh4lutNwclJY3ZToJ2qJgOIyPvAIuBSYLUPYzPGZCWrUVCLweioqsoDMx/g89WfZ1qX8GQCmw9v5txK5xIWYkmisOTkjqIizlAeqcoClVyJ45RPojLGeJfVKKjFYHTU5xc8z+hlozOV7xp3FqXHfUKTqk0ItyRRqHKSKF4FYkTkExEZD/wJvCYiZYG5vgzOGJOF9i85o6GmVQxGR337j7d5fsHzmcqXTq1EDTkLevb0Q1TGa9WTOE/ZzQZ+BFq5ip9U1d2u10N9GJsxJiupDdbFqNfT+JjxPPjTg5nKJ8+twMX7gmHhHKhXr/ADM94ThaqqiPyoqlHYvBDGBJZG/Yt0YkhrfMx4bp9+e6by4X9GcOP6UPj5Z7jwQj9EZiBnVU8rReRin0dijCmRRi0Z5TFJ9LmwD89fMwrmzYPISD9EZlLlpNdTa6C/iGzDmddacG42mvo0MmNMsffwTw8z6o/MveyblT2XT/t8SlBYOQ97mcKWk0SR9TyDxhiTB4nJiVz9xdX8tPGnTOuqHocZE3ZT7v5QP0RmPMm26klVt6nqNuAEoGm+jDEm1/Ye20uj0Y08JgmAadOgTu8TsOjhQo7MZCXbRCEiV4vIBmALsADYCmT9uGQuiUiwiPwpIt+7luuLyB8islFEvnTNp22MKQb+2PkHDd5qwKbDmzyuL5MIl1yDMwTpX2MLNTaTtZw0Zr8ItAH+UdX6OPNZLynAGDIOW/4KMFJVzwMOA4MK8FzGGD/5cOWHtPmoDSeSTmS5zawUoLprQW3am0CRk0SRqKpxQJCIBKnqfCC6IE4uIrWAK4EPXcsCdAGmujaZAPQuiHMZY/zjVNIp7v7+bv7z3X+8brchBS6tnKZAgn0bmMmxnDRmx4tIOWAhMElE9gPHCuj8o4DHgAjXcmUgXlWTXMs7cW5CMxGRwcBggDp1iv6wBcYUR7uP7qbvV335fefvXrfbVwaqZvzY2nSw7wIzuZKTO4pVQALwMPATsAlYl98Ti0gvYL+qrsjL/qo6VlWjVTW6SpUq+Q3HGFPAft3+KxeNvchrkuh8MIJjd22jaot7ztxBSDA0uwe6vldIkZrs5OSOorOqpgApOFVBiIjnWURypx1wtYj0xBnK/CzgLaCCiIS47ipq4cyqZ0zBKgGT/viLqvLBig94YOYDJKYkZrndlPVN6Tv2VyhXDs55zxJDAMvyjkJE7hGR1cCFIvJXmq8tQL4Thao+oaq1VLUecCPws6r2B+YDfV2bDcCGDjEFLXXSn6PbAD0z6U/sJH9HVuSdTDrJnTPu5J4f7vGaJFbs6Enf8cucJGECnreqp8+Bq3DeqK9K83WRqt7iw5geBx4RkY04bRYf+fBcpiQq5pP++MvOf3fS4ZMOfBzzMeEh4URVjcq0TZNKF7K77LO0HPc9hFnP96Iiy6onVT2CM+XpTb4OQlV/AX5xvd7MmZFqjSl4xXjSH39ZuG0h10+5nv3H91O3fF2qlq3Kst3L0m1zw4V9+eTaTykdWtpPUZq8sslmTclTjCf9KWyqyjt/vMNln17G/uP7aVe7HaeTT2dKEs8vCmFynUcsSRRRlihMyVNMJ/0pbCcSTzBw+kAe+OkBklKSuOLcK/h1x6/sObYn3XZfza7AM//7DWnb1k+RmvzKSa8nY4qXYjjpT2HbFr+Na7+6lpV7VlI6pDTNz2nOrE2zMm237Jfzif5iLtSu7YcoTUGxRGFKpmI06U9hm79lPv2m9uNgwkHKhpalQngFj89K7NxyLTW/m2A9m4oBSxTGmBxRVUYuGcljcx4j2TUO06nkU+w6mv5Rp7PLnM22h7ZRJrSMp8OYIsjaKIwx2UpITKD/N/15dPaj7iQBkJSSlG67DjuC2HfNYksSxYzdURhjvNpyeAt9vuzDqn2rvG43+J8IPnh8MZx/QSFFZgqLJQpjTJbmbJrDjV/fyKETh7xu98q283nsnd+hUqVCiswUJqt6MsZkoqq8+uurdJ/UPVOSqFehXrrliUlX89i4tZYkijG7ozDGpHPs9DHumH4HU9ZOSVdeMbwizc9pzsKtC9xl39/0PVeef2Vhh2gKmSUKY4zbxkMb6fNlH9bsX5OuvH1YKSqeOsyMrfMBeGpHfZ4fs57gkFB/hGkKmSUKYwwAMzfM5OZvbib+ZLy7LESCuSYEDh4/xYxQKJUEH28J5uZhz4MliRLD2iiMKeFUlZcWvsSVn1+ZLkkAJGsyXycmsyAUqh2HX47AzS2S4ben/RSt8Qe7ozCmBDt66igDvh3AtHXTPK5XoBIwKB4ergjVq6XuaCPtliSWKIwpodYfXE+fL/sQezDW4/qWe4O4L6IsN1Y9SumMo3DYSLslilU9GVMCfbf+Oy4cfaHHJNF//zn8/iEs39CJ23uMoHTGp6xtpN0Sx+4ojClBUjSFG6femKnrK8Dz5a7irpfnUu3UcXj1fRg8GIKC4JzKNtJuCWeJwpgSICkliYmrJnLHjDsyrRvWbhgvdnmRkA/GQatkGDMm/bDgNtJuiee3RCEitYFPgWo4bWZjVfUtEakEfAnUA7YC/VT1sL/iNKYo23dsHx+u/JDh84dnWte6+sXM3d6JcnsaQ1AI3H238yVS+IGagObPNook4FFVbQy0AYaISGNgGDBPVRsC81zLxpgcUlWW7FzCLd/cQu2RtTMliSAJYkq9x1jy3/2Ue+k1WOaatlTEkoTxyG93FKq6B9jjen1URGKBmsA1QCfXZhOAX4DH/RCiMUXKicQTfLHmC95d9i4r96z0uE3Hs85l4qJ61H72VWjSBBYsgA4dCjlSU9QERBuFiNQDWgB/ANVcSQRgL07VlKd9BgODAerUsa56puTacngL7y9/n4/+/Mg9gF+wBKebNwLgv2Hw2LodBM/aBW+8AfffD6H2dLXJnqiqfwMQKQcsAF5S1W9EJF5VK6RZf1hVK3o7RnR0tC5fvtzXoRoTMFI0hbmb5/Lu0nf5/p/vUZz/4+ga0XSKqMrkDT+xKyUl3T6/l4Y2wUBQLXh4hx+iNoFGRFaoanR22/n1jkJEQoGvgUmq+o2reJ+IVFfVPSJSHdjvvwiNCSzxJ+OZEDOB0ctGs+HQBgDCgsO4ockN3NfqPuYueZ2n1mTu+gpQO/UzYcouj+uNyYo/ez0J8BEQq6pvplk1AxgAvOz6Pt0P4RkTUFbvW83oZaOZ+NdEEhITAKh9Vm3uib6Hqy+4mol/TaT1h6097nvLUfiwGpRK7bpiT1WbXPLnHUU74FZgtYjEuMqexEkQX4nIIGAb0M9P8RnjV4nJiXy77ltGLxvNgm1n5oC4rP5l3NHiDv7e/zdP/vwkT/78pMf9/7sL7moAldINvyH2VLXJNX/2eloMZNUX77LCjMUUYbGTit1Tw3uP7WXcinGMWTGG3Ud3A1AurBw3R97M4ZOHmbJ2CvO2zPO4b7WgIFaGp1AjBcg0dbVAs7uL/M/HFL6A6PVkTJ7EToLZgyHJqYrh6DZnGYrcm6Gq8vvO33l36btMXTuVxJREwJl2NCExgf3H9zN25dgs9z/nGEwPG0Cr/l1hzl1nfiapwitDl7eK3M/FBAZLFKboWvRU5jfEpASnvIi8ISYkJjB59WTeXfYuMXtjMq3fGr/V437jKg1k4EPjub1vCBXPb8ZLD35FRK0GzkqRYneXZfzLEoUpurKaE6EIzJWw+fBm3lv2Hh//+TGHT+ZshJpxtYcw6Lx+SIcOsG8fHDmXiUOGQMUMvcdtbCZTwCxRmKIroo5T3eSpPAClaAqzNs5i9LLR/LDhhxzt063uZXzzb3fKvj0GNo2GfgecJ6mrVYPhmcdvMsYXbD4KU3S1f8mZGyGtAJwr4fCJw4z8fSR1R9Wl5+c9c5wkAGZvm8f4z4ZClSrw5ZcwaZIPIzXGM7ujMEVXavVKgNbHr9q7ihGLRjB17dR8HafHy1Ohy3UFFJUxuWeJwhRtAVYfn5icyMglI3l8bv7Gsbyj+R2M6DKC6hHVCygyY/LOEoUx+aSq/LjhR3pN7pXnY0ScgpF7mnH9dU9zVo/eEBxcgBEakz+WKIzJg9PJp1mwdQFD5wxl1b5VeTpGz4Y9+c8vR+l2QU/K3DUQzjkn/QbF8GFCUzRZojAmh46cPMLMjTOZvGYyM9bPyNMx/rMCro2vTpe5mwgLKw03Z7FhMXqY0BR9liiM8WLnvzuZsX4G3677ljmb5+R6/5oawbVLj3LdngpcesmNBD96I1x6afZVS8XgYUJTfFiiMCYNVWX1/tVMXzed6euns2LPilwf49xDcF1kP67t8QgXn6hIUMddzrMPuWl3KMIPE5rixxKFKfGSUpJYvH2xOzlsid+S62NE7oNrY+E6GhHV6Qbk8gFQq56zsuH5uQ+qiD1MaIo3SxSmRDp2+hizNs5i+vrp/LDhB/cUorkRXSOa6y7ow7WTVnL+JVfB8MuhRo2CCbD9S+nbKCAgHyY0JYMlClNi7D22lxnrZzB9/XTmbZ7HqeRTudpfFNrtCua65Ib0aXsHdf8z1FnRwQfBBvjDhKZksURR0pSgLpe69jPWzX+c6f/u5msNY3ni6VwfIxjocgSujYfeFeCc909CSCH92wTYw4Sm5LJEUZIUky6XJ5NOcjDhIHF/f0rc8reIS9jPwbBKxNXpwoEyNfjun+88tDPkPEn03g+tg6BVTYgOgbPKATWBiLqFlySMCSD2V18U5fWuIMC6XKoqR08fJe6vj4j741UOHt9LXHhl4ur15OBZ9TiYcJDtR7azae+fbDy6myRvBzt5CNbkbEyl1juh6ulQvmvgTA7UJgUeOQWXVoTqIUA5nPYAax8wBrBEEThy+uafn7sCH3a5TE5J5vDJw8QlxBF3Io64hDj2HtvL5sObid0yh9X7VrE52etbveNkHMRMzHc8GZ2eBaHVgGrAiL1QtSqMq59Fz6K6zs+/hFTRGZOdgE0UItIdeAunmvhDVX3ZzyH5Tm7e/PNzV5Cmy2WSwgkgQSGhbHWO7I3h0IlDHEw4yM5/dxJ7IJaVe1eycs/K/F9fIbsuojWd4mOJDv+XqDAoK0Dq4KsRdZ25HMB7zyJrHzDGTVTV3zFkIiLBwD/A5cBOYBlwk6qu9bR9dHS0Ll++vBAjLGBj68HRbXyeCK8mwgnXm/gJCeJEcGmOJx73d4QB4dlt0CEJSpeBMmWhdEUoWwqqCYSIayMJhh4TnNeekkC3sekTQAlq3DcmIxFZoarR2W4XoImiLfCcql7hWn4CQFX/52n7/CaK46eP8+OGH0lMSSQpJYnEZOd7UkqS17K05enKcnucQ/+wRSHwfhO+d1kwdBEodxAqhUDVMKgaDlUr1+Dsu7YQ9tH5nquHSlWG5BOZ767gTEIASwLGeJHTRBGoVU81gR1plncCrdNuICKDgcEAderk72nVuBNx9JvaL1/HKC6iQmtxQVgNaoRWpOKuWZSqCKFA6EEonQKlBcKDILzLSEpXqkp4jTqEh4QT/m8CpcuWJ7xsecLDIygdWppSwaUI/vBcz2/0GdXKsHz+NRAclnX10GVvOa9nDgBNTr9valXc4K2WGIwpAIGaKLKlqmOBseDcUeTnWOXCytG3cV9CgkIIDQolJCgk3evQ4JyVpS3PVdmWHwhZ/BShSScJwalGCQ0pTchlowltcivBEoyInAm4sKpLXFViAKR94DiiLvR8KP22WT2Q7OmNXkJBBFK8dFnd/KPzPbsHz3681fP+NiaSMQUmUBPFLqB2muVarjKfqFS6ElOun+Krw2fv7AugXLWcv/kXVkNrQQwjkdUbvbssi7uNtG/03q7XxkQyxucCtY0iBKcx+zKcBLEMuFlV//a0fZFvzA5kvr57SXvXklZEXafqKCfx5aTR2hiTSZFuo1DVJBG5D5iF0z3246yShPExX9+95PeuxcZEMsbnAjJRAKjqj8CP/o7D+FhBvNHbMw/G+FTAJgpTgtgbvTEBLcjfARhjjAlsliiMMcZ4ZYnCGGOMV5YojDHGeGWJwhhjjFeWKIwxxnhlicIYY4xXliiMMcZ4ZYnCGGOMV5YojDHGeGWJwhhjjFeWKIwxxnhlicIYY4xXliiMMcZ4ZYnCGGOMV5YojDHGeGWJwhhjjFd+SRQi8pqIrBORv0RkmohUSLPuCRHZKCLrReQKf8RnjDHmDH/dUcwBIlW1KfAP8ASAiDQGbgSaAN2B90Qk2E8xGmOMwU+JQlVnq2qSa3EJUMv1+hrgC1U9papbgI1AK58HFDsJxtaDN4Kc77GTfH5KY4wpKgKhjeIOYKbrdU1gR5p1O11lmYjIYBFZLiLLDxw4kPezx06C2YPh6DZAne+zB1uyMMYYF58lChGZKyJrPHxdk2abp4AkINfvyqo6VlWjVTW6SpUqeQ900VOQlJC+LCnBKTfGGEOIrw6sql29rReRgUAv4DJVVVfxLqB2ms1qucp85+j23JUbY0wJ469eT92Bx4CrVTXtx/kZwI0iUkpE6gMNgaU+DSaiTu7KjTGmhPFXG8W7QAQwR0RiRGQMgKr+DXwFrAV+AoaoarJPI2n/EoSUSV8WUsYpN8YY47uqJ29U9Twv614CCu9dulF/5/uip5zqpog6TpJILTfGmBLOL4ki4DTqb4nBGGOyEAjdY40xxgQwSxTGGGO8skRhjDHGK0sUxhhjvLJEYYwxxitLFMYYY7yyRGGMMcYrSxTGGGO8kjPj8RVdInIA2JaHXc8GDhZwOP5i1xK4itP12LUEprxeS11VzXb47WKRKPJKRJararS/4ygIdi2Bqzhdj11LYPL1tVjVkzHGGK8sURhjjPGqpCeKsf4OoADZtQSu4nQ9di2ByafXUqLbKIwxxmSvpN9RGGOMyYYlCmOMMV6V+EQhIteLyN8ikiIiRbKrnIh0F5H1IrJRRIb5O568EpGPRWS/iKzxdyz5JSK1RWS+iKx1/X096O+Y8kpEwkVkqYiscl3L8/6OKb9EJFhE/hSR7/0dS36JyFYRWe2aVnq5L85R4hMFsAa4Fljo70DyQkSCgdFAD6AxcJOINPZvVHk2Huju7yAKSBLwqKo2BtoAQ4rw7+UU0EVVmwHNge4i0sbPMeXXg0Csv4MoQJ1VtbmvnqUo8YlCVWNVdb2/48iHVsBGVd2sqqeBL4Br/BxTnqjqQuCQv+MoCKq6R1VXul4fxXlTqunfqPJGHcdci6GuryLbC0ZEagFXAh/6O5aiosQnimKgJrAjzfJOiugbUnElIvWAFsAf/o0k71xVNTHAfmCOqhbZawFGAY8BKf4OpIAoMFtEVojIYF+cIMQXBw00IjIXOMfDqqdUdXphx2NKDhEpB3wNPKSq//o7nrxS1WSguYhUAKaJSKSqFrm2JBHpBexX1RUi0snf8RSQS1V1l4hUBeaIyDrX3XmBKRGJQlW7+jsGH9oF1E6zXMtVZvxMREJxksQkVf3G3/EUBFWNF5H5OG1JRS5RAO2Aq0WkJxAOnCUin6nqLX6OK89UdZfr+34RmYZTHV2gicKqnoq+ZUBDEakvImHAjcAMP8dU4omIAB8Bsar6pr/jyQ8RqeK6k0BESgOXA+v8G1XeqOoTqlpLVevh/K/8XJSThIiUFZGI1NdAN3yQwEt8ohCRPiKyE2gL/CAis/wdU26oahJwHzALp8H0K1X9279R5Y2ITAZ+By4QkZ0iMsjfMeVDO+BWoIur22KM61NsUVQdmC8if+F8MJmjqkt4pA8AAAF9SURBVEW+W2kxUQ1YLCKrgKXAD6r6U0GfxIbwMMYY41WJv6MwxhjjnSUKY4wxXlmiMMYY45UlCmOMMV5ZojDGGOOVJQpjCoCIPCci/+dlfe8iPCigKeEsURhTOHrjjO5rTJFjz1EYk0ci8hQwAGegvB3ACuAIMBgIAzbiPHTXHPjete4IcB3QJeN2qppQyJdgTI5YojAmD0TkIpz5M1rjjJm2EhgDfKKqca5tRgD7VPUdERkPfK+qU13rKnvartAvxJgcKBGDAhrjA+2Baal3ASKSOr5WpOuNvwJQDmdoFU9yup0xfmdtFMYUrPHAfaoaBTyPM0JpfrYzxu8sURiTNwuB3iJS2jV651Wu8ghgj2uI8f5ptj/qWkc22xkTcCxRGJMHrmlOvwRWATNxRlUFeBpnJrtfST8U9xfAUBH5U0TO9bKdMQHHGrONMcZ4ZXcUxhhjvLJEYYwxxitLFMYYY7yyRGGMMcYrSxTGGGO8skRhjDHGK0sUxhhjvPp/ZpVUP0r97GsAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "%matplotlib inline\n",
+ "\n",
+ "import numpy as np\n",
+ "from sklearn.svm import SVR\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "# Generate sample data\n",
+ "X = np.sort(5*np.random.rand(40,1), axis=0)\n",
+ "y = X**3\n",
+ "y=y.ravel()\n",
+ "\n",
+ "# Add noise to targets\n",
+ "X[::4] +=3*(0.5 - np.random.rand(1))\n",
+ "y[::5] += 50 * (0.5 - np.random.rand(8))\n",
+ "\n",
+ "plt.plot(X,y, 'g^')\n",
+ "\n",
+ "#SVR Fit\n",
+ "svr_poly = SVR(kernel='poly', C=1e3, degree=3)\n",
+ "y_poly = svr_poly.fit(X, y).predict(X)\n",
+ "\n",
+ "# Plots\n",
+ "z = np.arange(0, 5, 0.1)\n",
+ "t = z**3\n",
+ "fig = plt.figure()\n",
+ "ax = fig.add_subplot(111)\n",
+ "plt.plot(z,z**3, 'r--', label='Cubic Function with No Noise')\n",
+ "lw = 2\n",
+ "plt.scatter(X, y, color='darkorange', label='Gaussian Cubic Noise')\n",
+ "plt.plot(X, y_poly, color='green', lw=lw, label='Polynomial model')\n",
+ "plt.xlabel('data')\n",
+ "plt.ylabel('target')\n",
+ "plt.title('Cubic Gaussian Distribution')\n",
+ "plt.legend()\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz
index 20c5ffe52..8f47939d0 100644
Binary files a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz and b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz differ
diff --git a/doc/pub/svm/ipynb/svm.ipynb b/doc/pub/svm/ipynb/svm.ipynb
index 3538e4fc9..a63b10e6f 100644
--- a/doc/pub/svm/ipynb/svm.ipynb
+++ b/doc/pub/svm/ipynb/svm.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Nov 3, 2018**\n",
+ "Date: **Nov 4, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -172,7 +172,136 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "as the function that determines the line that separates two classes (our two features). \n",
+ "as the function that determines the line $L$ that separates two classes (our two features), see the figur here. \n",
+ "\n",
+ "Define a vector $\\hat{\\beta}:\\left\\{\\beta_0,\\beta_1\\right\\}$. Let us label the values of $\\hat{\\beta}$ that satisfy this constraint as $\\overline{\\beta}$. \n",
+ "\n",
+ "Any two points $x_1$ and $x_2$ on the line $L$ will satisfy $\\hat{\\beta}(x_1-x_2)=0$. We normalize the solution and define"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\overline{\\beta} = \\frac{\\hat{\\beta}}{\\vert\\vert \\hat{\\beta}\\vert\\vert},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "which is vector normal to the line $L$. \n",
+ "\n",
+ "The signed distance from a point $x_0$ on $L$ to any point $x$ is then"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\overline{\\beta}(x-x_0) = \\frac{\\beta_1 x + \\beta_0}{\\vert\\vert \\hat{\\beta}\\vert\\vert}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## First attempt at a minimization approach\n",
+ "\n",
+ "How do we find the parameters $\\beta_0$ and $\\beta_0$? What we could\n",
+ "do is to define a cost function which now contains the set of all\n",
+ "misclassified points $M$ and attempt to minimize this function"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "C(\\beta_0,\\beta_1) = -\\sum_{i\\in M} y_i(\\beta_1x_1+\\beta_0).\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We could now for example define all values $y_i =1$ as misclassified in case we have $\\beta_1x_i+\\beta_0 < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\frac{\\partial C}{\\partial \\beta_0} = -\\sum_{i\\in M} y_i,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\frac{\\partial C}{\\partial \\beta_1} = -\\sum_{i\\in M} y_ix_i.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Solving the equations\n",
+ "\n",
+ "We can now use the Newton-Raphson method or gradient descent to solve the equations"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\beta_0 \\leftarrow \\beta_0 +\\eta \\frac{\\partial C}{\\partial \\beta_0},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\beta_1 \\leftarrow \\beta_1 +\\eta \\frac{\\partial C}{\\partial \\beta_1},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "where $\\eta$ is our by now well-known learning rate. \n",
+ "There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.\n",
+ "\n",
+ "## A better approach\n",
+ "A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning). \n",
+ "\n",
"## Examples with kernels"
]
},
diff --git a/doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf b/doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf
index 254e33143..ec79cbc31 100644
Binary files a/doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf and b/doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf differ
diff --git a/doc/pub/svm/pdf/svm-beamer.pdf b/doc/pub/svm/pdf/svm-beamer.pdf
index 7e47c65d6..9bade2271 100644
Binary files a/doc/pub/svm/pdf/svm-beamer.pdf and b/doc/pub/svm/pdf/svm-beamer.pdf differ
diff --git a/doc/pub/svm/pdf/svm-minted.pdf b/doc/pub/svm/pdf/svm-minted.pdf
index fab148e6b..3f667c2ab 100644
Binary files a/doc/pub/svm/pdf/svm-minted.pdf and b/doc/pub/svm/pdf/svm-minted.pdf differ
diff --git a/doc/src/SupportVMachines/svm.do.txt b/doc/src/SupportVMachines/svm.do.txt
index 305c36a93..29fd69633 100644
--- a/doc/src/SupportVMachines/svm.do.txt
+++ b/doc/src/SupportVMachines/svm.do.txt
@@ -115,7 +115,73 @@ Let us define the function
f(x) = \beta_0+\beta_1x = 0,
\]
!et
-as the function that determines the line that separates two classes (our two features).
+as the function that determines the line $L$ that separates two classes (our two features), see the figur here.
+
+Define a vector $\hat{\beta}:\left\{\beta_0,\beta_1\right\}$. Let us label the values of $\hat{\beta}$ that satisfy this constraint as $\overline{\beta}$.
+
+Any two points $x_1$ and $x_2$ on the line $L$ will satisfy $\hat{\beta}(x_1-x_2)=0$. We normalize the solution and define
+!bt
+\[
+\overline{\beta} = \frac{\hat{\beta}}{\vert\vert \hat{\beta}\vert\vert},
+\]
+!et
+which is vector normal to the line $L$.
+
+The signed distance from a point $x_0$ on $L$ to any point $x$ is then
+!bt
+\[
+\overline{\beta}(x-x_0) = \frac{\beta_1 x + \beta_0}{\vert\vert \hat{\beta}\vert\vert}.
+\]
+!et
+
+!split
+===== First attempt at a minimization approach =====
+
+How do we find the parameters $\beta_0$ and $\beta_0$? What we could
+do is to define a cost function which now contains the set of all
+misclassified points $M$ and attempt to minimize this function
+
+!bt
+\[
+C(\beta_0,\beta_1) = -\sum_{i\in M} y_i(\beta_1x_1+\beta_0).
+\]
+!et
+
+We could now for example define all values $y_i =1$ as misclassified in case we have $\beta_1x_i+\beta_0 < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us
+!bt
+\[
+\frac{\partial C}{\partial \beta_0} = -\sum_{i\in M} y_i,
+\]
+!et
+and
+!bt
+\[
+\frac{\partial C}{\partial \beta_1} = -\sum_{i\in M} y_ix_i.
+\]
+!et
+
+!split
+===== Solving the equations =====
+
+We can now use the Newton-Raphson method or gradient descent to solve the equations
+!bt
+\[
+\beta_0 \leftarrow \beta_0 +\eta \frac{\partial C}{\partial \beta_0},
+\]
+!et
+and
+!bt
+\[
+\beta_1 \leftarrow \beta_1 +\eta \frac{\partial C}{\partial \beta_1},
+\]
+!et
+where $\eta$ is our by now well-known learning rate.
+There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.
+
+!split
+===== A better approach =====
+A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
+
!split
===== Examples with kernels =====