diff --git a/doc/pub/week39/html/week39-bs.html b/doc/pub/week39/html/week39-bs.html index 746b595ba..65aa91964 100644 --- a/doc/pub/week39/html/week39-bs.html +++ b/doc/pub/week39/html/week39-bs.html @@ -59,6 +59,7 @@ Automatically generated HTML file from DocOnce source None, 'revisiting-our-logistic-regression-case'), ('The equations to solve', 2, None, 'the-equations-to-solve'), + ('To be added', 2, None, 'to-be-added'), ("Solving using Newton-Raphson's method", 2, None, @@ -223,53 +224,54 @@ MathJax.Hub.Config({
  • Optimization, the central part of any Machine Learning algortithm
  • Revisiting our Logistic Regression case
  • The equations to solve
  • -
  • Solving using Newton-Raphson's method
  • -
  • Brief reminder on Newton-Raphson's method
  • -
  • The equations
  • -
  • Simple geometric interpretation
  • -
  • Extending to more than one variable
  • -
  • Steepest descent
  • -
  • More on Steepest descent
  • -
  • The ideal
  • -
  • The sensitiveness of the gradient descent
  • -
  • Convex functions
  • -
  • Convex function
  • -
  • Conditions on convex functions
  • -
  • More on convex functions
  • -
  • Some simple problems
  • -
  • Standard steepest descent
  • -
  • Gradient method
  • -
  • Steepest descent method
  • -
  • Steepest descent method
  • -
  • Final expressions
  • -
  • Steepest descent example
  • -
  • Conjugate gradient method
  • -
  • Conjugate gradient method
  • -
  • Conjugate gradient method
  • -
  • Conjugate gradient method
  • -
  • Conjugate gradient method and iterations
  • -
  • Conjugate gradient method
  • -
  • Conjugate gradient method
  • -
  • Conjugate gradient method
  • -
  • Revisiting our first homework
  • -
  • Gradient descent example
  • -
  • The derivative of the cost/loss function
  • -
  • The Hessian matrix
  • -
  • Simple program
  • -
  • Gradient Descent Example
  • -
  • And a corresponding example using scikit-learn
  • -
  • Gradient descent and Ridge
  • -
  • Program example for gradient descent with Ridge Regression
  • -
  • Using gradient descent methods, limitations
  • -
  • Friday October 1
  • -
  • Stochastic Gradient Descent
  • -
  • Computation of gradients
  • -
  • SGD example
  • -
  • The gradient step
  • -
  • Simple example code
  • -
  • When do we stop?
  • -
  • Slightly different approach
  • -
  • Program for stochastic gradient
  • +
  • To be added
  • +
  • Solving using Newton-Raphson's method
  • +
  • Brief reminder on Newton-Raphson's method
  • +
  • The equations
  • +
  • Simple geometric interpretation
  • +
  • Extending to more than one variable
  • +
  • Steepest descent
  • +
  • More on Steepest descent
  • +
  • The ideal
  • +
  • The sensitiveness of the gradient descent
  • +
  • Convex functions
  • +
  • Convex function
  • +
  • Conditions on convex functions
  • +
  • More on convex functions
  • +
  • Some simple problems
  • +
  • Standard steepest descent
  • +
  • Gradient method
  • +
  • Steepest descent method
  • +
  • Steepest descent method
  • +
  • Final expressions
  • +
  • Steepest descent example
  • +
  • Conjugate gradient method
  • +
  • Conjugate gradient method
  • +
  • Conjugate gradient method
  • +
  • Conjugate gradient method
  • +
  • Conjugate gradient method and iterations
  • +
  • Conjugate gradient method
  • +
  • Conjugate gradient method
  • +
  • Conjugate gradient method
  • +
  • Revisiting our first homework
  • +
  • Gradient descent example
  • +
  • The derivative of the cost/loss function
  • +
  • The Hessian matrix
  • +
  • Simple program
  • +
  • Gradient Descent Example
  • +
  • And a corresponding example using scikit-learn
  • +
  • Gradient descent and Ridge
  • +
  • Program example for gradient descent with Ridge Regression
  • +
  • Using gradient descent methods, limitations
  • +
  • Friday October 1
  • +
  • Stochastic Gradient Descent
  • +
  • Computation of gradients
  • +
  • SGD example
  • +
  • The gradient step
  • +
  • Simple example code
  • +
  • When do we stop?
  • +
  • Slightly different approach
  • +
  • Program for stochastic gradient
  • @@ -304,7 +306,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 28, 2021

    +

    Sep 29, 2021


    @@ -328,7 +330,7 @@ MathJax.Hub.Config({

  • 9
  • 10
  • ...
  • -
  • 56
  • +
  • 57
  • »
  • diff --git a/doc/pub/week39/html/week39-reveal.html b/doc/pub/week39/html/week39-reveal.html index 42460a08e..bedb96af6 100644 --- a/doc/pub/week39/html/week39-reveal.html +++ b/doc/pub/week39/html/week39-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

     
    -

    Sep 28, 2021

    +

    Sep 29, 2021


    @@ -241,7 +241,7 @@ plt.legend() plt.show()

    -Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function RidgeCV instead of just calling the Ridge function. For RidgeCV we need to passe the array of \( \lambda \) values. +Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function RidgeCV instead of just calling the Ridge function. For RidgeCV we need to pass the array of \( \lambda \) values. By inspecting the figure we can in turn determine which is the optimal regularization parameter. This becomes however less functional in the long run. @@ -302,7 +302,10 @@ ypredictRidge = gridsearch.predict(X_test) print(f"R2 score: {R2(y_test,ypredictRidge)}")

    -By default the grid search function includes cross validation with five folds. The Scikit-Learn documentation contains more information on how to set the different parameters. +By default the grid search function includes cross validation with +five folds. The Scikit-Learn +documentation +contains more information on how to set the different parameters.

    If we take out the random noise, running the above codes results in \( \lambda=0 \) yielding the best fit. @@ -317,7 +320,10 @@ An alternative to the above manual grid set up, is to use a random search where the parameters are tuned from a random distribution (uniform below) for a fixed number of iterations. A model is constructed and evaluated for each combination of chosen parameters. -We repeat the previous example but now with a random search. +We repeat the previous example but now with a random search. Note +that values of \( \lambda \) are now limited to be within \( x\in +[0,1] \). This domain may not be the most relevant one for the specific +case under study.

    @@ -438,6 +444,17 @@ This defines what is called the Hessian matrix. +

    +

    To be added

    + +

    +We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. + +

    +Make link with linear regression and the Hessian matrix from linear regression. +

    + +

    Solving using Newton-Raphson's method

    diff --git a/doc/pub/week39/html/week39-solarized.html b/doc/pub/week39/html/week39-solarized.html index 7dc0d0002..458978331 100644 --- a/doc/pub/week39/html/week39-solarized.html +++ b/doc/pub/week39/html/week39-solarized.html @@ -79,6 +79,7 @@ div { text-align: justify; text-justify: inter-word; } None, 'revisiting-our-logistic-regression-case'), ('The equations to solve', 2, None, 'the-equations-to-solve'), + ('To be added', 2, None, 'to-be-added'), ("Solving using Newton-Raphson's method", 2, None, @@ -239,7 +240,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 28, 2021

    +

    Sep 29, 2021












    @@ -325,7 +326,7 @@ plt.legend() plt.show()

    -Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function RidgeCV instead of just calling the Ridge function. For RidgeCV we need to passe the array of \( \lambda \) values. +Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function RidgeCV instead of just calling the Ridge function. For RidgeCV we need to pass the array of \( \lambda \) values. By inspecting the figure we can in turn determine which is the optimal regularization parameter. This becomes however less functional in the long run. @@ -386,7 +387,10 @@ ypredictRidge = gridsearch.predict(X_test) print(f"R2 score: {R2(y_test,ypredictRidge)}")

    -By default the grid search function includes cross validation with five folds. The Scikit-Learn documentation contains more information on how to set the different parameters. +By default the grid search function includes cross validation with +five folds. The Scikit-Learn +documentation +contains more information on how to set the different parameters.

    If we take out the random noise, running the above codes results in \( \lambda=0 \) yielding the best fit. @@ -401,7 +405,10 @@ An alternative to the above manual grid set up, is to use a random search where the parameters are tuned from a random distribution (uniform below) for a fixed number of iterations. A model is constructed and evaluated for each combination of chosen parameters. -We repeat the previous example but now with a random search. +We repeat the previous example but now with a random search. Note +that values of \( \lambda \) are now limited to be within \( x\in +[0,1] \). This domain may not be the most relevant one for the specific +case under study.

    @@ -516,6 +523,17 @@ This defines what is called the Hessian matrix.











    +

    To be added

    + +

    +We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. + +

    +Make link with linear regression and the Hessian matrix from linear regression. + +

    +









    +

    Solving using Newton-Raphson's method

    diff --git a/doc/pub/week39/html/week39.html b/doc/pub/week39/html/week39.html index 6c91bc434..ed2405a4a 100644 --- a/doc/pub/week39/html/week39.html +++ b/doc/pub/week39/html/week39.html @@ -84,6 +84,7 @@ div { text-align: justify; text-justify: inter-word; } None, 'revisiting-our-logistic-regression-case'), ('The equations to solve', 2, None, 'the-equations-to-solve'), + ('To be added', 2, None, 'to-be-added'), ("Solving using Newton-Raphson's method", 2, None, @@ -244,7 +245,7 @@ MathJax.Hub.Config({

    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 28, 2021

    +

    Sep 29, 2021












    @@ -330,7 +331,7 @@ plt.legend() plt.show()

    -Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function RidgeCV instead of just calling the Ridge function. For RidgeCV we need to passe the array of \( \lambda \) values. +Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function RidgeCV instead of just calling the Ridge function. For RidgeCV we need to pass the array of \( \lambda \) values. By inspecting the figure we can in turn determine which is the optimal regularization parameter. This becomes however less functional in the long run. @@ -391,7 +392,10 @@ ypredictRidge = gridsearchprint(f"R2 score: {R2(y_test,ypredictRidge)}")

    -By default the grid search function includes cross validation with five folds. The Scikit-Learn documentation contains more information on how to set the different parameters. +By default the grid search function includes cross validation with +five folds. The Scikit-Learn +documentation +contains more information on how to set the different parameters.

    If we take out the random noise, running the above codes results in \( \lambda=0 \) yielding the best fit. @@ -406,7 +410,10 @@ An alternative to the above manual grid set up, is to use a random search where the parameters are tuned from a random distribution (uniform below) for a fixed number of iterations. A model is constructed and evaluated for each combination of chosen parameters. -We repeat the previous example but now with a random search. +We repeat the previous example but now with a random search. Note +that values of \( \lambda \) are now limited to be within \( x\in +[0,1] \). This domain may not be the most relevant one for the specific +case under study.

    @@ -521,6 +528,17 @@ This defines what is called the Hessian matrix.











    +

    To be added

    + +

    +We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. + +

    +Make link with linear regression and the Hessian matrix from linear regression. + +

    +









    +

    Solving using Newton-Raphson's method

    diff --git a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz index 2d12ed541..4f2bb3bb6 100644 Binary files a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz and b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz differ diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index 6e9d11a8c..2d9c9b2dd 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -10,7 +10,7 @@ " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Sep 28, 2021**\n", + "Date: **Sep 29, 2021**\n", "\n", "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", "\n", @@ -45,22 +45,12 @@ }, { "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAx1klEQVR4nO3deZyO9f7H8ddnFjOYQcOUZQxiji3OyFC2siU0oU6FitBJ2peTVh3nR4qSVqeoJlRHipRKi0IcS4xdtqwZZN+GMMvn98d9mTM0u7nnmnvuz/PxuB/u6/pe13W/r5zjc1/X93tfX1FVjDHGmLwKcDuAMcYY32KFwxhjTL5Y4TDGGJMvVjiMMcbkixUOY4wx+RLkdoCiUKlSJa1Zs6bbMYwxxqcsW7bsgKpGnr/eLwpHzZo1SUxMdDuGMcb4FBHZkdV6u1VljDEmX6xwGGOMyRcrHMYYY/LFa30cIpIAxAP7VPWyLNrrAe8DlwPPqOpoZ311YBJwCaDAeFV9zWmLAKYANYHtwC2qethb52CMv0lJSSEpKYlTp065HcUUodDQUKKioggODs7T9t7sHJ8AvImnCGTlEPAg0OO89anAP1R1uYiEA8tEZJaqrgOeBH5U1ZEi8qSz/IQ3whvjj5KSkggPD6dmzZqIiNtxTBFQVQ4ePEhSUhK1atXK0z5eu1WlqvPwFIfs2vep6lIg5bz1e1R1ufP+OLAeqOY0dwcmOu8n8ueiY4y5AKdOnaJixYpWNPyIiFCxYsV8XWUW6z4OEakJNAF+dlZdoqp7nPe/47mdZYwpRFY0/E9+/86LbeEQkTBgGvCwqh47v109z4PP9pnwIjJQRBJFJHH//v0FzrFizwpOpdr9XmOMOatYFg4RCcZTND5S1c8yNe0VkSrONlWAfdkdQ1XHq2qcqsZFRv7ph495cjLlJJ0+7ETMGzGMSxzHmbQzBTqOMSbvRITbb789Yzk1NZXIyEji4+MB2Lt3L/Hx8fz1r3+lQYMGdO3aFYDt27dTunRpYmNjM16TJv25i/Wrr76iSZMmGfuPGzcOgH/961+MHj06x2w1a9akUaNGNG7cmKuvvpodO/73+7iWLVtmuU+/fv2YOnVq/v4jnGfNmjUZ5xQREUGtWrWIjY2lY8eOeT7G559/zrp16y4ox1nFrnCI55rpPWC9qo45r3kGcIfz/g7gC29mKRNchik3TaF6ueoM+noQ9d6sx4SVE0hNT/Xmxxrj18qWLcvatWv5448/AJg1axbVqlXLaP/nP//JNddcw6pVq1i3bh0jR47MaKtduzYrV67MePXt2/ecY6ekpDBw4EC+/PJLVq1axYoVK2jbtm2+8s2ZM4fVq1fTtm1bnnvuuYz1CxcuLMDZ5k2jRo0yzqlbt2689NJLrFy5kh9++CHPx/CJwiEik4FFQF0RSRKRO0VkkIgMctori0gS8CgwxNmmHNAK6AO0F5GVzqurc9iRwDUi8ivQ0Vn2qva12rNgwAJm3jqTi0pfRP8v+vNz0s+572iMKbCuXbvy9ddfAzB58mR69+6d0bZnzx6ioqIylhs3bpzn4x4/fpzU1FQqVqwIQEhICHXr1i1QxhYtWrBr166M5bCwMMAzSun++++nbt26dOzYkX37/ndjZObMmdSrV4+mTZvy4IMPZlxFnThxggEDBtC8eXOaNGnCF1/k7Tvx999/T4sWLbj88su5+eabSU5OBuDJJ5+kQYMGNG7cmMcee4yFCxcyY8YMBg8eTGxsLFu2bCnQOZ/lteG4qto7l/bfgagsmv4LZNlTo6oHgQ4Xni5/RIQuMV3oXKcz83bMo1V0KwDGLBpDrQq16FGvh3UomhKp7YS2f1p3S8NbuLfZvZxMOUnXj7r+qb1fbD/6xfbjwMkD3PTJTee0ze03N0+f26tXL4YNG0Z8fDyrV69mwIABzJ8/H4D77ruPnj178uabb9KxY0f69+9P1apVAdiyZQuxsbEZx3njjTdo06ZNxnJERATdunWjRo0adOjQgfj4eHr37k1AQP6/Q3/77bf06NHjT+unT5/Oxo0bWbduHXv37qVBgwYMGDCAU6dOcffddzNv3jxq1ap1TjEcMWIE7du3JyEhgSNHjtC8eXM6duxI2bJls/38AwcO8Nxzz/HDDz9QtmxZRo0axZgxY7jvvvuYPn06GzZsQEQ4cuQIFSpUoFu3bsTHx3PTTTdle8y88ouHHBYWEeHqmlcDkJqeysRVE1m9dzVXRl3JqI6juKrGVS4nNKZkaNy4Mdu3b2fy5MkZfRhnXXvttWzdupVvv/2Wb775hiZNmrB27Vrgf7eqcvLuu++yZs0afvjhB0aPHs2sWbOYMGFCnrO1a9eOQ4cOERYWxvDhw//UPm/ePHr37k1gYCBVq1alffv2AGzYsIFLL70047cSvXv3Zvz48YDnymHGjBkZfSynTp3it99+o379+tnmWLx4MevWraNVK88X2TNnztCiRQvKly9PaGgod955J/Hx8RlXNYXJCkcBBQUEsWzgMiaunMjQuUO5esLVXBdzHa92fpU6EXXcjmdMocjpCqFMcJkc2yuVqZTnK4ysdOvWjccee4y5c+dy8ODBc9oiIiK49dZbufXWW4mPj2fevHk0bdo0z8du1KgRjRo1ok+fPtSqVStfhWPOnDlUqFCB2267jaFDhzJmzPldsfmnqkybNi1ft81UlWuuuYbJkyf/qW3JkiX8+OOPTJ06lTfffJPZs2dfcMbMil3nuC8JCgjizsvvZNMDmxjZYSQ/7/qZdE0HPH+pxpiCGzBgAEOHDqVRo0bnrJ89ezYnT54EPH0WW7ZsITo6Ok/HTE5OZu7cuRnLK1eupEaNGvnOFhQUxKuvvsqkSZM4dOjc3zlfddVVTJkyhbS0NPbs2cOcOXMAqFu3Llu3bmX79u0ATJkyJWOfa6+9ljfeeCPj340VK1bkmuHKK69kwYIFbN68GfD0k2zatInk5GSOHj1K165deeWVV1i1ahUA4eHhHD9+PN/nmhUrHIWgTHAZnmj9BDsf2clfKv4FgN7TevOP7/7BwZMHc9nbGJOVqKgoHnzwwT+tX7ZsGXFxcTRu3JgWLVrw97//nWbNmgH/6+M4+3r99dfP2VdVefHFF6lbty6xsbEMHTr0nKuN5557jqioqIxXTqpUqULv3r0ZO3bsOetvuOEGYmJiaNCgAX379qVFixYAlC5dmn//+9907tyZpk2bEh4eTvny5QF49tlnSUlJoXHjxjRs2JBnn3021/8+kZGRTJgwgd69e2f8t9iwYQPHjx8nPj6exo0b07p164wrol69evHSSy/RpEmTC+4cF3/4ZhwXF6dFOZFTanoqd395NxNWTSCsVBhPtHqCh654iLKlsu/oMqY4WL9+fY731c2FSU5OJiwsDFXlvvvuIyYmhkceecTtWEDWf/ciskxV487f1q44vCAoIIj3ur/H6kGraVuzLc/MfoaYN2JYtHOR29GMMS565513iI2NpWHDhhw9epS7777b7UgFYp3jXtTw4oZ80esLFvy2gBHzR1C3kqfj68DJA1QsbQ+SM8bfPPLII8XmCuNC2BVHEWgV3YqZt80konQE6ZpOpw860fr91ixOWux2NGP+xB9uX5tz5ffv3ApHEVNV7om7hy2HttDivRb0nNqTrYe3uh3LGMAzoc/BgwetePiRs/NxhIaG5nkf6xx3SfKZZF5a8BKjF40mJS2FH/v+SJsabXLf0RgvshkA/VN2MwBm1zluhcNlu4/v5vWfX2dYu2GUCizF2n1riYmIISQoxO1oxhg/Z6Oqiqmq4VUZ2XEkpQJLcSr1FJ0+6ET9sfX55JdP7HaBMaZYssJRjIQGhTKhxwTCQ8LpObUnLRNasuC3BW7HMsaYc1jhKGY61e7E8oHLSeiWwI4jO2j9fmuW7FridixjjMlghaMYCgwIpH+T/vz6wK8kdEugWVXP4xTmbJvDiTMnXE5njPF3VjiKsbKlytK/SX9EhEN/HCJ+cjz1xtZjytop1v9hjHGNFQ4fEVE6gu9v/57IMpH0mtaLdhPbsXrvardjGWP8kBUOH9IquhVL71rK29e9zdp9a4kbH8fu47vdjmWM8TPenHM8QUT2icjabNrricgiETktIo/lZV8R+ZeI7MpiLnK/ERgQyN1xd7PpgU1MumESVcM9U2au+n2Vy8mMMf7Cm1ccE4DOObQfAh4ERudz31dUNdZ5zbyghD4sonQEvS7rBcCSXUuIHRfLjVNutCsQY4zXea1wqOo8PMUhu/Z9qroUSMnvvuZcTSo3YWSHkXyz+Rvqj63P24lvZ8xEaIwxhc0X+zjuF5HVzu2si7LbSEQGikiiiCTu37+/KPMVueDAYJ5o/QRr71lLs6rNuOfre+j6UVcbeWWM8QpfKxxvAbWBWGAP8HJ2G6rqeFWNU9W4yMjIIornrtoRtZnVZxbvd3+fG+vfmDHfhxUQY0xh8qnCoap7VTVNVdOBd4DmbmcqbkSEfrH9GNh0IAAfr/2YGz+5kf0nSvZVlzGm6PhU4RCRKpkWbwCyHLFl/ufwH4eZ+etMGr3ViLnb57odxxhTAnhzOO5kYBFQV0SSROROERkkIoOc9soikgQ8CgxxtimX3b7OYV8UkTUishpoB/j+HIxedk+ze1h611IqhFagw6QOPD//ees4N8ZcEJuPw08cP32cgV8N5OO1HzOrzyw6XtrR7UjGmGLO5uPwc+Eh4fznxv8w9465GUUj+Uyyy6mMMb7ICocfERGurnk1AIm7E6n5ak2mrpvqcipjjK+xwuGnospFEVMxhps/vZkR80bYkF1jTJ5Z4fBTlcMqM+eOOdzW6DaGzBlC38/7cibtjNuxjDE+wAqHHwsNCuWDGz5geLvhfLj6Q8YvG+92JGOMDwhyO4Bxl4gw5KohNK/WnA61OrgdxxjjA+yKwwCeuc4DAwLZdWwX8f+Jt6fsGmOyZYXDnGPbkW38tOMn2k1sx57je9yOY4wphqxwmHO0jm7NN7d9w65ju+gwqQN7k/e6HckYU8xY4TB/0jq6NTNvm8mOozvoMKmDPSDRGHMOKxwmS1fVuIqven9FhdAKBIj9z8QY8z82qspkq12tdsyvOR8RISUthTRNIzQo1O1YxhiX2VdJkyMRIV3T6TGlB72n9SYtPc3tSMYYl1nhMLkKkACurX0tn2/4nAe+ecAeT2KMn7NbVSZPHrziQXYd28WLC1+kTkQdHm3xqNuRjDEuscJh8uyFji+w5fAWBs8aTP1K9ekS08XtSMYYF1jhMHkWIAFM7DGR09NOE1k20u04xhiXeHPq2AQR2SciWc4LLiL1RGSRiJwWkcfysq+IRIjILBH51fnzIm/lN1krW6osX/b+kriqnknBUtNTXU5kjClq3uwcnwB0zqH9EPAgMDof+z4J/KiqMcCPzrJxyeOzHufmT2+2znJj/IzXCoeqzsNTHLJr36eqS4GUfOzbHZjovJ8I9LjwpKagqoZX5fMNn/PK4lfcjmKMKUK+Nhz3ElU9++S934FLsttQRAaKSKKIJO7fb4/M8IaHrniIG+rdwBM/PMHCnQvdjmOMKSK+VjgyqOf+SLb3SFR1vKrGqWpcZKR15HqDiJDQPYHo8tH0nNqTAycPuB3JGFMEfK1w7BWRKgDOn/tczuP3KoRW4NObP+XEmRMs37Pc7TjGmCLga8NxZwB3ACOdP79wN44BuLzK5ex4eAfhIeFuRzHGFAFvDsedDCwC6opIkojcKSKDRGSQ015ZRJKAR4EhzjblstvXOexI4BoR+RXo6CybYiA8JBxVZdKqSWw7vM3tOMYYLxJ/GEoZFxeniYmJbsco8fad2EfMGzE0urgRP/X7icCAQLcjGWMugIgsU9W489f7Wh+HKcYuLnsxY7uOZcHOBby86GW34xhjvMQKhylUtzW6jR71ejB07lB+Pfir23GMMV5ghcMUKhFhbNexhASGcNeXd9mvyo0pgXxtVJXxAVXDq/J2/NuEBoUiIm7HMcYUMiscxit6XdYr472qWgExpgSxW1XGq15c8CL9vujndgxjTCGywmG86kzaGSatmsSsLbPcjmKMKSRWOIxXPdbyMepE1OH+b+7ndOppt+MYYwqBFQ7jVaFBobzR5Q02Hdxkj183poSwwmG8rnOdztxQ7waen/88R08ddTuOMeYC2agqUyRe7fwqB04eoHxoebejGGMukBUOUySiy0cTXT4a8HSYlwos5XIiY0xB2a0qU6QGfz+Y9hPb2y/KjfFhVjhMkapbqS4Ldi5g6rqpbkcxxhSQFQ5TpPrH9qfRxY144ocnbHiuMT7KCocpUoEBgYzuNJptR7bx5pI33Y5jjCkAKxymyHWq3YnOdTozZvEYzqSdcTuOMSafvDl1bIKI7BORtdm01xORRSJyWkQeO6+ts4hsFJHNIvJkpvUTRGSbiKx0XrHeym+86+3r3mbZwGU2usoYH+TNK44JQOcc2g8BDwKjM68UkUBgLNAFaAD0FpEGmTYZrKqxzmtloSY2RaZGhRpUDquMqlpfhzE+xmuFQ1Xn4SkO2bXvU9WlQMp5Tc2Bzaq6VVXPAB8D3b2V07gnLT2N9pPaM3jWYLejGGPyoTj2cVQDdmZaTnLWnTVCRFaLyCsiEpLdQURkoIgkikji/v37vZXVXIDAgEBiImIYt2wcvx39ze04xpg8Ko6FIydPAfWAZkAE8ER2G6rqeFWNU9W4yMjIospn8unZq54FYNhPw1xOYozJq+JYOHYB1TMtRznrUNU96nEaeB/PbS3jw6qXr849cfcwYeUENh3c5HYcY0weFMfCsRSIEZFaIlIK6AXMABCRKs6fAvQAshyxZXzLU62fIiQohNELR+e+sTHGdV57yKGITAbaApVEJAkYCgQDqOrbIlIZSATKAeki8jDQQFWPicj9wHdAIJCgqr84h/1IRCIBAVYCg7yV3xSdS8Iu4fvbv6dJlSZuRzHG5IH4w8Pm4uLiNDEx0e0YJg9UFc8FpTHGbSKyTFXjzl9fHG9VGT/1c9LP1B9bny2HtrgdxRiTAyscptiILh/N9iPbeeG/L7gdxRiTAyscptioEl6Fuy6/i4mrJrLjyA634xhjsmGFwxQrj7d6HEEYtWCU21GMMdmwwmGKlerlq9Mvth/vrXiP3cd3ux3HGJMFm3PcFDtPt3maTrU7UTmssttRjDFZsMJhip2aFWpSs0JNt2MYY7Jht6pMsaSqDPtpGGMWjXE7ijHmPDkWDhG5PdP7Vue13e+tUMaICMv3LGfE/BGcOHPC7TjGmExyu+J4NNP7N85rG1DIWYw5x+OtHufQH4dIWJHgdhRjTCa5FQ7J5n1Wy8YUqpbVW9KyekvGLB5Danqq23GMMY7cCodm8z6rZWMK3eMtH2f7ke1MXTfV7SjGGEduo6rqichqPFcXtZ33OMuXejWZMcD1da9nUNNBxETEuB3FGOPIrXDUL5IUxmQjQAJ4K/4tt2MYYzLJ8VaVqu7I/AKSgcuBSs6yMUViy6EtvLb4NbdjGGPIfTjuVyJymfO+Cp4Z9wYAHzgTLxlTJKaum8rD3z3Mij0r3I5ijN/LrXO8lqqenZ61PzBLVa8HrsCG45oidHfc3YSVCmP0Ipte1hi35VY4UjK97wDMBFDV40B6bgcXkQQR2SciWc4NLiL1RGSRiJwWkcfOa+ssIhtFZLOIPJlpfS0R+dlZP8WZl9yUcBVCK3B307uZsnYKvx39ze04xvi13ArHThF5QERuwNO38S2AiJTGmT88FxOAzjm0HwIeBM75GikigcBYoAvQAOgtIg2c5lHAK6paBzgM3JmHHKYEeKD5AyjKm0vedDuKMX4tt8JxJ9AQ6Af0VNUjzvorgfdzO7iqzsNTHLJr36eqSzn3ygagObBZVbeq6hngY6C7eCajbg+cHdQ/EeiRWw5TMtSoUIPbG99OuuZ6sWuM8aIch+Oq6j5gUBbr5wBzvBUKqAbszLSchKdfpSJwRFVTM62v5sUcppiZ0H0Cnu8Pxhi35Fg4RGRGTu2q2q1w4xQeERkIDASIjo52OY0pLGeLxrLdy2hSpQkBYg94Nqao5fYDwBZ4vvlPBn6m6J5PtQuonmk5yll3EKggIkHOVcfZ9X+iquOB8QBxcXH2eJQS5NvN39Lloy7M6DWD6+te73YcY/xObl/XKgNPA5cBrwHXAAdU9SdV/cmLuZYCMc4IqlJAL2CGqiqeW2Q3OdvdAXzhxRymGOpQqwNR5aJ4ZfErbkcxxi/l9svxNFX9VlXvwNMhvhmYm9e5OERkMrAIqCsiSSJyp4gMEpFBTntlEUnC8/j2Ic425ZyrifuB74D1wCeq+otz2CeAR0VkM54+j/fyfdbGpwUHBvNA8weYs30OK39f6XYcY/yOeL7E57CBSAhwHdAbqAnMABJUNctbRMVRXFycJiYmuh3DFKLDfxwm6pUobml4C+93z3WAnzGmAERkmarGnb8+t87xSXhuU80E/i/Tr8iNcdVFpS+if2x/pq6byqnUU4QGhbodyRi/keMVh4ikA2fn7cy8oQCqquW8mK3Q2BVHyXTg5AFCg0IJKxXmdhRjSqQCXXGoqo11NMVWpTKVAEjXdNI1naCA3AYJGmMKgxUG49P2Ju+l4b8bMnHlRLejGOM3rHAYn3Zx2YspFViKVxa/Qm4DPYwxhcMKh/FpIsIjVz7CL/t/4cdtP7odxxi/YIXD+Lzel/XmkrKX2A8CjSkiVjiMzwsJCuHeZvcy89eZbDiwwe04xpR4NgzFlAiD4gZRt2Jdal9U2+0oxpR4VjhMiXBx2YvpeVlPt2MY4xfsVpUpMVSV5+c/z7jEcW5HMaZEs8JhSgwRYc72OQyfN5yUtPMnlTTGFBYrHKZEeeTKR9h1fBefrvvU7SjGlFhWOEyJ0rlOZ+pWrGs/CDTGi6xwmBIlQAJ46IqHSNydyMKdC92OY0yJZKOqTInT9699mbtjLmWCy7gdxZgSyQqHKXHKlirLlJumuB3DmBLLblWZEmvb4W3M2DjD7RjGlDheKxwikiAi+0Qky1kDxeN1EdksIqtF5PJMbaNEZK3z6plp/QQR2SYiK51XrLfyG9/3zOxn6DO9D8dPH3c7ijElijevOCYAnXNo7wLEOK+BwFsAInIdcDkQC1wBPCYimWcaHKyqsc5rZeHHNiXFw1c+zLHTx0hYkeB2FGNKFK8VDlWdBxzKYZPuwCT1WAxUEJEqQANgnqqmquoJYDU5FyBjstS8WnNaVm/J60teJy09ze04xpQYbvZxVAN2ZlpOctatAjqLSBkRqQS0A6pn2m6Ec2vrFREJye7gIjJQRBJFJHH//v3eyG98wCNXPsLWw1v5ctOXbkcxpsQodp3jqvo9MBNYCEwGFgFnvy4+BdQDmgERwBM5HGe8qsapalxkZKR3Q5tiq0e9Hvyl4l/YeGCj21GMKTHcHI67i3OvJKKcdajqCGAEgIj8B9jkrN/jbHtaRN4HHiuytMYnBQUEseaeNZQKLOV2FGNKDDevOGYAfZ3RVVcCR1V1j4gEikhFABFpDDQGvneWqzh/CtADyHLEljGZnS0aO4/uzGVLY0xeeO2KQ0QmA22BSiKSBAwFggFU9W08t6O6ApuBk0B/Z9dgYL6nNnAMuF1VU522j0QkEhBgJTDIW/lNyfL6z6/z2PePsf3h7VQNr+p2HGN8mvjDg+Di4uI0MTHR7RjGRVsObSHmjRiebvM0z7V/zu04xvgEEVmmqnHnry92nePGeEPtiNp0r9edtxLfIvlMsttxjPFpVjiM3xjccjCH/jjEe8vfczuKMT7NCofxGy2rt6R1dGveW/GezdVhzAWwp+Mav5LQLYFLwi7BGXxhjCkAKxzGr8RUjAEgXdMRxAqIMQVgt6qM39l6eCuN32rMN5u/cTuKMT7JCofxO9XLVefY6WOMWjDK7SjG+CQrHMbvBAcG82iLR5m3Yx6Lkxa7HccYn2OFw/ilv1/+dy4KvciuOowpACscxi+FlQrjvmb38cWGL9hwYIPbcYzxKTaqyvitB694kJiKMdS+qLbbUYzxKVY4jN+KLBtJ37/2dTuGMT7HblUZv/fGz28wZPYQt2MY4zOscBi/t+HABl5c8KLN12FMHlnhMH7vidaeGYhH/neky0mM21SVtPQ0TqWe4lTqKbfjFFvWx2H8XnT5aPrH9ufdFe/ydJunqVaumtuRjJekpKWwZt8a1u5bS2hQKLc0vAWAVgmtWLtvLcdOH8vY9m/1/8bUW6YCUH9sfU6nnqZKeBWqhlelWng12tZsS496PTKOGxwYXOTn4xYrHMYAT7V5ioSVCYxaMIrXu7zudhxTyEYvHM2MjTNI3J3IH6l/ANCsarOMwtEmug1xVeIoF1KO4MBgggOCqVepXsb+Xet0Ze+Jvew+vps1e9fw7eZvSUlLoUe9HqSkpVBuZDmql6tOvUr1iK0cS5PKTWhRvQWVwyq7cr7e5tUZAEUkAYgH9qnqZVm0C/AanilkTwL9VHW50zYKuM7ZdLiqTnHW1wI+BioCy4A+qnompxw2A6DJizGLxtCkchPa1WrndhRzAdI1nTnb5vDVpq8Yc+0YRIS/z/g7a/atoUVUC1pEtSC2ciyXXnTpBV0lpKanEhQQRPKZZEb9dxQbDm5g/f71rD+wnnRNZ1THUTze6nH2n9jPy4teJq5qHM2qNiO6fLTPPFwzuxkAvV04rgKSgUnZFI6uwAN4CscVwGuqeoWIXAc8DHQBQoC5QAdVPSYinwCfqerHIvI2sEpV38ophxUOY0q+Pcf38O+l/2biqonsPLaTciHl2HDfBqqEV0FVi+wf65MpJ1mzdw1Vw6tSvXx15u+YT4dJHUhJTwEgskwkzao1Y0T7EcRWji3SbPnlytSxqjoPOJTDJt3xFBVV1cVABRGpAjQA5qlqqqqeAFYDnZ0rlPbAVGf/iUAPr52A8TsHTx7k4W8fZuvhrW5HMfkwf8d8arxagxHzR3DZxZfx8d8+5vd//E6V8CoARfoPc5ngMlwRdQXVy1cHoE2NNhx/6jhL71rK2K5jue4v17HjyA6CAzxXOxNWTqDGqzW46ZObGPXfUczeNvucvpbiyO0+jmpA5jGQSc66VcBQEXkZKAO0A9bhuT11RFVTz9v+T0RkIDAQIDo62ivhTclzKvUU45aN4+AfB/nghg/cjmNysOngJnYe3UmHSzvQvFpzHrnyEQY2HUjtiOL3JICQoBDiqsYRV/VPX96JLh9Ni6gWLN29lGnrpwEgCAceP0BE6QgSdyeSkpZCbOVYSgeXLuroWXK7cGRJVb8XkWbAQmA/sAhIy+cxxgPjwXOrqtBDmhKpWrlqPND8AUYvHM3jLR+n0SWN3I5kzrP/xH6GzB7Ceyveo05EHdbft56QoBBGXeObD6zscGkHOlzaAYADJw+QuDuR9fvXE1E6AoAX/vsCn63/jKCAIBpd3IhmVZvRKrqVq0898GofB4CI1AS+yqaPYxwwV1UnO8sbgbaquue87f4DfAh8g6eQVFbVVBFpAfxLVa/NKYP1cZj8OPTHIS597VKuqnEVM3rPcDuOcaSlpzFu2Tiemf0MyWeSuSfuHp5p8wyXhF3idjSv2n18N0t2LWHprqUs2b2ExN2JxETEsOSuJQDc+/W9lAosRZPKTagfWZ/6leoTHhJeKJ+dXR+H21ccM4D7ReRjPJ3jR1V1j4gEAhVU9aCINAYaA9+rqorIHOAmPCOr7gC+cCu8KZkiSkcwuOVghswZwsKdC2lZvaXbkQzww9YfuG/mfbSv1Z43u7xJ/cj6bkcqElXDq9KjXo+M34yoKof+OJTxfsvhLczfMT9jmDHAnU3u5N1u73otk1cLh4hMBtoClUQkCRgKBAOo6tvATDwjqjbjGY7b39k1GJjvdGgdA27P1K/xBPCxiDwHrADe8+Y5GP/00JUPsePojhI7Dt9XpKansnzPcppXa06n2p34se+PtKvZrtiOQioKIkLFMhUz3n93+3ekpqey5dAW1h9Yz/r964mpGOPdDN6+VVUc2K0qY3zPxgMb6ft5X1bvXc3mBzbbL/pd4MpwXGN83cYDGxn45UDOpOX4G1NTyD5c/SGXj7+czYc2M6H7BCsaxYwVDmNysPXwVt5Z/g5vLnnT7Sh+QVUZ+OVA+kzvQ9MqTVk9aDU9L+vpdixzHiscxuSgS0wXOtfpzLCfhrH/xH6345R4IkL5kPI81fopZt8x2640iikrHMbk4uVOL5N8Jpmhc4e6HaXEmr1tNkt2eYaXvnjNizzf4XmCAtwe9GmyY4XDmFw0iGzAPXH3MG7ZOFbsWeF2nBJFVRm7ZCydPuiUMQujP4+Y8hVW0o3Jg2HthlEupBx1Iuq4HaXESElL4YFvHmDcsnFc/5fr+fDGD92OZPLICocxeXBR6YsY0WGE2zFKjOQzyfT4uAc/bvuRJ1s9yXPtnyMwINDtWCaP7FaVMfmwfM9yWiW04vfk392O4tNCg0IJDwlnYo+JvNDxBSsaPsYKhzH5UDa4LIm7E3ns+8fcjuKTNh3cxO/JvxMUEMRnt3zm6oP6TMFZ4TAmH+pWqsuTrZ7kozUfMfPXmW7H8SmLkxbT8r2W9P/C82Qh6wT3XVY4jMmnp9s8zWUXX8ZdX97F4T8Oux3HJ3y58UvaT2xPhdAKvNnFfkzp66xwGJNPIUEhTOwxkX0n9vHGkjfcjlPsvbv8XXpM6UHDixuy8M6FxXKiJZM/NqrKmAK4vMrlzL1jLldGXel2lGLtVOopXl70Mtdceg1Tb5lKWKkwtyOZQmCFw5gCahXdCoDfk38nQAK4uOzFLicqPlLTU0nXdEKDQplzxxwqlq5IcGCw27FMIbFbVcZcgDNpZ2j5Xktu++w20tLzNbtxiXUy5SR/++Rv9J3eF1WlclhlKxoljBUOYy5AqcBSPN3maX7Y+gMv/PcFt+O47tAfh7jmg2v4cuOXtIluYyOnSigrHMZcoDub3MmtjW5l6Nyh/LT9J7fjuGb7ke20fK8ly3Yv45ObP+G+5ve5Hcl4iVcLh4gkiMg+EVmbTbuIyOsisllEVovI5ZnaXhSRX0RkvbONOOvnishGEVnpvOzGsnGViPD2dW9TJ6IOvaf19stflaelp9H1o67sPbGX7/t8z00NbnI7kvEib19xTAA659DeBYhxXgOBtwBEpCXQCmgMXAY0A67OtN9tqhrrvPZ5Ibcx+RIeEs6nN39Kq+hWlAku43acIhcYEMj468ezYMACrqpxldtxjJd5dVSVqs4TkZo5bNIdmKSeic8Xi0gFEakCKBAKlAIECAb2ejOrMReq8SWN+fTmTwHPMNSQwJASf49/wsoJ7D+xn8GtBtM6urXbcUwRcbuPoxqwM9NyElBNVRcBc4A9zus7VV2fabv3ndtUz0o2/88UkYEikigiifv328xtpugcO32MNu+3Yfi84W5H8RpVZfhPw+n/RX9mbZ1lI8r8jNuFI0siUgeoD0ThKS7tRaSN03ybqjYC2jivPlkdQ1XHq2qcqsZFRkYWRWxjAAgvFU6DyAYMnTuUiSsnuh2n0J1KPUXfz/vyz7n/pE/jPnx161f2dFs/43bh2AVUz7Qc5ay7AVisqsmqmgx8A7QAUNVdzp/Hgf8AzYs0sTG5EBHGx4+n46UdGTBjAJ+t/8ztSIUmLT2NjpM68uHqDxnebjgTe0ykVGApt2OZIuZ24ZgB9HVGV10JHFXVPcBvwNUiEiQiwXg6xtc7y5UAnPXxQJYjtoxxU0hQCNN7TueKalfQa2ovZm+b7XakQhEYEEifxn2Ydss0hlw1pMT34ZisebVzXEQmA22BSiKSBAzF09GNqr4NzAS6ApuBk0B/Z9epQHtgDZ6O8m9V9UsRKQt85xSNQOAH4B1vnoMxBRVWKoyvb/2a/l/0JyYixu04F2TSqklUCK1At7rduDvubrfjGJeJZ0BTyRYXF6eJiYluxzB+Li09jcVJizOeceULTqac5IGZD5CwMoH4v8Qzo9cMu8rwIyKyTFXjzl/v9q0qY/zG6z+/Tpv32zB64Wh84Qvbuv3raP5Oc95f+T5D2gxhes/pVjQMYE/HNabIDIobxKKkRQyeNZj1+9fzVvxbxbZjefuR7TR7pxllg8vy3e3fcU3ta9yOZIoRKxzGFJHSwaX5+KaPqTe3HsPnDeeX/b8w+W+TqXVRLbejZThx5gRlS5WlZoWaDGs7jN6NelM1vKrbsUwxY7eqjClCARLAsHbD+PTmT9l2ZBtHTh1xOxLg+W3G8/OfJ/rVaNbtXwfAP1r+w4qGyZIVDmNccFODm9j20DaaVGkCwKuLX+W3o78VeQ5V5YsNX9Dw3w15ZvYztK3ZlnIh5Yo8h/EtVjiMccnZhyHuPLqTZ2Y/Q/2x9fm/uf/HsdPHiuTz0zWddhPb0WNKD0ICQ5jVZxbTbplGVLmoIvl847uscBjjsurlq7Pu3nV0rtOZf/30Ly597VJG/nckyWeSC/2zTqWeYvr66agqARJAp9qdGBc/jlWDVtHx0o6F/nmmZLLfcRhTjCTuTmTI7CEsSlrErkd3EVYqjK2HtxJVLqrAI7BS01OZv2M+09ZPY/LayRz64xBL/r6EZtWaFXJ6U9Jk9zsOG1VlTDESVzWOb2//ln0n9hFWKgyA7h93J+lYEh0v7UiLqBY0r9acepXqUalMpT/tr6rsPr6boIAgLgm7hFW/r6L9pPYc+uMQpYNKc33d67m76d3EVf3TvwXG5JkVDmOKoYvLeia2VFWeb/88n234jLnb5zJ13VQA+sf2J6F7AumaTswbMQRIAMdOH+PwH4dJSU/h2aueZVi7YdSOqE33ut25LuY6OtfpTNlSZd08LVNCWOEwphgTEa6vez3X170egN+Tf2fZ7mUZhSUtPY2W1VuSmp5K+ZDyVAitQI3yNTImVQorFUZC9wTX8puSyfo4jDHGZMmeVWWMMaZQWOEwxhiTL1Y4jDHG5IsVDmOMMflihcMYY0y+WOEwxhiTL1Y4jDHG5IsVDmOMMfniFz8AFJH9wI4C7l4JOFCIcdxk51L8lJTzADuX4upCzqWGqkaev9IvCseFEJHErH456YvsXIqfknIeYOdSXHnjXOxWlTHGmHyxwmGMMSZfrHDkbrzbAQqRnUvxU1LOA+xciqtCPxfr4zDGGJMvdsVhjDEmX6xwGGOMyRcrHPkgIv8QERWRP0/27CNEZLiIrBaRlSLyvYhUdTtTQYjISyKywTmX6SJSwe1MBSUiN4vILyKSLiI+OQRURDqLyEYR2SwiT7qdp6BEJEFE9onIWrezXAgRqS4ic0RknfO/rYcK8/hWOPJIRKoDnYDf3M5ygV5S1caqGgt8BfzT5TwFNQu4TFUbA5uAp1zOcyHWAjcC89wOUhAiEgiMBboADYDeItLA3VQFNgHo7HaIQpAK/ENVGwBXAvcV5t+JFY68ewV4HPDp0QSqeizTYll89HxU9XtVTXUWFwNRbua5EKq6XlU3up3jAjQHNqvqVlU9A3wMdHc5U4Go6jzgkNs5LpSq7lHV5c7748B6oFphHT+osA5UkolId2CXqq4SEbfjXDARGQH0BY4C7VyOUxgGAFPcDuHHqgE7My0nAVe4lMWcR0RqAk2AnwvrmFY4HCLyA1A5i6ZngKfx3KbyCTmdi6p+oarPAM+IyFPA/cDQIg2YR7mdh7PNM3guyz8qymz5lZdzMaawiUgYMA14+Ly7DRfECodDVTtmtV5EGgG1gLNXG1HAchFprqq/F2HEPMvuXLLwETCTYlo4cjsPEekHxAMdtJj/ICkffye+aBdQPdNylLPOuEhEgvEUjY9U9bPCPLYVjlyo6hrg4rPLIrIdiFNVn3xypojEqOqvzmJ3YIObeQpKRDrj6XO6WlVPup3Hzy0FYkSkFp6C0Qu41d1I/k0833LfA9ar6pjCPr51jvufkSKyVkRW47n9VqjD9IrQm0A4MMsZWvy224EKSkRuEJEkoAXwtYh853am/HAGKdwPfIenE/YTVf3F3VQFIyKTgUVAXRFJEpE73c5UQK2APkB75/8fK0Wka2Ed3B45YowxJl/sisMYY0y+WOEwxhiTL1Y4jDHG5IsVDmOMMflihcMYY0y+WOEwfkdEki9g3/udJ8Ce85Rk8XjdaVstIpdnaqsiIl8579uefX+hRGRuXp6mKyLbc3uis4j8ICIXFUYuU/JZ4TAmfxYAHYEd563vAsQ4r4HAW5naHgXeKZJ0BfcBcK/bIYxvsMJh/JZzlfCS84PINSLS01kfICL/dub7mCUiM0XkJgBVXaGq27M4XHdgknosBiqISBWn7W/At1l8fnMRWSQiK0RkoYjUddb3E5HPnc/e7lzlPOpst1hEIjIdpo/z4661ItLc2b+iM9fKLyLyLiCZPvNzEVnmtA3MdJwZQO+C/rc0/sUKh/FnNwKxwF/xXEW85PxjfyNQE8/cEn3w/KI7N1k9Ibaa8xiOw6p6Oot9NgBtVLUJnnlRns/UdpmToxkwAjjpbLcIz5ONzyrjzK1yL5DgrBsK/FdVGwLTgehM2w9Q1aZAHPCgiFQEUNXDQMjZZWNyYs+qMv6sNTBZVdOAvSLyE55/qFsDn6pqOvC7iMy5gM+oAuzPpq08MFFEYvDMixKcqW2OM4/CcRE5CnzprF8DNM603WTwzCMhIuXEMxPiVXiKDqr6tYgczrT9gyJyg/O+Op5bawed5X1A1UzLxmTJrjiMKRzZPSH2DyA0m32G4ykQlwHXn7dd5iuU9EzL6Zz7he/8ZwZl+wwhEWmL58qqhar+FVhx3meGOnmNyZEVDuPP5gM9RSRQRCLxfFNfgqcD/G9OX8clQNs8HGsG0NfpN7kSOKqqe/BMa1szm33K87/Hj/cr4Dmc7Zdp7XzmUTxT0N7qrO8CnB0tVR7PbbOTIlIPz5SiONsJnvlCthcwh/EjVjiMP5sOrAZWAbOBx505Vqbh6aNYB3wILMczWyIi8qDzJNsoYLXT+QyeeU22ApvxjKC6F0BVTwBbRKROFp//IvCCiKyg4LeNTzn7vw2cfZLr/wFXicgveG5Z/eas/xYIEpH1wEg8U+6e1RRYnGk6XmOyZU/HNSYLIhKmqslOZ/ESoFVBJ+5y+hSaquqQQg1ZiETkNWCGqv7odhZT/FnnuDFZ+8rpaC4FDL+Q2R5VdboPjFZaa0XD5JVdcRhjjMkX6+MwxhiTL1Y4jDHG5IsVDmOMMflihcMYY0y+WOEwxhiTL/8PBg5bADYVH7UAAAAASUVORK5CYII=\n", - "text/plain": [ - "

    " - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], + "execution_count": null, + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "%matplotlib inline\n", "\n", @@ -114,7 +104,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Here we have performed a rather data greedy calculation as function of the regularization parameter $\\lambda$. There is no resampling here. The latter can easily be added by employing the function **RidgeCV** instead of just calling the **Ridge** function. For **RidgeCV** we need to passe the array of $\\lambda$ values.\n", + "Here we have performed a rather data greedy calculation as function of the regularization parameter $\\lambda$. There is no resampling here. The latter can easily be added by employing the function **RidgeCV** instead of just calling the **Ridge** function. For **RidgeCV** we need to pass the array of $\\lambda$ values.\n", "By inspecting the figure we can in turn determine which is the optimal regularization parameter.\n", "This becomes however less functional in the long run. \n", "\n", @@ -129,23 +119,12 @@ }, { "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "GridSearchCV(estimator=Ridge(),\n", - " param_grid={'alpha': array([1.00000000e-04, 4.64158883e-04, 2.15443469e-03, 1.00000000e-02,\n", - " 4.64158883e-02, 2.15443469e-01, 1.00000000e+00, 4.64158883e+00,\n", - " 2.15443469e+01, 1.00000000e+02])})\n", - "Best estimated lambda-value: 100.0\n", - "MSE score: 1.0892144853354966\n", - "R2 score: -0.0038332550504751595\n" - ] - } - ], + "execution_count": null, + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "import numpy as np\n", "from sklearn.model_selection import train_test_split\n", @@ -195,7 +174,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "By default the grid search function includes cross validation with five folds. The [Scikit-Learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV) contains more information on how to set the different parameters.\n", + "By default the grid search function includes cross validation with\n", + "five folds. The [Scikit-Learn\n", + "documentation](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV)\n", + "contains more information on how to set the different parameters.\n", "\n", "If we take out the random noise, running the above codes results in $\\lambda=0$ yielding the best fit. \n", "\n", @@ -206,26 +188,20 @@ "search where the parameters are tuned from a random distribution\n", "(uniform below) for a fixed number of iterations. A model is\n", "constructed and evaluated for each combination of chosen parameters.\n", - "We repeat the previous example but now with a random search." + "We repeat the previous example but now with a random search. Note\n", + "that values of $\\lambda$ are now limited to be within $x\\in\n", + "[0,1]$. This domain may not be the most relevant one for the specific\n", + "case under study." ] }, { "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "RandomizedSearchCV(estimator=Ridge(), n_iter=100,\n", - " param_distributions={'alpha': })\n", - "Best estimated lambda-value: 0.9849967686928113\n", - "MSE score: 1.0853136633465326\n", - "R2 score: -0.0002382102844775691\n" - ] - } - ], + "execution_count": null, + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "import numpy as np\n", "from sklearn.model_selection import train_test_split\n", @@ -357,6 +333,15 @@ "source": [ "This defines what is called the Hessian matrix.\n", "\n", + "\n", + "## To be added\n", + "\n", + "We will add here an example which computes the likelihood $p_i$, sets up the gradient and the Hessian matrix.\n", + "\n", + "Make link with linear regression and the Hessian matrix from linear regression.\n", + "\n", + "\n", + "\n", "## Solving using Newton-Raphson's method\n", "\n", "If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives. \n", @@ -1033,7 +1018,10 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -1068,7 +1056,10 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1086,7 +1077,10 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = guesses[-1]\n", @@ -1103,7 +1097,10 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def f1d(alpha):\n", @@ -1125,7 +1122,10 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1501,7 +1501,10 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = 2*np.random.rand(m,1)\n", @@ -1668,7 +1671,10 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "\n", @@ -1729,7 +1735,10 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -1813,7 +1822,10 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from random import random, seed\n", @@ -1994,7 +2006,10 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np \n", @@ -2056,7 +2071,10 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np \n", @@ -2095,7 +2113,10 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -2172,25 +2193,7 @@ ] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.5" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 4 } diff --git a/doc/src/week39/week39.do.txt b/doc/src/week39/week39.do.txt index af50f5af2..2acc140e7 100644 --- a/doc/src/week39/week39.do.txt +++ b/doc/src/week39/week39.do.txt @@ -78,7 +78,7 @@ plt.show() !ec -Here we have performed a rather data greedy calculation as function of the regularization parameter $\lambda$. There is no resampling here. The latter can easily be added by employing the function _RidgeCV_ instead of just calling the _Ridge_ function. For _RidgeCV_ we need to passe the array of $\lambda$ values. +Here we have performed a rather data greedy calculation as function of the regularization parameter $\lambda$. There is no resampling here. The latter can easily be added by employing the function _RidgeCV_ instead of just calling the _Ridge_ function. For _RidgeCV_ we need to pass the array of $\lambda$ values. By inspecting the figure we can in turn determine which is the optimal regularization parameter. This becomes however less functional in the long run. @@ -137,7 +137,10 @@ print(f"R2 score: {R2(y_test,ypredictRidge)}") !ec -By default the grid search function includes cross validation with five folds. The "Scikit-Learn documentation":"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV" contains more information on how to set the different parameters. +By default the grid search function includes cross validation with +five folds. The "Scikit-Learn +documentation":"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV" +contains more information on how to set the different parameters. If we take out the random noise, running the above codes results in $\lambda=0$ yielding the best fit. @@ -149,8 +152,10 @@ An alternative to the above manual grid set up, is to use a random search where the parameters are tuned from a random distribution (uniform below) for a fixed number of iterations. A model is constructed and evaluated for each combination of chosen parameters. -We repeat the previous example but now with a random search. - +We repeat the previous example but now with a random search. Note +that values of $\lambda$ are now limited to be within $x\in +[0,1]$. This domain may not be the most relevant one for the specific +case under study. !bc pycod @@ -258,6 +263,16 @@ $p(y_i\vert x_i,\bm{\beta})(1-p(y_i\vert x_i,\bm{\beta})$, we can obtain a compa !et This defines what is called the Hessian matrix. + +!split +===== To be added ===== + +We will add here an example which computes the likelihood $p_i$, sets up the gradient and the Hessian matrix. + +Make link with linear regression and the Hessian matrix from linear regression. + + + !split ===== Solving using Newton-Raphson's method =====