From 187aa16fc0f8303a0e2f12c2376a4b44f09255b1 Mon Sep 17 00:00:00 2001 From: mhjensen Date: Fri, 21 Sep 2018 06:17:07 +0200 Subject: [PATCH] typos in gradient methods --- doc/pub/Splines/html/._Splines-bs000.html | 50 +- doc/pub/Splines/html/._Splines-bs001.html | 48 +- doc/pub/Splines/html/._Splines-bs002.html | 54 +- doc/pub/Splines/html/._Splines-bs003.html | 48 +- doc/pub/Splines/html/._Splines-bs004.html | 50 +- doc/pub/Splines/html/._Splines-bs005.html | 51 +- doc/pub/Splines/html/._Splines-bs006.html | 103 ++- doc/pub/Splines/html/._Splines-bs007.html | 68 +- doc/pub/Splines/html/._Splines-bs008.html | 94 +-- doc/pub/Splines/html/._Splines-bs009.html | 75 ++- doc/pub/Splines/html/._Splines-bs010.html | 100 ++- doc/pub/Splines/html/._Splines-bs011.html | 94 +-- doc/pub/Splines/html/._Splines-bs012.html | 86 ++- doc/pub/Splines/html/._Splines-bs013.html | 80 +-- doc/pub/Splines/html/._Splines-bs014.html | 78 +-- doc/pub/Splines/html/._Splines-bs015.html | 86 ++- doc/pub/Splines/html/._Splines-bs016.html | 101 ++- doc/pub/Splines/html/._Splines-bs017.html | 93 ++- doc/pub/Splines/html/._Splines-bs018.html | 48 +- doc/pub/Splines/html/._Splines-bs019.html | 50 +- doc/pub/Splines/html/._Splines-bs020.html | 50 +- doc/pub/Splines/html/._Splines-bs021.html | 60 +- doc/pub/Splines/html/._Splines-bs022.html | 48 +- doc/pub/Splines/html/._Splines-bs023.html | 52 +- doc/pub/Splines/html/._Splines-bs024.html | 48 +- doc/pub/Splines/html/._Splines-bs025.html | 50 +- doc/pub/Splines/html/._Splines-bs026.html | 48 +- doc/pub/Splines/html/._Splines-bs027.html | 48 +- doc/pub/Splines/html/._Splines-bs028.html | 48 +- doc/pub/Splines/html/._Splines-bs029.html | 48 +- doc/pub/Splines/html/._Splines-bs030.html | 48 +- doc/pub/Splines/html/._Splines-bs031.html | 48 +- doc/pub/Splines/html/._Splines-bs032.html | 48 +- doc/pub/Splines/html/._Splines-bs033.html | 48 +- doc/pub/Splines/html/._Splines-bs034.html | 48 +- doc/pub/Splines/html/._Splines-bs035.html | 48 +- doc/pub/Splines/html/._Splines-bs036.html | 48 +- doc/pub/Splines/html/Splines-bs.html | 50 +- doc/pub/Splines/html/Splines-reveal.html | 593 +++++++++--------- doc/pub/Splines/html/Splines-solarized.html | 581 +++++++++-------- doc/pub/Splines/html/Splines.html | 581 +++++++++-------- doc/pub/Splines/ipynb/Splines.ipynb | 239 ++++--- .../Splines/ipynb/ipynb-Splines-src.tar.gz | Bin 210 -> 210 bytes doc/pub/Splines/pdf/Splines-minted.pdf | Bin 364005 -> 363461 bytes doc/src/Splines/Splines.do.txt | 448 ++++++------- 45 files changed, 2390 insertions(+), 2395 deletions(-) diff --git a/doc/pub/Splines/html/._Splines-bs000.html b/doc/pub/Splines/html/._Splines-bs000.html index 09897dcec..e3f254239 100644 --- a/doc/pub/Splines/html/._Splines-bs000.html +++ b/doc/pub/Splines/html/._Splines-bs000.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({
  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -193,7 +193,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 20, 2018

    +

    Sep 21, 2018


    diff --git a/doc/pub/Splines/html/._Splines-bs001.html b/doc/pub/Splines/html/._Splines-bs001.html index ad05fabe2..fa2e9e8ee 100644 --- a/doc/pub/Splines/html/._Splines-bs001.html +++ b/doc/pub/Splines/html/._Splines-bs001.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • diff --git a/doc/pub/Splines/html/._Splines-bs002.html b/doc/pub/Splines/html/._Splines-bs002.html index 2c97de8df..034502d3a 100644 --- a/doc/pub/Splines/html/._Splines-bs002.html +++ b/doc/pub/Splines/html/._Splines-bs002.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({
  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -185,11 +185,13 @@ direction of the negative gradient \( -\nabla F(\mathbf{x}) \).

    It can be shown that if $$ -\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ \gamma_k > 0 +\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), $$ +with \( \gamma_k > 0 \). +

    -for \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq +For \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq F(\mathbf{x}_k) \). This means that for a sufficiently small \( \gamma_k \) we are always moving towards smaller function values, i.e a minimum. diff --git a/doc/pub/Splines/html/._Splines-bs003.html b/doc/pub/Splines/html/._Splines-bs003.html index 90b01b5c8..1537aadc8 100644 --- a/doc/pub/Splines/html/._Splines-bs003.html +++ b/doc/pub/Splines/html/._Splines-bs003.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • diff --git a/doc/pub/Splines/html/._Splines-bs004.html b/doc/pub/Splines/html/._Splines-bs004.html index 9484ecc0c..47ec80c33 100644 --- a/doc/pub/Splines/html/._Splines-bs004.html +++ b/doc/pub/Splines/html/._Splines-bs004.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({
  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -177,7 +177,7 @@ MathJax.Hub.Config({

    The ideal

    -Ideally the sequence \( \{ \mathbf{x}_k \}_{k=0} \) converges to a global +Ideally the sequence \( \{\mathbf{x}_k \}_{k=0} \) converges to a global minimum of the function \( F \). In general we do not know if we are in a global or local minimum. In the special case when \( F \) is a convex function, all local minima are also global minima, so in this case diff --git a/doc/pub/Splines/html/._Splines-bs005.html b/doc/pub/Splines/html/._Splines-bs005.html index f9a3208bd..a4e58bab5 100644 --- a/doc/pub/Splines/html/._Splines-bs005.html +++ b/doc/pub/Splines/html/._Splines-bs005.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -177,7 +177,8 @@ MathJax.Hub.Config({

    The sensitiveness of the gradient descent

    -GD is sensitive to the choice of learning rate \( \gamma_k \). This is due +The gradient descent method +is sensitive to the choice of learning rate \( \gamma_k \). This is due to the fact that we are only guaranteed that \( F(\mathbf{x}_{k+1}) \leq F(\mathbf{x}_k) \) for sufficiently small \( \gamma_k \). The problem is to determine an optimal learning rate. If the learning rate is chosen too diff --git a/doc/pub/Splines/html/._Splines-bs006.html b/doc/pub/Splines/html/._Splines-bs006.html index 306b99cdb..1ba9ffe66 100644 --- a/doc/pub/Splines/html/._Splines-bs006.html +++ b/doc/pub/Splines/html/._Splines-bs006.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -172,54 +172,25 @@ MathJax.Hub.Config({

     

     

     

    - + -

    Gradient Descent Example

    +

    Convex functions

    -We revisit now our simple linear regression example with a linear polynomial. +Ideally we want our cost/loss function to be convex(concave). +

    +First we give the definition of a convex set: A set \( C \) in +\( \mathbb{R}^n \) is said to be convex if, for all \( x \) and \( y \) in \( C \) and +all \( t \in (0,1) \) , the point \( (1 − t)x + ty \) also belongs to +C. Geometrically this means that every point on the line segment +connecting \( x \) and \( y \) is in \( C \) as discussed below. - -

    # Importing various packages
-from random import random, seed
-import numpy as np
-import matplotlib.pyplot as plt
-from mpl_toolkits.mplot3d import Axes3D
-from matplotlib import cm
-from matplotlib.ticker import LinearLocator, FormatStrFormatter
-import sys
+
    
+The convex subsets of \( \mathbb{R} \) are the intervals of
+\( \mathbb{R} \). Examples of convex sets of \( \mathbb{R}^2 \) are the
+regular polygons (triangles, rectangles, pentagons, etc...).
 
-x = 2*np.random.rand(100,1)
-y = 4+3*x+np.random.randn(100,1)
-
-xb = np.c_[np.ones((100,1)), x]
-beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
-print(beta_linreg)
-beta = np.random.randn(2,1)
-
-eta = 0.1
-Niterations = 1000
-m = 100
-
-for iter in range(Niterations):
-    gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y)
-    beta -= eta*gradients
-
-print(beta)
-xnew = np.array([[0],[2]])
-xbnew = np.c_[np.ones((2,1)), xnew]
-ypredict = xbnew.dot(beta)
-ypredict2 = xbnew.dot(beta_linreg)
-plt.plot(xnew, ypredict, "r-")
-plt.plot(xnew, ypredict2, "b-")
-plt.plot(x, y ,'ro')
-plt.axis([0,2.0,0, 15.0])
-plt.xlabel(r'$x$')
-plt.ylabel(r'$y$')
-plt.title(r'Gradient descent example')
-plt.show()
-

    diff --git a/doc/pub/Splines/html/._Splines-bs007.html b/doc/pub/Splines/html/._Splines-bs007.html index 6b2852c4c..57c477e77 100644 --- a/doc/pub/Splines/html/._Splines-bs007.html +++ b/doc/pub/Splines/html/._Splines-bs007.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -174,27 +174,11 @@ MathJax.Hub.Config({ -

    And a corresponding example using scikit-learn

    +

    Convex function

    +Convex function: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below. - -

    # Importing various packages
-from random import random, seed
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.linear_model import SGDRegressor
-
-x = 2*np.random.rand(100,1)
-y = 4+3*x+np.random.randn(100,1)
-
-xb = np.c_[np.ones((100,1)), x]
-beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
-print(beta_linreg)
-sgdreg = SGDRegressor(n_iter = 50, penalty=None, eta0=0.1)
-sgdreg.fit(x,y.ravel())
-print(sgdreg.intercept_, sgdreg.coef_)
-

    diff --git a/doc/pub/Splines/html/._Splines-bs008.html b/doc/pub/Splines/html/._Splines-bs008.html index 86e9ac845..f0565b9ad 100644 --- a/doc/pub/Splines/html/._Splines-bs008.html +++ b/doc/pub/Splines/html/._Splines-bs008.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -172,24 +172,48 @@ MathJax.Hub.Config({

     

     

     

    - + -

    Convex functions

    +

    Conditions on convex functions

    -Ideally we want our cost/loss function to be convex(concave). +In the following we state first and second-order conditions which +ensures convexity of a function \( f \). We write \( D_f \) to denote the +domain of \( f \), i.e the subset of \( R^n \) where \( f \) is defined. For more +details and proofs we refer to: S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press.

    -First we give the definition of a convex set: A set \( C \) in -\( \mathbb{R}^n \) is said to be convex if, for all \( x \) and \( y \) in \( C \) and -all \( t \in (0,1) \) , the point \( (1 − t)x + ty \) also belongs to -C. Geometrically this means that every point on the line segment -connecting \( x \) and \( y \) is in \( C \) as discussed below. +

    +
    +

    +Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for +all \( x \) in the domain of \( f \)). Then \( f \) is convex if and only if \( D_f \) +is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds +for all \( x,y \in D_f \). This condition means that for a convex function +the first order Taylor expansion (right hand side above) at any point +a global under estimator of the function. To convince yourself you can +make a drawing of \( f(x) = x^2+1 \) and draw the tangent line to \( f(x) \) and +note that it is always below the graph. +

    +
    +

    -The convex subsets of \( \mathbb{R} \) are the intervals of -\( \mathbb{R} \). Examples of convex sets of \( \mathbb{R}^2 \) are the -regular polygons (triangles, rectangles, pentagons, etc...). +

    +
    +

    +Assume that \( f \) is twice +differentiable, i.e the Hessian matrix exists at each point in +\( D_f \). Then \( f \) is convex if and only if \( D_f \) is a convex set and its +Hessian is positive semi-definite for all \( x\in D_f \). For a +single-variable function this reduces to \( f''(x) \geq 0 \). Geometrically this means that \( f \) has nonnegative curvature +everywhere. +

    +
    + + +

    +This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.

    diff --git a/doc/pub/Splines/html/._Splines-bs009.html b/doc/pub/Splines/html/._Splines-bs009.html index 7ae79d754..15e45e732 100644 --- a/doc/pub/Splines/html/._Splines-bs009.html +++ b/doc/pub/Splines/html/._Splines-bs009.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -174,10 +174,33 @@ MathJax.Hub.Config({ -

    Convex function

    +

    More on convex functions

    -Convex function: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below. +The next result is of great importance to us and the reason why we are +going on about convex functions. In machine learning we frequently +have to minimize a loss/cost function in order to find the best +parameters for the model we are considering. + +

    +Ideally we want the +global minimum (for high-dimensional models it is hard to know +if we have local or global minimum). However, if the cost/loss function +is convex the following result provides invaluable information: + +

    +

    +
    +

    +Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \) +is minimal, where \( f \) is convex and differentiable. Then, any point +\( x^* \) that satisfies \( \nabla f(x^*) = 0 \) is a global minimum. +

    +
    + + +

    +This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.

    diff --git a/doc/pub/Splines/html/._Splines-bs010.html b/doc/pub/Splines/html/._Splines-bs010.html index c1bb98aa7..91bd8be2f 100644 --- a/doc/pub/Splines/html/._Splines-bs010.html +++ b/doc/pub/Splines/html/._Splines-bs010.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -174,47 +174,29 @@ MathJax.Hub.Config({ -

    Conditions on convex functions

    +

    Some simple problems

    -

    -In the following we state first and second-order conditions which -ensures convexity of a function \( f \). We write \( D_f \) to denote the -domain of \( f \), i.e the subset of \( R^n \) where \( f \) is defined. For more -details and proofs we refer to: S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press. +

      +
    1. Show that \( f(x)=x^2 \) is convex for \( x \in \mathbb{R} \) using the definition of convexity. Hint: If you re-write the definition, \( f \) is convex if the following holds for all \( x,y \in D_f \) and any \( \lambda \in [0,1] \) $\lambda f(x)+(1-\lambda)f(y)-f(\lambda x + (1-\lambda) y ) \geq 0$.
      2. +
    2. Using the second order condition show that the following functions are convex on the specified domain.
      3. -

      -

      -
      -

      -Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for -all \( x \) in the domain of \( f \)). Then \( f \) is convex if and only if \( D_f \) -is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds -for all \( x,y \in D_f \). This condition means that for a convex function -the first order Taylor expansion (right hand side above) at any point -a global under estimator of the function. To convince yourself you can -make a drawing of f(x) = x^2+1 and draw the tangent line to \( f(x) \) and -note that it is always below the graph. -

      -
      + +
    3. Let \( f(x) = x^2 \) and \( g(x) = e^x \). Show that \( f(g(x)) \) and \( g(f(x)) \) is convex for \( x \in \mathbb{R} \). Also show that if \( f(x) \) is any convex function than \( h(x) = e^{f(x)} \) is convex.
      4. +
    4. A norm is any function that satisfy the following properties
      5. -

      -

      -
      -

      -Assume that \( f \) is twice -differentiable, i.e the Hessian matrix exists at each point in -\( D_f \). Then \( f \) is convex if and only if \( D_f \) is a convex set and its -Hessian is positive semi-definite for all \( x\in D_f \). For a -single-variable function this reduces to \( f''(x) \geq -0 \). Geometrically this means that \( f \) has nonnegative curvature -everywhere. -

      -
      + +
    -

    -This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition. +Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).

    diff --git a/doc/pub/Splines/html/._Splines-bs011.html b/doc/pub/Splines/html/._Splines-bs011.html index 8a0925577..fc3bba3df 100644 --- a/doc/pub/Splines/html/._Splines-bs011.html +++ b/doc/pub/Splines/html/._Splines-bs011.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -172,35 +172,37 @@ MathJax.Hub.Config({

     

     

     

    - + -

    More on convex functions

    +

    Revisiting our first homework

    -The next result is of great importance to us and the reason why we are -going on about convex functions. In machine learning we frequently -have to minimize a loss/cost function in order to find the best -parameters for the model we are considering. +We will use linear regression as a case study for the gradient descent +methods. Linear regression is a great test case for the gradient +descent methods discussed in the lectures since it has several +desirable properties such as: -

    -Ideally we want the -global minimum (for high-dimensional models it is hard to know -if we have local or global minimum). However, if the cost/loss function -is convex the following result provides invaluable information: +

      +
    1. An analytical solution (recall homework set 1).
      2. +
    2. The gradient can be computed analytically.
      3. +
    3. The cost function is convex which guarantees that gradient descent converges for small enough learning rates
      4. +
    -

    -

    -
    -

    -Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \) -is minimal, where \( f \) is convex and differentiable. Then, any point -\( x^* \) that satisfies \( \nabla f(x^*) = 0 \) is a global minimum. -

    -
    +We revisit the example from homework set 1 where we had +$$ +y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100 +$$ +with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). +The linear regression model is given by +$$ +h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x, +$$ -

    -This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum. +such that +$$ +\hat{y}_i = \beta_0 + \beta_1 x_i. +$$

    diff --git a/doc/pub/Splines/html/._Splines-bs012.html b/doc/pub/Splines/html/._Splines-bs012.html index 29a039b11..27137f59b 100644 --- a/doc/pub/Splines/html/._Splines-bs012.html +++ b/doc/pub/Splines/html/._Splines-bs012.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -172,31 +172,29 @@ MathJax.Hub.Config({

     

     

     

    - + -

    Some simple problems

    +

    Gradient descent example

    -
      -
    1. Show that \( f(x)=x^2 \) is convex for \( x \in \mathbb{R} \) using the definition of convexity. Hint: If you re-write the definition, \( f \) is convex if the following holds for all \( x,y \in D_f \) and any \( \lambda \in [0,1] \) $\lambda f(x) + (1-\lambda)f(y) - f(\lambda x + (1-\lambda) y ) \geq 0. $
      2. -
    2. Using the second order condition show that the following functions are convex on the specified domain.
      3. +

      +Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \) -

      +

      +It is convenient to write \( \mathbf{\hat{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by +$$ +X \equiv \begin{bmatrix} +1 & x_1 \\ +\vdots & \vdots \\ +1 & x_{100} & \\ +\end{bmatrix}. +$$ -

    3. Let \( f(x) = x^2 \) and \( g(x) = e^x \). Show that \( f(g(x)) \) and \( g(f(x)) \) is convex for \( x \in \mathbb{R} \). Also show that if \( f(x) \) is any convex function than \( h(x) = e^{f(x)} \) is convex.
      4. -
    4. A norm is any function that satisfy the following properties
      5. +The loss function is given by +$$ +C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 +$$ - - -
    - -Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this). +and we want to find \( \beta \) such that \( C(\beta) \) is minimized.

    diff --git a/doc/pub/Splines/html/._Splines-bs013.html b/doc/pub/Splines/html/._Splines-bs013.html index 2a7103a97..372867cf3 100644 --- a/doc/pub/Splines/html/._Splines-bs013.html +++ b/doc/pub/Splines/html/._Splines-bs013.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -172,37 +172,19 @@ MathJax.Hub.Config({

     

     

     

    - + -

    Revisiting our first homework

    +

    The derivative of the cost/loss function

    -We will use linear regression as a case study for the gradient descent -methods. Linear regression is a great test case for the gradient -descent methods discussed in the lectures since it has several -desirable properties such as: - -

      -
    1. An analytical solution (recall homework set 1).
      2. -
    2. The gradient can be computed analytically.
      3. -
    3. The cost function is convex which guarantees that gradient descent converges for small enough learning rates
      4. -
    - -We revisit the example from homework set 1 where we had +Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as $$ -y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100 +\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ +\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ +\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), $$ -with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). -The linear regression model is given by -$$ -h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x, -$$ - -such that -$$ -\hat{y}_i = \beta_0 + \beta_1 x_i. -$$ +where \( X \) is the design matrix defined above.

    diff --git a/doc/pub/Splines/html/._Splines-bs014.html b/doc/pub/Splines/html/._Splines-bs014.html index 596f81cd4..9374c1ab0 100644 --- a/doc/pub/Splines/html/._Splines-bs014.html +++ b/doc/pub/Splines/html/._Splines-bs014.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -172,32 +172,18 @@ MathJax.Hub.Config({

     

     

     

    - + -

    Gradient descent example

    - -

    -Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \) - -

    -t is convenient to write \( \mathbf{\hat{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by +

    The Hessian matrix

    +The Hessian matrix of \( C(\beta) \) is given by $$ -\begin{equation} -X \equiv \begin{bmatrix} -1 & x_1 \\ -\vdots & \vdots \\ -1 & x_{100} & \\ -\end{bmatrix}. -\tag{1} -\end{equation} +\hat{H} \equiv \begin{bmatrix} +\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ +\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\ +\end{bmatrix} = 2X^T X. $$ -The loss function is given by -$$ -C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 -$$ - -and we want to find \( \beta \) such that \( C(\beta) \) is minimized. +This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite.

    diff --git a/doc/pub/Splines/html/._Splines-bs015.html b/doc/pub/Splines/html/._Splines-bs015.html index 1778f7a66..36d1bdfcf 100644 --- a/doc/pub/Splines/html/._Splines-bs015.html +++ b/doc/pub/Splines/html/._Splines-bs015.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -174,18 +174,44 @@ MathJax.Hub.Config({ -

    The derivative of the cost/loss function

    +

    Simple program

    -Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as +We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to $$ -\nabla_\beta C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ -\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ -\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), +\beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots $$ -where \( X \) is the design matrix defined above. +

    +We can use the expression we computed for the gradient and let use a +\( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating +when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). +

    +And finally we can compare our solution for \( \beta \) with the analytic result given by +\( \beta= (X^TX)^{-1} X^T \mathbf{y} \). +

    + + +

    import numpy as np
+
+"""
+The following setup is just a suggestion, feel free to write it the way you like.
+"""
+
+#Setup problem described in the exercise
+N  = 100 #Nr of datapoints
+M  = 2 #Nr of features
+x  = np.random.rand(N) #Uniformly generated x-values in [0,1]
+y  = 5*x**2 + 0.1*np.random.randn(N)
+X  = np.c_[np.ones(N),x] #Construct design matrix
+
+#Compute beta according to normal equations to compare with GD solution
+Xt_X_inv = np.linalg.inv(np.dot(X.T,X))
+Xt_y     = np.dot(X.transpose(),y)
+beta_NE = np.dot(Xt_X_inv,Xt_y)
+print(beta_NE)
+

    diff --git a/doc/pub/Splines/html/._Splines-bs016.html b/doc/pub/Splines/html/._Splines-bs016.html index d4a0a6cb0..8f6c8a419 100644 --- a/doc/pub/Splines/html/._Splines-bs016.html +++ b/doc/pub/Splines/html/._Splines-bs016.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -174,17 +174,52 @@ MathJax.Hub.Config({ -

    The Hessian matrix

    -The Hessian matrix of \( C(\beta) \) is given by -$$ -\hat{H} \equiv \begin{bmatrix} -\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ -\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\ -\end{bmatrix} = 2X^T X. -$$ +

    Gradient Descent Example

    -This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite. +

    +Another simple example is here +

    + +

    # Importing various packages
+from random import random, seed
+import numpy as np
+import matplotlib.pyplot as plt
+from mpl_toolkits.mplot3d import Axes3D
+from matplotlib import cm
+from matplotlib.ticker import LinearLocator, FormatStrFormatter
+import sys
+
+x = 2*np.random.rand(100,1)
+y = 4+3*x+np.random.randn(100,1)
+
+xb = np.c_[np.ones((100,1)), x]
+beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
+print(beta_linreg)
+beta = np.random.randn(2,1)
+
+eta = 0.1
+Niterations = 1000
+m = 100
+
+for iter in range(Niterations):
+    gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y)
+    beta -= eta*gradients
+
+print(beta)
+xnew = np.array([[0],[2]])
+xbnew = np.c_[np.ones((2,1)), xnew]
+ypredict = xbnew.dot(beta)
+ypredict2 = xbnew.dot(beta_linreg)
+plt.plot(xnew, ypredict, "r-")
+plt.plot(xnew, ypredict2, "b-")
+plt.plot(x, y ,'ro')
+plt.axis([0,2.0,0, 15.0])
+plt.xlabel(r'$x$')
+plt.ylabel(r'$y$')
+plt.title(r'Gradient descent example')
+plt.show()
+

    diff --git a/doc/pub/Splines/html/._Splines-bs017.html b/doc/pub/Splines/html/._Splines-bs017.html index d02aaae45..b6e338b6b 100644 --- a/doc/pub/Splines/html/._Splines-bs017.html +++ b/doc/pub/Splines/html/._Splines-bs017.html @@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source ('More on Steepest descent', 2, None, '___sec2'), ('The ideal', 2, None, '___sec3'), ('The sensitiveness of the gradient descent', 2, None, '___sec4'), - ('Gradient Descent Example', 2, None, '___sec5'), + ('Convex functions', 2, None, '___sec5'), + ('Convex function', 2, None, '___sec6'), + ('Conditions on convex functions', 2, None, '___sec7'), + ('More on convex functions', 2, None, '___sec8'), + ('Some simple problems', 2, None, '___sec9'), + ('Revisiting our first homework', 2, None, '___sec10'), + ('Gradient descent example', 2, None, '___sec11'), + ('The derivative of the cost/loss function', 2, None, '___sec12'), + ('The Hessian matrix', 2, None, '___sec13'), + ('Simple program', 2, None, '___sec14'), + ('Gradient Descent Example', 2, None, '___sec15'), ('And a corresponding example using _scikit-learn_', 2, None, - '___sec6'), - ('Convex functions', 2, None, '___sec7'), - ('Convex function', 2, None, '___sec8'), - ('Conditions on convex functions', 2, None, '___sec9'), - ('More on convex functions', 2, None, '___sec10'), - ('Some simple problems', 2, None, '___sec11'), - ('Revisiting our first homework', 2, None, '___sec12'), - ('Gradient descent example', 2, None, '___sec13'), - ('The derivative of the cost/loss function', 2, None, '___sec14'), - ('The Hessian matrix', 2, None, '___sec15'), - ('Simple program', 2, None, '___sec16'), + '___sec16'), ('Gradient descent and Ridge', 2, None, '___sec17'), ('Stochastic Gradient Descent', 2, None, '___sec18'), ('Computation of gradients', 2, None, '___sec19'), @@ -128,18 +128,18 @@ MathJax.Hub.Config({

  • More on Steepest descent
  • The ideal
  • The sensitiveness of the gradient descent
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
  • Gradient descent and Ridge
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -174,43 +174,26 @@ MathJax.Hub.Config({ -

    Simple program

    +

    And a corresponding example using scikit-learn

    -

    -We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to -$$ -\beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots -$$ - -

    -We can use the expression we computed for the gradient and let use a -\( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating -when \( ||\nabla_\beta C(\beta_k) || < \epsilon = 10^{-8} \). - -

    -And finally we can compare our solution for \( \beta \) with the analytic result given by -\( \beta= (X^TX)^{-1} X^T \mathbf{y} \).

    -

    import numpy as np
+
    # Importing various packages
+from random import random, seed
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.linear_model import SGDRegressor
 
-"""
-The following setup is just a suggestion, feel free to write it the way you like.
-"""
+x = 2*np.random.rand(100,1)
+y = 4+3*x+np.random.randn(100,1)
 
-#Setup problem described in the exercise
-N  = 100 #Nr of datapoints
-M  = 2 #Nr of features
-x  = np.random.rand(N) #Uniformly generated x-values in [0,1]
-y  = 5*x**2 + 0.1*np.random.randn(N)
-X  = np.c_[np.ones(N),x] #Construct design matrix
-
-#Compute beta according to normal equations to compare with GD solution
-Xt_X_inv = np.linalg.inv(np.dot(X.T,X))
-Xt_y     = np.dot(X.transpose(),y)
-beta_NE = np.dot(Xt_X_inv,Xt_y)
-print(beta_NE)
+xb = np.c_[np.ones((100,1)), x]
+beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
+print(beta_linreg)
+sgdreg = SGDRegressor(n_iter = 50, penalty=None, eta0=0.1)
+sgdreg.fit(x,y.ravel())
+print(sgdreg.intercept_, sgdreg.coef_)
 
    
 
    
 
    
diff --git a/doc/pub/Splines/html/._Splines-bs018.html b/doc/pub/Splines/html/._Splines-bs018.html
index 878874987..507b4a0f3 100644
--- a/doc/pub/Splines/html/._Splines-bs018.html
+++ b/doc/pub/Splines/html/._Splines-bs018.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/._Splines-bs019.html b/doc/pub/Splines/html/._Splines-bs019.html
index 464a234f4..dda06818b 100644
--- a/doc/pub/Splines/html/._Splines-bs019.html
+++ b/doc/pub/Splines/html/._Splines-bs019.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
@@ -183,7 +183,7 @@ the shortcomings of the Gradient descent method discussed above.
 
    
 The underlying idea of SGD comes from the observation that the cost
 function, which we want to minimize, can almost always be written as a
-sum over \( n \) datapoints \( \{\mathbf{x}_i\}_{i=1}^n \),
+sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
 $$
 C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
 \mathbf{\beta}). 
diff --git a/doc/pub/Splines/html/._Splines-bs020.html b/doc/pub/Splines/html/._Splines-bs020.html
index 953411294..2b13ebb75 100644
--- a/doc/pub/Splines/html/._Splines-bs020.html
+++ b/doc/pub/Splines/html/._Splines-bs020.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
@@ -187,7 +187,7 @@ $$
 
    
 Stochasticity/randomness is introduced by only taking the
 gradient on a subset of the data called minibatches.  If there are \( n \)
-datapoints and the size of each minibatch is \( M \), there will be \( n/M \)
+data points and the size of each minibatch is \( M \), there will be \( n/M \)
 minibatches. We denote these minibatches by \( B_k \) where
 \( k=1,\cdots,n/M \).
 
diff --git a/doc/pub/Splines/html/._Splines-bs021.html b/doc/pub/Splines/html/._Splines-bs021.html
index 062ea2901..6525dab00 100644
--- a/doc/pub/Splines/html/._Splines-bs021.html
+++ b/doc/pub/Splines/html/._Splines-bs021.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
@@ -175,21 +175,21 @@ MathJax.Hub.Config({
 
 
 
    SGD example 
    
-As an example, suppose we have \( 10 \) datapoints \( ( \mathbf{x}_1,
-\cdots, \mathbf{x}_{10} ) \) and we choose to have \( M=5 \) minibathces,
-then each minibatch contains two datapoints. In particular we have
+As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \) 
+and we choose to have \( M=5 \) minibathces,
+then each minibatch contains two data points. In particular we have
 \( B_1 = (\mathbf{x}_1,\mathbf{x}_2), \cdots, B_5 =
 (\mathbf{x}_9,\mathbf{x}_{10}) \). Note that if you choose \( M=1 \) you
-have only a single batch with all datapoints and on the other extreme,
+have only a single batch with all data points and on the other extreme,
 you may choose \( M=n \) resulting in a minibatch for each datapoint, i.e
 \( B_k = \mathbf{x}_k \).
 
 
    
 The idea is now to approximate the gradient by replacing the sum over
-all datapoints with a sum over the datapoints in one the minibatches
+all data points with a sum over the data points in one the minibatches
 picked at random in each gradient descent step 
 $$
-\nabla_\beta
+\nabla_{\beta}
 C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
 \mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
 c_i(\mathbf{x}_i, \mathbf{\beta}).
diff --git a/doc/pub/Splines/html/._Splines-bs022.html b/doc/pub/Splines/html/._Splines-bs022.html
index 6dc924616..c7901fea0 100644
--- a/doc/pub/Splines/html/._Splines-bs022.html
+++ b/doc/pub/Splines/html/._Splines-bs022.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/._Splines-bs023.html b/doc/pub/Splines/html/._Splines-bs023.html
index 9c98ea73d..ccc0d0fab 100644
--- a/doc/pub/Splines/html/._Splines-bs023.html
+++ b/doc/pub/Splines/html/._Splines-bs023.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
@@ -199,8 +199,8 @@ Taking the gradient only on a subset of the data has two important
 benefits. First, it introduces randomness which decreases the chance
 that our opmization scheme gets stuck in a local minima. Second, if
 the size of the minibatches are small relative to the number of
-datapoints (\( M < n \)), the computation of the gradient is much
-cheaper since we sum over the datapoints in the k-th minibatch and not
+datapoints (\( M <  n \)), the computation of the gradient is much
+cheaper since we sum over the datapoints in the \( k-th \) minibatch and not
 all \( n \) datapoints.
 
 
    
diff --git a/doc/pub/Splines/html/._Splines-bs024.html b/doc/pub/Splines/html/._Splines-bs024.html
index 1653551d2..084a9a24f 100644
--- a/doc/pub/Splines/html/._Splines-bs024.html
+++ b/doc/pub/Splines/html/._Splines-bs024.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/._Splines-bs025.html b/doc/pub/Splines/html/._Splines-bs025.html
index 49c8b196a..2568a5049 100644
--- a/doc/pub/Splines/html/._Splines-bs025.html
+++ b/doc/pub/Splines/html/._Splines-bs025.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
@@ -182,7 +182,7 @@ number of epochs in such a way that it becomes very small after a
 reasonable time such that we do not move at all.
 
 
    
-As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \).
+As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \).
 
 
    
 In this way we can fix the number of epochs, compute \( \beta \) and
diff --git a/doc/pub/Splines/html/._Splines-bs026.html b/doc/pub/Splines/html/._Splines-bs026.html
index a3bc73c10..069443c14 100644
--- a/doc/pub/Splines/html/._Splines-bs026.html
+++ b/doc/pub/Splines/html/._Splines-bs026.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/._Splines-bs027.html b/doc/pub/Splines/html/._Splines-bs027.html
index aa6377c6d..faf2c87d5 100644
--- a/doc/pub/Splines/html/._Splines-bs027.html
+++ b/doc/pub/Splines/html/._Splines-bs027.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/._Splines-bs028.html b/doc/pub/Splines/html/._Splines-bs028.html
index 07849d947..b6e89d1b7 100644
--- a/doc/pub/Splines/html/._Splines-bs028.html
+++ b/doc/pub/Splines/html/._Splines-bs028.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/._Splines-bs029.html b/doc/pub/Splines/html/._Splines-bs029.html
index acbc6471d..7ea3b65e7 100644
--- a/doc/pub/Splines/html/._Splines-bs029.html
+++ b/doc/pub/Splines/html/._Splines-bs029.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/._Splines-bs030.html b/doc/pub/Splines/html/._Splines-bs030.html
index 2ffe81cff..03448fbfc 100644
--- a/doc/pub/Splines/html/._Splines-bs030.html
+++ b/doc/pub/Splines/html/._Splines-bs030.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/._Splines-bs031.html b/doc/pub/Splines/html/._Splines-bs031.html
index ad740e7f8..0237a96c7 100644
--- a/doc/pub/Splines/html/._Splines-bs031.html
+++ b/doc/pub/Splines/html/._Splines-bs031.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/._Splines-bs032.html b/doc/pub/Splines/html/._Splines-bs032.html
index a4ecf522e..0c101dce5 100644
--- a/doc/pub/Splines/html/._Splines-bs032.html
+++ b/doc/pub/Splines/html/._Splines-bs032.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/._Splines-bs033.html b/doc/pub/Splines/html/._Splines-bs033.html
index 5ca815020..ea7e4344b 100644
--- a/doc/pub/Splines/html/._Splines-bs033.html
+++ b/doc/pub/Splines/html/._Splines-bs033.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/._Splines-bs034.html b/doc/pub/Splines/html/._Splines-bs034.html
index 33e8bbc77..f96293d33 100644
--- a/doc/pub/Splines/html/._Splines-bs034.html
+++ b/doc/pub/Splines/html/._Splines-bs034.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/._Splines-bs035.html b/doc/pub/Splines/html/._Splines-bs035.html
index a783f024b..63617ce8f 100644
--- a/doc/pub/Splines/html/._Splines-bs035.html
+++ b/doc/pub/Splines/html/._Splines-bs035.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/._Splines-bs036.html b/doc/pub/Splines/html/._Splines-bs036.html
index 4070ced3b..93e47979b 100644
--- a/doc/pub/Splines/html/._Splines-bs036.html
+++ b/doc/pub/Splines/html/._Splines-bs036.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
diff --git a/doc/pub/Splines/html/Splines-bs.html b/doc/pub/Splines/html/Splines-bs.html
index 09897dcec..e3f254239 100644
--- a/doc/pub/Splines/html/Splines-bs.html
+++ b/doc/pub/Splines/html/Splines-bs.html
@@ -49,21 +49,21 @@ Automatically generated HTML file from DocOnce source
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -128,18 +128,18 @@ MathJax.Hub.Config({
       
  • More on Steepest descent
    • 
       
  • The ideal
    • 
       
  • The sensitiveness of the gradient descent
    • 
-      
  • Gradient Descent Example
    • 
-      
  • And a corresponding example using scikit-learn
    • 
-      
  • Convex functions
    • 
-      
  • Convex function
    • 
-      
  • Conditions on convex functions
    • 
-      
  • More on convex functions
    • 
-      
  • Some simple problems
    • 
-      
  • Revisiting our first homework
    • 
-      
  • Gradient descent example
    • 
-      
  • The derivative of the cost/loss function
    • 
-      
  • The Hessian matrix
    • 
-      
  • Simple program
    • 
+      
  • Convex functions
    • 
+      
  • Convex function
    • 
+      
  • Conditions on convex functions
    • 
+      
  • More on convex functions
    • 
+      
  • Some simple problems
    • 
+      
  • Revisiting our first homework
    • 
+      
  • Gradient descent example
    • 
+      
  • The derivative of the cost/loss function
    • 
+      
  • The Hessian matrix
    • 
+      
  • Simple program
    • 
+      
  • Gradient Descent Example
    • 
+      
  • And a corresponding example using scikit-learn
    • 
       
  • Gradient descent and Ridge
    • 
       
  • Stochastic Gradient Descent
    • 
       
  • Computation of gradients
    • 
@@ -193,7 +193,7 @@ MathJax.Hub.Config({
 
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    
 
    
 
    
-
    Sep 20, 2018
     
+
    Sep 21, 2018
     
 
    
 
    
 
diff --git a/doc/pub/Splines/html/Splines-reveal.html b/doc/pub/Splines/html/Splines-reveal.html
index f78ef0d25..53afa7c72 100644
--- a/doc/pub/Splines/html/Splines-reveal.html
+++ b/doc/pub/Splines/html/Splines-reveal.html
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
 
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    
 
    
 
     
    
-
    Sep 20, 2018
     
+
    Sep 21, 2018
     
 
    
 
    
 
@@ -186,12 +186,14 @@ direction of the negative gradient \( -\nabla F(\mathbf{x}) \).
 It can be shown that if 
 
     
    
 $$
-\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ \gamma_k > 0
+\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k),
 $$
 
     
    
 
+with \( \gamma_k > 0 \).
+
 
    
-for \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq
+For \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq
 F(\mathbf{x}_k) \). This means that for a sufficiently small \( \gamma_k \)
 we are always moving towards smaller function values, i.e a minimum.
 
@@ -222,7 +224,7 @@ the learning rate within the context of Machine Learning.
 
    The ideal 
    
 
 
    
-Ideally the sequence \( \{ \mathbf{x}_k \}_{k=0} \) converges to a global
+Ideally the sequence \( \{\mathbf{x}_k \}_{k=0} \) converges to a global
 minimum of the function \( F \). In general we do not know if we are in a
 global or local minimum. In the special case when \( F \) is a convex
 function, all local minima are also global minima, so in this case
@@ -248,7 +250,8 @@ Note that the gradient is a function of \( \mathbf{x} =
 
    The sensitiveness of the gradient descent 
    
 
 
    
-GD is sensitive to the choice of learning rate \( \gamma_k \). This is due
+The gradient descent method 
+is sensitive to the choice of learning rate \( \gamma_k \). This is due
 to the fact that we are only guaranteed that \( F(\mathbf{x}_{k+1}) \leq
 F(\mathbf{x}_k) \) for sufficiently small \( \gamma_k \). The problem is to
 determine an optimal learning rate. If the learning rate is chosen too
@@ -263,10 +266,284 @@ randomness. One such method is that of Stochastic Gradient Descent
 
 
 
    
-
    Gradient Descent Example 
    
+
    Convex functions 
    
 
 
    
-We revisit now our simple linear regression example with a linear polynomial.
+Ideally we want our cost/loss function to be convex(concave).
+
+
    
+First we give the definition of a convex set: A set \( C \) in
+\( \mathbb{R}^n \) is said to be convex if, for all \( x \) and \( y \) in \( C \) and
+all \( t \in (0,1) \) , the point \( (1 − t)x + ty \) also belongs to
+C. Geometrically this means that every point on the line segment
+connecting \( x \) and \( y \) is in \( C \) as discussed below.
+
+
    
+The convex subsets of \( \mathbb{R} \) are the intervals of
+\( \mathbb{R} \). Examples of convex sets of \( \mathbb{R}^2 \) are the
+regular polygons (triangles, rectangles, pentagons, etc...).
+
    
+
+
+
    
+
    Convex function 
    
+
+
    
+Convex function: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if 
     
    
+$$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$
+
     
     for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below.
+
    
+
+
+
    
+
    Conditions on convex functions 
    
+
+
    
+In the following we state first and second-order conditions which
+ensures convexity of a function \( f \). We write \( D_f \) to denote the
+domain of \( f \), i.e the subset of \( R^n \) where \( f \) is defined. For more
+details and proofs we refer to: S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press.
+
+
    
+
    
+First order condition.
+
    
+Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for
+all \( x \) in the domain of \( f \)). Then \( f \) is convex if and only if \( D_f \)
+is a convex set and 
     
    
+$$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$
+
     
     holds
+for all \( x,y \in D_f \). This condition means that for a convex function
+the first order Taylor expansion (right hand side above) at any point
+a global under estimator of the function. To convince yourself you can
+make a drawing of \( f(x) = x^2+1 \) and draw the tangent line to \( f(x) \) and
+note that it is always below the graph.
+
    
+
+
    
+
    
+Second order condition.
+
    
+Assume that \( f \) is twice
+differentiable, i.e the Hessian matrix exists at each point in
+\( D_f \). Then \( f \) is convex if and only if \( D_f \) is a convex set and its
+Hessian is positive semi-definite for all \( x\in D_f \). For a
+single-variable function this reduces to \( f''(x) \geq 0 \). Geometrically this means that \( f \) has nonnegative curvature
+everywhere.
+
    
+
+
    
+This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.
+
    
+
+
+
    
+
    More on convex functions 
    
+
+
    
+The next result is of great importance to us and the reason why we are
+going on about convex functions. In machine learning we frequently
+have to minimize a loss/cost function in order to find the best
+parameters for the model we are considering.
+
+
    
+Ideally we want the
+global minimum (for high-dimensional models it is hard to know
+if we have local or global minimum). However, if the cost/loss function
+is convex the following result provides invaluable information:
+
+
    
+
    
+Any minimum is global for convex functions.
+
    
+Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \)
+is minimal, where \( f \) is convex and differentiable. Then, any point
+\( x^* \) that satisfies \( \nabla f(x^*) = 0 \) is a global minimum.
+
    
+
+
    
+This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.
+
    
+
+
+
    
+
    Some simple problems 
    
+
+
      
+
    1.  Show that \( f(x)=x^2 \) is convex for \( x \in \mathbb{R} \) using the definition of convexity. Hint: If you re-write the definition, \( f \) is convex if the following holds for all \( x,y \in D_f \) and any \( \lambda \in [0,1] \) $\lambda f(x)+(1-\lambda)f(y)-f(\lambda x + (1-\lambda) y ) \geq 0$.
      2. 
+
    2.  Using the second order condition show that the following functions are convex on the specified domain.
      3. 
+
+
        
+ 
      •  \( f(x) = e^x \) is convex for \( x \in \mathbb{R} \).
        • 
+ 
      •  \( g(x) = -\ln(x) \) is convex for \( x \in (0,\infty) \).
        • 
+
      
+
    3.  Let \( f(x) = x^2 \) and \( g(x) = e^x \). Show that \( f(g(x)) \) and \( g(f(x)) \) is convex for \( x \in \mathbb{R} \). Also show that if \( f(x) \) is any convex function than \( h(x) = e^{f(x)} \) is convex.
      4. 
+
    4.  A norm is any function that satisfy the following properties
      5. 
+
+
        
+ 
      •  \( f(\alpha x) = |\alpha| f(x) \) for all \( \alpha \in \mathbb{R} \).
        • 
+ 
      •  \( f(x+y) \leq f(x) + f(y) \)
        • 
+ 
      •  \( f(x) \leq 0 \) for all \( x \in \mathbb{R}^n \) with equality if and only if \( x = 0 \)
        • 
+
      
+
      
+
    
+
    
+
+Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).
+
    
+
+
+
    
+
    Revisiting our first homework 
    
+
+
    
+We will use linear regression as a case study for the gradient descent
+methods. Linear regression is a great test case for the gradient
+descent methods discussed in the lectures since it has several
+desirable properties such as:
+
+
      
+
    1.  An analytical solution (recall homework set 1).
      2. 
+
    2.  The gradient can be computed analytically.
      3. 
+
    3.  The cost function is convex which guarantees that gradient descent converges for small enough learning rates
      4. 
+
    
+
    
+
+We revisit the example from homework set 1 where we had 
+
     
    
+$$
+y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
+$$
+
     
    
+
+with \( x_i \in [0,1]  \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). 
+The linear regression model is given by 
+
     
    
+$$
+h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x,
+$$
+
     
    
+
+such that 
+
     
    
+$$
+\hat{y}_i = \beta_0 + \beta_1 x_i.
+$$
+
     
    
+
    
+
+
+
    
+
    Gradient descent example 
    
+
+
    
+Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
+
+
    
+It is convenient to write \( \mathbf{\hat{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2}  \) is the design matrix given by
+
     
    
+$$
+X \equiv \begin{bmatrix}
+1 & x_1  \\
+\vdots & \vdots  \\
+1 & x_{100} &  \\
+\end{bmatrix}.
+$$
+
     
    
+
+The loss function is given by 
+
     
    
+$$
+C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 
+$$
+
     
    
+
+and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
+
    
+
+
+
    
+
    The derivative of the cost/loss function 
    
+
+
    
+Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show  that the gradient can be written as
+
     
    
+$$
+\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
+\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
+\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), 
+$$
+
     
    
+
+where \( X \) is the design matrix defined above.
+
    
+
+
+
    
+
    The Hessian matrix 
    
+The Hessian matrix of \( C(\beta) \) is given by 
+
     
    
+$$
+\hat{H} \equiv \begin{bmatrix}
+\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1}  \\
+\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &  \\
+\end{bmatrix} = 2X^T X.
+$$
+
     
    
+
+This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite.
+
    
+
+
+
    
+
    Simple program 
    
+
+
    
+We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to 
+
     
    
+$$
+\beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots 
+$$
+
     
    
+
+
    
+We can use the expression we computed for the gradient and let use a
+\( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating
+when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \).
+
+
    
+And finally we can compare our solution for \( \beta \) with the analytic result given by 
+\( \beta= (X^TX)^{-1} X^T \mathbf{y} \).
+
    
+
+
+
    import numpy as np
+
+"""
+The following setup is just a suggestion, feel free to write it the way you like.
+"""
+
+#Setup problem described in the exercise
+N  = 100 #Nr of datapoints
+M  = 2 #Nr of features
+x  = np.random.rand(N) #Uniformly generated x-values in [0,1]
+y  = 5*x**2 + 0.1*np.random.randn(N)
+X  = np.c_[np.ones(N),x] #Construct design matrix
+
+#Compute beta according to normal equations to compare with GD solution
+Xt_X_inv = np.linalg.inv(np.dot(X.T,X))
+Xt_y     = np.dot(X.transpose(),y)
+beta_NE = np.dot(Xt_X_inv,Xt_y)
+print(beta_NE)
+
    
+
    
+
+
+
    
+
    Gradient Descent Example 
    
+
+
    
+Another simple example is here
 
    
 
 
@@ -313,7 +590,7 @@ plt.show()
 
 
 
    
-
    And a corresponding example using scikit-learn 
    
+
    And a corresponding example using scikit-learn 
    
 
 
    
 
@@ -337,284 +614,6 @@ sgdreg.fit(x,y.ravel())
 
    
 
 
-
    
-
    Convex functions 
    
-
-
    
-Ideally we want our cost/loss function to be convex(concave).
-
-
    
-First we give the definition of a convex set: A set \( C \) in
-\( \mathbb{R}^n \) is said to be convex if, for all \( x \) and \( y \) in \( C \) and
-all \( t \in (0,1) \) , the point \( (1 − t)x + ty \) also belongs to
-C. Geometrically this means that every point on the line segment
-connecting \( x \) and \( y \) is in \( C \) as discussed below.
-
-
    
-The convex subsets of \( \mathbb{R} \) are the intervals of
-\( \mathbb{R} \). Examples of convex sets of \( \mathbb{R}^2 \) are the
-regular polygons (triangles, rectangles, pentagons, etc...).
-
    
-
-
-
    
-
    Convex function 
    
-
-
    
-Convex function: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if 
     
    
-$$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$
-
     
     for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below.
-
    
-
-
-
    
-
    Conditions on convex functions 
    
-
-
    
-In the following we state first and second-order conditions which
-ensures convexity of a function \( f \). We write \( D_f \) to denote the
-domain of \( f \), i.e the subset of \( R^n \) where \( f \) is defined. For more
-details and proofs we refer to: S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press.
-
-
    
-
    
-First order condition.
-
    
-Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for
-all \( x \) in the domain of \( f \)). Then \( f \) is convex if and only if \( D_f \)
-is a convex set and 
     
    
-$$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$
-
     
     holds
-for all \( x,y \in D_f \). This condition means that for a convex function
-the first order Taylor expansion (right hand side above) at any point
-a global under estimator of the function. To convince yourself you can
-make a drawing of f(x) = x^2+1 and draw the tangent line to \( f(x) \) and
-note that it is always below the graph.
-
    
-
-
    
-
    
-Second order condition.
-
    
-Assume that \( f \) is twice
-differentiable, i.e the Hessian matrix exists at each point in
-\( D_f \). Then \( f \) is convex if and only if \( D_f \) is a convex set and its
-Hessian is positive semi-definite for all \( x\in D_f \). For a
-single-variable function this reduces to \( f''(x) \geq
-0 \). Geometrically this means that \( f \) has nonnegative curvature
-everywhere.
-
    
-
-
    
-This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.
-
    
-
-
-
    
-
    More on convex functions 
    
-
-
    
-The next result is of great importance to us and the reason why we are
-going on about convex functions. In machine learning we frequently
-have to minimize a loss/cost function in order to find the best
-parameters for the model we are considering.
-
-
    
-Ideally we want the
-global minimum (for high-dimensional models it is hard to know
-if we have local or global minimum). However, if the cost/loss function
-is convex the following result provides invaluable information:
-
-
    
-
    
-Any minimum is global for convex functions.
-
    
-Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \)
-is minimal, where \( f \) is convex and differentiable. Then, any point
-\( x^* \) that satisfies \( \nabla f(x^*) = 0 \) is a global minimum.
-
    
-
-
    
-This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.
-
    
-
-
-
    
-
    Some simple problems 
    
-
-
      
-
    1.  Show that \( f(x)=x^2 \) is convex for \( x \in \mathbb{R} \) using the definition of convexity. Hint: If you re-write the definition, \( f \) is convex if the following holds for all \( x,y \in D_f \) and any \( \lambda \in [0,1]  \) $\lambda f(x) + (1-\lambda)f(y) - f(\lambda x + (1-\lambda) y ) \geq 0. $
      2. 
-
    2.  Using the second order condition show that the following functions are convex on the specified domain.
      3. 
-
-
        
- 
      •  \( f(x) = e^x \) is convex for \( x \in \mathbb{R} \).
        • 
- 
      •  \( g(x) = -\ln(x) \) is convex for \( x \in (0,\infty) \).
        • 
-
      
-
    3.  Let \( f(x) = x^2 \) and \( g(x) = e^x \). Show that \( f(g(x)) \) and \( g(f(x)) \) is convex for \( x \in \mathbb{R} \). Also show that if \( f(x) \) is any convex function than \( h(x) = e^{f(x)} \) is convex.
      4. 
-
    4.  A norm is any function that satisfy the following properties
      5. 
-
-
        
- 
      •  \( f(\alpha x) = |\alpha| f(x) \) for all \( \alpha \in \mathbb{R} \).
        • 
- 
      •  \( f(x+y) \leq f(x) + f(y) \)
        • 
- 
      •  \( f(x) \leq 0 \) for all \( x \in \mathbb{R}^n \) with equality if and only if \( x = 0 \)
        • 
-
      
-
      
-
    
-
    
-
-Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).
-
    
-
-
-
    
-
    Revisiting our first homework 
    
-
-
    
-We will use linear regression as a case study for the gradient descent
-methods. Linear regression is a great test case for the gradient
-descent methods discussed in the lectures since it has several
-desirable properties such as:
-
-
      
-
    1.  An analytical solution (recall homework set 1).
      2. 
-
    2.  The gradient can be computed analytically.
      3. 
-
    3.  The cost function is convex which guarantees that gradient descent converges for small enough learning rates
      4. 
-
    
-
    
-
-We revisit the example from homework set 1 where we had 
-
     
    
-$$
-y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
-$$
-
     
    
-
-with \( x_i \in [0,1]  \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). 
-The linear regression model is given by 
-
     
    
-$$
-h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x,
-$$
-
     
    
-
-such that 
-
     
    
-$$
-\hat{y}_i = \beta_0 + \beta_1 x_i.
-$$
-
     
    
-
    
-
-
-
    
-
    Gradient descent example 
    
-
-
    
-Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
-
-
    
-t is convenient to write \( \mathbf{\hat{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2}  \) is the design matrix given by
-
     
    
-$$
-\begin{equation}
-X \equiv \begin{bmatrix}
-1 & x_1  \\
-\vdots & \vdots  \\
-1 & x_{100} &  \\
-\end{bmatrix}.
-\tag{1}
-\end{equation}
-$$
-
     
    
-
-The loss function is given by 
-
     
    
-$$
-C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 
-$$
-
     
    
-
-and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
-
    
-
-
-
    
-
    The derivative of the cost/loss function 
    
-
-
    
-Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show  that the gradient can be written as
-
     
    
-$$
-\nabla_\beta C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
-\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
-\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), 
-$$
-
     
    
-
-where \( X \) is the design matrix defined above.
-
    
-
-
-
    
-
    The Hessian matrix 
    
-The Hessian matrix of \( C(\beta) \) is given by 
-
     
    
-$$
-\hat{H} \equiv \begin{bmatrix}
-\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1}  \\
-\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &  \\
-\end{bmatrix} = 2X^T X.
-$$
-
     
    
-
-This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite.
-
    
-
-
-
    
-
    Simple program 
    
-
-
    
-We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to 
-
     
    
-$$
-\beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots 
-$$
-
     
    
-
-
    
-We can use the expression we computed for the gradient and let use a
-\( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating
-when \( ||\nabla_\beta C(\beta_k) || < \epsilon = 10^{-8} \).
-
-
    
-And finally we can compare our solution for \( \beta \) with the analytic result given by 
-\( \beta= (X^TX)^{-1} X^T \mathbf{y} \).
-
    
-
-
-
    import numpy as np
-
-"""
-The following setup is just a suggestion, feel free to write it the way you like.
-"""
-
-#Setup problem described in the exercise
-N  = 100 #Nr of datapoints
-M  = 2 #Nr of features
-x  = np.random.rand(N) #Uniformly generated x-values in [0,1]
-y  = 5*x**2 + 0.1*np.random.randn(N)
-X  = np.c_[np.ones(N),x] #Construct design matrix
-
-#Compute beta according to normal equations to compare with GD solution
-Xt_X_inv = np.linalg.inv(np.dot(X.T,X))
-Xt_y     = np.dot(X.transpose(),y)
-beta_NE = np.dot(Xt_X_inv,Xt_y)
-print(beta_NE)
-
    
-
    
-
-
 
    
 
    Gradient descent and Ridge 
    
 
@@ -689,7 +688,7 @@ the shortcomings of the Gradient descent method discussed above.
 
    
 The underlying idea of SGD comes from the observation that the cost
 function, which we want to minimize, can almost always be written as a
-sum over \( n \) datapoints \( \{\mathbf{x}_i\}_{i=1}^n \),
+sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
 
     
    
 $$
 C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
@@ -715,7 +714,7 @@ $$
 
    
 Stochasticity/randomness is introduced by only taking the
 gradient on a subset of the data called minibatches.  If there are \( n \)
-datapoints and the size of each minibatch is \( M \), there will be \( n/M \)
+data points and the size of each minibatch is \( M \), there will be \( n/M \)
 minibatches. We denote these minibatches by \( B_k \) where
 \( k=1,\cdots,n/M \).
 
    
@@ -723,22 +722,22 @@ minibatches. We denote these minibatches by \( B_k \) where
 
 
    
 
    SGD example 
    
-As an example, suppose we have \( 10 \) datapoints \( ( \mathbf{x}_1,
-\cdots, \mathbf{x}_{10} ) \) and we choose to have \( M=5 \) minibathces,
-then each minibatch contains two datapoints. In particular we have
+As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \) 
+and we choose to have \( M=5 \) minibathces,
+then each minibatch contains two data points. In particular we have
 \( B_1 = (\mathbf{x}_1,\mathbf{x}_2), \cdots, B_5 =
 (\mathbf{x}_9,\mathbf{x}_{10}) \). Note that if you choose \( M=1 \) you
-have only a single batch with all datapoints and on the other extreme,
+have only a single batch with all data points and on the other extreme,
 you may choose \( M=n \) resulting in a minibatch for each datapoint, i.e
 \( B_k = \mathbf{x}_k \).
 
 
    
 The idea is now to approximate the gradient by replacing the sum over
-all datapoints with a sum over the datapoints in one the minibatches
+all data points with a sum over the data points in one the minibatches
 picked at random in each gradient descent step 
 
     
    
 $$
-\nabla_\beta
+\nabla_{\beta}
 C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
 \mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
 c_i(\mathbf{x}_i, \mathbf{\beta}).
@@ -794,8 +793,8 @@ Taking the gradient only on a subset of the data has two important
 benefits. First, it introduces randomness which decreases the chance
 that our opmization scheme gets stuck in a local minima. Second, if
 the size of the minibatches are small relative to the number of
-datapoints (\( M < n \)), the computation of the gradient is much
-cheaper since we sum over the datapoints in the k-th minibatch and not
+datapoints (\( M <  n \)), the computation of the gradient is much
+cheaper since we sum over the datapoints in the \( k-th \) minibatch and not
 all \( n \) datapoints.
 
    
 
@@ -826,7 +825,7 @@ number of epochs in such a way that it becomes very small after a
 reasonable time such that we do not move at all.
 
 
    
-As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function 
     
    
+As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function 
     
    
 $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$
 
     
     goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \).
 
diff --git a/doc/pub/Splines/html/Splines-solarized.html b/doc/pub/Splines/html/Splines-solarized.html
index fc0bfdc82..752bc03f2 100644
--- a/doc/pub/Splines/html/Splines-solarized.html
+++ b/doc/pub/Splines/html/Splines-solarized.html
@@ -69,21 +69,21 @@ div { text-align: justify; text-justify: inter-word; }
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -147,7 +147,7 @@ MathJax.Hub.Config({
 
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    
 
    
 
    
-
    Sep 20, 2018
     
+
    Sep 21, 2018
     
 
    
 
    
 









    
@@ -178,11 +178,13 @@ direction of the negative gradient \( -\nabla F(\mathbf{x}) \).
 
    
 It can be shown that if 
 $$
-\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ \gamma_k > 0
+\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k),
 $$
 
+with \( \gamma_k > 0 \).
+
 
    
-for \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq
+For \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq
 F(\mathbf{x}_k) \). This means that for a sufficiently small \( \gamma_k \)
 we are always moving towards smaller function values, i.e a minimum.
 
@@ -211,7 +213,7 @@ the learning rate within the context of Machine Learning.
 
    The ideal 
    
 
 
    
-Ideally the sequence \( \{ \mathbf{x}_k \}_{k=0} \) converges to a global
+Ideally the sequence \( \{\mathbf{x}_k \}_{k=0} \) converges to a global
 minimum of the function \( F \). In general we do not know if we are in a
 global or local minimum. In the special case when \( F \) is a convex
 function, all local minima are also global minima, so in this case
@@ -237,7 +239,8 @@ Note that the gradient is a function of \( \mathbf{x} =
 
    The sensitiveness of the gradient descent 
    
 
 
    
-GD is sensitive to the choice of learning rate \( \gamma_k \). This is due
+The gradient descent method 
+is sensitive to the choice of learning rate \( \gamma_k \). This is due
 to the fact that we are only guaranteed that \( F(\mathbf{x}_{k+1}) \leq
 F(\mathbf{x}_k) \) for sufficiently small \( \gamma_k \). The problem is to
 determine an optimal learning rate. If the learning rate is chosen too
@@ -250,12 +253,267 @@ randomness. One such method is that of Stochastic Gradient Descent
 (SGD), see below.
 
 
    
-









    
+
 
-
    Gradient Descent Example 
    
+
    Convex functions 
    
 
 
    
-We revisit now our simple linear regression example with a linear polynomial.
+Ideally we want our cost/loss function to be convex(concave).
+
+
    
+First we give the definition of a convex set: A set \( C \) in
+\( \mathbb{R}^n \) is said to be convex if, for all \( x \) and \( y \) in \( C \) and
+all \( t \in (0,1) \) , the point \( (1 − t)x + ty \) also belongs to
+C. Geometrically this means that every point on the line segment
+connecting \( x \) and \( y \) is in \( C \) as discussed below.
+
+
    
+The convex subsets of \( \mathbb{R} \) are the intervals of
+\( \mathbb{R} \). Examples of convex sets of \( \mathbb{R}^2 \) are the
+regular polygons (triangles, rectangles, pentagons, etc...).
+
+
    
+









    
+
+
    Convex function 
    
+
+
    
+Convex function: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below.
+
+
    
+









    
+
+
    Conditions on convex functions 
    
+
+
    
+In the following we state first and second-order conditions which
+ensures convexity of a function \( f \). We write \( D_f \) to denote the
+domain of \( f \), i.e the subset of \( R^n \) where \( f \) is defined. For more
+details and proofs we refer to: S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press.
+
+
    
+
    
+First order condition.
+
    
+Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for
+all \( x \) in the domain of \( f \)). Then \( f \) is convex if and only if \( D_f \)
+is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds
+for all \( x,y \in D_f \). This condition means that for a convex function
+the first order Taylor expansion (right hand side above) at any point
+a global under estimator of the function. To convince yourself you can
+make a drawing of \( f(x) = x^2+1 \) and draw the tangent line to \( f(x) \) and
+note that it is always below the graph.
+
    
+
+
+
    
+
    
+Second order condition.
+
    
+Assume that \( f \) is twice
+differentiable, i.e the Hessian matrix exists at each point in
+\( D_f \). Then \( f \) is convex if and only if \( D_f \) is a convex set and its
+Hessian is positive semi-definite for all \( x\in D_f \). For a
+single-variable function this reduces to \( f''(x) \geq 0 \). Geometrically this means that \( f \) has nonnegative curvature
+everywhere.
+
    
+
+
+
    
+This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.
+
+
    
+









    
+
+
    More on convex functions 
    
+
+
    
+The next result is of great importance to us and the reason why we are
+going on about convex functions. In machine learning we frequently
+have to minimize a loss/cost function in order to find the best
+parameters for the model we are considering.
+
+
    
+Ideally we want the
+global minimum (for high-dimensional models it is hard to know
+if we have local or global minimum). However, if the cost/loss function
+is convex the following result provides invaluable information:
+
+
    
+
    
+Any minimum is global for convex functions.
+
    
+Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \)
+is minimal, where \( f \) is convex and differentiable. Then, any point
+\( x^* \) that satisfies \( \nabla f(x^*) = 0 \) is a global minimum.
+
    
+
+
+
    
+This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.
+
+
    
+









    
+
+
    Some simple problems 
    
+
+
      
+
    1.  Show that \( f(x)=x^2 \) is convex for \( x \in \mathbb{R} \) using the definition of convexity. Hint: If you re-write the definition, \( f \) is convex if the following holds for all \( x,y \in D_f \) and any \( \lambda \in [0,1] \) $\lambda f(x)+(1-\lambda)f(y)-f(\lambda x + (1-\lambda) y ) \geq 0$.
      2. 
+
    2.  Using the second order condition show that the following functions are convex on the specified domain.
      3. 
+
+
        
+ 
      •  \( f(x) = e^x \) is convex for \( x \in \mathbb{R} \).
        • 
+ 
      •  \( g(x) = -\ln(x) \) is convex for \( x \in (0,\infty) \).
        • 
+
      
+
+
    3.  Let \( f(x) = x^2 \) and \( g(x) = e^x \). Show that \( f(g(x)) \) and \( g(f(x)) \) is convex for \( x \in \mathbb{R} \). Also show that if \( f(x) \) is any convex function than \( h(x) = e^{f(x)} \) is convex.
      4. 
+
    4.  A norm is any function that satisfy the following properties
      5. 
+
+
        
+ 
      •  \( f(\alpha x) = |\alpha| f(x) \) for all \( \alpha \in \mathbb{R} \).
        • 
+ 
      •  \( f(x+y) \leq f(x) + f(y) \)
        • 
+ 
      •  \( f(x) \leq 0 \) for all \( x \in \mathbb{R}^n \) with equality if and only if \( x = 0 \)
        • 
+
      
+
+
    
+
+Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).
+
+
    
+
+
+
    Revisiting our first homework 
    
+
+
    
+We will use linear regression as a case study for the gradient descent
+methods. Linear regression is a great test case for the gradient
+descent methods discussed in the lectures since it has several
+desirable properties such as:
+
+
      
+
    1.  An analytical solution (recall homework set 1).
      2. 
+
    2.  The gradient can be computed analytically.
      3. 
+
    3.  The cost function is convex which guarantees that gradient descent converges for small enough learning rates
      4. 
+
    
+
+We revisit the example from homework set 1 where we had 
+$$
+y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
+$$
+
+with \( x_i \in [0,1]  \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). 
+The linear regression model is given by 
+$$
+h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x,
+$$
+
+such that 
+$$
+\hat{y}_i = \beta_0 + \beta_1 x_i.
+$$
+
+
    
+
+
+
    Gradient descent example 
    
+
+
    
+Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
+
+
    
+It is convenient to write \( \mathbf{\hat{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2}  \) is the design matrix given by
+$$
+X \equiv \begin{bmatrix}
+1 & x_1  \\
+\vdots & \vdots  \\
+1 & x_{100} &  \\
+\end{bmatrix}.
+$$
+
+The loss function is given by 
+$$
+C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 
+$$
+
+and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
+
+
    
+









    
+
+
    The derivative of the cost/loss function 
    
+
+
    
+Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show  that the gradient can be written as
+$$
+\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
+\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
+\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), 
+$$
+
+where \( X \) is the design matrix defined above.
+
+
    
+









    
+
+
    The Hessian matrix 
    
+The Hessian matrix of \( C(\beta) \) is given by 
+$$
+\hat{H} \equiv \begin{bmatrix}
+\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1}  \\
+\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &  \\
+\end{bmatrix} = 2X^T X.
+$$
+
+This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite.
+
+
    
+









    
+
+
    Simple program 
    
+
+
    
+We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to 
+$$
+\beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots 
+$$
+
+
    
+We can use the expression we computed for the gradient and let use a
+\( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating
+when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \).
+
+
    
+And finally we can compare our solution for \( \beta \) with the analytic result given by 
+\( \beta= (X^TX)^{-1} X^T \mathbf{y} \).
+
    
+
+
+
    import numpy as np
+
+"""
+The following setup is just a suggestion, feel free to write it the way you like.
+"""
+
+#Setup problem described in the exercise
+N  = 100 #Nr of datapoints
+M  = 2 #Nr of features
+x  = np.random.rand(N) #Uniformly generated x-values in [0,1]
+y  = 5*x**2 + 0.1*np.random.randn(N)
+X  = np.c_[np.ones(N),x] #Construct design matrix
+
+#Compute beta according to normal equations to compare with GD solution
+Xt_X_inv = np.linalg.inv(np.dot(X.T,X))
+Xt_y     = np.dot(X.transpose(),y)
+beta_NE = np.dot(Xt_X_inv,Xt_y)
+print(beta_NE)
+
    
+
    
+









    
+
+
    Gradient Descent Example 
    
+
+
    
+Another simple example is here
 
    
 
 
@@ -301,7 +559,7 @@ plt.show()
 
    
 









    
 
-
    And a corresponding example using scikit-learn 
    
+
    And a corresponding example using scikit-learn 
    
 
 
    
 
@@ -325,265 +583,6 @@ sgdreg.fit(x,y.ravel())
 
    
 
 
-
    Convex functions 
    
-
-
    
-Ideally we want our cost/loss function to be convex(concave).
-
-
    
-First we give the definition of a convex set: A set \( C \) in
-\( \mathbb{R}^n \) is said to be convex if, for all \( x \) and \( y \) in \( C \) and
-all \( t \in (0,1) \) , the point \( (1 − t)x + ty \) also belongs to
-C. Geometrically this means that every point on the line segment
-connecting \( x \) and \( y \) is in \( C \) as discussed below.
-
-
    
-The convex subsets of \( \mathbb{R} \) are the intervals of
-\( \mathbb{R} \). Examples of convex sets of \( \mathbb{R}^2 \) are the
-regular polygons (triangles, rectangles, pentagons, etc...).
-
-
    
-









    
-
-
    Convex function 
    
-
-
    
-Convex function: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below.
-
-
    
-









    
-
-
    Conditions on convex functions 
    
-
-
    
-In the following we state first and second-order conditions which
-ensures convexity of a function \( f \). We write \( D_f \) to denote the
-domain of \( f \), i.e the subset of \( R^n \) where \( f \) is defined. For more
-details and proofs we refer to: S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press.
-
-
    
-
    
-First order condition.
-
    
-Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for
-all \( x \) in the domain of \( f \)). Then \( f \) is convex if and only if \( D_f \)
-is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds
-for all \( x,y \in D_f \). This condition means that for a convex function
-the first order Taylor expansion (right hand side above) at any point
-a global under estimator of the function. To convince yourself you can
-make a drawing of f(x) = x^2+1 and draw the tangent line to \( f(x) \) and
-note that it is always below the graph.
-
    
-
-
-
    
-
    
-Second order condition.
-
    
-Assume that \( f \) is twice
-differentiable, i.e the Hessian matrix exists at each point in
-\( D_f \). Then \( f \) is convex if and only if \( D_f \) is a convex set and its
-Hessian is positive semi-definite for all \( x\in D_f \). For a
-single-variable function this reduces to \( f''(x) \geq
-0 \). Geometrically this means that \( f \) has nonnegative curvature
-everywhere.
-
    
-
-
-
    
-This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.
-
-
    
-









    
-
-
    More on convex functions 
    
-
-
    
-The next result is of great importance to us and the reason why we are
-going on about convex functions. In machine learning we frequently
-have to minimize a loss/cost function in order to find the best
-parameters for the model we are considering.
-
-
    
-Ideally we want the
-global minimum (for high-dimensional models it is hard to know
-if we have local or global minimum). However, if the cost/loss function
-is convex the following result provides invaluable information:
-
-
    
-
    
-Any minimum is global for convex functions.
-
    
-Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \)
-is minimal, where \( f \) is convex and differentiable. Then, any point
-\( x^* \) that satisfies \( \nabla f(x^*) = 0 \) is a global minimum.
-
    
-
-
-
    
-This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.
-
-
    
-









    
-
-
    Some simple problems 
    
-
-
      
-
    1.  Show that \( f(x)=x^2 \) is convex for \( x \in \mathbb{R} \) using the definition of convexity. Hint: If you re-write the definition, \( f \) is convex if the following holds for all \( x,y \in D_f \) and any \( \lambda \in [0,1]  \) $\lambda f(x) + (1-\lambda)f(y) - f(\lambda x + (1-\lambda) y ) \geq 0. $
      2. 
-
    2.  Using the second order condition show that the following functions are convex on the specified domain.
      3. 
-
-
        
- 
      •  \( f(x) = e^x \) is convex for \( x \in \mathbb{R} \).
        • 
- 
      •  \( g(x) = -\ln(x) \) is convex for \( x \in (0,\infty) \).
        • 
-
      
-
-
    3.  Let \( f(x) = x^2 \) and \( g(x) = e^x \). Show that \( f(g(x)) \) and \( g(f(x)) \) is convex for \( x \in \mathbb{R} \). Also show that if \( f(x) \) is any convex function than \( h(x) = e^{f(x)} \) is convex.
      4. 
-
    4.  A norm is any function that satisfy the following properties
      5. 
-
-
        
- 
      •  \( f(\alpha x) = |\alpha| f(x) \) for all \( \alpha \in \mathbb{R} \).
        • 
- 
      •  \( f(x+y) \leq f(x) + f(y) \)
        • 
- 
      •  \( f(x) \leq 0 \) for all \( x \in \mathbb{R}^n \) with equality if and only if \( x = 0 \)
        • 
-
      
-
-
    
-
-Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).
-
-
    
-
-
-
    Revisiting our first homework 
    
-
-
    
-We will use linear regression as a case study for the gradient descent
-methods. Linear regression is a great test case for the gradient
-descent methods discussed in the lectures since it has several
-desirable properties such as:
-
-
      
-
    1.  An analytical solution (recall homework set 1).
      2. 
-
    2.  The gradient can be computed analytically.
      3. 
-
    3.  The cost function is convex which guarantees that gradient descent converges for small enough learning rates
      4. 
-
    
-
-We revisit the example from homework set 1 where we had 
-$$
-y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
-$$
-
-with \( x_i \in [0,1]  \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). 
-The linear regression model is given by 
-$$
-h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x,
-$$
-
-such that 
-$$
-\hat{y}_i = \beta_0 + \beta_1 x_i.
-$$
-
-
    
-
-
-
    Gradient descent example 
    
-
-
    
-Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
-
-
    
-t is convenient to write \( \mathbf{\hat{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2}  \) is the design matrix given by
-$$
-\begin{equation}
-X \equiv \begin{bmatrix}
-1 & x_1  \\
-\vdots & \vdots  \\
-1 & x_{100} &  \\
-\end{bmatrix}.
-\label{_auto1}
-\end{equation}
-$$
-
-The loss function is given by 
-$$
-C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 
-$$
-
-and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
-
-
    
-









    
-
-
    The derivative of the cost/loss function 
    
-
-
    
-Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show  that the gradient can be written as
-$$
-\nabla_\beta C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
-\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
-\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), 
-$$
-
-where \( X \) is the design matrix defined above.
-
-
    
-









    
-
-
    The Hessian matrix 
    
-The Hessian matrix of \( C(\beta) \) is given by 
-$$
-\hat{H} \equiv \begin{bmatrix}
-\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1}  \\
-\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &  \\
-\end{bmatrix} = 2X^T X.
-$$
-
-This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite.
-
-
    
-









    
-
-
    Simple program 
    
-
-
    
-We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to 
-$$
-\beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots 
-$$
-
-
    
-We can use the expression we computed for the gradient and let use a
-\( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating
-when \( ||\nabla_\beta C(\beta_k) || < \epsilon = 10^{-8} \).
-
-
    
-And finally we can compare our solution for \( \beta \) with the analytic result given by 
-\( \beta= (X^TX)^{-1} X^T \mathbf{y} \).
-
    
-
-
-
    import numpy as np
-
-"""
-The following setup is just a suggestion, feel free to write it the way you like.
-"""
-
-#Setup problem described in the exercise
-N  = 100 #Nr of datapoints
-M  = 2 #Nr of features
-x  = np.random.rand(N) #Uniformly generated x-values in [0,1]
-y  = 5*x**2 + 0.1*np.random.randn(N)
-X  = np.c_[np.ones(N),x] #Construct design matrix
-
-#Compute beta according to normal equations to compare with GD solution
-Xt_X_inv = np.linalg.inv(np.dot(X.T,X))
-Xt_y     = np.dot(X.transpose(),y)
-beta_NE = np.dot(Xt_X_inv,Xt_y)
-print(beta_NE)
-
    
-
    
-
-
 
    Gradient descent and Ridge 
    
 
 
    
@@ -650,7 +649,7 @@ the shortcomings of the Gradient descent method discussed above.
 
    
 The underlying idea of SGD comes from the observation that the cost
 function, which we want to minimize, can almost always be written as a
-sum over \( n \) datapoints \( \{\mathbf{x}_i\}_{i=1}^n \),
+sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
 $$
 C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
 \mathbf{\beta}). 
@@ -672,7 +671,7 @@ $$
 
    
 Stochasticity/randomness is introduced by only taking the
 gradient on a subset of the data called minibatches.  If there are \( n \)
-datapoints and the size of each minibatch is \( M \), there will be \( n/M \)
+data points and the size of each minibatch is \( M \), there will be \( n/M \)
 minibatches. We denote these minibatches by \( B_k \) where
 \( k=1,\cdots,n/M \).
 
@@ -680,21 +679,21 @@ minibatches. We denote these minibatches by \( B_k \) where
 









    
 
 
    SGD example 
    
-As an example, suppose we have \( 10 \) datapoints \( ( \mathbf{x}_1,
-\cdots, \mathbf{x}_{10} ) \) and we choose to have \( M=5 \) minibathces,
-then each minibatch contains two datapoints. In particular we have
+As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \) 
+and we choose to have \( M=5 \) minibathces,
+then each minibatch contains two data points. In particular we have
 \( B_1 = (\mathbf{x}_1,\mathbf{x}_2), \cdots, B_5 =
 (\mathbf{x}_9,\mathbf{x}_{10}) \). Note that if you choose \( M=1 \) you
-have only a single batch with all datapoints and on the other extreme,
+have only a single batch with all data points and on the other extreme,
 you may choose \( M=n \) resulting in a minibatch for each datapoint, i.e
 \( B_k = \mathbf{x}_k \).
 
 
    
 The idea is now to approximate the gradient by replacing the sum over
-all datapoints with a sum over the datapoints in one the minibatches
+all data points with a sum over the data points in one the minibatches
 picked at random in each gradient descent step 
 $$
-\nabla_\beta
+\nabla_{\beta}
 C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
 \mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
 c_i(\mathbf{x}_i, \mathbf{\beta}).
@@ -747,8 +746,8 @@ Taking the gradient only on a subset of the data has two important
 benefits. First, it introduces randomness which decreases the chance
 that our opmization scheme gets stuck in a local minima. Second, if
 the size of the minibatches are small relative to the number of
-datapoints (\( M < n \)), the computation of the gradient is much
-cheaper since we sum over the datapoints in the k-th minibatch and not
+datapoints (\( M <  n \)), the computation of the gradient is much
+cheaper since we sum over the datapoints in the \( k-th \) minibatch and not
 all \( n \) datapoints.
 
 
    
@@ -779,7 +778,7 @@ number of epochs in such a way that it becomes very small after a
 reasonable time such that we do not move at all.
 
 
    
-As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \).
+As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \).
 
 
    
 In this way we can fix the number of epochs, compute \( \beta \) and
diff --git a/doc/pub/Splines/html/Splines.html b/doc/pub/Splines/html/Splines.html
index b7f5a50f9..eec927569 100644
--- a/doc/pub/Splines/html/Splines.html
+++ b/doc/pub/Splines/html/Splines.html
@@ -74,21 +74,21 @@ div { text-align: justify; text-justify: inter-word; }
               ('More on Steepest descent', 2, None, '___sec2'),
               ('The ideal', 2, None, '___sec3'),
               ('The sensitiveness of the gradient descent', 2, None, '___sec4'),
-              ('Gradient Descent Example', 2, None, '___sec5'),
+              ('Convex functions', 2, None, '___sec5'),
+              ('Convex function', 2, None, '___sec6'),
+              ('Conditions on convex functions', 2, None, '___sec7'),
+              ('More on convex functions', 2, None, '___sec8'),
+              ('Some simple problems', 2, None, '___sec9'),
+              ('Revisiting our first homework', 2, None, '___sec10'),
+              ('Gradient descent example', 2, None, '___sec11'),
+              ('The derivative of the cost/loss function', 2, None, '___sec12'),
+              ('The Hessian matrix', 2, None, '___sec13'),
+              ('Simple program', 2, None, '___sec14'),
+              ('Gradient Descent Example', 2, None, '___sec15'),
               ('And a corresponding example using _scikit-learn_',
                2,
                None,
-               '___sec6'),
-              ('Convex functions', 2, None, '___sec7'),
-              ('Convex function', 2, None, '___sec8'),
-              ('Conditions on convex functions', 2, None, '___sec9'),
-              ('More on convex functions', 2, None, '___sec10'),
-              ('Some simple problems', 2, None, '___sec11'),
-              ('Revisiting our first homework', 2, None, '___sec12'),
-              ('Gradient descent example', 2, None, '___sec13'),
-              ('The derivative of the cost/loss function', 2, None, '___sec14'),
-              ('The Hessian matrix', 2, None, '___sec15'),
-              ('Simple program', 2, None, '___sec16'),
+               '___sec16'),
               ('Gradient descent and Ridge', 2, None, '___sec17'),
               ('Stochastic Gradient Descent', 2, None, '___sec18'),
               ('Computation of gradients', 2, None, '___sec19'),
@@ -152,7 +152,7 @@ MathJax.Hub.Config({
 
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    
 
    
 
    
-
    Sep 20, 2018
     
+
    Sep 21, 2018
     
 
    
 
    
 









    
@@ -183,11 +183,13 @@ direction of the negative gradient \( -\nabla F(\mathbf{x}) \).
 
    
 It can be shown that if 
 $$
-\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ \gamma_k > 0
+\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k),
 $$
 
+with \( \gamma_k > 0 \).
+
 
    
-for \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq
+For \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq
 F(\mathbf{x}_k) \). This means that for a sufficiently small \( \gamma_k \)
 we are always moving towards smaller function values, i.e a minimum.
 
@@ -216,7 +218,7 @@ the learning rate within the context of Machine Learning.
 
    The ideal 
    
 
 
    
-Ideally the sequence \( \{ \mathbf{x}_k \}_{k=0} \) converges to a global
+Ideally the sequence \( \{\mathbf{x}_k \}_{k=0} \) converges to a global
 minimum of the function \( F \). In general we do not know if we are in a
 global or local minimum. In the special case when \( F \) is a convex
 function, all local minima are also global minima, so in this case
@@ -242,7 +244,8 @@ Note that the gradient is a function of \( \mathbf{x} =
 
    The sensitiveness of the gradient descent 
    
 
 
    
-GD is sensitive to the choice of learning rate \( \gamma_k \). This is due
+The gradient descent method 
+is sensitive to the choice of learning rate \( \gamma_k \). This is due
 to the fact that we are only guaranteed that \( F(\mathbf{x}_{k+1}) \leq
 F(\mathbf{x}_k) \) for sufficiently small \( \gamma_k \). The problem is to
 determine an optimal learning rate. If the learning rate is chosen too
@@ -255,12 +258,267 @@ randomness. One such method is that of Stochastic Gradient Descent
 (SGD), see below.
 
 
    
-









    
+
 
-
    Gradient Descent Example 
    
+
    Convex functions 
    
 
 
    
-We revisit now our simple linear regression example with a linear polynomial.
+Ideally we want our cost/loss function to be convex(concave).
+
+
    
+First we give the definition of a convex set: A set \( C \) in
+\( \mathbb{R}^n \) is said to be convex if, for all \( x \) and \( y \) in \( C \) and
+all \( t \in (0,1) \) , the point \( (1 − t)x + ty \) also belongs to
+C. Geometrically this means that every point on the line segment
+connecting \( x \) and \( y \) is in \( C \) as discussed below.
+
+
    
+The convex subsets of \( \mathbb{R} \) are the intervals of
+\( \mathbb{R} \). Examples of convex sets of \( \mathbb{R}^2 \) are the
+regular polygons (triangles, rectangles, pentagons, etc...).
+
+
    
+









    
+
+
    Convex function 
    
+
+
    
+Convex function: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below.
+
+
    
+









    
+
+
    Conditions on convex functions 
    
+
+
    
+In the following we state first and second-order conditions which
+ensures convexity of a function \( f \). We write \( D_f \) to denote the
+domain of \( f \), i.e the subset of \( R^n \) where \( f \) is defined. For more
+details and proofs we refer to: S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press.
+
+
    
+
    
+First order condition.
+
    
+Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for
+all \( x \) in the domain of \( f \)). Then \( f \) is convex if and only if \( D_f \)
+is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds
+for all \( x,y \in D_f \). This condition means that for a convex function
+the first order Taylor expansion (right hand side above) at any point
+a global under estimator of the function. To convince yourself you can
+make a drawing of \( f(x) = x^2+1 \) and draw the tangent line to \( f(x) \) and
+note that it is always below the graph.
+
    
+
+
+
    
+
    
+Second order condition.
+
    
+Assume that \( f \) is twice
+differentiable, i.e the Hessian matrix exists at each point in
+\( D_f \). Then \( f \) is convex if and only if \( D_f \) is a convex set and its
+Hessian is positive semi-definite for all \( x\in D_f \). For a
+single-variable function this reduces to \( f''(x) \geq 0 \). Geometrically this means that \( f \) has nonnegative curvature
+everywhere.
+
    
+
+
+
    
+This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.
+
+
    
+









    
+
+
    More on convex functions 
    
+
+
    
+The next result is of great importance to us and the reason why we are
+going on about convex functions. In machine learning we frequently
+have to minimize a loss/cost function in order to find the best
+parameters for the model we are considering.
+
+
    
+Ideally we want the
+global minimum (for high-dimensional models it is hard to know
+if we have local or global minimum). However, if the cost/loss function
+is convex the following result provides invaluable information:
+
+
    
+
    
+Any minimum is global for convex functions.
+
    
+Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \)
+is minimal, where \( f \) is convex and differentiable. Then, any point
+\( x^* \) that satisfies \( \nabla f(x^*) = 0 \) is a global minimum.
+
    
+
+
+
    
+This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.
+
+
    
+









    
+
+
    Some simple problems 
    
+
+
      
+
    1.  Show that \( f(x)=x^2 \) is convex for \( x \in \mathbb{R} \) using the definition of convexity. Hint: If you re-write the definition, \( f \) is convex if the following holds for all \( x,y \in D_f \) and any \( \lambda \in [0,1] \) $\lambda f(x)+(1-\lambda)f(y)-f(\lambda x + (1-\lambda) y ) \geq 0$.
      2. 
+
    2.  Using the second order condition show that the following functions are convex on the specified domain.
      3. 
+
+
        
+ 
      •  \( f(x) = e^x \) is convex for \( x \in \mathbb{R} \).
        • 
+ 
      •  \( g(x) = -\ln(x) \) is convex for \( x \in (0,\infty) \).
        • 
+
      
+
+
    3.  Let \( f(x) = x^2 \) and \( g(x) = e^x \). Show that \( f(g(x)) \) and \( g(f(x)) \) is convex for \( x \in \mathbb{R} \). Also show that if \( f(x) \) is any convex function than \( h(x) = e^{f(x)} \) is convex.
      4. 
+
    4.  A norm is any function that satisfy the following properties
      5. 
+
+
        
+ 
      •  \( f(\alpha x) = |\alpha| f(x) \) for all \( \alpha \in \mathbb{R} \).
        • 
+ 
      •  \( f(x+y) \leq f(x) + f(y) \)
        • 
+ 
      •  \( f(x) \leq 0 \) for all \( x \in \mathbb{R}^n \) with equality if and only if \( x = 0 \)
        • 
+
      
+
+
    
+
+Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).
+
+
    
+
+
+
    Revisiting our first homework 
    
+
+
    
+We will use linear regression as a case study for the gradient descent
+methods. Linear regression is a great test case for the gradient
+descent methods discussed in the lectures since it has several
+desirable properties such as:
+
+
      
+
    1.  An analytical solution (recall homework set 1).
      2. 
+
    2.  The gradient can be computed analytically.
      3. 
+
    3.  The cost function is convex which guarantees that gradient descent converges for small enough learning rates
      4. 
+
    
+
+We revisit the example from homework set 1 where we had 
+$$
+y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
+$$
+
+with \( x_i \in [0,1]  \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). 
+The linear regression model is given by 
+$$
+h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x,
+$$
+
+such that 
+$$
+\hat{y}_i = \beta_0 + \beta_1 x_i.
+$$
+
+
    
+
+
+
    Gradient descent example 
    
+
+
    
+Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
+
+
    
+It is convenient to write \( \mathbf{\hat{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2}  \) is the design matrix given by
+$$
+X \equiv \begin{bmatrix}
+1 & x_1  \\
+\vdots & \vdots  \\
+1 & x_{100} &  \\
+\end{bmatrix}.
+$$
+
+The loss function is given by 
+$$
+C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 
+$$
+
+and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
+
+
    
+









    
+
+
    The derivative of the cost/loss function 
    
+
+
    
+Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show  that the gradient can be written as
+$$
+\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
+\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
+\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), 
+$$
+
+where \( X \) is the design matrix defined above.
+
+
    
+









    
+
+
    The Hessian matrix 
    
+The Hessian matrix of \( C(\beta) \) is given by 
+$$
+\hat{H} \equiv \begin{bmatrix}
+\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1}  \\
+\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &  \\
+\end{bmatrix} = 2X^T X.
+$$
+
+This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite.
+
+
    
+









    
+
+
    Simple program 
    
+
+
    
+We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to 
+$$
+\beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots 
+$$
+
+
    
+We can use the expression we computed for the gradient and let use a
+\( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating
+when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \).
+
+
    
+And finally we can compare our solution for \( \beta \) with the analytic result given by 
+\( \beta= (X^TX)^{-1} X^T \mathbf{y} \).
+
    
+
+
+
    import numpy as np
+
+"""
+The following setup is just a suggestion, feel free to write it the way you like.
+"""
+
+#Setup problem described in the exercise
+N  = 100 #Nr of datapoints
+M  = 2 #Nr of features
+x  = np.random.rand(N) #Uniformly generated x-values in [0,1]
+y  = 5*x**2 + 0.1*np.random.randn(N)
+X  = np.c_[np.ones(N),x] #Construct design matrix
+
+#Compute beta according to normal equations to compare with GD solution
+Xt_X_inv = np.linalg.inv(np.dot(X.T,X))
+Xt_y     = np.dot(X.transpose(),y)
+beta_NE = np.dot(Xt_X_inv,Xt_y)
+print(beta_NE)
+
    
+
    
+









    
+
+
    Gradient Descent Example 
    
+
+
    
+Another simple example is here
 
    
 
 
@@ -306,7 +564,7 @@ plt.show()
 
    
 









    
 
-
    And a corresponding example using scikit-learn 
    
+
    And a corresponding example using scikit-learn 
    
 
 
    
 
@@ -330,265 +588,6 @@ sgdreg.fit(x,y.
 
    
 
 
-
    Convex functions 
    
-
-
    
-Ideally we want our cost/loss function to be convex(concave).
-
-
    
-First we give the definition of a convex set: A set \( C \) in
-\( \mathbb{R}^n \) is said to be convex if, for all \( x \) and \( y \) in \( C \) and
-all \( t \in (0,1) \) , the point \( (1 − t)x + ty \) also belongs to
-C. Geometrically this means that every point on the line segment
-connecting \( x \) and \( y \) is in \( C \) as discussed below.
-
-
    
-The convex subsets of \( \mathbb{R} \) are the intervals of
-\( \mathbb{R} \). Examples of convex sets of \( \mathbb{R}^2 \) are the
-regular polygons (triangles, rectangles, pentagons, etc...).
-
-
    
-









    
-
-
    Convex function 
    
-
-
    
-Convex function: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below.
-
-
    
-









    
-
-
    Conditions on convex functions 
    
-
-
    
-In the following we state first and second-order conditions which
-ensures convexity of a function \( f \). We write \( D_f \) to denote the
-domain of \( f \), i.e the subset of \( R^n \) where \( f \) is defined. For more
-details and proofs we refer to: S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press.
-
-
    
-
    
-First order condition.
-
    
-Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for
-all \( x \) in the domain of \( f \)). Then \( f \) is convex if and only if \( D_f \)
-is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds
-for all \( x,y \in D_f \). This condition means that for a convex function
-the first order Taylor expansion (right hand side above) at any point
-a global under estimator of the function. To convince yourself you can
-make a drawing of f(x) = x^2+1 and draw the tangent line to \( f(x) \) and
-note that it is always below the graph.
-
    
-
-
-
    
-
    
-Second order condition.
-
    
-Assume that \( f \) is twice
-differentiable, i.e the Hessian matrix exists at each point in
-\( D_f \). Then \( f \) is convex if and only if \( D_f \) is a convex set and its
-Hessian is positive semi-definite for all \( x\in D_f \). For a
-single-variable function this reduces to \( f''(x) \geq
-0 \). Geometrically this means that \( f \) has nonnegative curvature
-everywhere.
-
    
-
-
-
    
-This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.
-
-
    
-









    
-
-
    More on convex functions 
    
-
-
    
-The next result is of great importance to us and the reason why we are
-going on about convex functions. In machine learning we frequently
-have to minimize a loss/cost function in order to find the best
-parameters for the model we are considering.
-
-
    
-Ideally we want the
-global minimum (for high-dimensional models it is hard to know
-if we have local or global minimum). However, if the cost/loss function
-is convex the following result provides invaluable information:
-
-
    
-
    
-Any minimum is global for convex functions.
-
    
-Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \)
-is minimal, where \( f \) is convex and differentiable. Then, any point
-\( x^* \) that satisfies \( \nabla f(x^*) = 0 \) is a global minimum.
-
    
-
-
-
    
-This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.
-
-
    
-









    
-
-
    Some simple problems 
    
-
-
      
-
    1.  Show that \( f(x)=x^2 \) is convex for \( x \in \mathbb{R} \) using the definition of convexity. Hint: If you re-write the definition, \( f \) is convex if the following holds for all \( x,y \in D_f \) and any \( \lambda \in [0,1]  \) $\lambda f(x) + (1-\lambda)f(y) - f(\lambda x + (1-\lambda) y ) \geq 0. $
      2. 
-
    2.  Using the second order condition show that the following functions are convex on the specified domain.
      3. 
-
-
        
- 
      •  \( f(x) = e^x \) is convex for \( x \in \mathbb{R} \).
        • 
- 
      •  \( g(x) = -\ln(x) \) is convex for \( x \in (0,\infty) \).
        • 
-
      
-
-
    3.  Let \( f(x) = x^2 \) and \( g(x) = e^x \). Show that \( f(g(x)) \) and \( g(f(x)) \) is convex for \( x \in \mathbb{R} \). Also show that if \( f(x) \) is any convex function than \( h(x) = e^{f(x)} \) is convex.
      4. 
-
    4.  A norm is any function that satisfy the following properties
      5. 
-
-
        
- 
      •  \( f(\alpha x) = |\alpha| f(x) \) for all \( \alpha \in \mathbb{R} \).
        • 
- 
      •  \( f(x+y) \leq f(x) + f(y) \)
        • 
- 
      •  \( f(x) \leq 0 \) for all \( x \in \mathbb{R}^n \) with equality if and only if \( x = 0 \)
        • 
-
      
-
-
    
-
-Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).
-
-
    
-
-
-
    Revisiting our first homework 
    
-
-
    
-We will use linear regression as a case study for the gradient descent
-methods. Linear regression is a great test case for the gradient
-descent methods discussed in the lectures since it has several
-desirable properties such as:
-
-
      
-
    1.  An analytical solution (recall homework set 1).
      2. 
-
    2.  The gradient can be computed analytically.
      3. 
-
    3.  The cost function is convex which guarantees that gradient descent converges for small enough learning rates
      4. 
-
    
-
-We revisit the example from homework set 1 where we had 
-$$
-y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
-$$
-
-with \( x_i \in [0,1]  \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). 
-The linear regression model is given by 
-$$
-h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x,
-$$
-
-such that 
-$$
-\hat{y}_i = \beta_0 + \beta_1 x_i.
-$$
-
-
    
-
-
-
    Gradient descent example 
    
-
-
    
-Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
-
-
    
-t is convenient to write \( \mathbf{\hat{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2}  \) is the design matrix given by
-$$
-\begin{equation}
-X \equiv \begin{bmatrix}
-1 & x_1  \\
-\vdots & \vdots  \\
-1 & x_{100} &  \\
-\end{bmatrix}.
-\label{_auto1}
-\end{equation}
-$$
-
-The loss function is given by 
-$$
-C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 
-$$
-
-and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
-
-
    
-









    
-
-
    The derivative of the cost/loss function 
    
-
-
    
-Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show  that the gradient can be written as
-$$
-\nabla_\beta C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
-\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
-\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), 
-$$
-
-where \( X \) is the design matrix defined above.
-
-
    
-









    
-
-
    The Hessian matrix 
    
-The Hessian matrix of \( C(\beta) \) is given by 
-$$
-\hat{H} \equiv \begin{bmatrix}
-\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1}  \\
-\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &  \\
-\end{bmatrix} = 2X^T X.
-$$
-
-This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite.
-
-
    
-









    
-
-
    Simple program 
    
-
-
    
-We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to 
-$$
-\beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots 
-$$
-
-
    
-We can use the expression we computed for the gradient and let use a
-\( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating
-when \( ||\nabla_\beta C(\beta_k) || < \epsilon = 10^{-8} \).
-
-
    
-And finally we can compare our solution for \( \beta \) with the analytic result given by 
-\( \beta= (X^TX)^{-1} X^T \mathbf{y} \).
-
    
-
-
-
    import numpy as np
-
-"""
-The following setup is just a suggestion, feel free to write it the way you like.
-"""
-
-#Setup problem described in the exercise
-N  = 100 #Nr of datapoints
-M  = 2 #Nr of features
-x  = np.random.rand(N) #Uniformly generated x-values in [0,1]
-y  = 5*x**2 + 0.1*np.random.randn(N)
-X  = np.c_[np.ones(N),x] #Construct design matrix
-
-#Compute beta according to normal equations to compare with GD solution
-Xt_X_inv = np.linalg.inv(np.dot(X.T,X))
-Xt_y     = np.dot(X.transpose(),y)
-beta_NE = np.dot(Xt_X_inv,Xt_y)
-print(beta_NE)
-
    
-
    
-
-
 
    Gradient descent and Ridge 
    
 
 
    
@@ -655,7 +654,7 @@ the shortcomings of the Gradient descent method discussed above.
 
    
 The underlying idea of SGD comes from the observation that the cost
 function, which we want to minimize, can almost always be written as a
-sum over \( n \) datapoints \( \{\mathbf{x}_i\}_{i=1}^n \),
+sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
 $$
 C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
 \mathbf{\beta}). 
@@ -677,7 +676,7 @@ $$
 
    
 Stochasticity/randomness is introduced by only taking the
 gradient on a subset of the data called minibatches.  If there are \( n \)
-datapoints and the size of each minibatch is \( M \), there will be \( n/M \)
+data points and the size of each minibatch is \( M \), there will be \( n/M \)
 minibatches. We denote these minibatches by \( B_k \) where
 \( k=1,\cdots,n/M \).
 
@@ -685,21 +684,21 @@ minibatches. We denote these minibatches by \( B_k \) where
 









    
 
 
    SGD example 
    
-As an example, suppose we have \( 10 \) datapoints \( ( \mathbf{x}_1,
-\cdots, \mathbf{x}_{10} ) \) and we choose to have \( M=5 \) minibathces,
-then each minibatch contains two datapoints. In particular we have
+As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \) 
+and we choose to have \( M=5 \) minibathces,
+then each minibatch contains two data points. In particular we have
 \( B_1 = (\mathbf{x}_1,\mathbf{x}_2), \cdots, B_5 =
 (\mathbf{x}_9,\mathbf{x}_{10}) \). Note that if you choose \( M=1 \) you
-have only a single batch with all datapoints and on the other extreme,
+have only a single batch with all data points and on the other extreme,
 you may choose \( M=n \) resulting in a minibatch for each datapoint, i.e
 \( B_k = \mathbf{x}_k \).
 
 
    
 The idea is now to approximate the gradient by replacing the sum over
-all datapoints with a sum over the datapoints in one the minibatches
+all data points with a sum over the data points in one the minibatches
 picked at random in each gradient descent step 
 $$
-\nabla_\beta
+\nabla_{\beta}
 C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
 \mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
 c_i(\mathbf{x}_i, \mathbf{\beta}).
@@ -752,8 +751,8 @@ Taking the gradient only on a subset of the data has two important
 benefits. First, it introduces randomness which decreases the chance
 that our opmization scheme gets stuck in a local minima. Second, if
 the size of the minibatches are small relative to the number of
-datapoints (\( M < n \)), the computation of the gradient is much
-cheaper since we sum over the datapoints in the k-th minibatch and not
+datapoints (\( M <  n \)), the computation of the gradient is much
+cheaper since we sum over the datapoints in the \( k-th \) minibatch and not
 all \( n \) datapoints.
 
 
    
@@ -784,7 +783,7 @@ number of epochs in such a way that it becomes very small after a
 reasonable time such that we do not move at all.
 
 
    
-As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \).
+As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \).
 
 
    
 In this way we can fix the number of epochs, compute \( \beta \) and
diff --git a/doc/pub/Splines/ipynb/Splines.ipynb b/doc/pub/Splines/ipynb/Splines.ipynb
index 5328f5ba0..4f4ff5c1d 100644
--- a/doc/pub/Splines/ipynb/Splines.ipynb
+++ b/doc/pub/Splines/ipynb/Splines.ipynb
@@ -10,7 +10,7 @@
     "  \n",
     "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
     "\n",
-    "Date: **Sep 20, 2018**\n",
+    "Date: **Sep 21, 2018**\n",
     "\n",
     "Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
     "\n",
@@ -43,7 +43,7 @@
    "metadata": {},
    "source": [
     "$$\n",
-    "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ \\gamma_k > 0\n",
+    "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n",
     "$$"
    ]
   },
@@ -51,7 +51,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "for $\\gamma_k$ small enough, then $F(\\mathbf{x}_{k+1}) \\leq\n",
+    "with $\\gamma_k > 0$.\n",
+    "\n",
+    "For $\\gamma_k$ small enough, then $F(\\mathbf{x}_{k+1}) \\leq\n",
     "F(\\mathbf{x}_k)$. This means that for a sufficiently small $\\gamma_k$\n",
     "we are always moving towards smaller function values, i.e a minimum.\n",
     "\n",
@@ -83,7 +85,7 @@
     "\n",
     "## The ideal\n",
     "\n",
-    "Ideally the sequence $\\{ \\mathbf{x}_k \\}_{k=0}$ converges to a global\n",
+    "Ideally the sequence $\\{\\mathbf{x}_k \\}_{k=0}$ converges to a global\n",
     "minimum of the function $F$. In general we do not know if we are in a\n",
     "global or local minimum. In the special case when $F$ is a convex\n",
     "function, all local minima are also global minima, so in this case\n",
@@ -105,7 +107,8 @@
     "\n",
     "## The sensitiveness of the gradient descent\n",
     "\n",
-    "GD is sensitive to the choice of learning rate $\\gamma_k$. This is due\n",
+    "The gradient descent method \n",
+    "is sensitive to the choice of learning rate $\\gamma_k$. This is due\n",
     "to the fact that we are only guaranteed that $F(\\mathbf{x}_{k+1}) \\leq\n",
     "F(\\mathbf{x}_k)$ for sufficiently small $\\gamma_k$. The problem is to\n",
     "determine an optimal learning rate. If the learning rate is chosen too\n",
@@ -116,98 +119,7 @@
     "randomness. One such method is that of Stochastic Gradient Descent\n",
     "(SGD), see below.\n",
     "\n",
-    "## Gradient Descent Example\n",
     "\n",
-    "We revisit now our simple linear regression example with a linear polynomial."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [],
-   "source": [
-    "%matplotlib inline\n",
-    "\n",
-    "\n",
-    "# Importing various packages\n",
-    "from random import random, seed\n",
-    "import numpy as np\n",
-    "import matplotlib.pyplot as plt\n",
-    "from mpl_toolkits.mplot3d import Axes3D\n",
-    "from matplotlib import cm\n",
-    "from matplotlib.ticker import LinearLocator, FormatStrFormatter\n",
-    "import sys\n",
-    "\n",
-    "x = 2*np.random.rand(100,1)\n",
-    "y = 4+3*x+np.random.randn(100,1)\n",
-    "\n",
-    "xb = np.c_[np.ones((100,1)), x]\n",
-    "beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
-    "print(beta_linreg)\n",
-    "beta = np.random.randn(2,1)\n",
-    "\n",
-    "eta = 0.1\n",
-    "Niterations = 1000\n",
-    "m = 100\n",
-    "\n",
-    "for iter in range(Niterations):\n",
-    "    gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y)\n",
-    "    beta -= eta*gradients\n",
-    "\n",
-    "print(beta)\n",
-    "xnew = np.array([[0],[2]])\n",
-    "xbnew = np.c_[np.ones((2,1)), xnew]\n",
-    "ypredict = xbnew.dot(beta)\n",
-    "ypredict2 = xbnew.dot(beta_linreg)\n",
-    "plt.plot(xnew, ypredict, \"r-\")\n",
-    "plt.plot(xnew, ypredict2, \"b-\")\n",
-    "plt.plot(x, y ,'ro')\n",
-    "plt.axis([0,2.0,0, 15.0])\n",
-    "plt.xlabel(r'$x$')\n",
-    "plt.ylabel(r'$y$')\n",
-    "plt.title(r'Gradient descent example')\n",
-    "plt.show()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## And a corresponding example using **scikit-learn**"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [],
-   "source": [
-    "# Importing various packages\n",
-    "from random import random, seed\n",
-    "import numpy as np\n",
-    "import matplotlib.pyplot as plt\n",
-    "from sklearn.linear_model import SGDRegressor\n",
-    "\n",
-    "x = 2*np.random.rand(100,1)\n",
-    "y = 4+3*x+np.random.randn(100,1)\n",
-    "\n",
-    "xb = np.c_[np.ones((100,1)), x]\n",
-    "beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
-    "print(beta_linreg)\n",
-    "sgdreg = SGDRegressor(n_iter = 50, penalty=None, eta0=0.1)\n",
-    "sgdreg.fit(x,y.ravel())\n",
-    "print(sgdreg.intercept_, sgdreg.coef_)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
     "\n",
     "## Convex functions\n",
     "\n",
@@ -242,7 +154,7 @@
     "for all $x,y \\in D_f$. This condition means that for a convex function\n",
     "the first order Taylor expansion (right hand side above) at any point\n",
     "a global under estimator of the function. To convince yourself you can\n",
-    "make a drawing of f(x) = x^2+1 and draw the tangent line to $f(x)$ and\n",
+    "make a drawing of $f(x) = x^2+1$ and draw the tangent line to $f(x)$ and\n",
     "note that it is always below the graph.\n",
     "\n",
     "\n",
@@ -253,8 +165,7 @@
     "differentiable, i.e the Hessian matrix exists at each point in\n",
     "$D_f$. Then $f$ is convex if and only if $D_f$ is a convex set and its\n",
     "Hessian is positive semi-definite for all $x\\in D_f$. For a\n",
-    "single-variable function this reduces to $f''(x) \\geq\n",
-    "0$. Geometrically this means that $f$ has nonnegative curvature\n",
+    "single-variable function this reduces to $f''(x) \\geq 0$. Geometrically this means that $f$ has nonnegative curvature\n",
     "everywhere.\n",
     "\n",
     "\n",
@@ -285,7 +196,7 @@
     "\n",
     "## Some simple problems\n",
     "\n",
-    "1. Show that $f(x)=x^2$ is convex for $x \\in \\mathbb{R}$ using the definition of convexity. Hint: If you re-write the definition, $f$ is convex if the following holds for all $x,y \\in D_f$ and any $\\lambda \\in [0,1] $ $\\lambda f(x) + (1-\\lambda)f(y) - f(\\lambda x + (1-\\lambda) y ) \\geq 0. $\n",
+    "1. Show that $f(x)=x^2$ is convex for $x \\in \\mathbb{R}$ using the definition of convexity. Hint: If you re-write the definition, $f$ is convex if the following holds for all $x,y \\in D_f$ and any $\\lambda \\in [0,1]$ $\\lambda f(x)+(1-\\lambda)f(y)-f(\\lambda x + (1-\\lambda) y ) \\geq 0$.\n",
     "\n",
     "2. Using the second order condition show that the following functions are convex on the specified domain.\n",
     "\n",
@@ -375,25 +286,19 @@
     "\n",
     "Let $\\mathbf{y} = (y_1,\\cdots,y_n)^T$, $\\mathbf{\\hat{y}} = (\\hat{y}_1,\\cdots,\\hat{y}_n)^T$ and $\\beta = (\\beta_0, \\beta_1)^T$\n",
     "\n",
-    "t is convenient to write $\\mathbf{\\hat{y}} = X\\beta$ where $X \\in \\mathbb{R}^{100 \\times 2} $ is the design matrix given by"
+    "It is convenient to write $\\mathbf{\\hat{y}} = X\\beta$ where $X \\in \\mathbb{R}^{100 \\times 2} $ is the design matrix given by"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "\n",
-    "
    \n",
-    "\n",
     "$$\n",
-    "\\begin{equation}\n",
     "X \\equiv \\begin{bmatrix}\n",
     "1 & x_1  \\\\\n",
     "\\vdots & \\vdots  \\\\\n",
     "1 & x_{100} &  \\\\\n",
     "\\end{bmatrix}.\n",
-    "\\label{_auto1} \\tag{1}\n",
-    "\\end{equation}\n",
     "$$"
    ]
   },
@@ -429,7 +334,7 @@
    "metadata": {},
    "source": [
     "$$\n",
-    "\\nabla_\\beta C(\\beta) = (\\partial C(\\beta) / \\partial \\beta_0, \\partial C(\\beta) / \\partial \\beta_1)^T = 2\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
+    "\\nabla_{\\beta} C(\\beta) = (\\partial C(\\beta) / \\partial \\beta_0, \\partial C(\\beta) / \\partial \\beta_1)^T = 2\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
     "\\sum_{i=1}^{100}\\left( x_i (\\beta_0+\\beta_1x_i)-y_ix_i\\right) \\\\\n",
     "\\end{bmatrix} = 2X^T(X\\beta - \\mathbf{y}),\n",
     "$$"
@@ -483,7 +388,7 @@
    "source": [
     "We can use the expression we computed for the gradient and let use a\n",
     "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n",
-    "when $||\\nabla_\\beta C(\\beta_k) || < \\epsilon = 10^{-8}$. \n",
+    "when $||\\nabla_\\beta C(\\beta_k) || \\leq \\epsilon = 10^{-8}$. \n",
     "\n",
     "And finally we can compare our solution for $\\beta$ with the analytic result given by \n",
     "$\\beta= (X^TX)^{-1} X^T \\mathbf{y}$."
@@ -491,7 +396,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 1,
    "metadata": {
     "collapsed": false
    },
@@ -517,6 +422,98 @@
     "print(beta_NE)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Gradient Descent Example\n",
+    "\n",
+    "Another simple example is here"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "%matplotlib inline\n",
+    "\n",
+    "\n",
+    "# Importing various packages\n",
+    "from random import random, seed\n",
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from mpl_toolkits.mplot3d import Axes3D\n",
+    "from matplotlib import cm\n",
+    "from matplotlib.ticker import LinearLocator, FormatStrFormatter\n",
+    "import sys\n",
+    "\n",
+    "x = 2*np.random.rand(100,1)\n",
+    "y = 4+3*x+np.random.randn(100,1)\n",
+    "\n",
+    "xb = np.c_[np.ones((100,1)), x]\n",
+    "beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
+    "print(beta_linreg)\n",
+    "beta = np.random.randn(2,1)\n",
+    "\n",
+    "eta = 0.1\n",
+    "Niterations = 1000\n",
+    "m = 100\n",
+    "\n",
+    "for iter in range(Niterations):\n",
+    "    gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y)\n",
+    "    beta -= eta*gradients\n",
+    "\n",
+    "print(beta)\n",
+    "xnew = np.array([[0],[2]])\n",
+    "xbnew = np.c_[np.ones((2,1)), xnew]\n",
+    "ypredict = xbnew.dot(beta)\n",
+    "ypredict2 = xbnew.dot(beta_linreg)\n",
+    "plt.plot(xnew, ypredict, \"r-\")\n",
+    "plt.plot(xnew, ypredict2, \"b-\")\n",
+    "plt.plot(x, y ,'ro')\n",
+    "plt.axis([0,2.0,0, 15.0])\n",
+    "plt.xlabel(r'$x$')\n",
+    "plt.ylabel(r'$y$')\n",
+    "plt.title(r'Gradient descent example')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## And a corresponding example using **scikit-learn**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# Importing various packages\n",
+    "from random import random, seed\n",
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from sklearn.linear_model import SGDRegressor\n",
+    "\n",
+    "x = 2*np.random.rand(100,1)\n",
+    "y = 4+3*x+np.random.randn(100,1)\n",
+    "\n",
+    "xb = np.c_[np.ones((100,1)), x]\n",
+    "beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
+    "print(beta_linreg)\n",
+    "sgdreg = SGDRegressor(n_iter = 50, penalty=None, eta0=0.1)\n",
+    "sgdreg.fit(x,y.ravel())\n",
+    "print(sgdreg.intercept_, sgdreg.coef_)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -624,7 +621,7 @@
     "\n",
     "The underlying idea of SGD comes from the observation that the cost\n",
     "function, which we want to minimize, can almost always be written as a\n",
-    "sum over $n$ datapoints $\\{\\mathbf{x}_i\\}_{i=1}^n$,"
+    "sum over $n$ data points $\\{\\mathbf{x}_i\\}_{i=1}^n$,"
    ]
   },
   {
@@ -663,22 +660,22 @@
    "source": [
     "Stochasticity/randomness is introduced by only taking the\n",
     "gradient on a subset of the data called minibatches.  If there are $n$\n",
-    "datapoints and the size of each minibatch is $M$, there will be $n/M$\n",
+    "data points and the size of each minibatch is $M$, there will be $n/M$\n",
     "minibatches. We denote these minibatches by $B_k$ where\n",
     "$k=1,\\cdots,n/M$.\n",
     "\n",
     "## SGD example\n",
-    "As an example, suppose we have $10$ datapoints $( \\mathbf{x}_1,\n",
-    "\\cdots, \\mathbf{x}_{10} )$ and we choose to have $M=5$ minibathces,\n",
-    "then each minibatch contains two datapoints. In particular we have\n",
+    "As an example, suppose we have $10$ data points $(\\mathbf{x}_1,\\cdots, \\mathbf{x}_{10})$ \n",
+    "and we choose to have $M=5$ minibathces,\n",
+    "then each minibatch contains two data points. In particular we have\n",
     "$B_1 = (\\mathbf{x}_1,\\mathbf{x}_2), \\cdots, B_5 =\n",
     "(\\mathbf{x}_9,\\mathbf{x}_{10})$. Note that if you choose $M=1$ you\n",
-    "have only a single batch with all datapoints and on the other extreme,\n",
+    "have only a single batch with all data points and on the other extreme,\n",
     "you may choose $M=n$ resulting in a minibatch for each datapoint, i.e\n",
     "$B_k = \\mathbf{x}_k$.\n",
     "\n",
     "The idea is now to approximate the gradient by replacing the sum over\n",
-    "all datapoints with a sum over the datapoints in one the minibatches\n",
+    "all data points with a sum over the data points in one the minibatches\n",
     "picked at random in each gradient descent step"
    ]
   },
@@ -687,7 +684,7 @@
    "metadata": {},
    "source": [
     "$$\n",
-    "\\nabla_\\beta\n",
+    "\\nabla_{\\beta}\n",
     "C(\\mathbf{\\beta}) = \\sum_{i=1}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n",
     "\\mathbf{\\beta}) \\rightarrow \\sum_{i \\in B_k}^n \\nabla_\\beta\n",
     "c_i(\\mathbf{x}_i, \\mathbf{\\beta}).\n",
@@ -758,8 +755,8 @@
     "benefits. First, it introduces randomness which decreases the chance\n",
     "that our opmization scheme gets stuck in a local minima. Second, if\n",
     "the size of the minibatches are small relative to the number of\n",
-    "datapoints ($M < n$), the computation of the gradient is much\n",
-    "cheaper since we sum over the datapoints in the k-th minibatch and not\n",
+    "datapoints ($M <  n$), the computation of the gradient is much\n",
+    "cheaper since we sum over the datapoints in the $k-th$ minibatch and not\n",
     "all $n$ datapoints.\n",
     "\n",
     "## When do we stop?\n",
@@ -781,7 +778,7 @@
     "number of epochs in such a way that it becomes very small after a\n",
     "reasonable time such that we do not move at all.\n",
     "\n",
-    "As an example, let $e = 0,1,2,3,\\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \\cdot m + i$ where $m$ is the number of minibatches and $i=0,\\cdots,m-1$. Then the function $$\\gamma_j(t; t_0, t_1) = \\frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.\n",
+    "As an example, let $e = 0,1,2,3,\\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \\cdot m + i$ where $m$ is the number of minibatches and $i=0,\\cdots,m-1$. Then the function $$\\gamma_j(t; t_0, t_1) = \\frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.\n",
     "\n",
     "In this way we can fix the number of epochs, compute $\\beta$ and\n",
     "evaluate the cost function at the end. Repeating the computation will\n",
diff --git a/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz b/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz
index a1b16d968d9d315c255e4d3233039aacb3ac3232..34845ba2e0a79a2b86a8a7dac542f2c372d4c502 100644
GIT binary patch
literal 210
zcmb2|=3rP*uq2v+`R#efyh8>8t%=v>B>4$gsA3rmjg
zl;M1&

literal 210
zcmV;@04@I?iwFSoXro&I1MSbxY63A3$8pa-#XP~1b=P<<=)psww7x)OqZ|Ju+ayBw
z?Ohexlf6liem|EV!wkdB`{HH(x+wBj9zBdwO4qeqa;B%(RFhkMRhC-Hs+uaT>q=Lm
zhC-{6sK=f3P
M1F<0zSO6#h0CqQMdjJ3c

diff --git a/doc/pub/Splines/pdf/Splines-minted.pdf b/doc/pub/Splines/pdf/Splines-minted.pdf
index c5ac6574a2c5f4d1f5517a9ce5f554beead3072d..b63252b15c08903249e5f9d05ea78d04bfb1990c 100644
GIT binary patch
delta 48978
zcmZtNQ+p+VvY_GEwr$(CZQHh2oOEn-Y<6thwr$&(efHeVnY;Q4^*&$KaTdmKIz|F1
zC@VW#asdn#pjlft{;&l-VArrf5?i0SD@qNMzAlHuI(VCGOd$wpTfoXL_TN$(L=s=;
zqTy~l=4PQpi=|HxA(;wa#3^3Dsl=jL&fHNa=hMWaVyk1%ds<_t^+0CQ__hB)F>%bP
zbPZbK7Flc6(d~KOWPzTUNY#_SszcsK$7@l8_Ir9f;Mj#&tD_=oLeZ~$qNXD2`GBxh
zkPMq#FOux=ZzM_GmlA!1zvgqVp;^?zQ&~j8&hjr!M29^^SMTM7x;}f_@7@}-f^-ci
z@AbNWb+~-=GoAjk$2_Zg0i`00%F!IWjfEac<|J7%II(Be+h{6F~`nRs&
zV{2(ZldJmbYRD&+>erIvrZ%vvM8~4=T~c;uwkLI3*K^@Qs!f
z5T^14g=h{G&qsN+hhXXn0&4EYV@K4KM57O3cu}32l2XAB`a8a{BT!Jex6?g2
z(m9=^@G5?VO^-7AXVmoR-w4Mo7OE9oHrS&NU!LA+Q%AY+jJbpk;jI1F^&&S}7cC}3
zWY~-#kPO&a7&f`n-{ik+-m`=Y5T|bo64@tG
zs%Oq$V)oXf%Iu(9H4M!l78y-hJ!lLcV~e*QBT3dNYgN#Y9xb~VVZe}vNQOFh3CG>m
zmg0Lyf01p3QHO%nG2OYW-`?y)%OLX5=Mp|;qr1OHV(+^H4dT4oA-%
zEc&uyxs8Mm@x@UG(?eV{a&43$jF*0X@gn19la`$rhCqVSH)kO-x0c^E7zsPkuiL<<
zL63j)M26q0N^yijx4WANrwk->Vrqh7Eu+ss9uN0vHl>jk08Y<`IlRx!JDm%83LZo_
zg=^!7nXgBk$GoKEoJCllO0|$Q|JRe%#u$E;@*maX^X5MNt0OCk?32@jo8f+W&&-x>
zpF_-})XGWJaiGljk!?Id*_neBg=C)eDh_=o%r~~4v`WrSdiQk44B)!0=XG|4BmBjD
z8v+vXveUy804%Daf|gZBcF3;?(P=aZdLGyz&SG(?gc^=e`P8Rq>Wjfl)^Q_F%yqi
z>Idb4h1LkoC>i`G+tei<@g5V7H*H_V#opo5T#~Il=pUD%I(Zu390H94ysx7;b%=MQ*|L(k=Y75EyK{HbU
zN&#)@hf&T`0jJp5`iXlo`x>W&LIsT(`=0Z-wFi5O`y*~a-~DJ*EtGR}s3U!W@!~bF
zVyw~CKd@8ZbU)3{`;1X9#NI6Ms{~i?4;qar!>Yeh_ZTEM``DHE)YsaBXh#EYm)*7-
z;G99+N3hlQ`fT1X+K8{M-f)$E%g`r7(?ERH3YWX*eE7_{Af;2t8DN2amOXAfxc)bq
znkqeF@h?Ytk)RC{Ec;aO+kq(E-yJ&3p>m|v^c@C?*fjI4V~@|;vTTT#NW$!}ya!y`
zD3V6NV;e1m_Xj7+?R;9ubQr|-B-<5lz*cOzRPI@C+E+50P&nDg1&!i>&Qfa
zT~?MyL;yg$(}GZztYAm1Pf)S|3tfBD$NDv<`XPOj^A}N1h9W33fh*()zS^v9mmFUPOZ?z!zd}>Z5>aGk7g#cNnBmS|Y
zRuFeiEzwN@^@QLZvw@Z=2)GDc$4n(ToVxJKU=oc>2h^E2u6}*7@CBCj#6#mT3M-80ged;EPwCYj==h7CcRZgoJ|&ZM1o)banjoe>)pCNPo%38e?Kkw*Z_m*^uG=ddQWL_$0Xk?zY>{|!
zYSmd|O0|l#;cNrNK678b0xsl@*P!n)6j+2`kP2fw1o6ehdgMQgVdaWb|)2kmNIde
z9=<`7>!gSVdSJot4FNc=g5of(#8K9+D+q0~71;}Fb*L(g3-EHt>%X5nt4~l=ydu@6
zk?vsiZvxejM883~aZD-{z9Lc?H2KJTH^BWqXDTi1pg>YozKUyvko8j_2YsdB(?~8=
zf008%NnVirNYezL!T5*)5dE#5#~8wh5YDH7I;H-V=mA%lQNpRR?6Jq}D@k2&;I<8K
zp%-`43-R&^pheg4r(LJ5dsvxPR?&inU;a;@%hfNpEqv0ZiNg0A%{W}1;;+ZRr>nMW
z%~BF=!;t2~KVRJb{SFg0iNJR=tJogwiUNm(yEoKLYva#n?f^T$4b|R;d8F$VSkbb*
z3!tVo=vYs{e&hsL?xeEITiE}wx{iYY^~-jAJlL@qW-7bh73?5^($}y^w8@zI?gvXF
zc9vnCeMy6ppWUxAD{GFDP4jh$2Yx)c;kX`s)^Op{irxx_5@jQ9Y&*(K_S(Tgd{1x$
ztte^gtUDafBI8$0K3OZU4
zVSwahDP|$UrRB$xWijm*UYRXM$MJ+JAdM#oUwUG$Zpr|}!TTeWLG&zOFf`r#J;a6&
z+RQPJ@oF2vKLW=Xr#qkc8z&mwd0)E=fYNDG0rXYe?EUox0o<
ztc}ioW=lsGNnj0Pm8q}sIEKN&zd~XA8OZ_PxX_Ybwl}pYplmG8mZs;5w(sM`%=v6b
z3EF%omPBnDx4K2E;@#-`O+nG?*qZAY1rukCxbAF!YJtQMJj27Di}8}6<+|f
zz;OP;i&g){AQ14+Qow-^c6RVNoRt8mX=+%oxP1A+rcLygIEdB>kM^uB!x7t|^Pr5W
z{93U2nD56f;x~lgs22~dq|4xH3JGy^(7#OcUK&FJLObk>5MXAt#F4c~{=2~a2!^9{
z*Sfx1e^J9vVX%EU)VbY7PCfj9zY73GKiq{`S?eGTl2q@eI7j+nJ(fgfrq<|Fg!*ab
zp@$B}IXn6wc&(G}PSzTf*P8ubNQXeEcn2{hzFVG~t!yXj!pcsJEYlE7_=^Sh$NnX!!Hq6;T
zFu#Uq@lYHN9uv=>44Y2;y5K$sBHLLcbRzwGc41EutF9z_3naJZ*paVwQ+{eQW0$?8
z$8piPR0MFLt{CeZBI*Jk@n%4|Gu)Fqk-J{BErE|w@93#G8?I7}_?Jg76?ZH@;~rFg
zj1qR1Kp*kEIz@I&F+MT_L`EM)Jtp=CxWS|uE~Kel%sP$AsWG;{`$KX{BFd<>ecf1G
z2YmM5hcUY@2_1%c?*4wCnjVEX^4W^AE1*OB@%JsPjLiwvmY`UAtqQ=#!X&=GC5-|5
zwYX0OU7hlvM}TJd77TXBeikdLoD@1xx?4o)HSvupSk)^&yGx5>h}r^%jxprdX(gOi
zcTs8Q`Vl&L5B;%9=VMyeNdD?l=ZPAmMXyZ@oR-4O`RuJId2LN{*LobsX5H=?
z^Z2T%{0skPZg@KW<{W^`F$mE}+PWhyWT#iG1i?zk%2yH7tu~zPFEw)3A=XV6L2o!r
zw*$x@Vu&ZPEejbL%^6N2A%&5(;bDD#*>O1jbl2uASJ^zK&+q*;H@iG)@eEdek238b
zUP2TnJ;4^BB5H2Nh+H?rcD{`(mjujD<_7>D5)x@RmpoZ@yBKmRik{qYABz
zgLw38PZ10xB%uPj2_gm>5LBJW%X$)g+`?)PGvEhrPW!8<>+o$|3`H_CY?@;zm2KDl
zBnK~1dXy@Y)j~ivE7Q)%BcHz33KbqNEh?f&U$={DDoueo-@zoT37S*R7QI_lje2CN
zikmDi;r$0Wv(bc{fZ>$ZM>ol=PC+1;xalW)DI|0=uA}i1_+K`f6wve4X!X7Mp0p;>
z50I4BNXYM}Br=aFtAs9*@sSG=7|VWoh=#}5Ftey=`C)+crn|aZ8nornqqrgYj1eG<
zzj0}}>fwIpUECfCso1|XwhKbNy+5+YBZ*;>n&^w}>Gwyc2HJ1l9Jc!-9U%$$M30Uw
z%YhopBf-S|Ftx8Q{~E=Ss8f0HHrxOfVWuYtOK~`yafwa6U9qRSvjBOn{K!_TChKq+
zBR=yTHUc0aD(DL%I?CfUt(dc$_tnKHWN}SwwDM_-G4w>&dM~m9gm_ylY5w1uHQpafc|O^vyg|elWKJJ-0^1FO-V*nh3GHa&+4CgBFl0uh!2l(
zmFSHCK)W<01;G}rS(cyQRp4OjPI)7j^ls&
z--vQ9r)K|~IjsLb=CE=9U*Ly>dzy}ES4)`ZiYD;weF=O$z+Mtl9$ni+JHm8^l
ziKE~A`_Yhi)Y$p{3@#;wVXm!n`p}R@1xzLk^d+E_#n2M%cE8-2642}A_d1>)%*bV_
zU|a9CMc0-xW#*A+Js7$aJ^FkQaimywsPb?G`a^AyhPZHM<|Qh3QA^1!40
zRc^b9=|G-p7FDJ7ASf*K&=Y7BZVCt}|CKMvPbc}8P2E4)4+rWCFM%~s;`-iAZg<0
z?1C9m|39H7T~xq8KtfcQ4bh|!dqm3TwQGG5&h4GEkGX>lD-($Ep!*Tq$)An6uFvAEvt
zUBKCp>pGy89hhM@QE`Vucur%QnHs$#OW3~@dbj{^8le&X8Ofi_jHloups(r(?Y!v{Y*3)}u~N{k~~d6P$j-b^Xd}
zSwwVO?HEuAy?R{#cAkBk^?=T+TRhcXHO8Ec+!Gj!^CBTH1nx2K4m>7{FAGLX7-2a(
z>0!8h_ZHa0dQH9YH-OM{c$En&T2jg4Dq0*KZ_!;qAH8MY}0so+~>n
zW6^(j(IR;Z=p)5H2>0GS$lPQEE6`huf;1;%x(pC3O9M;oh1^|D7
zAPxwq4pGX-@d7_v5%d!W2HNRy)<5?8#JhRmAYI#cLu)8y!p-n8aPTA;DUxn5GX#ke
z#Mp-jcc(MT9=%r6lb(j72+G}=ZX;Zeqi`|8s~>~EayCv2I(rhg7|`66aQLpbLTksF
zgwMMvX6YX?T3~HoDD4~E(zj^9X5H8}rvOBP{A)BcqG0C7$Smy>+s?5_s(1+OW&~$I
z^tE3Lof{|OLK@FGe`$j5-F}Rdj~M;YG$cq53dKM_#|DKQgaV3x7EcQluK7ou(VcLk
zdy-saz#$X^ynWsZEFsWIBpPd)PhncoEDDix`#o}lIz-#7jIxz?H_m#0iSK;j?+qYJ
zNLai)4gie!&7)m1drJ()PUs-E(VF+Q8V_DBm#h#tFBKGq(MI)F{jt&HzqYs@ps_8m
zMa6ireL@pg9f+Q`jyH7&svs(OFU7dtbKmY;8Xzw$taKQNjhOuj1Bn<*ejcpQ!Bn@>
zd$u@aznMU%+1~#r6=mvQ6V>6HegQzMNk4YH;(-Bjh4~l2fQ6>=(G3pRQ89AKmw;Qc
zE3Go!;!ev9f)g=v+5jA4XolY8RiV(aCx)^W^PpF~KT18%fE+X;Fa5>N9|J1F#k586
z?Er4_;bUhLaWS>x!+!wsg~xFQamKp5KjnV`5jebzPlP+LN~IvjMJN&Itp!A%z>6wl
zhjJ5^<1Z69(25QhD#7t22uh60eySa0{{sbs1MhoB2@$`^je)X@!`Y
z>k)e#5b*8~4pz`?tp4rV^2K)YCG^<;sNi|`xwaOC$xF9?<^a8DqLxy3eOO-bbul1qYNEzTv%eICSbl;_o&fXQV+aoLK4qG&c!|qX{%s
zr0Cguf{(;5^uP#K%l~{=P$M4Ca)>IB?9sJHK^K>VE6!>Oom1B=-GLLMdmEyb8Y
z^-GD%h6Doi>6Af?<$^=LH#}L(4`fhB?3#CFpTl2EPezeG)gAq1m^J
zxPYjKKh?RdEvsDmcmVbolRaBl?yCj;@&iL6#I>6ogB=N1-K=k$<7Bjf;{#U^su|c2
zAA&pLB35oOCiF?8`ggn+(*VK0W=J`GWavNwM<+aO*IpZP10?
z8n7RpO3^1p#vQkT0!u}%fXQJ0YS42-AAcL>BnDyyH_i*g6hc5AvZ1dz*CmkL2(*Gg
zTy85gXiV|z3cjoWl@?9TwrjZSXjdr_UPmt*l(?PK@3wXCi&>N^N_cc8*%jBJQMlAJ
z?EPYWy9+E>8~`yo5p?vpgSRXr=<~zgiEy2?JW}G11=#g@uA9OJp0
zK!Y45YTn%;uYLvb7zU@OKxRK%ij^l&VYI+9N(~6YU0V|lHK$Ok{~BK6D}Vo<2%%GS
zkETA2(^}JIgNJ@ibUrnr-ltnZJh9a`Ytm#qzLXm@F@S|8ZZ3L%-UHy1cRTp}9vRZz
zFI3`Er;6&wXwz5ieHfj+g^Kl7(v69$vuG^N*39J&Jop^1Bbz>o-YsW7S;_E{RUQUr
zL}Tzt{7V?*Yd15fsC)86MzH&L>$G!RY7MT~OLaR|_O!Pn(#(PhPji6~M2v-z^*b;q
zkrMv1A7BSp_w^`^mdecF=;o<^cIiBn$wLUa06krQ5KqLM!Inw=0*r2fcV<9_lbsn|
zXE(E&xyN9{(KQyoS@@`8l|O(gz$2eq(1&y0nWhec%Hm01i8eDhFSz_Z=KN(W$#;ybQY%hC$r_Q653(Dy|ll<
z2ylRT=QlKNx*FJ4=ZN5(Je=yk${#f}2)wLK2hN725?HvB#X=dMJxV0VcRq#)`KwC<
zlqZ8|Cc~mG<_84V1*YqLliace$q@M&Dv3&lr}xM8o)=4A8i6<`c9sND6jv$lg$d9`
zUi2MHoOfOM2MkLt_^6ebVb{mhODNX9003d)IAz#u1Y6ZS)4_&K0vrF;HM>`VykHH
zXhardi1x6O#rswcSIW@T!NO4y_3RA}L3bA5Vu}hgD>Ed-s;wf%>-%+XJ1Tet4`BO8
z?a$bK^)8ssd
zG4+;EZKAYBt22h^;bz%E+HBplKylVz4)XPvZF>%P?M2GB3m)SI}
z=)Ig$^(av5mll@hs&6;*>7)-ibqoG?u8AJ>xN@^~&{4r>^Wg`^WCRf}7uh{|9OvZlS5S;gK0sv8r9Ph&h
z#MlH}@lOF#MO*b$dz7?xC83CRA@DQLB$lA0)GbQJkp7oGJLN0taq%*QIG6aA`Ff%#
zU@bXrSQfR$ucTYx_MF47_GDn6?8~8w#E}p`cYs(P{4GE{>Pr~kYcKQYtW!*~>VS95
zRuG>IrZa8EXGO@8k4j^H7=Qtctk{CThZFEI!fw~5?Hekye2^xSm$fNAI%B7{`u*1o
z(VJ_tq|rI+D!Pk=F}daS8n3)ODsKi1cXQ*C;TAYesY4*q#6hmRhxN#xnojzy0HUPdM213MG&A#rC_1-^m$rHpKTm{_>})3j&+(&>l)Em~(|0oeU(>GzZU+IYN-Wo?fxcW-^x^|~?CE+R6E
z+rI)~VQ9cX2UaGEF4pzZK8S!;7Qp*guXh*^yPLMvpV0dvcB4jHsJHLp&1bWbLmVdz|X$@
z*>KN@=&nn6mA}~E>lFbWaOETDVkmgs+WDc>EV|w7S$wzX=6rV3!ep(<9s*E?ByoQO
z7``z1dA>e=8*Wn!wLy<0B>9lG3#onb9
zliYmoCat(g8Brl@n}Yn+)fj*DiTa{OYpnTr-9OC??+a@Khn#HUes&SNKY!|X)Q$Uj
zswu(nrYig#U-jPm-C-lVsOUo+=_3pQlIME09J(|K0FJv^1g@`^ZqueZ9W#m4z~3Ah
ztn>E>#N?g6Mor{5D^74z!3{wN)fZB!Q5Av&O8g2o;S30CX6pnsJJNcQY#hdaf&+YO
zCV6;3S3mx$_yg2}J{6=f-Xz|}Upy^m_fOZ%nB(m#;CKt_lMY-v8$1?
zyra-DfN@5h$y2%Ed|7ol#Gi-&3yPjRoaA%x}8XDN5TyTQ40(X9%AR{-A1qv_Z{p0W;Z_>hon#zW7VSyCag
ze$%7hHJ)PG)EqNvSf7F5Id8ATMck_l%0@A1)hg$ns-RVdd)^mC%@s{?kznfYSc+8=OWCl4ruuHf
z5uS%DLsc~ceN4<>4+bg)>U3fGb_NKjwc~)-fqgFA!*J7PC71M3qK_9hQwfm)NXT*@
z8YCi6iu-Fz-KvMJ;V?YI@D}%ywQ_U2EZ*5n`;b+GNi33byug#Nd65th^`V);M8!(9
zB^ZUmb%3!$DTz#-F@aK%EdTf^hyrDpJOQCNb+PvNGmf1u>
zo5=W>?D|If_`wF^YJB$BM1fZWfEYWIza@P#I=eTM4%WjjKD6Fnb9uei)`0>n?xv&i
zoA;PeBt>c6zOp4#}JXg4`cBf`=+B_9rrt;hRZtiS$**oh(BPP~No=$KO;2#cg
zW-{V<9^l8sX^_@RY65?W=){ZNkQ0BXL!^iqT<>hboJkAUaK~X%q$Z~Uc1qms=Ro2t
z{a6{Sr-Aa~8tY*!)W3Fre)Dcx3dgt!!%{LO{F7r(vlS%717>o7g?QOdIbai^$*_)C
zg`_~tiuW7EtFwxl-g18HtCC{Mi5`%6iejB#+V;;?PN2Qzt;7U+A%ARx>sPOEQY^9l
zc+eAj&ga!xztA#Go(4|>C>Rn9bJcyWyX+u_9gM#xyb~3XjJ$vl
zWSNc^4f<0I`NEx}tYSkGn2iT7E!lWQNO}ZNCEwGx}bV%8r2YP!x!YBAGLP@YpdF
z%PI7d?0hJtul!JIZ8H`m9+g2=v(N(^2r}n6$6>y!))twU1>+o3_jzDuUu2NA%vU}S8WNa-?nd)vm|8-uA`8rlJ(I~h3y2$B
zbC~4;)XTOeoMeu^Px{Y{zY3Mlln@9OnL0%AET%~$QX0s<7qajZ%u6%};>=1zu8lCm
zuz3`qeZq!fR34d($Yb~bEtS$VU<2fbzUMii^ZH}(tCYIqz$~$4`qq6b=kV~-hj2K8
zSLI26tnJfOAAU87X^y1gcNLoh$!xFZf%6Ovgr%1lGfVBj5W8W?U4Cl(#76Y=VSWMU
zViFTeqoHeVcnIU-#^_BeIyDE^Q~edZ5l{dS8~YjmRdyaV_`Y;KcP&YIe?Q)70~>eU
z;iM4`7&l3yU9iLoR?|peRHpAO-(#RfNvqL&TsJRfv>VJT25&_#JPc3Lb?e$hUh;dGk>5XWDYc<|U)B;LY%?*kX{XGF{
z(qPzQT@tg<&(4{sNqWqiiZ4;?0o)}
zLvYTuqR>7hXSib?j*3t<`Ec2AL};k#Dqqf@5Z8ne7n~r1(
zT-b1tP8nVG&Wg+u*aXY8{I;lt<+&>0n{mn^U`*1xW`AMdh?3mY4k@IRe4UUlDK1%xxPJZgYZ|1(X3
zJyab#nHGPR7Wu(+CP|0PsP^XT{t$p+xW!=)%_zm?#Q1^-+$Viqz2?i-RHaJKe(a&|
z4~faf^v?3&pfy%yMJfzGwA0O(vDk(x}ej7jvbdf``9b
z@$xDy*H~&TguJb9RunWWsH=dy%Ba1TiG6#QUd)a>s&bn7_nWc={H7uRZPu#kHxtJk
zyuK^XW^8j^kB?LLd607L@QRvn^q|4O&6pWuqmD<+vvpeL=X~U15vVQ$XwiCa}w_{yxN8EAbfYsUzNfd-le?nR;DHAi#MlCRIFf-sy*f|pVX$dm?q=w?J
z4>dy8D}og`X>FITb<97kS4AB+wUyFURZ$wdIrC4y@tBs3z2>DZL)JA_(TM1%!F`nf
z;Pyq-F&i(Y&+<%ZB_8j^pqO=L1(&mYa0y#h&#%7%03B~$s02r6gFaIdRpNea2J|PM
zrVf%-&%Fn0(|pqw&O_VJemQB-po^!M=XPH=2NCJJE?LQv;PX0as0x3WI2K+LAUb4F
zEkmC{$?G-LQ6-9H%gPt266wpJn;Rw|x^WcS>Pf@w{~2X~w}b_~T8GZOsEP4H^RiML
z5XB)$0IL2_Oq>xHHld?as+5@<04Zn3Q%R(dDNKZyJ0>i1F>kPeVl{wO8+o(t(W9ds
zmnvZ#Tl6Fmfw)oQx(Vb`$;^wg0Ef12$=O*#t_KiELTTU$xpeXZfp?p1H08N{xkY-O
zLL~+q0ohettg
z05%eUyEsdZs_&}v>FL3~^AeAt1%z3MsS#t9%iN50xrmsS)Y2p>BMU8m{@=FTPf*ar
zO?W&IuDbP_K8tG02ynyp^_BGd=;O+rsl(AXeG_y(MMg%=OYQEisQ7@eyN=8@13AY7
zOQXb7(?N2vWgQPGiR=XOen5X}0e^`afUd32WI?y`Y)(R^UrS@Ae+dmQ*Gcjd)+17F
z@Q;RFyqbql!a`s=q@dOmX~4ovTw$*gLX#e)c!w3qg?x9m;;Q^qH^{qV7tz2YNmKww
z@1NOc3Wowosz)F?~A{l#_LI#!toM)a=Hy%goWT-(jI?^hW>_V$RK%ImK
zsa1P|r|UmjNQQ_upnYK}{ts;4XkxkM1wWycrcjkuST-#MVKmlG{Y_F2lTIaYMk~j&
z8HjG7m&d%J@y-1L+%Dyw?QibFG}IZcyTDZG#S@N>2#Vw^hk2^&NMjr*U|LP;N>h^z
zC9d|XsDiA7@li6fJVqw>n7*ZC0JI!FDU&*muwu0!`MOP#pc9?27T-kf0O%uhBk^BK
z_IC&VoaH?Vus5%SUgbSmj`ZPZilSXjN-3jE@&St8`$InJxbqD1;@uDp>-Mu^0KWm~
z1KYlqCv)uO3iZlE_PAL_*m@9X`+e{V`2`d?CVu=x!g80S`kiKDF*s%V~v&_UL&V_7`y3WBG8;ZBY
z4e&y+ypU0X&DjykhM;lQ
z!z6%HiH`&&Fklt_)hTo^uyNw9fTFk`l6!z*;((Mti-k#u9!k0D31_QA?zjih1)5b7I=2N^dU1hTdgF-~x{AO&
zQ0?{Y2Y!$ri_9khIRaPmWEP7|o3g8G4d4hs%T2-;9&5>CKUm!&N7@%Z!bk1)P4zd2
zcwuSTEL7{47&WgCDH1xQr*Eo}Cy*LK^VTy7nP#So@(3*gIBJnOyk}F3o}JlVGF8czf?ZR#yP8$)N@T8;)hf%%#$G`o
zChB`;K`>HdMP>=NFI}^L$rd`#IgTv2&v^|zW-{)RNT*)-p*T|bBrPm?^@;GLxF~U#
z%|)%RbK&g)*!$oveZe>3ilyo2v(@A{i%Be>=0klB37FF4o#%2lBYoyB`}Fv4H&J7T
zFO_>=TdVOoo)68fyK+{S!ewgXAL5WX)~-M}esxFwla|^fxIS3c~L-
zO^t|(PXv72cEu-JxXt*u*SDwy<##z=ZK@zrnk*lH)>0^}XCo~iO$A>e>rq8$@L+~IRvm8^DV|Sd^h05KWI}pxC-mqj^b%Y+Go6%C
z3g)6WDfO)>`jO(nln&kghB~CVBBFLVdSYn%>Rt(MVotOaGkBq;W=pPDpFK8>TF)W=+`~J_p1H=hdBn4c8#1l}
z5T0FntE=ojxk~x_7pIVvb{C#$IaPaLN5k&TRj}Eos!9xkmCBs6k^*4q{a!V@Zujor
z$vT@`X~o@Zez6b!mJs|Y?NgLhH{ZJktoJYdewhpw-41zsP0KWk*oN*eT7P$<*6TtB
zEoQ_h?qOX)vrod=rtw#6^H`t
z+@1(B1L3l@k$J*bO5=64D0;Yvwo1g(eywFBR=^Em}h*ALzRmK%J`JVI2H5
zkKY>fF}m;yI+Cb5BpR-8(Dj7@;L4y<&;F#7Pks2S<=65S(=5cz+++mk$T
z2ALtd8s_ztbHC3Iu68ZyWiXPc06d@!@Kzmq+1=ifRoH&-A)|j1ym!I?1<_HoS~Huw
z^3U;&y~Dd8=w$astVs0Dprb1T^wTaME@|9-Gc&&~VGnv_vD;&Jx)BC_pbGe7%5~r8
z@*6kJ$AL)IdxHn)!d)L!_ZPLIT?!E(tIP+>1suFYI~MfURTWoM?=FDYjGG0yjmajv
zxj(ih`Hm*UV>&yX7QH_lHJ-{9vGS2g&%MzzGMgK?XO#u=VEG>Ckx@O}&tS%0EmNvEyx7Dxz+aHA7$_=HHB#B%qgn<}Y3KC76Eamcc3K#ihNoAcM>4X=Pj7Ok_
zcQl+J(I2bbP2I@De)&h+nqkrQ*Ddn`oFZzbB2|dS(VXK*V9c4}|3?>WsT`Qb;kS-#vGFI9&PdgE%
z0j+A9Y{IDugTmO0(+FHSU=w4Z2tT6xR~x+#rC>-iK-V^YVf$DtH|WL0@L(eq%u`SH
z{99_Ra*~FPkpi93DCue(8Hwc7?!hpZdWOs+kuh(f$U=a5Y%bfz;4>CSCa$v3ABi#q
z?f8c?97q!&7k1+&k{hGp0Cq2}6FR^1!f4fjLL~`S@jA@CS6!TB(
z0-h$^Z0NS2yl&Ow#vX4t<1Na@#P)h*1^t%;??0iz_dF44KNTJjHWz6&7Yz$?lR_Ki
zU+s*;j{D&SvO|8YbVpZ4HJ1992|G>-t68zYTPL0xE>s-#MbG9Fb{rYOJ9?g^4r$Be
zsZd7W$(TjT@zh*LUEA1miz>r94!x0kZ=g_tae6>q*=O_+onsh+6X?dZ)BEqrP6^7M
z1?vmI1!$h@N!+Q6;tG*5Yv~Rj=D;{KBgX#%69<*O@_ZT}4jmcXD{gwHeqq8LI5+_a
zP23bAGQ$v~@m$54>cQ}Gt(#Fv?BxeSV$h|kC0aZtrV9ZeXif;wh@qC$W4gVg#ndGlZ75z?ALFf}sl-8=z4Wdn*te+tqM9iOoKM
zBWnTf8}X1rbc;);=mXSN+hv@5`6GiC|HIZfHV4{7?KZY;XJXs7ZQC~P*tRj@
z#I`x%#1q>S+c{6wsX8Cd`*w9zf9M~uSM9!f?boTCs2&FX;=DNC6MGM6`T;7Nst1Tt
z?Ul9F2Joo8Y&S)=C*24^n3zZKJCc=w-CubKyma4h5VfUM3WKJ_lNriMQggV$_`(t)07P0~
z4uN}$gbsd=0+%iIAo2d9qh`?4?v%7L>S64f|C*OKo!OJK>Se%Xj~i=H9wRN;eW
z4J(-xF3u84afizAR3?+xk6+f-AuD}Vd8(TRn2RlLi6?!{9JKw%rhUkHZmuFm%#IysRRjm*f47g8}u=?(@0NzpY5
z;jQWIrZOG0f^R*lBR%wtQ)yo2vQCSxc^bdQgPSUDF3)cKOhd!y
zJadF{d|1L~KgBNhpsgW;B9#!VWG^tym38(~0wpZdhL1woqm`>L@dm
zP7+KLLkFwt#=KnecbBUJfUDAX*%#lPP>?(1pLr*!D4p1-3S?1vlNU;4~OM
zx~;B6JAoTQFQk-_P5JMHy>xd~jMxWVGox!GV=wf-rY=06dxUJSZWh1!tQ)Zu_FR)!
z+8$gaCmfSg!O1AwAY{M0rPUqOU-i}t6tD%m%v6;Mw>NUCY__kA0M5MS6RQ)QFE{K8
z6)k~?Kk{s+Pd}iOL1fbY5d8jap+ZM%1&zuhAgETb5xRcngua4X8mD?A7LYY>tN1n`TEld6!5Vo9f|8u~P
zjdveGq*~x7YD!j)8BlmZk&7xYksyAv-D9E0;NPL}Ftzx|=@*6?tL{htn5Tp8pNl;g
znRd*fwcO(wezKG@Z=`Niu{}6A>+WQ(c_5H9_ir{4A?;I)>x@D2OMbFOTbn(P>VSBH
zqSo_js2%^J%3k&U>asI;)r%PZL7t+kkIZaDtC`Ym%Vp6)4A|>>4jV!9G<&FLA?X~?
z@B2hX$Cr(KZKyI=9(|C~)UqyDA7CVAA&WwZx_zQN
z+Sq6@y?#y%jmX1?fwn1mj2vGI-uatg1^s(xv0E_iqjGjkXyJ}!c>tCoMH0T>!Y3!K
zAM>D;Q8qT27|?Jq0P5uKmANi1%4pm+hDj}3Yp$wRJ@iDLvo33}m7u5FzY6`1kJD!Y
zqTM6&8r*vpU$4#(A#d*{Y9^i=j|I^)g9JWF&L7zdmiY18@0Abtc^-_BxqkR!#jMoU
z6OsWNOP`9(J@QmQZSa)h_7WSNPn6`>(@=gpf#M}I5KR=5XvLDk%(x*s@!QeO=HKlx
z+s&B}e8(C2a2sJFLCq1kVUGL2#mE=EqQbAj7|M|DhD*Pf3mP7usMl0Gy?49nd>P-`
zByrbAx3U_;f87JVZ`jDybt8Ibdkf=X^g?-33BRihLalR41+Eeazg6dNVuOjWij`j+
zsysX?-qon57JK70dN4xV+Y9n7vDt@9%MB
zXyWg~wxf|Nn7%-$ffAGd*}vsx{hz50Ru1O>tA;t!{pYmNh55bRFWS*9eA#L_T2YDl
z*>G7W5*cf93T7{i1rbxrZEG=+diZ182tbyY#IkrJ-M9t22xW-=@#kQC_&z1T>HKwj
z8GlFr+x_!+*eb4*N`)HYzK)om0A(bL$~y?=X!~)b8+UVDgT1Hs>mNQb%Z?s^saDT+
z_XwQFd$H|@AxjoMzzt8tJ(07&1VoM*>x!pR%vLQZ-!8k?_4$j9;(^l23e7J#g;AzZ
zgxLj&%C>~ulTFaZYGGrEqE>)@sQ>wH{}TWSr{Ac196xvn4<8sm4Nd#{
zy=H7{8P59AN2ta`CBu6IYFqo%)o>)$Tt6T3y|23icAKJ}6Dh-iCVt@XQN};>hYqSz
zJB*-Gbt=w`+KkL)>Dm%h?3gMT97e-y6|5e)9ep6Jq);cqmHE$dWxSU@YM9I
zkvTJe{sww
zTnPKbEJ=DIe$q>t5+_=YM?zC(`8E#Nvl7e)A(I8*o=Bi(x_J*IP0FX(SB=LrzL}_;
zyU;)#alaO*m#`t9>GQYaeTh#SwMT9opjeRsSa+3`aXYaWL)PUpgQZC8}<
z^VbHRHYO1?IyYasm};y)$x#7Gr+Q1KJ3@fYzbZJUw%m)XEq||RGGDCjGqV#y9;5hW
zi~7k6dM4NJ@2wVJ)S?ZCYQ1q2r$=2w@HY~fG({z{WOJc&!{JXE^}0XvzkU0h+IIPu
z6c++7)~9Zy7L!!8(q?eCkqr0@X*p^A!T+>9E9
zLA|6Sq-zJePP>b3Q6RZ{g+t#BPguO{K4-mWxt3o^S(Zn4BgXGyT%o*zyMT-NY!DoC
ze5?z9{Z$U!t9#A?e_+SPx`_#fiVc^xJYX$-TePr+zC~_}D6W8fY{!~CmcxwHgEB|@
zlhGxq{v5xug%VvE0rUxFifa#b
zq_^@Iy1-A|mG*E~tn+r(2{wU;2OR9_GvJ*>sjjIbbHixjff|
z>XK;k&~$z(T5g(O_SAzfx{)9z4K2E`_o98;xp2h}!eC{NcBZKd9vs4~#<_6XE%cDK
zxzJM?>x6+X@{Pa@mfPC&=hMt!@-VEC!%}|)c~Q&p2IZJ@Y@-tf%=+PZx8>}Dhy+RD
zJ!15FMlq!+jG3*DBs2smP2R<0jTh3<5}|>de1{lG0>uwMQdfIfsoItD2es;%7}6i2
za}fuJ8fC5ICeCej1fDB<$Yhwrqbb%YWu4c@_}ouDtBAB;WC
z#47rpz6}?0Aj<1?cgq;<$ZjBMqdJFEFv#mM7V3TiA_4D!-r$@MQ~i*(vNGBF&Ot@<
ztuF!5wh?%>n@tMyFb_rD8s3BfgYeXZLx&)`W1qo1t5`7cN|Hti?r@n37x`g-nJ&Im
zkrC=7VqM4x$5_wd^k=0RoS)%|zVF1V^>LKI_y+MGW)e$qq1A6ISC1drl6qt{+vUP|
z=n-%2{d}8YJT>HC1p^I#!H{`LKhZWYo^>A`D^G>Vb!u_M~ge1s$2~(1L=0G
z)deP?E)kLOMcQoPW;na#`E?o#?29BXF#dUZtr+twFuN>G==`vhO$~!~At!AV^kM!%
zHSj44Y%s}AXg1`#t@rLJ!l&Xngjay)Um<{CPx;~(K=N*E^2|Ty@R-P|khw8{13R?-
zUx!6R#UO!0w-)NRCMNRqm5Veg>}yr(phk>LDM4I`{Em6&Bz56~*oXdTmBG)edx>JF
zV+o|r6M3OZUtt5}N@!|w4-Y%<{Te^5>E}(Fdg6HkhXr_e
zv`SY^ic8x039AxzS7XaOnrRg!h1Rj7SBN@O%Cdt;=Tl4dY+P%8ZnE=6Alh$>tYzib
z!yDBf2cT3K7YL>ZE~M&qE!piemr;P{XW(Vgk}KcRk7b@#^Gix3TYY~QDtGRdcO?ZG
zQQbvrLh%*HSnh6@uo^ro^Xg$_&`J&U$Hvg?V6+eHo#~Wz*AYc((4k~ugjg}t;9i=+
zD+L|i*H(i2@z2ZcH)EA_x7c!1M$ZnE>7gzj8-}%&1{;@sWlm8r3x;>|LRSEkIw)i%
zt=FI+5vi|X;-4j&VE)i{m%?}VSFH5i3D6r#!<`YLWERtWQ
ziM_exXibBHG^F;wbhmW7-t?$pmv%(uoX4iQYTK1P4pQi8Gkn?OOS(89+RrV90j%AR
z<$ic(yaeq5t$Mm82+YI$Uz$f|iu}Zg}?w>Om#Bc%!V1?-EfB
zUOSq?sLb_zcg1&PUVF4r>bV;7QPG~fx4lxg_HdSumVi~9$J=g_qzXS1OWW_3QEUSSfhA1rHOiZ^YUv(|a4AQ_7jJBwQlyM|Y;@bXb*|4o6e}i)zu!v5s6K
z$L092=Dc2xrWP_4e!L*ZN&Hl1gpgybN1vua!=Vb16Q@>Uj6F2aWz_Om1oC_PKTfg)
zK6bMLgupLSy|D9cT6CGTa%uKMn;=cM3VJSW#c612hWEx*B-w;*;z^;$ey70Gt0+cu
z4KG$GC0g3}Xh$qGcl5AWDR%wyZLIvyQgLOZw^*wcsNMGUTIjLynv7((a^Gpw-BdRV
zj%qBr37x_P4
zB8c?0;nu}pt;F*T!I0uz5!r;Hp8KN~V#I5g#@EI8zFK2_L%kXfBexMtne@MUZXZ*j
zl>ZrnBRt4{SZh{$qNuQa8U7U(R1Czp^d)*%{|w?g(-%uZ3sx_P4E>br)vtZWD5vXP
zbw8T!H^^3e)?p|326iS*iB>o*kxu%<*?n;8#G=`WT)=k`nP%=6L)Eb)^`7RaLo10mJ-N7lY~~2@_b)LO|QJ5J0pRu?u0A=BAh@
z(KubKPG+nNM5iJAMR&5lR&wFx3oXRo>hg`2E$Fj$SIL=F^a?!IIxT;PU4;otkwIcb
z-7^=>#-x@rR+CM}pe^Q+Ql-qC3M3{oA$~b~Hc>{K85U&Tdd(iXZq993@w?yGvSQ5|
zP+r6QnyO%%RozNkXL9?9C6DmoTB@Q(9dC=%;rZFJj0b1D%K3
zxjYtU&MkCif>Fzsk~b>4WsA}v-SYQZur~uQIaEF&%n$MHpwqETB-UnE9PEaw`&^0b
zn^3SfnCAa2wcfQYPoq!k2vhp0qDCmgO_1
zv4#);>T|mti;BETX75dH0+w^25a`t$>J#7UyJA;dZm1RiaZVf7S7-OzpO0#S53}Ye
zBaw!N8aO$TzzI*`SNF;~<&T9nZm%hJ%F(_$8Op=Hi$7vCJHiR{IUe`_Zq{|yW$kewAq*<|W~uR4kr?kt^TM4>hV
z>Bnj8>7UDAjUt14L&Tb)(tho|z;w}8f6O)@N%fQ-lrjpD(F(n~RA}_1PJmbOf4<{U+lJZ_H
znq|gSg~_nV>9^w(OvNceCHST$8!cV|?1)2aJ`UUzlX)d8Z0+9xeXgjbWL64%0d(s|
z!y(+=Xb7N;#Q{GFXs{+V0Akde#bf16u3a8t^S_?RiX#Iu6gOHe(EfgVN~DXcd=9eW
z>$SO)f9T`u)|t=d))n9QAwwOyW)^QIz^M^DL2RsQ4}(JU(SblIunrkWejHPZT`M+JJ&;k{_+3X7CNT|R
z9PaN>kMDlrlxt`-y^d%ZlIU85)~a11st}$h+{`@lO=#svV#$1JHbP-SPgSzbP~MN~
zpR?OJd}-lW5!6S1u~$va2Q!I4o2>iaj&!;N8^$dl>gw!YOgWRfOIHC8#%~5mZCgGp
z8v_Q;z_vwVgWj_-B%56~{nZU6B#1q^O6Mys1d8FPh=^hXu}>RVu_M{{oJA!4YzXqb
zDy3d+?$=$+7_ltsdiN6#PKFR(q4BdR!q^|o^YJ_&8tUfKL2Qw%8j%5jutfwj$x>2j
zW?FZZ!dul$JFQ|wRg^lqjos1r@C6&HCJ%_tO}(8dw^IDm`a0J!++i)~M@W-E5e=yF
ztYn@M;Yu2_Furg|6cn6mxw(@`_oYLpws-d1rt83VkUoaC1by|1l|DZOLfRFK6CVA1ehuM3D}5>ncY
z70(-gxpNUU3Wp&T^Y8Sq?CU30k+#_Q`Fa5H1VY!?OYU>UHffId+oYXA@{=9(o3jE*
zn0{E`_%->a1a@`~mAUKSqVnlI#~Ea*cEq~rV3XC$MlMLxL5(uCM%Cxi)^r5o-!J(hJ_?dY%+ejDG$RLq+r4G}6<
zhQqBh0=9b*oSG$e6Us;a6NI)TjdRC+6&fo&@MdX~$kPDCj_OAU>3vT@TOO`6mA>FldaoP*7slfHUwUoaskh3sz$M;B
z!p7L+mQnE#ll};vTOGv#o(RDbC|CzW4y5h?5?ozIx2Yx>X3F{c>stZaNfKEn`Q_-Z
z#mJs0wvuc%x2w!-Z
    >Z)l*Z1R(wW)Si9dk0bNru^aEY%UXuNZ7s(pIc!IqcaV
zqBh;!JB-^>-92ZM%Yms}R&oye!0D4brHaVqV|JYD#djnW!27*-q?evh%w|o=R?Nx)
z`5A%T5st3&Q%ye!#x8YK6)rV<^p$ctg?w@yfUH~s3eA-_-CHu>bG9#WZz0O15mMF4Ra#{=
z9{dp2)BKwo62W+fgrYDw@OtI2R%@f$w>CFtv3=V`I7b$Lj%IHXe$$;XsG0ex$Uay5
zUR%6OA{dJk1}HMV*p6<}2zn|sG8lqfdPLxqjli7RyIlnQL?Ubh%2V?WV^~R&Jlhna
z7)nZLP_pP<7CGvb9QIX9~xmKU3g&z>6wqPKmnNUI&vAO8#tlH?P?&X3=fC_P9^PhSARv#S-
zVJTdTrMG7)&sCrX;GeM5%-ChTgTd32ElVAuAzv^;+(yl#?{d7GY9jf$_*;UhW6m*V3o0}U^A<<4>kH&E
zP>!5qmTKAIlxazH%#x^kl^(QOz9T8YmZ3M{2l%xf9!2d6z?gYD2N@s7c%{0)l^wp5l&TjlS
zDS?Rplcj_|P9N*imfiSg+ptD=(18-gOVp^UPV!_NMI|f@`6*9>F?}4d@BkRCg-GzU
zHc>ykt#s8RK)QY!@CuQHWn+ru7BMsCoR_4Suq$m2@MhJpfEBITK&fjQV{$MUi@P%#lM4(Ov+#Uj24TgH?9!7MDz&oi-eID6M8O(PfxO=G<3dSAw|>K)*MrTGBxI;r
zTLiTk=MT=ww;_z_qYDa5C!~vb{%P}DmMRjNEqR3ipyG9a%nn6vwy5z!FsV@UQLVgMJ
zv6DTvgq|&fgk>k&_EJl7nG{iamJ34ie~8P0le9CQ97H#BCwm0ba%#(-@X~wprc`s$
z+_j&%ew?zT9v~s|E5?1sH6^>80ub%o5v`CPPoVT|Jl$=aw1I0U_=cNH*sl_uG<^DB
z(NOhx+HOrD#uVIMR^`yXKwHcFFIOD%lOfQARE_DB^GDnZf2BPz>?-fJOQi}M=we8~
z-7+_(*`mt*!D0D?t1L_ec+fCZW<+D2J#ENZ7HJ;xv96L=^OAWI
z)&l1>M2)uQNFr_7%m;44{lPx_*`5|?4&u;wm%Z|OIiW(~!XOe-Ud%Kmm?R5iu24E6
zbxhXA4g^yL{}#3-c^I(q7BX5Jr0H7^B9F8kPEP+EOg4~U?q^$oNoO@WiDO5XEccoX
z*!@Y$Xv*mQo5SKEP#&xqO)bX~8JTAqgH9P_(SiWJ&)0LUc_(z+X`$S+s#sRFP#Z
zL)T#m=eMjN1@g{^3!m&HIMx6c(vrrruT@x2k1F`2Rc)g*A9k*O$|B{6Y&!;3nL#4_v#ac1(a
zs&Ala;D`21S#R`43U+rDi#qVNa@u}W54o3U8)L+bYiIkIUi>%=JH)c`Y>Dk42f#f=BJe)NMEwu29nEG-P#a@|666;_lsS00-1W8S;D+CF!Z3>+Zf
z2(3v8D|LYP7k>BZZzT~vm@slWW^54($(tBR?XUuZW9zGqik=dhFG_Hxp8)`AACF)%Dni7dF
z+lwG{$o_0b*KJ4jfysq)prFU93UoQ+)O?v)9sJW+IH%|1XHSyP-V~5?9crPZ{bsJ<%ofln2hELZWSwt5Uc2%pE$H>V!Hx;14!Gf()2t?E_UrmTc|ev1lhay
zyhdevH7-fpEdcETMb#;CZZRDmLf4KrQ)$k3p@ep2Wwo@IL9?lq}5-|9TN
z#B{1hVlpa@-%ztdadASsxAYDM{hqfg4ZGTR)_eH1dSe;2^oD;^#eCOZ#a)H+;4+%G?znz*;mtf9wEiaU%{nF{qTT9ZJn4j
zq4mJZK{wIdBFmwF>91dko
z!pg<@e|w#}dQSfZ3j14)=tsN$h84J
ziSCrKX=BjGMX=vZWzxL88FxB=xVPoZul=!FG6H(|{XR3!bYvH@#{aWr;8NOv)t|=A
zWTA+UPta@TI2T)Hj$Q<`WZhHf;&+tJ&wPm9a8CmDx9c^C+i4mGy9OMAtyRqyjX6H}uH4jr7ChM*17b|X98o88p=4JhB9Lt^~a=l#UUnfWXRFEk#HxNg!
zS{9I9tq9qFo>rDlG9VuL*)SQZT(h}+H1^4FR$9ESV);}D>8LP!UTuT+x`v?j)#pz-
z7gT_@iOiQQ2hXoiJ352-XMAmNPpmble%d=$Ce}ar_8m%`9dK9i%QaicX?mckmFs*n
zgJv3zeXD$^9G1<4t-aq%Q_$DSbl!!_P)1Cg%dN@zm9J#78$tf^%rpz4!SJb#>6>p(Nu`PdNt8$5_-?g0y
z@1)Y*ZzJS>9+F<0S~@8=W#E=j4{Fg`v6Ng>8{qY>a6pq@P!`@waV1IiZb-|M5mPSS
zG;5yG1c9zMsT2QYiCJO(aD}(yG0rt610l!N`F=MBpOo2=*BZBX0F|cd_a@vop$UAk
zMd10v
z_ef_r4w3o?`-NNV9t9-NrkRFO5@@_8Hi6C7vIGmuO_@S!p-g}UU=(nb58^X1LUWho
zTz0!I*Ho}y{=we$YDD@2-AdO_nLkRgOxCD}VL}T!ijpm>K3)MG2HuQbtPV&A*!VO@
z2pdPiB(oY5_f|EmbPXx>GNL5%vu;PTQ1dKbpcI8Q@sVKC>}^J3o$!O{oA21A@nWNG
zs}V%??TW@S;2nLPzk)uy2$EHf>s@+vsGmd|mWbW8lo{pPYo)I(k9ptun3S)cB|i4?
zo%sdH)#>GHR?31P@#^h?2mue_)BT`dSC&e!j^i)#ED|IlJ{TcE{7@hnPm^T#TAKD;
z$D6jod~;igw6khM)@W$%oOO!d{5~?GBtTOpNGM%e&K-p8WOme
zBV`e$6TC~(4z7+v+KWV*$8a_F-z3#d2}@}Cfkc#&!ksNB-Y^IyLje$rn@%2M)30n9
z#Cdsv4u#LNsmx*fU_@(R{(nF^C`fMmJJElk`hgh)r+@T@F
z45vqD+jKu-w__$?7yOSlG&Ufmt?rl(+&rwUwr2;Ko;}Q_Jk_w3h4K#y
zhCNZ)Nqj0J}uo3PNfK!o&cnBM&QKTaQ)DzUcgwN
z)bCPgKni_PHW>G5?fl<6|4og-R)7l0cNe88t&;ud32<?_zAJ1_bn|X!j$a2JMt*DI+26Fl(CCmK!LTTPwn=Jq(pGoQlIQMTJLbg
z;M;iCCn=eS<)y%&q|1;-5m%#4*{)fyrbTC?8u^1iYvzxW9h?PH*V2A}+qXP>l9v1Q
z`e$-E7%e;Io@^(Edv&SU)4r+*Gx+_RF*HhmJ9|=4B5neyS>C>F7%u(GswmI&-KuPXWhVm`^9xP
zb^WchO>lxtFK}{Iq3vmAk5^haxI~on3YP;BFway59WA;u##d~oS1UY`eQK6rO1f@d
zywz_g!<3)hH+IE(PcUeor)@(=$JV!g=>8i#fmeO+%JGJlp7hw_=tdwnGEaM50frZ<
z%%5kv*7<{RWlTY6qjORkB6)o^Bz^5zSxlNEIveh_Z|8iA6Q3Cbe6Y<^c$nAW
z%6G5g+yGRe-7&Y#5huH4Sb7ba{8ReESAAeBzm9i@zhv$ZY$)wBxZ9TlzxLy+`I8NW
z0MS9qeu^)NF?^bdgmw_^*++n>O=F7$n0P{#k%kG>mhLhDU1)yBj;XHP#NJ?1%SFCB
z;}CHeRp|KmE8;EeEy^psXXb?XU42{SX`+?K>C|V~KIQFe
z0tzDE4Tgh&s|5P#j${MrHrSjr}2D(8g2!tda6GeuSNh$k%`S7%B2M)TpBYAwOUAebn4|!^ZSIkp`c7!5YO3<;R2F0;E|3ElP9r}
z3m=^u5Ou~|##a(}YeBmdbtMBS63Y`DTPMSD6q=ZOGE)44_GIJ@pJD7EksSeDaw?$o
zZ=w7?u$LY4f7`xSM^h$uM-$?y9!8J$^Xq@%FxoUeb~PyM?d-I^QdeAM1G(VFP^F~ALbmV
zO$uuxJ9Fv4=?U*znUT2ufX$lt2V!)5aYAXisOqIF9gdw$Z-uUSE=>wbsRlki+#Ma(
zY_<{%H(BW=`USH@>1W}8q&LWUdXCwOwY<9!K+swDU+ZoZ$#nw&HXNi&ZeEYAbh$w`
ziueYAbjt19nMdZGR6jgO>~{N1)r|WT$E5Y&aoBTR09Tr+R_V1>ST8!;J3)W^zPnRG
zjzdN(f>u_1q&ny7Q;(yuE@V*fiLIJ!`tiT8Fbm87$HL8T&dso(Y|VAPim;#@&7Yx~
zut;40Z-@;GCntNVNjxpE;fT+L60lRBHI7Byk+*lFi393tvjD^UkxfOL|#m(-F{As+k-4IbTZA0rArS}6=_UN*ZFPNuk0
z6!fKNh5Tp}vKT=fCQXf^1=~o5H64cDMVf6jpjDPqX$Bka^R)Z&n2ZcJTuq>cW!0j6^<4
zmiomZ7^Ms$?7;A3D=n+34D(2MECL!eX
z>}^%wVeu_Db|m@For~Dmk&B;lW9k^ivRexYrPhMwu9U@)i*+xnSuS~xGFyfwYNe}4
zsRti*!wF%YrSJnmz+w?rOi#xiCW;gqA8cWIFhUr|t$}nvY-JfGiTmgJ<)uo$(r4^w
zF685%QxZ}j*ih~J!`(+Zk!a8DZ~yCqiR5Q>31DgP?apE}dpwn>j_~)f-ijFe#A>`b
z@pqNRaD8*?VhL3&Jnv0LOq70|>uT{k?pt!rg_qqF*TLu4cp;}=rh>5`?+v#Q@creC
z&svj2V=LMa
zC1$YgYl!_7l4|%3_OY2kW+=6CNU3!_r`F*M9o`kll|NeTr8m5CZErV@N#wp#?RD5;
z)GtQqYC$ZTLieuidN?WQs?K<6=KERQJbL?OF9V&m*nG0opKU;Y#w(rrAh&gBWZe6qO(#8&c9P#Bj54DvDC3B+XcKm&
zf($W$hSj49e|rlW4a^2FtKEw+8EoKqgrC1r#Wts|txXmQT>w``PfEAoEWGT<345Jy
z=V(L4GIdymacODa;pESuI5)(5kVtx8|zMtO8J_DYw
zU0?Uc`@TE)bH3wMwk{5PfbJLf1>$e7ThB)Czuz~RT1?yjo^RZ_th9Kg@+ErR;&>C@
zPT|m0Q@uFu1o+@FBcDp)jgsBfR;e!_IQk%C9T4^h6;=l>RITr|dbo)nNi`oF=pF#~
zI|ro|6IV0rqBF(`yidy}+ck&Qm38_&7Iy{$NgIasuy{NJ*)l}B35^*AZH9d^IAe4y
zGIYm#uF`YaDg|wkeQ2BJDM}v2g$RWx9W#HORCxLwRvO8DVn?^DBp9y!9qF!r5Nj+n
zDULrRQV5?LiUZK`%SWfmU}M0un?nI5cuk1dUz^1Jh6~
z47*>b&)~E=^_El24xuy{1HJU$A+s@{+aN@}0M&+A$k|2Utoo$8
zK7d9RqeY&vY(azE^pD_*y*({|UX`66Yx|M!5T#~sbyGteuicPF`6KZUJ_ef&v^&AyEY4N
zZAquetGac>_J79K)uH}ACxUUdFEOGxnc=<6>)OOk=S4uX^dfmN|fOwS!rKz9NBf
z@nrT8Lz00rb8u$v5kq1@u<&p*XYTPqPy=2b+G*EqoLiJz2wVTH{|@F61QvIg7;cnz{8QncqP#XvLuL+?
z3m7nP9TUbo6fZz2sRaFj1xkqO#H>`xiH9*LCq5s3
zac~f5n=lo=7Bt8Rs4)mT6wWz7cYX$*_Sk~x@u1Ub9Q@e3wMAO+JuG3kGoYLeUPMxD
zQ3L}bcmzU?e~!=p#R(E(@j@u>d%+kJ23REb$PRD_>(eS@Xc*xKVwSjxe)VrQBzk}Sh5*xWhFSrL
zi*w`#XkKGyI}xC$VbA#WyV>nKU2bAySz~Gp@%LRn*HeG3!=`ZE*LUnEu!ajt?9*&5
z!oOv&|1Af7rwaB8dg&po%vS;Ad?bDpDmNKRX^{k_Y6Znc8bw_b$L}4qbq|A6z!9O~1?Z{p#H4Q*6==tk&Z
zLQww@Q4oA3C42yZXf^Bx77h%|$0XDV`Qcqa6vUB{83N7?w8Kv<6chKA!(cehHvO-kPghR_%I)kO<4eiI7sgZc^}%j`b(rC3lBh{FF)>2hM|v{
zC5DIvem+AF-r;`ZM-P42f5pFe8_VKrc~uCJsbq~jF@dY#gb7f7otl7k1LUBfI3bsh
z!~#snOhNcBR3J=o2yAYV@;B@2j$Jdn&^Shq@{!4>(T|$Np
zhl+s6Dr(SINJAjcH>62;_eVOIW_fjdZtW*v#AB>uKn#7Jh+L!DE_9QO^}+;maQH<<
zW-r>g<-lJ|g0JP5G+ekJI2Bw#67^|R$9xQ64#As!QPqOP{mQJ4|7%osJ7g;S9m0Gk
z2#^rRW$qA@`8R;yyjPQGFjVP(k0qA0D27JKw?H@e}=m5!Ozk
zdJ*Ek{PNlS>{B(-`(pZdsR0EE4ix5#d&Q0VPl4j*YbNd!Wjg;C&RkNZu0+<9#;ld1
z^tGn`5~CCj65n0;Y`@`n&c56e@&LOGMw*H
zkGjxf9o-N=J&XBIF(I`r!t$TVlIYD@;e~d#T*|Y+Vc|^y;?{}-@n#v=uLc@V@zS;RD={F=
zMrvcVg_5k;Z&s6zs!nh(4MM|5KCbz6mskABa>)(&K~x!_=rnPHlGX^mN+n*0c?K=ea)-Xnj)Oq#_ytnInHTUtTHZCRe;`C^IM{
z5QRB^HT-eXU?qA)5S#P>Vk*EobW#}LMA(sG6I)nYF*I-c+D)|r@IVs$Wa{|-R$cmI
zEx+Bc5fzW^UEr>~Enx)tgU1~b^|{bfns2~h_u6O&7-g2%EIg~7rrt<(NlQ?m5Cgj%WkU+)$g}Nkg>|D;B)fNB1Vxym+CLKHqr^LS-2+Mhl=jzKbu)9M`|6lR
z@D>S1oB2hX=@D3dr=gF3CosgJsNHHXJwox*>^uUNf?w@CCsxsrXD&h@Ts^jrJ!j9m
zBo19l26#DI>EJt_?{*KYOMi(nWMFn$d?P`=p%hI>phQfFw{
z!>MB)e&-mv(xo#3Ci<>7`NLCafxc^=VO}8ba2-bYj9%07XwtF})BUvEC3xfMpHZ_i
zC1Q!QjX$a@b+lsBM6M%LHBGn2cbTih()bJ8;2fiMx(Q%|&xJ}g_3StodxO*ml+0Q!
zDxNLtZ;kB~`YgRsJ5{~Z(tn`FgDX!fHjNPzUa3l)oDp6eP7SEX~uBfrGky8_((^A6m5Jt2qO?WZc
zSaTKtWi#vn1O*%~)oD-Mv9i6nQ%2dU(D`2_4_kSza}L2;C}Qc5Dh$(>jq%Pz#}9v)
zEnQo(4a6DJM}a)nP0(0J7T?CHjFQ(-^Iy+-s&mAhT=>q9zU@bs3-(MWd0N;qE;onT
z_Ig$w5P@sqp1{u@1&^#8PvW6{hDLwW6?u%b8M12#(cmu{!d}&7@SG;
zX7SjzIkD|bY)@?4_B*jAwr$&X-q@MgwmF;s*4FOF{djMm>aObB)m``L^ZX9Do)nL-
zK7MLCP%w_LbVf;c5+Y}Jk6dyCU3L0{p9H>d6=^$sS6e@
zcBUW=)TuD=(>RCkKD+T@7RVEisW8Km#jDoe)4oTEc?%H==&POqIqkPT5VCjffJ@T*
zXy|=}>&{}5%3@{ky(*?@vr$f<&gKbF5EX=8@C>d~ZN2@*dTC`J#RaMir)p;TfuAmdgD)ZG{T9HnfdIHRzEHlw9S
z+yH-?;ryX3|8V_<=-(f5;ZR=S
z1BpI+z=-BnAjaQ(JrjG5_j#9Urb*>mD}YP=XGWu?{W}tvX3H~IkKqg-M0kFbx?pj`
z$HYrXW2c9BtXlbEHMIAu=urd+=PRn|FxN^|I<;^4!eN;fCs`rFF+}02iHVI`1})4W
zbEOfCf2
zzL;lQhe*sn`}UnLU6^GG2vC+jfbd-*H5n3l^zo3B`g2b+aRIoO(qOZoO=)!Y+*WG)!*Xw}??+0$4{sWfy#Xr7
zSfX;rfdPAvW`J3liPxk!6aPvrv1a4X$sVmL?V}Pq^y^i*g@o5N=;K8ZUVuaTs6-Vro^UJg_CqM*P&r_mT*MU0
z={YbtY!eRpQZ~dF+cw|b+TuNHu`Lc+;JxNtZQaQpg@L;CN#tNWK*t~ZAd#n+e#iU_
zMF8S126kE430qLHt{5;U@Q>9Edzm;kqXH3d?J(G0eEUf$WIoTJA;g=>Cm;)QV8emG_bt}_mubq=V
z<=97T>j9n@6N!rh2QJxm%3|12NAxOhmc>gLx8eu%B74;38Qc6^(jSldQ#hI
zmPV1WaB+IrWR5tkay6IM`J{RFUH-#XEUo=>6Z$e<99M=RTCoWcNeVz#GnyMu(v$HL
z#`U-9Sd@o~8{rCF=;nn4RTMkz;HRv)>3zS7%=re$C%FuDW$I*wJ6>rE$X=o(CM
z<&*{>?faKik|_JaV+X(~z?#_Mg>N>Qb|8*hVWUG17?eT7SJINFARu<03$OD0Na*c!
z;=klaS}GJ2%o6!fBZg~C56<=@VBn2FnAkz~jgrU%;vV)edYC?D&>uckJcwM7&UtUG
zi!_*BsaP@U6tdx19sG|
zA`4J6CZY)c-nEyaEVw}e?j$1gmPo1XHvl$e@&+=1%_1$ye#U7T3!hk%>xyU9%Vf$DrmUnmK^V4
z3aXl$poL<7YfHBBt^zgO6B`UKKV=w*R5|I|MLFph92XIOw0`UBJn`{`szFqtl}v}Zg#!3G)2Pqen<4c_)`sF%{+N=Q9Yk@6cdm2r-ll8`L+2j^7*cTZN>z*VNf7h8cF{w&#Q`)=73IERS80NX
z(%4KXTJ*&!Ka3APv!)J2YXw9|i0GpHDg4Txcikw->~8y3BWx$|4B7B_TnU-3#SGFl
zqT_g@T0e?n`B>J2QdMPT%uEkTNps*|o24{DEdkI!jpM%Pd-&J%%ZL&j&-D`k;z?yh
zr*8k2&|DO{M0)ueB2g1>hl4tP?s}+8UF9K7#B|1<3AE@yk$%d0IfgtNM
zHCH~a4v{t4g9UoAaN~TUM@>Evoy$Y_MmdP&`&A-sMd@Knt)onOtb`_V*?DP#cQvp9e
zaCl`Plv}eB&&WOFciE+WJ~DfZbfTkf@H?^Tm^``0=&i5zv}(>t^>9mJ(rYVcdNE?>
zFRTt!MSgxf7MB@P1>908L>{7fR({gXjF=VtwTkfq#b)_iJM+qG*@GhOj=u>{x!oG_
zj-7HD&FJt50}(2Tbt(Ic^{Lv(zQbNt!ObvMQ+}sh9-0C&oh&|EjF*BAZ+fvZc>G=&
znw;p+oX$Ix|ArvB^*ahIuUu$;?I1$s3K0Z;$%oWC#rln;NbfU*Ze->|7z;rT
znSU`lBi=t&7uqH(7#U6Cz9v+7+^l#}Tm7!mtVilP1fRt@vR$bQa0kd@#UJ<5^zu`i
z$gR%#&9yL~<3k%@CN{JcW+~!#X`Cb
z;()mS6|zQb8GK;aj0I3UVE;51Nx=9qvOHUCy@-(_%7fK-5pFzd6TGOgxMn(4K?+m>pNHJd}5V-LexiKAUDGXr-%p~7pFoBvRMuw_e^`)F3QNSp<`tph(w5o`ifhOBjnF;%@1
ze2uBb7CD2}tto(e|8EQ^MnZ~g1+^@kX>Ut{v7T~sSed=g9-FD779Jwd&G$nJn!)-^
zroM2xbL%pLrU!G;$0uMtrQh;!sBhw=^~|!?1qG!3vZ;m#XDsUe1Q%vS=I!Bfx5N0@
z#fE$3Q`)f**_$RJ=F?3Dz>OOD3B3-%ljSV>*c#a6iU2@IEbw-l3Xm1{lV7?L7Py3Q
zqE6g*80wv_j|o>2h`>*%Nw-sX-ZjD?+{cNF@wfMiXT~C)V5#}3Nyhfg|G4abqS%c<
z@{=vQSy{fcuU?A1i$6-buV!pC2)B$ICy-Fc)l&&Z?*YIRAb*=#iPK_O*MfQyL0;qI
zt@Djcwg7Gw%aWA>rDQ5L`b_@4S!Rw7x6rSLVDK^E60O~zG9d=_y$Q~waNk7=K9c#>2sh!dkPe8y3ba3h<{zURzs$sT^GXq-_b>nZ^w%a
zy8~`IS?3eLb9RRYKObN-y_o`A^M+TuK{CvJ9^zP@qUPPhq|`JjIdk@I{(PvrUDw4@
z?vqn?2VUsY{ah#t0%LUy*dea;oXZx1B+mEKE@wX3K2+$QwECFB*8QOn|!Nb!sQo0QY%D{pe1}M{3s&RC=
zPmk^xIshnwoJtU$zGTdrbAG9?x2R4+zJVlshNdP?p%PVL4=PqDYRv=k-2RnqVH_^K
zV65a>ME(_sTrJ^6)ZQbv{6tzNV?d_9V$^geRsk6*VNm|P9fb)_2i6%DCj`%ad#3Nr
zI!c_(Qqr|Z{4M?|W1_j(~^&3`Mh#~NCv8GTeM}=+QUx-6M$Y5*nccZ(#
z2fcEdDXG02GC)D-7BbMN(~zN4aIHFtCR49S;$S~!)|3vn9Sajf3`@k|4dGZ!W1FF&
zyu9aA>1besC-S8Jr0voH)IB*S%&fsAr1$2qo%*9$RXgyC6A#JT6oQNk|Q+x&55j0HD=e`laP4bSyF7cJFrfJ$u+HjNGscUOsLb
zu3Kb*p^-@=!=Ofnu+k3_8{BI71N(d;mYt(zdU|c~<3u(0C7Lf2ol(aw=D7;u^UeR}
zNMSQ1>@rN^iR
z-TG-Vx3Dv;+on$p%sdHKbg@OjOcNi;OYBkYWq&1yrI7v{dvadB#Ar*;jb@0B`F%EN
zu=4N{JHg)Zr0ZMA7|A8(piP~2Tm7%2R)wcYzPvYlPChd9fpyr_vk7nqi5HL8;c?v^
z(DEDUv~7P(jeW++o=Kw^d^*r8z@0=K&c$RENZ;uoaw)%y_;N)xGPJTcgT{za#xEDd
zAjeei+{wEwhz?Y+z(dRndwYzcRCr0dr=UJIkuudn7KnAy+*Ho>ad;~7ml8b%X6lqD
zpany*?K=*>@$f1)^Z^+74<0JcSo=M2uzvwJYG12oz3S#EZML$6-nQ;O-xPvhXrc1U
zK|AxVr#B2WwhTa7Fzj+F%cawdX?4(6@^P*M1S2KApB|Yh90D4Gar5{uQyllnvvEA9
zj^s5Qr3zvg*?SmoF^z~)ICyhaJr#HBD*RkHEd
z6h4U%?C862It4^{3;D0hqj^8j1=#-+c_Aqlj`G`TehV2==?jup?n_iA8YtmQq|?jI
zMRhSEhXTxc?1UR
z2V&FC*>>|ymqof`9XyOfSMVlEr~1beA<>0EYl9*8@c~+wI2K}=gscn7B^g|Q>ar(M
zlks7V>1+I;gT~aMVQJ^!ICpMn&3mO#Jc2ulJ-HM^Z}N{fUXeuLwnM*I$+?V&EUpb*
zY^U5s-ZQ@1nT;bB*L3)kQw%Z7ItLN1(L=G5)2T3m%lN?fk2>$2VP5EGJBm}~kmZ9M
zKk#Tin-Rca_`zFiJ=GoNm?VCXCaHFG`G#NIi9}q+c5egrk!jRqe3<0lvE>K?JqRao
zzlZpNys)fl@r%3|r3;Q`j&t}kd$MlUFMGi|0KSp*oevq4k#s2{0;4
zOc<4iFF>trS8L3}Gufbpr6*GE581EqgjBxz_ylYazsp(66}9N$`gjIc$U?ziN@0IE
zBAH&3bx3@?Y0fPeUYt@GP%PRM3Vklk1mvt#uoQPek(M5(8
zl$67IA|m-dUue<7qxktF5@ztaXLue`;;A=}bhukQvW3Q@dQ?Z~ynl3jHjh3B)G0kw
zI|Q+pVZd?0=@e_AZ+rFsEQoF`p|FAbv=s#0G%*%f(3?72ydioOSL|J6K3=^RYUIxEfvhe|
zRZEq@;rak?fwuR{ygI?L%OUOPfLQ`tbW28|DDgPuw>(g3La8KeSM~_Lb_o)+k+2?s
zUe0Aura9;g&PKU(0=WPKN%&ELIBYjFIc(V;(a6ycj8H5
z{0_y)#5qY}J7jj{5lM-fuplZjt8A_Rhon|f%%xcHfX3p3UVjt*l>*AAh?Z8Qqa=Af
zJ9l(JC7)%E{gbk1@5gdZXHzl&z{miybOK!h7qqgWE^~U~u+pfOzV@s%cy4RIn~DE|
zOjHDG?4M$fhTG{~+)cP)i`{1Zdwr)1jvuzx(ae*nyqnuZd2p2LwwwBLkY-KY;a<`Q
zGH`xfllUJ!T;&>Q|9bA_EBwdO0$12qp;4FY_KV48*|MN$Nd!l)EZYVksK5K>)8x&v
zwXNhX-fZ%D(d5OBeqRF+}&m
zD+mRxb$0IdXk264RH?vtOmOClbMfwVv4&VNyO&SwONK*>nEC
zu;l`7ed;IXCM&JcUU*f9eoeBQjzrD(#gapPtp*Ve~g6t&hPpK7Bx$9TI`w-
z$f&QSBPN|t_izUwW76A+&^(7Ra?J^YfE$+uGS{s%BAP&b;=*~nlr23`ap?e|`^vgntdo)P;`z>k#tVG6l3o03V*hNRA)}16
zYNI*~j5raIW4n!GjiFb**
zvNSkvH-G%X2IC|kbYy{3*wTjMhh(b$iX8vtfA*WLg8BhY)YG=5c*0MP?;kpT;^p#u
zr($~8N}ntsA}GRJovraZsLJB8TpT`OY&|?(8Bcm&xciL2e%tm0+o1>9p=`x+cNsp=
zLa2;JOk5roW2gv1Fs8@4e`B}ES2dSQTH(Dv-Qu(ciN{V7vX;&RbpX0(@FFh7we+Pfr_kp5Chv1$<_f;U9P`!Whpxly
zj1U7duL@B9$Xv(eO8_=E#@WYh!&|QqJTN`O(|b^CWCI8tGG_1Xo_{yrgtpX&;0VCd
zu(O&ioTg_3-xe9^<}p&w%hW&&lomIT3&m-110n88_t#9HcbdXe&HkjI1{(FJiMs(W
zrRk3EKX;q(yWHS!+>(x4m)U)&b@=F0m*NLHLIaV#N)RqY
z`Is%#{$CHtuD`Qt3vKbcZsmgoJH5yXTFg3EwwRELUadPQ{zXU1V4uA-U){R(y0tdL
zC3%qR*C0_LiVEhg@N1LZrU-7{GOo1H8Y~EmC~ThmEi$R7p}9p*gGrCTzp3%sZ#e-l
z^ks&WM-zZeDOV#^MllTuTEPm095k2x5wHGf`7(M$8qL?G6P1?urj5m_=w2b~@6~W_
zrvo;8tCmY)?uzBgLI2xj@9{5gh{@NVWQlfMY5CB^rp8YSA}^I8G8GbOk~loIiH@$M
zTO6%Vv)2=hCl|s93H?rDjcoNaeXwLe3&Nv2MqyY6n;?$_R#GbiD$SnQ2QYbP=%;rK
zzdG0?8%cns+LHi!*`gXc^v)kH|MJZ0ae`6P
zH
z8g1^@Q);TD+mHe&Vy#y#A{EK9y}!C1yoeP8M=%o4USDNCOr@#qSK#uV{^nD@Cisgx
zh*)b_&|9d5t9RC}-r;{K5hcv;{KyOoR9?bLh;UfjfFco1tShe$x0z#%7x8*M3`}ER
z3v2z-e_PcRU*6pATE9yzFNQH%
zCugtL;7-VO`SrXwMlYFpoS}4ns#~|mBT-Dnx>1u#`q0=(P(4tjnggZKD&>u4>xW7bTktf
zew}C$n{J5}*zoEA4-;$v!60!gbkJpHwke91pBrxvc)hE_y@_!o9YprNba)?9?!9Iy
zcX4^O5L=Wn^~aw*C#|FCKM8g0(n+2!XvkwKvFgA05_aGI-Z5EPxZH?f?Xlq!o_lR0
zCrFrh6ku#oVEY89oIBl)0u?ysqP>njmIjUa0eAds2TYUS_VFh`xnxt{LqWT6-6ED>
z{f^hmBqB476jdd_Yj`iKNb<)?GO}{hEZ@p+3{WaTed~a+~>@#WAEVpHCWWP1YD0pC9)_zKc>&N5JYrT
zQn8_>l&nCoq_D{ue+c4127q?~p{&qPu^f$|
zznPp?=dSZ5Q6;T(J;2+BWa7f(y4sj9EePkdDwGQk^wXExgXEu)ouzZK!ZKkML@C`|
z^i)rh3Qq;qt4fCt8gjrq60DaULmWbi+6F${jh``n`qvrRcK0tH??vVKmwsLaA5ZGk
zyFY2ka|@RO_ELWhL|J+swFp2$JA>jD^60JEb#*vYbybC^PV6zZZJ*{idbd~3j{o``
z06^}^76Ag)DM~MkI<{Z$-N^?~K|)lM
zl4vEWpx1;)yh{g4Tf^BKYXQ3eIn=_xW-a3{?MnRLbcsH<-JU=FsFQ1CIkLzoZYKff
zCO^=vy24X7Oyo8dP>nz;1k2UGhJ%K<8y_x1m=1FolA?HWuYc5WxrH!Mm-0!{xR|NtJ
z`{T0#P&gJgGQTMGw!rcDfCFZh#MoZJLPZ{hCP+ePC(vDK5hZAL_s>4e6xtMfqJP)ZrK7~CYF4YuxOa{nod|`(CVNK3t~=FPmD6&jM#Z#W#*EaFE7+Fw
zEKIX?an32*T)LWL!D{Ulq@=A|d{2Ju~`YrLT6f{z}xsj%_7(-;{Nfg@D*4*y8|n#)`c
z)jeBNsXa{UiRD$Xk$rn*OM`y+MbjMW()Tesw)?1_I<-gH!nR6hr(F=)Mdk`15oAE$
z3d}=T(ru3;nO)(wL$c$_Hc~}=2=LjXuAeQAv*n+5EX|1CMs4w%J~-(w2&QKxWit8$
z7AC%6zcdu7u1;#Ddfb_C&ifn9&{axh>w{MMn}vqF4I$P;7>edyPGs~0%qz`6|A>#N
zl|`nfx()p?S{6?Pcfz$Nn^*#1Rf_QAWFJR9_75J6G_veGSd-IdnYPtJp;k4)gH~!<
zc#mqk#7`=mVSn5rd6#`rv;$zk?b$fE&H;3QPB4x+Qr?XsD%2pm(6?}Sw@q%Y!f?_ikJx)M+fP?px=63_)WP`1|ede9q8
zP>$BfQBb(hZ=Zd#JXo~^7&lL=SQ3~8@3*jO8f;SQTj)g%ZesZ@+>8U)GXEBGJ%hV2
zfwS=Nv{q9>JpSjDg?tcDh~LTzDG1;1zOz;;6$n$4Z{?q62u5o#W}en>>wcZgx6sZS
z@*M`0CDqCWGK`jmnUjT+jfIYxm5P~}iUy8J-oaGN$kmL5T8x*KnT4B~hxLCXsd%oC
zF`(>;bkX>DJRERL5?0PGt|Z)C%>Uck#=*wJ@!t_sm%3Um_*^aDp|vNYFA7@Ge&sUq
z$%h=c*UrX#G(VM8boLO%Cw?5R-k%@)LX$*XsV62>Z^&$B$iTQ4BL5D$zAywfpe0w>
z7SqwtTIrgB`OQi0^EszBi4~sKCnqC@_NZYpT%Br*H&Zd7T9ucDNkEdHlBvt^W-QHV
zo|caSq&r~PC(31b%;@-+C+3taXzjoe-G-!Gn`j^L@E>zrdm2>sl*_6PVafG84WyZ9
zM{VaW6!>(@CcNl(E^&=
zsVi_e5M!IT`a7c@ba6zV<3nMP3CJF0^DTjZz^kGLYem?CJVL0mLR3~U{$kX~-vxAd
z`3R&Xb${T9sOMm+*+ETC3K47QYjBiAb#ZnBcx#&eKy*Uc<>}GDApQzq1*1fElGcD6
zQ$%Hj3uJ>+GTs7-fS#1#_jZN+n=xu)G(ZI6z8X9j0>zIKMWHN~f?UUr5*6SqgxBN&
zppLSTM&G6fwH?3*)1#3RZ4cWBgwDmg8~(|p@D^+=Y}LQ2`=gj`fW}f
zC=YvilOzEvyGm`W7*7rJ%aJs&?}@|;kivQ{{3AQmd0wr~=jYeU44@{XqJl}`r~ag3
zY{m9@YT;o+x?<&<@V%GsMwh?uO+H!0Hez`rmi@Ly8=3GG(6{$gZTV`QR;pLAWOuP=
z!nGVSL>aA#>|#{Lrr@RSxwF;CV7`@{qiJA2f&=G#ucv%gmbcBpxfVU}?;^SzAn(z0
zBZE_bYvd9?V*yR<-{*%}?mw~*gQkDPd1ryE9%^$c#NzdSdWY%_VY1a>2Is99J>jg)
zX1&WBiaB#?Nk977N;tj$>@C|!ez7!T;NG$~4`-Q0p&0n1d_7N{MUmm$rvG)u2=TwG
z{^W$YS#~Hkmblk2B+k)sc-<#J0jzC>_SxtbPCa;CJ>qp4TZ5a>n88uOxQSJhFv$(wbiHXVnKl?yoM>KG)D~U$
z)9)iu=`@Q|4v!>y`mEfSW)BDUZa`f*5^twQGh@+@34(Qo+YC1Ata9!@=qu
z*(kCPWvSH(919Q9f^m)`i)C|6o68W+Iv^f%Pmpapdg2MVXIb7&@{9H$p%oT@*mlY2
zr|QrY*t&6}uVhklB|Y%ZHRbI^G%{eUXyc{A9smi6VQKhgfbBQAwnPLI?9~f&8%-IeDL5E%
z9$N}5oedG0L~aE%#+r~ag3R#U%BW($jnAo{0n>1Z26F%<412hFO5=5_#@GQ)<~0KX
z-eFASsnYD8`^s$qaQr^zB~8>?GVHQn?_ko-F~-O_gC^5ffJ~}c-~b;jk$IQWB3E;-
zJWNpRLhVvebup%|!YxhN7k5lQ2_`+`7^KjtD%tQmAztYTODoJPXK~cz{z7_73)}O{
zt)*xgxSm{4j+r=|0+hdb@?=e{s%~2z_g%)o?OUwmbP1LR(2gD2n<`GKsUJK6KQFN+4%fvd4!-s
z(^MxxeMgPj_wU%Om{&V05qDrqt{*x#NEtesOu=XCfX1
zsDE(S7v?EoWCW!im8C$c@XO;<(@m(c6oNHq+T|^t@&-Hz*kw>P3p(@DYY)0fT-{1B
z__12=1-yX|Gy5NL+e5{p5zSgLtyTSq4Fy-SEwJWL#M7v;1dei+024P3dgAT4ztDmNw`%K`TyizFBvMY||X+uGGYq}`31q?zU`
zSEOrUhIW#kk`QW>I!Y?Uah@9M?JKIk8LJ}zw9|mqe#k7jS_7RQeEvF5=ksn##~^SX
z$tf!1;l2v3IkMtv3h?9vFXsN4ahea=&LFdh%)q3Lhfx?N#O^9u(`0sT7@HQCQkX{U
zWJfX~nw8x}oI$oF^s(cojCPFldoHmfx0wGS4HeidN;XZ>|2T{B@nbk=^7tV8Pi~?S
z;6+6NBp^ZbBT2-VT;I{U|7wxaV1S2Rq`^p-yeL1zZHrP|e`TpEn43eEG;-aQl9+n+
z@c!NK#Z;&^*H1C?m|j~u!bRH?D8J>mshFM{Hn!Vq$v{Qee(a#{VOxQKbO&4AtF{?X
zpzcC_Jn7|c&Q}lD%s|oKB{@&3E1VtuUCQlyCuiSy=)B;
zygF)6G}*RE>5!5ef(Zv^EC|8!4TAnL49l7hB1mKbJ@d}Wft#jouV<*Ol+GZxi
zN_oNM(<@nq>^=HxKP#new0D$lbR4wcxVT9h!qY%n*e+zGrfP$s;;>ldgkOFGXydT+
zwx?HT;UBOqd(jf1);m@YNB;g2wrTXfde5mo7OuIz=4lbkjA8@Vd}FV3W2aBYyMsI&
za=xBoOK0?FrX_#TToFxDQi(C+W;4|
zrNUVHJbum%Cz|_ur#RG;FP$$ma6{GBaiVIl3V5qtCoeB-7!5f`Hd
zxoY(Vbij*SKo3DN{M(MXaEqQOlu8hj+KH6N>>bM@)F)bU%
zZDlTBU#Sb9;=hPJ>fZ~+9M~O75|8#-^``0_KTjL9=ttR?N`Y$)(tdp&EVvBQtP=pR
z<^pis(x2mGLWt)!C%k?vatX?X
zTxLuhl-tZ3{C6js1A}Z=>`~$V&kTw_9=zO~maEi5!c$A9)-KIW7RSTc#d_OKd#L-%
z1{7FChN8gEVTuT>1Rh)_M#{q4Or#8pwUm(F2zveQj_0rbbNVO%#3G}xl8lE&G4Xuc
z0G;bkcUJ?4h0o3#L@gX+C7kCoaCLxZWUl6?LBd@x`7p6O<8NAniUn(^Eo9++VWP5t6&8F8bf`ZMk-Wl)DOZSr)~SFmFxAQ6a$i30
zYb9tjOp&NHuSEwCkA-Vtx5uzcFiA2EXmXc`e}dxUdwG3*H}JQ*oODYfN?X|bC
z(MPmLgnOW@gryrFt%6=i+30OTZBxGCemULclQ6_IYrWdC4Cfq3-esUF*S;lh0S#Y?
zB-Ytjn+POIKoro|K@A)O5=i)$1E%te94#2G7m+469R~rx<*8EO+>bD#NP&-k7LnqO
zz*i-bjF?hH@j~2Ya8w%n{ow;=WdL4`;?yfQOS7fm0$$W@YV#%I%#r*rrhi$uwEon}4pgB~JsfJb}~u%_4_AIvYuENE7&$#+{!GThy#wfUom)
zk85=uS3W=sH3K_dEivA*<@=^i@S(%Dk&g@4-SAAVfKUB+qz5;hXwPrXuxArO;EeC{
z(MMhT69SxstBiT#AIK8t;`~05BMo@Lt+C8ruYc>6K^kziBHK1KW|>ng--^dHBM
zmV$uX&vf0PIm>C`ZmqCq0+Coos>INbsUn`z;T?~WT**Xp?~N*zh&Z
z|BuJ}*4QnA7K%E$$08}r>Z+TZp9-P&q9B05E#D@goQ9X6$mT|lKEhd?cr*wK`-8gc
zZovIPYpETIdz;;lJXApT@2=5zPka&t9Xyp$X&3&^7hZ#wXD@Q9_xsAhRz(BD+{V+3
zQ|EUrEHCxH6#j)nojDz=q`9ydd>DrT0SRA6uTsvXbdtRGxsB{DlN%8{S&82-qy+#v
zYvBG`msO|M_eIUG2|qrbmMRe@Yj#8{02)9{=6OAAoZ`^E<~1}Qb+gfm6Di2i&NnWM
zn>9}NMZ#+1eN)c-YSPu(ifBEKk(#@LBn4VFC7mn+`B0?($UtNA9yhbOaq^@#WihXZ
zhqX>BA$ORsQ_O-`tdswO8e>p;<_KX*dgTD(L6N>aCO<{x@yq+
zImD<5`wJqv(qy@ud#go1M8&hvxZsFtiMqUe{@wn}Atd5YG|sEZH#EZd5{qK9+!v@P7_ZxOGE$@(Ok(I=UgRNV{S(|PT;3F&I)c6*HECu
zugTJ1c}#B9Akg@bqv)p$s1CFy)afcbNZpXm|E;#jKv-44AlDq(vtKvk9k!qQHWB*s
zC8z|@mZCb8GN`MvkWYT}*bi_r8%`-aoU(hD>-b(4!Gh_qX*HC~!F}4`t=qdtPMX6r
z1T0A#-$faUyEl-OMef`tb%@PWC$m8rmjuq+R~warEOYhgrkH~UMpgaASlZe=L@n&P
z=pr~Z=0|+ny-n?gn^3#8y=P1*Y%_7cfZXDQ;CPUw+}k~eggduyGZ1T=f_bOH$wX0X
zMY{SIc(6*3p0q*9U!^E{eZ8a2Wanz%_W2H}b2@Zqg>HozmmDpv1%+Xn&Fb9!%_=$|
z2m!6|ATFW8L2Xzlss=AgB=H4rpTV}Is(NEs9$57;9_Of6NFvE-+z{a*rw&sLO(o75
zd6$!l;>~k6{w|df0B{lpd$*7(Dcd7X71A3tSJ0ohyRED<7xK>)H>cB|J%=g9@jAKQ
z=W#)~Cq9zWk55o5hv5Y|H{Csi?%&}}@~sWNiD<8mJFbh~NRk6f3Eaq{9dpZ}T-TsZ
z8>qwdCX~bAF@SKn;H-f%9?QW~^r~rZYG{4O?&YHzsAI4q2Z-V@=yRv*;PjnbGMloq
zsAdO0b_i&CRhr`KJt+9k$0sVn(Jv3^pwb**#0a9#$NN1j-S
zT;H*+LpyV%97dXkdv9trhFJ0R%Rl$=IZ0;x5ga&})3fHV^No{x(|cb3W{e3x+xVco
zWwN#Hm)}^&N-fahDbUVNOI_NR|~B
z`X_vx4L9cXLq)s&8LXEzNBoORUe?Qo73@03+XYD9J{s%ZhfF7Fksm_i6MbVpy_t*vDzt)z(
zhHW3zcr7)kv*&82Mo}}k2As2MT&=vGVRX9bmV9fTiboYIBu4SBF7eUW!0&h-?k;{$Q?JU6
znHdF68`D(~iEdn9sn#5fEck&vhr$14Xp1_m26?WzAcafMI;#kC7iu?2EnafADIgL(
z1-;ZbbB4@Nh}z46k@G?pl1YI@v54}~UU+T;Xw^0sv}fQ4l6Oy~oPzZ$*I=Fso<1Q*
z9pRLU{{tU0t&MeLYgw00q6s>6k~nWU@!qf_5;0KqFsF{batA9TG7$p0@#SL*$dpMa
z&T0ELg}2tpQNI!4KorMjD`fl{w{dW@ct?PG$~KZb6d&U!sTj
ze|24M7Lxz2>r%|i+R7Q>jNCqFr`M{bw%8XwpFij~Rg_iG@-i}vWT$k5Xp$NAhptMA
zb)mx6w_Xh>7#kU_Wk*MAO4HMe)9bzgZ2i5adA51(r=A7P%>iuYMmup@KhoC|`Jd6A
zRnjxnPE`~etJV=j9YNgt{a2v|tAO$DBJn`6j9}PR0G!VbiK2#3yH3bcLW&S1eTZ3I
z*07JXS{)MY<{k3Gh>Rv=Wj@|=pSJ$G{Ddf|^#+{BgEqsWumHOdIlC}hlMr2#FyDhn
z*m@tGUom>eOFoMV*MnnwY!m9+#)x%u8}ahJE8oR$OOOaseX5?e6BC#?=}c2r=~UhY*w#viW(AAl@e?{p!}-e*
zNr5H$$5tE<<2Pya#hQ{`6oOD(>h{gge-<$VEZj&`J3q1l7aXCc2Qbszd|uS9bAs-f
zp|N+!bOV959)B0Aw<}nwaL-Ob$|BATaFf6h0q{tiWm?RIc{cqK&>`@cF~$-UvmJRF
z!YhF_kb3aeSQWU4@R7J2O6O1fR-$h9y`f2Go3(x_){ec^Y|#?WD`yhy?K1!kYxjRQ
zTIIuRzqXUK1@143RWdEG_BQ+-OPDUG8-8`BAcY5
z;V;0q2SO0OzkqCZq$CIkDaLQ_Qk-_v^Je;9w)sUE4!Atkvqx%}xa`n@ti#?MR=kEKj&X4h|jrdyV&M!f&Y|ISIPp7rQO%BKnq)V
zx2`+m_gL@6MefK%OO@9vbLrdKb*2`uO223-13+CcR^CLeC(6gK4^(*a$aS?}*FYjut#_sZ8XCPHhWS!}l%adL``
z@p}kyIrGh?YO@d_>P5_8Mmu^aV{k`>4u)r-5Gh%9WuBAl)d
z67WN#X&22qFlg#80Yg>Vk3Vt~3@&S)XarYxh*qD|Qh`GdNi`%_SvXsUA|
z+9UV4duTv5tV|ZiFhdMgUkmzc><;@{@o~K~iFZDU9SAgv7)b?As9&zjT3hf*&|K&K
zSxU;}fW@Bv+V6h5apI+MQ;|fjb`AL|23RA;^u9CV7$SIG7Xcl_B^+W}szFv*F*|!w
z?0r<}5j+q){tYjsO(TBjds&x6WL6$1T^BY>yr+BB{^BlbGt@<3!||SRl|5|P;M`gJ
z;5*1@9xvU>In2>YJxOx40}bHr=gHgu#^w0J%i}e}uoEai{@|q@C!aJ5qNa7+0dOal
z(aAh&UHYs;#69`#$C7B$VqKpo)(v783e)xX4)RY!EJgXQFUGcPW4lH)EUZ(zDvZ}S
z^*3$}nG*QNk<{{a5W_>utdV&30_&%;7Bm|?O-FC0Rlt1}1Hspubob&+hqF)wbUx9np0u9f2>
za0_0^^sNwoy~h0h`m7ml+R5y0ezf;Inn9=0ySw}l1ww3r8_Y1gu_WS4vNjBnCso0X
zr}*WLqc2JPJlG^v;dNa_k29v@YT+!R@s7A0uXgGwE$bZ>IILTzx6p2;02H3Zm0;}*
z<2k8<*ajhXI}C7g+Ei+JU>l;TaaHsqXE8_~*-DLMilf-uof@t9chpwh8zf=VMJ?UU
z+jjm49rVxUGd@$85|dcOIM$8c2nJBo1+KZ8L*^`oYj)X(D_}%WI)#MO?d3jISWVze
zL^v=QPG5z>N^_ejhtlH*0vOk+PsbIqP;!shIQp5zuTOyV+;IgCn(FS_+jjO<&N%Z1
z0jpOx5;9wC*IbW8iAs04u&?kd%s8$gG2<{a9v@cMY~#f|A#eiRd{RfpRzY1laUdcV
z4jIUFc4g`+nd5A0y>_~5LT-mDbG23RM{mSwhkL&*)iP;yN|bn10Ir5tagj~4!7v6o
z*RuhK)hm&zHjx+6q#ku|kE!k=Up1aa-*t5t6m;M*Zv#2xmhTaT<1|i!VI+>MecjDn
z{11mYlDG$*@Rp#I)4>yE%^PVhQsJAo3P!ybSg$e!={~vj8oh<{hlQWOOJPYcQRr&x
zTuA&{ghdS4LiNhzbRcJW4Ig*0yE`70#01%A;PM`;;j#Vi1QH2pVqKbw
zLI952#ECG!L9dk8pTgEUE;m14aru?@u^jVjU(QrK
zVvCB$J7u2*_{N5_oK7%|T({*RuVKey;0b^PcRY}QywGJ!5g&+LQP@$n^{fr7nA|jZ
zddZ9Jr@$pKiFSYoHGhYK^LaT7X3Q!gof3ncys}7HAIsM?pJw{YfYuk&j!nC8#`7z3{b30c1yU?l@%BsNMFy*e%`K7w3&e&~
zl;OLHWn#rsDonp3kqAaUGANRAjt6CE|3C;|1dItIQzmMRV6ZsfGaHB9r2{Kc!jV%&
zWdB(PD2rtJvAbhbm}0+)lg)w;N_i?C
z(7cH^d$cl^8>ZqA*VrMn>IAz-u);9k>SmJ0xT@JSuWY`fxa}!8a-Aw9^JJY!s>ZRG
za)LKM@dSr9G-D4WuqYIxI~)1s3spXe3RIi|uCT(AT6BYq5iak+?O4b8efOzhUtcj*kN>F7zjQhE71
z<-PnMGj<6fFh&5%^Rs!CEh0;p^eRWdOJtKeTe|9S+(p9mowrlyRh!4}5+nL)e6ApI
zD)Au$v&LH;&+(_RV3z1C0tyPs!9KPx~VCo(Oq^_@Hr*@*Iths~Sbzn2*%rU7ra)N$*mG+pO?73|ffxjAx;(
zn8s?}%K7wo^uE;E$p0y)5!ii=Z}bENd(y_&yE!R?NzA+8%+sJAL#?_RQ`9+u*llgFc6Co8}Y8@?qGR
zhCM5Ij#E@|^bVY(xZ;o|adas+?^s>8hZc7(m0o(ut2f^?Oc8oP*i@WXIPa{+li@kT
zjpqq^C#n#oAs9U3#?K#W5mvQso
zC{(G%2;a_r1t-s;laWSE#9!02UcK%&!eYJ^5*RZ(Q)d?^Gb7vone2_NU|5st712OB
zI9dJ|E~mA1>^H~I{buWs<}f>~XPs)v+!Mkuv^uF9>F2P{L>i$iA`+n<3loz~em`?r
z|AvxSPDIq0CSW>qW;|9pQe&!#DLNopDgYw6Zu3R{_zG%-oD
zA~V)%L?QNnuM@8Re!VM`!44)$jE3~BFh{j%Vj(kwOStHPOh2>pbOOl*
z>TML8t(ZBV_TfYJ!=w6dgb$`o3&t7pmF6Nrvl&eY7
z(O-S}Z7^)kmjg)?*mTsGEbp;~rZ8o_1IR5^j+aNIBs<|;uV1H2Od}cLR%L_GHK+Mc
zx7k~1q!?8pCOf!M^F<-5Z+Ws+^0|FK`SfxlIDtr7-*QkV@Vtn^nnbOI`Xqx9>%;#M
zf1Q<+;2%}d(Z~8zL()i4=vyQ)6_75x)1zP@QtiOS3)}v-&Q_YS|1JN%r=kcU0I=%L
za^hu@{Yr-|8lVzw6;8v^CjJ?`P-?#ReQp7F!ImWIOM#%IHuK0N(hD7D^P0Go#dnmj+z2jWRH$M3
z?3gRy1ILMA;Bb}lJAri%1+yY({S`G4!5))2XKpO_&V0mhyivyhho_@x01S#c4h-EB
z+gx9?lqw<8Mq8e9&A)?tE}IOM5%sH3%-|6XaFy`sB1vSqq!c-7KJ#Si8d2G%ts5ao
z(R^6AXX5#U@$0Mx)VV~W)@*jHks^5ja%5nU_mm>9y;VOFpqDRK%t-5`&3+93><3(?&Y4eLbc)7
zJS^Z?z-o6oFsZHsR~e6;_Nr|L6Lc5)_bANpa5R*XqdV_vJUX+T}wI?I;0
z{anEF(Z68vHVh0518l>*wO<-oU|1L%9C$gjPPf@IGXF)A1vv#b+jn-;)7w$k%|+l!
zOMBt*jzpYqAxE<&<>DlGX8+(F_s}%9kQegr4Suon(GgOa@Iq~*7~7B4KnRL3%+N8j)_rj#0(3#HRT8LT9qJ+qRe7KGByYhsvYupaDo
zXhK5WHXO%p+*fGle-Kv~bDmAgRBm2!WKhsUlB&nk0Gdq>;fW1!C!aX0i=s)nrS5DI
z45C8drMB~Mvom#%6tSMk=IW{*YY~(I51+}%K&2=fW$<&_iG$Y64jz$X8@AZ|2?$L4
zU3{%V1pJ!hMt*Z*16j8uaT&aLX>(rH(^B0}W2gn?r&d#Vg~cZm5>j(nxh$I3+u{~w
z5A`Zzfc~7kKohync2k&~2DbR00$}39$F<=e?hqlp^-SuK@ErDJna(M<`v%x4g)JVe
zVNshAV}?983S?*uanwvwchPRVvuxoxL(xY)`*(;BVek;&5mkd9Q3VE(VcF_ZHsNU_
z=!u87ipl1JZ>aGDQC?Wy2)~*o>OK5LabW^kK&-Iy7IZ#hsIjqVON%c>x^Cw$mUPq8
z`GU@09b+)>LQ)MPq4Nrc*-mIaoQXeAG5f(>9k#-J@pTZ>Y=xGR(F=MR*O?y>nJ_dH
zP8!)tYbz~3LnA#dt%}_BdN{n7dXVTg`&J|UzUeIqQ>mItcW&2j@5;zU-$lbF>#Ewz
z0NWM!obl#?B$$cyHQTn!D&gW%P;A(orLqmo>9&y)?p7`X5bx`buBnl5Ss5LTTJ>1x
z8+O_2J*>H)M6nC1PZke`e&NSa668-2U>sS_XrqNTcv#zwIZItpS@_#3A8)?OUjih^}z&;2-_!b9~NB-7oR
z$)CjMZXQe$1)Z=0nA&{1zDwd&>%|L>A|BZEKBU+GPJblTc^9!0XS5%98;#utV4$+8
z%SWTO11F8nbHC{u$pOTwv`=EnJ+y}`B9)Fb_-m|ojAXUPxv^E@+vY#`dD7j9+BX?7
zK+o8>yEu|$hr@}u2@wxm8WvK!56*vXBF(bY;Zfdau`m#LM-$9q9upj
zADkdH8Lwq4y^OMTZI3Lh?cH(!-0T4XRFp1vEn5L{+L!&Soz3HyzA}uRE}mHj%T*~C
zfQBHy`AL|Zr+GN3IEaP)^eb3seGXTyQRJUHW(fgMJWRrBanr5Yl~JWrw>na*IaNfq
z7}QeJUrKv$_zn~8LZ`xSd4Kq}aa72~9?cN^L2$rA54V|ZeNxZxt%T0zw
zJB=m1gx&gW(AXazC^s>88i+CvAh1{kJw41;&(%g0;`plDQ$*cEU$0;9)7_gU^&Za#
zObK(9h@Fg5TmK!bCH2YpZ6R5GJA7qNCW?3Z<$_F03}jB1ZFpSR}=JwUbk##
zhvWQxqn%*htDZ>EjP)Y`C0lZ>H7ey=WgsFkImWe*Q;<<
z12qy82?BtRypO(mgAD^|Bh5cmPFa6fbFs>@Z^UWl{6qIa{qI)Zx7;=L@vbbV)N9=G
z=Lv=6UGD0H%Z-2jF|48EB8;U|39X%L`i|mx1(sRc@56y43q;mt*TaR67^OVt`9PSAE_S9qd0`l3&FucVSP(IDak2j|ARqq^AhVr+2A|u?`|Mv8a^{;%y%h8)0oqP;*;5q>&
zt_9EnawQQd9uaJ8OzZ2Lm@vCLXpR;x&-wUw$VV;ZwVTYMbiId
zjR3MuBAo2JixT|s{vD=N&n6CF+W`Re3^8>)T+%EoYLRSD+E^+3U<8t6^{>9g(!${<
z$LW@0N*@|MqcPs2hx~#&OAL9E1hg_%p~ySn;+K;4DS74#jwCPU3-kj!ZzY8VVVVj`
zh-op3SelLNNo$J-?Uh3_dU5>3l+YT>3fv;Okc`j`E^WAf&KwK~L7Gos3eVV${n3!-C!
zQM36M3B7Sa%5)1Tw>n_G*?)~uVxo%1da5v9VNAubDuEUsMZSkP*V=td{
zz5tT8eG-}?`QEo3>Z9~!9gPqDC9A_e4&tre1boH=2eJcg#obAeDY-#E^GCb;8&$U^
z-s8qpN*#4%^ajr9%39Yyl?Q{OM~BNXn!!z;bUS%eOD6&gn0lVcSB!ud6mpe6wX218xlVs>I^FnV{ltfv
zFpNswo5LR}KTdFbO+=Ru?R+V#90LS+(y4TNX^{Q0rWdQJ{wB4~8ex$;
z9!XdbEXm$7S9$5n#X10ple)PS@TjeZD>8UyV1P;}uRLh9NZ+t;u2Xa5&_~ER+8=bBm#3`2f6D?TF4j$ExXc
zR<4+HL7syqGUdJ)Ba_cccmO*Try-@l%#5-i#uQIIi)G)`wfy&&|NyorW(AG7lqQb1~xAIOo(c=(xaA^@lrtDwTdWy&W
zkw3KvhJ^pg5#%M_Uy#O4(jI65O9pl34MnC}$J)m8p*BL=CHXEM@@LcgJ
z9yH4Vi)FIkf8-wzSmYmdV>Fs7I!V+{=M6QPO$3!#ya*H#1*PvNk&WX-xE^%nX)$Ax
z?+g1Nv+GZr!VgK~6z2%VfWg&ffF9M7k3&pvYmg={EuVBdY5kHhd-#B7rld%~m3~tA
zE1k;;sQLlmIK#r_Q
z|3;3{t#gJ)$B>3&AkZ}#V0iNbNBKlSRugxctkZ$7{@9{(%HjkRt8mpI&_(jlG
z7Ks9t0i(QVDnpV<#_tns$%aSX5G~Si*G-D)Ax#v+X{)8cN=}XKSoD2*3q`e2<%fSP
z|8Wgkt%5Na6I29}E#s}T3-XJlV^_u3#<&YXGd=WaA?L+uAA|&dVjWE3h
z{zU><9nu~Kb)~1hWQ&&&W7Y{8Ws}YlXGKxaALzc*kL@1gCR(8h4ZaVn%_x~xSknr2$Vm#TFI
zIQ$*L#Uqlf+;vSWyg+)j<&BgoJb;)BNq1EVZR
zJR_0*q7n)NtvI!*DUZ;6dR$J?k3N(kA6@8f3zqKY%2VID5!11I>^9dFLxev7A=l49
zD5YBpiiOb81PFP)@$Ttr-xGMG4eatn>
zIc2zPNVO2MS`xUlbghWCa3uP4X7%;}L5~;G>2#(|PHGc^K?MT6f#%C$zDBv+KW8QT
z^>%qbzH0RgYkOFgG$Q}u0JWM?2Bup8H-O3o#_b*}HdF84
zU;Q9%6yS?TROYoi?3q|VqPqymA?I=qP0eYh;&HeK|x@>8gpQFZ=5Vf86f#8jDV&6G0Z28eG)
zRN$e8#{IQ=a6+x!FxiXqHvi7e0e|-ge3|z*Iq@)m(pdi$J
z;M-D~r^xc~XcFh?>@R`*nG`(R2Y?I@sYX^z!DD*lGH7E_3B#|DFC~A9hOWum=9p-&
zuU>6Ws*f0SvGjU7XF&zX+$l6foPcMQOCY>v$$q;t2RnA1n4?ktgAZUD8aIKLM_{I`
ztJGn}=+c8iU7QK8JKSzbI#|qSOr_1ukcu@d(+v+VNoNA(y_`O9nj3>t1zd;&|5WyX
z4LMC8lAngjDPo>a&z#Ab;vp=hOYof6vs}vkwHhStj|P@4Fs>CCaLg=j73~KNiYxC)
z&zj&iCrTC-{VL;{<{D&SE)0CsmrCl|l+URWg?eCyjHf5z9j&WEmG(fJe_KRosbC$*`^g|2!x=WezX?vursd#Z`z^8_ypA|=ROJv*cfP?ahr48H%enJAs7
z@2N4}Y%k6o}V<80zZ!z`lSCybcL+FKsq5qglsSw;$k>~<6tx}F$GkR}xT`EA|
z3Kx8mLzC&900>tv05Lxc5G6^WI~c_}z~Ye
zyN4=AvN-8@37f`LC*>p<>S4(2kWyA4ef#S9M4+7YE4N+%K)@o$H@bGn*3vX{Kx$
zP^wFVR?K&LyoyFe8Dk4ZGkw>UHJJQGPQ`P;8RE?{=6B6GK0^z5_|&|YouwjBm+*O%
zq<)KCxT)j?AVrQ@dT6OpM&sEwWEB1-vr~Q7-Zx62SC2=jNPaD;iU}i=qw?C1J-Gc4
z7piWz{kuH^vSi)`yKy_Gj7AU5*fI)Z=IY-;)ch`ik2i2x8d4G>6D;St7>XNL|LqSV
zb~JA8TFoJbPf!b4b&uzMjFWM5r%pCpz_l#47+0bU2*m~aqJ@laOuV1@J4apoGz$0A
zRSa?LAA{AfzQd63c&ceG&K927Z}A&criWY#sFs;qis3Bo?^T^TsY3*PY_RyM@@h+FwfyHXXUUYnugSU@srS1sc*sR4^!Fl=Sq8U?~*JuURNIHBri3
z{4BKq7!XB*oE5xE3?rPDTW5E9g)X%7BZMW*f_kTTg=459i2Grh_Ugb
z2z8Yf7q9snnduTo9}tM|PkuKrbQsy{tLH*Y?R4t3`IX1@6nk%>433&<>`fDOZeR>m
z@_hte;G$8RSt;!@c1+&(`Zc{dmi&|VGYmKfr16LH*>{|?2aLMYd7KIYP*TC)F!E(J
zS9;S{!omkS+10WxKlK9qikcah;wIC6JuD`D->qXAw%{jUb9)##UeN-6U8z9Twx8CJ
z@_tbdq(o$3=^<}RQ0YM>Y5Tf^L)WZ>0({gy>M4ZGBaJ-PN15Enr#f~T+Uq!}6@$S6
zMRz!F`%ia!3V|jnQ-P?si*~0J!QrO+j-qHYYNRwBGift5+PU`G3&PLqH4<7j0>
zaG0z@Yrrg^e?)C^sRVMgXhlHG(HGUqYzDu3?c=w9;={`_$m0a-^d5;i(O!otwQ)c}9@
zRIQC4Ur}>HaM3Y31!pFQVFUx-f29llQZYGU!nvtn5g(SYXp5Mk)X&ddG$rhhkq(W~
z3Vd$bGCQ32h>+&$6Wcqn?$u-7XKo0LwPL}AC3S*CsKh~oSO~wXpNpfFWJ$)f&8pO0
zanf;mv{b#FY)#|ka{3e?ty&{UX$d=*2EvF`>Nyv|yQ|`x;5d%|FA!f#yMbIPS{TMl
z7|#`ukQ>iFsjgt4@6Qs6_LGNx^sWyRqxtw5#vT*=Mu21zk?_yeU;CZGT>lOulP;uD
z{XQc6i0^uDo&;w>AYh*gwQRR9{}j~f!n5i}#DZy`o%yow_sc6rFj%UuvGhOUcy4*08ESsLv(_51WHV9fa;UyE+oe0*C`H
zzY(NW5?Se+&ee{DnvuD0v6FyJxTlVZGK$9`@2?b}Si|
z>LPvHFLWlJ;uHA|JVyN;O`M5kgx`3QZ!JMAHA|y=z}fHsD#BlG+008h%^*ZpbXf}L
z%5(}9#Rg=S1MctYgsDg~9T2-IGz@-8<@iErtX+x_MHm;l*jY=%C(R@IGOv15y@_oD
z1vbOstklY)d@vtgi8G}Hc@wl2GCE`Bs#6}dhhXDf;CR^cywg)j0Wn!omTxi2@O}oy
zDOih7&`bXS8TA5Fp0cSkhVo_IMxtznuqzu!mj`W(_4aUkAD&$iP~7*pQX2!C5{U!C
zftfo))``?}|CG2(*N>;k=8(nfWjmP8{NcB~Iqzq0=#rkHdCdJ4HdW-}U@AHIKMR>U
z>n+W`-Mc;TNbl~uZry-`qp*S}i@m?%S8dz*)jY2OCUm@9e}-RFLoP#^h*6sN=Y6r6
zJQrgSdb#eeFM6#Va2PqeNNpIC;KkpU)&35FLn%w^EPXy80Tr|C%PKCp4M7I!0^P{>
zfZ+RgLo$R06RuI25ls@wA@-hR*tC>5
z=3U
zUg@Ed?oI%s@*?B=Zk80K6$*qD5HfA#1AS2!%^^EEq-j=mF){8fW
zlP;rnnqDsHGs-+$6;15&6mFjud`fJqWT}b)^pjGgc;3EXG?#`MDTQP7TSeNc)gqg%|4icETILI_+!o@4lhu*|
z4zcOKw@ab+-*x8*&iHTd@7w?Ug`{qPjvNsx`)*H~TWxWVuc~7|?t$XsUD2`tbPTJj
zF4Aoh*ALXx_@Y+n1QRgg66H%?*Oyr);btPPhX{*&Yw4=oq=dRXb7`+xuahF_D(ieW>9m^&OTdR2No+lBH~gT2U99FU
z`k)VEwhcG+VPde0y(WO%4+hn61GB)k(;+Y*)<=00HtvpGvmz1xN*I&l5z$I(sj(f|
zyW9UDf;|Za|NlnxPf`Lj8Zc8*JrfOJT3agZu;qW-zhiz47FR&mX+ce#5qS!Zr^kEu
z0d@^K(d3T8(l_UTJVaP(5cr6qSiW(wLMpnD$~tv4Exwwu3&y;JyYH~i*`Bx11~oPH
z$cp9f!$>E6D>HTJi$urvj@@cMsviC=NA<3)8qbm_UvG=omsd-BgYrWBF94xJ=S`W{
zX&zzc&hPtNdwcI%+=z5yiR}0J*8D!*@%o%di@o0k!{QZM^OGJfSxUSp;<{6a^
zGBvjAAi|k*n>Ol@MisnHO#l{?K^+ZvDm3U10sBygSqU=&sEZVB`$0+J1S0oQ25U|@
zvT|mmL@|6>7>{{mjTtDo^TlCG5wk3rxCAV3=>6A1DN1jf{2JBxKigs|+EQn_uC5r8
z5@WXQP=*Gu0Tx;DMLh0)ILJ+}*Y@w>$WwYG(Wn!cZC$}~*`qb*1OO01F^x$hn5LW|
zP9_jAmt0@xNBuow7KY)G+%DeL?q~}{x(wn0wm&#$AmdCR!AgcA`wD@vjt
z0p`)CL`6iIflg1|E(R7f&YXgI_qx4vQ)|ZD{Q3HykJL?C3>beb6%S5Pyk^YZfGVO1Br(DC?irK&QX1(B
z>WpE#k#rOjhA=P{0^Z8ec=u;5>y_e3xix{*Y;eka0N3S@3!!QmO~NVJr_6=vvT0IA
zk@W!sYn{_S8(#mu9_;ArH+HWoMUBC`uH;|B0+oXPY3wDwQ{iBvS~Yj!p}d_
z>DAha0q~9eI#e593s0!)!z}5qU)2ow&gX>}T^|;FQpgZQv(xIUomW<6aq32{10pF>
zZxVVoEfpb4ydNg2<^Hs$XfjOekUl?6JNw5i~42
zT{1T@iL4InA%>7Tz*37?Cs~wnV6kWg~8iT^SaI$HeDBGH~iPoB_8(#*!|C#K2eywMjMBw&6+dD1^*GCXKU
zkl*#SNppv^wUZKm^NhY5=tl*v;H96i=lQ?aXx7uwMFg3n2afX_cTJ;b`+i9{m}i1n
z1ERwU0_Jz#PHSI}L3uaQih`PoY=BDifQ~6TEgXAfnJ&E8E?$;3r|tx7zp#gUNJGFV
zMQHY@rog9n#vkT?NxU1i@O}UmdNAnKm!+Gl*9Ed|Sf4{;xjlOr`AyRJo6WE_Jz0gu(TaBs6Db3$NW!d;=sJ;rrT$G;tKNv8A7
znJ#CQ4y$PTs>D5dD&eACLLCZPzL=Q<))Zck%(84HAl2G6ig4nWo}bMLN`u@7w2H@{UOC$L-pw6&`rP{0vME4
zS>g}{i+f0PBY)Mlr{*fdOCXX*mg-6>l-a`32;nU*a{f_*?+5@NhmAia`E%e^|T
zQ}^cPp!fqt*t+q!*3b>tQqr-TsG~<=B7~$zJsY&y6&zA4tj5_{kHFk@h78LB{g==W
z8B-V~DsS*74rdZOC~)4JBeIl11MU{+3MxcVXqy?kRT-QNO)H)Pb#dSlIE=Dj4P(c$
zux4%1GwZX!rpT$eR6k06i~WC@IH1!fE7_=R)%!57=6xz5?CgcPIiL_#KAVQZVceF&
zH?fq~GugYP>gyrM4_lda`y!Fta0f%C)z-4G7_c-G90uMc#La&i9)1!M03jW#M{xPz
zR&_%#;q8yw8EV&jsW!;r;UK}nxLw{_XT^%)c+KnBPsmT@OT`H0*DOkt(>PfE6ND1ndGJtX53bNDhD?ssZ)^8|Pfq6ug7TL0Lt}$~cvXNQD9G$E
zXUMUrGgdg5=SVV~Tya6c0YMc9*gwQwHMJcihOmyxf$f*IlEc*Lk&x6m&4Ym#fT%j(A>Xr)@2?%|
zj@u*TwUbuf8<2;4a;fUYiRwVQBaoqy;S3?^y
zdKS`s-BioeANi@)X|>3i;bIxnskmWWVyktD4>3edAa}zMrt>Wc(
zC>u3Bkzg-Bxv5d^#d7Qs8|&$Cg#mS^~>FOBdvL39laY
zfKON8Jq)JE-rEe?^2mYPcqZQss8>D@EBbA?ewlv6FB4n+%}(ot6_n`NueqLBTp81B
z`F@;X8b!wh0IuDB&e?)BI$!3Nx&b#?Wj)vs0H3egushq{%7X#|O8C*mlVJfbzW+p&
ziXivHYcaHiTflJ<4zZ=1$Yr!$>GK4tM{C^{;>*>}G#(nl#%SVhZsT=Zl75x>-xipW
z<*t_LxP^~uBt7n^Oc$KQo=NXjdx2V+%VA8B_YVnxWMg`#d2rAY>*}0c3yKvOHrjL~
z?EXF_iOS9GVDNs41Gr96`Li@4tF*-x=wQx+?A2rzO|m)udQ_Z&xiTVlY3N_Y1<6{C
z*BeXw9_+LtoMH*ZE*$zo(4fWM#r9-_FV^OHsGU3aWz2GYPs6wMGXv6vl8qWV6~bTW
zvZ5qF&y>~rsy!JW^J@`GSz!>vYJj3WVdlM^(eICo4f{E^+o^eG6xrRejFMHKgqI!q
zKgqP(hFSg!&z0cUp8O)1$jdNH+e?G
zy_LELN#I`b1m}f4N|3%>@iQhh7aOi{~-DC(Wj!uWhRd
z&17rEe5?675vgiDf3MQa&N{;AM0>su(hS4`<^Fg%n+Zsjm}`DDT$zav4jC*ChoaFC
z9e3X@{X`FR+)7DWD(wSdZ%{Hdt2TmlaGKNV*ukku$Y7f*#{Q9>c=?@yr9OwD(f6L@
z7N2uKjb|fK}f{uB4^h)H|=A9otLAjO}ptHSZe|c&Umk36hTzl#Bh(
z|1F#7XvN`jh0U}aCPooL*Ym5|4RHV(
z92RP)>StYqD=gP~%hqIB1xrxFGpv-ZT;FkN&a8(jqTZ9$&SrI#66P?0&~_922VysD
zBZ}kXtag1lmon$(t(VsiKTfr9dA1m@9GA=6pB|Kbw=&G2lZ(?Nzun!VO&`l%O@N74
zj%T;l3WJ{QZV!G)GdzuzMrFS
z_s-VRYu%$Q9x_%A>B5a-GFG0VicKS
zy4Pi0VH+aOhW*rLj#@U3+4ecGPB;|{&1TkQL$pVW)w}NiJSrT{9e9#@#nZnE4Meto
z1(d8EWz*$TZzGub4!agZ%a)L`vBMJx>E=^QF*mzrVdba5lg`xgkPVO@hiP43TmrG6
zy=+7P9buHg>cUefyhtTotpNKbv$2lGPQ9yg-+U0~fk31uHj3BcA*NH!c7n))=D?k#
z9_v2A8(6uI8rUm5Mu9aMX*B(xI5|?RR5H2@v)Q64bOt=Ixv`(k>4VbH3`M5f_+Hj}
z^3BVG{s*H5FXvXWrHCN5*Rx)>VaFib^XaGBC*^B*vyBr{b%01;00cAM)Ix`u
zyipI={2$)Y+V*VN%MRMCI!ThC74NbnFQrc+LA%EM=Uld6?MnU{?>Loc4EoCcI&`^C
zT4Ccy^FpCxUP#NgUU4+a!Rm;mLi7>6MGEl4H$ZEJyUavoOS~lH
zP;z1RU!Tbrs^mUe2Q0DCGE`0p?Q}2x^z*Pu&qQC^0A@EyL4at?Sj*Yhja$2x{8V}2
zLwOK3v?`H*A2;TLp|0A8ZFk()kC+9l@u;GvLch)z@uxJpo&_V_aUvG%iOL4>ACqXM
zy-F9*k!F}i86+UgyK&^ko}Ss(D)9U@k)9oi4-r9RBhq?0k*@48d#-k0$>)5eLCKY|mRYwM0Tx9Cih@DB%fbvLOVMit8+>RMDP55>xL)B8BT5wPw?;e)Q3F{*87i5cOn4WhWx^
zk((Cu=f#5>+I%q4gBe}(ulcKa2MOn#VZMG9F|M=-()wHpRu2h==G^wfcnr%os&zz3
zb8&PuxO2LkWa;#rItB*9ifUy#9@wr3GCG>>DL}JVmYSSeIHUATfi?s4^9nb^Iz*^{
zETZS|T;x${fgjQni{@HI^6fX{MM=dnkHDn?a_;zz!nN$bJ+2iwwMiU@{rxQ;&N&ne
zO>F4p5X&zg9clbj-Skd$i>Q&+%-~mhJp0vyvxs&+G^=h0flxo=N<)VI^xYs9!@M=#
z2H36Zd3Q;H*Wg8xeMSAJI}4_TBzMg_ph|F!1YUK~MV2?hbU3MPO)n-!F&|_`)5PDk
zw*i+FK03><&y_#751|Q``bNm@Bf1LGkD-2O>5ZMZV4mtT7(8FJT-LdD2E%|&L3!Rb
zeS3EU4rWrj&UD8{!Oa2r6fy>uE1wp9K(}XSl-bBRO5>n%R4kw)8&_r`zZ_Zw*-LJ*#F`6>ekvCdXkp~>aYRwWCh9Wj&}IlQhYHClgzVd>TX=>
zuqxa-oO*W;Z>9_Z8k6Gnty=N=s6E+&$D80IE;{krq+r2u=OmELOMArI
zUm92U(vWJ1C5?8|J03M!0Nl0EhH5>^)U0UaltSWkWWhM{CMgakIxnbqM3a_{8!#!$e
zY3yyEW|j}G;XYEMW^4gEOym|S86i^$X1yGZovhdmL>dK9q?snCnFhG~|3lV02Is;=
zZMv~-+qUgw$F^-dd1BkzF?MX*wr$%^=ACcOsZ({Pe{`>|?w?(?P*>jz@Xj_#xJu_!
zV;&1F>8HwP=^uV7EH4WOmn^?Sc)~e~=}C~2$5;4ZrCx^kB@YK|Yf%pGMqA`+lI6rP;4kO%lM?(yt&kj1!
zY{#_Ydj(BPOKg>e4hKd6>@=swW*>8;OQi0kODsO7@N05FhYwG>sypK+1GUcxgIZ1t
z$J543yS=@4dVa8kp0rs}kYde+Y&=@FX?ih(`ick>uKR&c;J_9(de?ufwtO=J?4
zMsp+8TZxUzDhON!Du2}cm&GSam90{KXBi9DgG()m3oz!+GReyZ3_80iR?zW>AE*ii
zKp5hT%fXU3)22rw(`5H~?npAvYgq*IVJtsi9E@Uib@Wf(lB$aN-Z8N2sVx|`wf){y01UoZ0Z)nvWQ(~<
z)UidI6Qm-3v>#xA7cY!}JkSBiX)^yi0$+6bi~&`JbvxD(FP^i;^vEL5>O8Tm;K
z^7oD8Bb#2hcZlpH-Lf%E-rZ2@_ZE{9M*b=3ht&HFX33m$8EyiW>eaI;X6hx**Sm`O
zBHoDU9}JIb37dV{548Q@;*B^nP?v|voaU3FrYp1g{4l8;j}slFa0zmFnKO@r!b!<4&v?atQn0A0{~60+og(*61Sx#8HvWnL2KX*KX(6jw}a`08-O*|RU>Kcc9TuoSbF
z-JP7nl==IBVO8faY1d)gr3j>k*m2hcW8jMuThLDRQV;d^baeB$B9~LMa>q<@*^fwv
z`k0vAvdu`Ma%X(D23mIQ7pRh7U-ADg+H7Gp|40Hiq;j+>>_8XEZVc;SI|>1Opj
zq%=;lopf2Dxd}|b7EfSLobq7G*eyg3Q7;`l*@HKK;4>&%&3x=6Kdh<#U@TW~4@zhA
zCY-dIC9cwj!^vIX)vyk56C<7AA#e_WcO4)XvZg36RF7juSM*!~x2l0%=^_`SWllp&
zv8wam0d7cy&=e`k7i_puh&EkfG)JxmcDHEutBoT{szmpxrNy8kR@xMs~WgoA446Sf(sNGd8v1no@+FM4mO0Y%E3n4mARMxvGI
zq!h}CCnjA1NkpD)zn3wmlyI4X$*6?q{wlI-M31u@(y=T|xTX)mM#x4W`sQ^=R?OZx6&CC8P@XC=GU@F9Z6?S;$PM}339AHVXxo}t@skCG{o`~2R>GJIK
zUvojYlcGMk#Y*q8M_jW)z`4E_ws09y1w52iucVP9K?Xe5y%6;5=m$4lRS4e(kq^-f
zLB74aOyvePtE>962cOJ?`GXwbW9ToKuJtL0DhEQYdgs*Qh(gdt7%u@~^Ysf!j46%?
z0OA@#7k4xng!31Qw}WBLq)Y>)`11~6z*`w1{+YC&!i@++kv7C_gA3v9LkrjL01W3<
z{LJ943?2Bq9WHPYfe)5-Y}x5|WMRle_itjrakKPuiKn*a0jwuL2TdN-W*g>5e@v)d
zb+58DOj6yT;N3ZDw{4rtuWeQ6Iqe+Mo6rA7=`tV4gjm{!I))3agx4S?O!Nz25`w_o
zMRsxtLeku9IvdRc=hbUAR2(!=dB*R|bn7C@X?ZjgEaw}1R+QDqU
zWLUE(2HBsBi=nzn(GVFUt#wDwkSYqP2XMMLEQ^ED!_we~Q3Mum7>c#PlG@XriXMwHW(T@AOt1n
z14F&adoe*>n;UraT%%E#L?lL@={+3&R>?$IPJzm(c&SxA(H5u-b-t(jwo_**((jhe
z-hGbSoXOwc?*Gi9{>Kguj2w+8?2baoKOBuyTO_Ga`+8qU)ZG|_as%MqG{HZZ?nW>H
z(8C74o}WTc%N3U1JbByQf?Ci0=&;LWu!Ox;>T)^`7
zF9rjh){EeDgwZRZeEJX@i1zA+@y?hZXuS^3_^9mrNk0YV01$#l*@YsFU&ngYS7Tl-
z-8M@>U0*53^QoPu1d&2Aep8iKBUCSie*Ix0r|0T}A3N=3&
znv>vRpUNmMptcQ3Rpm5>#8YoJQa-sOX^A%dyL8R8m;9E-WfM9ZC#=8r_*l(Ju{>2d
zhAY{qJawL~ea6l2Ci)NYd8Y5Ws=KQeM|5n>0Xt8`0_%$qYgKfmHf60|u7Sd~K4$ah
zGM@*Q+IWLC4)3#7qjr;h^}@5T3f9!5nu*1OWkvV_U}{Mh{=o6XJahG!FqD?peo^e?
zQWE}Q2OYS!Xn0t)1Fy&EMX=}{r98-&Cel%pB=a5G(C+NI9qCYVLaIX@X&WZTUDntkG!7&KVZ4#EG21wK!cf9M(gR
zM6+Nvz}Ja`Gqe3>iz1{i@>Oqs}0+U#}iVR)7XE9ORBK{k3vHY3A
zLB81j+Og%v!UV?=WM0b%CU!_n-@C#FHM6P)5NI*x2<5DLST%5O6$(3HBBakDK71;Q%oCo7vINImP%`Z}LbZ_%8
zfSpi2zF*ANLnx-um73J7K|{`by&B29FL8;DDr5HtzaB&0;uH)n$Omn|=Vs3Z>;}l!
z4zl7uR?~2=WjNgu6ka$4S~H{kip(5e^)Piq54S!BBc&F@Tgc338UvG~9qLY^7QwLlUhq{F=c#rJ`*nvcnZk4Yq&!Yr*%
zGlwD~U%xiRqP(9YGDtV0g8oE_RBi#4Kyt8r33jGjOQkp`2QnphD2!03
z9R-6obCP4oKkL9|OY$!JSH#s;#T!?I)s(9la96>#+hKxrR$7q8J*^lUky;$sGUFaK<`O{8N{x|-hh{g%K51B_hX1FbmICD5^4_UgV_ah1wRO9v6}Aw
z?`$I-cp3y1oQQ>m>;HUHUDn=q*yKV1+!_$O8XVbAMH!H|MGI{CqwD4StwF3A`-m54
z#uQc4V<&A5MgQFM9x1jIkI7%@`m|$4l5h`Z?%#2;HWe8Xi1klWDt%HYp^1rYKsHr)
zF`d1CA14#e=JEZqD+N+W<3;a%vE(NuXDkq#1rVjpOo~tyXAlPuP2YwO8R=lm#vL^Rhd}Cl96`nauZf
zQty4UQFd3=cHU|LTa8v5&9#g09300uxY1w9#FUp9x2%ZC*J(1nYTX301DP45@wYoS
zaP<`v(V*Mz0A+-KA(YUP&uZ)Y@d7R@c?nn5^;3~!l?s6qFr21MP4q)_vzM68!r%}H`{-6
z%)F=qKJF7)SE}&6f~iP*kG;k@qx%lxv-|Sg%X>w16g<*j^*`mhH7j;^GSRN}Ru96*uin%3t=Z}+Jb3^-@B@KT-{5YiNgNm`(Jze!;N|`slshG~yp`9<{n6;?
zsh9WHGo?knwzj}lokM-W1JT&o|D^`jc^l98{#x<@`~BEeI2Xyc+_jpPT*OsgXK2f2pL|>ckL9w0T6%?rFZV@d=J(<8a`FA{r;v`
zii_$Nz0g?C#TycN8=dfaKkP3QoF#xPwX2%vseihm@lxzY^z~K~mUe_HPMJz*CuR!?jjGIspK(q8hY!I3`6G@Fz@2s5eP|(iziU(-}c0nQ#!D0K0^rVDX*j{0UJ%ahM8JaHZR(w7mEG$^g@)H)!
zxE{Pn#52^rq84qgLX&z}T8bM9t<%&G&idp}2JP;})NN
zys5cvp%gG2d9+Z=3p%TMFwoTlZi$)J^-irKFTAs|QS@bXE5ukRyh4O3CXWq4Y&(GF
ztTZJrrEHjKo7?D(`KI&f#kxE;y33Qalxpn5h&!>Ol=MsE6UMwYr2S{~H=r|5GeIj)
zJ9vv~Y7*Nr9Tk!IN0kh4p9IFo-q=&-U}<|kJ-TWvd-)C_@6WX*F8}?m*|(C4E-TqT
zT0S&iRjZRu;Xs|R{YuF*5Ne)S88-kt2np!jenn&*@@6Aa{$Me>0BeN8i+lTrJkLH6JDvUBIQ3yc&ja|!t$sotinjf3IOU?s~@;;)?
zBTe!k8--v}YDaG4(MY@vg^V-}hHHl}WGKVgf^OEL-r-4v9PtzG6%IPlM1ahf!pWBN
zHXjA(h^5puQW-sIcD~R&6y2Jdau+8pm_M1T4=F1Rdr30FP}Mo
zB>eed%3RRA-lRPh?r7CI%vj&Ouf6cCDN
zhCqy<3ObmRLw4Egw$($OW6IvEZT6G5@IpcUH6dCBDc&<=cD%}Ec{c)lhqC^=Sd{Jb
zjv+SeBtcE+2NHskp?V!!j&QI7(%N~fj_Y+*12)?G@;KoaAwqryO)?cskSYXH9}OsW
zFdGvzSs#q_8!@BAel_}6bQ8&Q@C)hoLa`vQY_bQmA+nGw#^az{`q-7#w$!GJ!LWZE
z=`0A|!FhtNmhHRX$nOH6O3;j{TWY}%H>+PWbV&Ah)+)B)Ee_oGZ?F%?gBpR>_vjIk7$O*W!a_NDs)Crc!&XO%$a<)Gu=dJyNBs|Yi7%d-Rg3oDd=i*SrBzy
zPp2YL7#W0OAnOywgHP#7wb%D+qp^P6Fb29a*uY$c5@Ty!!Xzz>y4;tP?qIc_7YCDm&h<
z7h0`rS7>IxaBOXK(-XRva5La#n8+yfdtC#3r6BTVkF5kay}3a|p#@ag1gq3AgF9c-
zYAE@S@VJ5f@=D5#?)^Fe+;Fz{KLVJTeV-#n5>ECObd^ya1(NxVoPD%Qwv+k?I7!(@
zuIso@5ced5|8d{gT&NnLf-(}I_2V=4l%dQUB}YLmppGerbCiU?sP{}_q19}mfbJl+!m`E16h?6M%2x<{8<=82a^R?nfgk$;)P7#Mv
zB80OhKZ*%#YRyXpME8MZs~;bHYwiOE5aKnwft>EL`$f28~cs;AqthcQF>|Et__b0ya+q5^aO4@13-qZhx~
zcF*5CAg*spWo##ZIDZQXo2#=~f
z%h1{wKhIFPY23bR`s_=rh6mW54vloSu07?OsQYEa`8s}3a1e-ah1w3r*kUmfhx+I?
zNSHz~kCk-_G}6vq4`MZ2zH$a{Y|A4rOT4}#RQ;UG?DjlB^M04
zDdReM1XqyVmHEeBpJ=RJZojXi%|7fflPRa>sPFgDm4r*7@{JevhuSdG6JhY0)08Vd
zFJEXtD{0IGPq1`{WuAzNbfErS1RMXm-MU-8^|Go&-AV2j@!!^RF3+EmGO){H&(&I~S1I1s=by;6XdbrAX3x~hbV1#V49W1u1=E%5EqKD{zQHS1O2-v-Mr
zF`B}PExx(d7>KOGuJt1}*VN`ex^<;Ddh<0_$}Q`2b2K)M_#8zQR8H2Lo|eZIn*k6T
zZmpzE);N>5n>@QGwfCz1+yRz2)!1Ku3M3&Dl_hmN5+JY0#^3>ew(1ldMTerCklKLm
zP_K~kt%&IMvpmW;kg22JEt4PEPD8+Qv&nQ-PN)4BK?Kh-gbY1iVmdYp=^)?+7+2Ng
zb{}lHwwT9Jl5LlIMi41z)~JoC(OGl^bGxzJd_hfOh+#(O?0!M~vB*TEzqQfL6V5$f4
z-8P~G%scTzpqoEk4Qv}%7#*PFV(l+%)sBiq*_@XnxtcD0Tl<+zq!$ACu8=HWba2YM
z>_y>}7b#U$02z|=a<-WVKVyFbuuzZ=o3Of~S3NT+!;{
zz+up8=N2tkJnt{UIsFYM{=+qn-#V?cQ6qg!ovg74#nr%)N9d4M3ZW4)=gkuR98v<1
ze+6Qo5J)S#v=AUvmzvtFIgydE5N1puB;a1y$gG}2_)3vdppzuG#SwMo&hXBHl^XlI
z2r1dd>Qn&}qF4*u??4Hr
zQJ0;=fP-GkGMpSYq}y9$Rxzh&{i9&_+5#&_$vq1&DGd}BT@j?PA3#}-9wnqu45pJH
zeeaD^?0e;s%Tbj=%|SC+5~A{+8V18NG+t1+vzHm^p%jtApsG
zvUSig{tre(0UTOqj}@W`PMkY%$`otli^qtf|JJS8mtBhh2mwS1QjwxaeGiutmEe#r
zM53aIi|_{0>OLjDLB17%_hKBs-DKBNQgsS2gvFPwcEY!r{3EzQ;FNZ)WU)cu_TO^4
zM)16B8_zTP@{h5eU-FNyKtALv_-{_o2(Q#11@ngymU4+``@A)(P%2y~l&XF{;q#E7
z%_pR-11xp?yWVKSdO2aPBUz=NO@Ym~VK=u_J@o9t;bTNLPrbpcWO3Bt{!USOOd`OO
zQ_tLdxkx68rY`F~&1}-cHlBNrr?Cz`>sYraP;X8t{!Hw!r3!~Zv516Vik?1dr4m6t
znnb7oGSx=c%tj&2IG+c$vWzrrjN$?aeEPqu{w{}lr{jfmdu2DdUjcC~Mi+-0CFluG
zOf->B(?Niy`X;@adYy&|hST5G=05-<$JL3>{2BCA6?7yI93w59&%x84g#%?T8gIjW
z8ST^csz=)ak=$iv{JOhu3_*c_wt~C1z^EU@(Bvg_`7J{UK#d~MI)QJ8Gpg=9ga8pc
z(gzrazhK38ADTvdU!o}g!Po?Xlc|P4<}NkDxkCD0EOt4FL}~n;fTI#JCNlt=?ffXsB21)((
zFdvc>^6XZ^@^5#7iXy8fjjM2~6-?!2?{<6iQumeHOX!hog!nqGyXV2{oe`@X-3_6}d^U4P
zG((sdf*&3RTH1{qFq+V=Qb)o|z_1bl7T8no~q{#OL3@oGXy{sf@H|_JJz6%IMfN#>ez|
zY!H1+uO*1NXng2CFYwTXHq`ti$TwqwWm=cz|41CH|D$Q=@d@A{B;KcW_8;{-tmz8E
z#-f<+e*G)<`h9&`gav7hdZ2#dv1Iwe$>}AEP5zQMS2K6!-_q*&>|4e?a54W~Mvy9U
z2h#DUf$`LH+WnF-_j*4|{=-J*iU6kAYN@2dN-VRb7Wy1Vu+d!?PJ)Xn*#>>*m$)
zNm<8EK2X^`d#l2DZMO6Fv)#e9RXI52t96>mYq+R1K^W7p@n1^29oZNMp=tWh&!?5%
z-=mxgW~n|ZY!9W7>CYQyh)|DPfR}4>=cbw^`}K}NS_gpPxH5;Uzf-1}7TAyw-8|dq
zT`2PG)K@E&RuUbV*rT@mJRY=8FD8!Om{`ykCYSh#AiZiPUXL$2qcb2tzR`_QS``me
zIv@r&KLj9t3ZYjp{;enrYRIw(x~6_#9oXz!=-F9
zU|b^Ao%6g(vT}}ls=E1~(3g3UUo9k)$UyFAxKPc`fRee?hnMU{&Zp695D)ZTK*wBD09x=NWg5ks
zHs01t|5wPj0i{W*54c}Z5%J@bhp`-}sp9Mx;kNzZCbx3K?^moDJFi~z4Rk-^`>~eL
zu{mO*YI{}wwcv5H$Hi#P#g`rBa@Rkk1v6Z-W^UxVT)Zh9eCE!~?Y9)9XC9PHG~K3+
z{sSOq%jI$?Q?+^9?=moEFi}JibjYgJrRC1(@J-rTilyGUJ(+F(Hbn8vvfEpEF95r-
z>chsec2Tl1N1DwxQM38e9Fw=GMMRMLBaBEaKm$jbTzf9~D%D|77(NO=Ob2!Pk<`&o
z33m?XDo^o)9S2|fbN?{h3;c*8@%6tem&qfJh#*WH|2rTD;bc#)XQBoy<80b*a=q{y
z7|ALL?TpY-Cl#^<3tZ8aHtHD$Y-e^g2aD8W>%@|KnYCxWT)rlRcPOT%axuVM{Si|-
zYTq^C(V0EFTBW_n72MVB_Mcp$BbG*!Xw2@!CYOfGJByjdIu#SYr%Q?o(Vek>SpPKS
zMm#hu+D`;~eFR4=BI*PTT6E91QBBZ{s&Ric)hwu$!n$<2x4yn@ou0c_R-X83w_UTW
zPV1MvW^=<%T*G)Q4A8&0wd2ky9l!`^)$C1bz=LU!_w1Fz3CxzjEZ6~B23B1=Js@x3JkV=e
z>mZNl7U^>*8c8NqByn}QhH?44&QjDI=wxtkAP)2SG{^gGv}}7b&=IY>w-0vu@!h`q
zIa6S27G(}kk6X4n&Gb4P-ZqrJyu0Zg==blIQS6EEommW^`sK%S<{?Z5b=XW9Ohafb
zBI0m!1Av6>)x!W=A}NMAo1)sdEM8YjV0mfBJtNhfqV=vg?5~$6`RJHeS9FwS>R1;S
zHeYb+>fBQFV-sf{fVC9_<$Wn_9JvEKFz2&26ww+>5ilDyb0S#9`oJ9ZGY1JrWw6_u
zJDupBl{C_|j@hempHU(8J@*Zyuv#4?nd3dsHy%A++ZBKS4s*%PL&KCu`7Fy7&n5pE
ze~%8K2P9myZC|b|8nRLm+Y-<#TOV#aG!%6hLieB@Hf_&#+QzIEn89>B+n52;@1^xAgMzSKA
zQy2v#PB{Rg2EmqfZ`M;H{Yz6Bh{`L_V_gj-k|_@R(oQ|K(n)T_;AyDu7{!fNK<-qr
zX~-~f)WfYfbPBE+NB~4yJ^GgLpOdqqyPjos2E0z<4A!W1U)q&tbJ+z1LCD_dlz74+
zhV4{Sj6|{fHQmIJaV#)VF%m_*JOU$xCHb}#fhRy}!88>Jd@!UXHUGnWrE^iW4&Q}o
z;0qSS1~)SVIc*j{4pqU`n3R{uz+5QCB_E#d(VTROHs-;MN)@tSf9yz$WkP#d5#-kB
z2UOY24voiPDG%fm0zQi@&ReSdt6AcXeKTm@v3E=kLiOXe(k^HxoEqsOia2yw@JyOc
z@Gam|Y0PR%GIV8G8a
zw)gQqiR=hDF#)5C*C8g%OQ=jS&~DhUY^OVqpr4%d&hB{ENJWZ-$+?x%YVRC5)u`tRn3M~
zE|J_o;9zU(^{=93j0KmJU`Lx2R=q8^McC*o7eQ=MH^SQ~7B%zD$!=;#h(L+u68{4l
zoCSiJ0_~znK@OxADDeyssz4<Sii8T8`NI#)h$lcCR
z?E6TQpeab+Fe)Tasi3m&q#0nbP_O{Vq@f@&HfY}^`8!auyLw#;lsk|M+jTbdR;%ld
zc|Q$tf<};j1v0yY2??-jHgof_acp@eWcs29z
zz&J+}^=hv;aD!QnYKy2n^CDOU!ETuE3`3#>v4jweJr)`iJA8k2;WK1AsF~rNiB`#j1Iwr4ljNqgcyLuPu|KIMrr{u)#Yxo7$eOTIb92a)kx{
zZ-A{i^j>Npl`t`Th^pCSM;yXq2w8*OfK
zsmtF4`rJ5P0lQp<2c0eM|`
zwIH`9mE4XP4w^+O;
zP(!XMY6@r=0u$%!#x#F%W2{RPQN9pq0bxZnOeKm+(_%Y8$OtLwLzJHNWN2Du)5wz5
z9J{U$l~vN&c+FzCA=8Nn-VmxLe{$RBjQQVwZh+ldlrJX_ZP+spAV;_W=z$iPZuOIPgF6e
zdamS|ZZ}me7736Ipe4q88}uEUMi9#FHW*Wc7$)BJ!|vphPV3CM$%}RO5RixQrL`0i
zJ@P5Jof$zmNbz$TamP6(DIGr{>=fGpaW8M~sX2F>mW}$jZ~p+rn2Ht&Nv%Ot+uz4Q
z8WT+#YoV?vAEbo~I*A{?yw+rftd-20x5vZ*uaw
zt`WPHeP))+qi659-1L@B8BKZ-tHXdintX^vWiz!F9ldqjI5_QS{2vhf6>U3^j2MGFfZf&E5q6ttZ2q;jR1`2&hxU0){F`Xb8K*_N=
zP#Ntqf8Zk<4>EZaowLm@#^*7Rj@s#GxF}MAO8b4-2NhaKri_R9X{f85oTsC%v@);Z@XE&mgtxjLm<|JpkGyyS8SW!DMGks|Qo@c)$?kWkqbl*==fD!XK51P68#
zp1gNqf1lv+GshPvd&E!ETZZ$_F4LVKxi*>fV%Jh
zc>-V}$gea&l+PG&zqu$;at8DW=85qL>l6?=)(cjx&M|C8hIw)Bm|?c!NM+rz3VyBZ
z4zYbYy-Sxqw-#6(W3nr)OG$jCVRx>MAZ)BSBt{wV_gc2IVYc#1^scO{4ITgtML33@
ze{k0*xLNBTi1%2ClQluD{zwdNWy}8YATNYhXN1IjB>t7hoTEq4cVaO@#Rp$()E$eaw
z;b<-m>}!4?WB83al8N6@L6}+ocd*-d;aCd|%=Vv&)>z}s0}cE?A#meYuqZSF$N#4^
z&GG-VV(HehQ^1u(`nfTvC)6q5*v$O9o(>~E@kg4p83m?%DeWOJV6#xn9juh(w`UfM
zcWd9J2`d7uiF*KLMKj1Y;
zBSyaT84P{cvc-2JSJXK~2=eU?4_;
z!T?<`LT5K6BTm$sIvu7Ouw1klTJ${TnvpA*k!2AC0E(s_JQVg7@lz0RjRfVu{D1IL
zP6%yb1Z5gY1y4D40oa&`wqZ)0c{x!q2%T!%AK-a-C_8a9&;wE6$;dkBLL>zcRLwdW
z7ILBnAUP-%ONeGMEWIg${Yv8(sKLtEAb?!;crH;K@fs=^16FE?6r`+W3ONh2ka}d)
z086f82<{yb^d|h7QU;}bo03WdGO8>KQ-*0d_Lkvav{+h|dF7blTqey!7UH62Q`}I>
zR5;L}DR7Zu)5D?DY9kqy#iO7|+ZiIpAsYdg>uh2B(JB(+<(q?%dw&PQXnn
zugq9pYt5#h0bZxY?A>#*&+Gf*k{cxDuPo*E1^uboZKXO{vMG&s&!BxwEJz~;BihN<$M2ufycX>|;sv}V(o1sZ%!fVN&q1UzQ~i4zth|Kk?Y05bVP9Su(3C@i|^xo+2!!%
z;PANdh2^{9!Zm5dOvYL(eY84S-8JK-&l!Eqt#EzSO}$3I$8z_zTcyvUqyFK(h2E}7
zEmWn?5o_(Ws8d%9;{N>xg>#03t+Cg3|M1t7V~3-p=3=GaYIn9i<5oISEZ~AHtW#va
zAzQhP@>Aq_GCz`V{Fs23Ujrn`88mFRhAwK1
z97m0KFg({ucmh4H=_vkq+_mW_GYI)RQZsaPYzdZ9bRAV-cd4Pwyg)#bI+ClWDY
zzTV3bW|mwvuZ~Q5nDqj0d{_rMsH5)?5Ax>j;v0mc(_qGeu;bLnXA-;Pv@H7Zy^8nB
z-{*7Wr}xvN{_V+RRgXd59AKg@$lrOfGb@Q=yD)@!22q+FA+8tcn&e6L-
zQwy{(o3p1*Z09Bqn@h!05o@M47;ZpPF3?uET$Hk5G3U&3Enzd#YGC~McRjw+
za-csvp711$yMU@-Kz6qBnC{g`yH89)jY~ppM?!rZCneJ<1%PQKY%@!Fi%1j8y{NM}
z$82pT3BI~bT%mNIGk@BQ*DtYkP?y;;^p#AbwX4(hFli4nPYa)0Mjja6yzIcBTfLxl
zK~1xbPL!0p#YoduUDO!r&R&N^?VTT7Y9hVAes;yLOss}JUJFSj63F+p=@$UL*I5L>
zN^k1p!_Nfn0CM@)Ca>A;ur4Dc4aA>P+lol2iaV3)TKjpe26mWQqYe<|-wAwshq#L?)yqknCS6P&|Cd7_wm4E#^AM1d!>(<@7TwIj|FK=p#*rX=i$<
zUT$n-G$Yj00vEC`XKnUH4FpNi!!y~2e~dLKNsQLI>C(8t1v|qF5Xb74_}Ub
z4l31&;fplu;r7olxD?^O@CxI5xv(Rw#2j6f%>CLMJG=eo*3f|2()Wjf(Hikmu+4xu
z{*Nv5o@fWm^*?s&ojLF^0tgq=|Ljm$x!C?+AICz(%*4&kmfoKPf)2*Q%E6gj?@bA4
zc6ZZ6+iBs@16jZ)-TrS7Z8P;Du?{JcNc~B7g^3{A-d-Sw2LV~2R+f2;{kh@xo^iR&
z+Z|+uyPfc;?q1oM6I@&{OK%C)1Vtw@q~hg-OotMZ)7yr*$cJlohFEuWslNoe=Kov0
z1-ymFr>l$g^we-8L`ww)lGYdmA`SrB0#SqvV3=3r|aE
zA3eRioYcKO1vm?BU32UH1ppowU4XoRx`=P&f!11i=dj;8nR5+wG0e!zxhla=Zn;LR|{=n}UgM)|$
z8q(ImPa#=@KR*F!{-pH_2Su4Fsr}UtFC!9Dnap
znuEVQdERR{{50C)Lal)w0YAS{k{}sEIz#*z-`-zN3GMB&xLLuQfqUukyR)xd1+Fg>
z*WCng=L^j3uJj&APnJglfrY#SZPm@za;HLWPKfC08dmW$`8O*dfL)@Z9o$U51K`?F
zu@2y0K6{(oiR!AiYXK^C&#uS52oUXFAgRYcqmUE>Z={WoBEY&p`M5x5CWn3Rw9mKW
z@8l;xP{BG#hOiBR17QR$
zTv)wDw87UzK>L6pCW>F>?^!OUd&j<)NbHuT7NojH#y$+oa6o@K@PD_1&wpoUzBL%Y
z6)Y^^X)%cwe|Oe?TUwlfxj%mxUw+LjfWC&ar9lsFjE}zzcoM=v#vm;#!Gkuqz7~%F
zN53IS5SzT9z*k$oc)DF6e7}~tzZ)DDYnp>K6mzwc*XjCuy{>hU%}yYo{Cq&Zy1i;b
znifCN|8dLy04GSt3tSq@@$$MK!ojCaO#)lcdhoz&V7@O9AX*O(S_3$SwUE2JN6@b}
zqhJTvr+0o4AbA8PFc=r$HgAz{V0R>s7=ux|KE%D0AMy`KXdo`LA0lC(vMWL|u)UN|
zPLv}MN188TJ|OuL!3fO0@-N7K=IZ_Fm=
z`WC=9FJF~Zy5Joq@SOpYI?53v6zS~;C4sFnw_f#D0THc*PQrFxh5uyW4bhmv^nO_C
z{tsqI>i+#Gu*Bxz`ov)^PrDIwK0p}Dt+tmarumg{4AkWMF&q;yrg_#t?jzLiH>i$A
z|10dbT}-=oCu3KGVl}V&BX9JTb0xGxGzSm>L8PA}VA9^!s-?>22SCLjnB4Ktywp
zmcjR41jeK|7a1#9Kg2>kjtyv;`Wc7+a-;?n$&t@pPE2{ZBeyo;aq^Ek^H2Kd!T=sf
z?BvLe0VzaxKB6CrBB$ZYX-82Cm@OO1+3LU-z(;Ej3q@WZc(cKcOPHX4c&##C)gEAC
zqTWjzwNZ7-fo3HUO+z<3YfQW-2%Kh>|3d?HG#(PWqs`m1p|<|s`(ika@r+?o^6nMH
zuY%n(t9LKtX_?c(4n#_*DeC%k+yPqmOrpqc3o~09+Fp>1@njDPeTM6_D!0~H)n0Q0
zLc)RukMQ){acpx$+8obmo3D5~u$XaaN&Dv!(={U#
zLr5FykP=0??i1V=VB#ae39ngabwT|L5|DkD!#_#A#FIGuV&Y}LY6x;oU;y#oiR{m1
zE`y1zzqm=yY1-JtkOmkl`;-ECwRVZ;5b%_Q_S<4C^;d7qn8pX;$Gg#^`3U@3>wbD~
zwu{BTQTJAjt52kC*ZM!3FQ!p5#$P>}FD=F*i
z{uff#(x;C8*1G+B!DJ@B#iSyIDZ~SDhD}A8|7IAOhg}VU_yfB1yYs9W+0xRew3*V^
zZR-}NgHTip@*p{w#%-3@s@sFQR4sVc>$sCZtfUDEx;1a@Y`Hz
zNOkrh(ObUy?~3Ua|CRvcbChfJi~l;bjyo^3YW9D83qRpD$+7KuY|dPGPtUz=@di|O
z;y{}Qs2)S^tqL41IsrAP&PtW6e$Ds8l*uw-?sFKKLuMY~3K9r=vl20_8i#zY5LF1k
zcRfqWE~ZTi4|nny_y*)%a?#wNTaJ}WE@k`y2G$EtZsp~_bcB5C`2XgmHS_>qBC?7r
zEXak{OSZ15L$N}`t@H0DOzMJmgH1+(e}_r#;Elk>6UnRJ)B!AVTPbgfnR=SYjMc8Z
zBG`Q`XgBVI=edc(aMHke_bn;>g-$Z_;den~EO%!hE~P=iUH*-F&;
zRGu>fz2w&4qn<#jugvWW%H5e1V9GFp;p=IsFU&T7VgHy=gfbjGTW`yWS7?&`*K&k*
z9>u}9_z(79lOKQ_z1QE;zsiDcn=4o-^`X&u()adAW!`7jHf`16M?hmW;R3dR8O%SGcA1@Z<3%!Eh@eb#nK
z*B}d+5M|l+)#3Df@k~7Cwda8;Zx^X_Z3)hEQ~}OxMp;X7{ez+%PVTO)0vo)Tv
zuh}F22_m?_rq-rQ!`J^R>>Q#riP|k)Y1^u_ZQHhO+x*hDZB*K}ZQHh8+12;Iw|mfo
z9`3W=v(G4E#fpe$Z!eDZ9ei|(^x8p>EqHnp9(OMn-!fF=nuGF
zWTeajxkK@cfSuMX*ZSc>Zu~p|@&||ov4WfQeNGAMN4JFk&OB_k=)Rx%C|pY{VJKn(
zE!LI2KU^Es2~2v7{@7TP@ez%xcUX)Tv7-KNHUw85#*fQFY#Ym`!
z4j}IlcG`UHv&2Bl?e_$QMz<;tyGqIVfo0g;&R&XGP2n-wDf`CP2!}PwrkPND5O7I;
zaDLLWZ&{C~k@}}u?LknP2|oQ6mlbtziY;vTZjI(wioHxeGeX4>BCsS;Vi&ph#4$?&
zVgtNVj4F8iGH$B-lIkXDlW68pn&Us{bMtPZ~fcp0`pFbiJp8wQBE5$CzY!KzoDZ
ziO!UxsuRP{TTL3d10&!^DiG5Jv_J8FG9BCAI*GVeplNJXk9g9vgKtv12xw~G
z@|)IcYx6|_@~MAj?J}9>1r5)KNxM|elwZJbdGXO%HdOPtU4Z#^Bnz1g6LkaCe4b&n
zB!ksGcj2;1g_oxv`5SsaV{2?JoA^K)DoFFU;0g%Nit;ZzH9L7B=M?W>pTCq~VnVo@
z$UuhyX-__R2Uap~A?s7i6=suPQWVhnmPLG?kl6)bi3V
zSG_6^Om!p9)e@MZ!*NjU6UiWW{AE5>#)Ft&-BC}4J-^$&HSnLsCF6k#GyRA=qvtF=
zxP+(JkGx1W9}}A&Dp$|h9j8T&o_)(WgV<~L!@X3OUje0YS(Jd=2;?#-l%*3(d04<{udUO4d-SSk>YB
zZ!wS9w~Sw^sul3g(X<%ZvnyWXZ@Y55ZqUnXU>dBhzk){7lHqE@h}Xy5D)HM(M1Mt6^}DE>i~fqux%kQ)5JG=OwRRM21PL`6K?(JR%LDlyMiGg?{nS%
z*rs;8Pw@F@BR6`1RM@s-XGyjsUu$r7{jjrzh?~!v)_V`p$1OLs$x>D=d%n!Nuu^N%XvA^={kGoZT`RcbjsZ!>_G3Uk
zg4owcg_>8;Qs##v2|jS@=H$fEz%2Q@QnX&73tMkg7d|1IUi=h=RVDxNS2pLZgkdt5
zfO0iw4~=L+!OCk@KLB2G?A-$u>YzH-or0BM;L-JRb3YDm`#wV{7&wiSuF4rhH?f#4
z!D)fq1be3d>Lx{!jr}QCwbDX%=VPFY*GBO@K~O>Xsf+
z$eWioRO;7sD~>BfF3|7`U6cD7)b9IJ?}E}+sbk?+LQ
z%%)q*+Xf~y;#@pH+YdGcLy~d97r`WV;0mFYTfmMf{UoEw(H38r+2*?U+@mAG=og;*
zs_P!3F-dS9=fhTdWBRswL;R0;ys?yaH0=hLit8fDCE$91&qvEom)&z*ed^2zPf=Ky
zdNSAo&14vB0kxe~d{#g=WJp8_d0%SfLQX|vo1P5vvyn$#`lpzoJ-T`C5;vk^k7iH@
zh-xbqtv^X)!>&h&>yeZ;l9ZTuiS!o=Osl+l!3MRBXf3ECuN|Sp5S(htFXjujvT>9U
z7jl^F54?(8N!P+M=I!*&>lF`4mF|i024KN(5kR!wX-auZ&81&+ZAO`Kf7|?+Naad4
z3Tgp73H#_pL-Y#p?u`mQKAXHITEcM+sRnD$x_W2N)HD?(JKz3v>&=;hyuskd{F2~S
zin(kKF<(vPs6hTN`2zBmVfl)zFe4EaXZyvqrj^!j;!qz3|NZ+AJuz3r2DNBoaaYJK
zJV31=pH}L8ZbLC;5di3Ao8+W4qTI&h5of3+rgZB@E3|%29c-KPrqhyt2KwQ_0@RJc=B`RJ$LDVqx%fm!`0fM&=PsI8)6|3nuVb@qDQ*
z!M{k--fSEMsvfhkKUz7_&&Avi=itom9x%?f^02f|FMFQgBwU1M6Fa*ZcXzEoSKYp;
zsG;C(tuJNJtnXYGtteXFV`M!dXGeM26cVMaY4<&&S+tkC&VVjZb~VR5JP_e%0s@6=
z@J_jt6+C_S8*A56-_!=%^?j&rCP0Y%(QLzVYk(@YX@0@nE}*zkz9FLKf#qfW-=iWJ
z!5wlw?M!v+4eikDxNb10kLXrbDJyfs%fEQXUNb%aaT0ZtCVFUJRJ)scW{}!73yF;D
z2xj6G)ezp2_Rm~2qw4z8*)Kf)U;|t)J9)cS4A-^t9Y_7dDekmS8>%8wWsE>}*t1Uw
zSju0nl7{FWG4HHTrQT)s%R36Cvw)hrQ`%Cw`y%1r(p*zxu5_z77K!Ga&@J|jFvrik
zYKK93ePG%2rQk7RQq%~`pP!{CQHwM@gjVL766UQ}p9}V`Xv1i3?3_2n9u+FD*U<%U
z*P3)jFugaMUDkn)=1CBquv6vB5cp|TP}q4%9x#sJJIypiu*`)k!5hx@SbzbUd+cR#
zxcxDc-=`<_97631CEj$W>9r2HMWJ#LJEpinL45k(+u1tgqn3zvc6kJ545SGho@H*P
zXYT5g4j%-%SY{sJFd)*)ttWxXj_=1Nx$HEsk;osW{N9cRF7+v=OK9C%r^j&htrJCI
zh-w{pQRbYbl0%Ut9pjWYp#b&tEVjgm3^7>UvpbR5CAEfm`y;RT_LBTEUy+d$9LpNS
zXYk^9%KDIC)7iaSeoF3(+WNFTY(uQBrWxtE7-$u)6P+gzw;oLMpZwFp4(vm32DQ>+JQDtQAk3
z)x7FN;Ux=n=ipZQevP__;?vKr&7*
z?22i(I?z;|QyCm)DIJsXCGjEZjP91&O4m*^&AzH*SZ|dPn@_yjGTNddv&m9RLc^DW
z*HF_s7_R4H>aqZr^IXU2tXrdWWO~`OWKc|eQ1*o
z0sE`(cGI(&|8Da%uj||2)!+Q8VH_g0T-8rzMi|43*7C5#qd`oJNGVAMSeFf*e3T4}
z!{$+;uIa1v>!61T8B#o3r*&P@Q{wVdm;}G4D%RP3H09CwR*94JI@6q4{h@Q|A?<}k
z%ks)7ujGV`4gkDJG06>EBFu&C;}Ve%%yQEomTvz_3JvkVoa>E)ld@2VZb~`nvz;*h
zD=z-I+r7m>e7qYF@f>uGmXofrGjG^3921Iuur$v}coG?Kd#oA|C6uSA<3NE@4K08E
zX$w|M9U4-Aa#Fk?Z>a)}ChQVhE)w9ou(|hw9G!2v2w;vLvfK;BE1h2pp7&aIs*@Gu
z@^WsBF?kq*WxT2TG@KhJRyQ*W9x3to?0;RE2A&nqdNWFDvI`p$uApphh+crUXxQ=4
z{}zKs_5SuIeSF5rWXCeW#4-F9*18g;9@b;7YAaT{-{DY4!uaecsJ4FyR1)vXN-b7bL~d~#99HD`
zj?Be>V;T(&ip5~=kux*kC`t_>&wLKjoL6EC0;qI&kso=878)<+%KM(!;qehTt(hGu
zzQIQL1>)$W`4(cka@{IGUh=?t%fvx>6CIa&g`2E~ItJ7FXH<+fIc;r8cyMeb?k1M@
zHTa(=_0=4x?bY4VHUw{hg~8+6w6S91m&L;P*muoM_|-abo&HR^ZcDc0xDA&^GBRQ?
z0I;bL(ok(SEO{fnumBI+Bd=n1yFP^@5Pt48?mGr?r^-#pyF+g;C~>Mz)}0bG{B6lQ
zjNOq6Bctk=>ag5HKXA;3`8MWs@~&7SqH=p3kv2en%j00O2;!x)K*=Dx?PL)kbm2(?O#6%yP#3p`Bb
zXs#9RzmznHAu>rtOyAYI)ka3((iC`jd_I`YSbr`W3sSgg$=;wMZ7w|L3ONlyQ+KY!
z##Q@3p`$uc(@^+(45x$WN2AxI6Oj&klzc*Txk}*B2Q!p7?ios&TYEMb)7YQ{|1}$X
zhdsot?cRkAZ@~w*;Ks$)WC`CP3wne*lR2o&@_dHqim%5oG~bQ@rk8FDi`4OxCL
z(LMF;)l|=Xd96`zFU*U2T~;y+zk;wc9>kHD
zcvE%p$>*fZ+;U?UCwh+!N`V4*TU9&fq4L~Ed&J-ua>v=${sF%3g@eD@10<5~M?GgJ
zbzYb&dEGnB<_)VLmfM<1+=x;*KB?FLRwHJHKQBFS9P)5uhii0L#!%^8B-4Ax
z%O*D7{psrHS~ZeFWU+9v0DO#PO56ea9arz2hWayTUVy~pGC6r4S$OT=79DTHY_uT!
zu@4YG6Vhx|nDqnAYYFL;5PLAxK$~ZGt2qi{wmu5LvsBNyvJvK)HW5WUYn@W$Id)IZreb
z^*FKqTK7|8uTN-rpt2kj<7}1N
z2MMj2UYiJ0>K(pb-YNn1=X=ljbpJQkk6F{5wU7V@A*Kw~b4EB!1{&^wgv!5U{!vM?
z(HG0&q|irb08j?vI{2SHdw(Ghw5x_c>D~<6F_(PWQ$`#TE#B3M+Wl*FPgQJV-4x;b
z_aI-)+l2!ygOMP#21-&{98sebF<9$Py%)2A{z>dqI&hMwG9mApKz19TRE%A)(v
z^FGx_2?Zf~0j!AlS8ClO^66M~PE@asE3%L#PN_G?LK%t&4+%+ZwZL{+J;%UPIByHt
z;U$;1tQmaW;Ef|UEs
zY&08;jZg_K<=wQ!-xh-uZ9>r%#xGL+AQP?Cj
z(q7mF6g#NrKai1WXl0ij(!eJlA`H{?N5;LgYiOZb<$E#Tx0MFY}yRy$CbE`DnZ1Xlj`$A}dk1l(fl%ZT3{-t*+Y*QIE1NLtcb35+!+
z)sczn8KaOXARgg2?v#dNED~*2ccs>=2>6oYjg(qdi3JOUXtzt29f_e$i!qsJPk}WI?bNGLj!G&((XA|RZZF7=t}9nZ_Sw&Ce2S4`XnV_j>t)q$E&(}E)qQz
z0@#awn>TR;K9EeCh%~<_2(q4mT#nLTnTaj%w|XXxP4+u(ZtaFp3x?+i1L^jgLy;rn
zye7D0O)NyG59h2<($ch(S8Hn)`GkRav8>G4BaFXwVbaRK%-=g>R(a05C55yaBN1x>
zaEWrHX#=f4I?YRsZxZo{^qA^{6G}7@07eTrb&x(!*#+^?38s(mc?w#eD}1jhZ_nJ|
z)>*$~?wLFEK6^F~8?2v^ut2>Y~4){OQK4p~LTo&seEE!`G%7fG1;5mQ$RI~}N`0S^{!
zwoXVo{=TLuyoc_&U4i_t6lbKen|B(k!XGY13uu%JdKleK3$6Nn6Jz^N#5=;rgkg=|
z?KLlXf@bOLFqL7;?7WEOOAret}Dy6uq6%+nms#%L=d9^tMH#N~x&dIO*ccTM*9
z9~w;T%AhNC?2r_LQ0~2Z98W3;001erREok;X@CQDN@23NH_}JdPmSV|+aG2+%6k>g
ziH7@gRhd5S+28s`)Im7~M^pCly3pDPxf|CB(78=_qb$M9>7H4+E7n?mM`6lcU^Zeh01u3t>%RFL
zUzuITw}!&*)rT2t7uj{&-AtlRm0Uw}8?w{Q#@d$*_%j*DVa)PGuaBp&LFA}J?9h+l
zBFLeR+Bkn65epYmN04<1Z3#Uykf@f=4{e8jUbi40FWMP-YK`TfQsY%dg@M5-(uO=T
z4&FwzEv2ZToN7X#>Ae+fz?o|p7LJlpOdO{VBv_K=#BB`C{)dh{uCINK8
z=g7x?nQqq}?RDc{1VP-IP>HB7r;vG=mMB2W?uey&}4X!&LP}Kj(R#RU>5W0T~jESD0poM
zH8?+#GyK3Vx=f^k69b790IKnk7Umw2&5!X+9h!G;
zQgnYIxhdm8cG8St6au3lQ+@bcz0{VdRh6ZW({%$F;_=B5N~O@^I5zmcTRzHhN=c{6
zT9ZFsP@_y4*fqc~b6Q!8MTaU_3w5*99!2VqDjCX(6*8n95XECuznoy>qb(O
zFO^xgJc2#403TAm%8tEyzG}#CsAVVnA+OBaoJsH
zo=I=bvB*XHg3@5)T+uNovsAcp?Hw6wbN?<$@xK8VhGH)@kBbu;u(l!<1xdvbzEe?NTo+qes8C^iuCbE6_KunD
zLAt(G2j2u!Z$W8EHldDdYV5&3_~`U-Hq}oRTvI6ns`*|lq$!Zz59b?_aBZ2-rmHI%
z{^ACb0IvC&L4s(g5ChJkF3J9HxcM-H5d0~AQgsWgUd;$9sjJ-8313$uvV_*h;n+Je
zhX!%rlDS0MOGodN^w{khBiEgHiHxhEH(m({6n&NQ_Ol;>x>pIO(eSY;byYgU)JJuU
z&A3=WHC}pi;O5%-CE{?O0GzBlrgfV2`rt1+(b#Z9}q4%YoH;bg|$&g1C
zpi!P_Y%K|=8MMur_J%|=-lT7kENXnMtOj#-KrB?8nmJNLar}Cv)VYC$G%-3XZ2S@u
zPI_zp$fd4t4;#XY#on|^c79ZP%7+%ixx?}{FT
z8~#yTXdCVtmx^k3Ge@XYtQv;fBa>|_2%3HEMkZRAJtvGcW;^n?%#kOYJ%;FEW8GnE
z<{ee0Tf|9-06+0BDG5A|139taRwkt1Tv@d~hOxGhwSI0eeds2Uq=#u)>Jj(Zg
z&Pq7|gQ~^jAdh@Hj3Q+ruEyrWO^K{vizmE%oH67&uoo}J3tarqYO#D3pmKL-a+f0}
z`3fyY_3pKj-B?xIf|XjHUG($leq1q~f?EF%)>!;DQFj<3^VTA}UgT#QW}Ac@V!kHq
zhm5Fw2mVO+ep=A%4Xt;Ex-u=Bek*Mom#A5Q#-)bzrwi#Z@x*+3nx)v+1y&}ryJD7;rcTTp%$H{ITRxk?u$OCY2L>VvG4ll>~Ed_K6{^G
zTJn^Sc&GG4v#FFK)%k^tWFCg4)#z0Byc;;)y={ZNzOEiAYe=6cxtcn=;xhsKX`|ml
zx~d`foQzV%OW!U8U<1k_R5SBaB2rCc1cofp3g|#@YSiC|SnK^ls&WEca=@Wy3VIz5
zz~Eu)8rH3r^?-6uV*N8n>L}<@S8*q)fsUVKSImYR%96U_sxnGxI*WDw!b$3Q5`vMU
zG4HcClW-kor$6WoFXu$+2lK#Aq~F2G^Nq`9hefGyKS$B^nn2Ve|Ats3
zMF`oR`RV<#z9CvO24}+F6j3hod$%A-Obg9OPXhFu&xu_uuX-o@_r7^Hl*+_kl1ZbN
zIe+F?JGTfr0Gv5piu!uN-_ya)60_CyYqS_55Q$@D9eJRWX%7zV$=G^O*jJ}j22wcD
zZZ)e-@=D^F?5c8kT!tZIaEJ=Lg$a?}gX-|Pso&i@{IF>_P{c>MQ0>P;8bVsg&}i*y
z*;J)6sp#u^CbPble|1m|8Gf{#1h9z;>Xvo#14%{i0LsxsP_6l~*Mg4U-QOAMh8w!}
zo}l3QQjVYvp)V!2dDSvoEKgNh3kI6vi6Cn;XdzG~H4z6wN-=i2&oLz{Go+-I+3yTj
zOCN`>5sXya$}OuS43=}wQwxe)Np4|RGRd75?kY9TEaZ?uNV_#I@$4-TQx7K(OTX8d
zjblL605krEQnVY8eXX|g+kSC_!ovhMtVCGJT*}#%`K!F#t6#(Pbzja+B6mSgm{jV`
z`*1#DC*8(fwHZ!DCUi;(ZSQcq=D}gl1Z9v89B;M`Q`g1HaQeoyGrqCD-mDbZk7&-!
z$>1GvLmSyyT;CO<#nIq|pnCaF62jR0lj-6K04RnJu)Vsf&1|_rf|KEiqI;nF4^+N?
zZ&x#MQ?*Q0j;6wP^ZpvK@jYPLj`68|pe1<}>?UiTc2aR&mCz^at5xI6oFNk;eA<_4
ze=UcK;iuo8klvu`;VDgL>g;>zHUjS?qBny%xvD-lIqY5Po+KtQeg3`=IP5I3Rwv_a
z0ccA^F)N}E!A5E0*x`;e_Og@9g7c(&UpObu|AMHfRhEWt3+2vFy
z{?6cJHR+`U>F32h)(XqK2-l!^!H&)Bj^k6p!f}E!^eGGKUs4|!r}zRuhdiELdftQ^zc
zP5~#6;$>9}ubnHGQB^=y>)2zIu^3WCle%!4n=>%P#(SK_go2$%AHWn7m+wj$uN@@*
zr95WG`X1cG>1+5Wj;2d3ls;a%WdSEYM;4msXg})^UZuO_h0$^~046gC6aTp@0?@bs
z^%HxlVQr%&0I>kIEf%HZjm75I{aft)(OH1dI$A!SkLH9(}tp>%etX
z{qnTCIch~5GBjr>fmgU|x&+2iC7=x)-<}rbffi(%*thxUbNl#hb#By>m||z-{JqNJ
z?d{w$nLhTOY>K{+9V>jU?yiRBl?)r1t7Njf=hdQgZpg-<
z@zf5(e+m|3C&yzx#;Jr384zO@b!3VmtSg)!7r_Kp52s!8AWKUFN$j@sFXsIk^6rF@
zAMWfc^x<0fsAj6^kp%->?X-Rft?)orbYc+hiyBJE|56EGQXNuo0Q!9yuHxS#Ly$wL0UUBz{<@b5Q8%qsg~LTO7-
zkQHhx-`XA;?`;;Iq$wZ!u7O3=K;kp~DuNpf@x-KKQ-eol@$gqXY7vDWxng=A@~H0k
zyO%1X9I;sW2sZ#8z3Fg67CqiI|1#=`NB<89fD+;wo3I|j#|E0FK_z1FeobOFD@B>^&&LCNE^sM
zcP0&JhmTU_jwEs9ycDWRu#vE8aesV;usm@-kjL+$v_Ei45l#{vYg$PiCl0HpU^D8c
zf02aRwymFMKA=;2F#@_U>{w$po8z&=7$4$Q1dA$);}BLzpt6
z#ZdJz7tDOXQ%6#~XJc&Ch(TVfSEeZ2Oeomub`*mcj&ZH$>&c0b6D36(s_
zdo!Ja1U~C|J$NuspF|ZBFK8+`LM~l)IWAZY!R
z6t2=2{e#NX3&2QYF-d{A`nMDLwBss2&dWxRJPp7n7E)T*pmMW0#!e4saV!viiR6Z=
z)mD<2jGTIqwRM89-InVy(L|?F70)5#t%b<5^7xLgpc-6;iPFicR019YfZ|_$w;fS(
z+}0Trb$Ia{Mp#jN2R&OTcsXmh!-0Q1KKYba)2LYKOvfi!k!N&BOU&!7w5dj?vDBIy
z65zb@k>NlR4hk=LW^~B7pJe9@bo>LNTTVFiUtAvtQyP#sC|WCH2~ZF8KOEwpKy3xB
z1KL6ZVr%`!MnH%DVB+pQ?z1|g6^aKj|Dsz=mRVeOY6U*eEwmD|M5*!097#l
zxBGuH8m+I5px9QxjQ`o3cIgTP-^%(MELQ4&9M&wrP9gt^K1Z-H2!?+(C5%j*X>v|r
zxc^5uI)TLkv8FnM;bQ)S6~!!_oLvYw{x8!(z{<+-Ur`>{I@b0$>?{AUMMnHA-67_#
zB!M~s+)>HK_D$Pt@OOeu6m5Ic@q{C`b^YE`&SH|8Xz>Oscf~FW14UeT`mot{XEAgs
z3{^sCqzWM=OK8=xNUbL7;)f?ZXwvgdl@O)W9@7HSZT5~i?A;bfwkya(XhINK6CSDn
zs;7D}MM1ze@H9^yo2YogLP13Q6VT_IkY(_RgFWB(5Zq4P-ZtpR{H2_B`rps!iK~X*e5`Fpb$ScN%0(aLm`q7HE=^=&w#Xm
zG_T1q>+-L3+XzT(i<3D1=sJ9X*6cbEGOQqD3tJu*ABZyZlc`Lu8kqod2
zlJ~H$(HpTcN^bi*?O-hO>b0Ex$U>0a3c^!xi6%B1eYD=hB
zbTTzlB+f1RZo-r)gHQ`z#%igf+C@Ln8uZ?k8!HL-{&3#<-Z=PfpqJ!UXjsOR9(_7o
zBrfXdZ`;0h?LV%wR6j17gB0yb)BVS|(nt}(K)WhwX1wjG_iwXuPmY`%!lmbZ#MuUb
zpJJluW3%h6OSttnuNIo9qFxT5Ju^2CH81rk?ep1waPi~2v3oZCB8luNYNsjF>5+H9#Mk5-N<<4uD{ag#bieStOE
zrAuj7#;e_+-*u`19;T?nWomg^&ysjM!j#2KUoiMCuA;R#vXBNNK%e?fQR#2q`h;e2
znJUCoa6ZZGt|aM(j&B_>_OaltcZhC!k$0k3(@#CJz?>>1A{bG}TvskBf}{oTR=xk#
z07Zi07|CI+FL@Hykth(q(#R32>qel$zO$v#D@zG#j2~vp{idc|R{NLxePfzCreK;)
zU69bI-7G$Fu4x!ki5meeRWSulbETIbuyJZ1{3G`K1GE9zvVJ#VfGAq0ALm*GO0$YL
z!68r6T}+X$m?_nF=(w$j&twP?8xYnGXplInuSBZ~IBT|F7?#jzTLFE(X1lcbs_1}l6Rs05T&%OT)uCa(
zoam~jdnz~)qg%pUZZ2;T{D%)LLTBW6m#W9KXa*9Nx^qB~wlSwg6QVGAA0<3N&RsgG
zMb#i--2TFdCk1OO-xU#Oi3g^WW<4%3d~*P?V611i8|IKAAsNAgM1x>pG1FTc>S%iE
zAP9OM+`J)p#z|YVHr*UR9afR%SjlAy*<(p5xdQ#3aTT$}aq1Qd@{E$mN4^6*#!R&U
zHlVN{x`lL(hHGRGh#&-7Kr6PWAmAy4E;-q#O?*`2l|g0FAk)jkhM$f`#KfsMP+A>`
zRdhn=%0X3)y*#yMnn)ii^2Hcw&fwsc
zA*d)x0`FC^+Y^IN{OsKwM3ZP|RC7zk7Q
z*b2T^3X^ji&L#rTj!`mYS3rPf)1{^$gO~rB86u4}Jk>Ipyrky9%yqW3xkn&7mu*8n
zC&tUV`Cvd~qA*J#rSveBdJF9qpG!!F)c9++LECPfhmNc5OYpvJ+!|B09y*A<4j@-I
zB8dybfoy_o4mS!*D@p_bD1nmHP`?{!{D}@klmL@VqILu9JIZGc5!A(F-eB-%LfouO
zni~C1M-d2CkPUI^J10}NAq?~xc*V;Y8hF$q0jzt8q*AlG{zdfS;@&__|gHQR>z+u5s+r@2ScCPhTSJsSdTe*&VDOW4;EN|
z6SR~l&x1A}(ZPDY80p4QKb^>ly9AvT6VC_SP?b2q)Da(CyoC|oAyZBN3{^p6CzM_j
z5)hZ}C#hy-?tM=U$)<0^LqPYygWQ%tl{3_AJ#?KziDqbJBq^^0xPda>%T(E1EJcb9
z$hs4OoJeBP`(5HS>T9b94+;r{bK;?#m2x8MRG_XKF+-Pjy%
zJr@AsY5zrcHKSG9LyUWMX8vaG8)z=zlt;ci6-|--p3|x!kqK0(0>6m
z78MOf
zF>x#?vP8IVmY~ly9yL*KGrFk^0n4deEu{_KwGsbmpa}ca`@JVHgj9
zQDVlc>ZaV3cBfrB!|aG@++Ta?AOTAMGM)!Nx7U-#GTNinP+PLn
zKU@~(QTkWm0mX@@97D=?hZ28Q?(!ZN-nPt#sm)6dVU~V@6n&^WjA%}Te+B|jw10kh
z|Ne)rBIm(DPycUp8j#z>AxroHW#*v~!?5xS{>12U(nG$FUJr$S%I)N!lR|EC+3~r9
zmzVFmsS1F4cB11;DK`z{c=)w4%1Z}+@`qad*p!ow{8-gfRX!}K(MfA&!qZdtubLlq
zFMl`=7bY^&z`gU%+b?%!1$%fv+)lT5GpJMz$OI5YTr0#|4#=@nz6%P4FL!=L9r~Z|
z2g~>9J}*gwZn|9tgMS67U=~U9CNjsz#{>m0A%8<{ul%Oxabzy<|NLc&Yo(yIr
ze2*{v@Z!x_t%rM?5er%u6?%8y)A?H#(XTUEA({8l
z&(n_sU(6Cr!ybshux=7mp;1r2qzdbr_*EuR3*nG;SS=F!I_~1|Va*G{K|c8SIP%QX
zON2FDbD|za4NixTjdKz+xopsrZBA2AfTixh+DWz&$ZS5izHD*lyDacHmDP2(d3SJn
z7}WYnk4#UbkcBBoxGjtYz;AU9O&j;K81K;{|3omvL1wPdGvlFq)%Y@`^fS2!NDlyH
z0{Jm@bp^iUFW&uK4G6#gg}zFB5UhD*Bh2~KC;!avJsjD@SN8jGVCKL{8ZrenAih!D
z?1th+ye_j*{CNMY??0NuB~NR`SAy0r?B^1^bYL9lXK^QOUfJ9NhA-pzzn9iBwCVYX
zY~kqZwr5weuePyGPAMcICr8srAP>P$ds&gzChV`i-=5NuG&t6pJ6EBr;OdM#iX_KE
zFCk@r4OHulVoSFu4j~<
z|9p$tSvc`tKMEcK_S?wdvK7`3qZ7IGVrtBfSN&F1^x=sW|M8?+#%y0LOFQqwO*ND6
zw@F)$DL>_9No2jP4|?_p2(SD!hr!QQUvnCfsHi(3Bw=7~o7|0Ag*$qFAhI^Cox|1H
z1EZ__01iGqn|ZsnLWX#YuA&Of!s(n=$Z7rF!I6^zwU#%)bEUey8qhtsaY^^U$z5cr
z>ECm1`~aalTaJNR#wjiL|S2t#_Fq|(V$cN-5z}dbT)%hjE7)?RE9_Q
z5gHY1#?4uP%wNKcPc|pUer&UheyReiQf0$Vf}i2eIQdtmR?LW76w|}HAVve)85WbGQ
z_u9#R7p`~S%T%9;E*7Yn!U&$yy{|zhvC;H-^5-Zn|mp~pzcHZI3
zl;ihIZ(5NTTa;|tAme(@>9z4AySQ&~{
zo^IaM1vtbVPBbmY-QVZ1prS!C2)2;iqjF$B7J`_ef;nG?{M}3hvv6gzUZq^OdeAm7
zY~lp~s3CRG_uJw8qrM)S;F>`Tl0W#D98m9<1Kkz`_SDYASdWMM
z+x{`ml`~UrS9UG`2+`yRB7bIo9H~f21Kxw9K+7N%KXv?oEqsCz1t|k
zjV>uJ!x1~FAKyv7y%z7zk{HuC9^L|O-aGcMqy{rNRqpb#)FF5DMg<>?Qu9)Ka2P0M>ZUMWu~F}J@M3}uRPRzk{EHG
zFsg*tbjf;zsr{KxWaHiiLUh~g6(dRjZcR!e6qHM3`$8dqsL5>R3-C%}YBKVJ?8MQ%
z>>!7gvw_GzgCVMuT|IRtp@)$e$;ef}YXg%b^}~!{y(V%@Ks13?V+Kevr4^zMv_Zb}
zoESeFXl#)fEF|BL)on`EoTH>!bM75oJj`@5%_l0JuIoIpb~y~}@@0#j&ij>slXk~_
z-&2!!w=7g6$FImDRzBkrR8P+bDWh!oe-OgWW*vhcAt#M&{4ex^fr}AAokafu)nmbZ
z5)%T+U*U1U3Ld(@1D+1G?@;9PiNulasyw{3^+r6Z(MIV8TX}|xwn~t$f5l|}w2VXV|
z3Mz4?!8~EuQ1ff8k)Vs#ek*XTFoISNc21GRV};wm8|9dphb4;M9Yg8SQi?Z~RNxW^
zfnGvc7`>ddpqX7lez8Gx}^x5`LYL}3g-U91nQ_WCGHG=6=kjk%uYndo_=(c_^8&#bM
z8z1tYTE$??WU>U1dh?+YmV#O3H$g3b7hBkDbu|~^7IeTjyT`PC3e>V#T63Fp;Hr8N
z7CsPCH;O~(#6>~aItyh1yHQQgrRA~u
zcxfjN=-*ZqQqRk}F}}Noy`gP4HwEe033trb&e=l5Y~}Y$%U*3CHv<&H^&YLc5kIb>
zw(j;iWq0A&+6pjK&CXof5P;cvdhJlw7>9(|epNB;YHdUK;>>0MHb1CWBn<{5)?**p
z&Afrx@djJYl`qejvHV7MAUw0+B{q*-v8tTDRKWAv5I!~_kZg-q113*_PnknlZ{j>c
z_%Saxb~IMq;IiI!Lz6ow?=vJco$;HVdwL^70m(aX}&_*T9ip%eGLaf|l6JQ$`fOQEg14c0x?g
ze4|QhF*gaO2Af+1(Ll}45e^9ma&_@|29`Zb^>cNft}6*besK#!FIbkmxF*JGn82&lP(K}go+
zIsRBa=?Qbhr`Qfe*e#nifHyRBW5JoIdNX5>RlJz7Xp5}NyLrrAZOgZSH&oDW|683g
z^REJ2Ai|rthB**a{%@P@|FXFu&!)JFxAm{GYeioEH!9L-_ZUKP6HV$RyDVMfId|4U
z!WIH9>20!rk@5GiQ09O(NkCe&+IgQHySzyV**W9yZK2Fhks_IG>Uf-#rM;z6N0tOo
zZ1P_<5i=A80;zA4$RXA=heXfokbZpI2NFoT3ir4kc-cYWEJi-$7J?-MUR>{z70
z?)Hk&IL~&Y_mxQi&{~oJ=>OX!$HdH>78DLf1k0ieO|NX}WlF%r%#v0g4u)#Y_^;7g
z37THg&dmPbBGZ2>5dv)ZM3Fdds3xF>=y?z_d#}+|
z;Y#g6LS7q=v<`fw!JZ$sk08%BSgADfM
zTg(^WMs5wPK$H(u9yMgYA-4zJ$KLOPH8D>5-V)QEAwU
zqu(s=oW6uTjw)XvvRQaN_GOZNYZ1*Za&&9Li=g-o$rK4g*B)BqMDK-lcSwH|d2pl*z
zEDNRu!<<2Wzko1E2`nSb6jQqClxIh?LGJI~y50(l;E=TJD6ro!%#3Wz(4?fI@?y~c
E3+wk@@&Et;

diff --git a/doc/src/Splines/Splines.do.txt b/doc/src/Splines/Splines.do.txt
index a1465d76d..950f1cad9 100644
--- a/doc/src/Splines/Splines.do.txt
+++ b/doc/src/Splines/Splines.do.txt
@@ -26,11 +26,12 @@ direction of the negative gradient $-\nabla F(\mathbf{x})$.
 It can be shown that if 
 !bt
 \[
-\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ \gamma_k > 0
+\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k),
 \]
 !et
+with $\gamma_k > 0$.
 
-for $\gamma_k$ small enough, then $F(\mathbf{x}_{k+1}) \leq
+For $\gamma_k$ small enough, then $F(\mathbf{x}_{k+1}) \leq
 F(\mathbf{x}_k)$. This means that for a sufficiently small $\gamma_k$
 we are always moving towards smaller function values, i.e a minimum.
 
@@ -54,7 +55,7 @@ the learning rate within the context of Machine Learning.
 !split 
 ===== The ideal =====
 
-Ideally the sequence $\{ \mathbf{x}_k \}_{k=0}$ converges to a global
+Ideally the sequence $\{\mathbf{x}_k \}_{k=0}$ converges to a global
 minimum of the function $F$. In general we do not know if we are in a
 global or local minimum. In the special case when $F$ is a convex
 function, all local minima are also global minima, so in this case
@@ -76,7 +77,8 @@ Note that the gradient is a function of $\mathbf{x} =
 !split 
 ===== The sensitiveness of the gradient descent ===== 
 
-GD is sensitive to the choice of learning rate $\gamma_k$. This is due
+The gradient descent method 
+is sensitive to the choice of learning rate $\gamma_k$. This is due
 to the fact that we are only guaranteed that $F(\mathbf{x}_{k+1}) \leq
 F(\mathbf{x}_k)$ for sufficiently small $\gamma_k$. The problem is to
 determine an optimal learning rate. If the learning rate is chosen too
@@ -87,10 +89,217 @@ Many of these shortcomings can be alleviated by introducing
 randomness. One such method is that of Stochastic Gradient Descent
 (SGD), see below.
 
+
+!split 
+===== Convex functions ===== 
+
+Ideally we want our cost/loss function to be convex(concave).
+
+First we give the definition of a convex set: A set $C$ in
+$\mathbb{R}^n$ is said to be convex if, for all $x$ and $y$ in $C$ and
+all $t \in (0,1)$ , the point $(1 − t)x + ty$ also belongs to
+C. Geometrically this means that every point on the line segment
+connecting $x$ and $y$ is in $C$ as discussed below.
+
+The convex subsets of $\mathbb{R}$ are the intervals of
+$\mathbb{R}$. Examples of convex sets of $\mathbb{R}^2$ are the
+regular polygons (triangles, rectangles, pentagons, etc...).
+
+!split
+===== Convex function =====
+
+_Convex function_: Let $X \subset \mathbb{R}^n$ be a convex set. Assume that the function $f: X \rightarrow \mathbb{R}$ is continuous, then $f$ is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all $x_1, x_2 \in X$ and for all $t \in [0,1]$. If $\leq$ is replaced with a strict inequaltiy in the definition, we demand $x_1 \neq x_2$ and $t\in(0,1)$ then $f$ is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the value of the function on the interval $[x_1,x_2]$ is always below the line as illustrated below.
+
+!split
+===== Conditions on convex functions =====
+
+In the following we state first and second-order conditions which
+ensures convexity of a function $f$. We write $D_f$ to denote the
+domain of $f$, i.e the subset of $R^n$ where $f$ is defined. For more
+details and proofs we refer to: "S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press":"http://stanford.edu/boyd/cvxbook/, 2004".
+
+!bblock First order condition
+Suppose $f$ is differentiable (i.e $\nabla f(x)$ is well defined for
+all $x$ in the domain of $f$). Then $f$ is convex if and only if $D_f$
+is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds
+for all $x,y \in D_f$. This condition means that for a convex function
+the first order Taylor expansion (right hand side above) at any point
+a global under estimator of the function. To convince yourself you can
+make a drawing of $f(x) = x^2+1$ and draw the tangent line to $f(x)$ and
+note that it is always below the graph.  
+!eblock
+
+!bblock Second order condition 
+Assume that $f$ is twice
+differentiable, i.e the Hessian matrix exists at each point in
+$D_f$. Then $f$ is convex if and only if $D_f$ is a convex set and its
+Hessian is positive semi-definite for all $x\in D_f$. For a
+single-variable function this reduces to $f''(x) \geq 0$. Geometrically this means that $f$ has nonnegative curvature
+everywhere.
+!eblock
+
+This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.
+
+!split
+===== More on convex functions =====
+
+The next result is of great importance to us and the reason why we are
+going on about convex functions. In machine learning we frequently
+have to minimize a loss/cost function in order to find the best
+parameters for the model we are considering. 
+
+Ideally we want the
+global minimum (for high-dimensional models it is hard to know
+if we have local or global minimum). However, if the cost/loss function
+is convex the following result provides invaluable information:
+
+!bblock Any minimum is global for convex functions
+Consider the problem of finding $x \in \mathbb{R}^n$ such that $f(x)$
+is minimal, where $f$ is convex and differentiable. Then, any point
+$x^*$ that satisfies $\nabla f(x^*) = 0$ is a global minimum.
+!eblock
+
+This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.
+
+!split
+===== Some simple problems =====
+
+o Show that $f(x)=x^2$ is convex for $x \in \mathbb{R}$ using the definition of convexity. Hint: If you re-write the definition, $f$ is convex if the following holds for all $x,y \in D_f$ and any $\lambda \in [0,1]$ $\lambda f(x)+(1-\lambda)f(y)-f(\lambda x + (1-\lambda) y ) \geq 0$.
+
+o Using the second order condition show that the following functions are convex on the specified domain.
+ * $f(x) = e^x$ is convex for $x \in \mathbb{R}$.
+ * $g(x) = -\ln(x)$ is convex for $x \in (0,\infty)$.
+o Let $f(x) = x^2$ and $g(x) = e^x$. Show that $f(g(x))$ and $g(f(x))$ is convex for $x \in \mathbb{R}$. Also show that if $f(x)$ is any convex function than $h(x) = e^{f(x)}$ is convex.
+
+o A norm is any function that satisfy the following properties
+ * $f(\alpha x) = |\alpha| f(x)$ for all $\alpha \in \mathbb{R}$.
+ * $f(x+y) \leq f(x) + f(y)$
+ * $f(x) \leq 0$ for all $x \in \mathbb{R}^n$ with equality if and only if $x = 0$
+
+Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).
+
+!split 
+===== Revisiting our first homework =====
+
+We will use linear regression as a case study for the gradient descent
+methods. Linear regression is a great test case for the gradient
+descent methods discussed in the lectures since it has several
+desirable properties such as:
+
+o An analytical solution (recall homework set 1).
+o The gradient can be computed analytically.
+o The cost function is convex which guarantees that gradient descent converges for small enough learning rates
+
+We revisit the example from homework set 1 where we had 
+!bt
+\[
+y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
+\]
+!et
+with $x_i \in [0,1] $ chosen randomly with a uniform distribution. Additionally $\xi_i$ represents stochastic noise chosen according to a normal distribution $\cal {N}(0,1)$. 
+The linear regression model is given by 
+!bt
+\[
+h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x,
+\] 
+!et
+such that 
+!bt
+\[
+\hat{y}_i = \beta_0 + \beta_1 x_i.
+\]
+!et
+
+!split 
+===== Gradient descent example =====
+
+Let $\mathbf{y} = (y_1,\cdots,y_n)^T$, $\mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T$ and $\beta = (\beta_0, \beta_1)^T$
+
+It is convenient to write $\mathbf{\hat{y}} = X\beta$ where $X \in \mathbb{R}^{100 \times 2} $ is the design matrix given by
+!bt
+\[
+X \equiv \begin{bmatrix}
+1 & x_1  \\
+\vdots & \vdots  \\
+1 & x_{100} &  \\
+\end{bmatrix}.
+\]
+!et
+The loss function is given by 
+!bt
+\[
+C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 
+\]
+!et
+and we want to find $\beta$ such that $C(\beta)$ is minimized.
+
+!split
+===== The derivative of the cost/loss function =====
+
+Computing $\partial C(\beta) / \partial \beta_0$ and $\partial C(\beta) / \partial \beta_1$ we can show  that the gradient can be written as
+!bt
+\[
+\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
+\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
+\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), 
+\]
+!et
+where $X$ is the design matrix defined above.
+
+!split
+===== The Hessian matrix =====
+The Hessian matrix of $C(\beta)$ is given by 
+!bt
+\[
+\hat{H} \equiv \begin{bmatrix}
+\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1}  \\
+\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &  \\
+\end{bmatrix} = 2X^T X.
+\]
+!et
+This result implies that $C(\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite.
+
+!split
+===== Simple program =====
+
+We can now write a program that minimizes $C(\beta)$ using the gradient descent method with a constant learning rate $\gamma$ according to 
+!bt
+\[
+\beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots 
+\]
+!et
+
+We can use the expression we computed for the gradient and let use a
+$\beta_0$ be chosen randomly and let $\gamma = 0.001$. Stop iterating
+when $||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8}$. 
+
+And finally we can compare our solution for $\beta$ with the analytic result given by 
+$\beta= (X^TX)^{-1} X^T \mathbf{y}$.
+!bc pycod 
+import numpy as np
+
+"""
+The following setup is just a suggestion, feel free to write it the way you like.
+"""
+
+#Setup problem described in the exercise
+N  = 100 #Nr of datapoints
+M  = 2 #Nr of features
+x  = np.random.rand(N) #Uniformly generated x-values in [0,1]
+y  = 5*x**2 + 0.1*np.random.randn(N)
+X  = np.c_[np.ones(N),x] #Construct design matrix
+
+#Compute beta according to normal equations to compare with GD solution
+Xt_X_inv = np.linalg.inv(np.dot(X.T,X))
+Xt_y     = np.dot(X.transpose(),y)
+beta_NE = np.dot(Xt_X_inv,Xt_y)
+print(beta_NE)
+!ec
+
 !split
 ===== Gradient Descent Example =====
 
-We revisit now our simple linear regression example with a linear polynomial.
+Another simple example is here
 !bc pycod
 
 # Importing various packages
@@ -156,214 +365,7 @@ print(sgdreg.intercept_, sgdreg.coef_)
 
 !ec
 
-!split 
-===== Convex functions ===== 
 
-Ideally we want our cost/loss function to be convex(concave).
-
-First we give the definition of a convex set: A set $C$ in
-$\mathbb{R}^n$ is said to be convex if, for all $x$ and $y$ in $C$ and
-all $t \in (0,1)$ , the point $(1 − t)x + ty$ also belongs to
-C. Geometrically this means that every point on the line segment
-connecting $x$ and $y$ is in $C$ as discussed below.
-
-The convex subsets of $\mathbb{R}$ are the intervals of
-$\mathbb{R}$. Examples of convex sets of $\mathbb{R}^2$ are the
-regular polygons (triangles, rectangles, pentagons, etc...).
-
-!split
-===== Convex function =====
-
-_Convex function_: Let $X \subset \mathbb{R}^n$ be a convex set. Assume that the function $f: X \rightarrow \mathbb{R}$ is continuous, then $f$ is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all $x_1, x_2 \in X$ and for all $t \in [0,1]$. If $\leq$ is replaced with a strict inequaltiy in the definition, we demand $x_1 \neq x_2$ and $t\in(0,1)$ then $f$ is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the value of the function on the interval $[x_1,x_2]$ is always below the line as illustrated below.
-
-!split
-===== Conditions on convex functions =====
-
-In the following we state first and second-order conditions which
-ensures convexity of a function $f$. We write $D_f$ to denote the
-domain of $f$, i.e the subset of $R^n$ where $f$ is defined. For more
-details and proofs we refer to: "S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press":"http://stanford.edu/boyd/cvxbook/, 2004".
-
-!bblock First order condition
-Suppose $f$ is differentiable (i.e $\nabla f(x)$ is well defined for
-all $x$ in the domain of $f$). Then $f$ is convex if and only if $D_f$
-is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds
-for all $x,y \in D_f$. This condition means that for a convex function
-the first order Taylor expansion (right hand side above) at any point
-a global under estimator of the function. To convince yourself you can
-make a drawing of f(x) = x^2+1 and draw the tangent line to $f(x)$ and
-note that it is always below the graph.  
-!eblock
-
-!bblock Second order condition 
-Assume that $f$ is twice
-differentiable, i.e the Hessian matrix exists at each point in
-$D_f$. Then $f$ is convex if and only if $D_f$ is a convex set and its
-Hessian is positive semi-definite for all $x\in D_f$. For a
-single-variable function this reduces to $f''(x) \geq
-0$. Geometrically this means that $f$ has nonnegative curvature
-everywhere.
-!eblock
-
-This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.
-
-!split
-===== More on convex functions =====
-
-The next result is of great importance to us and the reason why we are
-going on about convex functions. In machine learning we frequently
-have to minimize a loss/cost function in order to find the best
-parameters for the model we are considering. 
-
-Ideally we want the
-global minimum (for high-dimensional models it is hard to know
-if we have local or global minimum). However, if the cost/loss function
-is convex the following result provides invaluable information:
-
-!bblock Any minimum is global for convex functions
-Consider the problem of finding $x \in \mathbb{R}^n$ such that $f(x)$
-is minimal, where $f$ is convex and differentiable. Then, any point
-$x^*$ that satisfies $\nabla f(x^*) = 0$ is a global minimum.
-!eblock
-
-This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.
-
-!split
-===== Some simple problems =====
-
-o Show that $f(x)=x^2$ is convex for $x \in \mathbb{R}$ using the definition of convexity. Hint: If you re-write the definition, $f$ is convex if the following holds for all $x,y \in D_f$ and any $\lambda \in [0,1] $ $\lambda f(x) + (1-\lambda)f(y) - f(\lambda x + (1-\lambda) y ) \geq 0. $
-
-o Using the second order condition show that the following functions are convex on the specified domain.
- * $f(x) = e^x$ is convex for $x \in \mathbb{R}$.
- * $g(x) = -\ln(x)$ is convex for $x \in (0,\infty)$.
-o Let $f(x) = x^2$ and $g(x) = e^x$. Show that $f(g(x))$ and $g(f(x))$ is convex for $x \in \mathbb{R}$. Also show that if $f(x)$ is any convex function than $h(x) = e^{f(x)}$ is convex.
-
-o A norm is any function that satisfy the following properties
- * $f(\alpha x) = |\alpha| f(x)$ for all $\alpha \in \mathbb{R}$.
- * $f(x+y) \leq f(x) + f(y)$
- * $f(x) \leq 0$ for all $x \in \mathbb{R}^n$ with equality if and only if $x = 0$
-
-Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).
-
-!split 
-===== Revisiting our first homework =====
-
-We will use linear regression as a case study for the gradient descent
-methods. Linear regression is a great test case for the gradient
-descent methods discussed in the lectures since it has several
-desirable properties such as:
-
-o An analytical solution (recall homework set 1).
-o The gradient can be computed analytically.
-o The cost function is convex which guarantees that gradient descent converges for small enough learning rates
-
-We revisit the example from homework set 1 where we had 
-!bt
-\[
-y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
-\]
-!et
-with $x_i \in [0,1] $ chosen randomly with a uniform distribution. Additionally $\xi_i$ represents stochastic noise chosen according to a normal distribution $\cal {N}(0,1)$. 
-The linear regression model is given by 
-!bt
-\[
-h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x,
-\] 
-!et
-such that 
-!bt
-\[
-\hat{y}_i = \beta_0 + \beta_1 x_i.
-\]
-!et
-
-!split 
-===== Gradient descent example =====
-
-Let $\mathbf{y} = (y_1,\cdots,y_n)^T$, $\mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T$ and $\beta = (\beta_0, \beta_1)^T$
-
-t is convenient to write $\mathbf{\hat{y}} = X\beta$ where $X \in \mathbb{R}^{100 \times 2} $ is the design matrix given by
-!bt
-\[
-\begin{equation}
-X \equiv \begin{bmatrix}
-1 & x_1  \\
-\vdots & \vdots  \\
-1 & x_{100} &  \\
-\end{bmatrix}.
-\end{equation}
-\]
-!et
-The loss function is given by 
-!bt
-\[
-C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 
-\]
-!et
-and we want to find $\beta$ such that $C(\beta)$ is minimized.
-
-!split
-===== The derivative of the cost/loss function =====
-
-Computing $\partial C(\beta) / \partial \beta_0$ and $\partial C(\beta) / \partial \beta_1$ we can show  that the gradient can be written as
-!bt
-\[
-\nabla_\beta C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
-\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
-\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), 
-\]
-!et
-where $X$ is the design matrix defined above.
-
-!split
-===== The Hessian matrix =====
-The Hessian matrix of $C(\beta)$ is given by 
-!bt
-\[
-\hat{H} \equiv \begin{bmatrix}
-\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1}  \\
-\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &  \\
-\end{bmatrix} = 2X^T X.
-\]
-!et
-This result implies that $C(\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite.
-
-!split
-===== Simple program =====
-
-We can now write a program that minimizes $C(\beta)$ using the gradient descent method with a constant learning rate $\gamma$ according to 
-!bt
-\[
-\beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots 
-\]
-!et
-
-We can use the expression we computed for the gradient and let use a
-$\beta_0$ be chosen randomly and let $\gamma = 0.001$. Stop iterating
-when $||\nabla_\beta C(\beta_k) || < \epsilon = 10^{-8}$. 
-
-And finally we can compare our solution for $\beta$ with the analytic result given by 
-$\beta= (X^TX)^{-1} X^T \mathbf{y}$.
-!bc pycod 
-import numpy as np
-
-"""
-The following setup is just a suggestion, feel free to write it the way you like.
-"""
-
-#Setup problem described in the exercise
-N  = 100 #Nr of datapoints
-M  = 2 #Nr of features
-x  = np.random.rand(N) #Uniformly generated x-values in [0,1]
-y  = 5*x**2 + 0.1*np.random.randn(N)
-X  = np.c_[np.ones(N),x] #Construct design matrix
-
-#Compute beta according to normal equations to compare with GD solution
-Xt_X_inv = np.linalg.inv(np.dot(X.T,X))
-Xt_y     = np.dot(X.transpose(),y)
-beta_NE = np.dot(Xt_X_inv,Xt_y)
-print(beta_NE)
-!ec
 
 !split 
 ===== Gradient descent and Ridge =====
@@ -432,7 +434,7 @@ the shortcomings of the Gradient descent method discussed above.
 
 The underlying idea of SGD comes from the observation that the cost
 function, which we want to minimize, can almost always be written as a
-sum over $n$ datapoints $\{\mathbf{x}_i\}_{i=1}^n$,
+sum over $n$ data points $\{\mathbf{x}_i\}_{i=1}^n$,
 !bt
 \[
 C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
@@ -454,27 +456,27 @@ computed as a sum over $i$-gradients
 
 Stochasticity/randomness is introduced by only taking the
 gradient on a subset of the data called minibatches.  If there are $n$
-datapoints and the size of each minibatch is $M$, there will be $n/M$
+data points and the size of each minibatch is $M$, there will be $n/M$
 minibatches. We denote these minibatches by $B_k$ where
 $k=1,\cdots,n/M$.
 
 !split
 ===== SGD example =====
-As an example, suppose we have $10$ datapoints $( \mathbf{x}_1,
-\cdots, \mathbf{x}_{10} )$ and we choose to have $M=5$ minibathces,
-then each minibatch contains two datapoints. In particular we have
+As an example, suppose we have $10$ data points $(\mathbf{x}_1,\cdots, \mathbf{x}_{10})$ 
+and we choose to have $M=5$ minibathces,
+then each minibatch contains two data points. In particular we have
 $B_1 = (\mathbf{x}_1,\mathbf{x}_2), \cdots, B_5 =
 (\mathbf{x}_9,\mathbf{x}_{10})$. Note that if you choose $M=1$ you
-have only a single batch with all datapoints and on the other extreme,
+have only a single batch with all data points and on the other extreme,
 you may choose $M=n$ resulting in a minibatch for each datapoint, i.e
 $B_k = \mathbf{x}_k$.
 
 The idea is now to approximate the gradient by replacing the sum over
-all datapoints with a sum over the datapoints in one the minibatches
+all data points with a sum over the data points in one the minibatches
 picked at random in each gradient descent step 
 !bt
 \[
-\nabla_\beta
+\nabla_{\beta}
 C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
 \mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
 c_i(\mathbf{x}_i, \mathbf{\beta}).
@@ -522,8 +524,8 @@ Taking the gradient only on a subset of the data has two important
 benefits. First, it introduces randomness which decreases the chance
 that our opmization scheme gets stuck in a local minima. Second, if
 the size of the minibatches are small relative to the number of
-datapoints ($M < n$), the computation of the gradient is much
-cheaper since we sum over the datapoints in the k-th minibatch and not
+datapoints ($M <  n$), the computation of the gradient is much
+cheaper since we sum over the datapoints in the $k-th$ minibatch and not
 all $n$ datapoints.
 
 !split
@@ -547,7 +549,7 @@ Another approach is to let the step length $\gamma_j$ depend on the
 number of epochs in such a way that it becomes very small after a
 reasonable time such that we do not move at all.
 
-As an example, let $e = 0,1,2,3,\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \cdot m + i$ where $m$ is the number of minibatches and $i=0,\cdots,m-1$. Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.
+As an example, let $e = 0,1,2,3,\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \cdot m + i$ where $m$ is the number of minibatches and $i=0,\cdots,m-1$. Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.
 
 In this way we can fix the number of epochs, compute $\beta$ and
 evaluate the cost function at the end. Repeating the computation will