From 999bca6d696b2d63a351d59c3fc65206734259a5 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Tue, 17 Sep 2024 14:12:18 +0200 Subject: [PATCH] added video from exercise session --- doc/pub/week38/html/._week38-bs003.html | 1 + doc/pub/week38/html/week38-reveal.html | 2 + doc/pub/week38/html/week38-solarized.html | 1 + doc/pub/week38/html/week38.html | 1 + doc/pub/week38/ipynb/ipynb-week38-src.tar.gz | Bin 193 -> 192 bytes doc/pub/week38/ipynb/week38.ipynb | 1119 ++++++++++++------ doc/src/week38/week38.do.txt | 1 + 7 files changed, 732 insertions(+), 393 deletions(-) diff --git a/doc/pub/week38/html/._week38-bs003.html b/doc/pub/week38/html/._week38-bs003.html index 9c7a8c5bd..25b7cd9ff 100644 --- a/doc/pub/week38/html/._week38-bs003.html +++ b/doc/pub/week38/html/._week38-bs003.html @@ -329,6 +329,7 @@ MathJax.Hub.Config({
  • Resampling techniques, cross-validation examples included here, see also the lectures from last week on the bootstrap method
  • Exercise for week 38 on the bias-variance tradeoff, see also the video from the lab session from week 37 at https://youtu.be/omLmp_kkie0
  • Work on project 1, in particular resampling methods like cross-validation and bootstrap.
    • +
  • Video on cross-validation from exercise session
    • diff --git a/doc/pub/week38/html/week38-reveal.html b/doc/pub/week38/html/week38-reveal.html index fc091a161..3cbc48741 100644 --- a/doc/pub/week38/html/week38-reveal.html +++ b/doc/pub/week38/html/week38-reveal.html @@ -255,6 +255,8 @@ MathJax.Hub.Config({

  • Exercise for week 38 on the bias-variance tradeoff, see also the video from the lab session from week 37 at https://youtu.be/omLmp_kkie0

  • Work on project 1, in particular resampling methods like cross-validation and bootstrap.
    • + +

  • Video on cross-validation from exercise session
    • diff --git a/doc/pub/week38/html/week38-solarized.html b/doc/pub/week38/html/week38-solarized.html index f6750d75a..0675122f4 100644 --- a/doc/pub/week38/html/week38-solarized.html +++ b/doc/pub/week38/html/week38-solarized.html @@ -324,6 +324,7 @@ MathJax.Hub.Config({
  • Resampling techniques, cross-validation examples included here, see also the lectures from last week on the bootstrap method
  • Exercise for week 38 on the bias-variance tradeoff, see also the video from the lab session from week 37 at https://youtu.be/omLmp_kkie0
  • Work on project 1, in particular resampling methods like cross-validation and bootstrap.
    • +
  • Video on cross-validation from exercise session
    • diff --git a/doc/pub/week38/html/week38.html b/doc/pub/week38/html/week38.html index 4f5f82b21..51d5d5167 100644 --- a/doc/pub/week38/html/week38.html +++ b/doc/pub/week38/html/week38.html @@ -401,6 +401,7 @@ MathJax.Hub.Config({
  • Resampling techniques, cross-validation examples included here, see also the lectures from last week on the bootstrap method
  • Exercise for week 38 on the bias-variance tradeoff, see also the video from the lab session from week 37 at https://youtu.be/omLmp_kkie0
  • Work on project 1, in particular resampling methods like cross-validation and bootstrap.
    • +
  • Video on cross-validation from exercise session
    • diff --git a/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz b/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz index c5cf6a1d44d6ef1f0234a5925d0d908c165e7621..076993818441afcc05dda104bde7ccb5e43108c2 100644 GIT binary patch literal 192 zcmV;x06+g9iwFP`a_MFO1MSaC3c@fD2H>uHia9|^(wAGoE?fvAULd7Y8*7uAq-bw% zAD}D6O%WmA=4Y5;m^ozY%`OY<-Fk~5gd|Y}W2Q+wC1G{>B`ISmC5!=~VayT_ z$Z{vWbjET$oYGWhlm_LyxnZmftix3x+Fs^R uC^TaWG+sNQ5xDGu7lp7wNq+HLtxlRZCh&j#jN>?t^R)-Ox&J=^2mk=ab5gGW literal 193 zcmV;y06za8iwFR{1?gr01MSaC3c@fD2H>uHia9|^(k68)*o6y0#0#V}wXrs-Ns9LN z_5r$5+!PV=ZGMIshM7aQ-t4l--CeL4gb+&-7&B#bN<`**f-wc0u#^x)SV{rmn8qvs zwA@KAopsy{r!>`BC@a*vxnZm;dp+4d@j vLJK>#z=&%njR3BC;6)*w)QVri*65Srjg7)zKjV3x=Y8z~ntWR^00;m8`tezt diff --git a/doc/pub/week38/ipynb/week38.ipynb b/doc/pub/week38/ipynb/week38.ipynb index a01340aed..68998261e 100644 --- a/doc/pub/week38/ipynb/week38.ipynb +++ b/doc/pub/week38/ipynb/week38.ipynb @@ -2,8 +2,10 @@ "cells": [ { "cell_type": "markdown", - "id": "54608ce4", - "metadata": {}, + "id": "a811ba80", + "metadata": { + "editable": true + }, "source": [ "\n", @@ -12,8 +14,10 @@ }, { "cell_type": "markdown", - "id": "702fc25a", - "metadata": {}, + "id": "e5014a9c", + "metadata": { + "editable": true + }, "source": [ "# Week 38: Logistic Regression and Optimization\n", "**Morten Hjorth-Jensen**, Department of Physics and Center for Computing in Science Education, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", @@ -23,8 +27,10 @@ }, { "cell_type": "markdown", - "id": "708a3ac3", - "metadata": {}, + "id": "023eb6d1", + "metadata": { + "editable": true + }, "source": [ "## Plans for week 38, lecture Monday September 16\n", "\n", @@ -41,8 +47,10 @@ }, { "cell_type": "markdown", - "id": "ca760e72", - "metadata": {}, + "id": "e981c015", + "metadata": { + "editable": true + }, "source": [ "## Suggested reading and videos\n", " * Readings and Videos:\n", @@ -62,8 +70,10 @@ }, { "cell_type": "markdown", - "id": "a9fd5329", - "metadata": {}, + "id": "11590c09", + "metadata": { + "editable": true + }, "source": [ "## Plans for the lab sessions\n", "\n", @@ -75,21 +85,27 @@ "\n", " * Exercise for week 38 on the bias-variance tradeoff, see also the video from the lab session from week 37 at \n", "\n", - " * Work on project 1, in particular resampling methods like cross-validation and bootstrap." + " * Work on project 1, in particular resampling methods like cross-validation and bootstrap.\n", + "\n", + " * [Video on cross-validation from exercise session](https://youtu.be/T9jjWsmsd1o)" ] }, { "cell_type": "markdown", - "id": "63b2922e", - "metadata": {}, + "id": "57e011be", + "metadata": { + "editable": true + }, "source": [ "## Material for lecture Monday September 16" ] }, { "cell_type": "markdown", - "id": "d5f9f534", - "metadata": {}, + "id": "0896e712", + "metadata": { + "editable": true + }, "source": [ "## Logistic Regression\n", "\n", @@ -108,8 +124,10 @@ }, { "cell_type": "markdown", - "id": "0143d162", - "metadata": {}, + "id": "44bb3650", + "metadata": { + "editable": true + }, "source": [ "## Classification problems\n", "\n", @@ -132,8 +150,10 @@ }, { "cell_type": "markdown", - "id": "ef85093d", - "metadata": {}, + "id": "921c6771", + "metadata": { + "editable": true + }, "source": [ "## Optimization and Deep learning\n", "\n", @@ -154,8 +174,10 @@ }, { "cell_type": "markdown", - "id": "99320f0f", - "metadata": {}, + "id": "f80e9666", + "metadata": { + "editable": true + }, "source": [ "## Basics\n", "\n", @@ -177,8 +199,10 @@ }, { "cell_type": "markdown", - "id": "44bcf7a3", - "metadata": {}, + "id": "952f8119", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y_i = \\begin{bmatrix} 0 & \\mathrm{no}\\\\ 1 & \\mathrm{yes} \\end{bmatrix}.\n", @@ -187,8 +211,10 @@ }, { "cell_type": "markdown", - "id": "403aa0cd", - "metadata": {}, + "id": "9b587b40", + "metadata": { + "editable": true + }, "source": [ "## Linear classifier\n", "\n", @@ -203,8 +229,10 @@ }, { "cell_type": "markdown", - "id": "7703e42f", - "metadata": {}, + "id": "bfb711d7", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -219,8 +247,10 @@ }, { "cell_type": "markdown", - "id": "025d7cc2", - "metadata": {}, + "id": "0acaaf3c", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n", "$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors." @@ -228,8 +258,10 @@ }, { "cell_type": "markdown", - "id": "02656176", - "metadata": {}, + "id": "73564ce7", + "metadata": { + "editable": true + }, "source": [ "## Some selected properties\n", "\n", @@ -253,8 +285,10 @@ }, { "cell_type": "markdown", - "id": "190a0e98", - "metadata": {}, + "id": "ef6011fd", + "metadata": { + "editable": true + }, "source": [ "## Simple example\n", "\n", @@ -264,8 +298,11 @@ { "cell_type": "code", "execution_count": 1, - "id": "d3aa4736", - "metadata": {}, + "id": "3444ad7b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "%matplotlib inline\n", @@ -328,8 +365,10 @@ }, { "cell_type": "markdown", - "id": "b52054ec", - "metadata": {}, + "id": "01d01242", + "metadata": { + "editable": true + }, "source": [ "## Plotting the mean value for each group\n", "\n", @@ -339,8 +378,11 @@ { "cell_type": "code", "execution_count": 2, - "id": "8c348f42", - "metadata": {}, + "id": "143c59fe", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "agegroupmean = np.array([0.1, 0.133, 0.250, 0.333, 0.462, 0.625, 0.765, 0.800])\n", @@ -355,8 +397,10 @@ }, { "cell_type": "markdown", - "id": "6df16129", - "metadata": {}, + "id": "42136436", + "metadata": { + "editable": true + }, "source": [ "We are now trying to find a function $f(y\\vert x)$, that is a function which gives us an expected value for the output $y$ with a given input $x$.\n", "In standard linear regression with a linear dependence on $x$, we would write this in terms of our model" @@ -364,8 +408,10 @@ }, { "cell_type": "markdown", - "id": "9a89924c", - "metadata": {}, + "id": "e8a7f059", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(y_i\\vert x_i)=\\beta_0+\\beta_1 x_i.\n", @@ -374,8 +420,10 @@ }, { "cell_type": "markdown", - "id": "bf2297a8", - "metadata": {}, + "id": "f1c0bcf8", + "metadata": { + "editable": true + }, "source": [ "This expression implies however that $f(y_i\\vert x_i)$ could take any\n", "value from minus infinity to plus infinity. If we however let\n", @@ -391,8 +439,10 @@ }, { "cell_type": "markdown", - "id": "a865cd76", - "metadata": {}, + "id": "e4fd2845", + "metadata": { + "editable": true + }, "source": [ "## The logistic function\n", "\n", @@ -411,8 +461,10 @@ }, { "cell_type": "markdown", - "id": "4137be07", - "metadata": {}, + "id": "f4bb77ad", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(t) = \\frac{1}{1+\\mathrm \\exp{-t}}=\\frac{\\exp{t}}{1+\\mathrm \\exp{t}}.\n", @@ -421,16 +473,20 @@ }, { "cell_type": "markdown", - "id": "42c4a103", - "metadata": {}, + "id": "47fc800d", + "metadata": { + "editable": true + }, "source": [ "Note that $1-p(t)= p(-t)$." ] }, { "cell_type": "markdown", - "id": "9b193f51", - "metadata": {}, + "id": "0fe9154b", + "metadata": { + "editable": true + }, "source": [ "## Examples of likelihood functions used in logistic regression and nueral networks\n", "\n", @@ -440,8 +496,11 @@ { "cell_type": "code", "execution_count": 3, - "id": "02f9e114", - "metadata": {}, + "id": "150c4acd", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "\"\"\"The sigmoid function (or the logistic curve) is a\n", @@ -502,8 +561,10 @@ }, { "cell_type": "markdown", - "id": "cf81cc5c", - "metadata": {}, + "id": "9c1d64b9", + "metadata": { + "editable": true + }, "source": [ "## Two parameters\n", "\n", @@ -512,8 +573,10 @@ }, { "cell_type": "markdown", - "id": "905fc981", - "metadata": {}, + "id": "d1929423", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -525,8 +588,10 @@ }, { "cell_type": "markdown", - "id": "2c734796", - "metadata": {}, + "id": "1698b9e7", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n", "\n", @@ -535,8 +600,10 @@ }, { "cell_type": "markdown", - "id": "c72d9c5f", - "metadata": {}, + "id": "eff2f862", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n", @@ -545,8 +612,10 @@ }, { "cell_type": "markdown", - "id": "bc0af9ee", - "metadata": {}, + "id": "640f9f45", + "metadata": { + "editable": true + }, "source": [ "## Maximum likelihood\n", "\n", @@ -560,8 +629,10 @@ }, { "cell_type": "markdown", - "id": "7362c186", - "metadata": {}, + "id": "f94fafba", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -572,16 +643,20 @@ }, { "cell_type": "markdown", - "id": "553b6180", - "metadata": {}, + "id": "5d457b2e", + "metadata": { + "editable": true + }, "source": [ "from which we obtain the log-likelihood and our **cost/loss** function" ] }, { "cell_type": "markdown", - "id": "4a4286b1", - "metadata": {}, + "id": "683657ba", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n", @@ -590,8 +665,10 @@ }, { "cell_type": "markdown", - "id": "5873ef6a", - "metadata": {}, + "id": "3d17d95b", + "metadata": { + "editable": true + }, "source": [ "## The cost function rewritten\n", "\n", @@ -600,8 +677,10 @@ }, { "cell_type": "markdown", - "id": "97f8406f", - "metadata": {}, + "id": "76cd7541", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", @@ -610,8 +689,10 @@ }, { "cell_type": "markdown", - "id": "d492a29a", - "metadata": {}, + "id": "b88061e7", + "metadata": { + "editable": true + }, "source": [ "The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n", "Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that" @@ -619,8 +700,10 @@ }, { "cell_type": "markdown", - "id": "b384d3ea", - "metadata": {}, + "id": "3c95fe37", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", @@ -629,8 +712,10 @@ }, { "cell_type": "markdown", - "id": "257c19a6", - "metadata": {}, + "id": "4f573bed", + "metadata": { + "editable": true + }, "source": [ "This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n", "in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression." @@ -638,8 +723,10 @@ }, { "cell_type": "markdown", - "id": "4fcc7b46", - "metadata": {}, + "id": "08a700a8", + "metadata": { + "editable": true + }, "source": [ "## Minimizing the cross entropy\n", "\n", @@ -652,8 +739,10 @@ }, { "cell_type": "markdown", - "id": "cccc7482", - "metadata": {}, + "id": "9bd6709b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n", @@ -662,16 +751,20 @@ }, { "cell_type": "markdown", - "id": "ff305cbe", - "metadata": {}, + "id": "98c81b67", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "id": "ab1aa23e", - "metadata": {}, + "id": "5540b76a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n", @@ -680,8 +773,10 @@ }, { "cell_type": "markdown", - "id": "6799086b", - "metadata": {}, + "id": "0018d823", + "metadata": { + "editable": true + }, "source": [ "## A more compact expression\n", "\n", @@ -693,8 +788,10 @@ }, { "cell_type": "markdown", - "id": "cda3c102", - "metadata": {}, + "id": "ee63f4f9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -703,8 +800,10 @@ }, { "cell_type": "markdown", - "id": "b5be8ca2", - "metadata": {}, + "id": "413ff641", + "metadata": { + "editable": true + }, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -712,8 +811,10 @@ }, { "cell_type": "markdown", - "id": "28868ab9", - "metadata": {}, + "id": "337a2c56", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -722,8 +823,10 @@ }, { "cell_type": "markdown", - "id": "e92418db", - "metadata": {}, + "id": "8c3e92fe", + "metadata": { + "editable": true + }, "source": [ "## Extending to more predictors\n", "\n", @@ -732,8 +835,10 @@ }, { "cell_type": "markdown", - "id": "5533c3f5", - "metadata": {}, + "id": "ba84fae7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n", @@ -742,16 +847,20 @@ }, { "cell_type": "markdown", - "id": "9312b78f", - "metadata": {}, + "id": "bddd73d3", + "metadata": { + "editable": true + }, "source": [ "Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to" ] }, { "cell_type": "markdown", - "id": "edbb0614", - "metadata": {}, + "id": "fce6aba6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n", @@ -760,8 +869,10 @@ }, { "cell_type": "markdown", - "id": "394f6fae", - "metadata": {}, + "id": "63325aad", + "metadata": { + "editable": true + }, "source": [ "## Including more classes\n", "\n", @@ -772,8 +883,10 @@ }, { "cell_type": "markdown", - "id": "883253c4", - "metadata": {}, + "id": "1c5878f6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\beta_{10}+\\beta_{11}x_1,\n", @@ -782,16 +895,20 @@ }, { "cell_type": "markdown", - "id": "7c00e2f2", - "metadata": {}, + "id": "2c8a1b85", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "id": "6a4795d0", - "metadata": {}, + "id": "cced4ec8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n", @@ -800,16 +917,20 @@ }, { "cell_type": "markdown", - "id": "deed99f4", - "metadata": {}, + "id": "6efd1ce1", + "metadata": { + "editable": true + }, "source": [ "and so on till the class $C=K-1$ class" ] }, { "cell_type": "markdown", - "id": "3ca93d73", - "metadata": {}, + "id": "933753b8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n", @@ -818,8 +939,10 @@ }, { "cell_type": "markdown", - "id": "89672774", - "metadata": {}, + "id": "ba94450f", + "metadata": { + "editable": true + }, "source": [ "and the model is specified in term of $K-1$ so-called log-odds or\n", "**logit** transformations." @@ -827,8 +950,10 @@ }, { "cell_type": "markdown", - "id": "8a75d099", - "metadata": {}, + "id": "8f174f5d", + "metadata": { + "editable": true + }, "source": [ "## More classes\n", "\n", @@ -848,8 +973,10 @@ }, { "cell_type": "markdown", - "id": "780a498d", - "metadata": {}, + "id": "9ba36ed7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n", @@ -858,16 +985,20 @@ }, { "cell_type": "markdown", - "id": "f19ad43b", - "metadata": {}, + "id": "b5b5ecc6", + "metadata": { + "editable": true + }, "source": [ "It is easy to extend to more predictors. The final class is" ] }, { "cell_type": "markdown", - "id": "25956ba0", - "metadata": {}, + "id": "e6b33699", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n", @@ -876,8 +1007,10 @@ }, { "cell_type": "markdown", - "id": "4cb5ea2b", - "metadata": {}, + "id": "b49a6a23", + "metadata": { + "editable": true + }, "source": [ "and they sum to one. Our earlier discussions were all specialized to\n", "the case with two classes only. It is easy to see from the above that\n", @@ -891,8 +1024,10 @@ }, { "cell_type": "markdown", - "id": "1d6c4caf", - "metadata": {}, + "id": "e9bfd38c", + "metadata": { + "editable": true + }, "source": [ "## Searching for Optimal Regularization Parameters $\\lambda$\n", "\n", @@ -908,8 +1043,11 @@ { "cell_type": "code", "execution_count": 4, - "id": "2cc35ceb", - "metadata": {}, + "id": "5dc4fd0e", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -960,8 +1098,10 @@ }, { "cell_type": "markdown", - "id": "d5c16758", - "metadata": {}, + "id": "70944449", + "metadata": { + "editable": true + }, "source": [ "Here we have performed a rather data greedy calculation as function of the regularization parameter $\\lambda$. There is no resampling here. The latter can easily be added by employing the function **RidgeCV** instead of just calling the **Ridge** function. For **RidgeCV** we need to pass the array of $\\lambda$ values.\n", "By inspecting the figure we can in turn determine which is the optimal regularization parameter.\n", @@ -970,8 +1110,10 @@ }, { "cell_type": "markdown", - "id": "67640d94", - "metadata": {}, + "id": "1538973c", + "metadata": { + "editable": true + }, "source": [ "## Grid Search\n", "\n", @@ -983,8 +1125,11 @@ { "cell_type": "code", "execution_count": 5, - "id": "6aee13c4", - "metadata": {}, + "id": "1c1fdba0", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -1033,8 +1178,10 @@ }, { "cell_type": "markdown", - "id": "fbf16323", - "metadata": {}, + "id": "dd6f78eb", + "metadata": { + "editable": true + }, "source": [ "By default the grid search function includes cross validation with\n", "five folds. The [Scikit-Learn\n", @@ -1046,8 +1193,10 @@ }, { "cell_type": "markdown", - "id": "b32612ca", - "metadata": {}, + "id": "20b7afcb", + "metadata": { + "editable": true + }, "source": [ "## Randomized Grid Search\n", "\n", @@ -1064,8 +1213,11 @@ { "cell_type": "code", "execution_count": 6, - "id": "ec7749b1", - "metadata": {}, + "id": "4810b670", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -1115,8 +1267,10 @@ }, { "cell_type": "markdown", - "id": "8e0c1ec4", - "metadata": {}, + "id": "0696dfc9", + "metadata": { + "editable": true + }, "source": [ "## Wisconsin Cancer Data\n", "\n", @@ -1128,8 +1282,11 @@ { "cell_type": "code", "execution_count": 7, - "id": "9edb0920", - "metadata": {}, + "id": "c55d1159", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", @@ -1152,8 +1309,10 @@ }, { "cell_type": "markdown", - "id": "72beef44", - "metadata": {}, + "id": "b83cd520", + "metadata": { + "editable": true + }, "source": [ "## Using the correlation matrix\n", "\n", @@ -1164,8 +1323,11 @@ { "cell_type": "code", "execution_count": 8, - "id": "bf09a6eb", - "metadata": {}, + "id": "5497a1d8", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", @@ -1206,8 +1368,10 @@ }, { "cell_type": "markdown", - "id": "2d081372", - "metadata": {}, + "id": "e9552a3c", + "metadata": { + "editable": true + }, "source": [ "## Discussing the correlation data\n", "\n", @@ -1229,8 +1393,11 @@ { "cell_type": "code", "execution_count": 9, - "id": "cd0fad75", - "metadata": {}, + "id": "623ddee7", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)" @@ -1238,8 +1405,10 @@ }, { "cell_type": "markdown", - "id": "0cf534b2", - "metadata": {}, + "id": "7a61e306", + "metadata": { + "editable": true + }, "source": [ "and then" ] @@ -1247,8 +1416,11 @@ { "cell_type": "code", "execution_count": 10, - "id": "6429451b", - "metadata": {}, + "id": "859552c6", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "correlation_matrix = cancerpd.corr().round(1)" @@ -1256,8 +1428,10 @@ }, { "cell_type": "markdown", - "id": "ed0138cc", - "metadata": {}, + "id": "43d915d7", + "metadata": { + "editable": true + }, "source": [ "Diagonalizing this matrix we can in turn say something about which\n", "features are of relevance and which are not. This leads us to\n", @@ -1267,8 +1441,10 @@ }, { "cell_type": "markdown", - "id": "9b32814b", - "metadata": {}, + "id": "5c8e892e", + "metadata": { + "editable": true + }, "source": [ "## Other measures in classification studies: Cancer Data again" ] @@ -1276,8 +1452,11 @@ { "cell_type": "code", "execution_count": 11, - "id": "bfe200b0", - "metadata": {}, + "id": "08b680f2", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", @@ -1316,8 +1495,10 @@ }, { "cell_type": "markdown", - "id": "13691108", - "metadata": {}, + "id": "fe0c7fda", + "metadata": { + "editable": true + }, "source": [ "## Optimization, the central part of any Machine Learning algortithm\n", "\n", @@ -1335,8 +1516,10 @@ }, { "cell_type": "markdown", - "id": "9622bff1", - "metadata": {}, + "id": "9df4ecc4", + "metadata": { + "editable": true + }, "source": [ "## Revisiting our Logistic Regression case\n", "\n", @@ -1350,8 +1533,10 @@ }, { "cell_type": "markdown", - "id": "bffe481f", - "metadata": {}, + "id": "a0a65501", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -1363,16 +1548,20 @@ }, { "cell_type": "markdown", - "id": "f967b579", - "metadata": {}, + "id": "8dc1194c", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$." ] }, { "cell_type": "markdown", - "id": "929b2860", - "metadata": {}, + "id": "62d70952", + "metadata": { + "editable": true + }, "source": [ "## The equations to solve\n", "\n", @@ -1385,8 +1574,10 @@ }, { "cell_type": "markdown", - "id": "4d50befc", - "metadata": {}, + "id": "41787d1e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -1395,8 +1586,10 @@ }, { "cell_type": "markdown", - "id": "0c55b82a", - "metadata": {}, + "id": "86fc7282", + "metadata": { + "editable": true + }, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -1404,8 +1597,10 @@ }, { "cell_type": "markdown", - "id": "69de59e3", - "metadata": {}, + "id": "8f4c640e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -1414,16 +1609,20 @@ }, { "cell_type": "markdown", - "id": "5615f25a", - "metadata": {}, + "id": "f37d28ca", + "metadata": { + "editable": true + }, "source": [ "This defines what is called the Hessian matrix." ] }, { "cell_type": "markdown", - "id": "48df73cf", - "metadata": {}, + "id": "9b5cb6dd", + "metadata": { + "editable": true + }, "source": [ "## Solving using Newton-Raphson's method\n", "\n", @@ -1434,8 +1633,10 @@ }, { "cell_type": "markdown", - "id": "cb6d2d95", - "metadata": {}, + "id": "f474d68a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n", @@ -1444,16 +1645,20 @@ }, { "cell_type": "markdown", - "id": "7f1e6122", - "metadata": {}, + "id": "ff928190", + "metadata": { + "editable": true + }, "source": [ "or in matrix form as" ] }, { "cell_type": "markdown", - "id": "830f7b44", - "metadata": {}, + "id": "a9e9efc2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n", @@ -1462,8 +1667,10 @@ }, { "cell_type": "markdown", - "id": "1860e409", - "metadata": {}, + "id": "93061994", + "metadata": { + "editable": true + }, "source": [ "The right-hand side is computed with the old values of $\\beta$. \n", "\n", @@ -1472,8 +1679,10 @@ }, { "cell_type": "markdown", - "id": "49bb5265", - "metadata": {}, + "id": "08acd443", + "metadata": { + "editable": true + }, "source": [ "## Brief reminder on Newton-Raphson's method\n", "\n", @@ -1490,8 +1699,10 @@ }, { "cell_type": "markdown", - "id": "9142c2ca", - "metadata": {}, + "id": "caa94b50", + "metadata": { + "editable": true + }, "source": [ "## The equations\n", "\n", @@ -1504,8 +1715,10 @@ }, { "cell_type": "markdown", - "id": "d98a052f", - "metadata": {}, + "id": "ac3e7ef2", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -1518,8 +1731,10 @@ }, { "cell_type": "markdown", - "id": "3d013035", - "metadata": {}, + "id": "6bd1aafd", + "metadata": { + "editable": true + }, "source": [ "For small enough values of the function and for well-behaved\n", "functions, the terms beyond linear are unimportant, hence we obtain" @@ -1527,8 +1742,10 @@ }, { "cell_type": "markdown", - "id": "67e37396", - "metadata": {}, + "id": "699697a1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(x)+(s-x)f'(x)\\approx 0,\n", @@ -1537,16 +1754,20 @@ }, { "cell_type": "markdown", - "id": "aaad3cb4", - "metadata": {}, + "id": "4efbdd72", + "metadata": { + "editable": true + }, "source": [ "yielding" ] }, { "cell_type": "markdown", - "id": "1c4c599d", - "metadata": {}, + "id": "4bd64a59", + "metadata": { + "editable": true + }, "source": [ "$$\n", "s\\approx x-\\frac{f(x)}{f'(x)}.\n", @@ -1555,16 +1776,20 @@ }, { "cell_type": "markdown", - "id": "947ebe9a", - "metadata": {}, + "id": "358dc6db", + "metadata": { + "editable": true + }, "source": [ "Having in mind an iterative procedure, it is natural to start iterating with" ] }, { "cell_type": "markdown", - "id": "f7a44bbf", - "metadata": {}, + "id": "8a007c48", + "metadata": { + "editable": true + }, "source": [ "$$\n", "x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n", @@ -1573,8 +1798,10 @@ }, { "cell_type": "markdown", - "id": "6db59422", - "metadata": {}, + "id": "e0828d1d", + "metadata": { + "editable": true + }, "source": [ "## Simple geometric interpretation\n", "\n", @@ -1593,8 +1820,10 @@ }, { "cell_type": "markdown", - "id": "d8287387", - "metadata": {}, + "id": "26efa0c4", + "metadata": { + "editable": true + }, "source": [ "## Extending to more than one variable\n", "\n", @@ -1604,8 +1833,10 @@ }, { "cell_type": "markdown", - "id": "80c8b46c", - "metadata": {}, + "id": "8af30001", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n", @@ -1615,16 +1846,20 @@ }, { "cell_type": "markdown", - "id": "49a9d35c", - "metadata": {}, + "id": "77528641", + "metadata": { + "editable": true + }, "source": [ "which we Taylor expand to obtain" ] }, { "cell_type": "markdown", - "id": "78a67613", - "metadata": {}, + "id": "d10154f0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n", @@ -1639,16 +1874,20 @@ }, { "cell_type": "markdown", - "id": "5eb2659a", - "metadata": {}, + "id": "58a6cb05", + "metadata": { + "editable": true + }, "source": [ "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" ] }, { "cell_type": "markdown", - "id": "8a8fe9da", - "metadata": {}, + "id": "87917443", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", @@ -1660,16 +1899,20 @@ }, { "cell_type": "markdown", - "id": "71b1c4c8", - "metadata": {}, + "id": "316440eb", + "metadata": { + "editable": true + }, "source": [ "we can rephrase Newton's method as" ] }, { "cell_type": "markdown", - "id": "881e1a96", - "metadata": {}, + "id": "4ec22184", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n", @@ -1680,16 +1923,20 @@ }, { "cell_type": "markdown", - "id": "6551a36e", - "metadata": {}, + "id": "9da35b82", + "metadata": { + "editable": true + }, "source": [ "where we have defined" ] }, { "cell_type": "markdown", - "id": "c0e75b50", - "metadata": {}, + "id": "61c4f7fc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", @@ -1700,8 +1947,10 @@ }, { "cell_type": "markdown", - "id": "ac897f51", - "metadata": {}, + "id": "ffd39c16", + "metadata": { + "editable": true + }, "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", @@ -1713,8 +1962,10 @@ }, { "cell_type": "markdown", - "id": "8c19a119", - "metadata": {}, + "id": "de590520", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent\n", "\n", @@ -1728,8 +1979,10 @@ }, { "cell_type": "markdown", - "id": "782860fa", - "metadata": {}, + "id": "6a0e0292", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n", @@ -1738,8 +1991,10 @@ }, { "cell_type": "markdown", - "id": "5ec3bde0", - "metadata": {}, + "id": "ec6877a5", + "metadata": { + "editable": true + }, "source": [ "with $\\gamma_k > 0$.\n", "\n", @@ -1750,8 +2005,10 @@ }, { "cell_type": "markdown", - "id": "ddb67ab3", - "metadata": {}, + "id": "b7e72c2f", + "metadata": { + "editable": true + }, "source": [ "## More on Steepest descent\n", "\n", @@ -1763,8 +2020,10 @@ }, { "cell_type": "markdown", - "id": "5a8be57f", - "metadata": {}, + "id": "cae90d84", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n", @@ -1773,8 +2032,10 @@ }, { "cell_type": "markdown", - "id": "4a60b86d", - "metadata": {}, + "id": "1ab31b86", + "metadata": { + "editable": true + }, "source": [ "The parameter $\\gamma_k$ is often referred to as the step length or\n", "the learning rate within the context of Machine Learning." @@ -1782,8 +2043,10 @@ }, { "cell_type": "markdown", - "id": "9ff0c7db", - "metadata": {}, + "id": "87d0d18e", + "metadata": { + "editable": true + }, "source": [ "## The ideal\n", "\n", @@ -1808,8 +2071,10 @@ }, { "cell_type": "markdown", - "id": "0c9c1201", - "metadata": {}, + "id": "c92a82a1", + "metadata": { + "editable": true + }, "source": [ "## The sensitiveness of the gradient descent\n", "\n", @@ -1828,8 +2093,10 @@ }, { "cell_type": "markdown", - "id": "d8f01f9e", - "metadata": {}, + "id": "b5a9af46", + "metadata": { + "editable": true + }, "source": [ "## Convex functions\n", "\n", @@ -1848,8 +2115,10 @@ }, { "cell_type": "markdown", - "id": "fc87a9cf", - "metadata": {}, + "id": "77ee5272", + "metadata": { + "editable": true + }, "source": [ "## Convex function\n", "\n", @@ -1858,8 +2127,10 @@ }, { "cell_type": "markdown", - "id": "a0b661cc", - "metadata": {}, + "id": "282df4c7", + "metadata": { + "editable": true + }, "source": [ "## Conditions on convex functions\n", "\n", @@ -1893,8 +2164,10 @@ }, { "cell_type": "markdown", - "id": "180f4fbd", - "metadata": {}, + "id": "e435596b", + "metadata": { + "editable": true + }, "source": [ "## More on convex functions\n", "\n", @@ -1919,8 +2192,10 @@ }, { "cell_type": "markdown", - "id": "e1ddb722", - "metadata": {}, + "id": "7bc1bf29", + "metadata": { + "editable": true + }, "source": [ "## Some simple problems\n", "\n", @@ -1947,8 +2222,10 @@ }, { "cell_type": "markdown", - "id": "f30489e3", - "metadata": {}, + "id": "90fef1a2", + "metadata": { + "editable": true + }, "source": [ "## Revisiting our first homework\n", "\n", @@ -1969,8 +2246,11 @@ { "cell_type": "code", "execution_count": 12, - "id": "d8c2af3d", - "metadata": {}, + "id": "1c59342a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = 2*np.random.rand(m,1)\n", @@ -1979,8 +2259,10 @@ }, { "cell_type": "markdown", - "id": "a9356dfb", - "metadata": {}, + "id": "79d0e3da", + "metadata": { + "editable": true + }, "source": [ "with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", "The linear regression model is given by" @@ -1988,8 +2270,10 @@ }, { "cell_type": "markdown", - "id": "1c332ff0", - "metadata": {}, + "id": "ec79a08a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n", @@ -1998,16 +2282,20 @@ }, { "cell_type": "markdown", - "id": "6e4cff1f", - "metadata": {}, + "id": "fa4910ae", + "metadata": { + "editable": true + }, "source": [ "such that" ] }, { "cell_type": "markdown", - "id": "239b1e2c", - "metadata": {}, + "id": "e7665e13", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n", @@ -2016,8 +2304,10 @@ }, { "cell_type": "markdown", - "id": "6c8f0818", - "metadata": {}, + "id": "90ffc363", + "metadata": { + "editable": true + }, "source": [ "## Gradient descent example\n", "\n", @@ -2028,8 +2318,10 @@ }, { "cell_type": "markdown", - "id": "3b591ee3", - "metadata": {}, + "id": "3aa073fa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "X \\equiv \\begin{bmatrix}\n", @@ -2042,16 +2334,20 @@ }, { "cell_type": "markdown", - "id": "4c93f254", - "metadata": {}, + "id": "e1ddc571", + "metadata": { + "editable": true + }, "source": [ "The cost/loss/risk function is given by (" ] }, { "cell_type": "markdown", - "id": "16996523", - "metadata": {}, + "id": "5709f3d7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n", @@ -2060,16 +2356,20 @@ }, { "cell_type": "markdown", - "id": "3262b60e", - "metadata": {}, + "id": "b7b3b90f", + "metadata": { + "editable": true + }, "source": [ "and we want to find $\\beta$ such that $C(\\beta)$ is minimized." ] }, { "cell_type": "markdown", - "id": "dfcf41ff", - "metadata": {}, + "id": "6651ef6c", + "metadata": { + "editable": true + }, "source": [ "## The derivative of the cost/loss function\n", "\n", @@ -2078,8 +2378,10 @@ }, { "cell_type": "markdown", - "id": "4fbc9b44", - "metadata": {}, + "id": "646be0cc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -2090,16 +2392,20 @@ }, { "cell_type": "markdown", - "id": "1972249c", - "metadata": {}, + "id": "b6f528c2", + "metadata": { + "editable": true + }, "source": [ "where $X$ is the design matrix defined above." ] }, { "cell_type": "markdown", - "id": "7e4e1d18", - "metadata": {}, + "id": "ae40f47b", + "metadata": { + "editable": true + }, "source": [ "## The Hessian matrix\n", "The Hessian matrix of $C(\\beta)$ is given by" @@ -2107,8 +2413,10 @@ }, { "cell_type": "markdown", - "id": "64c801f1", - "metadata": {}, + "id": "592c656d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -2120,16 +2428,20 @@ }, { "cell_type": "markdown", - "id": "16f76226", - "metadata": {}, + "id": "aaff093b", + "metadata": { + "editable": true + }, "source": [ "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite." ] }, { "cell_type": "markdown", - "id": "51aca262", - "metadata": {}, + "id": "dd177bee", + "metadata": { + "editable": true + }, "source": [ "## Simple program\n", "\n", @@ -2138,8 +2450,10 @@ }, { "cell_type": "markdown", - "id": "3e7e9594", - "metadata": {}, + "id": "94ead835", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n", @@ -2148,8 +2462,10 @@ }, { "cell_type": "markdown", - "id": "31584669", - "metadata": {}, + "id": "75c4e856", + "metadata": { + "editable": true + }, "source": [ "We can use the expression we computed for the gradient and let use a\n", "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", @@ -2161,8 +2477,10 @@ }, { "cell_type": "markdown", - "id": "850d7725", - "metadata": {}, + "id": "228edb14", + "metadata": { + "editable": true + }, "source": [ "## Gradient Descent Example\n", "\n", @@ -2172,8 +2490,11 @@ { "cell_type": "code", "execution_count": 13, - "id": "20e6fb89", - "metadata": {}, + "id": "46647c95", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "\n", @@ -2226,8 +2547,10 @@ }, { "cell_type": "markdown", - "id": "36c762c7", - "metadata": {}, + "id": "e0bb3c65", + "metadata": { + "editable": true + }, "source": [ "## And a corresponding example using **scikit-learn**" ] @@ -2235,8 +2558,11 @@ { "cell_type": "code", "execution_count": 14, - "id": "1ba65efc", - "metadata": {}, + "id": "d29a0ccf", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -2259,8 +2585,10 @@ }, { "cell_type": "markdown", - "id": "845acf36", - "metadata": {}, + "id": "7d20e2cc", + "metadata": { + "editable": true + }, "source": [ "## Gradient descent and Ridge\n", "\n", @@ -2269,8 +2597,10 @@ }, { "cell_type": "markdown", - "id": "ae354b8b", - "metadata": {}, + "id": "52a46927", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n", @@ -2279,16 +2609,20 @@ }, { "cell_type": "markdown", - "id": "a6349642", - "metadata": {}, + "id": "446851f9", + "metadata": { + "editable": true + }, "source": [ "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows" ] }, { "cell_type": "markdown", - "id": "d4a4a710", - "metadata": {}, + "id": "dc10da38", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -2299,16 +2633,20 @@ }, { "cell_type": "markdown", - "id": "3a8c4b43", - "metadata": {}, + "id": "e15d77aa", + "metadata": { + "editable": true + }, "source": [ "We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by" ] }, { "cell_type": "markdown", - "id": "a4efb50c", - "metadata": {}, + "id": "89cd7379", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n", @@ -2317,8 +2655,10 @@ }, { "cell_type": "markdown", - "id": "d180548c", - "metadata": {}, + "id": "20a6a0b6", + "metadata": { + "editable": true + }, "source": [ "## The Hessian matrix for Ridge Regression\n", "The Hessian matrix of Ridge Regression for our simple example is given by" @@ -2326,8 +2666,10 @@ }, { "cell_type": "markdown", - "id": "87acd12f", - "metadata": {}, + "id": "2bcf31af", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -2339,8 +2681,10 @@ }, { "cell_type": "markdown", - "id": "869f478b", - "metadata": {}, + "id": "3f9a5445", + "metadata": { + "editable": true + }, "source": [ "This implies that the Hessian matrix is positive definite, hence the stationary point is a\n", "minimum.\n", @@ -2351,8 +2695,10 @@ }, { "cell_type": "markdown", - "id": "9035cf3a", - "metadata": {}, + "id": "003f6d0d", + "metadata": { + "editable": true + }, "source": [ "## Program example for gradient descent with Ridge Regression" ] @@ -2360,8 +2706,11 @@ { "cell_type": "code", "execution_count": 15, - "id": "10e7bb54", - "metadata": {}, + "id": "bb679580", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from random import random, seed\n", @@ -2418,8 +2767,10 @@ }, { "cell_type": "markdown", - "id": "4a349d39", - "metadata": {}, + "id": "2050684c", + "metadata": { + "editable": true + }, "source": [ "## Using gradient descent methods, limitations\n", "\n", @@ -2438,8 +2789,10 @@ }, { "cell_type": "markdown", - "id": "33424a86", - "metadata": {}, + "id": "6b20b26d", + "metadata": { + "editable": true + }, "source": [ "## Challenge yourself the coming weekend\n", "\n", @@ -2448,16 +2801,20 @@ }, { "cell_type": "markdown", - "id": "facda6dc", - "metadata": {}, + "id": "3570021a", + "metadata": { + "editable": true + }, "source": [ "## Lab session: Material from last week and relevant for the first project" ] }, { "cell_type": "markdown", - "id": "71006b8a", - "metadata": {}, + "id": "c5f36ff0", + "metadata": { + "editable": true + }, "source": [ "## Various steps in cross-validation\n", "\n", @@ -2479,8 +2836,10 @@ }, { "cell_type": "markdown", - "id": "8a7b6bd2", - "metadata": {}, + "id": "a6e47b16", + "metadata": { + "editable": true + }, "source": [ "## How to set up the cross-validation for Ridge and/or Lasso\n", "\n", @@ -2493,8 +2852,10 @@ }, { "cell_type": "markdown", - "id": "01381fbd", - "metadata": {}, + "id": "5b7545c5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -2507,8 +2868,10 @@ }, { "cell_type": "markdown", - "id": "db6812a1", - "metadata": {}, + "id": "fa3a49a2", + "metadata": { + "editable": true + }, "source": [ "* Evaluate the prediction performance of these models on the test set by $C[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\beta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n", "\n", @@ -2519,8 +2882,10 @@ }, { "cell_type": "markdown", - "id": "e98af5ba", - "metadata": {}, + "id": "685304e1", + "metadata": { + "editable": true + }, "source": [ "## Cross-validation in brief\n", "\n", @@ -2545,8 +2910,10 @@ }, { "cell_type": "markdown", - "id": "e3fb483e", - "metadata": {}, + "id": "9cac2104", + "metadata": { + "editable": true + }, "source": [ "## Code Example for Cross-validation and $k$-fold Cross-validation\n", "\n", @@ -2555,21 +2922,13 @@ }, { "cell_type": "code", - "execution_count": 1, - "id": "b78643de", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAGwCAYAAACHJU4LAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABmvklEQVR4nO3dd1yV5f/H8ddhHTYIAoJMQUUUHLhQc+U2s+EqM7VpqWnDyva2rNTKhlrZ0LLSLHOPcu+BIjhQUVBUFJDNYZz79wff+EWOFIHrnMPn+Xicx4Nzn3u87+PB8+G6r/u6dJqmaQghhBBCmCAr1QGEEEIIIa5GChUhhBBCmCwpVIQQQghhsqRQEUIIIYTJkkJFCCGEECZLChUhhBBCmCwpVIQQQghhsmxUB7gZRqOR1NRUXFxc0Ol0quMIIYQQ4jpomkZOTg5+fn5YWV27zcSsC5XU1FQCAgJUxxBCCCFEJaSkpODv73/Ndcy6UHFxcQHKTtTV1VVxGiGEEEJcj+zsbAICAsq/x6/FrAuVvy/3uLq6SqEihBBCmJnr6bYhnWmFEEIIYbKkUBFCCCGEyZJCRQghhBAmy6z7qFyv0tJSiouLVccQFs7W1hZra2vVMYQQwqJYdKGiaRrnzp3j0qVLqqOIWsLd3Z169erJuD5CCFFFLLpQ+btI8fb2xtHRUb48RLXRNI38/HzS0tIA8PX1VZxICCEsg8UWKqWlpeVFiqenp+o4ohZwcHAAIC0tDW9vb7kMJIQQVcBiO9P+3SfF0dFRcRJRm/z9eZM+UUIIUTUstlD5m1zuETVJPm9CCFG1LL5QEUIIIYT5kkJFCCGEECZLChWhlE6n47ffflMdQwghhImSQkUIIcR1KzVqXDiVSsGlHNVRRC0hhYqZk7tLqkdRUZHqCEKYlPPZhUz+NY4Wr69m+V2P4FDHlcMNIomd+R2a0ag6nrBgtapQ0TSN/KKSGn9omnZDOY1GI++99x5hYWHo9XoCAwN5++23OXnyJDqdjp9//pmuXbtib2/PvHnzMBqNvPHGG/j7+6PX62nRogUrV64s319RURHjxo3D19cXe3t7goODmTJlSvnrr732GoGBgej1evz8/HjiiSf+M+PkyZNp3779ZcujoqJ49dVXAdi1axc9e/akbt26uLm50aVLF/bu3XtD78X1nsOlS5d45JFH8PHxwd7enmbNmrF06dLy1xctWkTTpk3R6/UEBwfz4YcfVth/cHAwb731FqNGjcLNzY2HH34YgK1bt9K5c2ccHBwICAjgiSeeIC8vr1LnIIS52j9/Cbe/u5IfdyaTYyjhjKs3AOFJB2kxfiS7u9+BIS9fcUphqSx2wLcrKSguJeKVVTV+3IQ3euNod/1v9eTJk5kzZw7Tp0+nU6dOnD17lsOHD5e//txzz/Hhhx8yd+5c9Ho9H330ER9++CGzZs2iZcuWfP3119x+++3Ex8fTsGFDPv74Y5YsWcLPP/9MYGAgKSkppKSkALBw4UKmT5/OggULaNq0KefOnWP//v3/mXH48OG8++67HD9+nNDQUADi4+OJi4tj4cKFAOTk5DBy5Eg+/vhjAD788EP69etHYmIiLi4u1/1+ANc8B6PRSN++fcnJyWHevHmEhoaSkJBQPuDanj17GDJkCK+99hpDhw5l69atPP7443h6ejJq1KjyY7z//vu8/PLLvPTSSwDExcXRu3dv3nzzTb766isuXLjAuHHjGDduHHPnzr2h/EKYqwNfLqDJoyN4Nawtn499lxf6R9DmrT6kJb7I8Vffpe3Cr2mz4Q9iO/am6fa12NrrVUcWFkan3eif+yYkOzsbNzc3srKycHV1rfBaYWEhSUlJhISEYG9vD0B+UYnJFyo5OTl4eXkxc+ZMHnrooQqvnTx5kpCQEGbMmMGECRPKl9evX5+xY8fywgsvlC9r27Ytbdq04dNPP+WJJ54gPj6etWvXXjbOx7Rp05g1axYHDx7E1tb2hs6refPmDBo0iJdffhmAF154gbVr17Jz584rrl9aWkqdOnX44YcfuO2224CyzrSLFy/mjjvuuOaxrnUOq1evpm/fvhw6dIhGjRpdtu3w4cO5cOECq1evLl/27LPPsmzZMuLj44GyFpWWLVuyePHi8nXuv/9+HBwcmDVrVvmyzZs306VLF/Ly8so/V/90pc+dEObq1JY91O1+C05FBexteytN1y9F71Dxcx339c80fHQE9iVF7OgzlHYrFihKK8zJtb6//61Wtag42FqT8EZvJce9XocOHcJgMHDrrbdedZ3WrVuX/5ydnU1qaiodO3assE7Hjh3LW0ZGjRpFz549ady4MX369OG2226jV69eAAwePJgZM2bQoEED+vTpQ79+/RgwYAA2Nv/90Rg+fDhff/01L7/8Mpqm8eOPPzJx4sTy19PS0njllVf4888/OX/+PKWlpeTn55OcnHzd78ffrnUOsbGx+Pv7X7FIgbL3dODAgZe9PzNmzKC0tLS85eWf7yuUtcQcO3aM+fPnly/TNA2j0UhSUhJNmjS54fMQwlyUGIooHnYPTkUFJDRsQbO/lmLncHnxHfnAEGLzC4kaP4rWq35hy5LH6Hh7FwWJhaWqVYWKTqe7oUswKvw9X8y1ODk5Xbbs360MmqaVL2vVqhVJSUmsWLGCtWvXMmTIEHr06MHChQsJCAjgyJEjrFmzhrVr1/L444/z/vvvs2HDhv9sYbn33nt5/vnn2bt3LwUFBaSkpDBs2LDy10eNGsWFCxeYMWMGQUFB6PV6YmJiKtVR9Vrn8F/v2T/fi38u+7d/v69Go5FHH330in12AgMDb/gchDAnu599i/anE8m2d8Z7+WLsHK/eQthi3P2sOnSMmbkepB0sZk2vYlztb6yFVoirqVWdac1Bw4YNcXBwYN26dde1vqurK35+fmzevLnC8q1bt1b4i9/V1ZWhQ4cyZ84cfvrpJxYtWkRGRgZQVhzdfvvtfPzxx6xfv55t27YRFxf3n8f29/enc+fOzJ8/n/nz59OjRw98fHzKX9+0aRNPPPEE/fr1K+/IevHixes6r6ud65XOISoqitOnT3P06NErbhcREXHF96dRo0bXnDiwVatWxMfHExYWdtnDzs6u0uchhKlLO3ScZl+UdTg/NPFF6oYF/+c2XWa8Sm5kC85nG5iy/PB/ri/E9TLt5oVayN7enueee45nn30WOzs7OnbsyIULF4iPj7/q5aBJkybx6quvEhoaSosWLZg7dy6xsbHllyymT5+Or68vLVq0wMrKil9++YV69erh7u7ON998Q2lpKe3atcPR0ZHvv/8eBwcHgoKCrivv8OHDee211ygqKmL69OkVXgsLC+P777+ndevWZGdnM2nSpOtqMbqSa51Dly5d6Ny5M3fffTfTpk0jLCyMw4cPo9Pp6NOnD08//TRt2rThzTffZOjQoWzbto2ZM2fy2WefXfOYzz33HO3bt2fs2LE8/PDDODk5cejQIdasWcMnn3xSqfMQwhycHPsMbYvyORLSlDZvTrqubextrXn3rkiGzt7O5jU7OeEPDdpGVnNSUStoCmVnZ2sTJkzQAgMDNXt7ey0mJkbbuXPndW+flZWlAVpWVtZlrxUUFGgJCQlaQUFBVUauEaWlpdpbb72lBQUFaba2tlpgYKD2zjvvaElJSRqg7du377L1X3/9da1+/fqara2t1rx5c23FihXlr8+ePVtr0aKF5uTkpLm6umq33nqrtnfvXk3TNG3x4sVau3btNFdXV83JyUlr3769tnbt2uvOmpmZqen1es3R0VHLycmp8NrevXu11q1ba3q9XmvYsKH2yy+/aEFBQdr06dPL1wG0xYsX/+dxrnUOmqZp6enp2ujRozVPT0/N3t5ea9asmbZ06dLy1xcuXKhFRESUv5/vv/9+hf3/O9ffdu7cqfXs2VNzdnbWnJyctKioKO3tt9++ak5z/twJoWmalpKRp90z/F1tY1AL7fCvK294++8feFErsrLW9rTpXg3phKW41vf3vym962fo0KEcPHiQzz//HD8/P+bNm8f06dNJSEigfv36/7n9jd71I0R1k8+dMHev/H6Q77adokOoJz88fPlYSf8laf0OgrrFYIXG8VUbCe11SzWkFObuRu76UdZHpaCggEWLFjF16lQ6d+5MWFgYr732GiEhIXz++eeqYgkhRK2VllPIgl1l4xON6xZWqX2EdG3HvpiyO/KyJ79SZdlE7aWsUCkpKaG0tPSyvzodHBwu6/j4N4PBQHZ2doWHqB6bNm3C2dn5qo+q9s4771z1WH379q3y4wkhLpcw+W0mrvuaXk4FxIR6Vno/dae+BUDzvRs4sye+quKJWkpZZ1oXFxdiYmJ48803adKkCT4+Pvz444/s2LGDhg0bXnGbKVOm8Prrr9dw0tqpdevWxMbG1tjxxowZw5AhQ674WmU74Aohrl9pcQnh82bRNesCu2/vctkt/TciqFNrDjRtR1T8DlLe/oD6v8pIzqLylPZROX78OA888AAbN27E2tqaVq1a0ahRI/bu3UtCQsJl6xsMBgwGQ/nz7OxsAgICpI+KMBnyuRPmav+cH2n+yL1ccnDB/vxZ7F0uH6/pRsR+/j0tHr+fbHtnbFJP41jHrYqSCktgFn1UAEJDQ9mwYQO5ubmkpKSwc+dOiouLCQkJueL6er0eV1fXCg8hhBA3r3TWbAAO97zjposUgKiH7uGMhy/WpSVs+XntTe9P1F4mMY6Kk5MTTk5OZGZmsmrVKqZOnao6khBC1BoXjycTtXcjAL5Pj6+SfVrZ2rBz6ixeOVhAg5J69KySvYraSGmLyqpVq1i5ciVJSUmsWbOGbt260bhxY0aPHq0ylhBC1CrHP52LjWbkaFATgjq3qbL93jK0F/kOzuxPucSxtNwq26+oXZQWKllZWYwdO5bw8HDuv/9+OnXqxOrVq294Fl8hhBCV57rkVwAyBtxVpfut66ynayMvAFb8eaBK9y1qD6WXfoYMGXLVOz2EEEJUv3OZ+exxqY+X4ylCHhtZ5fsf4VXM08+Nx9OQS+ndqVjbmkSPA2FGZFLCWmjUqFHccccd11yna9euTJw4sUbyCCHUWRZ/npd6j2Xs+3/gE3HloSFuRkyX5vhnX8An6wIJ83+v8v0LyyeFigm6UiGxcOFC7O3tmTp1Kq+99ho6ne6yx9q10rNeCHFjlh5IBaBvc/9q2b/e2YlDncsGbSyY/2O1HENYNilUzMCXX37J8OHDmTlzJs8++ywATZs25ezZsxUenTt3VpxUCGFOLpxMpXTHTnSakX6RvtV2HKfhwwBouHUtJYaiajuOsEy1s1DJy7v6o7Dw+tctKPjvdW/S1KlTGTduHD/88AMPPfRQ+XIbGxvq1atX4WFnZwdAXFwc3bt3x8HBAU9PTx555BFyc6/e4z4vL4/7778fZ2dnfH19+fDDD286txDC9CV9OY8l3z3FL0vfwdu1+gYoDB96G5mOrtTJz+LwT0ur7TjCMtXOQsXZ+eqPu++uuK6399XX/fccNMHBl69zE55//nnefPNNli5dyt3/znUV+fn59OnThzp16rBr1y5++eUX1q5dy7hx4666zaRJk/jrr79YvHgxq1evZv369ezZs+emsgshTJ/tihUAFLdtV63HsdHbkRjTA4C8HxZU67GE5ZHu1yZqxYoV/P7776xbt47u3btf9npcXFyFyQEjIiLYuXMn8+fPp6CggO+++w4np7LRJWfOnMmAAQN477338PHxqbCf3NxcvvrqK7777jt69iwbkunbb7/F3796rlcLIUxDYU4ejeO2A+A1rGpvS74S+3uGwLpfCdu8htLiErn7R1y32vlJucZlEKytKz5PS7v6ulb/apA6ebLSkf4tKiqKixcv8sorr9CmTRtcXFwqvN64cWOWLFlS/lyv1wNw6NAhmjdvXl6kAHTs2BGj0ciRI0cuK1SOHz9OUVERMTEx5cs8PDxo3LhxlZ2LEML0HP15KVHFhaS5eBLas1O1H6/J8DtY9FkfVgVF82BSOu0a+fz3RkJQWwsVpxuYx6K61v0P9evXZ9GiRXTr1o0+ffqwcuXKCsWKnZ0dYWFhl22nadpVZz290nKFc1IKIRQq+LXsVuGk9t3w/vcfXdXA1l7PpknvsDo2lZDjGVKoiOtWO/uomInAwEA2bNhAWloavXr1Ijs7+z+3iYiIIDY2lrx/dOTdsmULVlZWNGrU6LL1w8LCsLW1Zfv27eXLMjMzOXr0aNWchBDCJNXfsQEA/e0DauyY3cK9Afjr8DVaqoX4FylUTJy/vz/r168nPT2dXr16kZWVdc31hw8fjr29PSNHjuTgwYP89ddfjB8/nhEjRlx22QfA2dmZBx98kEmTJrFu3ToOHjzIqFGjsKqBv7CEEGqk7kvAPz2VEp0VYcNqrlDp0siLyPPH6LtoNmfj5I8hcX3k28gM1K9fnw0bNnDp0iV69uzJpUuXrrquo6Mjq1atIiMjgzZt2jBo0CBuvfVWZs6cedVt3n//fTp37sztt99Ojx496NSpE9HR0dVwJkIIU7A5346h90zhq2FP41y3To0d193RjqmbvubJLT+Q/O1PNXZcYd50mhl3UsjOzsbNzY2srCxcXV0rvFZYWEhSUhIhISHY21ff+ABC/JN87oQ5GPfDXpYeOMuEWxvyZM/LLwlXp22PTCJmzgcciIwh6sDWGj22MB3X+v7+N2lREUKIWsRo1Nh6PB2ATg3r1vjx6907CIBGh/ZQcCmnxo8vzI8UKkIIUYuc2LiT8b9/Qu/kfbQIcK/x4wd3bsM5d2/sS4o48uNvNX58YX6kUBFCiFrk4s+/MXrPHzyesBJb65r/CtBZWXGqfVcADL//UePHF+bH4gsVM+6CI8yQfN6EqXPYugmAgk5dlGWwH1h2p1HQjg1oRqOyHMI8WGyhYmtrC5TNfSNETfn78/b3508IU2IsKSXkyH4APPrcqixHo2EDKbK2wSk/h+SEE8pyCPNgsSPTWltb4+7uTtr/hsB3dHS86oitQtwsTdPIz88nLS0Nd3d3rP89FYMQJuDUpp2EFOaSb2tPgxoYNv9qHNxdeOalb1ic78JrubaMUJZEmAOLLVQA6tWrB1BerAhR3dzd3cs/d0KYmrQV6wgBjodFEqm3U5oluHNbSlcfZUviRUa0D1KaRZg2iy5UdDodvr6+eHt7U1xcrDqOsHC2trbSkiJMmvWWLQDktmmvOAl0DKvLB6uPsvX4RUpLjVgr6NgrzINFFyp/s7a2li8QIUStZ59yEgCXHt3UBgEi67vxyqZv6J6wheOt59Hotu6qIwkTJSWsEELUAqcz87ntnvfp8tjXNBjYU3UcbKytaFl4keBLZ0n/fbnqOMKESaEihBC1wO6TmaDT4d60EY6uzqrjAFDUpSsALpvXK80hTJsUKkIIUQvsPJkBQJugmpuE8L/Uu/s2ABomHqAwO1dxGmGqpFARQohaYOikEcz+9S06W2erjlIuMKYlaa510ZcWk/jrKtVxxL+czy5kw9ELZOQVKc0hhYoQQli4S8lnaX58P70St9Ms3F91nHI6KytOtYwBIHf5SsVpxL/tXr6ZGW9+x9PfblOaQwoVIYSwcCeXrQUg2SsAjxDTKVQA6FZ2t0+dXWq/DMXl6n49i8XznuGR9fOU5pBCRQghLFzh5rIi4FzTVoqTXM7/zj4c8/Bnl0cIeYUy3pUp8TgcB4C+XVulOWrFOCpCCFGbOe7fC4DWpo3iJJfzjQqn46RvOXOpgJCULDo1rKs6kgAMRcVQUABAve4dlWaRFhUhhLBgmtFI8PF4ADy7q5vf51rahngAsCMpXXES8bejafn0fPAzbnl+Ib7Nw5VmkUJFCCEs2OldcbgW5mKwtiWoq/qh86+kXYgHNqUlnNmyW3UU8T8HzlwCIDjMH52V2lJBLv0IIYQFO3b8LOfqR2Dn7EBze73qOFcUY1/IgY+GYm00UvjMHdi7OKmOVOvFnc4CyqY6UE1aVIQQwoJtcA5g8H1T+W3qN6qjXFVgszDy9Y7oS4s5sfwv1XEEcO/zo/jhxxfoUHBWdRS1hUpJSQkvvfQSISEhODg40KBBA9544w2MRqPKWEIIYTH2n74EQItA0xmR9t90VlacatoagKyV6xSnEYU5eTQ5cYAOyQcIDfNTHUftpZ/33nuPL774gm+//ZamTZuye/duRo8ejZubGxMmTFAZTQghzF6RoZiTSefA2p7m/u6q41xTSadOsGMNzju3qo5S6yVv2EEjYymZjm7Ua9pQdRy1LSrbtm1j4MCB9O/fn+DgYAYNGkSvXr3YvVs6VAkhxM06tW4rez4YzMIFkwnydFQd55q8+5fN6ByaeIDiQoPiNLVb5p+bAEgOjVDekRYUFyqdOnVi3bp1HD16FID9+/ezefNm+vXrd8X1DQYD2dnZFR5CCCGuLGP9JqzQsHNxRqfTqY5zTUG3tOWSgwuOxYWcWLVRdZxazWr3LgDyW7ZWnKSM0kLlueee45577iE8PBxbW1tatmzJxIkTueeee664/pQpU3Bzcyt/BAQE1HBiIYQwH1a7yr5w8lqY3oi0/2ZlY01Sk7KcmdJPRal6hw8A4NgpRnGSMkoLlZ9++ol58+bxww8/sHfvXr799ls++OADvv322yuuP3nyZLKyssofKSkpNZxYCCHMh9eh/33hdDTN8VP+7eJdQ5na+X6W1W+hOkqtlXUmjYALZd+tQb26KE5TRmln2kmTJvH8888zbNgwACIjIzl16hRTpkxh5MiRl62v1+vR601zHAAhhDAluRczCTx/CgD/3qbxhfNf6o28l89yAnEpseE1o4a1lWlfrrJERw6fIj8kGp+SXJoEqb/jBxS3qOTn52P1r4461tbWcnuyEELcpJS/tmGFxnk3LzxDzOMyeRNfF5z1NuQUlnDorPRBVGE77owa8jpfTP1RdZRySguVAQMG8Pbbb7Ns2TJOnjzJ4sWLmTZtGnfeeafKWEIIYfaytuwA4GyI2nlaboSNtRXd3I3cdmgjp39fqTpOrbQ/5RKASd3OrvTSzyeffMLLL7/M448/TlpaGn5+fjz66KO88sorKmMJIYTZ2+/iy+lmt+LeubPqKDdk+P6VtF/yMbsv9oHHh6qOU6toRiOnDiWBzokWge6q45TTaZqmqQ5RWdnZ2bi5uZGVlYWrq6vqOEIIYTL6zNjI4XM5zB4RTa+m9VTHuW4Hv/uVZiPv5py7N/Uyz6uOU6ukxh7Cr2UESR5++J5Jwt7ertqOdSPf3+pHchFCCFGlDCWlHEvLBaCZCUwqdyNC+nenRGdFvUtpnE9IVB2nVjm7ej0AJc6u1Vqk3CgpVIQQwsIcjz9J6Pkk6uqt8HWzVx3nhjh5unOyfhgAp5euVZymdineuh2AjKbNFSepSAoVIYSwMLk/LWTV1+P46tc3TX5E2itJj4oGoGTjZsVJahe3g7EAWLUzrXF3pFARQggLo+3dA0B+eITiJJVj3fkWADz3y7xvNaUov5CQU4cB8OllWh2wpVARQggL434kHgC7NtGKk1SOf/8eAASfOUZe+iW1YWqJk+u2YF9SxCUHFwLayqUfIYQQ1aTEUETQ6WMAeHc2rSb861WvWUMm3/sqt4z5iv2ZJarj1AoZq/8C4GSj5uisrRWnqUgKFSGEsCCnt8diX1JEnp0D9VtHqY5TaTn9b+esqxd7TmWqjlIrbHEP5ruW/Unvc5vqKJeRQkUIISzIxY3bAEgOaIiVjWn9ZXwjWgfVAWC3FCrVTtM0frIP5pVej+H8yEOq41xG6ci0QgghqlbJnr0AZIc3U5zk5rTxseex7b/Q4o8TGO//y6yLLlN3OrOAtBwDNlY6mge4q45zGSlUhBDCgiwLv4XNGUaibzO9Jvwb0TjAk5CtP+FYXMiJzbto0NU8+9uYg6N/7SD6dAK61tHY25peQSiXfoQQwkJomsZSx0BmdhiG14A+quPcFBu9HSdCy1qFLqz8U3Eay+b05SwWzX+WZ9Z/qzrKFUmhIoQQFiItx0BmfjFWOgjzdlYd56bltmoLgNXWrYqTWDavA2Xj7tjd0klxkiuTQkUIISzEye376XV0G+2tc02yCf9GOXYvG3jML36v4iSWKyctneCzJwAIuO1WxWmuTAoVIYSwENrixcxe/DZPr/1KdZQqEXzbrRjRUT/jLBePnVQdxyKdXP4n1pqR1Dr18GrcQHWcK5JCRQghLITNwYMAFDU17zt+/ubqU5eTfmVfnil/yASF1SF33UYAUiNaKk5ydVKoCCGEhfA4UTZXi0Mr0/3SuVEXIltRaGPH+cNJqqNYJJc9ZTMml7aPUZzk6uT2ZCGEsADFhQb8z50CwKdTa8Vpqs6F518mMvJemgZ7Yd73MZmeovxCQhPjAPC5rZfiNFcnLSpCCGEBTm+Pxc5YQq6dI/UiG6uOU2WiIkMptrYlPjWLwuJS1XEsSty5HO655x0+7P0IQbeYbnErhYoQQliA9G27AEgJCENnZTn/tQd4OODloqe4VONAyiXVcSzK9lNZxPo1JnH4IyY3EeE/Wc6nWQgharHiffsByA6znNYUAJ1Ox8Tjf7Hqq8cpfe9d1XEsyo6kDADaNfBQnOTapI+KEEJYgEVt+jPf4EH/2023U2RlhTrpaHwxmdgd21VHsRglhiL6fv4mdeo1pt1jpj09gbSoCCGEBdhS4sLSJp3xurWz6ihVrk6PLgAEH92PsUT6qVSFE6s3cc/upby5dhbhfm6q41yTFCpCCGHmLuUXcTarEIBG9VwUp6l6IT1vodDGDveCHFK271MdxyJkLF8DwInwlljZmvbFFSlUhBDCzJ3cvIeHd/xK30vHcLW3VR2nytk52nMiJAKA8yvWKU5jGRy2bQagsOMtipP8NylUhBDCzBlWrubF9V/z6I5FqqNUm6z/TVCokwkKb1ppcQkNDpe1THn2M93xU/4mhYoQQpg5XVzZoF0F4U0VJ6k+jl3L/vKvd1AmKLxZSeu24GLIJ0fvSIOeHVXH+U9SqAghhJlzO3YIALuWUYqTVJ/gAT057lGfbb7hXLyUpzqOWUv/YxUAJxq3wNrE+6eA3J4shBBmzVhSSsCZEwDU7dBGcZrq41bfh0EvzicxLZdZZ3Lo7e6kOpLZyo8vK2wLOndTnOT6SIuKEEKYsbOxCTgWF2KwtqV+G8ttUQFoHVwHgD2nMhUnMV+GklIev+VR2j7+LR6PPaQ6znWRQkUIIczY+S3/GzrfNxgbvZ3iNNUrOsgDK2MpqTv3q45itvYlX6KguBSjry8NmwSpjnNd5NKPEEKYMcOBgwBkhjRSnKT6tdMbOPDRMOxKiyl8egD2LnL550ZtSbwAQMewuuh0OsVpro8UKkIIYcYW3zKIt0uDGN65EZbbQ6WMf0QIGbZ6nIsKOLxyA+GD+6mOZHZ6PnEfrYs1Clq8pzrKdZNLP0IIYcbis0s5WC8Mz3YtVUepdjorK5LDWwBwad0GtWHMUPb5izRN3EeXpL20aBqoOs51U1qoBAcHo9PpLnuMHTtWZSwhhDALRqPG8Qu5AIR5OytOUzMM7comXdTvlAkKb9Txn5dirRlJ8QqgXqT5XCpUWqjs2rWLs2fPlj/WrCmbe2Dw4MEqYwkhhFk4l3CcF5fNZMT+lQR6OKqOUyPce3YFIPjIfjSjUW0YM1O0cjUAZ1qb/iBv/6S0j4qXl1eF5++++y6hoaF06dLliusbDAYMBkP58+zs7GrNJ4QQpuzi5h2M2LecpHoh2Fh/ojpOjWjQqzOFNnbUyc8ieUcsgTGtVEcyG767yub30fcx/WHz/8lk+qgUFRUxb948Hnjggav2RJ4yZQpubm7lj4CAgBpOKYQQpiP/f3f8ZASFKU5Sc+wc7TkR3ASA88v/VJzGfJyLO0rghRRKdVY0GHyb6jg3xGQKld9++41Lly4xatSoq64zefJksrKyyh8pKSk1F1AIIUyM1eHDABQ1ClecpGYl3T6UaZ2Gs9EjRHUUs3Fq/kIAjjZohpuv13+sbVpM5vbkr776ir59++Ln53fVdfR6PXq9vgZTCSGE6XI7mQiAbbMIxUlqlv1DD/Cx7W4aGJ14WnUYM7Gz0J6ioBZY9eitOsoNM4lC5dSpU6xdu5Zff/1VdRQhhDALmtGIb+pJADzatlCapaZFB5UNpX/iQh4ZeUV4OFn2iLw3y1BSyueuEeQPe4ul4zupjnPDTOLSz9y5c/H29qZ///6qowghhFm4mHgSV0MepTori5/j59/cHe1o7VBE7yNbObx2q+o4Jm9XUib5RaV4u+hp6ueqOs4NU16oGI1G5s6dy8iRI7GxMYkGHiGEMHnndscBcKZuffROtePW5H96dtM8Zv32Drp581RHMXmH/1iHV24GXRp5mc2w+f+kvFBZu3YtycnJPPDAA6qjCCGE2dgb0pyoCQuY9dQ01VHU6Fg2Foj7vl2Kg5i+nu89y65P72dweoLqKJWivFDp1asXmqbRqJH5jJInhBCqJablkG3vjGuLZqqjKOHbtxsADU4ewpCXrziN6TqzJ56gtGRKdFaE39lTdZxKUV6oCCGEuHHH0v43dL5X7Rg6/9/82zYn3ckdfWkxJ5avVx3HZJ3+cREAR8Ka4+pTV3GaypFCRQghzNBDHz/Hq2tnEW5r+O+VLZDOyopTTaMBuLRyreI0pst+zSoAcrqbZ2sKmMjtyUIIIa7fpeSz9IjfBEBe/e8Up1GnuOMtsHMdTjvkzp8ryb2YSXh8WR8en2F3KU5TedKiIoQQZubsjn0AnHP3xsnTXW0Yhbz6l7UShB3dT3Fh7WxZupaj3y5EX1rMGU8/gju3UR2n0qRQEUIIM5Oz9wAA5/1DFSdRK7hLO168cxK9HvyUg+fzVMcxOaW//QZASude6KzM9+teLv0IIYSZMSaU3WZaENpQcRK1rGysOT9wCKcPnWfnyUxaBnmojmQyikuNPNlhNG3cwnnwUfO97APSoiKEEGbH8UTZHD+6prVrjp8rad+grDjZkZShOIlp2ZmUwWmdAxvb9aFJzw6q49wUKVSEEMLMeKccB8C1Ze0aOv9K2tdz5MGdixnywTOUFpeojmMyVsefA6BHEx+srcxvNNp/kks/QghhRnKzcrEtKus46tuupeI06oUHevDklh9xLsrn2J9bCevdWXUk5TSjke4vjMHeuyExg15SHeemSYuKEEKYkePZJUSPn0+P53/GPaCe6jjK2ejtON64OQAXl61RnMY0HFu5iS4JW5iwZQHtG/mojnPTpFARQggzciwtF3Q66ob4q45iMvJjOgGg37pZcRLTcPH7HwE43DwGe1fzH7lYChUhhDAjiX8Pne9t/l9AVaVO71sBCD60F2NJqeI0amlGIwFrlgJgvOtuxWmqhhQqQghhRtp98BJfLXydDueOqI5iMhr06UK+rZ46+dkkb9mjOo5Sx9dsxj/9DIU2doQ/fK/qOFVCChUhhDAjYQd2cOvxXQQ6WauOYjLsHO05ERYJwPk/VilOo9aFr74HIKFlJ5zr1lGcpmpIoSKEEGaiMCcPv/RUAOq1a6E2jInJadcRg7UNF5POqI6ijGY0ErS27LKPNmSo4jRVR25PFkIIM3F29wFCNCPZeic8w4JUxzEp+icn0NzjFhzdXelr1LAy87FDKiPu8GlOezXA3pBPxIPDVMepMtKiIoQQZiJjVywAqX4hZj13S3VoFhGElZMTGXlFHD6XozqOEktO5PL4nS/w+uercajjqjpOlZFPuhBCmImiuLI5frKDwxQnMT12Nla0CykbTn/r4XOK09Q8o1FjedxZAPq2sqzWNilUhBDCTOiPHgbA2KSJ4iSm6e7cEyyb+wRtnhytOkqNO7h5Hw4nEnHW29C1sZfqOFVK+qgIIYSZyCzRkWPngENUM9VRTFJERCAN0k6Qn3mGovxC7BztVUeqMQVvTWHdmoWsGvQo9ra9VcepUtKiIoQQZqCk1MiYPhOJnPgzHncNUB3HJAV3aUeGkxuOxQaO/bFWdZwaU5iTR5ONKwCoP9CyihSQQkUIIcxCckY+xaUaDnY21PeUUWmvxMrGmqSo9gBkLV2pOE3NiZ81H1dDHufcvIgYdrvqOFVOChUhhDADfw+d38DLqVbeenu9Srt3B6DO1o2Kk9Qc63llg7wl9b0LKxvLGwhQChUhhDADzp/MYO2cMTy4Z4nqKCbNf1DZZbGwpHhy0tIVp6l+F48n0+zAVgD8xj2sOE31kEJFCCHMgF3CQcIyTuNrU7sn3fsvfi2acNrTDxvNyPFFy1XHqXbHPpqDjWbkaFATgjpGq45TLeSuHyGEMAPuJ48BoJc7fv7TsS592XnkBBfybGmhOkw1068q64uTOchyRqL9NylUhBDCxBlLSvE7dwoAzzYt1IYxA8Vvvs1T3+0m0OjII5qGTmeZfXriTmcx5PYX6XdsB68/YZmXfUAu/QghhMk7n5CIU3EhRVY2+EVLi8p/6RDqiZ21FckZ+Zy4mKc6TrX5YWcyxda2MHQI7oG+quNUGylUhBDCxKXtjAUg1csfW72d2jBmwElvQ9vgOjRJO8HBxZY5nkpubgF/7EsB4J62gYrTVC8pVIQQwsQVxMYBkBEUqjiJ+XgsYRUr5j5Bg4+mqI5SLeLf/ojlnzzA40kbyuc4slRSqAghhIk7U2JDvHcD8iMiVUcxG/6DbgOg8ZG95KVfUhumGnjM/4bArPN09rK12D44f5NCRQghTNwPzXvTf/THZEycpDqK2QiMaUmqhy92pSUkLvhDdZwqlbh8Aw1TjlBkbUPjSWNVx6l2yguVM2fOcN999+Hp6YmjoyMtWrRgz549qmMJIYRJ0DSNY/8blTbMS4bOv146KytS2nYGwPCHZRUqmdM+BuBA+57UCfJTnKb6KS1UMjMz6dixI7a2tqxYsYKEhAQ+/PBD3N3dVcYSQgiTcSErn5y8Qqx0ZcPni+tnP7Ds8k/Qjg1oRqPiNFUj81QqURuWAuA43vJbU0DxOCrvvfceAQEBzJ07t3xZcHCwukBCCGFi0pau4dC0Iexu1Br7Kf1VxzErjYYNxDDOlnqX0jixfgcNuseojnTTDr81jZiSIo75N6LJ4L6q49QIpS0qS5YsoXXr1gwePBhvb29atmzJnDlzrrq+wWAgOzu7wkMIISxZXmwc+tJinPQyPueNcnB34VBk2WzKZ7//WXGam1dsKCL0528ByHzoMXRWyntv1AilZ3nixAk+//xzGjZsyKpVqxgzZgxPPPEE33333RXXnzJlCm5ubuWPgICAGk4shBA17FACAAVhjRQHMU/p459i+NC3eKfZANVRbtqKwxd5ZOBkfm3dj6hnHlEdp8YoLVSMRiOtWrXinXfeoWXLljz66KM8/PDDfP7551dcf/LkyWRlZZU/UlJSajixEELULKfjiQBYN41QnMQ8tRraj+0NWhJ/oYCTZj5K7debk4j1a0zylOnonRxVx6kxSgsVX19fIiIq/vI1adKE5OTkK66v1+txdXWt8BBCCEtW70wSAO7RzRUnMU91nOyIaeAJwKr4c4rTVN7eUxnEplzCztqK4e2CVMepUUovenbs2JEjR45UWHb06FGCgmrXP4IQQlxJ1pk06uZmAODbvqXiNObrDm/oNmsO4auzYe+fquNUSsnQe3izyIakRyfg5aJXHadGKS1UnnzySTp06MA777zDkCFD2LlzJ7Nnz2b27NkqYwkhhEk4u2MvbsB5Ny98vCx7mPTq1DXcG4/dS7BCI+3QcbybmNdUBCc37KTtjtW0RkdKszdUx6lxSi/9tGnThsWLF/Pjjz/SrFkz3nzzTWbMmMHw4cNVxhJCCJNwKqeU5Y06EN+8o+ooZq1uoxCONiibdTppzjzFaW7chZfLipPYNt0I6hitOE3NU36/22233cZtt92mOoYQQpicXZ4hfHnnC4zuGEx31WHM3KU+t8FncTgvWwLTXlUd57ql7kug5eYVALi89rLiNGrUjpuwhRDCDCX+PXS+twydf7MCHx4BQJOj+0g7dFxxmuuX8sLr2GhGDjRtR8N+XVXHUUIKFSGEMFGXEpNA02jo7aI6itnza9GEQ6FRWKFx4pMvVce5LucTEmmxZjEA1pMnK06jjhQqQghhgvIysvj93WHETx9MQ32p6jgWIXvQUAC8lixSnOT6HJ/0GvrSYg6FRhFxj/kPWFdZUqgIIYQJSt0eC4DBTk8dPy+1YSxE+LgHSHX1YkO9JhxJTlcd55pOpefxRMMBfNH2LrS336k1w+VfSe09cyGEMGFZe/cDcNavgeIklsPNvx6vfryMN3o8wuL4C6rjXNOMtYlctHdh66PPETG0dk9GKYWKEEKYoOKD8QDkNghTnMSy3B3tD8CivacpLjUqTnNliUnn+W3faQAm9WqsOI16UqgIIYQJsk88CoAW3kRxEsvSPdwHL0cbGsXtYPdPK1XHuaLcQUP54ccXeMAtl0h/N9VxlFM+jooQQojL1U0pu4XWqUWU4iSWxc7Gig+Or6DLTx+xPyEG7u2nOlIFB79fTMu9GyjRWeHXUS77wU20qJSUlLB27VpmzZpFTk4OAKmpqeTm5lZZOCGEqI0Mefn4XjwDgE+7FmrDWKDQx0cDEBm3nXNxRxWn+X8lhiKcnnsGgD19hxLUuY3iRKahUoXKqVOniIyMZODAgYwdO5YLF8o6JU2dOpVnnnmmSgMKIURtk5yayZy2d7Ey4ha8GoeojmNx/Ns1J75xNFZoJL0zTXWccntenkrI2RNk2TvT+PMPVccxGZUqVCZMmEDr1q3JzMzEwcGhfPmdd97JunXrqiycEELURkcLrHiv6yi+eHxKrb4ttToVjXkMgCa//0DBpRzFacpmym706fsAHBrzNO6BvooTmY5K/QZs3ryZl156CTs7uwrLg4KCOHPmTJUEE0KI2ioxreyLU4bOrz5RY0eS6uGLe0EOB975WHUcDj80njr52Zz0CSZ6Su0dhfZKKlWoGI1GSksvHynx9OnTuLjIUM9CCHEzcvfuxys3k4ZeTqqjWCxrWxuSRzwMgO83szCWqBv9d9fhVJz27QEg9/1p2NrrlWUxRZUqVHr27MmMGTPKn+t0OnJzc3n11Vfp18+0elALIYS5ufejyez6dATtjuxUHcWiNXtpItl6J3J1NmzYEq8kQ2FxKc8tPcodIz7k68kzaTbiTiU5TFmlbk+ePn063bp1IyIigsLCQu69914SExOpW7cuP/74Y1VnFEKIWqOkqJj655MB8GrdXHEay+Zctw5zP/6JN44bCY/NosstGlZWuhrNMPPPY5y4mIeXuxN3P/lIjR7bXFSqRcXPz4/Y2FgmTZrEo48+SsuWLXn33XfZt28f3t7eVZ1RCCFqjbOxh9CXFlNgo6delIxKWt3uvK8nzvZ2HDqbzar4czV67MRlf6F/83XsSop5c2BT3Bxta/T45qLSA745ODgwevRoRo8eXZV5hBCiVkvfGUsAkOoTSKitjMlZ3dwd7RjdKYQ5K+I4+sYH9PrhPaxr4H3PSUvH8f7hjM84SxM3a3o0u6Paj2muKtWi8u2337Js2bLy588++yzu7u506NCBU6dOVVk4IYSobQri4gDIDApVnKT2eLBDEMu/m8iEhdPY/fLUaj+eZjRy5M77qJ9xlrPuPrSZLWOmXEulCpV33nmnfPyUbdu2MXPmTKZOnUrdunV58sknqzSgEELUJjaHDwNQ3ChccZLaw81Jz/n7HgCg0SfvkXW6ei8B7XptOq23rqREZ0XWV9/iVl+6TFxLpQqVlJQUwsLKZvT87bffGDRoEI888ghTpkxh06ZNVRpQCCFqE/eTxwDQRzZVnKR2af3eiyTVC6FOfjaHH55Qbcc5+sc6oqa8AMCuB58k/K7e1XYsS1GpQsXZ2Zn09HQAVq9eTY8ePQCwt7enoKCg6tIJIUQtomkaX7YcwJetB+LRub3qOLWKjd6O/A+mA9Bu5c/EfbOoyo9xPiERj3uHYF9SRGzzTrT7/L0qP4YlqvQ4Kg899BAPPfQQR48epX///gDEx8cTFBRUpQGFEKK2OJtVyE/hXXi35yPUbxmhOk6t03T4QHb0HgyAz4QxVXoJ6FJ+Ee/MWou+qICkeiGErV2ClY11le3fklWqUPn000+JiYnhwoULLFq0CE9PTwD27NnDvffeW6UBhRCitkhMK5t9PriuE7bWMsePCpE/ziHFKwDv7IvsGTmeUqN20/vMKSxm5Nc7+d0hkHEPvo9+xTKc69apgrS1Q6XuwXJ3d+eDDz7gwIEDpKWlsWTJEgCio6OrNJwQQtQm6dv3EH06gcDgNqqj1FqOddw48813bB73FE83vZOBSxN47fbK9xfKSErhrc9Wst/ajzqOtrz45P34+chUMzeiUoXKypUruf/++0lPT0fTKlabOp3uivMACSGEuLb6875k0ZpFbDM+BmO6qo5TazXs15Wji5eS+eM+vtl6Ejd7Gyb2aHjDM1mf2rIHqzvu4NWcTE4/NI1Xxg+lkRQpN6xSbYvjxo1j8ODBpKamYjQaKzykSBFCiMpxPZEIgG2k9E9RrX9zP17oV3aLeMGU99jZbxhF+YXXta1mNLL7rY/x6taJgIunKXBw4oPhrWlW3606I1usSrWopKWl8dRTT+Hj41PVeYQQolbSjEZ8U5MAcI+WOX5MwSOdQ/E+l8yAqd9irRk50XAvJbNm0+i27lfd5vjqTRRMfIrWh3YDEN84Gp+lC6kXFlxDqS1PpQqVQYMGsX79ekJDZeREIYSoChmnzuBZkIMRHfXbtVQdR/zPHUO6se/8VwQ/P5EGqcdhwK3EN2xJ7h134dahLbqWLcnWrNmfcommEx4kZt96AAzWtuwd8ThtZ39QI0PyWzKd9u9OJtchPz+fwYMH4+XlRWRkJLa2FSdSeuKJJ6os4LVkZ2fj5uZGVlYWrq6uNXJMIYSoDvHzf6fpfXdwxsOX+umpquOIf8lIOs2J+8fQYssKbDRj+fJbH/qc454BADy98XvG7FjI/nY98Pt0Gn5yi/lV3cj3d6XKvB9++IFVq1bh4ODA+vXr0en+f1psnU5XY4WKEEJYitz9ZXP8XAxoQH3FWcTlPEL88di0lPMJiZx4/1Ocdu+g3qlE6jjYUt/dgSa+rrjGPM2lJlNo3ShEdVyLUqlC5aWXXuKNN97g+eefx+oGe0ELIYS4gvgEAApCGykOIq7FJ6IhPnNnlD9fqC5KrVGpQqWoqIihQ4dKkSKEEFXk9+jeLC12o2t/mftFiH+qVKUxcuRIfvrpp6rOIoQQtdZafX2+b3UbdXp0UR1FCJNSqRaV0tJSpk6dyqpVq4iKirqsM+20adOuaz+vvfYar7/+eoVlPj4+nDtXvVNsCyGEKcnKLyYtxwBAQ29nxWmEMC2VKlTi4uJo2bLs9rmDBw9WeO2fHWuvR9OmTVm7dm35c2trmaRJCFG7JO89yN1x6zgf2gQXe9v/3kCIWqRShcpff/1VdQFsbKhXr16V7U8IIcxN/orVfLh8OgeatgceVh1HCJOivDdsYmIifn5+hISEMGzYME6cOHHVdQ0GA9nZ2RUeQghh7rSD8QDkh8kdP0L8m9JCpV27dnz33XesWrWKOXPmcO7cOTp06EB6evoV158yZQpubm7lj4CAgBpOLIQQVc/x+FEArJrKAGFC/FulRqatLnl5eYSGhvLss8/y1FNPXfa6wWDAYDCUP8/OziYgIEBGphVCmLXz7t74ZF3g8KIVhN/VR3UcIapdtY9MW12cnJyIjIwkMTHxiq/r9Xr0en0NpxJCiOqTnZaOT9YFAHzbRytOI4TpUd5H5Z8MBgOHDh3C19dXdRQhhKgRqdv2ApDm4ombn5fiNEKYHqWFyjPPPMOGDRtISkpix44dDBo0iOzsbEaOHKkylhBC1Jic3fsBOO/fQHESIUyT0ks/p0+f5p577uHixYt4eXnRvn17tm/fTlBQkMpYQghRYzZFdOCTwa/TPSqASNVhhDBBSguVBQsWqDy8EEIod6DQho0Nounds5nqKEKYJJPqoyKEELVN4vlcABp6uyhOIoRpMqm7foQQojbJTb/E0CWzSawbSEOvHqrjCGGSpFARQghFUrft5YltP3HRuQ51nKeqjiOESZJLP0IIoUjWnv/d8VNf7vgR4mqkUBFCCEVK4srm+MkJlTl+hLgaKVSEEEIRh2NHANDJHD9CXJUUKkIIoYh3ynEAnFtGKU4ihOmSQkUIIRTIz8yiXsY5AHzbt1KcRgjTJYWKEEIokLojFis0Mh3d8AjxVx1HCJMltycLIYQCBzyDefCR2XR2LeVN1WGEMGFSqAghhAKJ6QWcquOH1j5QdRQhTJpc+hFCCAVk6Hwhro+0qAghhAJ9vpxCuM6RJne9pDqKECZNChUhhKhhhdm53LnlN6w1Ixc831IdRwiTJpd+hBCihp3Zvg9rzcglBxfqhgSojiOESZNCRQghaljmrlgAzvqFoLOS/4aFuBb5DRFCiBpWsv8AANlhjRUnEcL0SaEihBA1zOHoobIfmkWqDSKEGZBCRQghapjPqUQAXFq3UBtECDMghYoQQtSgnPRLeGRnAFD/lnaK0whh+uT2ZCGEqEFHc40Me+oXooszWFDfW3UcIUyeFCpCCFGDjpzLpdjaFrsmzVRHEcIsyKUfIYSoQUfOZQMQXk+GzhfiekiLihBC1KCO014hMiMbl+gXgCaq4whh8qRQEUKIGqIZjbTZvY46+dkcc3lZdRwhzIJc+hFCiBqSfuwUdfKzKdVZ4d8hWnUcIcyCFCpCCFFDUrfsBuBM3frYuzorTiOEeZBCRQghakj+nlgALgY3VBtECDMihYoQQtQQ6/iDABiaNFWcRAjzIYWKEELUEPcTRwHQt2iuOIkQ5kMKFSGEqAGlpUZKDEUY0eEV00p1HCHMhtyeLIQQNSA5s4C+oz7G3VjEntZRquMIYTakRUUIIWrAkXM5APgH1MXaxlpxGiHMh8kUKlOmTEGn0zFx4kTVUYQQosr9Xag09nFVnEQI82ISl3527drF7NmziYqS5lAhhGVq/faz/HzsKBe8nwOkM60Q10t5i0pubi7Dhw9nzpw51KlT55rrGgwGsrOzKzyEEMIc+Cfspe3pBALc9KqjCGFWlBcqY8eOpX///vTo0eM/150yZQpubm7lj4CAgBpIKIQQN6cwOxf/C6cB8O3URnEaIcyL0kJlwYIF7N27lylTplzX+pMnTyYrK6v8kZKSUs0JhRDi5iVv3Im1ZiTDyY26DYNVxxHCrCjro5KSksKECRNYvXo19vb217WNXq9Hr5dmUyGEebm0dRcAZ4Ia42GlvCFbCLOirFDZs2cPaWlpREf//wyipaWlbNy4kZkzZ2IwGLC2llv4hBDmzxgbC0Bek2ZqgwhhhpQVKrfeeitxcXEVlo0ePZrw8HCee+45KVKEEBbD7WgCADYtW6gNIoQZUlaouLi40KxZxb8unJyc8PT0vGy5EEKYK6NR46TenTrOHtTt1E51HCHMjkmMoyKEEJYqOSOfxwY8i52NFQkdo/97AyFEBSZVqKxfv151BCGEqFIJZ8vGewqv54KNDJ0vxA2T7udCCFGNjpy8AECErwydL0RlSKEihBDVqNvLY9k5cwS9ju9QHUUIsySFihBCVCPfk0fwzsvEt4G/6ihCmCUpVIQQoppknkrFJ6vs0o9/F7njR4jKkEJFCCGqyZkN2wE47emHi5eH4jRCmCcpVIQQoprk7tgNQFqDcMVJhDBfUqgIIUQ1sY47AIChaZTiJEKYLylUhBCimngeOwSAQ9tWipMIYb5MasA3IYSwFIXFpfwZ0Jw0G0eCO7dXHUcIsyWFihBCVINjabm81e1B3B1t2RcRpjqOEGZLLv0IIUQ1SEgtGzo/wtcVnU6nOI0Q5ksKFSGEqAapew/iZMiniQydL8RNkUJFCCGqQZ/3niVuxlB6JW5XHUUIsyaFihBCVLESQxFBKYlYoeHbvoXqOEKYNSlUhBCiiiVv2YNDiYFcO0f82zRXHUcIsyaFihBCVLGL67cCcCo4HCsba8VphDBvUqgIIUQV03btAiC7mbSmCHGzpFARQogq5n44DgDbtm0UJxHC/EmhIoQQVai40EBwSiIA9bp1VJxGCPMnI9MKIUQVSjyTyY/dHyQq/RSDWjdTHUcIsyeFihBCVKED6UV83+o2OoR6MthKGq2FuFnyWySEEFUo7kwWAJH+boqTCGEZpEVFCCGqkOPyP2hS6kKUT6TqKEJYBJ2maZrqEJWVnZ2Nm5sbWVlZuLrKfBpCCLWK8gvB1QW70hLO7D5I/eimqiMJYZJu5PtbLv0IIUQVSd64A7vSErLsnfFr2UR1HCEsghQqQghRRdL/3ARAcoMIdNKRVogqIb9JQghRRax27AAgt5UM9CZEVZFCRQghqki9Q/sBcOwsA70JUVWkUBFCiCpwKfksARdSAAjq3VVtGCEsiBQqQghRBU6tWg9AilcA7oG+asMIYUFkHBUhhKgCG70b88GQN+ge4MRo1WGEsCDSoiKEEFVg58ViNoW0wmbQ3aqjCGFRlBYqn3/+OVFRUbi6uuLq6kpMTAwrVqxQGUkIIW6Y0agRm3IJgJYB7kqzCGFplBYq/v7+vPvuu+zevZvdu3fTvXt3Bg4cSHx8vMpYQghxQ5L3HOTxlV/S69Qewuu5qI4jhEVR2kdlwIABFZ6//fbbfP7552zfvp2mTWXoaSGEeUj7YzWP7VhIQsYxbKxfUR1HCItiMp1pS0tL+eWXX8jLyyMmJuaK6xgMBgwGQ/nz7OzsmoonhBBXpW3fDkBW82jFSYSwPMo708bFxeHs7Ixer2fMmDEsXryYiIiIK647ZcoU3Nzcyh8BAQE1nFYIIS7nFb8PAH2nDoqTCGF5lM+eXFRURHJyMpcuXWLRokV8+eWXbNiw4YrFypVaVAICAmT2ZCGEMtnnL+Lk64O1ZuTC4eN4NW6gOpIQJu9GZk9WfunHzs6OsLAwAFq3bs2uXbv46KOPmDVr1mXr6vV69Hp9TUcUQoirSlqyhuaakTMevtSXIkWIKqf80s+/aZpWodVECCFMWf669QCkRslEhEJUB6UtKi+88AJ9+/YlICCAnJwcFixYwPr161m5cqXKWEIIcd1sE/43nEKnW9QGEcJCKS1Uzp8/z4gRIzh79ixubm5ERUWxcuVKevbsqTKWEEJcl8LiUu697QXqtz/NNyP7qY4jhEVSWqh89dVXKg8vhBA3ZX/KJYqMGrlBDQgMra86jhAWyeT6qAghhLnYdTIDgLbBHuh0OsVphLBMyu/6EUIIc9XipQl8mpFNacSLqqMIYbGkUBFCiEooMRTRcs96nIoKOO4r4zgJUV3k0o8QQlTCybVbcSoqIFvvRHDXdqrjCGGxpFARQohKuLhyLQBJ4S2wtpXGaSGqixQqQghRCfqtWwDIbyfz+whRnaRQEUKIG1RiKCI0fhcAngP6KE4jhGWT9kohLFRG0mlSVm8kraCEPY3bciHHQInRyKC572FrZwP+/jg0b4Z/7y54hshM5Dfi2Ir1hBvyyLJ3JrR3Z9VxhLBoUqgIYSEKc/I4/N2vFP26GL/9O/FPP4MHsNM/gi+GTy1f78Utq/HOy6yw7Qm/UM5370vA+IfxbxtVw8nNT8LxNAp9G1ESGERr6Z8iRLWS3zAhzJimaew+mYHxgQeJ3LKKFsWFFV4/5R1IQcNwRnUIxsfVHjsbK44UTeT4mdPYJZ/CM+kowedP0iD1OA3mzeTQqqWMfncBj3VrSNsQD0VnZfoWOIey8/5pvHV7BK1VhxHCwkmhIoQZys3J5+f95/lhZzLH0nL5IjUNx+JCzrl5capTT+wH3kZwv1sJqu9NENDlnxt3eqXCvjKSTnN83iL0Py/g65BO/HX0In8dvUivMA9eamZPYPuWNXlqJi/PUMK+5LIWqU6NvBWnEcLySWdaIcxIRtJptg1/nKJ6fnw970+OpeXiYGvN4Yef5PDiVfhknKPd0vk0f/ge3Opf35eoR4g/bV6eQFTcNiZ+/Rr3tA3ExkpH/QVzqdepLdsfepoSQ1E1n5n52LfnCPr8PPzrOBDk6ag6jhAWT6dpmqY6RGVlZ2fj5uZGVlYWrq4yMqSwXOlJKSROfJHmy3/GocQAwHc9RqB7803uaOGHi71tlR7v+IVczt8+mA7bVwJwJKQp7ksX4xPRsEqPY4623zGS1kvmsfaecfSZ/5HqOEKYpRv5/pYWFSFMWPb5i2wb/jgOjRvRfsn3OJQYSAwMZ9/0Lxm+/GtGtA+q8iIFINTLmZgty9j1xgyy7Z1pnBSPTdu2JCz4o8qPZW58dm7GRjPi1aKJ6ihC1ArSoiKECSosLuX7jYn0v7sLfllpABwNbILh1ddoNmoQOqua+xvjzJ54Cm+/g9DUY5TorNjz3Nu0m/J8jR3flFw8dpK6DUMAyDx5hjpBfooTCWGepEVFCDNVbChi/o5TdHn/L95ec5yFEd045RPEvulf0jDpIJEPDKnRIgWgfnRT/OL3srtjX2w0I63fe5EfvluNGf+NU2knFywB4Jh/QylShKghctePECbAWFLK3ve/wPfDd/ij1zjOB0ZR390B/w/ewr9NMEGKx+pwcHcheuNSto0Yx28ZNvyUUMzJFYeZ3DccnU6nNFtN0i1fDsCFDl0JU5xFiNpCChUhFNKMRg589RPOr71C69RjADy+93f6jB3GPe0C0dtYK074/3RWVsTM/4z4TSdg2SFmbzyBdV4uz94dXeOtPCqUGIpouK9sfp86Q+9SnEaI2sPy/3cRwkQdXriCQ41b0fyRewlNPUaO3pFtoyYSvWUlozqGmFSR8k8P3dKAd++KxDsnnTsfG8SOkU+ojlQjji5ehWthLpmOrjQc0EN1HCFqDWlREaKGHT6XTep9D9F93S8AFNrYETvwPsKnvUVMoK/idNdnWNtA/H87R6P0ZJj3KdtdXGj/2RTVsarVajyY3+tx2vg4cIcMmy9EjZEWFSFqyIkLuTz5Uyx9P9rEEodASnRW7Oh5N1kH4mm/8CvczaRI+Vund55l20NPA9D+83fZPeVTxYmq17LUEua37If100+pjiJErSKFihDV7OSGnezu2Je59z3H4n1n0DQoGTqU1G17abd6IT5NzLdbZsycD9h+5ygAol6eSMJPy9QGqiYpGfkkpuVibaWjcyMv1XGEqFWk/VKIanJ8zRayXnqVFjv/JBgNXxcvzg+9j/G9Ioj0d1Mdr8q0/flL9sacotXuv/AbfQ8pQRsIsLD5gY7MnMv9e+K40Os23ByqfoA9IcTVSYuKEFVIMxo5+P1i9rfoRGivTrTauQ4rNPa26U7BwkXMfiDGoooUACsba5qs+Z2jQU1wL8ghY/gosguLVceqUgHffM4ba2cx6twe1VGEqHWkUBGiChhKSvlldwoLewyn2f130Xz/Fozo2BPTm6S/ttNq5zrCet2iOma1cXB3wWPdSv5q2olHez/FkwtiMRotY0C4tEPHaZwUD0CDh+9TnEaI2kcu/QhxE87sOciSg+f5KkXjYq6BFv5t6G+7mLied+L3yvNEt2uuOmKNqRsaiOfKP8j4YhvrDqfx0bpEnuzZSHWsm5Y0Zx7ewOGQZoSHh6qOI0StIy0qQtygovxC9nwwm7iIttRvHYnLjA+5mGugnqs9vR8YSFHyadot+5GAWlSk/C3K352372gGwJHPvmHfp98rTnTznJeVDZuf1XeA4iRC1E7SoiLEdTAWl3Do52XkffM9jTevIrowt2w5OhraFPHFfa24tYkPttZS+w9uHUDRb0sY/tsUcpc7ktyqKYExrVTHqpSLx04SnhgLQMBDctlHCBWkUBHiKkpKjew5lcmahPMMGDeE5qfiy19Lc/HkxG2DCZw0nnYtIxSmNE1DXnqIhJ+/JCIxlot33k1e/D6cPN1Vx7phxz7+ivaaseyyj/w7C6GEFCpC/EP2+YskfreQkmXLGX/LI6QVl024V8c3gpBzJznSqRcOI0fQZNgAvGV00quytdfjvXwxF1pFE3z+JHsGDKPV5uVmNyfQmYTjFFtZk3XXYNVRhKi1dJoZz9WenZ2Nm5sbWVlZuLq6qo4jzFBhTh7Hf19D9qq1uG3fTNiJeOyMJQCMGvQasZExdG/sTf8gRzpF+qN3clSc2LwcXriC0KEDsDWWsn3CK7Sf8brqSNftWFouPaZtwLMwh9XP9cDTt67qSEJYjBv5/pY/CUWtkpFrYP+ZLGKTL6EtWsjYL1+jaWnFMT+SvQJIvaUHT43pQ0TXNthIv5NKCx/Ul+3jXqD9x28S/cnbHO7UnvBBfVXHui6/x54BoHmLUClShFBIChVhkTSjkfRjpzi7dS95u/Zgu3cPvkfj+Cz6Dua16g9Ao2I3niot5qKzByej2mLs0pX6d/UjsHUkgYrzW5J2019jz84dRG9fzfpZP+PZuzteLnrVsa6ptLiEXau2g21dBrbwUx1HiFpNaaEyZcoUfv31Vw4fPoyDgwMdOnTgvffeo3HjxipjCTNSYiji3LlMThVbc/xCLmlxR+j3/nP4nTlB3YIc/v13cIuzR9nqNYQW/u60CmxK8kMxBLRrQV0z6zthTnRWVoQv/YmXJ33G997NWf/jXuY92M6kW6riv13IgmmjWBfekY5vblAdR4haTWmhsmHDBsaOHUubNm0oKSnhxRdfpFevXiQkJODk5KQymjARuYYSzl/MpuCvjRSeTKbk9Bl0J5NwOJ2Mx7kUfDLPsyGqJy/2HgeAkyGfZ47tB6BUZ0Vq3fpcCArD0KIVLrd0oGfPWxjk+89J5YJr/qRqISdPd0a++wS/ztzM9hMZvL/yEJP7N1Ud66pKv5gFgFPDEOxtrRWnEaJ2U1qorFy5ssLzuXPn4u3tzZ49e+jcubOiVKK6aJpGrqGES/nFZGblwZ9/UpR2kdILFzGmp2OVno71pUwcLqaxNyCC9zoMJ9dQgkNRIYemD7rqfgOyLxDq5USwpxON6rmw0+8LPJtHUD+mFQEuTgTU4DmKqwvzdub9wc15ddZauj/2PPuemkjLiQ+qjnWZC0dOELl3IwD1nh6vOI0QwqT6qGRlZQHg4eFxxdcNBgMGg6H8eXZ2do3ksnSa0UhJUTGG3HyK8wowoKPQyRVDiZGiAgO2e3ZRkl9AaUEhxoICSgsKMBYY0AoLyPAJ4FiLDuQWlVCUnUP/mW9gk5+LTUEedvl56AvzsS/Mx8FQwKpGMUzqOwEAu5Jijn447KqZLhRBbnTZ3Tc2Ls4c9m9MibMzhZ7eFAcEYd0wDKcmDanbvAmdGjVgnc0//urtE16t75eovH6RvjhnbKddykFyn3uClPatTG6m5WPvfkKMZuRQaBRNurRVHUeIWs9kChVN03jqqafo1KkTzZo1u+I6U6ZM4fXXq//2xkNns9m2dDNN/lwCmgZGDdDQGY2gaWhGjUMde3K6cQs0NDxSkmi77Afg/9f9ezudpnGgQy8SI9tj1DQ8ziXTY9Gcste1/+2z7A1AZzSyp31P9rfujoZGnbRUBs2fBhrl+9QZjejQ0JUa2d76Vv6M6UepUcP94lkmzXkZK2MJ1qWlWBlLsSotxdpY9vMfrXozq9sISowaHpcusnTmA9gYS7E2GrHWjNgCf09eP79Fn/JLKW4FOez/+J6rvle/RXThwwFlPUFsS4t5dfOyq67rmZsJgL2tFXXcXDgUFIHR3gGDmzvF7h4Y69RB5+mJjb8fHuGN+bNTDN6u9jjrbeD1wzfzTypMSMzc6STs3kpEYiwX7ryb/IR9ONYxjRmlDXn5NFz4HQB5o0yvtUeI2shkCpVx48Zx4MABNm/efNV1Jk+ezFNPPVX+PDs7m4CAqm/YP3o+h60rt/PAr7Ouus7SfEfmXyz7z7XDyf2MX/7jVdddVerOT4b6ALQ6c4TJG/646rrrbbxYpm8CQHjaKd7es/6q625xqs92n/YABGZm0ij50FXXtb2UycXcIgBsio04Fhuuuq7eWIqTnTV6W2tcHVw47VmfYls7Suz0lNjaUWpnR4mtHqOdHcYmrbinbQCOdjY46W3Ylj8ZKxdnrN1csXFxxcbdFX0dN/QebjTz8eJwgN//X/OfHH/VDMJy2ert8F76KxejWxNyLondA+4heuNSkxgMbv/UL2ibm0GaiydRTz2iOo4QAhMZ8G38+PH89ttvbNy4kZCQkOverroGfIs7ncXGX/8kes0i0OlAp0OzsgJ06KzKnh/p2JOz4S3Q6aDOuRQiV/0K/3sNndX/frYCnY6UVh240Kysedsp/TyN1ywp3w86HVhZlf98oWlL0pu2QKfTYZ+dSfCfywAdWOnQWVmDDtDp0NnaktuwCblNo7Cx0mFjKMBr91Z0NrZY2dpiZWuDla0NOhsbrGxt0Xy80QICsLHSYa1p6FNPY2Vjg7WdDVZ2ttjY2mLn4oydkwPWNtJ5UFS/hJ+W0eie27HRjGx/6nXaf/iK0jyaprE3vC3RR3ez/dFnaf/Fe0rzCGHJbuT7W2mhomka48ePZ/Hixaxfv56GDRve0PYyMq0Q5m37+JdoP/NtiqxsOLFwGeF39lKWZf2RNB6Zs5VBiZt47ovncatwd5gQoirdyPe30rbWsWPHMm/ePH744QdcXFw4d+4c586do6CgQGUsIUQNaffRG+xteyvJ7vV4bcNpLuZe/ZJkddI0jelrjlJkY4vDww9KkSKECVHaoqLT6a64fO7cuYwaNeo/t5cWFSHMX+7FTIbN2sbBHI2YBp58/2DbGh8MbtPa3YxedQZbez0bn+1m8iPnCmHuzKZFRdO0Kz6up0gRQlgG57p1mPHwLTjaWbPtRDpz5v1Vo8c3lpRSf+Qw1n05hud98qRIEcLEqO9mL4So9cK8XZg6KIoHdy7moQd6s+/juTV27N1vzKBB6nHqFORwx52dauy4QojrI4WKEMIk3BblR0+XImyNpTSaNJbEZdXfspJ1+hwNP3wTgIQHxuNW36fajymEuDFSqAghTEb0T3M42KQ1TkUF1B1yJ8nb9lbr8Q4/+AR18rM4WS+Y6GmvVeuxhBCVI4WKEMJk2NrrCd64msSAxtTJz8KuX1/OJyRWy7HivllEu9W/AJA37SNs7aVvihCmSAoVIYRJca5bB88Na0nxCqDepTSKO3cldX/VTqGQkXQa3/FlI8/u6D2YpvfcXqX7F0JUHSlUhBAmxyPEH5u1q0n18MU/PZUFL83k+IXcKtl3SamRl5YkEOsTykmfYKIWfFUl+xVCVA8pVIQQJsk3KhybzZv4dOBYPm7al6GztrEvOfOm9qlpGi8uPsjys8WMG/YaRWvX4eDuUkWJhRDVQQoVIYTJ8m4SyrB5H9LUz5WLuUU8MmMN296eWal9GUtK+f2Zqfy86xRWOvjk3mgaNWtQxYmFEFVNChUhhEnzdNaz4JH29G7syYeL3yXmpfHsbdeTjKTT172Pgks57LulH3dMe57Jf83ljYHN6BkhtyILYQ6kUBFCmDwXe1s+H9EG265dKLayptXOtVhHNGHbo8+Sn5l11e00o5G4r38mI7Qx0dtXU2RlQ8vbu3Jf+6AaTC+EuBlK5/q5WTLXjxC1z7GVG7B68EEapB4HIM/OgYQOPcl56FG8u3ZEb2NF3v6DFP7+B16/LyT0TNntzefdvLgwczbN7rtDYXohBNzY97dNDWUSQogqEdanC6UnD7PrnZn4fvI+/umptFm/hCdcGrMkruzvrgd3/cbLf34JgMHaln19h9D0q49o5u2pMroQohKkUBFCmB1rWxvavDoR7eUnSPhlBdnzF1DSrh11NT1GTSO5cQv2XeqCoeMthD/zOO0DfVVHFkJUklz6EUIIIUSNupHvb+lMK4QQQgiTJYWKEEIIIUyWFCpCCCGEMFlSqAghhBDCZEmhIoQQQgiTJYWKEEIIIUyWFCpCCCGEMFlSqAghhBDCZEmhIoQQQgiTJYWKEEIIIUyWFCpCCCGEMFlSqAghhBDCZEmhIoQQQgiTJYWKEEIIIUyWjeoAN0PTNKBsumghhBBCmIe/v7f//h6/FrMuVHJycgAICAhQnEQIIYQQNyonJwc3N7drrqPTrqecMVFGo5HU1FRcXFzQ6XRVuu/s7GwCAgJISUnB1dW1SvdtCuT8zJ+ln6Olnx9Y/jnK+Zm/6jpHTdPIycnBz88PK6tr90Ix6xYVKysr/P39q/UYrq6uFvsBBDk/S2Dp52jp5weWf45yfuavOs7xv1pS/iadaYUQQghhsqRQEUIIIYTJkkLlKvR6Pa+++ip6vV51lGoh52f+LP0cLf38wPLPUc7P/JnCOZp1Z1ohhBBCWDZpURFCCCGEyZJCRQghhBAmSwoVIYQQQpgsKVSEEEIIYbKkULkOt99+O4GBgdjb2+Pr68uIESNITU1VHatKnDx5kgcffJCQkBAcHBwIDQ3l1VdfpaioSHW0KvX222/ToUMHHB0dcXd3Vx3npn322WeEhIRgb29PdHQ0mzZtUh2pymzcuJEBAwbg5+eHTqfjt99+Ux2pSk2ZMoU2bdrg4uKCt7c3d9xxB0eOHFEdq0p9/vnnREVFlQ8SFhMTw4oVK1THqhZTpkxBp9MxceJE1VGqzGuvvYZOp6vwqFevnrI8Uqhch27duvHzzz9z5MgRFi1axPHjxxk0aJDqWFXi8OHDGI1GZs2aRXx8PNOnT+eLL77ghRdeUB2tShUVFTF48GAee+wx1VFu2k8//cTEiRN58cUX2bdvH7fccgt9+/YlOTlZdbQqkZeXR/PmzZk5c6bqKNViw4YNjB07lu3bt7NmzRpKSkro1asXeXl5qqNVGX9/f9599112797N7t276d69OwMHDiQ+Pl51tCq1a9cuZs+eTVRUlOooVa5p06acPXu2/BEXF6cujCZu2O+//67pdDqtqKhIdZRqMXXqVC0kJER1jGoxd+5czc3NTXWMm9K2bVttzJgxFZaFh4drzz//vKJE1QfQFi9erDpGtUpLS9MAbcOGDaqjVKs6depoX375peoYVSYnJ0dr2LChtmbNGq1Lly7ahAkTVEeqMq+++qrWvHlz1THKSYvKDcrIyGD+/Pl06NABW1tb1XGqRVZWFh4eHqpjiCsoKipiz5499OrVq8LyXr16sXXrVkWpxM3IysoCsNjfudLSUhYsWEBeXh4xMTGq41SZsWPH0r9/f3r06KE6SrVITEzEz8+PkJAQhg0bxokTJ5RlkULlOj333HM4OTnh6elJcnIyv//+u+pI1eL48eN88sknjBkzRnUUcQUXL16ktLQUHx+fCst9fHw4d+6colSisjRN46mnnqJTp040a9ZMdZwqFRcXh7OzM3q9njFjxrB48WIiIiJUx6oSCxYsYO/evUyZMkV1lGrRrl07vvvuO1atWsWcOXM4d+4cHTp0ID09XUmeWluoXKmz0L8fu3fvLl9/0qRJ7Nu3j9WrV2Ntbc3999+PZsKD+t7o+QGkpqbSp08fBg8ezEMPPaQo+fWrzDlaCp1OV+G5pmmXLROmb9y4cRw4cIAff/xRdZQq17hxY2JjY9m+fTuPPfYYI0eOJCEhQXWsm5aSksKECROYN28e9vb2quNUi759+3L33XcTGRlJjx49WLZsGQDffvutkjw2So5qAsaNG8ewYcOuuU5wcHD5z3Xr1qVu3bo0atSIJk2aEBAQwPbt2022KfNGzy81NZVu3boRExPD7Nmzqzld1bjRc7QEdevWxdra+rLWk7S0tMtaWYRpGz9+PEuWLGHjxo34+/urjlPl7OzsCAsLA6B169bs2rWLjz76iFmzZilOdnP27NlDWloa0dHR5ctKS0vZuHEjM2fOxGAwYG1trTBh1XNyciIyMpLExEQlx6+1hcrfhUdl/N2SYjAYqjJSlbqR8ztz5gzdunUjOjqauXPnYmVlHg1tN/NvaK7s7OyIjo5mzZo13HnnneXL16xZw8CBAxUmE9dL0zTGjx/P4sWLWb9+PSEhIaoj1QhN00z6/8zrdeutt152B8zo0aMJDw/nueees7giBcq+6w4dOsQtt9yi5Pi1tlC5Xjt37mTnzp106tSJOnXqcOLECV555RVCQ0NNtjXlRqSmptK1a1cCAwP54IMPuHDhQvlrKu+br2rJyclkZGSQnJxMaWkpsbGxAISFheHs7Kw23A166qmnGDFiBK1bty5vAUtOTraYfkW5ubkcO3as/HlSUhKxsbF4eHgQGBioMFnVGDt2LD/88AO///47Li4u5a1jbm5uODg4KE5XNV544QX69u1LQEAAOTk5LFiwgPXr17Ny5UrV0W6ai4vLZf2J/u6/aCn9jJ555hkGDBhAYGAgaWlpvPXWW2RnZzNy5Eg1gVTecmQODhw4oHXr1k3z8PDQ9Hq9FhwcrI0ZM0Y7ffq06mhVYu7cuRpwxYclGTly5BXP8a+//lIdrVI+/fRTLSgoSLOzs9NatWplUbe2/vXXX1f8txo5cqTqaFXiar9vc+fOVR2tyjzwwAPln08vLy/t1ltv1VavXq06VrWxtNuThw4dqvn6+mq2traan5+fdtddd2nx8fHK8ug0zYR7hAohhBCiVjOPzghCCCGEqJWkUBFCCCGEyZJCRQghhBAmSwoVIYQQQpgsKVSEEEIIYbKkUBFCCCGEyZJCRQghhBAmSwoVIYQQQpgsKVSEsEBdu3Zl4sSJqmNcUXp6Ot7e3pw8eRKA9evXo9PpuHTpUrUet7LH+eabb3B3d7+hbdq0acOvv/56Q9sIIa5MChUhxH86e/Ys9957L40bN8bKyuqqRdCiRYuIiIhAr9cTERHB4sWLL1tnypQpDBgwwOJmtv6nl19+meeffx6j0ag6ihBmTwoVIcR/MhgMeHl58eKLL9K8efMrrrNt2zaGDh3KiBEj2L9/PyNGjGDIkCHs2LGjfJ2CggK++uorHnrooZqKrkT//v3Jyspi1apVqqMIYfakUBHCwmVmZnL//fdTp04dHB0d6du3L4mJiRXWmTNnDgEBATg6OnLnnXcybdq0Cpc7goOD+eijj7j//vtxc3O74nFmzJhBz549mTx5MuHh4UyePJlbb72VGTNmlK+zYsUKbGxsrjnzeHp6Ovfccw/+/v44OjoSGRnJjz/+WGGdrl27Mn78eCZOnEidOnXw8fFh9uzZ5OXlMXr0aFxcXAgNDWXFihWX7X/Lli00b94ce3t72rVrR1xcXIXXv/nmGwIDA8vfi/T09AqvHz9+nIEDB+Lj44OzszNt2rRh7dq1FdaxtramX79+l+UWQtw4KVSEsHCjRo1i9+7dLFmyhG3btqFpGv369aO4uBgo++IeM2YMEyZMIDY2lp49e/L222/f8HG2bdtGr169Kizr3bs3W7duLX++ceNGWrdufc39FBYWEh0dzdKlSzl48CCPPPIII0aMqNAyA/Dtt99St25ddu7cyfjx43nssccYPHgwHTp0YO/evfTu3ZsRI0aQn59fYbtJkybxwQcfsGvXLry9vbn99tvL34sdO3bwwAMP8PjjjxMbG0u3bt146623Kmyfm5tLv379WLt2Lfv27aN3794MGDCA5OTkCuu1bduWTZs2Xd+bJ4S4OmXzNgshqs3f084fPXpUA7QtW7aUv3bx4kXNwcFB+/nnnzVNK5vSvX///hW2Hz58uObm5nbNff+bra2tNn/+/ArL5s+fr9nZ2ZU/HzhwoPbAAw9UWOevv/7SAC0zM/Oq59OvXz/t6aefrpChU6dO5c9LSko0JycnbcSIEeXLzp49qwHatm3bKhxnwYIF5eukp6drDg4O2k8//aRpmqbdc889Wp8+fSoce+jQoVd9L/4WERGhffLJJxWW/f7775qVlZVWWlp6zW2FENcmLSpCWLBDhw5hY2NDu3btypd5enrSuHFjDh06BMCRI0do27Zthe3+/fx66XS6Cs81TauwrKCgAHt7+2vuo7S0lLfffpuoqCg8PT1xdnZm9erVl7VYREVFlf9sbW2Np6cnkZGR5ct8fHwASEtLq7DdPy87eXh4VHgvDh06dNllqX8/z8vL49lnnyUiIgJ3d3ecnZ05fPjwZfkcHBwwGo0YDIZrnq8Q4tpsVAcQQlQfTdOuuvzvAuLfxcS1truWevXqce7cuQrL0tLSygsGgLp165KZmXnN/Xz44YdMnz6dGTNmEBkZiZOTExMnTqSoqKjCera2thWe63S6Csv+PqfrufPmn+/Ff5k0aRKrVq3igw8+ICwsDAcHBwYNGnRZvoyMDBwdHXFwcPjPfQohrk5aVISwYBEREZSUlFTo35Gens7Ro0dp0qQJAOHh4ezcubPCdrt3777hY8XExLBmzZoKy1avXk2HDh3Kn7ds2ZKEhIRr7mfTpk0MHDiQ++67j+bNm9OgQYPLOv/ejO3bt5f/nJmZydGjRwkPDwfK3q9/vv7v9f/ON2rUKO68804iIyOpV69e+Zgw/3Tw4EFatWpVZbmFqK2kUBHCgjVs2JCBAwfy8MMPs3nzZvbv3899991H/fr1GThwIADjx49n+fLlTJs2jcTERGbNmsWKFSsua2WJjY0lNjaW3NxcLly4QGxsbIWiY8KECaxevZr33nuPw4cP895777F27doKY6707t2b+Pj4a7aqhIWFsWbNGrZu3cqhQ4d49NFHL2upuRlvvPEG69at4+DBg4waNYq6detyxx13APDEE0+wcuVKpk6dytGjR5k5cyYrV668LN+vv/5KbGws+/fv5957771iq82mTZsu61wshLhxUqgIYeHmzp1LdHQ0t912GzExMWiaxvLly8svk3Ts2JEvvviCadOm0bx5c1auXMmTTz55WV+Sli1b0rJlS/bs2cMPP/xAy5Yt6devX/nrHTp0YMGCBcydO5eoqCi++eYbfvrppwr9YyIjI2ndujU///zzVfO+/PLLtGrVit69e9O1a1fq1atXXkhUhXfffZcJEyYQHR3N2bNnWbJkCXZ2dgC0b9+eL7/8kk8++YQWLVqwevVqXnrppQrbT58+nTp16tChQwcGDBhA7969L2s5OXPmDFu3bmX06NFVlluI2kqnVeZitBDCoj388MMcPny4Wm6vXb58Oc888wwHDx7Eysoy/1aaNGkSWVlZzJ49W3UUIcyedKYVQvDBBx/Qs2dPnJycWLFiBd9++y2fffZZtRyrX79+JCYmcubMGQICAqrlGKp5e3vzzDPPqI4hhEWQFhUhBEOGDGH9+vXk5OTQoEEDxo8fz5gxY1THEkIIKVSEEEIIYbos8wKxEEIIISyCFCpCCCGEMFlSqAghhBDCZEmhIoQQQgiTJYWKEEIIIUyWFCpCCCGEMFlSqAghhBDCZEmhIoQQQgiT9X+fZkvEqg05YAAAAABJRU5ErkJggg==", - "text/plain": [ - "
    " - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "execution_count": 16, + "id": "1134c2ed", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", @@ -2661,35 +3020,9 @@ "\n", "plt.show()" ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f03f3f0d", - "metadata": {}, - "outputs": [], - "source": [] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.18" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week38/week38.do.txt b/doc/src/week38/week38.do.txt index 0b176358c..45ec2c7ce 100644 --- a/doc/src/week38/week38.do.txt +++ b/doc/src/week38/week38.do.txt @@ -37,6 +37,7 @@ DATE: September 16-20, 2024 * Resampling techniques, cross-validation examples included here, see also the lectures from last week on the bootstrap method * Exercise for week 38 on the bias-variance tradeoff, see also the video from the lab session from week 37 at URL:"https://youtu.be/omLmp_kkie0" * Work on project 1, in particular resampling methods like cross-validation and bootstrap. + * "Video on cross-validation from exercise session":"https://youtu.be/T9jjWsmsd1o" !eblock