diff --git a/doc/pub/week44/html/._week44-bs001.html b/doc/pub/week44/html/._week44-bs001.html index 91470901e..c0c5388b9 100644 --- a/doc/pub/week44/html/._week44-bs001.html +++ b/doc/pub/week44/html/._week44-bs001.html @@ -340,6 +340,9 @@ MathJax.Hub.Config({
  • Note: Thursday's lecture is digital only due to High-school post-education day
  • Friday: Decision trees and ensemble models (bagging and random forests)
    • +
    diff --git a/doc/pub/week44/html/week44-reveal.html b/doc/pub/week44/html/week44-reveal.html index eb36876f1..c63319700 100644 --- a/doc/pub/week44/html/week44-reveal.html +++ b/doc/pub/week44/html/week44-reveal.html @@ -207,6 +207,11 @@ MathJax.Hub.Config({

  • Friday: Decision trees and ensemble models (bagging and random forests)
    • + +

    diff --git a/doc/pub/week44/html/week44-solarized.html b/doc/pub/week44/html/week44-solarized.html index 121c3d6d6..5d8f605a6 100644 --- a/doc/pub/week44/html/week44-solarized.html +++ b/doc/pub/week44/html/week44-solarized.html @@ -299,6 +299,9 @@ MathJax.Hub.Config({
  • Note: Thursday's lecture is digital only due to High-school post-education day
  • Friday: Decision trees and ensemble models (bagging and random forests)
    • +
    Videos diff --git a/doc/pub/week44/html/week44.html b/doc/pub/week44/html/week44.html index f9ed34672..cc37a5b48 100644 --- a/doc/pub/week44/html/week44.html +++ b/doc/pub/week44/html/week44.html @@ -376,6 +376,9 @@ MathJax.Hub.Config({
  • Note: Thursday's lecture is digital only due to High-school post-education day
  • Friday: Decision trees and ensemble models (bagging and random forests)
    • +
    Videos diff --git a/doc/pub/week44/ipynb/Results/FigureFiles/baggingboot.png b/doc/pub/week44/ipynb/Results/FigureFiles/baggingboot.png index 71702772a..5a9dfa070 100644 Binary files a/doc/pub/week44/ipynb/Results/FigureFiles/baggingboot.png and b/doc/pub/week44/ipynb/Results/FigureFiles/baggingboot.png differ diff --git a/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz b/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz index 9a08c14e8..9b80f40d6 100644 Binary files a/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz and b/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz differ diff --git a/doc/pub/week44/ipynb/week44.ipynb b/doc/pub/week44/ipynb/week44.ipynb index 5f6c07b59..693c4ef6b 100644 --- a/doc/pub/week44/ipynb/week44.ipynb +++ b/doc/pub/week44/ipynb/week44.ipynb @@ -2,8 +2,10 @@ "cells": [ { "cell_type": "markdown", - "id": "1e9d457a", - "metadata": {}, + "id": "c783cfeb", + "metadata": { + "editable": true + }, "source": [ "\n", @@ -12,8 +14,10 @@ }, { "cell_type": "markdown", - "id": "8da83506", - "metadata": {}, + "id": "3896f306", + "metadata": { + "editable": true + }, "source": [ "# Week 44: Decision Trees, Ensemble methods and Random Forests\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", @@ -25,8 +29,10 @@ }, { "cell_type": "markdown", - "id": "0d38a800", - "metadata": {}, + "id": "7f793430", + "metadata": { + "editable": true + }, "source": [ "## Overview of week 44\n", "\n", @@ -38,6 +44,8 @@ "\n", "* Friday: Decision trees and ensemble models (bagging and random forests)\n", "\n", + " * [Video of lecture](https://youtu.be/9QcU8VcXxRU)\n", + "\n", "**Videos.**\n", "\n", "1. [Video on Decision trees](https://www.youtube.com/watch?v=RmajweUFKvM&ab_channel=Simplilearn)\n", @@ -49,8 +57,10 @@ }, { "cell_type": "markdown", - "id": "d31105a9", - "metadata": {}, + "id": "8f731670", + "metadata": { + "editable": true + }, "source": [ "## Digression First\n", "\n", @@ -65,8 +75,10 @@ }, { "cell_type": "markdown", - "id": "07fe25d8", - "metadata": {}, + "id": "5204298c", + "metadata": { + "editable": true + }, "source": [ "## Decision trees, overarching aims\n", "\n", @@ -94,8 +106,10 @@ }, { "cell_type": "markdown", - "id": "2d6686ff", - "metadata": {}, + "id": "13462217", + "metadata": { + "editable": true + }, "source": [ "## Basics of a tree\n", "\n", @@ -112,8 +126,10 @@ }, { "cell_type": "markdown", - "id": "ce62a1a1", - "metadata": {}, + "id": "1dcb8496", + "metadata": { + "editable": true + }, "source": [ "## A Sketch of a Tree, Regression problem\n", "\n", @@ -124,8 +140,10 @@ }, { "cell_type": "markdown", - "id": "cf8c3236", - "metadata": {}, + "id": "65cc5600", + "metadata": { + "editable": true + }, "source": [ "## A Sketch of a Tree, Classification problem\n", "\n", @@ -135,8 +153,10 @@ }, { "cell_type": "markdown", - "id": "d450c9c5", - "metadata": {}, + "id": "503138f3", + "metadata": { + "editable": true + }, "source": [ "## A typical Decision Tree with its pertinent Jargon, Classification Problem\n", "\n", @@ -151,8 +171,10 @@ }, { "cell_type": "markdown", - "id": "6065bf9d", - "metadata": {}, + "id": "931cadfd", + "metadata": { + "editable": true + }, "source": [ "## General Features\n", "\n", @@ -172,8 +194,10 @@ }, { "cell_type": "markdown", - "id": "dab43c19", - "metadata": {}, + "id": "80af3452", + "metadata": { + "editable": true + }, "source": [ "## How do we set it up?\n", "\n", @@ -193,8 +217,10 @@ }, { "cell_type": "markdown", - "id": "f35f5834", - "metadata": {}, + "id": "99d353f2", + "metadata": { + "editable": true + }, "source": [ "## Decision trees and Regression" ] @@ -202,8 +228,11 @@ { "cell_type": "code", "execution_count": 1, - "id": "6caa26f7", - "metadata": {}, + "id": "c42147eb", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "%matplotlib inline\n", @@ -300,8 +329,10 @@ }, { "cell_type": "markdown", - "id": "9eb4d7cf", - "metadata": {}, + "id": "5c54b1f4", + "metadata": { + "editable": true + }, "source": [ "## Building a tree, regression\n", "\n", @@ -320,8 +351,10 @@ }, { "cell_type": "markdown", - "id": "53c984bd", - "metadata": {}, + "id": "a7fe1577", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sum_{j=1}^J\\sum_{i\\in R_j}(y_i-\\overline{y}_{R_j})^2,\n", @@ -330,8 +363,10 @@ }, { "cell_type": "markdown", - "id": "a9906c4c", - "metadata": {}, + "id": "76468ec2", + "metadata": { + "editable": true + }, "source": [ "where $\\overline{y}_{R_j}$ is the mean response for the training observations \n", "within box $j$." @@ -339,8 +374,10 @@ }, { "cell_type": "markdown", - "id": "96557142", - "metadata": {}, + "id": "908a9534", + "metadata": { + "editable": true + }, "source": [ "## A top-down approach, recursive binary splitting\n", "\n", @@ -359,8 +396,10 @@ }, { "cell_type": "markdown", - "id": "f30bcd98", - "metadata": {}, + "id": "b8082569", + "metadata": { + "editable": true + }, "source": [ "## Making a tree\n", "\n", @@ -370,8 +409,10 @@ }, { "cell_type": "markdown", - "id": "ea89bbb5", - "metadata": {}, + "id": "c03ba845", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left\\{X\\vert x_j < s\\right\\},\n", @@ -380,16 +421,20 @@ }, { "cell_type": "markdown", - "id": "1516b6cc", - "metadata": {}, + "id": "923b79d6", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "id": "893e24a3", - "metadata": {}, + "id": "ddb1c706", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left\\{X\\vert x_j \\geq s\\right\\},\n", @@ -398,16 +443,20 @@ }, { "cell_type": "markdown", - "id": "c4b356f7", - "metadata": {}, + "id": "974a1a13", + "metadata": { + "editable": true + }, "source": [ "so that we obtain the lowest MSE, that is" ] }, { "cell_type": "markdown", - "id": "be0d9aca", - "metadata": {}, + "id": "0879e55b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sum_{i:x_i\\in R_j}(y_i-\\overline{y}_{R_1})^2+\\sum_{i:x_i\\in R_2}(y_i-\\overline{y}_{R_2})^2,\n", @@ -416,8 +465,10 @@ }, { "cell_type": "markdown", - "id": "d4f3358d", - "metadata": {}, + "id": "8d20dc23", + "metadata": { + "editable": true + }, "source": [ "which we want to minimize by considering all predictors\n", "$x_1,x_2,\\dots,x_p$. We consider also all possible values of $s$ for\n", @@ -447,8 +498,10 @@ }, { "cell_type": "markdown", - "id": "7a339714", - "metadata": {}, + "id": "b8e00a18", + "metadata": { + "editable": true + }, "source": [ "## Pruning the tree\n", "\n", @@ -469,8 +522,10 @@ }, { "cell_type": "markdown", - "id": "33f066ed", - "metadata": {}, + "id": "de4b7a52", + "metadata": { + "editable": true + }, "source": [ "## Cost complexity pruning\n", "\n", @@ -479,8 +534,10 @@ }, { "cell_type": "markdown", - "id": "753ee3d1", - "metadata": {}, + "id": "9e950ab1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sum_{m=1}^{\\overline{T}}\\sum_{i:x_i\\in R_m}(y_i-\\overline{y}_{R_m})^2+\\alpha\\overline{T},\n", @@ -489,8 +546,10 @@ }, { "cell_type": "markdown", - "id": "55e0c324", - "metadata": {}, + "id": "5eae9abd", + "metadata": { + "editable": true + }, "source": [ "is as small as possible. Here $\\overline{T}$ is \n", "the number of terminal nodes of the tree $T$ , $R_m$ is the\n", @@ -515,8 +574,10 @@ }, { "cell_type": "markdown", - "id": "813adcf0", - "metadata": {}, + "id": "abe1c0ed", + "metadata": { + "editable": true + }, "source": [ "## Schematic Regression Procedure\n", "\n", @@ -539,8 +600,10 @@ }, { "cell_type": "markdown", - "id": "c48b2ef5", - "metadata": {}, + "id": "93731a82", + "metadata": { + "editable": true + }, "source": [ "## A Classification Tree\n", "\n", @@ -560,8 +623,10 @@ }, { "cell_type": "markdown", - "id": "fddbe38b", - "metadata": {}, + "id": "f86f2095", + "metadata": { + "editable": true + }, "source": [ "## Growing a classification tree\n", "\n", @@ -585,8 +650,10 @@ }, { "cell_type": "markdown", - "id": "a3fba89d", - "metadata": {}, + "id": "6e26c38e", + "metadata": { + "editable": true + }, "source": [ "## Classification tree, how to split nodes\n", "\n", @@ -602,8 +669,10 @@ }, { "cell_type": "markdown", - "id": "22166b43", - "metadata": {}, + "id": "72e74778", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p_{mk} = \\frac{1}{N_m}\\sum_{x_i\\in R_m}I(y_i=k).\n", @@ -612,8 +681,10 @@ }, { "cell_type": "markdown", - "id": "c2e4074c", - "metadata": {}, + "id": "ce861605", + "metadata": { + "editable": true + }, "source": [ "We let $p_{mk}$ represent the majority class of observations in region\n", "$m$. The three most common ways of splitting a node are given by\n", @@ -623,8 +694,10 @@ }, { "cell_type": "markdown", - "id": "b8e1e02c", - "metadata": {}, + "id": "470fddfc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p_{mk} = \\frac{1}{N_m}\\sum_{x_i\\in R_m}I(y_i\\ne k) = 1-p_{mk}.\n", @@ -633,16 +706,20 @@ }, { "cell_type": "markdown", - "id": "25506a4a", - "metadata": {}, + "id": "7bf4f22b", + "metadata": { + "editable": true + }, "source": [ "* Gini index $g$" ] }, { "cell_type": "markdown", - "id": "8e859f7d", - "metadata": {}, + "id": "d09f93cf", + "metadata": { + "editable": true + }, "source": [ "$$\n", "g = \\sum_{k=1}^K p_{mk}(1-p_{mk}).\n", @@ -651,16 +728,20 @@ }, { "cell_type": "markdown", - "id": "75118af9", - "metadata": {}, + "id": "47a9f98d", + "metadata": { + "editable": true + }, "source": [ "* Information entropy or just entropy $s$" ] }, { "cell_type": "markdown", - "id": "7ad7a4a9", - "metadata": {}, + "id": "6f55e4f8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "s = -\\sum_{k=1}^K p_{mk}\\log{p_{mk}}.\n", @@ -669,8 +750,10 @@ }, { "cell_type": "markdown", - "id": "80392de1", - "metadata": {}, + "id": "b24a20a2", + "metadata": { + "editable": true + }, "source": [ "## Visualizing the Tree, Classification" ] @@ -678,8 +761,11 @@ { "cell_type": "code", "execution_count": 2, - "id": "448f2c77", - "metadata": {}, + "id": "bc028f62", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import os\n", @@ -719,8 +805,10 @@ }, { "cell_type": "markdown", - "id": "976d6f54", - "metadata": {}, + "id": "55cb02a7", + "metadata": { + "editable": true + }, "source": [ "## Visualizing the Tree, The Moons" ] @@ -728,8 +816,11 @@ { "cell_type": "code", "execution_count": 3, - "id": "245a63bb", - "metadata": {}, + "id": "43c8786b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Common imports\n", @@ -760,8 +851,10 @@ }, { "cell_type": "markdown", - "id": "96f4a124", - "metadata": {}, + "id": "685cb93a", + "metadata": { + "editable": true + }, "source": [ "## Other ways of visualizing the trees\n", "\n", @@ -771,8 +864,11 @@ { "cell_type": "code", "execution_count": 4, - "id": "89e96fb7", - "metadata": {}, + "id": "f86f0669", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.datasets import load_iris\n", @@ -786,8 +882,10 @@ }, { "cell_type": "markdown", - "id": "3f85a09f", - "metadata": {}, + "id": "80b1c9ce", + "metadata": { + "editable": true + }, "source": [ "## Printing out as text\n", "\n", @@ -798,8 +896,11 @@ { "cell_type": "code", "execution_count": 5, - "id": "b6c15b05", - "metadata": {}, + "id": "82a35973", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.datasets import load_iris\n", @@ -814,8 +915,10 @@ }, { "cell_type": "markdown", - "id": "64b8d1dd", - "metadata": {}, + "id": "285c0f97", + "metadata": { + "editable": true + }, "source": [ "## Algorithms for Setting up Decision Trees\n", "\n", @@ -832,8 +935,10 @@ }, { "cell_type": "markdown", - "id": "696574e0", - "metadata": {}, + "id": "2769cab5", + "metadata": { + "editable": true + }, "source": [ "## The CART algorithm for Classification\n", "\n", @@ -847,8 +952,10 @@ }, { "cell_type": "markdown", - "id": "69cfbe7f", - "metadata": {}, + "id": "67932aa2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(k,t_k) = \\frac{m_{\\mathrm{left}}}{m}G_{\\mathrm{left}}+ \\frac{m_{\\mathrm{right}}}{m}G_{\\mathrm{right}},\n", @@ -857,8 +964,10 @@ }, { "cell_type": "markdown", - "id": "68b5217a", - "metadata": {}, + "id": "06167e51", + "metadata": { + "editable": true + }, "source": [ "where $G_{\\mathrm{left/right}}$ measures the impurity of the left/right subset and $m_{\\mathrm{left/right}}$\n", " is the number of instances in the left/right subset\n", @@ -872,8 +981,10 @@ }, { "cell_type": "markdown", - "id": "d7eeda5d", - "metadata": {}, + "id": "20620ae7", + "metadata": { + "editable": true + }, "source": [ "## The CART algorithm for Regression\n", "\n", @@ -883,8 +994,10 @@ }, { "cell_type": "markdown", - "id": "f0978df9", - "metadata": {}, + "id": "b434cc65", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(k,t_k) = \\frac{m_{\\mathrm{left}}}{m}\\mathrm{MSE}_{\\mathrm{left}}+ \\frac{m_{\\mathrm{right}}}{m}\\mathrm{MSE}_{\\mathrm{right}}.\n", @@ -893,16 +1006,20 @@ }, { "cell_type": "markdown", - "id": "40902e6c", - "metadata": {}, + "id": "2b09a9ef", + "metadata": { + "editable": true + }, "source": [ "Here the MSE for a specific node is defined as" ] }, { "cell_type": "markdown", - "id": "091dc8c0", - "metadata": {}, + "id": "bb805fe4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{MSE}_{\\mathrm{node}}=\\frac{1}{m_\\mathrm{node}}\\sum_{i\\in \\mathrm{node}}(\\overline{y}_{\\mathrm{node}}-y_i)^2,\n", @@ -911,16 +1028,20 @@ }, { "cell_type": "markdown", - "id": "49903d3e", - "metadata": {}, + "id": "31ba0b95", + "metadata": { + "editable": true + }, "source": [ "with" ] }, { "cell_type": "markdown", - "id": "b4bff5f6", - "metadata": {}, + "id": "457cdfae", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\overline{y}_{\\mathrm{node}}=\\frac{1}{m_\\mathrm{node}}\\sum_{i\\in \\mathrm{node}}y_i,\n", @@ -929,8 +1050,10 @@ }, { "cell_type": "markdown", - "id": "cd13ac16", - "metadata": {}, + "id": "6fee2e9b", + "metadata": { + "editable": true + }, "source": [ "the mean value of all observations in a specific node.\n", "\n", @@ -940,8 +1063,10 @@ }, { "cell_type": "markdown", - "id": "ec454ba0", - "metadata": {}, + "id": "ffb1412b", + "metadata": { + "editable": true + }, "source": [ "## Why binary splits?\n", "\n", @@ -953,8 +1078,10 @@ }, { "cell_type": "markdown", - "id": "e1419e8a", - "metadata": {}, + "id": "d1ac8567", + "metadata": { + "editable": true + }, "source": [ "## Computing a Tree using the Gini Index\n", "\n", @@ -977,8 +1104,10 @@ }, { "cell_type": "markdown", - "id": "acf7278e", - "metadata": {}, + "id": "982a3f12", + "metadata": { + "editable": true + }, "source": [ "## The Table\n", "\n", @@ -1003,8 +1132,10 @@ }, { "cell_type": "markdown", - "id": "cf09301a", - "metadata": {}, + "id": "ebcecdc0", + "metadata": { + "editable": true + }, "source": [ "## Computing the various Gini Indices\n", "\n", @@ -1018,8 +1149,10 @@ }, { "cell_type": "markdown", - "id": "ac1071fd", - "metadata": {}, + "id": "14ef43ec", + "metadata": { + "editable": true + }, "source": [ "## Computing the various Gini Indices, Hours slept\n", "\n", @@ -1030,8 +1163,10 @@ }, { "cell_type": "markdown", - "id": "ae964258", - "metadata": {}, + "id": "37ea3246", + "metadata": { + "editable": true + }, "source": [ "## Computing the various Gini Indices, Hours studied\n", "\n", @@ -1044,8 +1179,10 @@ }, { "cell_type": "markdown", - "id": "c50f3fa1", - "metadata": {}, + "id": "4b5d78ee", + "metadata": { + "editable": true + }, "source": [ "## A possible code using Scikit-Learn" ] @@ -1053,8 +1190,11 @@ { "cell_type": "code", "execution_count": 6, - "id": "7550c1fe", - "metadata": {}, + "id": "da857ad8", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Common imports\n", @@ -1121,8 +1261,10 @@ }, { "cell_type": "markdown", - "id": "73371791", - "metadata": {}, + "id": "149b6737", + "metadata": { + "editable": true + }, "source": [ "## Further example: Computing the Gini index\n", "\n", @@ -1163,8 +1305,10 @@ }, { "cell_type": "markdown", - "id": "07b54f10", - "metadata": {}, + "id": "9f2c2b49", + "metadata": { + "editable": true + }, "source": [ "## Simple Python Code to read in Data and perform Classification" ] @@ -1172,8 +1316,11 @@ { "cell_type": "code", "execution_count": 7, - "id": "727954e5", - "metadata": {}, + "id": "cf2c645b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Common imports\n", @@ -1247,8 +1394,10 @@ }, { "cell_type": "markdown", - "id": "ca1867f9", - "metadata": {}, + "id": "cf269868", + "metadata": { + "editable": true + }, "source": [ "## Computing the Gini Factor\n", "\n", @@ -1263,8 +1412,11 @@ { "cell_type": "code", "execution_count": 8, - "id": "17d5a20a", - "metadata": {}, + "id": "e01adb33", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Split a dataset based on an attribute and an attribute value\n", @@ -1331,8 +1483,10 @@ }, { "cell_type": "markdown", - "id": "8400e1fe", - "metadata": {}, + "id": "a1de546e", + "metadata": { + "editable": true + }, "source": [ "## Regression trees" ] @@ -1340,8 +1494,11 @@ { "cell_type": "code", "execution_count": 9, - "id": "8f030fe1", - "metadata": {}, + "id": "fcdcb884", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Quadratic training set + noise\n", @@ -1355,8 +1512,11 @@ { "cell_type": "code", "execution_count": 10, - "id": "7de07d7a", - "metadata": {}, + "id": "759e0cf1", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.tree import DecisionTreeRegressor\n", @@ -1367,8 +1527,10 @@ }, { "cell_type": "markdown", - "id": "1d114a7d", - "metadata": {}, + "id": "3ecc42b9", + "metadata": { + "editable": true + }, "source": [ "## Final regressor code" ] @@ -1376,8 +1538,11 @@ { "cell_type": "code", "execution_count": 11, - "id": "0cf732d7", - "metadata": {}, + "id": "60a03548", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.tree import DecisionTreeRegressor\n", @@ -1423,8 +1588,11 @@ { "cell_type": "code", "execution_count": 12, - "id": "7713257a", - "metadata": {}, + "id": "a783779b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "tree_reg1 = DecisionTreeRegressor(random_state=42)\n", @@ -1459,8 +1627,10 @@ }, { "cell_type": "markdown", - "id": "3e0b8e66", - "metadata": {}, + "id": "2745b639", + "metadata": { + "editable": true + }, "source": [ "## Pros and cons of trees, pros\n", "\n", @@ -1481,8 +1651,10 @@ }, { "cell_type": "markdown", - "id": "b73d29d8", - "metadata": {}, + "id": "18106586", + "metadata": { + "editable": true + }, "source": [ "## Disadvantages\n", "\n", @@ -1507,8 +1679,10 @@ }, { "cell_type": "markdown", - "id": "86083d18", - "metadata": {}, + "id": "4193991b", + "metadata": { + "editable": true + }, "source": [ "## Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods\n", "\n", @@ -1536,8 +1710,10 @@ }, { "cell_type": "markdown", - "id": "66d972ae", - "metadata": {}, + "id": "86e2b66a", + "metadata": { + "editable": true + }, "source": [ "## An Overview of Ensemble Methods\n", "\n", @@ -1550,8 +1726,10 @@ }, { "cell_type": "markdown", - "id": "ca6bf352", - "metadata": {}, + "id": "6480a2d8", + "metadata": { + "editable": true + }, "source": [ "## Why Voting?\n", "\n", @@ -1572,8 +1750,10 @@ }, { "cell_type": "markdown", - "id": "94d89cf7", - "metadata": {}, + "id": "8f8e6b9b", + "metadata": { + "editable": true + }, "source": [ "## Tossing coins\n", "\n", @@ -1601,8 +1781,10 @@ }, { "cell_type": "markdown", - "id": "ec65ac92", - "metadata": {}, + "id": "be6951af", + "metadata": { + "editable": true + }, "source": [ "## Standard imports first" ] @@ -1610,8 +1792,11 @@ { "cell_type": "code", "execution_count": 13, - "id": "6904dbea", - "metadata": {}, + "id": "f8aedad8", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Common imports\n", @@ -1655,8 +1840,10 @@ }, { "cell_type": "markdown", - "id": "5ea8eea1", - "metadata": {}, + "id": "3232d714", + "metadata": { + "editable": true + }, "source": [ "## Simple Voting Example, head or tail" ] @@ -1664,8 +1851,11 @@ { "cell_type": "code", "execution_count": 14, - "id": "ed3edd5f", - "metadata": {}, + "id": "7d33cecb", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "\n", @@ -1695,8 +1885,10 @@ }, { "cell_type": "markdown", - "id": "913397e8", - "metadata": {}, + "id": "59f9c81a", + "metadata": { + "editable": true + }, "source": [ "## Using the Voting Classifier\n", "\n", @@ -1706,8 +1898,11 @@ { "cell_type": "code", "execution_count": 15, - "id": "d06e933d", - "metadata": {}, + "id": "08866444", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", @@ -1756,8 +1951,10 @@ }, { "cell_type": "markdown", - "id": "6f392050", - "metadata": {}, + "id": "291eda26", + "metadata": { + "editable": true + }, "source": [ "## Voting and Bagging" ] @@ -1765,8 +1962,11 @@ { "cell_type": "code", "execution_count": 16, - "id": "6f49c5ce", - "metadata": {}, + "id": "a220ad51", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", @@ -1792,8 +1992,11 @@ { "cell_type": "code", "execution_count": 17, - "id": "fafb0ab3", - "metadata": {}, + "id": "475e945d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.metrics import accuracy_score\n", @@ -1807,8 +2010,11 @@ { "cell_type": "code", "execution_count": 18, - "id": "58bfd587", - "metadata": {}, + "id": "a0443168", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "log_clf = LogisticRegression(random_state=42)\n", @@ -1824,8 +2030,11 @@ { "cell_type": "code", "execution_count": 19, - "id": "a5e2060a", - "metadata": {}, + "id": "e96466b9", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.metrics import accuracy_score\n", @@ -1838,8 +2047,10 @@ }, { "cell_type": "markdown", - "id": "bc47caa6", - "metadata": {}, + "id": "95d87928", + "metadata": { + "editable": true + }, "source": [ "## Bagging\n", "\n", @@ -1858,8 +2069,10 @@ }, { "cell_type": "markdown", - "id": "7a7ce88c", - "metadata": {}, + "id": "09eac3f8", + "metadata": { + "editable": true + }, "source": [ "## More bagging\n", "\n", @@ -1888,8 +2101,10 @@ }, { "cell_type": "markdown", - "id": "3a2bfa3b", - "metadata": {}, + "id": "97b706bf", + "metadata": { + "editable": true + }, "source": [ "## Making your own Bootstrap: Changing the Level of the Decision Tree\n", "\n", @@ -1899,173 +2114,13 @@ }, { "cell_type": "code", - "execution_count": 42, - "id": "f5f20a2c", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Polynomial degree: 1\n", - "Error: 0.1077869092797025\n", - "Bias^2: 0.09665219931376287\n", - "Var: 0.01113470996593965\n", - "0.1077869092797025 >= 0.09665219931376287 + 0.01113470996593965 = 0.10778690927970253\n", - "Polynomial degree: 2\n", - "Error: 0.07073328352158705\n", - "Bias^2: 0.05660265596808508\n", - "Var: 0.01413062755350194\n", - "0.07073328352158705 >= 0.05660265596808508 + 0.01413062755350194 = 0.07073328352158702\n", - "Polynomial degree: 3\n", - "Error: 0.03491402051096134\n", - "Bias^2: 0.025645913680458925\n", - "Var: 0.009268106830502446\n", - "0.03491402051096134 >= 0.025645913680458925 + 0.009268106830502446 = 0.034914020510961374\n", - "Polynomial degree: 4\n", - "Error: 0.03122805721680159\n", - "Bias^2: 0.020459888416421317\n", - "Var: 0.01076816880038027\n", - "0.03122805721680159 >= 0.020459888416421317 + 0.01076816880038027 = 0.031228057216801587\n", - "Polynomial degree: 5\n", - "Error: 0.029360432630668315\n", - "Bias^2: 0.01933417827965857\n", - "Var: 0.010026254351009753\n", - "0.029360432630668315 >= 0.01933417827965857 + 0.010026254351009753 = 0.029360432630668322\n", - "Polynomial degree: 6\n", - "Error: 0.030277866384743817\n", - "Bias^2: 0.020595346780014352\n", - "Var: 0.00968251960472946\n", - "0.030277866384743817 >= 0.020595346780014352 + 0.00968251960472946 = 0.030277866384743814\n", - "Polynomial degree: 7\n", - "Error: 0.030892885889348087\n", - "Bias^2: 0.02014767854486778\n", - "Var: 0.010745207344480309\n", - "0.030892885889348087 >= 0.02014767854486778 + 0.010745207344480309 = 0.03089288588934809\n", - "Polynomial degree: 8\n", - "Error: 0.03110939965212513\n", - "Bias^2: 0.020817454714933955\n", - "Var: 0.010291944937191167\n", - "0.03110939965212513 >= 0.020817454714933955 + 0.010291944937191167 = 0.03110939965212512\n", - "Polynomial degree: 9\n", - "Error: 0.03129578341811742\n", - "Bias^2: 0.020886306078419455\n", - "Var: 0.01040947733969797\n", - "0.03129578341811742 >= 0.020886306078419455 + 0.01040947733969797 = 0.03129578341811742\n", - "Polynomial degree: 10\n", - "Error: 0.031606699705781\n", - "Bias^2: 0.021032809061206647\n", - "Var: 0.010573890644574355\n", - "0.031606699705781 >= 0.021032809061206647 + 0.010573890644574355 = 0.031606699705781005\n", - "Polynomial degree: 11\n", - "Error: 0.03148677989704399\n", - "Bias^2: 0.021152336559874207\n", - "Var: 0.010334443337169787\n", - "0.03148677989704399 >= 0.021152336559874207 + 0.010334443337169787 = 0.031486779897043994\n", - "Polynomial degree: 12\n", - "Error: 0.032116346271443774\n", - "Bias^2: 0.021860481703725315\n", - "Var: 0.010255864567718462\n", - "0.032116346271443774 >= 0.021860481703725315 + 0.010255864567718462 = 0.032116346271443774\n", - "Polynomial degree: 13\n", - "Error: 0.03209100097235015\n", - "Bias^2: 0.02177267731801543\n", - "Var: 0.01031832365433474\n", - "0.03209100097235015 >= 0.02177267731801543 + 0.01031832365433474 = 0.03209100097235017\n", - "Polynomial degree: 14\n", - "Error: 0.031537537813106746\n", - "Bias^2: 0.02082938849956035\n", - "Var: 0.010708149313546408\n", - "0.031537537813106746 >= 0.02082938849956035 + 0.010708149313546408 = 0.03153753781310675\n", - "Polynomial degree: 15\n", - "Error: 0.03145124683797013\n", - "Bias^2: 0.021111527370668133\n", - "Var: 0.010339719467301988\n", - "0.03145124683797013 >= 0.021111527370668133 + 0.010339719467301988 = 0.03145124683797012\n", - "Polynomial degree: 16\n", - "Error: 0.03209105344712722\n", - "Bias^2: 0.021338345379319522\n", - "Var: 0.010752708067807702\n", - "0.03209105344712722 >= 0.021338345379319522 + 0.010752708067807702 = 0.032091053447127225\n", - "Polynomial degree: 17\n", - "Error: 0.03196914233431665\n", - "Bias^2: 0.02166629218762961\n", - "Var: 0.010302850146687045\n", - "0.03196914233431665 >= 0.02166629218762961 + 0.010302850146687045 = 0.03196914233431666\n", - "Polynomial degree: 18\n", - "Error: 0.03165248155309402\n", - "Bias^2: 0.02132799291304245\n", - "Var: 0.010324488640051558\n", - "0.03165248155309402 >= 0.02132799291304245 + 0.010324488640051558 = 0.03165248155309401\n", - "Polynomial degree: 19\n", - "Error: 0.03235209768543048\n", - "Bias^2: 0.022074472952423024\n", - "Var: 0.010277624733007463\n", - "0.03235209768543048 >= 0.022074472952423024 + 0.010277624733007463 = 0.032352097685430486\n", - "Polynomial degree: 20\n", - "Error: 0.03174154546980151\n", - "Bias^2: 0.021139865993692355\n", - "Var: 0.010601679476109165\n", - "0.03174154546980151 >= 0.021139865993692355 + 0.010601679476109165 = 0.03174154546980152\n", - "Polynomial degree: 21\n", - "Error: 0.03249366272788966\n", - "Bias^2: 0.022392964161879643\n", - "Var: 0.010100698566010037\n", - "0.03249366272788966 >= 0.022392964161879643 + 0.010100698566010037 = 0.03249366272788968\n", - "Polynomial degree: 22\n", - "Error: 0.03100387126675639\n", - "Bias^2: 0.02122260145527001\n", - "Var: 0.009781269811486361\n", - "0.03100387126675639 >= 0.02122260145527001 + 0.009781269811486361 = 0.03100387126675637\n", - "Polynomial degree: 23\n", - "Error: 0.0320452032176362\n", - "Bias^2: 0.021849700000696155\n", - "Var: 0.010195503216940061\n", - "0.0320452032176362 >= 0.021849700000696155 + 0.010195503216940061 = 0.032045203217636216\n", - "Polynomial degree: 24\n", - "Error: 0.0315431807559056\n", - "Bias^2: 0.021311765078499363\n", - "Var: 0.01023141567740624\n", - "0.0315431807559056 >= 0.021311765078499363 + 0.01023141567740624 = 0.031543180755905606\n", - "Polynomial degree: 25\n", - "Error: 0.03167739763326645\n", - "Bias^2: 0.020906889239215253\n", - "Var: 0.010770508394051193\n", - "0.03167739763326645 >= 0.020906889239215253 + 0.010770508394051193 = 0.03167739763326645\n", - "Polynomial degree: 26\n", - "Error: 0.03091217303915449\n", - "Bias^2: 0.021114234118115666\n", - "Var: 0.009797938921038828\n", - "0.03091217303915449 >= 0.021114234118115666 + 0.009797938921038828 = 0.030912173039154493\n", - "Polynomial degree: 27\n", - "Error: 0.03181795943603612\n", - "Bias^2: 0.0217612448640803\n", - "Var: 0.01005671457195582\n", - "0.03181795943603612 >= 0.0217612448640803 + 0.01005671457195582 = 0.03181795943603612\n", - "Polynomial degree: 28\n", - "Error: 0.031719847034916156\n", - "Bias^2: 0.021338073424295095\n", - "Var: 0.01038177361062107\n", - "0.031719847034916156 >= 0.021338073424295095 + 0.01038177361062107 = 0.03171984703491616\n", - "Polynomial degree: 29\n", - "Error: 0.03138781693308297\n", - "Bias^2: 0.021135473862578174\n", - "Var: 0.010252343070504813\n", - "0.03138781693308297 >= 0.021135473862578174 + 0.010252343070504813 = 0.031387816933082985\n", - "Simple tree: 0.6586458006727302\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAj0AAAGeCAYAAAB/8bDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABedklEQVR4nO3deXwTdcI/8M/kTpomvWihpS1HOeXyxOK60EUEVFbBZRHEXQQvFB9Zn4UVFdDVfcBr1fUBfR6PtY8IeIH+RNCFCniwoKy63AoFSqFI76Rpmnt+f0wyTdq0NKVt2ubz5jWvmcyR+SYdkk++8/3OCKIoiiAiIiLq5hTRLgARERFRR2DoISIiopjA0ENEREQxgaGHiIiIYgJDDxEREcUEhh4iIiKKCQw9REREFBMYeoiIiCgmqKJdgPbm8/lQUlKC+Ph4CIIQ7eIQERFRC4iiiJqaGqSnp0OhaJs6mm4fekpKSpCZmRntYhAREVErFBcXo3fv3m3yXN0+9MTHxwOQ3jSTyRTl0hAREVFLWK1WZGZmyt/jbaHbh57AKS2TycTQQ0RE1MW0ZdMUNmQmIiKimMDQQ0RERDGBoYeIiIhiQrdv00NERLFDFEV4PB54vd5oF4XOQ6lUQqVSdejlZBh6iIioW3C5XDh79izsdnu0i0ItZDAY0KtXL2g0mg7ZH0MPERF1eT6fDydOnIBSqUR6ejo0Gg0vSNuJiaIIl8uFsrIynDhxAgMGDGizCxA2h6GHiIi6PJfLBZ/Ph8zMTBgMhmgXh1pAr9dDrVajqKgILpcLOp2u3ffJhsxERNRtdERtAbWdjv578eggIiKimMDQQ0RERDGBoYeIiIhiAkMPERFRFM2ZMweCIOCee+5ptOzee++FIAiYM2cOAKC0tBR33303srKyoNVq0bNnT0ycOBH//Oc/5W369OkDQRAaDStXruyol9RpsfcWERFRlGVmZmL9+vV4/vnnodfrAQAOhwPr1q1DVlaWvN7NN98Mt9uN/Px89OvXD+fOnUNBQQEqKytDnu/Pf/4z7rzzzpB5bXm38q6KoYeIiLodURRR547OVZn1amXE1wi65JJLcPz4cWzYsAG33norAGDDhg3IzMxEv379AADV1dX46quvsGPHDowdOxYAkJ2djSuuuKLR88XHx6Nnz54X+Eq6n5gJPTUON0ymaJeCiIg6Qp3bi6HLPovKvg/9eSIMmsi/Xm+//Xb8/e9/l0PPG2+8gblz52LHjh0AAKPRCKPRiA8//BBXXnkltFptWxY7JsRMm55TFbwsORERdV633XYbvvrqK5w8eRJFRUX4+uuvMXv2bHm5SqXCm2++ifz8fCQkJOCqq67Cww8/jH379jV6rj/96U9ySAoMgfAUy2KmpudkRS1yo10IIiLqEHq1Eof+PDFq+26NlJQUXH/99cjPz4coirj++uuRkpISss7NN9+M66+/Hl9++SX++c9/4tNPP8XTTz+N1157TW7sDACLFi0KeQwAGRkZrSpXdxIzoaeINT1ERDFDEIRWnWKKtrlz52LBggUAgFWrVoVdR6fTYcKECZgwYQKWLVuGO+64A8uXLw8JOSkpKcjJyemIIncpMXN662RFbbSLQERE1KxJkybB5XLB5XJh4sSW1VQNHToUtbX8jmuJrheDW4k1PURE1NkplUocPnxYng5WUVGB6dOnY+7cuRgxYgTi4+Oxd+9ePP3007jxxhtD1q2pqcHPP/8cMs9gMMAU4z16Yij01EIUxYi7ERIREXWkpoKJ0WjE6NGj8fzzz6OwsBButxuZmZm488478fDDD4esu2zZMixbtixk3t13341XXnml3crdFQiiKIrRLkR7slqtMJvNyFz4Lr574tdIMbKLHxFRd+NwOHDixAn07dsXOp0u2sWhFmru7xb4/rZYLG1WQxUzbXoA4EQ5z3kSERHFqtgKPWUMPURERLEqpkLPcdb0EBERxayYCj0nym3RLgIRERFFSYyFHtb0EBERxaqYCj0nK+zw+rp1ZzUiIiJqQsyEHrVSAZfHh5LqumgXhYiIiKIgZkJPZpIeAE9xERERxaqYCT3ZyQYADD1ERNR5jBs3DgsXLmxyeZ8+ffDCCy90WHm6u5i5DUWf5DjsPFHL0ENERF3Gt99+i7i4uGgXo9uIuKbHZrNh4cKFSE9Ph06nw6hRo7B+/frzbnf69GksXLgQY8eORUJCAgRBwJtvvtnk+tu2bUNubi4MBgNSUlIwZ84clJaWRlpcWXYSa3qIiKhr6dGjBwwGQ7SL0W1EHHqmTZuG/Px8LF++HFu2bMHll1+OmTNnYu3atc1ud+zYMbz99tvQaDS47rrrml13586dmDx5MtLS0vDRRx/hxRdfxLZt2zB+/Hg4nc5IiwxAqukBGHqIiKhz8Xg8WLBgARISEpCcnIxHH30UgdtiNjy99de//hXDhw9HXFwcMjMzce+998Jmq78GXVFREaZMmYLExETExcXhoosuwubNmzv6JXVaEZ3e2rx5M7Zu3Yq1a9di5syZAIC8vDwUFRVh0aJFmDFjBpRKZdhtf/nLX6KsrAwAsHfvXqxbt67J/SxatAgDBw7E+++/D5VKKmLfvn1x1VVX4Y033sD8+fMjKTYAIDtFSsqnq+xwerzQqsKXk4iIugFRBNz26OxbbQAEocWr5+fnY968edizZw/27t2Lu+66C9nZ2bjzzjsbratQKPC3v/0Nffr0wYkTJ3Dvvfdi8eLFWL16NQDgvvvug8vlwhdffIG4uDgcOnQIRqOxzV5aVxdR6Nm4cSOMRiOmT58eMv/222/HrFmzsGfPHowZMybstgpFyyqVzpw5g2+//RYrVqyQAw8AjBkzBgMHDsTGjRtbFXpSjFoYtSrYnB4UV9qRkxof8XMQEVEX4bYD/5UenX0/XAJoWt4OJzMzE88//zwEQcCgQYOwf/9+PP/882FDT3Cj5759++KJJ57A/Pnz5dBz6tQp3HzzzRg+fDgAoF+/fhf2WrqZiE5vHThwAEOGDAkJIwAwYsQIefmFCjxH4Dkb7qe1+xAEAX1TpIPwOG88SkREncSVV14JIahmKDc3F0ePHoXX62207vbt2zFhwgRkZGQgPj4ev/vd71BRUYHaWul77T/+4z/w5JNP4qqrrsLy5cuxb9++DnsdXUFENT0VFRVhU2NSUpK8/EIFniPwnA33c759OJ3OkHY/VqtVnu6bEof9Zyxs10NE1N2pDVKNS7T23Q6Kiopw3XXX4Z577sETTzyBpKQkfPXVV5g3bx7cbjcA4I477sDEiRPxySef4B//+AdWrFiB5557Dvfff3+7lKmribjLutDMecrmlrXVfs63jxUrVuDxxx8PuyxQ08PQQ0TUzQlCRKeYomn37t2NHg8YMKBRG9m9e/fC4/Hgueeek5uMvPvuu42eLzMzE/fccw/uueceLFmyBK+++ipDj19Ep7eSk5PD1rRUVlYCCF87E6nk5GQA4WuNKisrz7uPJUuWwGKxyENxcbG8rF8P/+kthh4iIuokiouL8eCDD+LHH3/EunXr8NJLL+GBBx5otF7//v3h8Xjw0ksv4fjx43jrrbfwyiuvhKyzcOFCfPbZZzhx4gS+++47fP755xgyZEhHvZROL6LQM3z4cBw+fBgejydk/v79+wEAw4YNu+ACBZ4j8JwN93O+fWi1WphMppAhgN3WiYios/nd736Huro6XHHFFbjvvvtw//3346677mq03qhRo/DXv/4VTz31FIYNG4a3334bK1asCFnH6/Xivvvuw5AhQzBp0iQMGjRIbuRMgCAGLgbQAlu2bMF1112H9evXY8aMGfL8yZMnY9++fTh16lSTXdaD7d27F5dffjn+/ve/Y86cOY2Wjx49Gna7HT/88IP8fLt370Zubi5efvll3HPPPS0tMqxWK8xmMywWC0S1HiMf/wcAYP9j1yJep27x8xARUeflcDhw4sQJ9O3bFzqdLtrFoRZq7u8W/P0dXIFxISJq0zN58mRMmDAB8+fPh9VqRU5ODtatW4dPP/0Ua9askQPKvHnzkJ+fj8LCQmRnZ8vbv//++wCA48ePA5DCT+D6Ab/5zW/k9Z566ilMmDAB06dPx7333ovS0lI89NBDGDZsGG6//fZWv1izXo0UowblNheKKuwYlmFu9XMRERFR1xJxQ+YNGzbgkUcewbJly1BZWYnBgwdj3bp1uOWWW+R1vF4vvF4vGlYiNby+z6pVq7Bq1SoACFl33Lhx2Lx5M5YtW4YpU6bAYDDghhtuwDPPPAOtVhtpkUP0TYlDuc2F4+W1DD1EREQxJKLTW11Rw+qxxe//G+/uPY0/XDMQD1wzINrFIyKiNsDTW11TR5/eivjeW11d3xTpdNqJctt51iQiIqLuJAZDD3twERERxaKYCz3B1+rp5mf2iIiIKEjshB5/wMlKMkAQgBqHBxW1rigXioiIiDpK7IQeexUAQKdWIiNBD4CnuIiIiGJJ7ISe6pPypNyuh3dbJyIiihmxE3qqiuTJQOjhPbiIiKg7EgQBH374YbSL0enETuipbhx6TjL0EBFRlEyZMgXXXHNN2GX//Oc/IQgCvvvuu1Y999mzZzF58uQLKV63FDuhJ0xND9v0EBFRtMybNw+ff/45ioqKGi174403MGrUKFxyySURPafLJXXQ6dmz5wXfwaA7ip3QE1TT0y9wgcKKWvh87LZOREQd74YbbkBqairefPPNkPl2ux3vvPMObrrpJsycORO9e/eGwWDA8OHDsW7dupB1x40bhwULFuDBBx9ESkoKJkyYAKDx6a0//elPGDhwIAwGA/r164elS5fC7XbLyx977DGMGjUKb731Fvr06QOz2YxbbrkFNTU18jo+nw9PPfUUcnJyoNVqkZWVhb/85S/y8jNnzmDGjBlITExEcnIybrzxRpw8ebLt3rA2EDuhp+qkPJmRqIdaKcDl8aHEUhe9MhERUbsQRRF2tz0qQ0uvAadSqfC73/0Ob775Zsg27733HlwuF+644w5ceuml2LRpEw4cOIC77roLt912G/bs2RPyPPn5+VCpVPj666/xP//zP2H3FR8fjzfffBOHDh3Ciy++iFdffRXPP/98yDqFhYX48MMPsWnTJmzatAk7d+7EypUr5eVLlizBU089haVLl+LQoUNYu3Yt0tLSAEhBLS8vD0ajEV988QW++uorGI1GTJo0Sa596gxi595bD8XD9PhZQCOd2rrmrztxrNSGt+ZdgasH9IhyKYmI6EI0vIeT3W3H6LWjo1KWPbP2wKA2tGjdI0eOYMiQIfj888+Rl5cHABg7diwyMjKwdu3aRutff/31GDJkCJ599lkAUk2PxWLB999/H7KeIAjYuHEjbrrpprD7feaZZ/DOO+9g7969AKSanmeeeQY///wz4uPjAQCLFy/GF198gd27d6OmpgY9evTAf//3f+OOO+5o9HxvvPEGnn76aRw+fBiCIACQTrUlJCTgww8/xLXXXhu2HB19762I77LepVWdBNIuAiC16zlWasOJ8lqGHiIiiorBgwdjzJgxeOONN5CXl4fCwkJ8+eWX+Mc//gGv14uVK1finXfewZkzZ+B0OuF0OhEXFxfyHJdddtl59/P+++/jhRdewLFjx2Cz2eDxeBoFiT59+siBBwB69eqF0tJSAMDhw4fhdDoxfvz4sM//r3/9C8eOHQvZHpBCTWFhYYvei44QW6Gn8oQcevoFuq3zWj1ERN2OXqXHnll7zr9iO+07EvPmzcOCBQuwatUq/P3vf0d2djbGjx+PZ555Bs8//zxeeOEFDB8+HHFxcVi4cGGj00UNQ1BDu3fvxi233ILHH38cEydOhNlsxvr16/Hcc8+FrKdWq0MeC4IAn88nvSZ986/J5/Ph0ksvxdtvv91oWY8enadiIbZCT9UJeZI9uIiIui9BEFp8iinafvvb3+KBBx7A2rVrkZ+fjzvvvBOCIODLL7/EjTfeiNmzZwOQgsXRo0cxZMiQiJ7/66+/RnZ2Nh555BF5XrgeY80ZMGAA9Ho9CgoKwp7euuSSS/DOO+8gNTW1zU5FtYfYacgMSDU9fgw9RETUGRiNRsyYMQMPP/wwSkpKMGfOHABATk4Otm7dil27duHw4cO4++678fPPP0f8/Dk5OTh16hTWr1+PwsJC/O1vf8PGjRsjeg6dToc//elPWLx4Mf7v//4PhYWF2L17N15//XUAwK233oqUlBTceOON+PLLL3HixAns3LkTDzzwAE6fPh1xmdtLbIWeMDU9p6vscHl80SoRERER5s2bh6qqKlxzzTXIysoCACxduhSXXHIJJk6ciHHjxqFnz55NNkxuzo033og//OEPWLBgAUaNGoVdu3Zh6dKlET/P0qVL8Z//+Z9YtmwZhgwZghkzZshtfgwGA7744gtkZWVh2rRpGDJkCObOnYu6urpOVfMTW723evYDHvgBgNSdcdjyz1Dr8mLbg2ORk2qMbkGJiKjVmusFRJ1XR/feiq2aHksx4PUAkM739u3BU1xERESxInZCj0ID+DxS8PHrG7gyc7ktWqUiIiKiDhI7oSchWxqzBxcREVFMip3Qk+gPPUE9uHitHiIiotgRe6GHNT1EREQxKfZCT1BNTx9/6CmtccLm9ESjVERE1Ia6eYfkbqej/16xE3rkNj0n5VlmvRopRg0A4CRre4iIuqzALRTsdnuUS0KRCPy9Gt4Co73Ezm0oEvtI48oTgCgC/rvA9k2JQ7nNhePltRiWYY5e+YiIqNWUSiUSEhJCLpYXuNs3dT6iKMJut6O0tBQJCQlQKpUdst/YCT3m3gAEwF0L2EqB+DQAUuj59mQVTrAxMxFRl9azZ08AkIMPdX4JCQny360jxE7oUWkBcyZgOSU1ZvaHnj5yY2Zeq4eIqCsTBAG9evVCamoq3G53tItD56FWqzushicgdkIPACT1kUJP5Qkg60oA9d3WT1TwPDARUXegVCo7/MuUuobYacgMAIl9pXFIt3X/VZnLbGz1T0RE1I3FVuhJ8oeeoG7r2ckGCAJgdXhQWeuKUsGIiIiovcVW6AlT06NTK5Fu1gPgRQqJiIi6s9gKPWFqegCgn/9u68cZeoiIiLqt2Ao9gZoeezngrJFn83YURERE3V9shR6dCTAkS9OVYe7BxWv1EBERdVuxFXqAJnpwsaaHiIiou4u90BOmXU+/QLf1ilr4fOy2TkRE1B3FXugJU9OTkaiHWinA5fGhxFIXpYIRERFRe4q90BOmpkepEJCVZAAAnCznlZmJiIi6o9gLPWFqeoCgKzPzHlxERETdUuyFnkBNj+U04Km/AjOv1UNERNS9xV7oMaYBagMg+oDqU/Js9uAiIiLq3mIv9AgCu60TERHFoNgLPUAT3dal0FNcaYfL44tGqYiIiKgdxWboSewjjYNqenrEaxGnUcInAqcq2YOLiIiou4nN0BOmpkcQBPTtwVNcRERE3VVshh52WyciIoo5sRl6AjU9VScBX337HTZmJiIi6r5iM/SYMwFBCXgcgO1neXbfFOmqzAw9RERE3U/Eocdms2HhwoVIT0+HTqfDqFGjsH79+hZtW1paijlz5iAlJQUGgwG5ubkoKChotJ7T6cQzzzyDYcOGIS4uDmlpaZg8eTJ27doVaXHDU6qBhExpujK423rg9BZDDxERUXcTceiZNm0a8vPzsXz5cmzZsgWXX345Zs6cibVr1za7ndPpxPjx41FQUIAXX3wRH330EdLS0jBp0iTs3LkzZN0777wTDz30EG666SZ8/PHHWLVqFcrKyjB27Fh88803kRY5vHDX6kmWTm+dszpR6/S0zX6IiIioU1BFsvLmzZuxdetWrF27FjNnzgQA5OXloaioCIsWLcKMGTOgVCrDbvv666/jwIED2LVrF3Jzc+VtR44cicWLF2PPnj0ApHC0du1azJo1C08++aS8/VVXXYX09HS8/fbbuOKKK1r1YkMk9QWObw+p6TEb1EiO06Ci1oUT5bUYlmG+8P0QERFRpxBRTc/GjRthNBoxffr0kPm33347SkpK5ODS1LaDBg2SAw8AqFQqzJ49G9988w3OnDkjFUihgEKhgNkcGjhMJhMUCgV0Ol0kRW5akz242JiZiIioO4oo9Bw4cABDhgyBShVaQTRixAh5eXPbBtYLt+3BgwcBAGq1Gvfeey/y8/Px4Ycfwmq14uTJk7jzzjthNptx5513RlLkpsnX6jkeMpuhh4iIqHuK6PRWRUUF+vXr12h+UlKSvLy5bQPrnW/b559/HmazGTfffDN8/i7lWVlZ+Pzzz5GTk9NsGZ1OJ5xOp/zYarWGXzGx8QUKAfAChURERN1UxA2ZBUFo1bJItv3LX/6CZ599Fo899hi2b9+Ojz76CIMGDcKECRPw/fffN7uPFStWwGw2y0NmZmb4FQM1PY5qoK5Knh24B9dxhh4iIqJuJaLQk5ycHLY2p7KyEgDC1uREuu3hw4exbNkyPP7441i6dCnGjRuHX//61/jkk0+QkJCABx98sNkyLlmyBBaLRR6Ki4vDr6iJA4xp/kKE6bZeZoMois3ui4iIiLqOiELP8OHDcfjwYXg8od259+/fDwAYNmxYs9sG1mtu23//+98QRRGXX355yHpqtRojR45stt0QAGi1WphMppChSWEaM2cnGyAIgNXhQWWtq9l9ERERUdcRUeiZOnUqbDYbPvjgg5D5+fn5SE9Px+jRo5vd9siRIyE9vDweD9asWYPRo0cjPT0dAOTx7t27Q7Z3Op347rvv0Lt370iK3LwwNx7VqZVIN+sBsF0PERFRdxJRQ+bJkydjwoQJmD9/PqxWK3JycrBu3Tp8+umnWLNmjXyNnnnz5iE/Px+FhYXIzs4GAMydOxerVq3C9OnTsXLlSqSmpmL16tX48ccfsW3bNnkfv/jFL3D55Zfjscceg91uxy9/+UtYLBa89NJLOHHiBN566622e/XNdFs/U12HE+W1uKxP06fsiIiIqOuIKPQAwIYNG/DII49g2bJlqKysxODBg7Fu3Trccsst8jperxderzekTYxWq0VBQQEWL16M+++/H3a7HaNGjcKWLVswduxYeT2FQoGtW7fimWeewXvvvYdnn30WRqMRQ4cOxebNmzF58uQLfMlB5JqekyGz+6bE4atj5azpISIi6kYEsZu31rVarTCbzbBYLI3b9xR/C7x+DWDKAB48JM9+46sT+POmQ5g8rCdenn1pB5eYiIiImv3+bqXYvMt6QKCmx1oCuB3ybF6rh4iIqPuJ7dBjSAY08QBEoLpInt0v6KrMPl+3rggjIiKKGbEdegQBSOojTQf14MpI0EOtFOD0+HDW6gi/LREREXUpsR16gLA9uFRKBbKSDACAE2U8xUVERNQdMPSEuVYPEHRl5nJbR5eIiIiI2gFDT2L4u63368F7cBEREXUnDD1JTV+gEGAPLiIiou6CoSepnzSuKgJ8Xnl2n2Qp9Jxk6CEiIuoWGHpMGYBCDfjcgPWMPDtwequ4qg4ujy9apSMiIqI2wtCjUAKJ0v3Bghszp8ZrYdAo4fWJKK6yR6lwRERE1FYYeoCw3dYFQahv18Nu60RERF0eQw/QTLd1NmYmIiLqLhh6gLA1PUD97SjYbZ2IiKjrY+gBmq7pkW88ygsUEhERdXUMPUBQTc9JQKy/wWj9VZlZ00NERNTVMfQA9b23nFbAXinP7uu/Vs85qxO1Tk80SkZERERthKEHANR6ID5dmg5q12M2qJEcpwHA2h4iIqKujqEn4Dw9uE5WMPQQERF1ZQw9AU3ceLQPr9VDRETULTD0BCT1kca88SgREVG3xNATkBj+9Bav1UNERNQ9MPQEyHdbD3+tnuNlNohB3dmJiIioa2HoCQg0ZLadA1z1tTp9kuMgCIDV4UFlrStKhSMiIqILxdAToE8EdAnSdNVJebZOrUS6WQ+Ap7iIiIi6MoaeYE10W++fKl2ZubCUt6MgIiLqqhh6gvHGo0RERN0WQ0+wpmp6ghozExERUdfE0BOsqZqeHtLpreO8QCEREVGXxdATrImann7+mp6iSjtcHl9Hl4qIiIjaAENPsEBNj6UY8NbfVb2nSQeDRgmvT8SpSnuUCkdEREQXgqEnWHwvQKkFfB4p+PgJgiDX9rBdDxERUdfE0BNMoQAS+0jTjXpw+dv1sAcXERFRl8TQ01BS+Luts6aHiIioa2PoaaipG4/6e3AVsgcXERFRl8TQ01CgpifoVhQAr9VDRETU1TH0NBS423qDmp6+/qsyV9ndqOKNR4mIiLochp6GEoNqekRRnm3QqJBu1gEAjpeztoeIiKirYehpKCELEBSAuxawlYYsYrseIiKirouhpyGVBjD1lqarwt+Dq5DteoiIiLochp5wkvpI4yZ6cPEeXERERF0PQ084Td54lD24iIiIuiqGnnCavPGoVNNzqtIOt5c3HiUiIupKGHrCaaKmp5dJB51aAbdXRDFvPEpERNSlMPSE00RNj0Ih1N+Di+16iIiIuhSGnnACNT32csBZE7JIbtfDa/UQERF1KQw94ehMgCFZmmYPLiIiom6BoacpieHvtl5/Dy6GHiIioq6EoacpSU10W08JXJWZp7eIiIi6EoaepiQ21W1dqumpqHXBYnd3dKmIiIiolRh6mhK423qDmp44rQo9TdKNRwvZmJmIiKjLiDj02Gw2LFy4EOnp6dDpdBg1ahTWr1/fom1LS0sxZ84cpKSkwGAwIDc3FwUFBWHXra2txbJlyzBw4EBotVokJycjLy8PR48ejbTIrSN3Wz/ZaFE/tushIiLqclSRbjBt2jR8++23WLlyJQYOHIi1a9di5syZ8Pl8mDVrVpPbOZ1OjB8/HtXV1XjxxReRmpqKVatWYdKkSdi2bRvGjh0rr2uz2ZCXl4eSkhI89NBDGDFiBCwWC3bt2gW7vYMuChg4vWU9DXhc0o1I/fr1iMOuwgq26yEiIupCIgo9mzdvxtatW+WgAwB5eXkoKirCokWLMGPGDCiVyrDbvv766zhw4AB27dqF3NxceduRI0di8eLF2LNnj7zuo48+isOHD2Pfvn3o16+fPP/Xv/51xC+w1YypgDoOcNcC1aeAlBx5Uf0FChl6iIiIuoqITm9t3LgRRqMR06dPD5l/++23o6SkJCS4hNt20KBBcuABAJVKhdmzZ+Obb77BmTNnAAB2ux2vvfYapk+fHhJ4OpwgAIl9pOkG7Xr6p/JaPURERF1NRKHnwIEDGDJkCFSq0AqiESNGyMub2zawXrhtDx48CAD417/+hdraWgwYMADz589HYmIiNBoNLrvsMnzyySfnLaPT6YTVag0ZWq2pG4+mSG16iirs8PrE1j8/ERERdZiIQk9FRQWSkpIazQ/Mq6iouOBtAzU+Tz31FPbv34//+7//w8aNG2EymTBlyhR89tlnzZZxxYoVMJvN8pCZmdmyFxdOEzU9GQl6aFUKuLw+nK7ijUeJiIi6goh7bwmC0KplLd3W5/MBADQaDbZs2YIpU6bg+uuvx6ZNm9CrVy888cQTze5jyZIlsFgs8lBcXNzs+s1q5sajff21PWzMTERE1DVEFHqSk5PD1uZUVlYCQNianEi3TU6W7nk1ZswYxMfHy+sZDAaMHTsW3333XbNl1Gq1MJlMIUOrJYa/KjPAbutERERdTUShZ/jw4Th8+DA8Hk/I/P379wMAhg0b1uy2gfWa2zZcu58AURShUHTg9RTlW1GcBPw1UAH9ewRuR8HQQ0RE1BVElCCmTp0Km82GDz74IGR+fn4+0tPTMXr06Ga3PXLkSEgPL4/HgzVr1mD06NFIT08HAPTq1Qu5ubn4+uuvQxoh2+127Ny5E1deeWUkRb4w5kxAUAIeB1BzNmRRfU0PT28RERF1BRGFnsmTJ2PChAmYP38+Xn31VWzfvh133XUXPv30Uzz99NPyNXrmzZsHlUqFoqIiedu5c+fioosuwvTp07F27Vps27YNv/3tb/Hjjz/iqaeeCtnPs88+i5qaGkycOBEffvghPvroI0yaNAnl5eXnbdPTppRqIMHfELqJG48eL2dNDxERUVcQ8bmiDRs24LbbbsOyZcswadIk7NmzB+vWrcOtt94qr+P1euH1eiGK9d25tVotCgoKkJeXh/vvvx9TpkzB2bNnsWXLlpCrMQNSe56CggJotVrceuutmDVrFtRqNXbs2BFynZ8OcZ4bj5bVOGF18MajREREnZ0gBieTbshqtcJsNsNisbSuUfOmPwB73wCu/k9g/LKQRVf8ZRtKa5z48L6rMCozoW0KTERERBf+/R0G77J+PoG7rVc214OL7XqIiIg6O4ae82m22zpvR0FERNRVMPScTxMXKATqb0fBCxQSERF1fgw955PYF4AAOKoBW2nIov6s6SEiIuoyGHrOR2Oob9dzLvSGqoHQc6KiljceJSIi6uQYeloi7SJpfO5QyOyMRD00KgVcHh9KquuiUDAiIiJqKYaelgiEntLQ0KNUCOiTbADAdj1ERESdHUNPS6QOlcYNTm8B9Vdm5j24iIiIOjeGnpYI1PSU/Qh4Q2+2ymv1EBERdQ0MPS2R2BdQG6Qbj1YeD1nEHlxERERdA0NPSygUQOoQabr0YMgiuaannDU9REREnRlDT0vJ7XpCGzMHrsp8zuqEzelpuBURERF1Egw9LSV3Ww+t6THr1UgxagCwXQ8REVFnxtDTUnK39YONFgV6cLFdDxERUefF0NNSqf7QU3UScIbW6PRPZQ8uIiKizo6hp6XikgFjmjRdejhkkXytnnLW9BAREXVWDD2RaOIUV6AHV2Epa3qIiIg6K4aeSMg9uBqGHqmm52RFLXy88SgREVGnxNATibRh0rhBt/XMRD3USgEOtw8lFt54lIiIqDNi6IlEmr+mp/QgINbX6KiUCmQnBxozs10PERFRZ8TQE4mUQYCgBOqqgJqzIYv6pbAHFxERUWfG0BMJtQ5IzpGmm7gyM++2TkRE1Dkx9EQq+BRXEN6Di4iIqHNj6IlUavjbUfBu60RERJ0bQ0+k5HtwhZ7e6u+v6TlrcaCWNx4lIiLqdBh6IhU4vVV2BPC65dkJBg2S4qQbj57glZmJiIg6HYaeSJmzAI0R8LmBimMhiwI9uArZg4uIiKjTYeiJlELRzJWZea0eIiKizoqhpzXSwoceuTEzT28RERF1Ogw9rRG4HUVp+Gv18AKFREREnQ9DT2vIp7cahp7601u88SgREVHnwtDTGoHTW5ZTgMMiz85KMkClEFDn9uJnqyNKhSMiIqJwGHpaQ58ImDKk6dLD8my1UoGsJAMANmYmIiLqbBh6WqvJHlyBxsxs10NERNSZMPS0VpM9uNhtnYiIqDNi6GmtJntw8QKFREREnRFDT2sF9+AS63tq9eONR4mIiDolhp7WShkIKFSA0wJYTsuzA7eiOFNdhzqXN1qlIyIiogYYelpLpZGCDxByiispToMEgxoAbzxKRETUmTD0XAj5FNcBeZYgCLzxKBERUSfE0HMh0i6Sxo2uzMx2PURERJ0NQ8+FCISeJnpw8Vo9REREnQdDz4UInN4q/wnwuOTZ/VJY00NERNTZMPRcCHNvQGsGfB4p+PjlpAYuUGiDKPLGo0RERJ0BQ8+FEISwV2bOSoqDUiGg1uXFOaszSoUjIiKiYAw9F0pu11MfejQqBTIT9QCk2h4iIiKKPoaeCxV8ZeYggR5chbxWDxERUafA0HOh5G7rTd14lDU9REREnQFDz4VKHSKNa0qAuip5tlzTwx5cREREnULEocdms2HhwoVIT0+HTqfDqFGjsH79+hZtW1paijlz5iAlJQUGgwG5ubkoKChodpu6ujoMHDgQgiDg2WefjbS47U9nBsxZ0nTQKa7AVZlZ00NERNQ5RBx6pk2bhvz8fCxfvhxbtmzB5ZdfjpkzZ2Lt2rXNbud0OjF+/HgUFBTgxRdfxEcffYS0tDRMmjQJO3fubHK7pUuXora2k9eWhOnBFajpOVNdB4ebNx4lIiKKNlUkK2/evBlbt27F2rVrMXPmTABAXl4eioqKsGjRIsyYMQNKpTLstq+//joOHDiAXbt2ITc3V9525MiRWLx4Mfbs2dNom2+++QYvvfQS3n77bUyfPj3S19Zx0i4Cfvo0pAdXilGDeJ0KNQ4PTlbUYnBPUxQLSERERBHV9GzcuBFGo7FRALn99ttRUlISNrgEbzto0CA58ACASqXC7Nmz8c033+DMmTMh67tcLsydOxf33XcfLrvsskiK2fFSG9f0CIKA/rwHFxERUacRUeg5cOAAhgwZApUqtIJoxIgR8vLmtg2sF27bgwdDez/9+c9/Rm1tLZ544olIihgd8rV6DgM+nzw7cA+uwlK26yEiIoq2iE5vVVRUoF+/fo3mJyUlycub2zaw3vm2/eGHH/D000/j448/RlxcHMrKylpcRqfTCaez/irIVqu1xdu2WnIOoNQALhtgOQUk9gGA+poeXquHiIgo6iJuyCwIQquWtXRbj8eDuXPnYsaMGZg4cWKkxcOKFStgNpvlITMzM+LniJhSDaQMkqaDGzOzBxcREVGnEVHoSU5ODlubU1lZCQBha3Ii3faFF17A8ePHsXz5clRXV6O6ulqurXE4HKiurobX23RvqCVLlsBischDcXFxy1/ghZAvUhjUbT2oTQ9vPEpERBRdEYWe4cOH4/Dhw/B4PCHz9+/fDwAYNmxYs9sG1mtu2wMHDsBisWDAgAFITExEYmIiRo4cCUDqvp6YmBj2eQK0Wi1MJlPI0CEC3daDenBlJxugEIAapwdlNt54lIiIKJoiCj1Tp06FzWbDBx98EDI/Pz8f6enpGD16dLPbHjlyJKSHl8fjwZo1azB69Gikp6cDAB566CFs3749ZFi3bh0A4J577sH27duRk5MTSbE7Rmrj21Ho1Er0TjQAAApL2a6HiIgomiJqyDx58mRMmDAB8+fPh9VqRU5ODtatW4dPP/0Ua9aska/RM2/ePOTn56OwsBDZ2dkAgLlz52LVqlWYPn06Vq5cidTUVKxevRo//vgjtm3bJu9j8ODBGDx4cMh+T548CQDo378/xo0bdwEvtx0FTm9VFAJuB6DWAZB6cJ2qtON4uQ25/ZOjWEAiIqLYFnFD5g0bNuC2227DsmXLMGnSJOzZswfr1q3DrbfeKq/j9Xrh9XpD2rFotVoUFBQgLy8P999/P6ZMmYKzZ89iy5YtGDt2bNu8mmiK7wnoEwHRC5T/KM/ul8Jr9RAREXUGgtjNW9harVaYzWZYLJb2b9/z9+uBoq+Am14GRs0CAKzZXYRHPzyAvEE98Pfbr2jf/RMREXUT7fH9zbust6W0xu16+vNu60RERJ0CQ09bCnPj0f7+qzKfrrLD6eGNR4mIiKKFoactpfm77JfWX6unR7wWRq0KPhEoqrBHqWBERETE0NOWevh7ndnOAbXlAKQrTQfuwcUrMxMREUUPQ09b0hrl+26Fux0F2/UQERFFD0NPWwtziqu+MTNreoiIiKKFoaetpTZuzBx8Dy4iIiKKDoaethamB1dwm55uflkkIiKiTouhp60FTm+VHQF8Uhf1vilxUCoEWB0eLHznB1TVuqJYQCIiotjE0NPWkvoBKh3gtgNVJwFINx5dMnkwFALw0Q8luPaFL7D10LnolpOIiCjGMPS0NYUS6DFImg46xXXH1f3wwfwx6N8jDmU1Ttz5f3vx4Ls/wGJ3R6mgREREsYWhpz2E6cEFABdnJeKT/7gad/2yHwQB2PDdGVz7wk5sP1IahUISERHFFoae9iD34DrQaJFOrcTD1w3B+/fkom9KHM5Znbj9zW+x+P1/w+pgrQ8REVF7YehpD/KNRw81ucql2UnY/B9XY94v+kIQgHf3nsbE57/AFz+VdVAhiYiIYgtDT3sIhJ7K44Cr6ftt6TVKLL1hKN65KxfZyQactTjwuze+wZIN+2FzejqosERERLFBELv5hWOsVivMZjMsFgtMJlPH7fjp/oC9HLjzcyDj0vOubnd58PSnP+LNXScBABkJejz9mxG4KielnQtKRJ2Z0+NFSbUDxZV2nLXUwahVo6dZh15mHVLjtVAp2++3q9cnoqS6Dqcq7SiqsKOoshanKuw4VWlHgkGNX+T0wNUDUjC0lwkKhdBu5aDY1B7f3ww97SX/18CJncCv/xu45LYWb/bPwgosev/fOF1VBwC47cpsPDR5MOK0qvYqKXVRVocbJdV1OFvtQImlTp6udXkgQIAgQBrkaQECAvP8j4OXo359vUaJNJMOaSatPO4Rr4NJp4IgdN4vN7vLg5Jqh/ReWOpwptqBcpsTKoUAlUIBtUqARqkImVYrFVApBaiVCmlZ0HRgmUalgFGrQrxOhXidGga1ss2+5D1eH85aHCiusuN0VR1OV9pRXFWH01V2FFfW4VyNA019SisEIDVeh14JUgjqZdbL455mHdITdOhhbD4YOdze+lBTUStPn6q043SVHW7v+b8ikuI0uConBVcPkIZeZn1r344W8Xh9OF5ei4MlFljsbiQbtegRr0WKUYseRi1M+s59nFLLMPS0QtRCz6dLgN2rgdHzgckrI9q01unBii2HsWb3KQBAZpIez/xmJK7sl9weJaUL5Pb6YHN4IAJQ+78w1UoFlBfwpehwe3HW4sDZ6jqUWOq/xOu/0B1ROQWqUyukEBSvQ6pJi9T4+mAU/NiobfsvHa9PRFmNE2eq60Lej8Djkuo6VHXQJSAEATBqVTDp1IjXqUICUbxOBaOufpm0XA2DRonSGgeKK+sDTXGVHWctDnh9zX8M69VK9E7UIz1BLwe7c1YHPOfZDmgcjHqa9KhxuOWam3NWZ7Pba5QK9E7So09yHLKSDMhONiAz0YAz1XX48mgZ/llYgVqXN2SbnFQjfpGTgl8OTMHovskX9KPN7vLg8NkaHDprxaESCw6VWHHk5xo4Pb5my5xi1NQHIf9YmqerXxavRXyYY1UURbi9ItxeH1wenzT2+uD2iqGPPf55Xi8EQUBmoh69Ew3QqZWtfr2REEXp/8RP52w4WlqDo6U2nKqwQ4QIpUIBtUKAUiHI4V2pEKBWKKBUClArBKiUCukHgVL6UaDyz1MrBcT5j+nAcWzSq+XHHfX6GHpaIWqh57u3gP+3AOj7S+D3H7fqKb46Wo4/fbAPZ6qlWp85Y/pg8aRBMGhY63Oh3F4fnB4fHG4vHG4vap1e1DjcqHF6UOPwwObwoMbhhs3/uKbBY2nsRo3D0+SHr0IAVA1qD9QKAWqVvwZBIdUgqBT1QclSJ9XeVLTwqt0JBjV6mfXISPD/yk/QIV6nBkQRIgBRlD4YRQA+/zQC8yFCFP3z/dPSMhE2pxelNQ6UWp04Z5W+YK2Olocsg0aJRIMGav8HrUohhUCV/LjB/MBjpQCl/8NXqRDg8frk0PezpWVf8katCukJOqQnSCGhh1ErfYn5RP+XlC902v/l1ty00+NDrf9v35IyREqjVCAjUY/e/i/NzCQ9MhMN6J2oR2aSAclxmkZfzD6fiHKbUwrHljr/2CGH5bOWlgejeJ0K2ckGZCfFISvZgOwkgzROjkNPk67ZAO/2+vD9qWp8dbQMXxwtx77T1QjepVop4JKsRH8tUA8MyzA3+XwVNicOllhx6KwVB0usOFhiwYny2rA1XXEaJYamm5Bi1KLC5kK5zYmyGidqIvwxoFEpYNKp4fHVB5yW1G41RRCAXiYdspPj0CdFeg+zk/zjZEOrAqAoivjZ6sDRczYcLbXh6LkaeRzJ/8u2olEqYNJLId/kD/smvQrx2tCAlJ6gx8SLerZ6Pww9rRC10HPmO+DVPMCQDCwqlP4ntEKNw43/2nwE676Ran0yEvSYenEGpoxMx6Ce8W1Z4qgSRREurw8Olw91bq80uLyoc3tQFzTP4fKGLA+EFofbB4fHC6d/HJjn9Pjg9K8jhxyP77y/rDsDvVqJXgk6ZCTUn7LISND7f7HrkZ6g69AA7HB7pRBU4/AHISdKrQ6U1tQHo1Jr5F86kVAqBPQ0Se9JINj0SpBCXyDkmHTqdtu/KIpwuH2ocbpDw7B/2uoPwsGhuMbhQY3Tg1qnBylGjT/M+INNklRrkhqvbZc2MV6fiAqbEyUWB37214ydszoQp5VCTlaSAX2S45BgULdZzZzF7sauwnJ8eawcX/xUJp+qD0gwqHFV/xT8YkAKEvTqoJBjabLWqUe8FhelmzC0lwkXpZsxNN2E7CRD2PfM4fairMaJcpsT5TaXPF0/LzDtanFtqUKQwlHwaU/psfSDRatSwOUVUVxpP+9zphi16JNcH4Kyk6W/QXayAWa9GiUWB346V4NjQbU3x87Zmvx/pRCAPslxyEk1YkCaEf1SjFApBXi8Ijw+KcB5fVKA9/pEeIKmpWXSOh5f/Ty3Vwr5VocH1jq3fGzbnJ4mT7WGM6K3Gf9vwS9avkEDDD2tELXQ47ID/5UOQAT+8ycgPu2Cnm7nT2V46IN9OGtxyPMGpBpxw4h03DCyF/r77+Temfh8IsprnThb7Qj5NRr41V5a44Td5fGHGy+ilUM0KgXitdIpifpTFWrE+6t3jf5fMvWnMBo81qph1KkgAHD7P0DcHp887fHXFrg8gQ+h0OngGoV4rVoOOmZ9230RdSS7y4NSqxNVdhd8ogiP/0PX4wse++ofe5uY7xOhEOAPeFLISY1vvtaBOhdRFFFUYceXx8rx5U/SqbDzheK+KXEYKgccE4amm5Aar2uX8tW5vCi3OVHj8ECjEoLCjEIOM5GcqhZFEZW1Lpz0t486WWHHKf+4qKL2vKdf1UqhyVompUJAn2QDBqbFY0CqETn+cd+UuA473eTzibC5/AE/EIbq3KhxumGtk4K+1f9DwFrnQWaSAQ9NHtzq/TH0tELUQg8AvHQpUHEMuG0j0P9XF/x0dpcHWw+dw6Z9Z7HzxzK4vPWnVYb0MuGGEb0wZUQ6spINF7yv8xFFERW1LvxsqW9jUmKRwkygYe05q6NV1cQqhdSQVq9WymOdOsxjjQI6lTRPp1ZCq1JAq1ZCFzTWqeuXSdMKaFXSWKdWQqNUsNcJUQfxeH349+lqfHm0HF8fK4fD7cPQXlKwuSjdhMG9TDB2404bljo3TlXYcbKiFkUVtf7G49Lj0hqplkutFNA3JQ4DUuPl2puBafHokxwHjSq2rjLD0NMKUQ0979wGHP5/wLV/AcYsaNOnttS5/QGoBF8dLQ85bz+ytxk3jEjH9SN6IT3hwnpReLw+FFXacfRcjb+xnHQe+Xh5LVzNNCQMCDSiDPQkCe1dokWcViWFGbUSOn+gUbdjF1wios7I7vKgwuZCT7OOn4F+DD2tENXQs2MlsGMFMHIWMPXldttNVa0Lnx38GR/vK8E/CytCThNdmp2IG0b0wvXDeyHV1HQVscfrw6lKuxRsztXgp0C4KasNqVEKJgjS+en0Bl1kA+1Nepr1SI3X8j8wERFFjKGnFaIaeg5/DLwzG+g1Erj7iw7ZZVmNE58eOIuP953Ftycr5UZnggBc0ScJN4xMx+V9ElFUUV9789N5am70aiUGpBmRkypVsw5MMyKnRzx6mnUxV91KREQdg6GnFaIaeioKgZcuAZRa4OESQNmx56p/tjiwef9ZbNpXgu9OVZ93fZ1agQGp8RiQZsSAVCncDEyLR0aCnu1eiIioQ7XH93f3bTHWGST2BdQGwG2X7sPVY2CH7r6nWYe5v+iLub/oi9NVdnyy7yw27TuLE+W16JNiwMDUeOSkGTEwNR4D0+LRO5HhhoiIui+GnvakUACpQ4Az/wJKD3Z46AnWO9GAu8f2x91j+0etDERERNHEBhntLXWoND53MLrlICIiinEMPe0t7SJpfO5QdMtBREQU4xh62lsg9JSypoeIiCiaGHraW6o/9FSdBJw1US0KERFRLGPoaW9xyYDRf5fZ0iPRLQsREVEMY+jpCGn+xsw8xUVERBQ1DD0dgT24iIiIoo6hpyOkDZPG7MFFREQUNQw9HSFweuvcAaB73/WDiIio02Lo6Qgpg6T7bzmqgW3LGXyIiIiigKGnI6h1wLVPStNfvwh88iDgC39HcyIiImofDD0dZfRdwJQXAQjA3jeAD+8BvJ5ol4qIiChmMPR0pEvnADe/BihUwL53gPd+D3ic0S4VERFRTGDo6WjDfwPMWCO18TmyCVg7A3DVRrtURERE3R5DTzQMmgzc+h6gjgOObwfemgbUVUe7VERERN0aQ0+09BsL/O4jQGcGincD+VOA2vJol4qIiKjbYuiJpszLgTmfAHE9gJ/3AX+fDFhLol0qIiKibomhJ9p6Dgdu3wKYMoDyn4A3JgGVJ6JdKiIiom6HoaczSBkAzP0USOwLVBdJwYd3ZCeizqi2HPjpM6BwO9siUpejinYByC8hSwo+b00FSg8Bb14HzN4ApI+KdsmIKJpEEagtk2qALcXS6fDUoYCxR/vv2+eTaqCL99QPFcdC10kZCGRcWj+kDQNUmvYvG1ErCKLYve+JYLVaYTabYbFYYDKZol2c87NXAmtuBkq+A7QmYNa7QHZutEtFnZHHJX0JVhcBVUX1Y7dduhaUQukfq5p5rA6/XJ8A9BgC9BgEaI3RfqXRI4rSJSVctYAmThoEoe3343VLf8vKE0DVCf/4ZP3gsjXexpAshZ8eg4HUIdJ06mBAn9j6crhqgTPfBYWcb6Tb5zSUMlAqc1WYU/FKLdBrBJBxmRSCel8q1WJf6PvmqgUsZwDracByun7a4wSMadIQ37N+HN9T+gxtj79XZ+Os8b8npwHbOUBjlI4PQ5I01id1ySDaHt/fEYcem82GRx99FO+++y4qKysxePBgPPTQQ7jlllvOu21paSkWL16MTZs2wW63Y+TIkXjyyScxfvx4eR2r1YqXXnoJW7duxZEjR2Cz2dC3b1/Mnj0bDzzwAHQ6XUQvsMuFHgBwWIF1twBFXwMqPXDL20DO+PNvR92LzwfUnG0cagLjmhJA7IDbmSRk+79Qh9SPUwYAKm377/tC+HxSAHTVAk6r9P/KaZHGDkv9vEbTwevUAKK3/jkFpdTjUmfyj/2D1hz6ONw6aoP09wwJNv6x5XTofhoRAHNvabCd87f7a+KjO76XPwj5Q1DqUH94jW+8ruVMaC3Oz/sBX4Mrxav0QO/LgMwrgMwrpWlDkrSstgI48y//sFca11U13o8+yR+ALquvEQo8ByAF+JoSqTyW0/5gcwawnqn/Mg8Xvs5HpQfi0wBjz/Dj+F7StCGp84Yjn1f6m1tOA9Wn6t8PeTglHavno4mvD0HBgSg4GAUv08RJ758yeieEOkXoufbaa/Htt99i5cqVGDhwINauXYvXXnsNb7/9NmbNmtXkdk6nE5dddhmqq6uxcuVKpKamYtWqVfjkk0+wbds2jB07FgBw4MAB5OXl4bbbbsO4ceNgNBrx5ZdfYuXKlbjqqquwdetWCBEcnF0y9ACAyw68+zvg2FZAqQF+8wYwZEq0S0WA9Ovf55F+YXpd0uBxSr98fW7/PLd/cPnnBT0OWc9Tv47HJX24BUKNpVha1hyVHkjMloJJYKwzS+XzeaQPTHk6+LG76eVet1SO0sNAbWn4/QpKIDknNAilDgWS+ko1RpHwOKVw4ayRajScNv+4RgosgeDirpOm3Xbp/4fbP89lr5/vrqvfxuOIrBzRptIBiX2kWpGkvqHTCVmhIdNll047lR4Gyg5L49LD0jHTFHOWFIJSBkrhq/ib8OvHpwNZo4FM/9BzOKBUt+w1iCJQebw+CJ3eK/VMDXccJ/aVaqWsZwBbKZoMccG0JqnThznDP86U3hfbOWmo+VkabOekINtSglIqiyHJ/+UfGCc2eOwfB9ZtafD3ef3Hb139sRk4XgODxyGVORD8LMXSYC1pHETD0SVI70d8mnR82Cukoa7ywn4cKdSAWi8NKl3QtL5+uqllpgxgxPRW7zrqoWfz5s24/vrrsXbtWsycOVOef+211+LgwYM4deoUlMrwH3irV6/Gfffdh127diE3Vzpd4/F4MHLkSBiNRuzZswcAUFsrXZ04Li4uZPtnn30WixYtwpdffolf/OIXLX6BXTb0ANKX4IY7gEMfSf8pb1oNjDx/jVq3JorSB6SlWPrVYy0BvE7pV73orf/iFr3+sa/xvJDl/nFwcPE66wNNSLBx1S9ryQd0WxCU0q/7xGzpSzAheJwtte9oz1+oteX1X6ilh+qnnU38slTppC/V1KGAKV36YHfapA/zkEBjA1w10tjnbr/yAwAE6csyUPPSomlz6Hy1XvpiclhCB6dVqoGQ51mbWMcifanpk/yBpm/9OLGPNG3sCSgusG+JwwqU/Sj9rcqO+P9mRwDbz028NQop1GQGhRxz77Y9pjxO4NwB4HRQbVDDdkGAdFpMDjP+Wq3AdCDo6Mwt36/LLr3umnMNxkHBqOZnKRS0ljrOH4QSpZo0jwNwBwUajz/QnO/Hy/kISuk9SMisf2/MvaUga+4tvTfhavIA6bPRaZGaTtgrgsZBoShkXuWFB6WA9EuAu7a3evOoh54777wT69evR1VVFVSq+iqvdevWYdasWfj6668xZsyYsNtOmDABxcXFOHIktFfSihUr8PDDD+P06dPIyMhoct9ffPEFxo4d2yhwnU+XDj2AVBPw8X8AP7wtPZ7wZ2DkTMCY2rHlEEWpKv7UHuliihWF0gdQXApgSKkfG5JC50V6CsTrkX6FBkJNdbFUfVvt/9VTXSwFj85EUEgf2EqN9Is4MCjU/nkq/1gjtZcJXi94HYVaes+Ca21MGVGtXg5LFKWwGRKEDklftp661j+v2iC1RdDGS+2INPH+djQG6ctFY/D/koyTxhqDtI3aIK2n1gc9NtSvp9Z3jlMXXk/0/pb2yvpaobKfpFMYWaOldjfRaLNlrwRKvpcCgTkDMPWWjv1o/J08LqmheODLv64qdDoQAoKX1VW1PhTINSIG/7Sh/jjVxEk/Fsy9pVobsz/kxPeMvAb1QoiiFFblWqngGqm68zwOWj8hC8hb0upitMf3d0T/Aw8cOIAhQ4aEBB4AGDFihLy8qdBz4MABXH311Y3mB7Y9ePBgs6Hn888/BwBcdNFFkRS561OqgF//t/RFsOcVYOsyaTBlAOkXS0PGJUCvUaHnyC+UxwWc/bcUcIr3SGGnqVMdzdHEA3HJ4YORxij90gqEmepTUlV3s20bIIWM+F7+D4QM/5easr4hbmBaUDQxz99oN3ieSlsfTFRaKcSoNP4wo248T+VfV6ntfKGkvQmC9L6bM4AB19TP93mlRreB2qDaMn948QcZjTHosSlo2j/uyA/1aIjmcWJIAvpcJQ2dgSGp87RTVGnqj+eWCq49CQQjly000KiDAk3gdI9Kd+G1eR1BEPzlj6wNbVcQ0f/CiooK9OvXr9H8pKQkeXlz2wbWi3Tbffv24emnn8bUqVPlkNQUp9MJp7O+JsBqjeC8bmelUACTVkpf8t+/Jf2itvob+R3ZVL9eYt/6EJR+MdBrZNNVng3ZK+sbM57aI/Uea9gmQqmRwlXWaKlbqrNGqg6tLQfs5f5xULWpzyOdwnDVSF+GLX69av+HUKZU25Hg/8WTkCVNmzJa3saAOo5CCST3l4YhN0S7NETtR6GQTmldSE85ioqIf3o014j4fA2MW7PtyZMnccMNNyAzMxOvvfbaecu3YsUKPP744+ddr8sRBGDMAmlw2qRamJLvpXBS8r3UeLDK3xvk4IbARv5raPhDUPolQM9h0q+NimP+gOOvySn/qfE+9UlA1pX15/rTL2558hdFqa1DbUVQIAoKRrXlUmiKT5PCjNkfaBKypC6n3f1XPxERdbiIQk9ycnLYGpnKSqkhWLianAvZtqioCHl5eVCpVCgoKGj2+QOWLFmCBx98UH5stVqRmZl53u26FK2xcVV1XRVQ8kN9CDrzvdTts/xHafj3Omk9QSnV/jR17Y1At9SsK6XeOa09xy4IQb+Eclr3HERERG0ootAzfPhwrFu3Dh6PJ6Rdz/79+wEAw4YNa3bbwHrBmtq2qKgI48aNgyiK2LFjB3r37t2iMmq1Wmi1nfz6Ie1Bnwj0z5OGAFupvzboe+mCYyXfSe0sHNVSW5SMS6QanKwrgd5XSG1viIiIuqmIem9t2bIF1113HdavX48ZM2bI8ydPnox9+/Y122X95Zdfxr333ovdu3dj9OjRAKQu66NGjYLRaMTu3bvldU+dOoWxY8fC6/Vix44dYdsRtVSX773VlkRRagdUWy5dV6WzX1yOiIhiVtR7b02ePBkTJkzA/PnzYbVakZOTg3Xr1uHTTz/FmjVr5MAzb9485Ofno7CwENnZ2QCAuXPnYtWqVZg+fbp8ccLVq1fjxx9/xLZt2+R9lJaWIi8vD2fPnsXrr7+O0tJSlJbW9xrq3bt3i2t9qAEh6KquREREMSbihswbNmzAI488gmXLlsm3oVi3bl3IbSi8Xi+8Xi+CK5G0Wi0KCgqwePFi3H///bDb7Rg1ahS2bNkiX40ZAA4dOoTjx48DAGbPnt1o/8uXL8djjz0WabGJiIgoxvGGo0RERNTptMf3dxe4ShIRERHRhWPoISIiopjA0ENEREQxgaGHiIiIYgJDDxEREcUEhh4iIiKKCQw9REREFBMYeoiIiCgmMPQQERFRTGDoISIiopjA0HMeNpcNbxx4AxuObkA3v2MHERFRtxbxDUdjhdvrxns/vYdX/v0KqpxVAICD5QexZPQSqBR824iIiLoafns3IIoi/lH0D/ztu7/hVM0pAEB6XDrO1p7Fuz+9i1J7KZ4e+zT0Kn2US0pERESR4OmtIHt/3ovZm2fjjzv/iFM1p5CkS8LSK5di07RN+Ou4v0Kj0GDH6R2447M7UOmojHZxiYiIKAKC2M0bqrTk1vTHq4/j+e+ex47iHQAAvUqPORfNwe8v+j3i1HHyet+Xfo8FBQtgdVmRFZ+FV655BZmmzA54FURERLGlJd/fkYrp0FNqL8XqH1Zj47GN8Ik+KAUlbh5wM+aPmo8UfUrY5ztuOY75W+ejpLYESbokrBq/CsNShnXESyEiIooZDD2tEO5Ns7ls+PvBv+OtQ2+hzlMHAPhV5q/wwKUPoJ+533mfs8xehvsK7sPhysPQq/R4duyz+GXvX7br6yAiIoolDD2tEPym6Y16vP/T+3jl36/IbXJG9hiJ/7zsP3Fx6sURPW+tuxYP7ngQu0p2QSkosSx3GaYNmNYeL4GIiCjmMPS0QuBN27BvA14/9jqKrEUAgGxTNhZeshDjs8ZDEIRWPbfb58Zjux7D/yv8fwCA+SPnY/7I+a1+PiIiIpIw9LRC4E0b8vIQKPVKJOmSMH/kfNw88GaoFeoLfn5RFPHfP/w3/nff/wIApuZMxdLcpW3y3ERERLGqPUJPzFynR6/UY+7IuZhz0ZyQHlkXShAE3H/x/UgzpOEve/6Cjcc2orSuFH8d+1cY1IY22w8AOL1O2Fw2JOmSukVtksPjgFap7RavhTont9cNu8cOo9oIpUIZ7eJQNySKIqwuKxweB1INqfw86+Ripqan8Gwh+vU8fyPlC7GjeAcW7VwEh9eBoclDsWr8qiZ7gbWEKIo4Vn0Mu0p24Z8l/8Tec3vh9DqhV+mRYcxA7/jeyIzPRG+jNM6Mz0S6MR0apabtXlQbEEURZ2vP4nDlYRyuOIwjlUdwuOIwSutKoVKokKhNRKJOGpJ0SUjSJcnzknRJ9cu0STBpTVAIbXt5Ka/PC5fPBZfXBafXCZc3aDpovtPrhNvrrp/2uUPmqxQq6FV6GFQG6NV66FX1g0FlqH/sX9ZVagNFUUSdpw4WpwUWlwUWpwXVzmpYnBZYXdaQx8HzXD4X4tXxMGvNMGlMMGlN0lhjanZenDqu0RdHcBmqndWNBovTgipHVaPlte5aAIBCUMCsMYccZw2Pu0RdIhK10nSCLqHL/H0CvD4vbG6bNLhsqHHVoMZVA5vbJo9tLmm5VqlFij4FyfpkJOuS5ekkXRKvOO/nE32odlajzF6G8rpylNpLUV5XjrI66XGZvUyednqdAIA4dRwGJAzAgERpGJg4EAMSB8CkaZtailjD01ut0B5vWnP2le3DgoIFqHJWIcOYgZeveRl9zX1bvH1FXQV2n90tB52yurKI9i9AQM+4nlIYahCKesf3hllrjvQlRcQn+lBkLZKDzeFKabA4LW3y/EpBiQRtgvxFFXg9bp8bbp8bHp8HHp9HnpbHXjc8YoOxf7mI6PwXUCvUIcFIr9JDo9RAKSihUqigEBRQKpRQCdK0PC94eZhpAPCKXvhEH7w+/1isH4uiGPI4eBwYHB5HSMhx+9wd9r4oBSXiNfEwaUzQqrRymAp8sXSUeE28HI4StAkQBEE6VkRADPwT5an6+eHm+Y8xAQIUggIKQQFBEKBA/bRSUErT/nUEQZD/rsHz6tx1qHH7A43Lhhq3NLZ77Bf8mgUISNAmSGGoQSAKnk7Rp8CsNXd4MPT6vLC6rCFBt85TBx988vvuE6Vpn+gLeSxP+/9GwcsdXgfK7eUorStFuV0KNhV1FfCInhaXTSko4RW9YZf1jOsph6FAEOpr6gu1sn3eP1EUpTMDgcAbdJw0DMLhlts9dggQoFaqoVY0GBrOU6qhUqjCrpMel44Zg2e0+nUw9LRCR4ceACiyFmH+tvkorilGgjYBL/3qJYxKHRV2XZfXhe9Lv5dDzuHKwyHLdUodLu15Kcb0GoMx6WOQbcpGSW0JimuKcbrmdP3YJo0DXfCbEq+JR3pcOsxaM+I18SGDSWOSptWN5xnUhkY1LG6fG8erj+NQxSEp5FQexo+VP4b98FUJKuQk5mBw0mAMSRqCoclD0dfcF3WeOlQ6KlHlqJLHVU5pWn7sH2rcNZH9IVpBISigVWqhUWqgVUhjjVIjz5MfN1imVqjh8XlQ56lDnacOdo9dnq7z1KHOXT+vqQ/Gzk6tUMOsNSNBmyDXzAQeB2ppguepFWrUuGrk2h+rywqr0yrXBIWb5/K5zluGwP4C4TcwHTyYtVKtToI2AXqVHlaXNeR4qnRUospZFXrc+Y+9amc1fKKvg97VtqdT6mDUGGFUGxGviYdRbYRREzStNqLOW4eKugpU1FWgvK4cFY4KVDoqI37dOqUOceq48Pvxl0EuR5gyiaKIKmeDGjpHgxq8oOVWp7XDf6Qk6ZKQok9BD30PaWzogR76HvI4RZ+CFH0KlAoliixFOFp9FD9V/YSjVUdxtOooSmpLwj6vSqFCX3PfkDDUx9QHHp8Hdo8ddrc9/Ng/Xeepk+fVumtD5te4ajr0h0pThqcMx9rr17Z6e4aeVohG6AGkGpsFBQtwoOIAtEotnv7l0/hV1q8giiKOW45jV8ku7CrZhX+d+1ejoDI4aTBy03MxJn0MLk69GFqltkX7FEURFY6K0DBUU4zTNmlcXlfe6tejEBTyB5ZJY4IIEYXVhWH/Y+mUOgxMGoghSUOkIXkIchJyLvi0m8vrCglFVQ7pC0ohKKBW1P/aaGocdlqphkpQyQGmvav2RVGE2+cODUdBgcjtc8u1Mx7R02jaK3rrx0HTHp9HWs/ngSAIITUHTY4V9Y8brq9RakIChEljgl6lb/f2Cg6PIyQIObyOkFBjUBnavQyB2oTgcGRxWiBChBD4J0hjAPJ02HnB8wXINQwNayGCa+F88IXM94m+kHl6lV4KF2p/kAiajlfHt7r2wOvzotpZjQqHPwjVSUEoMB0IR4H50aohBSCfNk3QJiBOHSfXggVqz0LG/tqzpqYDnx/BISYQapJ1yRdcG1PjqsGx6mP4qfInHK2WgtBPVT/B5ra10bvRNAFC+NDpnzZpTI3CaLwmHnHqOIiQPqvcXndITXrw4+DlLp+r0bI0Qxp+f9HvW11+hp5WiFboAQC7245FXyzCF6e/gEJQIC8zD/vL96PUXhqyXoo+BWPSxyA3PRdX9rrygtoBna88Z2xncM5+Tj7fb3VZ5engITDf6rI2+4shXh2PwclS7c3gpMEYmjwU2aZstgsg6sYC7YcathUKfhw4XRIyHdTmyOa2QYDQuKZOF1Rb5z+1GKi5C9QkdrX2Vg2Jooifa3+WaoSCaoZO15yGVqWFQWWQBrVBbiMYeBynipPnG9RSW8HgxwaVQQ4w4WrouxKGnlaIZugBAI/Pgyd3P4kPjn4gz9Mqtbgs7TLkpuciNz0XAxIGdNoW/4Fzww3DkMfnQU5iDnobe3fashNR5xU4ldaVv5SpfbHLehekUqiwPHc5hiYPxRnbGVzZ60pcknZJi09ZRZsgCNCpdNCpdOhh6BHt4hBRN8GwQ9HA0NMBBEHAbwf9NtrFICIiimmM2kRERBQTGHqIiIgoJjD0EBERUUxg6CEiIqKYwNBDREREMYGhh4iIiGICQw8RERHFBIYeIiIiigkMPURERBQTGHqIiIgoJjD0EBERUUxg6CEiIqKYwNBDREREMYGhh4iIiGICQw8RERHFBIYeIiIiigkMPURERBQTGHqIiIgoJjD0EBERUUxg6CEiIqKYwNBDREREMSHi0GOz2bBw4UKkp6dDp9Nh1KhRWL9+fYu2LS0txZw5c5CSkgKDwYDc3FwUFBSEXXfbtm3Izc2FwWBASkoK5syZg9LS0kiLS0RERASgFaFn2rRpyM/Px/Lly7FlyxZcfvnlmDlzJtauXdvsdk6nE+PHj0dBQQFefPFFfPTRR0hLS8OkSZOwc+fOkHV37tyJyZMnIy0tDR999BFefPFFbNu2DePHj4fT6Yy0yEREREQQRFEUW7ry5s2bcf3112Pt2rWYOXOmPP/aa6/FwYMHcerUKSiVyrDbrl69Gvfddx927dqF3NxcAIDH48HIkSNhNBqxZ88eed0rrrgCtbW1+Pe//w2VSgUA2LVrF6666iqsXr0a8+fPb/ELtFqtMJvNsFgsMJlMLd6OiIiIoqc9vr8jqunZuHEjjEYjpk+fHjL/9ttvR0lJSUhwCbftoEGD5MADACqVCrNnz8Y333yDM2fOAADOnDmDb7/9FrfddpsceABgzJgxGDhwIDZu3BhJkYmIiIgARBh6Dhw4gCFDhoSEEQAYMWKEvLy5bQPrhdv24MGDIc/R1LrN7QOQTqNZrdaQgYiIiCii0FNRUYGkpKRG8wPzKioqLnjbwLipdZvbBwCsWLECZrNZHjIzM5tdn4iIiGJDxA2ZBUFo1bJIt21q3fPtY8mSJbBYLPJQXFzc7PpEREQUG1TnX6VecnJy2JqWyspKAOFrZyLdNjk5GUD4WqPKyspm9wEAWq0WWq222XWIiIgo9kRU0zN8+HAcPnwYHo8nZP7+/fsBAMOGDWt228B6zW0bGDe1bnP7ICIiImpKRKFn6tSpsNls+OCDD0Lm5+fnIz09HaNHj2522yNHjoT08PJ4PFizZg1Gjx6N9PR0AEBGRgauuOIKrFmzBl6vV1539+7d+PHHHzFt2rRIikxEREQEIMLr9ADSNXn27t2Lp556Cjk5OVi3bh1effVVrFmzBrfeeisAYN68ecjPz0dhYSGys7MBSL2qLr30UlitVqxcuRKpqalYvXo1Pv74Y2zbtg1jx46V97Fjxw5MmDABU6ZMwb333ovS0lI89NBDMJvN2Lt3b0Snr3idHiIioq4n6tfpAYANGzbgtttuw7JlyzBp0iTs2bMH69atkwMPAHi9Xni9XgTnKa1Wi4KCAuTl5eH+++/HlClTcPbsWWzZsiUk8ADAuHHjsHnzZpw9exZTpkzB/fffj7y8PBQUFLC9DhEREbVKxDU9XQ1reoiIiLqeTlHTQ0RERNQVMfQQERFRTGDoISIiopjA0ENEREQxgaGHiIiIYgJDDxEREcUEhh4iIiKKCQw9REREFBMiust6VxS49qLVao1ySYiIiKilAt/bbXkN5W4fempqagAAmZmZUS4JERERRaqiogJms7lNnqvb34bC5/OhpKQE8fHxEAQh2sWJOqvViszMTBQXF/O2HG2I72vb43vaPvi+tj2+p+3DYrEgKysLVVVVSEhIaJPn7PY1PQqFAr179452MTodk8nE/5ztgO9r2+N72j74vrY9vqftQ6Fou+bHbMhMREREMYGhh4iIiGICQ0+M0Wq1WL58ObRabbSL0q3wfW17fE/bB9/Xtsf3tH20x/va7RsyExEREQGs6SEiIqIYwdBDREREMYGhh4iIiGICQ08M2LFjBwRBCDvs3r072sXrEmpqarB48WJce+216NGjBwRBwGOPPRZ23e+++w7XXHMNjEYjEhISMG3aNBw/frxjC9xFtPR9nTNnTtjjd/DgwR1f6E7s888/x9y5czF48GDExcUhIyMDN954I/71r381WpfHacu19H3lcRqZH374Addffz2ysrKg1+uRlJSE3NxcrFmzptG6bXW8dvuLE1K9//qv/0JeXl7IvGHDhkWpNF1LRUUF/vd//xcjR47ETTfdhNdeey3sekeOHMG4ceMwatQovPvuu3A4HFi2bBmuvvpq/PDDD+jRo0cHl7xza+n7CgB6vR6ff/55o3lU7+WXX0ZFRQUeeOABDB06FGVlZXjuuedw5ZVX4rPPPsOvfvUrADxOI9XS9xXgcRqJ6upqZGZmYubMmcjIyEBtbS3efvtt3HbbbTh58iQeffRRAG18vIrU7W3fvl0EIL733nvRLkqX5fP5RJ/PJ4qiKJaVlYkAxOXLlzdab/r06WJKSoposVjkeSdPnhTVarW4ePHijipul9HS9/X3v/+9GBcX18Gl63rOnTvXaF5NTY2YlpYmjh8/Xp7H4zQyLX1feZy2jdGjR4uZmZny47Y8Xnl6i6gFAtXUzfF4PNi0aRNuvvnmkEvRZ2dnIy8vDxs3bmzvYnY5LXlfqeVSU1MbzTMajRg6dCiKi4sB8DhtjZa8r9R2UlJSoFJJJ6La+nhl6Ikh9913H1QqFUwmEyZOnIivvvoq2kXqVgoLC1FXV4cRI0Y0WjZixAgcO3YMDocjCiXrHurq6tCzZ08olUr07t0bCxYsQGVlZbSL1elZLBZ89913uOiiiwDwOG0rDd/XAB6nkfP5fPB4PCgrK8Pq1avx2Wef4U9/+hOAtj9e2aYnBpjNZjzwwAMYN24ckpOTcezYMTzzzDMYN24cPvnkE0ycODHaRewWKioqAABJSUmNliUlJUEURVRVVaFXr14dXbQub+TIkRg5cqTcBm3nzp14/vnnUVBQgG+//RZGozHKJey87rvvPtTW1uKRRx4BwOO0rTR8XwEep61177334n/+538AABqNBn/7299w9913A2j745WhJwZcfPHFuPjii+XHV199NaZOnYrhw4dj8eLFDD1trLnTNTyV0zp/+MMfQh5PmDABF198MX7zm9/g1VdfbbScJEuXLsXbb7+Nl156CZdeemnIMh6nrdfU+8rjtHUefvhh3HHHHSgtLcXHH3+MBQsWoLa2Fn/84x/lddrqeGXoiVEJCQm44YYb8Morr6Curo69C9pAcnIygPpfJsEqKyshCAISEhI6uFTd19SpUxEXF8fLLjTh8ccfx5NPPom//OUvWLBggTyfx+mFaep9bQqP0/PLyspCVlYWAOC6664DACxZsgS///3v2/x4ZZueGCb6b7vGX3Vto3///tDr9di/f3+jZfv370dOTg50Ol0UStZ9iaIIhYIfYw09/vjjeOyxx/DYY4/h4YcfDlnG47T1mntfm8PjNDJXXHEFPB4Pjh8/3ubHK/8KMaqqqgqbNm3CqFGj+AHXRlQqFaZMmYINGzagpqZGnn/q1Cls374d06ZNi2Lpup/3338fdrsdV155ZbSL0qk88cQTeOyxx/Doo49i+fLljZbzOG2d872vTeFxGrnt27dDoVCgX79+bX688i7rMWDWrFnIysrCZZddhpSUFBw9ehTPPfccCgsLsWXLFlxzzTXRLmKXsGXLFtTW1qKmpgZz587F9OnT8dvf/haAVCVrMBhw5MgRXH755bjkkkvw0EMPyRfRqqys5EXfmnC+97WsrAyzZs3CLbfcgpycHAiCgJ07d+KFF15A//79sWfPHsTFxUX5VXQOzz33HP74xz9i0qRJYb+YA1+8PE4j05L3taioiMdphO666y6YTCZcccUVSEtLQ3l5Od577z288847WLRoEZ5++mkAbXy8XvBVhKjTW7FihThq1CjRbDaLSqVS7NGjhzh16lTxm2++iXbRupTs7GwRQNjhxIkT8np79+4Vx48fLxoMBtFkMok33XSTeOzYsegVvJM73/taWVkpTp06VezTp4+o1+tFjUYjDhgwQFy8eLFYXV0d7eJ3KmPHjm3yvWz4cc/jtOVa8r7yOI3cG2+8IV599dViSkqKqFKpxISEBHHs2LHiW2+91WjdtjpeWdNDREREMYFteoiIiCgmMPQQERFRTGDoISIiopjA0ENEREQxgaGHiIiIYgJDDxEREcUEhh4iIiKKCQw9REREFBMYeoiIiCgmMPQQERFRTGDoISIiopjA0ENEREQx4f8Dsnuo/FrkZW8AAAAASUVORK5CYII=\n", - "text/plain": [ - "
    " - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "execution_count": 20, + "id": "18fc9dab", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "\n", "import matplotlib.pyplot as plt\n", @@ -2077,7 +2132,7 @@ "\n", "n = 100\n", "n_boostraps = 100\n", - "maxdepth = 30\n", + "maxdepth = 8\n", "\n", "# Make data set.\n", "x = np.linspace(-3, 3, n).reshape(-1, 1)\n", @@ -2129,8 +2184,10 @@ }, { "cell_type": "markdown", - "id": "79f3b548", - "metadata": {}, + "id": "c4507499", + "metadata": { + "editable": true + }, "source": [ "## Random forests\n", "\n", @@ -2150,8 +2207,10 @@ }, { "cell_type": "markdown", - "id": "f51ddbb5", - "metadata": {}, + "id": "30c44f26", + "metadata": { + "editable": true + }, "source": [ "$$\n", "m\\approx \\sqrt{p}.\n", @@ -2160,8 +2219,10 @@ }, { "cell_type": "markdown", - "id": "5307d349", - "metadata": {}, + "id": "182dde82", + "metadata": { + "editable": true + }, "source": [ "In building a random forest, at\n", "each split in the tree, the algorithm is not even allowed to consider\n", @@ -2183,8 +2244,10 @@ }, { "cell_type": "markdown", - "id": "f5df6740", - "metadata": {}, + "id": "441e79b9", + "metadata": { + "editable": true + }, "source": [ "## Random Forest Algorithm\n", "The algorithm described here can be applied to both classification and regression problems.\n", @@ -2207,8 +2270,10 @@ }, { "cell_type": "markdown", - "id": "e1f7886c", - "metadata": {}, + "id": "906bdc82", + "metadata": { + "editable": true + }, "source": [ "## Random Forests Compared with other Methods on the Cancer Data" ] @@ -2216,8 +2281,11 @@ { "cell_type": "code", "execution_count": 21, - "id": "4e31e8c8", - "metadata": {}, + "id": "d837183f", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", @@ -2285,8 +2353,10 @@ }, { "cell_type": "markdown", - "id": "3de40f94", - "metadata": {}, + "id": "837cb618", + "metadata": { + "editable": true + }, "source": [ "Recall that the cumulative gains curve shows the percentage of the\n", "overall number of cases in a given category *gained* by targeting a\n", @@ -2299,8 +2369,10 @@ }, { "cell_type": "markdown", - "id": "9bff02d3", - "metadata": {}, + "id": "bb3be0d1", + "metadata": { + "editable": true + }, "source": [ "## Compare Bagging on Trees with Random Forests" ] @@ -2308,8 +2380,11 @@ { "cell_type": "code", "execution_count": 22, - "id": "23f8734b", - "metadata": {}, + "id": "51a8089f", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "bag_clf = BaggingClassifier(\n", @@ -2320,8 +2395,11 @@ { "cell_type": "code", "execution_count": 23, - "id": "59bd532d", - "metadata": {}, + "id": "ad835b2a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "bag_clf.fit(X_train, y_train)\n", @@ -2335,8 +2413,10 @@ }, { "cell_type": "markdown", - "id": "e24309a7", - "metadata": {}, + "id": "3b670a57", + "metadata": { + "editable": true + }, "source": [ "## Boosting, a Bird's Eye View\n", "\n", @@ -2353,8 +2433,10 @@ }, { "cell_type": "markdown", - "id": "8f84b7a8", - "metadata": {}, + "id": "fca2f8ea", + "metadata": { + "editable": true + }, "source": [ "## What is boosting? Additive Modelling/Iterative Fitting\n", "\n", @@ -2365,8 +2447,10 @@ }, { "cell_type": "markdown", - "id": "a8144faa", - "metadata": {}, + "id": "f083d00f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f_M(x) = \\sum_{i=1}^M \\beta_m b(x;\\gamma_m),\n", @@ -2375,8 +2459,10 @@ }, { "cell_type": "markdown", - "id": "9f523032", - "metadata": {}, + "id": "5e67f8ab", + "metadata": { + "editable": true + }, "source": [ "where $\\beta_m$ are the expansion parameters to be determined in a\n", "minimization process and $b(x;\\gamma_m)$ are some simple functions of\n", @@ -2390,8 +2476,10 @@ }, { "cell_type": "markdown", - "id": "b0d77841", - "metadata": {}, + "id": "0a3b13c4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sigma(t) = \\frac{1}{1+\\exp{(-t)}},\n", @@ -2400,8 +2488,10 @@ }, { "cell_type": "markdown", - "id": "3e086177", - "metadata": {}, + "id": "b4cd2760", + "metadata": { + "editable": true + }, "source": [ "where $t=\\gamma_0+\\gamma_1 x$ and the parameters $\\gamma_0$ and\n", "$\\gamma_1$ were determined by the Logistic Regression fitting\n", @@ -2412,8 +2502,10 @@ }, { "cell_type": "markdown", - "id": "15ad95dc", - "metadata": {}, + "id": "cbba0135", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{y},\\boldsymbol{f}) = \\frac{1}{n} \\sum_{i=0}^{n-1}(y_i-f(x_i))^2.\n", @@ -2422,8 +2514,10 @@ }, { "cell_type": "markdown", - "id": "6287183a", - "metadata": {}, + "id": "9a94bd46", + "metadata": { + "editable": true + }, "source": [ "In this case the function $f(x)$ was replaced by the design matrix\n", "$\\boldsymbol{X}$ and the unknown linear regression parameters $\\boldsymbol{\\beta}$,\n", @@ -2433,8 +2527,10 @@ }, { "cell_type": "markdown", - "id": "27de8464", - "metadata": {}, + "id": "7ddc4cca", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -2443,16 +2539,20 @@ }, { "cell_type": "markdown", - "id": "7d8e41cb", - "metadata": {}, + "id": "ab04ee35", + "metadata": { + "editable": true + }, "source": [ "In iterative fitting or additive modeling, we minimize the cost function with respect to the parameters $\\beta_m$ and $\\gamma_m$." ] }, { "cell_type": "markdown", - "id": "f45f6f2a", - "metadata": {}, + "id": "f1065910", + "metadata": { + "editable": true + }, "source": [ "## Iterative Fitting, Regression and Squared-error Cost Function\n", "\n", @@ -2477,8 +2577,10 @@ }, { "cell_type": "markdown", - "id": "af260098", - "metadata": {}, + "id": "e225cab9", + "metadata": { + "editable": true + }, "source": [ "## Squared-Error Example and Iterative Fitting\n", "\n", @@ -2491,8 +2593,10 @@ }, { "cell_type": "markdown", - "id": "f1a85e34", - "metadata": {}, + "id": "86833101", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\beta_m,\\gamma_m) = \\mathrm{argmin}_{\\beta,\\lambda}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta b(x;\\gamma))^2=\\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta(1+\\gamma x_i))^2.\n", @@ -2501,8 +2605,10 @@ }, { "cell_type": "markdown", - "id": "a809e54d", - "metadata": {}, + "id": "09b937d0", + "metadata": { + "editable": true + }, "source": [ "We start our iteration by simply setting $f_0(x)=0$. \n", "Taking the derivatives with respect to $\\beta$ and $\\gamma$ we obtain" @@ -2510,8 +2616,10 @@ }, { "cell_type": "markdown", - "id": "21b8fb9c", - "metadata": {}, + "id": "0aa5e94a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial {\\cal C}}{\\partial \\beta} = -2\\sum_{i}(1+\\gamma x_i)(y_i-\\beta(1+\\gamma x_i))=0,\n", @@ -2520,16 +2628,20 @@ }, { "cell_type": "markdown", - "id": "d40d3542", - "metadata": {}, + "id": "cef4a5df", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "id": "30b388a2", - "metadata": {}, + "id": "a87160e1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial {\\cal C}}{\\partial \\gamma} =-2\\sum_{i}\\beta x_i(y_i-\\beta(1+\\gamma x_i))=0.\n", @@ -2538,16 +2650,20 @@ }, { "cell_type": "markdown", - "id": "8f5986c5", - "metadata": {}, + "id": "053210aa", + "metadata": { + "editable": true + }, "source": [ "We can then rewrite these equations as (defining $\\boldsymbol{w}=\\boldsymbol{e}+\\gamma \\boldsymbol{x})$ with $\\boldsymbol{e}$ being the unit vector)" ] }, { "cell_type": "markdown", - "id": "605abac6", - "metadata": {}, + "id": "20c89e86", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\gamma \\boldsymbol{w}^T(\\boldsymbol{y}-\\beta\\gamma \\boldsymbol{w})=0,\n", @@ -2556,16 +2672,20 @@ }, { "cell_type": "markdown", - "id": "f67ecd40", - "metadata": {}, + "id": "2da6a7a0", + "metadata": { + "editable": true + }, "source": [ "which gives us $\\beta = \\boldsymbol{w}^T\\boldsymbol{y}/(\\boldsymbol{w}^T\\boldsymbol{w})$. Similarly we have" ] }, { "cell_type": "markdown", - "id": "85880a97", - "metadata": {}, + "id": "e539d7b7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta\\gamma \\boldsymbol{x}^T(\\boldsymbol{y}-\\beta(1+\\gamma \\boldsymbol{x}))=0,\n", @@ -2574,8 +2694,10 @@ }, { "cell_type": "markdown", - "id": "e801e0c6", - "metadata": {}, + "id": "b9d48dde", + "metadata": { + "editable": true + }, "source": [ "which leads to $\\gamma =(\\boldsymbol{x}^T\\boldsymbol{y}-\\beta\\boldsymbol{x}^T\\boldsymbol{e})/(\\beta\\boldsymbol{x}^T\\boldsymbol{x})$. Inserting\n", "for $\\beta$ gives us an equation for $\\gamma$. This is a non-linear equation in the unknown $\\gamma$ and has to be solved numerically. \n", @@ -2586,8 +2708,10 @@ }, { "cell_type": "markdown", - "id": "05e0fc0e", - "metadata": {}, + "id": "d731116e", + "metadata": { + "editable": true + }, "source": [ "## Iterative Fitting, Classification and AdaBoost\n", "\n", @@ -2600,8 +2724,10 @@ }, { "cell_type": "markdown", - "id": "d1c5a166", - "metadata": {}, + "id": "243e0159", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{\\overline{err}}=\\frac{1}{n} \\sum_{i=0}^{n-1} I(y_i\\ne G(x_i)).\n", @@ -2610,8 +2736,10 @@ }, { "cell_type": "markdown", - "id": "4476c4c7", - "metadata": {}, + "id": "55a6a0d1", + "metadata": { + "editable": true + }, "source": [ "The iterative procedure starts with defining a weak classifier whose\n", "error rate is barely better than random guessing. The iterative\n", @@ -2624,8 +2752,10 @@ }, { "cell_type": "markdown", - "id": "8b35c8d2", - "metadata": {}, + "id": "1b2d75e9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f_M(x) = \\sum_{i=1}^M \\beta_m b(x;\\gamma_m),\n", @@ -2634,16 +2764,20 @@ }, { "cell_type": "markdown", - "id": "a533e45a", - "metadata": {}, + "id": "f6cee5c0", + "metadata": { + "editable": true + }, "source": [ "will be a function of" ] }, { "cell_type": "markdown", - "id": "b1e33a99", - "metadata": {}, + "id": "66050e09", + "metadata": { + "editable": true + }, "source": [ "$$\n", "G_M(x) = \\mathrm{sign} \\sum_{i=1}^M \\alpha_m G_m(x).\n", @@ -2652,8 +2786,10 @@ }, { "cell_type": "markdown", - "id": "4199e7a8", - "metadata": {}, + "id": "56a630c2", + "metadata": { + "editable": true + }, "source": [ "## Adaptive Boosting, AdaBoost\n", "\n", @@ -2662,8 +2798,10 @@ }, { "cell_type": "markdown", - "id": "db2fda6f", - "metadata": {}, + "id": "5333aca1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f_m(x) = f_{m-1}(x)+\\beta_mG_m(x).\n", @@ -2672,8 +2810,10 @@ }, { "cell_type": "markdown", - "id": "dbb7d266", - "metadata": {}, + "id": "58533db8", + "metadata": { + "editable": true + }, "source": [ "The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the\n", "exponential cost/loss function defined as" @@ -2681,8 +2821,10 @@ }, { "cell_type": "markdown", - "id": "70431c63", - "metadata": {}, + "id": "c8f78e0d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{y},\\boldsymbol{f}) = \\sum_{i=0}^{n-1}\\exp{(-y_i(f_{m-1}(x_i)+\\beta G(x_i))}.\n", @@ -2691,8 +2833,10 @@ }, { "cell_type": "markdown", - "id": "9e678e2c", - "metadata": {}, + "id": "80ed0e5b", + "metadata": { + "editable": true + }, "source": [ "We optimize $\\beta$ and $G$ for each value of $m=1:M$ as we did in the regression case.\n", "This is normally done in two steps. Let us however first rewrite the cost function as" @@ -2700,8 +2844,10 @@ }, { "cell_type": "markdown", - "id": "d2957ac5", - "metadata": {}, + "id": "2c203531", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{y},\\boldsymbol{f}) = \\sum_{i=0}^{n-1}w_i^{m}\\exp{(-y_i\\beta G(x_i))},\n", @@ -2710,16 +2856,20 @@ }, { "cell_type": "markdown", - "id": "abebd5ea", - "metadata": {}, + "id": "5d0b9e28", + "metadata": { + "editable": true + }, "source": [ "where we have defined $w_i^m= \\exp{(-y_if_{m-1}(x_i))}$." ] }, { "cell_type": "markdown", - "id": "9d7d7c9f", - "metadata": {}, + "id": "2bd73caf", + "metadata": { + "editable": true + }, "source": [ "## Building up AdaBoost\n", "\n", @@ -2728,8 +2878,10 @@ }, { "cell_type": "markdown", - "id": "9fdaa6db", - "metadata": {}, + "id": "d1ca1da0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "G_m(x) = \\mathrm{sign} \\sum_{i=0}^{n-1} w_i^m I(y_i \\ne G_(x_i)),\n", @@ -2738,8 +2890,10 @@ }, { "cell_type": "markdown", - "id": "f4cbd741", - "metadata": {}, + "id": "ac0baf63", + "metadata": { + "editable": true + }, "source": [ "which is the classifier that minimizes the weighted error rate in predicting $y$.\n", "\n", @@ -2748,8 +2902,10 @@ }, { "cell_type": "markdown", - "id": "8310198c", - "metadata": {}, + "id": "97d3112a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\exp{-(\\beta)}\\sum_{y_i=G(x_i)}w_i^m+\\exp{(\\beta)}\\sum_{y_i\\ne G(x_i)}w_i^m,\n", @@ -2758,16 +2914,20 @@ }, { "cell_type": "markdown", - "id": "b37fe48c", - "metadata": {}, + "id": "9ea22020", + "metadata": { + "editable": true + }, "source": [ "which can be rewritten as" ] }, { "cell_type": "markdown", - "id": "9e909739", - "metadata": {}, + "id": "5fae693c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\exp{(\\beta)}-\\exp{-(\\beta)})\\sum_{i=0}^{n-1}w_i^mI(y_i\\ne G(x_i))+\\exp{(-\\beta)}\\sum_{i=0}^{n-1}w_i^m=0,\n", @@ -2776,16 +2936,20 @@ }, { "cell_type": "markdown", - "id": "d8ad59f0", - "metadata": {}, + "id": "e7721f66", + "metadata": { + "editable": true + }, "source": [ "which leads to" ] }, { "cell_type": "markdown", - "id": "ba2d48e0", - "metadata": {}, + "id": "beecad56", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_m = \\frac{1}{2}\\log{\\frac{1-\\mathrm{\\overline{err}}}{\\mathrm{\\overline{err}}}},\n", @@ -2794,16 +2958,20 @@ }, { "cell_type": "markdown", - "id": "5e1dcb00", - "metadata": {}, + "id": "5aa189b2", + "metadata": { + "editable": true + }, "source": [ "where we have redefined the error as" ] }, { "cell_type": "markdown", - "id": "45c824f3", - "metadata": {}, + "id": "fa4335a9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{\\overline{err}}_m=\\frac{1}{n}\\frac{\\sum_{i=0}^{n-1}w_i^mI(y_i\\ne G(x_i)}{\\sum_{i=0}^{n-1}w_i^m},\n", @@ -2812,16 +2980,20 @@ }, { "cell_type": "markdown", - "id": "c916fad2", - "metadata": {}, + "id": "d0360603", + "metadata": { + "editable": true + }, "source": [ "which leads to an update of" ] }, { "cell_type": "markdown", - "id": "070d7540", - "metadata": {}, + "id": "50af7ce6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f_m(x) = f_{m-1}(x) +\\beta_m G_m(x).\n", @@ -2830,16 +3002,20 @@ }, { "cell_type": "markdown", - "id": "4b1b22a8", - "metadata": {}, + "id": "f0fafe85", + "metadata": { + "editable": true + }, "source": [ "This leads to the new weights" ] }, { "cell_type": "markdown", - "id": "cc2bf04a", - "metadata": {}, + "id": "47a5a700", + "metadata": { + "editable": true + }, "source": [ "$$\n", "w_i^{m+1} = w_i^m \\exp{(-y_i\\beta_m G_m(x_i))}\n", @@ -2848,8 +3024,10 @@ }, { "cell_type": "markdown", - "id": "ea775ce7", - "metadata": {}, + "id": "a98e7bee", + "metadata": { + "editable": true + }, "source": [ "## Adaptive boosting: AdaBoost, Basic Algorithm\n", "\n", @@ -2866,8 +3044,10 @@ }, { "cell_type": "markdown", - "id": "ae10c0a4", - "metadata": {}, + "id": "f7884f46", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{err}=\\frac{1}{n}\\sum_{i=0}^{n-1}I(y_i\\ne G(x_i)),\n", @@ -2876,16 +3056,20 @@ }, { "cell_type": "markdown", - "id": "dfe87ab2", - "metadata": {}, + "id": "a2ac5c4b", + "metadata": { + "editable": true + }, "source": [ "where the function $I()$ is one if we misclassify and zero if we classify correctly." ] }, { "cell_type": "markdown", - "id": "875649dd", - "metadata": {}, + "id": "8a029ef0", + "metadata": { + "editable": true + }, "source": [ "## Basic Steps of AdaBoost\n", "\n", @@ -2898,8 +3082,10 @@ }, { "cell_type": "markdown", - "id": "5aac54e6", - "metadata": {}, + "id": "29784fc8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{\\overline{err}}_m=\\frac{\\sum_{i=0}^{n-1}w_i^m I(y_i\\ne G(x_i))}{\\sum_{i=0}^{n-1}w_i},\n", @@ -2908,8 +3094,10 @@ }, { "cell_type": "markdown", - "id": "a3e84a60", - "metadata": {}, + "id": "5ba646f1", + "metadata": { + "editable": true + }, "source": [ "1. Then we start looping over all attempts at classifying, namely we start an iterative process for $m=1:M$, where $M$ is the final number of classifications. Our given classifier could for example be a plain decision tree.\n", "\n", @@ -2934,8 +3122,10 @@ }, { "cell_type": "markdown", - "id": "d80edac5", - "metadata": {}, + "id": "5e96905a", + "metadata": { + "editable": true + }, "source": [ "## AdaBoost Examples\n", "\n", @@ -2945,8 +3135,11 @@ { "cell_type": "code", "execution_count": 24, - "id": "04a2ab5b", - "metadata": {}, + "id": "82cfb9f3", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.ensemble import AdaBoostClassifier\n", @@ -2973,25 +3166,7 @@ ] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.10" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week44/week44.do.txt b/doc/src/week44/week44.do.txt index 2b93ae0ab..39b7256ea 100644 --- a/doc/src/week44/week44.do.txt +++ b/doc/src/week44/week44.do.txt @@ -10,6 +10,7 @@ DATE: today * "Video of lecture":"https://youtu.be/7jexGH5SOOE" * _Note_: Thursday's lecture is digital only due to "High-school post-education day":"https://www.uio.no/om/samarbeid/skole/fagped-dag/" * Friday: Decision trees and ensemble models (bagging and random forests) + * "Video of lecture":"https://youtu.be/9QcU8VcXxRU" !bblock Videos o "Video on Decision trees":"https://www.youtube.com/watch?v=RmajweUFKvM&ab_channel=Simplilearn"