lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
06b09eb681a311aec756811f6f464148168d0bcb
FYS-STK4155/doc/LectureNotes/_build/html/week46.html
T
Morten Hjorth-Jensen 06b09eb681 update
2023-11-21 06:16:54 +01:00

3451 lines
262 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Week 46: Decision Trees, Ensemble methods and Random Forests &#8212; Applied Data Analysis and Machine Learning</title>
<link href="_static/css/theme.css" rel="stylesheet">
<link href="_static/css/index.ff1ffe594081f20da1ef19478df9384b.css" rel="stylesheet">
<link rel="stylesheet"
href="_static/vendor/fontawesome/5.13.0/css/all.min.css">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
<link rel="stylesheet" type="text/css" href="_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-book-theme.css?digest=c3fdc42140077d1ad13ad2f1588a4309" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-main.c949a650a448cc0ae9fd3441c0e17fb0.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-variables.06eb56fa6e07937060861dad626602ad.css" />
<link rel="preload" as="script" href="_static/js/index.be7d3bbb2ef33a8344ce.js">
<script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/clipboard.min.js"></script>
<script src="_static/copybutton.js"></script>
<script>let toggleHintShow = 'Click to show';</script>
<script>let toggleHintHide = 'Click to hide';</script>
<script>let toggleOpenOnPrint = 'true';</script>
<script src="_static/togglebutton.js"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown, .tag_hide_input div.cell_input, .tag_hide-input div.cell_input, .tag_hide_output div.cell_output, .tag_hide-output div.cell_output, .tag_hide_cell.cell, .tag_hide-cell.cell';</script>
<script src="_static/sphinx-book-theme.d59cb220de22ca1c485ebbdc042f0030.js"></script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"
const thebe_selector = ".thebe,.cell"
const thebe_selector_input = "pre"
const thebe_selector_output = ".output, .cell_output"
</script>
<script async="async" src="_static/sphinx-thebe.js"></script>
<script>window.MathJax = {"options": {"processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script>
<script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Week 47: From Decision Trees to Ensemble Methods, Random Forests and Boosting Methods and Summary of Course" href="week47.html" />
<link rel="prev" title="Week 45, Recurrent Neural Networks" href="week45.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="None">
<!-- Google Analytics -->
</head>
<body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
<div class="container-fluid" id="banner"></div>
<div class="container-xl">
<div class="row">
<div class="col-12 col-md-3 bd-sidebar site-navigation show" id="site-navigation">
<div class="navbar-brand-box">
<a class="navbar-brand text-wrap" href="index.html">
<!-- `logo` is deprecated in Sphinx 4.0, so remove this when we stop supporting 3 -->
<img src="_static/logo.png" class="logo" alt="logo">
<h1 class="site-logo" id="site-title">Applied Data Analysis and Machine Learning</h1>
</a>
</div><form class="bd-search d-flex align-items-center" action="search.html" method="get">
<i class="icon fas fa-search"></i>
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search this book..." aria-label="Search this book..." autocomplete="off" >
</form><nav class="bd-links" id="bd-docs-nav" aria-label="Main">
<div class="bd-toc-item active">
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="intro.html">
Applied Data Analysis and Machine Learning
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
About the course
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="schedule.html">
Teaching schedule with links to material
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="teachers.html">
Teachers and Grading
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="textbooks.html">
Textbooks
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Review of Statistics with Resampling Techniques and Linear Algebra
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="statistics.html">
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="linalg.html">
2. Linear Algebra, Handling of Arrays and more Python Features
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
From Regression to Support Vector Machines
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter1.html">
3. Linear Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html">
4. Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
5. Resampling Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
6. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapteroptimization.html">
7. Optimization, the central part of any Machine Learning algortithm
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
8. Support Vector Machines, overarching aims
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Decision Trees, Ensemble Methods and Boosting
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
9. Decision trees, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Dimensionality Reduction
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
11. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="clustering.html">
12. Clustering and Unsupervised Learning
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Deep Learning Methods
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter9.html">
13. Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter10.html">
14. Building a Feed Forward Neural Network
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter11.html">
15. Solving Differential Equations with Deep Learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter12.html">
16. Convolutional Neural Networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter13.html">
17. Recurrent neural networks: Overarching view
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Weekly material, notes and exercises
</span>
</p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek34.html">
Exercises week 34
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week34.html">
Week 34: Introduction to the course, Logistics and Practicalities
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek35.html">
Exercises week 35
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week35.html">
Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek36.html">
Exercises week 36
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week36.html">
Week 36: Statistical interpretation of Linear Regression and Resampling techniques
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek37.html">
Exercises week 37
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week37.html">
Week 37: Statistical interpretations and Resampling Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek38.html">
Exercises week 38
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week38.html">
Week 38: Logistic Regression and Optimization
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek39.html">
Exercises week 39
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week39.html">
Week 39: Optimization and Gradient Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week40.html">
Week 40: Gradient descent methods (continued) and start Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek41.html">
Exercises week 41
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week41.html">
Week 41 Neural networks and constructing a neural network code
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek42.html">
Exercises week 42
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week42.html">
Week 42 Constructing a Neural Network code with introduction to Tensor flow
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek43.html">
Exercises weeks 43 and 44
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week43.html">
Week 43: Deep Learning: Constructing a Neural Network code and solving differential equations
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week44.html">
Week 44, Convolutional Neural Networks (CNN)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week45.html">
Week 45, Recurrent Neural Networks
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
Week 46: Decision Trees, Ensemble methods and Random Forests
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week47.html">
Week 47: From Decision Trees to Ensemble Methods, Random Forests and Boosting Methods and Summary of Course
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Projects
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="project1.html">
Project 1 on Machine Learning, deadline October 9 (midnight), 2023
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="project2.html">
Project 2 on Machine Learning, deadline November 17 (Midnight)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="project3.html">
Project 3 on Machine Learning, deadline December 18 (midnight), 2023
</a>
</li>
</ul>
</div>
</nav> <!-- To handle the deprecated key -->
<div class="navbar_extra_footer">
Powered by <a href="https://jupyterbook.org">Jupyter Book</a>
</div>
</div>
<main class="col py-md-3 pl-md-4 bd-content overflow-auto" role="main">
<div class="topbar container-xl fixed-top">
<div class="topbar-contents row">
<div class="col-12 col-md-3 bd-topbar-whitespace site-navigation show"></div>
<div class="col pl-md-4 topbar-main">
<button id="navbar-toggler" class="navbar-toggler ml-0" type="button" data-toggle="collapse"
data-toggle="tooltip" data-placement="bottom" data-target=".site-navigation" aria-controls="navbar-menu"
aria-expanded="true" aria-label="Toggle navigation" aria-controls="site-navigation"
title="Toggle navigation" data-toggle="tooltip" data-placement="left">
<i class="fas fa-bars"></i>
<i class="fas fa-arrow-left"></i>
<i class="fas fa-arrow-up"></i>
</button>
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn" aria-label="Download this page"><i
class="fas fa-download"></i></button>
<div class="dropdown-buttons">
<!-- ipynb file if we had a myst markdown file -->
<!-- Download raw file -->
<a class="dropdown-buttons" href="_sources/week46.ipynb"><button type="button"
class="btn btn-secondary topbarbtn" title="Download source file" data-toggle="tooltip"
data-placement="left">.ipynb</button></a>
<!-- Download PDF via print -->
<button type="button" id="download-print" class="btn btn-secondary topbarbtn" title="Print to PDF"
onclick="printPdf(this)" data-toggle="tooltip" data-placement="left">.pdf</button>
</div>
</div>
<!-- Source interaction buttons -->
<!-- Full screen (wrap in <a> to have style consistency -->
<a class="full-screen-button"><button type="button" class="btn btn-secondary topbarbtn" data-toggle="tooltip"
data-placement="bottom" onclick="toggleFullScreen()" aria-label="Fullscreen mode"
title="Fullscreen mode"><i
class="fas fa-expand"></i></button></a>
<!-- Launch buttons -->
</div>
<!-- Table of contents -->
<div class="d-none d-md-block col-md-2 bd-toc show noprint">
<div class="tocsection onthispage pt-5 pb-3">
<i class="fas fa-list"></i> Contents
</div>
<nav id="bd-toc-nav" aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#plan-for-week-46">
Plan for week 46
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#decision-trees-overarching-aims">
Decision trees, overarching aims
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basics-of-a-tree">
Basics of a tree
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-typical-decision-tree-with-its-pertinent-jargon-classification-problem">
A typical Decision Tree with its pertinent Jargon, Classification Problem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#general-features">
General Features
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#how-do-we-set-it-up">
How do we set it up?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#decision-trees-and-regression">
Decision trees and Regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#building-a-tree-regression">
Building a tree, regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-top-down-approach-recursive-binary-splitting">
A top-down approach, recursive binary splitting
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#making-a-tree">
Making a tree
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#pruning-the-tree">
Pruning the tree
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#cost-complexity-pruning">
Cost complexity pruning
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#schematic-regression-procedure">
Schematic Regression Procedure
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-classification-tree">
A Classification Tree
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#growing-a-classification-tree">
Growing a classification tree
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#classification-tree-how-to-split-nodes">
Classification tree, how to split nodes
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#visualizing-the-tree-classification">
Visualizing the Tree, Classification
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#visualizing-the-tree-the-moons">
Visualizing the Tree, The Moons
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-ways-of-visualizing-the-trees">
Other ways of visualizing the trees
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#printing-out-as-text">
Printing out as text
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#algorithms-for-setting-up-decision-trees">
Algorithms for Setting up Decision Trees
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-cart-algorithm-for-classification">
The CART algorithm for Classification
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-cart-algorithm-for-regression">
The CART algorithm for Regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#why-binary-splits">
Why binary splits?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#computing-a-tree-using-the-gini-index">
Computing a Tree using the Gini Index
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-table">
The Table
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#computing-the-various-gini-indices">
Computing the various Gini Indices
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#computing-the-various-gini-indices-hours-slept">
Computing the various Gini Indices, Hours slept
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#computing-the-various-gini-indices-hours-studied">
Computing the various Gini Indices, Hours studied
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-possible-code-using-scikit-learn">
A possible code using Scikit-Learn
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#further-example-computing-the-gini-index">
Further example: Computing the Gini index
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#simple-python-code-to-read-in-data-and-perform-classification">
Simple Python Code to read in Data and perform Classification
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#computing-the-gini-factor">
Computing the Gini Factor
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#regression-trees">
Regression trees
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#final-regressor-code">
Final regressor code
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#pros-and-cons-of-trees-pros">
Pros and cons of trees, pros
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#disadvantages">
Disadvantages
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#ensemble-methods-from-a-single-tree-to-many-trees-and-extreme-boosting-meet-the-jungle-of-methods">
Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#an-overview-of-ensemble-methods">
An Overview of Ensemble Methods
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#why-voting">
Why Voting?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#tossing-coins">
Tossing coins
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#standard-imports-first">
Standard imports first
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#simple-voting-example-head-or-tail">
Simple Voting Example, head or tail
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#using-the-voting-classifier">
Using the Voting Classifier
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#voting-and-bagging">
Voting and Bagging
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#bagging">
Bagging
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#more-bagging">
More bagging
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#making-your-own-bootstrap-changing-the-level-of-the-decision-tree">
Making your own Bootstrap: Changing the Level of the Decision Tree
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-forests">
Random forests
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-forest-algorithm">
Random Forest Algorithm
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-forests-compared-with-other-methods-on-the-cancer-data">
Random Forests Compared with other Methods on the Cancer Data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#compare-bagging-on-trees-with-random-forests">
Compare Bagging on Trees with Random Forests
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#boosting-a-bird-s-eye-view">
Boosting, a Bird’s Eye View
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#what-is-boosting-additive-modelling-iterative-fitting">
What is boosting? Additive Modelling/Iterative Fitting
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#iterative-fitting-regression-and-squared-error-cost-function">
Iterative Fitting, Regression and Squared-error Cost Function
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#squared-error-example-and-iterative-fitting">
Squared-Error Example and Iterative Fitting
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#iterative-fitting-classification-and-adaboost">
Iterative Fitting, Classification and AdaBoost
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#adaptive-boosting-adaboost">
Adaptive Boosting, AdaBoost
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#building-up-adaboost">
Building up AdaBoost
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#adaptive-boosting-adaboost-basic-algorithm">
Adaptive boosting: AdaBoost, Basic Algorithm
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basic-steps-of-adaboost">
Basic Steps of AdaBoost
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#adaboost-examples">
AdaBoost Examples
</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div id="main-content" class="row">
<div class="col-12 col-md-9 pl-md-3 pr-md-0">
<!-- Table of contents that is only displayed when printing the page -->
<div id="jb-print-docs-body" class="onlyprint">
<h1>Week 46: Decision Trees, Ensemble methods and Random Forests</h1>
<!-- Table of contents -->
<div id="print-main-content">
<div id="jb-print-toc">
<div>
<h2> Contents </h2>
</div>
<nav aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#plan-for-week-46">
Plan for week 46
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#decision-trees-overarching-aims">
Decision trees, overarching aims
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basics-of-a-tree">
Basics of a tree
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-typical-decision-tree-with-its-pertinent-jargon-classification-problem">
A typical Decision Tree with its pertinent Jargon, Classification Problem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#general-features">
General Features
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#how-do-we-set-it-up">
How do we set it up?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#decision-trees-and-regression">
Decision trees and Regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#building-a-tree-regression">
Building a tree, regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-top-down-approach-recursive-binary-splitting">
A top-down approach, recursive binary splitting
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#making-a-tree">
Making a tree
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#pruning-the-tree">
Pruning the tree
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#cost-complexity-pruning">
Cost complexity pruning
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#schematic-regression-procedure">
Schematic Regression Procedure
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-classification-tree">
A Classification Tree
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#growing-a-classification-tree">
Growing a classification tree
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#classification-tree-how-to-split-nodes">
Classification tree, how to split nodes
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#visualizing-the-tree-classification">
Visualizing the Tree, Classification
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#visualizing-the-tree-the-moons">
Visualizing the Tree, The Moons
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-ways-of-visualizing-the-trees">
Other ways of visualizing the trees
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#printing-out-as-text">
Printing out as text
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#algorithms-for-setting-up-decision-trees">
Algorithms for Setting up Decision Trees
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-cart-algorithm-for-classification">
The CART algorithm for Classification
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-cart-algorithm-for-regression">
The CART algorithm for Regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#why-binary-splits">
Why binary splits?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#computing-a-tree-using-the-gini-index">
Computing a Tree using the Gini Index
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-table">
The Table
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#computing-the-various-gini-indices">
Computing the various Gini Indices
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#computing-the-various-gini-indices-hours-slept">
Computing the various Gini Indices, Hours slept
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#computing-the-various-gini-indices-hours-studied">
Computing the various Gini Indices, Hours studied
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-possible-code-using-scikit-learn">
A possible code using Scikit-Learn
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#further-example-computing-the-gini-index">
Further example: Computing the Gini index
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#simple-python-code-to-read-in-data-and-perform-classification">
Simple Python Code to read in Data and perform Classification
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#computing-the-gini-factor">
Computing the Gini Factor
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#regression-trees">
Regression trees
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#final-regressor-code">
Final regressor code
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#pros-and-cons-of-trees-pros">
Pros and cons of trees, pros
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#disadvantages">
Disadvantages
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#ensemble-methods-from-a-single-tree-to-many-trees-and-extreme-boosting-meet-the-jungle-of-methods">
Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#an-overview-of-ensemble-methods">
An Overview of Ensemble Methods
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#why-voting">
Why Voting?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#tossing-coins">
Tossing coins
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#standard-imports-first">
Standard imports first
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#simple-voting-example-head-or-tail">
Simple Voting Example, head or tail
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#using-the-voting-classifier">
Using the Voting Classifier
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#voting-and-bagging">
Voting and Bagging
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#bagging">
Bagging
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#more-bagging">
More bagging
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#making-your-own-bootstrap-changing-the-level-of-the-decision-tree">
Making your own Bootstrap: Changing the Level of the Decision Tree
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-forests">
Random forests
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-forest-algorithm">
Random Forest Algorithm
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-forests-compared-with-other-methods-on-the-cancer-data">
Random Forests Compared with other Methods on the Cancer Data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#compare-bagging-on-trees-with-random-forests">
Compare Bagging on Trees with Random Forests
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#boosting-a-bird-s-eye-view">
Boosting, a Bird’s Eye View
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#what-is-boosting-additive-modelling-iterative-fitting">
What is boosting? Additive Modelling/Iterative Fitting
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#iterative-fitting-regression-and-squared-error-cost-function">
Iterative Fitting, Regression and Squared-error Cost Function
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#squared-error-example-and-iterative-fitting">
Squared-Error Example and Iterative Fitting
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#iterative-fitting-classification-and-adaboost">
Iterative Fitting, Classification and AdaBoost
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#adaptive-boosting-adaboost">
Adaptive Boosting, AdaBoost
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#building-up-adaboost">
Building up AdaBoost
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#adaptive-boosting-adaboost-basic-algorithm">
Adaptive boosting: AdaBoost, Basic Algorithm
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basic-steps-of-adaboost">
Basic Steps of AdaBoost
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#adaboost-examples">
AdaBoost Examples
</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div>
<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
doconce format html week46.do.txt --no_mako -->
<!-- dom:TITLE: Week 46: Decision Trees, Ensemble methods and Random Forests --><div class="tex2jax_ignore mathjax_ignore section" id="week-46-decision-trees-ensemble-methods-and-random-forests">
<h1>Week 46: Decision Trees, Ensemble methods and Random Forests<a class="headerlink" href="#week-46-decision-trees-ensemble-methods-and-random-forests" title="Permalink to this headline"></a></h1>
<p><strong>Morten Hjorth-Jensen</strong>, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</p>
<p>Date: <strong>Week 46, November 13-17</strong></p>
<div class="section" id="plan-for-week-46">
<h2>Plan for week 46<a class="headerlink" href="#plan-for-week-46" title="Permalink to this headline"></a></h2>
<p><strong>Active learning sessions on Tuesday and Wednesday.</strong></p>
<ul class="simple">
<li><p>Work and Discussion of project 2</p></li>
<li><p>Discussion of project 3 as well</p></li>
</ul>
<p><strong>Material for the lecture on Thursday November 16, 2023.</strong></p>
<ul class="simple">
<li><p>Thursday: Basics of decision trees, classification and regression algorithms and ensemble models</p></li>
<li><p>Readings and Videos:</p>
<ul>
<li><p>These lecture notes</p></li>
<li><p><a class="reference external" href="https://youtu.be/PMswUwhYa7k">Video of lecture</a></p></li>
<li><p><a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesNov16.pdf">Whiteboard notes</a></p></li>
<li><p><a class="reference external" href="https://www.youtube.com/watch?v=RmajweUFKvM&amp;ab_channel=Simplilearn">Video on Decision trees</a></p></li>
<li><p>Decision Trees: Geron’s chapter 6 covers decision trees while ensemble models, voting and bagging are discussed in chapter 7. See also lecture from <a class="reference external" href="https://www.uio.no/studier/emner/matnat/math/STK-IN4300/h20/slides/lecture_7.pdf">STK-IN4300, lecture 7</a>. Chapter 9.2 of Hastie et al contains also a good discussion.</p></li>
</ul>
</li>
</ul>
</div>
<div class="section" id="decision-trees-overarching-aims">
<h2>Decision trees, overarching aims<a class="headerlink" href="#decision-trees-overarching-aims" title="Permalink to this headline"></a></h2>
<p>We start here with the most basic algorithm, the so-called decision
tree. With this basic algorithm we can in turn build more complex
networks, spanning from homogeneous and heterogenous forests (bagging,
random forests and more) to one of the most popular supervised
algorithms nowadays, the extreme gradient boosting, or just
XGBoost. But let us start with the simplest possible ingredient.</p>
<p>Decision trees are supervised learning algorithms used for both,
classification and regression tasks.</p>
<p>The main idea of decision trees
is to find those descriptive features which contain the most
<strong>information</strong> regarding the target feature and then split the dataset
along the values of these features such that the target feature values
for the resulting underlying datasets are as pure as possible.</p>
<p>The descriptive features which reproduce best the target/output features are normally said
to be the most informative ones. The process of finding the <strong>most
informative</strong> feature is done until we accomplish a stopping criteria
where we then finally end up in so called <strong>leaf nodes</strong>.</p>
</div>
<div class="section" id="basics-of-a-tree">
<h2>Basics of a tree<a class="headerlink" href="#basics-of-a-tree" title="Permalink to this headline"></a></h2>
<p>A decision tree is typically divided into a <strong>root node</strong>, the <strong>interior nodes</strong>,
and the final <strong>leaf nodes</strong> or just <strong>leaves</strong>. These entities are then connected by so-called <strong>branches</strong>.</p>
<p>The leaf nodes
contain the predictions we will make for new query instances presented
to our trained model. This is possible since the model has
learned the underlying structure of the training data and hence can,
given some assumptions, make predictions about the target feature value
(class) of unseen query instances.</p>
</div>
<div class="section" id="a-typical-decision-tree-with-its-pertinent-jargon-classification-problem">
<h2>A typical Decision Tree with its pertinent Jargon, Classification Problem<a class="headerlink" href="#a-typical-decision-tree-with-its-pertinent-jargon-classification-problem" title="Permalink to this headline"></a></h2>
<!-- dom:FIGURE: [DataFiles/cancer.png, width=600 frac=0.8] -->
<!-- begin figure -->
<p><img src="DataFiles/cancer.png" width="600"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<p>This tree was produced using the Wisconsin cancer data (discussed here as well, see code examples below) using <strong>Scikit-Learn</strong>’s decision tree classifier. Here we have used the so-called <strong>gini</strong> index (see below) to split the various branches.</p>
</div>
<div class="section" id="general-features">
<h2>General Features<a class="headerlink" href="#general-features" title="Permalink to this headline"></a></h2>
<p>The overarching approach to decision trees is a top-down approach.</p>
<ul class="simple">
<li><p>A leaf provides the classification of a given instance.</p></li>
<li><p>A node specifies a test of some attribute of the instance.</p></li>
<li><p>A branch corresponds to a possible values of an attribute.</p></li>
<li><p>An instance is classified by starting at the root node of the tree, testing the attribute specified by this node, then moving down the tree branch corresponding to the value of the attribute in the given example.</p></li>
</ul>
<p>This process is then repeated for the subtree rooted at the new
node.</p>
</div>
<div class="section" id="how-do-we-set-it-up">
<h2>How do we set it up?<a class="headerlink" href="#how-do-we-set-it-up" title="Permalink to this headline"></a></h2>
<p>In simplified terms, the process of training a decision tree and
predicting the target features of query instances is as follows:</p>
<ol class="simple">
<li><p>Present a dataset containing of a number of training instances characterized by a number of descriptive features and a target feature</p></li>
<li><p>Train the decision tree model by continuously splitting the target feature along the values of the descriptive features using a measure of information gain during the training process</p></li>
<li><p>Grow the tree until we accomplish a stopping criteria create leaf nodes which represent the <em>predictions</em> we want to make for new query instances</p></li>
<li><p>Show query instances to the tree and run down the tree until we arrive at leaf nodes</p></li>
</ol>
<p>Then we are essentially done!</p>
</div>
<div class="section" id="decision-trees-and-regression">
<h2>Decision trees and Regression<a class="headerlink" href="#decision-trees-and-regression" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%</span><span class="k">matplotlib</span> inline
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">PolynomialFeatures</span>
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LinearRegression</span>
<span class="n">steps</span><span class="o">=</span><span class="mi">250</span>
<span class="n">distance</span><span class="o">=</span><span class="mi">0</span>
<span class="n">x</span><span class="o">=</span><span class="mi">0</span>
<span class="n">distance_list</span><span class="o">=</span><span class="p">[]</span>
<span class="n">steps_list</span><span class="o">=</span><span class="p">[]</span>
<span class="k">while</span> <span class="n">x</span><span class="o">&lt;</span><span class="n">steps</span><span class="p">:</span>
<span class="n">distance</span><span class="o">+=</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
<span class="n">distance_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">distance</span><span class="p">)</span>
<span class="n">x</span><span class="o">+=</span><span class="mi">1</span>
<span class="n">steps_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">steps_list</span><span class="p">,</span><span class="n">distance_list</span><span class="p">,</span> <span class="n">color</span><span class="o">=</span><span class="s1">&#39;green&#39;</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s2">&quot;Random Walk Data&quot;</span><span class="p">)</span>
<span class="n">steps_list</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">asarray</span><span class="p">(</span><span class="n">steps_list</span><span class="p">)</span>
<span class="n">distance_list</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">asarray</span><span class="p">(</span><span class="n">distance_list</span><span class="p">)</span>
<span class="n">X</span><span class="o">=</span><span class="n">steps_list</span><span class="p">[:,</span><span class="n">np</span><span class="o">.</span><span class="n">newaxis</span><span class="p">]</span>
<span class="c1">#Polynomial fits</span>
<span class="c1">#Degree 2</span>
<span class="n">poly_features</span><span class="o">=</span><span class="n">PolynomialFeatures</span><span class="p">(</span><span class="n">degree</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">include_bias</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">X_poly</span><span class="o">=</span><span class="n">poly_features</span><span class="o">.</span><span class="n">fit_transform</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="n">lin_reg</span><span class="o">=</span><span class="n">LinearRegression</span><span class="p">()</span>
<span class="n">poly_fit</span><span class="o">=</span><span class="n">lin_reg</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_poly</span><span class="p">,</span><span class="n">distance_list</span><span class="p">)</span>
<span class="n">b</span><span class="o">=</span><span class="n">lin_reg</span><span class="o">.</span><span class="n">coef_</span>
<span class="n">c</span><span class="o">=</span><span class="n">lin_reg</span><span class="o">.</span><span class="n">intercept_</span>
<span class="nb">print</span> <span class="p">(</span><span class="s2">&quot;2nd degree coefficients:&quot;</span><span class="p">)</span>
<span class="nb">print</span> <span class="p">(</span><span class="s2">&quot;zero power: &quot;</span><span class="p">,</span><span class="n">c</span><span class="p">)</span>
<span class="nb">print</span> <span class="p">(</span><span class="s2">&quot;first power: &quot;</span><span class="p">,</span> <span class="n">b</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="nb">print</span> <span class="p">(</span><span class="s2">&quot;second power: &quot;</span><span class="p">,</span><span class="n">b</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="n">z</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">steps</span><span class="p">,</span> <span class="mf">.01</span><span class="p">)</span>
<span class="n">z_mod</span><span class="o">=</span><span class="n">b</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">*</span><span class="n">z</span><span class="o">**</span><span class="mi">2</span><span class="o">+</span><span class="n">b</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">*</span><span class="n">z</span><span class="o">+</span><span class="n">c</span>
<span class="n">fit_mod</span><span class="o">=</span><span class="n">b</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">*</span><span class="n">X</span><span class="o">**</span><span class="mi">2</span><span class="o">+</span><span class="n">b</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">*</span><span class="n">X</span><span class="o">+</span><span class="n">c</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">z</span><span class="p">,</span> <span class="n">z_mod</span><span class="p">,</span> <span class="n">color</span><span class="o">=</span><span class="s1">&#39;r&#39;</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s2">&quot;2nd Degree Fit&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">&quot;Polynomial Regression&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s2">&quot;Steps&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s2">&quot;Distance&quot;</span><span class="p">)</span>
<span class="c1">#Degree 10</span>
<span class="n">poly_features10</span><span class="o">=</span><span class="n">PolynomialFeatures</span><span class="p">(</span><span class="n">degree</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">include_bias</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">X_poly10</span><span class="o">=</span><span class="n">poly_features10</span><span class="o">.</span><span class="n">fit_transform</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="n">poly_fit10</span><span class="o">=</span><span class="n">lin_reg</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_poly10</span><span class="p">,</span><span class="n">distance_list</span><span class="p">)</span>
<span class="n">y_plot</span><span class="o">=</span><span class="n">poly_fit10</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_poly10</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y_plot</span><span class="p">,</span> <span class="n">color</span><span class="o">=</span><span class="s1">&#39;black&#39;</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s2">&quot;10th Degree Fit&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c1">#Decision Tree Regression</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">DecisionTreeRegressor</span>
<span class="n">regr_1</span><span class="o">=</span><span class="n">DecisionTreeRegressor</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="n">regr_2</span><span class="o">=</span><span class="n">DecisionTreeRegressor</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
<span class="n">regr_3</span><span class="o">=</span><span class="n">DecisionTreeRegressor</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="mi">7</span><span class="p">)</span>
<span class="n">regr_1</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">distance_list</span><span class="p">)</span>
<span class="n">regr_2</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">distance_list</span><span class="p">)</span>
<span class="n">regr_3</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">distance_list</span><span class="p">)</span>
<span class="n">X_test</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">steps</span><span class="p">,</span> <span class="mf">0.01</span><span class="p">)[:,</span> <span class="n">np</span><span class="o">.</span><span class="n">newaxis</span><span class="p">]</span>
<span class="n">y_1</span> <span class="o">=</span> <span class="n">regr_1</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="n">y_2</span> <span class="o">=</span> <span class="n">regr_2</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="n">y_3</span><span class="o">=</span><span class="n">regr_3</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="c1"># Plot the results</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">scatter</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">distance_list</span><span class="p">,</span> <span class="n">s</span><span class="o">=</span><span class="mf">2.5</span><span class="p">,</span> <span class="n">c</span><span class="o">=</span><span class="s2">&quot;black&quot;</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s2">&quot;data&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">X_test</span><span class="p">,</span> <span class="n">y_1</span><span class="p">,</span> <span class="n">color</span><span class="o">=</span><span class="s2">&quot;red&quot;</span><span class="p">,</span>
<span class="n">label</span><span class="o">=</span><span class="s2">&quot;max_depth=2&quot;</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">X_test</span><span class="p">,</span> <span class="n">y_2</span><span class="p">,</span> <span class="n">color</span><span class="o">=</span><span class="s2">&quot;green&quot;</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s2">&quot;max_depth=5&quot;</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">X_test</span><span class="p">,</span> <span class="n">y_3</span><span class="p">,</span> <span class="n">color</span><span class="o">=</span><span class="s2">&quot;m&quot;</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s2">&quot;max_depth=7&quot;</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s2">&quot;Data&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s2">&quot;Darget&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">&quot;Decision Tree Regression&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>2nd degree coefficients:
zero power: -2.877168286047408
first power: -0.052836183775253814
second power: -1.4881344092288954e-05
</pre></div>
</div>
<img alt="_images/week46_9_1.png" src="_images/week46_9_1.png" />
<img alt="_images/week46_9_2.png" src="_images/week46_9_2.png" />
</div>
</div>
</div>
<div class="section" id="building-a-tree-regression">
<h2>Building a tree, regression<a class="headerlink" href="#building-a-tree-regression" title="Permalink to this headline"></a></h2>
<p>There are mainly two steps</p>
<ol class="simple">
<li><p>We split the predictor space (the set of possible values <span class="math notranslate nohighlight">\(x_1,x_2,\dots, x_p\)</span>) into <span class="math notranslate nohighlight">\(J\)</span> distinct and non-non-overlapping regions, <span class="math notranslate nohighlight">\(R_1,R_2,\dots,R_J\)</span>.</p></li>
<li><p>For every observation that falls into the region <span class="math notranslate nohighlight">\(R_j\)</span> , we make the same prediction, which is simply the mean of the response values for the training observations in <span class="math notranslate nohighlight">\(R_j\)</span>.</p></li>
</ol>
<p>How do we construct the regions <span class="math notranslate nohighlight">\(R_1,\dots,R_J\)</span>? In theory, the
regions could have any shape. However, we choose to divide the
predictor space into high-dimensional rectangles, or boxes, for
simplicity and for ease of interpretation of the resulting predictive
model. The goal is to find boxes <span class="math notranslate nohighlight">\(R_1,\dots,R_J\)</span> that minimize the
MSE, given by</p>
<div class="math notranslate nohighlight">
\[
\sum_{j=1}^J\sum_{i\in R_j}(y_i-\overline{y}_{R_j})^2,
\]</div>
<p>where <span class="math notranslate nohighlight">\(\overline{y}_{R_j}\)</span> is the mean response for the training observations
within box <span class="math notranslate nohighlight">\(j\)</span>.</p>
</div>
<div class="section" id="a-top-down-approach-recursive-binary-splitting">
<h2>A top-down approach, recursive binary splitting<a class="headerlink" href="#a-top-down-approach-recursive-binary-splitting" title="Permalink to this headline"></a></h2>
<p>Unfortunately, it is computationally infeasible to consider every
possible partition of the feature space into <span class="math notranslate nohighlight">\(J\)</span> boxes. The common
strategy is to take a top-down approach</p>
<p>The approach is top-down because it begins at the top of the tree (all
observations belong to a single region) and then successively splits
the predictor space; each split is indicated via two new branches
further down on the tree. It is greedy because at each step of the
tree-building process, the best split is made at that particular step,
rather than looking ahead and picking a split that will lead to a
better tree in some future step.</p>
</div>
<div class="section" id="making-a-tree">
<h2>Making a tree<a class="headerlink" href="#making-a-tree" title="Permalink to this headline"></a></h2>
<p>In order to implement the recursive binary splitting we start by selecting
the predictor <span class="math notranslate nohighlight">\(x_j\)</span> and a cutpoint <span class="math notranslate nohighlight">\(s\)</span> that splits the predictor space into two regions <span class="math notranslate nohighlight">\(R_1\)</span> and <span class="math notranslate nohighlight">\(R_2\)</span></p>
<div class="math notranslate nohighlight">
\[
\left\{X\vert x_j &lt; s\right\},
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\left\{X\vert x_j \geq s\right\},
\]</div>
<p>so that we obtain the lowest MSE, that is</p>
<div class="math notranslate nohighlight">
\[
\sum_{i:x_i\in R_j}(y_i-\overline{y}_{R_1})^2+\sum_{i:x_i\in R_2}(y_i-\overline{y}_{R_2})^2,
\]</div>
<p>which we want to minimize by considering all predictors
<span class="math notranslate nohighlight">\(x_1,x_2,\dots,x_p\)</span>. We consider also all possible values of <span class="math notranslate nohighlight">\(s\)</span> for
each predictor. These values could be determined by randomly assigned
numbers or by starting at the midpoint and then proceed till we find
an optimal value.</p>
<p>For any <span class="math notranslate nohighlight">\(j\)</span> and <span class="math notranslate nohighlight">\(s\)</span>, we define the pair of half-planes where
<span class="math notranslate nohighlight">\(\overline{y}_{R_1}\)</span> is the mean response for the training
observations in <span class="math notranslate nohighlight">\(R_1(j,s)\)</span>, and <span class="math notranslate nohighlight">\(\overline{y}_{R_2}\)</span> is the mean
response for the training observations in <span class="math notranslate nohighlight">\(R_2(j,s)\)</span>.</p>
<p>Finding the values of <span class="math notranslate nohighlight">\(j\)</span> and <span class="math notranslate nohighlight">\(s\)</span> that minimize the above equation can be
done quite quickly, especially when the number of features <span class="math notranslate nohighlight">\(p\)</span> is not
too large.</p>
<p>Next, we repeat the process, looking
for the best predictor and best cutpoint in order to split the data
further so as to minimize the MSE within each of the resulting
regions. However, this time, instead of splitting the entire predictor
space, we split one of the two previously identified regions. We now
have three regions. Again, we look to split one of these three regions
further, so as to minimize the MSE. The process continues until a
stopping criterion is reached; for instance, we may continue until no
region contains more than five observations.</p>
</div>
<div class="section" id="pruning-the-tree">
<h2>Pruning the tree<a class="headerlink" href="#pruning-the-tree" title="Permalink to this headline"></a></h2>
<p>The above procedure is rather straightforward, but leads often to
overfitting and unnecessarily large and complicated trees. The basic
idea is to grow a large tree <span class="math notranslate nohighlight">\(T_0\)</span> and then prune it back in order to
obtain a subtree. A smaller tree with fewer splits (fewer regions) can
lead to smaller variance and better interpretation at the cost of a
little more bias.</p>
<p>The so-called Cost complexity pruning algorithm gives us a
way to do just this. Rather than considering every possible subtree,
we consider a sequence of trees indexed by a nonnegative tuning
parameter <span class="math notranslate nohighlight">\(\alpha\)</span>.</p>
<p>Read more at the following <a class="reference external" href="https://scikit-learn.org/stable/auto_examples/tree/plot_cost_complexity_pruning.html#sphx-glr-auto-examples-tree-plot-cost-complexity-pruning-py">Scikit-Learn link on pruning</a>.</p>
</div>
<div class="section" id="cost-complexity-pruning">
<h2>Cost complexity pruning<a class="headerlink" href="#cost-complexity-pruning" title="Permalink to this headline"></a></h2>
<p>For each value of <span class="math notranslate nohighlight">\(\alpha\)</span> there corresponds a subtree <span class="math notranslate nohighlight">\(T \in T_0\)</span> such that</p>
<div class="math notranslate nohighlight">
\[
\sum_{m=1}^{\overline{T}}\sum_{i:x_i\in R_m}(y_i-\overline{y}_{R_m})^2+\alpha\overline{T},
\]</div>
<p>is as small as possible. Here <span class="math notranslate nohighlight">\(\overline{T}\)</span> is
the number of terminal nodes of the tree <span class="math notranslate nohighlight">\(T\)</span> , <span class="math notranslate nohighlight">\(R_m\)</span> is the
rectangle (i.e. the subset of predictor space) corresponding to the <span class="math notranslate nohighlight">\(m\)</span>-th terminal node.</p>
<p>The tuning parameter <span class="math notranslate nohighlight">\(\alpha\)</span> controls a trade-off between the subtree’s
complexity and its fit to the training data. When <span class="math notranslate nohighlight">\(\alpha = 0\)</span>, then the
subtree <span class="math notranslate nohighlight">\(T\)</span> will simply equal <span class="math notranslate nohighlight">\(T_0\)</span>,
because then the above equation just measures the
training error.
However, as <span class="math notranslate nohighlight">\(\alpha\)</span> increases, there is a price to pay for
having a tree with many terminal nodes. The above equation will
tend to be minimized for a smaller subtree.</p>
<p>It turns out that as we increase <span class="math notranslate nohighlight">\(\alpha\)</span> from zero
branches get pruned from the tree in a nested and predictable fashion,
so obtaining the whole sequence of subtrees as a function of <span class="math notranslate nohighlight">\(\alpha\)</span> is
easy. We can select a value of <span class="math notranslate nohighlight">\(\alpha\)</span> using a validation set or using
cross-validation. We then return to the full data set and obtain the
subtree corresponding to <span class="math notranslate nohighlight">\(\alpha\)</span>.</p>
</div>
<div class="section" id="schematic-regression-procedure">
<h2>Schematic Regression Procedure<a class="headerlink" href="#schematic-regression-procedure" title="Permalink to this headline"></a></h2>
<p><strong>Building a Regression Tree.</strong></p>
<ol class="simple">
<li><p>Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.</p></li>
<li><p>Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of <span class="math notranslate nohighlight">\(\alpha\)</span>.</p></li>
<li><p>Use for example <span class="math notranslate nohighlight">\(K\)</span>-fold cross-validation to choose <span class="math notranslate nohighlight">\(\alpha\)</span>. Divide the training observations into <span class="math notranslate nohighlight">\(K\)</span> folds. For each <span class="math notranslate nohighlight">\(k=1,2,\dots,K\)</span> we:</p></li>
</ol>
<ul class="simple">
<li><p>repeat steps 1 and 2 on all but the <span class="math notranslate nohighlight">\(k\)</span>-th fold of the training data.</p></li>
<li><p>Then we valuate the mean squared prediction error on the data in the left-out <span class="math notranslate nohighlight">\(k\)</span>-th fold, as a function of <span class="math notranslate nohighlight">\(\alpha\)</span>.</p></li>
<li><p>Finally we average the results for each value of <span class="math notranslate nohighlight">\(\alpha\)</span>, and pick <span class="math notranslate nohighlight">\(\alpha\)</span> to minimize the average error.</p></li>
</ul>
<ol class="simple">
<li><p>Return the subtree from Step 2 that corresponds to the chosen value of <span class="math notranslate nohighlight">\(\alpha\)</span>.</p></li>
</ol>
</div>
<div class="section" id="a-classification-tree">
<h2>A Classification Tree<a class="headerlink" href="#a-classification-tree" title="Permalink to this headline"></a></h2>
<p>A classification tree is very similar to a regression tree, except
that it is used to predict a qualitative response rather than a
quantitative one. Recall that for a regression tree, the predicted
response for an observation is given by the mean response of the
training observations that belong to the same terminal node. In
contrast, for a classification tree, we predict that each observation
belongs to the most commonly occurring class of training observations
in the region to which it belongs. In interpreting the results of a
classification tree, we are often interested not only in the class
prediction corresponding to a particular terminal node region, but
also in the class proportions among the training observations that
fall into that region.</p>
</div>
<div class="section" id="growing-a-classification-tree">
<h2>Growing a classification tree<a class="headerlink" href="#growing-a-classification-tree" title="Permalink to this headline"></a></h2>
<p>The task of growing a
classification tree is quite similar to the task of growing a
regression tree. Just as in the regression setting, we use recursive
binary splitting to grow a classification tree. However, in the
classification setting, the MSE cannot be used as a criterion for making
the binary splits. A natural alternative to MSE is the <strong>classification
error rate</strong>. Since we plan to assign an observation in a given region
to the most commonly occurring error rate class of training
observations in that region, the classification error rate is simply
the fraction of the training observations in that region that do not
belong to the most common class.</p>
<p>When building a classification tree, either the Gini index or the
entropy are typically used to evaluate the quality of a particular
split, since these two approaches are more sensitive to node purity
than is the classification error rate.</p>
</div>
<div class="section" id="classification-tree-how-to-split-nodes">
<h2>Classification tree, how to split nodes<a class="headerlink" href="#classification-tree-how-to-split-nodes" title="Permalink to this headline"></a></h2>
<p>If our targets are the outcome of a classification process that takes
for example <span class="math notranslate nohighlight">\(k=1,2,\dots,K\)</span> values, the only thing we need to think of
is to set up the splitting criteria for each node.</p>
<p>We define a PDF <span class="math notranslate nohighlight">\(p_{mk}\)</span> that represents the number of observations of
a class <span class="math notranslate nohighlight">\(k\)</span> in a region <span class="math notranslate nohighlight">\(R_m\)</span> with <span class="math notranslate nohighlight">\(N_m\)</span> observations. We represent
this likelihood function in terms of the proportion <span class="math notranslate nohighlight">\(I(y_i=k)\)</span> of
observations of this class in the region <span class="math notranslate nohighlight">\(R_m\)</span> as</p>
<div class="math notranslate nohighlight">
\[
p_{mk} = \frac{1}{N_m}\sum_{x_i\in R_m}I(y_i=k).
\]</div>
<p>We let <span class="math notranslate nohighlight">\(p_{mk}\)</span> represent the majority class of observations in region
<span class="math notranslate nohighlight">\(m\)</span>. The three most common ways of splitting a node are given by</p>
<ul class="simple">
<li><p>Misclassification error</p></li>
</ul>
<div class="math notranslate nohighlight">
\[
p_{mk} = \frac{1}{N_m}\sum_{x_i\in R_m}I(y_i\ne k) = 1-p_{mk}.
\]</div>
<ul class="simple">
<li><p>Gini index <span class="math notranslate nohighlight">\(g\)</span></p></li>
</ul>
<div class="math notranslate nohighlight">
\[
g = \sum_{k=1}^K p_{mk}(1-p_{mk}).
\]</div>
<ul class="simple">
<li><p>Information entropy or just entropy <span class="math notranslate nohighlight">\(s\)</span></p></li>
</ul>
<div class="math notranslate nohighlight">
\[
s = -\sum_{k=1}^K p_{mk}\log{p_{mk}}.
\]</div>
</div>
<div class="section" id="visualizing-the-tree-classification">
<h2>Visualizing the Tree, Classification<a class="headerlink" href="#visualizing-the-tree-classification" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="kn">import</span> <span class="n">load_breast_cancer</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">DecisionTreeClassifier</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.metrics</span> <span class="kn">import</span> <span class="n">confusion_matrix</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">export_graphviz</span>
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">Image</span>
<span class="kn">from</span> <span class="nn">pydot</span> <span class="kn">import</span> <span class="n">graph_from_dot_data</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="n">cancer</span> <span class="o">=</span> <span class="n">load_breast_cancer</span><span class="p">()</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">cancer</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="n">cancer</span><span class="o">.</span><span class="n">feature_names</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="o">.</span><span class="n">from_codes</span><span class="p">(</span><span class="n">cancer</span><span class="o">.</span><span class="n">target</span><span class="p">,</span> <span class="n">cancer</span><span class="o">.</span><span class="n">target_names</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">get_dummies</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="n">tree_clf</span> <span class="o">=</span> <span class="n">DecisionTreeClassifier</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
<span class="n">tree_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="n">export_graphviz</span><span class="p">(</span>
<span class="n">tree_clf</span><span class="p">,</span>
<span class="n">out_file</span><span class="o">=</span><span class="s2">&quot;DataFiles/cancer.dot&quot;</span><span class="p">,</span>
<span class="n">feature_names</span><span class="o">=</span><span class="n">cancer</span><span class="o">.</span><span class="n">feature_names</span><span class="p">,</span>
<span class="n">class_names</span><span class="o">=</span><span class="n">cancer</span><span class="o">.</span><span class="n">target_names</span><span class="p">,</span>
<span class="n">rounded</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">filled</span><span class="o">=</span><span class="kc">True</span>
<span class="p">)</span>
<span class="n">cmd</span> <span class="o">=</span> <span class="s1">&#39;dot -Tpng DataFiles/cancer.dot -o DataFiles/cancer.png&#39;</span>
<span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> mean radius mean texture mean perimeter mean area mean smoothness \
0 17.99 10.38 122.80 1001.0 0.11840
1 20.57 17.77 132.90 1326.0 0.08474
2 19.69 21.25 130.00 1203.0 0.10960
3 11.42 20.38 77.58 386.1 0.14250
4 20.29 14.34 135.10 1297.0 0.10030
.. ... ... ... ... ...
564 21.56 22.39 142.00 1479.0 0.11100
565 20.13 28.25 131.20 1261.0 0.09780
566 16.60 28.08 108.30 858.1 0.08455
567 20.60 29.33 140.10 1265.0 0.11780
568 7.76 24.54 47.92 181.0 0.05263
mean compactness mean concavity mean concave points mean symmetry \
0 0.27760 0.30010 0.14710 0.2419
1 0.07864 0.08690 0.07017 0.1812
2 0.15990 0.19740 0.12790 0.2069
3 0.28390 0.24140 0.10520 0.2597
4 0.13280 0.19800 0.10430 0.1809
.. ... ... ... ...
564 0.11590 0.24390 0.13890 0.1726
565 0.10340 0.14400 0.09791 0.1752
566 0.10230 0.09251 0.05302 0.1590
567 0.27700 0.35140 0.15200 0.2397
568 0.04362 0.00000 0.00000 0.1587
mean fractal dimension ... worst radius worst texture \
0 0.07871 ... 25.380 17.33
1 0.05667 ... 24.990 23.41
2 0.05999 ... 23.570 25.53
3 0.09744 ... 14.910 26.50
4 0.05883 ... 22.540 16.67
.. ... ... ... ...
564 0.05623 ... 25.450 26.40
565 0.05533 ... 23.690 38.25
566 0.05648 ... 18.980 34.12
567 0.07016 ... 25.740 39.42
568 0.05884 ... 9.456 30.37
worst perimeter worst area worst smoothness worst compactness \
0 184.60 2019.0 0.16220 0.66560
1 158.80 1956.0 0.12380 0.18660
2 152.50 1709.0 0.14440 0.42450
3 98.87 567.7 0.20980 0.86630
4 152.20 1575.0 0.13740 0.20500
.. ... ... ... ...
564 166.10 2027.0 0.14100 0.21130
565 155.00 1731.0 0.11660 0.19220
566 126.70 1124.0 0.11390 0.30940
567 184.60 1821.0 0.16500 0.86810
568 59.16 268.6 0.08996 0.06444
worst concavity worst concave points worst symmetry \
0 0.7119 0.2654 0.4601
1 0.2416 0.1860 0.2750
2 0.4504 0.2430 0.3613
3 0.6869 0.2575 0.6638
4 0.4000 0.1625 0.2364
.. ... ... ...
564 0.4107 0.2216 0.2060
565 0.3215 0.1628 0.2572
566 0.3403 0.1418 0.2218
567 0.9387 0.2650 0.4087
568 0.0000 0.0000 0.2871
worst fractal dimension
0 0.11890
1 0.08902
2 0.08758
3 0.17300
4 0.07678
.. ...
564 0.07115
565 0.06637
566 0.07820
567 0.12400
568 0.07039
[569 rows x 30 columns]
malignant benign
0 1 0
1 1 0
2 1 0
3 1 0
4 1 0
.. ... ...
564 1 0
565 1 0
566 1 0
567 1 0
568 0 1
[569 rows x 2 columns]
</pre></div>
</div>
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="visualizing-the-tree-the-moons">
<h2>Visualizing the Tree, The Moons<a class="headerlink" href="#visualizing-the-tree-the-moons" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Common imports</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">DecisionTreeClassifier</span>
<span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="kn">import</span> <span class="n">make_moons</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">export_graphviz</span>
<span class="kn">from</span> <span class="nn">pydot</span> <span class="kn">import</span> <span class="n">graph_from_dot_data</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span>
<span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">make_moons</span><span class="p">(</span><span class="n">n_samples</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">noise</span><span class="o">=</span><span class="mf">0.25</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">53</span><span class="p">)</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span><span class="n">y</span><span class="p">,</span><span class="n">random_state</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="n">tree_clf</span> <span class="o">=</span> <span class="n">DecisionTreeClassifier</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
<span class="n">tree_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="n">export_graphviz</span><span class="p">(</span>
<span class="n">tree_clf</span><span class="p">,</span>
<span class="n">out_file</span><span class="o">=</span><span class="s2">&quot;DataFiles/moons.dot&quot;</span><span class="p">,</span>
<span class="n">rounded</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">filled</span><span class="o">=</span><span class="kc">True</span>
<span class="p">)</span>
<span class="n">cmd</span> <span class="o">=</span> <span class="s1">&#39;dot -Tpng DataFiles/moons.dot -o DataFiles/moons.png&#39;</span>
<span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="other-ways-of-visualizing-the-trees">
<h2>Other ways of visualizing the trees<a class="headerlink" href="#other-ways-of-visualizing-the-trees" title="Permalink to this headline"></a></h2>
<p><strong>Scikit-Learn</strong> has also another way to visualize the trees which is very useful, here with the Iris data.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="kn">import</span> <span class="n">load_iris</span>
<span class="kn">from</span> <span class="nn">sklearn</span> <span class="kn">import</span> <span class="n">tree</span>
<span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">load_iris</span><span class="p">(</span><span class="n">return_X_y</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">tree_clf</span> <span class="o">=</span> <span class="n">tree</span><span class="o">.</span><span class="n">DecisionTreeClassifier</span><span class="p">()</span>
<span class="n">tree_clf</span> <span class="o">=</span> <span class="n">tree_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="c1"># and then plot the tree</span>
<span class="n">tree</span><span class="o">.</span><span class="n">plot_tree</span><span class="p">(</span><span class="n">tree_clf</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[Text(0.5, 0.9166666666666666, &#39;X[2] &lt;= 2.45\ngini = 0.667\nsamples = 150\nvalue = [50, 50, 50]&#39;),
Text(0.4230769230769231, 0.75, &#39;gini = 0.0\nsamples = 50\nvalue = [50, 0, 0]&#39;),
Text(0.5769230769230769, 0.75, &#39;X[3] &lt;= 1.75\ngini = 0.5\nsamples = 100\nvalue = [0, 50, 50]&#39;),
Text(0.3076923076923077, 0.5833333333333334, &#39;X[2] &lt;= 4.95\ngini = 0.168\nsamples = 54\nvalue = [0, 49, 5]&#39;),
Text(0.15384615384615385, 0.4166666666666667, &#39;X[3] &lt;= 1.65\ngini = 0.041\nsamples = 48\nvalue = [0, 47, 1]&#39;),
Text(0.07692307692307693, 0.25, &#39;gini = 0.0\nsamples = 47\nvalue = [0, 47, 0]&#39;),
Text(0.23076923076923078, 0.25, &#39;gini = 0.0\nsamples = 1\nvalue = [0, 0, 1]&#39;),
Text(0.46153846153846156, 0.4166666666666667, &#39;X[3] &lt;= 1.55\ngini = 0.444\nsamples = 6\nvalue = [0, 2, 4]&#39;),
Text(0.38461538461538464, 0.25, &#39;gini = 0.0\nsamples = 3\nvalue = [0, 0, 3]&#39;),
Text(0.5384615384615384, 0.25, &#39;X[2] &lt;= 5.45\ngini = 0.444\nsamples = 3\nvalue = [0, 2, 1]&#39;),
Text(0.46153846153846156, 0.08333333333333333, &#39;gini = 0.0\nsamples = 2\nvalue = [0, 2, 0]&#39;),
Text(0.6153846153846154, 0.08333333333333333, &#39;gini = 0.0\nsamples = 1\nvalue = [0, 0, 1]&#39;),
Text(0.8461538461538461, 0.5833333333333334, &#39;X[2] &lt;= 4.85\ngini = 0.043\nsamples = 46\nvalue = [0, 1, 45]&#39;),
Text(0.7692307692307693, 0.4166666666666667, &#39;X[1] &lt;= 3.1\ngini = 0.444\nsamples = 3\nvalue = [0, 1, 2]&#39;),
Text(0.6923076923076923, 0.25, &#39;gini = 0.0\nsamples = 2\nvalue = [0, 0, 2]&#39;),
Text(0.8461538461538461, 0.25, &#39;gini = 0.0\nsamples = 1\nvalue = [0, 1, 0]&#39;),
Text(0.9230769230769231, 0.4166666666666667, &#39;gini = 0.0\nsamples = 43\nvalue = [0, 0, 43]&#39;)]
</pre></div>
</div>
<img alt="_images/week46_41_1.png" src="_images/week46_41_1.png" />
</div>
</div>
</div>
<div class="section" id="printing-out-as-text">
<h2>Printing out as text<a class="headerlink" href="#printing-out-as-text" title="Permalink to this headline"></a></h2>
<p>Alternatively, the tree can also be exported in textual format with the function exporttext.
This method doesn’t require the installation of external libraries and is more compact:</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="kn">import</span> <span class="n">load_iris</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">DecisionTreeClassifier</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">export_text</span>
<span class="n">iris</span> <span class="o">=</span> <span class="n">load_iris</span><span class="p">()</span>
<span class="n">decision_tree</span> <span class="o">=</span> <span class="n">DecisionTreeClassifier</span><span class="p">(</span><span class="n">random_state</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">max_depth</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="n">decision_tree</span> <span class="o">=</span> <span class="n">decision_tree</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">iris</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="n">iris</span><span class="o">.</span><span class="n">target</span><span class="p">)</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">export_text</span><span class="p">(</span><span class="n">decision_tree</span><span class="p">,</span> <span class="n">feature_names</span><span class="o">=</span><span class="n">iris</span><span class="p">[</span><span class="s1">&#39;feature_names&#39;</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="n">r</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>|--- petal width (cm) &lt;= 0.80
| |--- class: 0
|--- petal width (cm) &gt; 0.80
| |--- petal width (cm) &lt;= 1.75
| | |--- class: 1
| |--- petal width (cm) &gt; 1.75
| | |--- class: 2
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="algorithms-for-setting-up-decision-trees">
<h2>Algorithms for Setting up Decision Trees<a class="headerlink" href="#algorithms-for-setting-up-decision-trees" title="Permalink to this headline"></a></h2>
<p>Two algorithms stand out in the set up of decision trees:</p>
<ol class="simple">
<li><p>The CART (Classification And Regression Tree) algorithm for both classification and regression</p></li>
<li><p>The ID3 algorithm based on the computation of the information gain for classification</p></li>
</ol>
<p>We discuss both algorithms with applications here. The popular library
<strong>Scikit-Learn</strong> uses the CART algorithm. For classification problems
you can use either the <strong>gini</strong> index or the <strong>entropy</strong> to split a tree
in two branches.</p>
</div>
<div class="section" id="the-cart-algorithm-for-classification">
<h2>The CART algorithm for Classification<a class="headerlink" href="#the-cart-algorithm-for-classification" title="Permalink to this headline"></a></h2>
<p>For classification, the CART algorithm splits the data set in two subsets using a single feature <span class="math notranslate nohighlight">\(k\)</span> and a threshold <span class="math notranslate nohighlight">\(t_k\)</span>.
This could be for example a threshold set by a number below a certain circumference of a malign tumor.</p>
<p>How do we find these two quantities?
We search for the pair <span class="math notranslate nohighlight">\((k,t_k)\)</span> that produces the purest subset using for example the <strong>gini</strong> factor <span class="math notranslate nohighlight">\(G\)</span>.
The cost function it tries to minimize is then</p>
<div class="math notranslate nohighlight">
\[
C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}},
\]</div>
<p>where <span class="math notranslate nohighlight">\(G_{\mathrm{left/right}}\)</span> measures the impurity of the left/right subset and <span class="math notranslate nohighlight">\(m_{\mathrm{left/right}}\)</span>
is the number of instances in the left/right subset</p>
<p>Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets
and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the
<span class="math notranslate nohighlight">\(max\_depth\)</span> hyperparameter), or if it cannot find a split that will reduce impurity. A few other
hyperparameters control additional stopping conditions such as the <span class="math notranslate nohighlight">\(min\_samples\_split\)</span>,
<span class="math notranslate nohighlight">\(min\_samples\_leaf\)</span>, <span class="math notranslate nohighlight">\(min\_weight\_fraction\_leaf\)</span>, and <span class="math notranslate nohighlight">\(max\_leaf\_nodes\)</span>.</p>
</div>
<div class="section" id="the-cart-algorithm-for-regression">
<h2>The CART algorithm for Regression<a class="headerlink" href="#the-cart-algorithm-for-regression" title="Permalink to this headline"></a></h2>
<p>The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the
training set in a way that minimizes say the <strong>gini</strong> or <strong>entropy</strong> impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now</p>
<div class="math notranslate nohighlight">
\[
C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}.
\]</div>
<p>Here the MSE for a specific node is defined as</p>
<div class="math notranslate nohighlight">
\[
\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2,
\]</div>
<p>with</p>
<div class="math notranslate nohighlight">
\[
\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i,
\]</div>
<p>the mean value of all observations in a specific node.</p>
<p>Without any regularization, the regression task for decision trees,
just like for classification tasks, is prone to overfitting.</p>
</div>
<div class="section" id="why-binary-splits">
<h2>Why binary splits?<a class="headerlink" href="#why-binary-splits" title="Permalink to this headline"></a></h2>
<p>It is custom to split to a tree uising binary splits. The reason is
that multiway splits fragment the data too quickly, leaving
insufficient data at the next level down. Multiway splits can be
achieved by a series of binary split and this is normally preferred.</p>
</div>
<div class="section" id="computing-a-tree-using-the-gini-index">
<h2>Computing a Tree using the Gini Index<a class="headerlink" href="#computing-a-tree-using-the-gini-index" title="Permalink to this headline"></a></h2>
<p>Consider the following example with attributes/features and two
possible outcomes (classes) for each attribute. Assume we wish to find some
correlations between the average grade of a student as function of the
number of hours studied and hours slept. We want also to correlate the
grade in a given course with the general trend, whether the students
recently has gotten grades below average or above.</p>
<p>We have three features/attributes</p>
<ol class="simple">
<li><p>Trend of average grades before present course, classified as either below or above the average grade of the whole class</p></li>
<li><p>The number of hours studies, classified again as either higher (more than 3 hours per day) or lower . Here we have used a standard for one <span class="math notranslate nohighlight">\(ECTS\)</span> which is scaled to 25-30 hours of work for a semester which lasts 18 weeks, with 15 weeks of lectures and 3 weeks for exams, assuming a total of 30 ECTS per semester.</p></li>
<li><p>The number of hours slept as high for more than <span class="math notranslate nohighlight">\(8\)</span> hours and below for less than 8 hours of sleep, classified again as either high or low</p></li>
<li><p>The final grade whether it is above or below average</p></li>
</ol>
</div>
<div class="section" id="the-table">
<h2>The Table<a class="headerlink" href="#the-table" title="Permalink to this headline"></a></h2>
<table class="dotable" border="1">
<thead>
<tr><th align="center">Grade Trend</th> <th align="center">Hours slept</th> <th align="center">Hours Studied</th> <th align="center">Grade</th> </tr>
</thead>
<tbody>
<tr><td align="center"> Above </td> <td align="center"> Low </td> <td align="center"> High </td> <td align="center"> Above </td> </tr>
<tr><td align="center"> Below </td> <td align="center"> High </td> <td align="center"> Low </td> <td align="center"> Below </td> </tr>
<tr><td align="center"> Above </td> <td align="center"> Low </td> <td align="center"> High </td> <td align="center"> Above </td> </tr>
<tr><td align="center"> Above </td> <td align="center"> High </td> <td align="center"> High </td> <td align="center"> Above </td> </tr>
<tr><td align="center"> Below </td> <td align="center"> Low </td> <td align="center"> High </td> <td align="center"> Below </td> </tr>
<tr><td align="center"> Above </td> <td align="center"> Low </td> <td align="center"> Low </td> <td align="center"> Below </td> </tr>
<tr><td align="center"> Below </td> <td align="center"> High </td> <td align="center"> High </td> <td align="center"> Below </td> </tr>
<tr><td align="center"> Below </td> <td align="center"> Low </td> <td align="center"> High </td> <td align="center"> Below </td> </tr>
<tr><td align="center"> Above </td> <td align="center"> Low </td> <td align="center"> Low </td> <td align="center"> Below </td> </tr>
<tr><td align="center"> Above </td> <td align="center"> High </td> <td align="center"> High </td> <td align="center"> Above </td> </tr>
</tbody>
</table></div>
<div class="section" id="computing-the-various-gini-indices">
<h2>Computing the various Gini Indices<a class="headerlink" href="#computing-the-various-gini-indices" title="Permalink to this headline"></a></h2>
<p>In computations we will translate all classes into numbers. Being
these binary classes, they can easily be split into ones and zeros.</p>
<p><strong>Gini index for Average trend.</strong></p>
<p><a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2022/NotesNov32022.pdf">See handwritten notes November 3</a></p>
</div>
<div class="section" id="computing-the-various-gini-indices-hours-slept">
<h2>Computing the various Gini Indices, Hours slept<a class="headerlink" href="#computing-the-various-gini-indices-hours-slept" title="Permalink to this headline"></a></h2>
<p><strong>Gini index for hour slept.</strong></p>
<p><a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2022/NotesNov32022.pdf">See handwritten notes November 3</a></p>
</div>
<div class="section" id="computing-the-various-gini-indices-hours-studied">
<h2>Computing the various Gini Indices, Hours studied<a class="headerlink" href="#computing-the-various-gini-indices-hours-studied" title="Permalink to this headline"></a></h2>
<p><strong>Gini index for hour studied.</strong></p>
<p><a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2022/NotesNov32022.pdf">See handwritten notes November 3</a></p>
<p>For final tree, see the above handwritten notes</p>
</div>
<div class="section" id="a-possible-code-using-scikit-learn">
<h2>A possible code using Scikit-Learn<a class="headerlink" href="#a-possible-code-using-scikit-learn" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Common imports</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">DecisionTreeClassifier</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">export_graphviz</span>
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">StandardScaler</span><span class="p">,</span> <span class="n">OneHotEncoder</span>
<span class="kn">from</span> <span class="nn">sklearn.compose</span> <span class="kn">import</span> <span class="n">ColumnTransformer</span>
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">Image</span>
<span class="kn">from</span> <span class="nn">pydot</span> <span class="kn">import</span> <span class="n">graph_from_dot_data</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="c1"># Where to save the figures and data files</span>
<span class="n">PROJECT_ROOT_DIR</span> <span class="o">=</span> <span class="s2">&quot;Results&quot;</span>
<span class="n">FIGURE_ID</span> <span class="o">=</span> <span class="s2">&quot;Results/FigureFiles&quot;</span>
<span class="n">DATA_ID</span> <span class="o">=</span> <span class="s2">&quot;DataFiles/&quot;</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">PROJECT_ROOT_DIR</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">PROJECT_ROOT_DIR</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">):</span>
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">,</span> <span class="n">fig_id</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">data_path</span><span class="p">(</span><span class="n">dat_id</span><span class="p">):</span>
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">,</span> <span class="n">dat_id</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">save_fig</span><span class="p">(</span><span class="n">fig_id</span><span class="p">):</span>
<span class="n">plt</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="n">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">)</span> <span class="o">+</span> <span class="s2">&quot;.png&quot;</span><span class="p">,</span> <span class="nb">format</span><span class="o">=</span><span class="s1">&#39;png&#39;</span><span class="p">)</span>
<span class="n">infile</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">data_path</span><span class="p">(</span><span class="s2">&quot;grades.csv&quot;</span><span class="p">),</span><span class="s1">&#39;r&#39;</span><span class="p">)</span>
<span class="c1"># Read the experimental data with Pandas</span>
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">display</span>
<span class="n">grades</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="n">infile</span><span class="p">)</span>
<span class="n">grades</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">grades</span><span class="p">)</span>
<span class="n">display</span><span class="p">(</span><span class="n">grades</span><span class="p">)</span>
<span class="c1"># Features and targets</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">grades</span><span class="o">.</span><span class="n">loc</span><span class="p">[:,</span> <span class="n">grades</span><span class="o">.</span><span class="n">columns</span> <span class="o">!=</span> <span class="s1">&#39;Grade&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">values</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">grades</span><span class="o">.</span><span class="n">loc</span><span class="p">[:,</span> <span class="n">grades</span><span class="o">.</span><span class="n">columns</span> <span class="o">==</span> <span class="s1">&#39;Grade&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">values</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="c1"># Then do a Classification tree</span>
<span class="n">tree_clf</span> <span class="o">=</span> <span class="n">DecisionTreeClassifier</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="n">tree_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Train set accuracy with Decision Tree: </span><span class="si">{:.2f}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">tree_clf</span><span class="o">.</span><span class="n">score</span><span class="p">(</span><span class="n">X</span><span class="p">,</span><span class="n">y</span><span class="p">)))</span>
<span class="c1">#transfer to a decision tree graph</span>
<span class="n">export_graphviz</span><span class="p">(</span>
<span class="n">tree_clf</span><span class="p">,</span>
<span class="n">out_file</span><span class="o">=</span><span class="s2">&quot;DataFiles/grade.dot&quot;</span><span class="p">,</span>
<span class="n">rounded</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">filled</span><span class="o">=</span><span class="kc">True</span>
<span class="p">)</span>
<span class="n">cmd</span> <span class="o">=</span> <span class="s1">&#39;dot -Tpng DataFiles/grade.dot -o DataFiles/grades.png&#39;</span>
<span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>Grade Trend</th>
<th>Hours slept</th>
<th>Hours Studied</th>
<th>Grade</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>1</td>
<td>0</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<th>1</th>
<td>0</td>
<td>1</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<th>2</th>
<td>1</td>
<td>0</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<th>3</th>
<td>1</td>
<td>1</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<th>4</th>
<td>0</td>
<td>0</td>
<td>1</td>
<td>0</td>
</tr>
<tr>
<th>5</th>
<td>1</td>
<td>0</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<th>6</th>
<td>0</td>
<td>1</td>
<td>1</td>
<td>0</td>
</tr>
<tr>
<th>7</th>
<td>0</td>
<td>0</td>
<td>1</td>
<td>0</td>
</tr>
<tr>
<th>8</th>
<td>1</td>
<td>0</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<th>9</th>
<td>1</td>
<td>1</td>
<td>1</td>
<td>1</td>
</tr>
</tbody>
</table>
</div></div><div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[1 0 1]
[0 1 0]
[1 0 1]
[1 1 1]
[0 0 1]
[1 0 0]
[0 1 1]
[0 0 1]
[1 0 0]
[1 1 1]]
Train set accuracy with Decision Tree: 1.00
</pre></div>
</div>
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="further-example-computing-the-gini-index">
<h2>Further example: Computing the Gini index<a class="headerlink" href="#further-example-computing-the-gini-index" title="Permalink to this headline"></a></h2>
<p>The next example we will look at is a classical one in many Machine
Learning applications. Based on various meteorological features, we
have several so-called attributes which decide whether we at the end
will do some outdoor activity like skiing, going for a bike ride etc
etc. The table here contains the feautures <strong>outlook</strong>, <strong>temperature</strong>,
<strong>humidity</strong> and <strong>wind</strong>. The target or output is whether we ride
(True=1) or whether we do something else that day (False=0). The
attributes for each feature are then sunny, overcast and rain for the
outlook, hot, cold and mild for temperature, high and normal for
humidity and weak and strong for wind.</p>
<p>The table here summarizes the various attributes and</p>
<table class="dotable" border="1">
<thead>
<tr><th align="center">Day</th> <th align="center">Outlook </th> <th align="center">Temperature</th> <th align="center">Humidity</th> <th align="center"> Wind </th> <th align="center">Ride</th> </tr>
</thead>
<tbody>
<tr><td align="center"> 1 </td> <td align="center"> Sunny </td> <td align="center"> Hot </td> <td align="center"> High </td> <td align="center"> Weak </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 2 </td> <td align="center"> Sunny </td> <td align="center"> Hot </td> <td align="center"> High </td> <td align="center"> Strong </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 3 </td> <td align="center"> Overcast </td> <td align="center"> Hot </td> <td align="center"> High </td> <td align="center"> Weak </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 4 </td> <td align="center"> Rain </td> <td align="center"> Mild </td> <td align="center"> High </td> <td align="center"> Weak </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 5 </td> <td align="center"> Rain </td> <td align="center"> Cool </td> <td align="center"> Normal </td> <td align="center"> Weak </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 6 </td> <td align="center"> Rain </td> <td align="center"> Cool </td> <td align="center"> Normal </td> <td align="center"> Strong </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 7 </td> <td align="center"> Overcast </td> <td align="center"> Cool </td> <td align="center"> Normal </td> <td align="center"> Strong </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 8 </td> <td align="center"> Sunny </td> <td align="center"> Mild </td> <td align="center"> High </td> <td align="center"> Weak </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 9 </td> <td align="center"> Sunny </td> <td align="center"> Cool </td> <td align="center"> Normal </td> <td align="center"> Weak </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 10 </td> <td align="center"> Rain </td> <td align="center"> Mild </td> <td align="center"> Normal </td> <td align="center"> Weak </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 11 </td> <td align="center"> Sunny </td> <td align="center"> Mild </td> <td align="center"> Normal </td> <td align="center"> Strong </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 12 </td> <td align="center"> Overcast </td> <td align="center"> Mild </td> <td align="center"> High </td> <td align="center"> Strong </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 13 </td> <td align="center"> Overcast </td> <td align="center"> Hot </td> <td align="center"> Normal </td> <td align="center"> Weak </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 14 </td> <td align="center"> Rain </td> <td align="center"> Mild </td> <td align="center"> High </td> <td align="center"> Strong </td> <td align="center"> 0 </td> </tr>
</tbody>
</table></div>
<div class="section" id="simple-python-code-to-read-in-data-and-perform-classification">
<h2>Simple Python Code to read in Data and perform Classification<a class="headerlink" href="#simple-python-code-to-read-in-data-and-perform-classification" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Common imports</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">DecisionTreeClassifier</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">export_graphviz</span>
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">StandardScaler</span><span class="p">,</span> <span class="n">OneHotEncoder</span>
<span class="kn">from</span> <span class="nn">sklearn.compose</span> <span class="kn">import</span> <span class="n">ColumnTransformer</span>
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">Image</span>
<span class="kn">from</span> <span class="nn">pydot</span> <span class="kn">import</span> <span class="n">graph_from_dot_data</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="c1"># Where to save the figures and data files</span>
<span class="n">PROJECT_ROOT_DIR</span> <span class="o">=</span> <span class="s2">&quot;Results&quot;</span>
<span class="n">FIGURE_ID</span> <span class="o">=</span> <span class="s2">&quot;Results/FigureFiles&quot;</span>
<span class="n">DATA_ID</span> <span class="o">=</span> <span class="s2">&quot;DataFiles/&quot;</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">PROJECT_ROOT_DIR</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">PROJECT_ROOT_DIR</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">):</span>
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">,</span> <span class="n">fig_id</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">data_path</span><span class="p">(</span><span class="n">dat_id</span><span class="p">):</span>
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">,</span> <span class="n">dat_id</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">save_fig</span><span class="p">(</span><span class="n">fig_id</span><span class="p">):</span>
<span class="n">plt</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="n">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">)</span> <span class="o">+</span> <span class="s2">&quot;.png&quot;</span><span class="p">,</span> <span class="nb">format</span><span class="o">=</span><span class="s1">&#39;png&#39;</span><span class="p">)</span>
<span class="n">infile</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">data_path</span><span class="p">(</span><span class="s2">&quot;rideclass.csv&quot;</span><span class="p">),</span><span class="s1">&#39;r&#39;</span><span class="p">)</span>
<span class="c1"># Read the experimental data with Pandas</span>
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">display</span>
<span class="n">ridedata</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="n">infile</span><span class="p">,</span><span class="n">names</span> <span class="o">=</span> <span class="p">(</span><span class="s1">&#39;Outlook&#39;</span><span class="p">,</span><span class="s1">&#39;Temperature&#39;</span><span class="p">,</span><span class="s1">&#39;Humidity&#39;</span><span class="p">,</span><span class="s1">&#39;Wind&#39;</span><span class="p">,</span><span class="s1">&#39;Ride&#39;</span><span class="p">))</span>
<span class="n">ridedata</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">ridedata</span><span class="p">)</span>
<span class="c1"># Features and targets</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">ridedata</span><span class="o">.</span><span class="n">loc</span><span class="p">[:,</span> <span class="n">ridedata</span><span class="o">.</span><span class="n">columns</span> <span class="o">!=</span> <span class="s1">&#39;Ride&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">values</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">ridedata</span><span class="o">.</span><span class="n">loc</span><span class="p">[:,</span> <span class="n">ridedata</span><span class="o">.</span><span class="n">columns</span> <span class="o">==</span> <span class="s1">&#39;Ride&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">values</span>
<span class="c1"># Create the encoder.</span>
<span class="n">encoder</span> <span class="o">=</span> <span class="n">OneHotEncoder</span><span class="p">(</span><span class="n">handle_unknown</span><span class="o">=</span><span class="s2">&quot;ignore&quot;</span><span class="p">)</span>
<span class="c1"># Assume for simplicity all features are categorical.</span>
<span class="n">encoder</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="c1"># Apply the encoder.</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">encoder</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="c1"># Then do a Classification tree</span>
<span class="n">tree_clf</span> <span class="o">=</span> <span class="n">DecisionTreeClassifier</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="n">tree_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Train set accuracy with Decision Tree: </span><span class="si">{:.2f}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">tree_clf</span><span class="o">.</span><span class="n">score</span><span class="p">(</span><span class="n">X</span><span class="p">,</span><span class="n">y</span><span class="p">)))</span>
<span class="c1">#transfer to a decision tree graph</span>
<span class="n">export_graphviz</span><span class="p">(</span>
<span class="n">tree_clf</span><span class="p">,</span>
<span class="n">out_file</span><span class="o">=</span><span class="s2">&quot;DataFiles/ride.dot&quot;</span><span class="p">,</span>
<span class="n">rounded</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">filled</span><span class="o">=</span><span class="kc">True</span>
<span class="p">)</span>
<span class="n">cmd</span> <span class="o">=</span> <span class="s1">&#39;dot -Tpng DataFiles/cancer.dot -o DataFiles/cancer.png&#39;</span>
<span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> (0, 0) 1.0
(0, 7) 1.0
(0, 9) 1.0
(0, 13) 1.0
(1, 3) 1.0
(1, 5) 1.0
(1, 8) 1.0
(1, 12) 1.0
(2, 3) 1.0
(2, 5) 1.0
(2, 8) 1.0
(2, 11) 1.0
(3, 1) 1.0
(3, 5) 1.0
(3, 8) 1.0
(3, 12) 1.0
(4, 2) 1.0
(4, 6) 1.0
(4, 8) 1.0
(4, 12) 1.0
(5, 2) 1.0
(5, 4) 1.0
(5, 10) 1.0
(5, 12) 1.0
(6, 2) 1.0
: :
(8, 12) 1.0
(9, 3) 1.0
(9, 4) 1.0
(9, 10) 1.0
(9, 12) 1.0
(10, 2) 1.0
(10, 6) 1.0
(10, 10) 1.0
(10, 12) 1.0
(11, 3) 1.0
(11, 6) 1.0
(11, 10) 1.0
(11, 11) 1.0
(12, 1) 1.0
(12, 6) 1.0
(12, 8) 1.0
(12, 11) 1.0
(13, 1) 1.0
(13, 5) 1.0
(13, 10) 1.0
(13, 12) 1.0
(14, 2) 1.0
(14, 6) 1.0
(14, 8) 1.0
(14, 11) 1.0
Train set accuracy with Decision Tree: 0.73
</pre></div>
</div>
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="computing-the-gini-factor">
<h2>Computing the Gini Factor<a class="headerlink" href="#computing-the-gini-factor" title="Permalink to this headline"></a></h2>
<p>The above functions (gini, entropy and misclassification error) are
important components of the so-called CART algorithm. We will discuss
this algorithm below after we have discussed the information gain
algorithm ID3.</p>
<p>In the example here we have converted all our attributes into numerical values <span class="math notranslate nohighlight">\(0,1,2\)</span> etc.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Split a dataset based on an attribute and an attribute value</span>
<span class="k">def</span> <span class="nf">test_split</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">dataset</span><span class="p">):</span>
<span class="n">left</span><span class="p">,</span> <span class="n">right</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(),</span> <span class="nb">list</span><span class="p">()</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">dataset</span><span class="p">:</span>
<span class="k">if</span> <span class="n">row</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> <span class="o">&lt;</span> <span class="n">value</span><span class="p">:</span>
<span class="n">left</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">right</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
<span class="k">return</span> <span class="n">left</span><span class="p">,</span> <span class="n">right</span>
<span class="c1"># Calculate the Gini index for a split dataset</span>
<span class="k">def</span> <span class="nf">gini_index</span><span class="p">(</span><span class="n">groups</span><span class="p">,</span> <span class="n">classes</span><span class="p">):</span>
<span class="c1"># count all samples at split point</span>
<span class="n">n_instances</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="nb">sum</span><span class="p">([</span><span class="nb">len</span><span class="p">(</span><span class="n">group</span><span class="p">)</span> <span class="k">for</span> <span class="n">group</span> <span class="ow">in</span> <span class="n">groups</span><span class="p">]))</span>
<span class="c1"># sum weighted Gini index for each group</span>
<span class="n">gini</span> <span class="o">=</span> <span class="mf">0.0</span>
<span class="k">for</span> <span class="n">group</span> <span class="ow">in</span> <span class="n">groups</span><span class="p">:</span>
<span class="n">size</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">group</span><span class="p">))</span>
<span class="c1"># avoid divide by zero</span>
<span class="k">if</span> <span class="n">size</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">score</span> <span class="o">=</span> <span class="mf">0.0</span>
<span class="c1"># score the group based on the score for each class</span>
<span class="k">for</span> <span class="n">class_val</span> <span class="ow">in</span> <span class="n">classes</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="p">[</span><span class="n">row</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">group</span><span class="p">]</span><span class="o">.</span><span class="n">count</span><span class="p">(</span><span class="n">class_val</span><span class="p">)</span> <span class="o">/</span> <span class="n">size</span>
<span class="n">score</span> <span class="o">+=</span> <span class="n">p</span> <span class="o">*</span> <span class="n">p</span>
<span class="c1"># weight the group score by its relative size</span>
<span class="n">gini</span> <span class="o">+=</span> <span class="p">(</span><span class="mf">1.0</span> <span class="o">-</span> <span class="n">score</span><span class="p">)</span> <span class="o">*</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="n">n_instances</span><span class="p">)</span>
<span class="k">return</span> <span class="n">gini</span>
<span class="c1"># Select the best split point for a dataset</span>
<span class="k">def</span> <span class="nf">get_split</span><span class="p">(</span><span class="n">dataset</span><span class="p">):</span>
<span class="n">class_values</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">dataset</span><span class="p">))</span>
<span class="n">b_index</span><span class="p">,</span> <span class="n">b_value</span><span class="p">,</span> <span class="n">b_score</span><span class="p">,</span> <span class="n">b_groups</span> <span class="o">=</span> <span class="mi">999</span><span class="p">,</span> <span class="mi">999</span><span class="p">,</span> <span class="mi">999</span><span class="p">,</span> <span class="kc">None</span>
<span class="k">for</span> <span class="n">index</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">dataset</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="o">-</span><span class="mi">1</span><span class="p">):</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">dataset</span><span class="p">:</span>
<span class="n">groups</span> <span class="o">=</span> <span class="n">test_split</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">row</span><span class="p">[</span><span class="n">index</span><span class="p">],</span> <span class="n">dataset</span><span class="p">)</span>
<span class="n">gini</span> <span class="o">=</span> <span class="n">gini_index</span><span class="p">(</span><span class="n">groups</span><span class="p">,</span> <span class="n">class_values</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;X</span><span class="si">%d</span><span class="s1"> &lt; </span><span class="si">%.3f</span><span class="s1"> Gini=</span><span class="si">%.3f</span><span class="s1">&#39;</span> <span class="o">%</span> <span class="p">((</span><span class="n">index</span><span class="o">+</span><span class="mi">1</span><span class="p">),</span> <span class="n">row</span><span class="p">[</span><span class="n">index</span><span class="p">],</span> <span class="n">gini</span><span class="p">))</span>
<span class="k">if</span> <span class="n">gini</span> <span class="o">&lt;</span> <span class="n">b_score</span><span class="p">:</span>
<span class="n">b_index</span><span class="p">,</span> <span class="n">b_value</span><span class="p">,</span> <span class="n">b_score</span><span class="p">,</span> <span class="n">b_groups</span> <span class="o">=</span> <span class="n">index</span><span class="p">,</span> <span class="n">row</span><span class="p">[</span><span class="n">index</span><span class="p">],</span> <span class="n">gini</span><span class="p">,</span> <span class="n">groups</span>
<span class="k">return</span> <span class="p">{</span><span class="s1">&#39;index&#39;</span><span class="p">:</span><span class="n">b_index</span><span class="p">,</span> <span class="s1">&#39;value&#39;</span><span class="p">:</span><span class="n">b_value</span><span class="p">,</span> <span class="s1">&#39;groups&#39;</span><span class="p">:</span><span class="n">b_groups</span><span class="p">}</span>
<span class="n">dataset</span> <span class="o">=</span> <span class="p">[[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">],</span>
<span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">],</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">],</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">],</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">],</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">],</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">],</span>
<span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">],</span>
<span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">],</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">],</span>
<span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">],</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">],</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">],</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">]]</span>
<span class="n">split</span> <span class="o">=</span> <span class="n">get_split</span><span class="p">(</span><span class="n">dataset</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Split: [X</span><span class="si">%d</span><span class="s1"> &lt; </span><span class="si">%.3f</span><span class="s1">]&#39;</span> <span class="o">%</span> <span class="p">((</span><span class="n">split</span><span class="p">[</span><span class="s1">&#39;index&#39;</span><span class="p">]</span><span class="o">+</span><span class="mi">1</span><span class="p">),</span> <span class="n">split</span><span class="p">[</span><span class="s1">&#39;value&#39;</span><span class="p">]))</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>X1 &lt; 0.000 Gini=0.408
X1 &lt; 0.000 Gini=0.408
X1 &lt; 1.000 Gini=0.394
X1 &lt; 2.000 Gini=0.394
X1 &lt; 2.000 Gini=0.394
X1 &lt; 2.000 Gini=0.394
X1 &lt; 1.000 Gini=0.394
X1 &lt; 0.000 Gini=0.408
X1 &lt; 0.000 Gini=0.408
X1 &lt; 2.000 Gini=0.394
X1 &lt; 0.000 Gini=0.408
X1 &lt; 1.000 Gini=0.394
X1 &lt; 1.000 Gini=0.394
X1 &lt; 2.000 Gini=0.394
X2 &lt; 0.000 Gini=0.408
X2 &lt; 0.000 Gini=0.408
X2 &lt; 0.000 Gini=0.408
X2 &lt; 1.000 Gini=0.407
X2 &lt; 2.000 Gini=0.407
X2 &lt; 2.000 Gini=0.407
X2 &lt; 2.000 Gini=0.407
X2 &lt; 1.000 Gini=0.407
X2 &lt; 2.000 Gini=0.407
X2 &lt; 1.000 Gini=0.407
X2 &lt; 1.000 Gini=0.407
X2 &lt; 1.000 Gini=0.407
X2 &lt; 0.000 Gini=0.408
X2 &lt; 1.000 Gini=0.407
X3 &lt; 0.000 Gini=0.408
X3 &lt; 0.000 Gini=0.408
X3 &lt; 0.000 Gini=0.408
X3 &lt; 0.000 Gini=0.408
X3 &lt; 1.000 Gini=0.367
X3 &lt; 1.000 Gini=0.367
X3 &lt; 1.000 Gini=0.367
X3 &lt; 0.000 Gini=0.408
X3 &lt; 1.000 Gini=0.367
X3 &lt; 1.000 Gini=0.367
X3 &lt; 1.000 Gini=0.367
X3 &lt; 0.000 Gini=0.408
X3 &lt; 1.000 Gini=0.367
X3 &lt; 0.000 Gini=0.408
X4 &lt; 0.000 Gini=0.408
X4 &lt; 1.000 Gini=0.405
X4 &lt; 0.000 Gini=0.408
X4 &lt; 0.000 Gini=0.408
X4 &lt; 0.000 Gini=0.408
X4 &lt; 1.000 Gini=0.405
X4 &lt; 1.000 Gini=0.405
X4 &lt; 0.000 Gini=0.408
X4 &lt; 0.000 Gini=0.408
X4 &lt; 0.000 Gini=0.408
X4 &lt; 1.000 Gini=0.405
X4 &lt; 1.000 Gini=0.405
X4 &lt; 0.000 Gini=0.408
X4 &lt; 1.000 Gini=0.405
Split: [X3 &lt; 1.000]
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="regression-trees">
<h2>Regression trees<a class="headerlink" href="#regression-trees" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Quadratic training set + noise</span>
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span>
<span class="n">m</span> <span class="o">=</span> <span class="mi">200</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">m</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span> <span class="o">*</span> <span class="p">(</span><span class="n">X</span> <span class="o">-</span> <span class="mf">0.5</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">y</span> <span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">m</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="mi">10</span>
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">DecisionTreeRegressor</span>
<span class="n">tree_reg</span> <span class="o">=</span> <span class="n">DecisionTreeRegressor</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">tree_reg</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>DecisionTreeRegressor(max_depth=2, random_state=42)
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="final-regressor-code">
<h2>Final regressor code<a class="headerlink" href="#final-regressor-code" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">DecisionTreeRegressor</span>
<span class="n">tree_reg1</span> <span class="o">=</span> <span class="n">DecisionTreeRegressor</span><span class="p">(</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">,</span> <span class="n">max_depth</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="n">tree_reg2</span> <span class="o">=</span> <span class="n">DecisionTreeRegressor</span><span class="p">(</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">,</span> <span class="n">max_depth</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
<span class="n">tree_reg1</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="n">tree_reg2</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">plot_regression_predictions</span><span class="p">(</span><span class="n">tree_reg</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">axes</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.2</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="n">ylabel</span><span class="o">=</span><span class="s2">&quot;$y$&quot;</span><span class="p">):</span>
<span class="n">x1</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linspace</span><span class="p">(</span><span class="n">axes</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">axes</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="mi">500</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">tree_reg</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">x1</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="n">axes</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s2">&quot;$x_1$&quot;</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">18</span><span class="p">)</span>
<span class="k">if</span> <span class="n">ylabel</span><span class="p">:</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="n">ylabel</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">18</span><span class="p">,</span> <span class="n">rotation</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="s2">&quot;b.&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">x1</span><span class="p">,</span> <span class="n">y_pred</span><span class="p">,</span> <span class="s2">&quot;r.-&quot;</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="sa">r</span><span class="s2">&quot;$\hat</span><span class="si">{y}</span><span class="s2">$&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">11</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span>
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">121</span><span class="p">)</span>
<span class="n">plot_regression_predictions</span><span class="p">(</span><span class="n">tree_reg1</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="k">for</span> <span class="n">split</span><span class="p">,</span> <span class="n">style</span> <span class="ow">in</span> <span class="p">((</span><span class="mf">0.1973</span><span class="p">,</span> <span class="s2">&quot;k-&quot;</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.0917</span><span class="p">,</span> <span class="s2">&quot;k--&quot;</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.7718</span><span class="p">,</span> <span class="s2">&quot;k--&quot;</span><span class="p">)):</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">([</span><span class="n">split</span><span class="p">,</span> <span class="n">split</span><span class="p">],</span> <span class="p">[</span><span class="o">-</span><span class="mf">0.2</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="n">style</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">text</span><span class="p">(</span><span class="mf">0.21</span><span class="p">,</span> <span class="mf">0.65</span><span class="p">,</span> <span class="s2">&quot;Depth=0&quot;</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">15</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">text</span><span class="p">(</span><span class="mf">0.01</span><span class="p">,</span> <span class="mf">0.2</span><span class="p">,</span> <span class="s2">&quot;Depth=1&quot;</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">13</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">text</span><span class="p">(</span><span class="mf">0.65</span><span class="p">,</span> <span class="mf">0.8</span><span class="p">,</span> <span class="s2">&quot;Depth=1&quot;</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">13</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">(</span><span class="n">loc</span><span class="o">=</span><span class="s2">&quot;upper center&quot;</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">18</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">&quot;max_depth=2&quot;</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">14</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">122</span><span class="p">)</span>
<span class="n">plot_regression_predictions</span><span class="p">(</span><span class="n">tree_reg2</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">ylabel</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
<span class="k">for</span> <span class="n">split</span><span class="p">,</span> <span class="n">style</span> <span class="ow">in</span> <span class="p">((</span><span class="mf">0.1973</span><span class="p">,</span> <span class="s2">&quot;k-&quot;</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.0917</span><span class="p">,</span> <span class="s2">&quot;k--&quot;</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.7718</span><span class="p">,</span> <span class="s2">&quot;k--&quot;</span><span class="p">)):</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">([</span><span class="n">split</span><span class="p">,</span> <span class="n">split</span><span class="p">],</span> <span class="p">[</span><span class="o">-</span><span class="mf">0.2</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="n">style</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="k">for</span> <span class="n">split</span> <span class="ow">in</span> <span class="p">(</span><span class="mf">0.0458</span><span class="p">,</span> <span class="mf">0.1298</span><span class="p">,</span> <span class="mf">0.2873</span><span class="p">,</span> <span class="mf">0.9040</span><span class="p">):</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">([</span><span class="n">split</span><span class="p">,</span> <span class="n">split</span><span class="p">],</span> <span class="p">[</span><span class="o">-</span><span class="mf">0.2</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="s2">&quot;k:&quot;</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">text</span><span class="p">(</span><span class="mf">0.3</span><span class="p">,</span> <span class="mf">0.5</span><span class="p">,</span> <span class="s2">&quot;Depth=2&quot;</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">13</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">&quot;max_depth=3&quot;</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">14</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/week46_72_0.png" src="_images/week46_72_0.png" />
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">tree_reg1</span> <span class="o">=</span> <span class="n">DecisionTreeRegressor</span><span class="p">(</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">tree_reg2</span> <span class="o">=</span> <span class="n">DecisionTreeRegressor</span><span class="p">(</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">,</span> <span class="n">min_samples_leaf</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
<span class="n">tree_reg1</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="n">tree_reg2</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="n">x1</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linspace</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">500</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">y_pred1</span> <span class="o">=</span> <span class="n">tree_reg1</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">x1</span><span class="p">)</span>
<span class="n">y_pred2</span> <span class="o">=</span> <span class="n">tree_reg2</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">x1</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">11</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span>
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">121</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="s2">&quot;b.&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">x1</span><span class="p">,</span> <span class="n">y_pred1</span><span class="p">,</span> <span class="s2">&quot;r.-&quot;</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="sa">r</span><span class="s2">&quot;$\hat</span><span class="si">{y}</span><span class="s2">$&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.2</span><span class="p">,</span> <span class="mf">1.1</span><span class="p">])</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s2">&quot;$x_1$&quot;</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">18</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s2">&quot;$y$&quot;</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">18</span><span class="p">,</span> <span class="n">rotation</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">(</span><span class="n">loc</span><span class="o">=</span><span class="s2">&quot;upper center&quot;</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">18</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">&quot;No restrictions&quot;</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">14</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">122</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="s2">&quot;b.&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">x1</span><span class="p">,</span> <span class="n">y_pred2</span><span class="p">,</span> <span class="s2">&quot;r.-&quot;</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="sa">r</span><span class="s2">&quot;$\hat</span><span class="si">{y}</span><span class="s2">$&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.2</span><span class="p">,</span> <span class="mf">1.1</span><span class="p">])</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s2">&quot;$x_1$&quot;</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">18</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">&quot;min_samples_leaf=</span><span class="si">{}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">tree_reg2</span><span class="o">.</span><span class="n">min_samples_leaf</span><span class="p">),</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">14</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/week46_73_0.png" src="_images/week46_73_0.png" />
</div>
</div>
</div>
<div class="section" id="pros-and-cons-of-trees-pros">
<h2>Pros and cons of trees, pros<a class="headerlink" href="#pros-and-cons-of-trees-pros" title="Permalink to this headline"></a></h2>
<ul class="simple">
<li><p>White box, easy to interpret model. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches discussed earlier (think of support vector machines)</p></li>
<li><p>Trees are very easy to explain to people. In fact, they are even easier to explain than linear regression!</p></li>
<li><p>No feature normalization needed</p></li>
<li><p>Tree models can handle both continuous and categorical data (Classification and Regression Trees)</p></li>
<li><p>Can model nonlinear relationships</p></li>
<li><p>Can model interactions between the different descriptive features</p></li>
<li><p>Trees can be displayed graphically, and are easily interpreted even by a non-expert (especially if they are small)</p></li>
</ul>
</div>
<div class="section" id="disadvantages">
<h2>Disadvantages<a class="headerlink" href="#disadvantages" title="Permalink to this headline"></a></h2>
<ul class="simple">
<li><p>Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches</p></li>
<li><p>If continuous features are used the tree may become quite large and hence less interpretable</p></li>
<li><p>Decision trees are prone to overfit the training data and hence do not well generalize the data if no stopping criteria or improvements like pruning, boosting or bagging are implemented</p></li>
<li><p>Small changes in the data may lead to a completely different tree. This issue can be addressed by using ensemble methods like bagging, boosting or random forests</p></li>
<li><p>Unbalanced datasets where some target feature values occur much more frequently than others may lead to biased trees since the frequently occurring feature values are preferred over the less frequently occurring ones.</p></li>
<li><p>If the number of features is relatively large (high dimensional) and the number of instances is relatively low, the tree might overfit the data</p></li>
<li><p>Features with many levels may be preferred over features with less levels since for them it is <em>more easy</em> to split the dataset such that the sub datasets only contain pure target feature values. This issue can be addressed by preferring for instance the information gain ratio as splitting criteria over information gain</p></li>
</ul>
<p>However, by aggregating many decision trees, using methods like
bagging, random forests, and boosting, the predictive performance of
trees can be substantially improved.</p>
</div>
<div class="section" id="ensemble-methods-from-a-single-tree-to-many-trees-and-extreme-boosting-meet-the-jungle-of-methods">
<h2>Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods<a class="headerlink" href="#ensemble-methods-from-a-single-tree-to-many-trees-and-extreme-boosting-meet-the-jungle-of-methods" title="Permalink to this headline"></a></h2>
<p>As stated above and seen in many of the examples discussed here about
a single decision tree, we often end up overfitting our training
data. This normally means that we have a high variance. Can we reduce
the variance of a statistical learning method?</p>
<p>This leads us to a set of different methods that can combine different
machine learning algorithms or just use one of them to construct
forests and jungles of trees, homogeneous ones or heterogenous
ones. These methods are recognized by different names which we will
try to explain here. These are</p>
<ol class="simple">
<li><p>Voting classifiers</p></li>
<li><p>Bagging and Pasting</p></li>
<li><p>Random forests</p></li>
<li><p>Boosting methods, from adaptive to Extreme Gradient Boosting (XGBoost)</p></li>
</ol>
<p>We discuss these methods here.</p>
</div>
<div class="section" id="an-overview-of-ensemble-methods">
<h2>An Overview of Ensemble Methods<a class="headerlink" href="#an-overview-of-ensemble-methods" title="Permalink to this headline"></a></h2>
<!-- dom:FIGURE: [DataFiles/ensembleoverview.png, width=600 frac=0.8] -->
<!-- begin figure -->
<p><img src="DataFiles/ensembleoverview.png" width="600"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure --></div>
<div class="section" id="why-voting">
<h2>Why Voting?<a class="headerlink" href="#why-voting" title="Permalink to this headline"></a></h2>
<p>The idea behind boosting, and voting as well can be phrased as follows:
<strong>Can a group of people somehow arrive at highly
reasoned decisions, despite the weak judgement of the individual
members?</strong></p>
<p>The aim is to create a good classifier by combining several weak classifiers.
<strong>A weak classifier is a classifier which is able to produce results that are only slightly better than guessing at random.</strong></p>
<p>The basic approach is to apply repeatedly (in boosting this is done in an iterative way) a weak classifier to modifications of the data.
In voting we simply apply the law of large numbers while in boosting we give more weight to misclassified data in
each iteration.</p>
<p>Decision trees play an important role as our weak classifier. They serve as the basic method.</p>
</div>
<div class="section" id="tossing-coins">
<h2>Tossing coins<a class="headerlink" href="#tossing-coins" title="Permalink to this headline"></a></h2>
<p>The simplest case is a so-called voting ensemble. To illustrate this,
think of yourself tossing coins with a biased outcome of 51 per cent
for heads and 49% for tails. With only few tosses,
you may not clearly see this distribution for heads and tails. However, after some
thousands of tosses, there will be a clear majority of heads. With 2000 tosses
you should see approximately 1020 heads and 980 tails.</p>
<p>We can then state that the outcome is a clear majority of heads. If
you do this ten thousand times, it is easy to see that there is a 97%
likelihood of a majority of heads.</p>
<p>Another example would be to collect all polls before an
election. Different polls may show different likelihoods for a
candidate winning with say a majority of the popular vote. The majority vote
would then consist in many polls indicating that this candidate will
actually win.</p>
<p>The example here shows how we can implement the coin tossing case,
clealry demostrating that after some tosses we see the <a class="reference external" href="https://en.wikipedia.org/wiki/Law_of_large_numbers">law of large</a>
numbers kicking in.</p>
</div>
<div class="section" id="standard-imports-first">
<h2>Standard imports first<a class="headerlink" href="#standard-imports-first" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Common imports</span>
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">Image</span>
<span class="kn">from</span> <span class="nn">pydot</span> <span class="kn">import</span> <span class="n">graph_from_dot_data</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">DecisionTreeClassifier</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">export_graphviz</span>
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">StandardScaler</span><span class="p">,</span> <span class="n">OneHotEncoder</span>
<span class="kn">from</span> <span class="nn">sklearn.compose</span> <span class="kn">import</span> <span class="n">ColumnTransformer</span>
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">Image</span>
<span class="kn">from</span> <span class="nn">pydot</span> <span class="kn">import</span> <span class="n">graph_from_dot_data</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="c1"># Where to save the figures and data files</span>
<span class="n">PROJECT_ROOT_DIR</span> <span class="o">=</span> <span class="s2">&quot;Results&quot;</span>
<span class="n">FIGURE_ID</span> <span class="o">=</span> <span class="s2">&quot;Results/FigureFiles&quot;</span>
<span class="n">DATA_ID</span> <span class="o">=</span> <span class="s2">&quot;DataFiles/&quot;</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">PROJECT_ROOT_DIR</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">PROJECT_ROOT_DIR</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">):</span>
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">,</span> <span class="n">fig_id</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">data_path</span><span class="p">(</span><span class="n">dat_id</span><span class="p">):</span>
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">,</span> <span class="n">dat_id</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">save_fig</span><span class="p">(</span><span class="n">fig_id</span><span class="p">):</span>
<span class="n">plt</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="n">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">)</span> <span class="o">+</span> <span class="s2">&quot;.png&quot;</span><span class="p">,</span> <span class="nb">format</span><span class="o">=</span><span class="s1">&#39;png&#39;</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="simple-voting-example-head-or-tail">
<h2>Simple Voting Example, head or tail<a class="headerlink" href="#simple-voting-example-head-or-tail" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Common imports</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">matplotlib.colors</span> <span class="kn">import</span> <span class="n">ListedColormap</span>
<span class="n">plt</span><span class="o">.</span><span class="n">rcParams</span><span class="p">[</span><span class="s1">&#39;axes.labelsize&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="mi">14</span>
<span class="n">plt</span><span class="o">.</span><span class="n">rcParams</span><span class="p">[</span><span class="s1">&#39;xtick.labelsize&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="mi">12</span>
<span class="n">plt</span><span class="o">.</span><span class="n">rcParams</span><span class="p">[</span><span class="s1">&#39;ytick.labelsize&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="mi">12</span>
<span class="n">heads_proba</span> <span class="o">=</span> <span class="mf">0.51</span>
<span class="n">coin_tosses</span> <span class="o">=</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">10000</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="o">&lt;</span> <span class="n">heads_proba</span><span class="p">)</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">int32</span><span class="p">)</span>
<span class="n">cumulative_heads_ratio</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">cumsum</span><span class="p">(</span><span class="n">coin_tosses</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">10001</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">8</span><span class="p">,</span><span class="mf">3.5</span><span class="p">))</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">cumulative_heads_ratio</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">10000</span><span class="p">],</span> <span class="p">[</span><span class="mf">0.51</span><span class="p">,</span> <span class="mf">0.51</span><span class="p">],</span> <span class="s2">&quot;k--&quot;</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s2">&quot;51%&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">10000</span><span class="p">],</span> <span class="p">[</span><span class="mf">0.5</span><span class="p">,</span> <span class="mf">0.5</span><span class="p">],</span> <span class="s2">&quot;k-&quot;</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s2">&quot;50%&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s2">&quot;Number of coin tosses&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s2">&quot;Heads ratio&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">(</span><span class="n">loc</span><span class="o">=</span><span class="s2">&quot;lower right&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">10000</span><span class="p">,</span> <span class="mf">0.42</span><span class="p">,</span> <span class="mf">0.58</span><span class="p">])</span>
<span class="n">save_fig</span><span class="p">(</span><span class="s2">&quot;votingsimple&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/week46_83_0.png" src="_images/week46_83_0.png" />
</div>
</div>
</div>
<div class="section" id="using-the-voting-classifier">
<h2>Using the Voting Classifier<a class="headerlink" href="#using-the-voting-classifier" title="Permalink to this headline"></a></h2>
<p>We can use the voting classifier on other data sets, here the exciting binary case of two distinct objects using the make moons functionality of <strong>Scikit-Learn</strong>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="kn">import</span> <span class="n">make_moons</span>
<span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">make_moons</span><span class="p">(</span><span class="n">n_samples</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span> <span class="n">noise</span><span class="o">=</span><span class="mf">0.30</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="kn">import</span> <span class="n">RandomForestClassifier</span>
<span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="kn">import</span> <span class="n">VotingClassifier</span>
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LogisticRegression</span>
<span class="kn">from</span> <span class="nn">sklearn.svm</span> <span class="kn">import</span> <span class="n">SVC</span>
<span class="n">log_clf</span> <span class="o">=</span> <span class="n">LogisticRegression</span><span class="p">(</span><span class="n">solver</span><span class="o">=</span><span class="s2">&quot;liblinear&quot;</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">rnd_clf</span> <span class="o">=</span> <span class="n">RandomForestClassifier</span><span class="p">(</span><span class="n">n_estimators</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">svm_clf</span> <span class="o">=</span> <span class="n">SVC</span><span class="p">(</span><span class="n">gamma</span><span class="o">=</span><span class="s2">&quot;auto&quot;</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">voting_clf</span> <span class="o">=</span> <span class="n">VotingClassifier</span><span class="p">(</span>
<span class="n">estimators</span><span class="o">=</span><span class="p">[(</span><span class="s1">&#39;lr&#39;</span><span class="p">,</span> <span class="n">log_clf</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;rf&#39;</span><span class="p">,</span> <span class="n">rnd_clf</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;svc&#39;</span><span class="p">,</span> <span class="n">svm_clf</span><span class="p">)],</span>
<span class="n">voting</span><span class="o">=</span><span class="s1">&#39;hard&#39;</span><span class="p">)</span>
<span class="n">voting_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">sklearn.metrics</span> <span class="kn">import</span> <span class="n">accuracy_score</span>
<span class="k">for</span> <span class="n">clf</span> <span class="ow">in</span> <span class="p">(</span><span class="n">log_clf</span><span class="p">,</span> <span class="n">rnd_clf</span><span class="p">,</span> <span class="n">svm_clf</span><span class="p">,</span> <span class="n">voting_clf</span><span class="p">):</span>
<span class="n">clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span> <span class="n">accuracy_score</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_pred</span><span class="p">))</span>
<span class="n">log_clf</span> <span class="o">=</span> <span class="n">LogisticRegression</span><span class="p">(</span><span class="n">solver</span><span class="o">=</span><span class="s2">&quot;liblinear&quot;</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">rnd_clf</span> <span class="o">=</span> <span class="n">RandomForestClassifier</span><span class="p">(</span><span class="n">n_estimators</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">svm_clf</span> <span class="o">=</span> <span class="n">SVC</span><span class="p">(</span><span class="n">gamma</span><span class="o">=</span><span class="s2">&quot;auto&quot;</span><span class="p">,</span> <span class="n">probability</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">voting_clf</span> <span class="o">=</span> <span class="n">VotingClassifier</span><span class="p">(</span>
<span class="n">estimators</span><span class="o">=</span><span class="p">[(</span><span class="s1">&#39;lr&#39;</span><span class="p">,</span> <span class="n">log_clf</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;rf&#39;</span><span class="p">,</span> <span class="n">rnd_clf</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;svc&#39;</span><span class="p">,</span> <span class="n">svm_clf</span><span class="p">)],</span>
<span class="n">voting</span><span class="o">=</span><span class="s1">&#39;soft&#39;</span><span class="p">)</span>
<span class="n">voting_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">sklearn.metrics</span> <span class="kn">import</span> <span class="n">accuracy_score</span>
<span class="k">for</span> <span class="n">clf</span> <span class="ow">in</span> <span class="p">(</span><span class="n">log_clf</span><span class="p">,</span> <span class="n">rnd_clf</span><span class="p">,</span> <span class="n">svm_clf</span><span class="p">,</span> <span class="n">voting_clf</span><span class="p">):</span>
<span class="n">clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span> <span class="n">accuracy_score</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_pred</span><span class="p">))</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>LogisticRegression 0.864
RandomForestClassifier 0.872
SVC 0.888
VotingClassifier 0.896
LogisticRegression 0.864
RandomForestClassifier 0.872
SVC 0.888
VotingClassifier 0.912
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="voting-and-bagging">
<h2>Voting and Bagging<a class="headerlink" href="#voting-and-bagging" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="kn">import</span> <span class="n">make_moons</span>
<span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">make_moons</span><span class="p">(</span><span class="n">n_samples</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span> <span class="n">noise</span><span class="o">=</span><span class="mf">0.30</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="kn">import</span> <span class="n">RandomForestClassifier</span>
<span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="kn">import</span> <span class="n">VotingClassifier</span>
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LogisticRegression</span>
<span class="kn">from</span> <span class="nn">sklearn.svm</span> <span class="kn">import</span> <span class="n">SVC</span>
<span class="n">log_clf</span> <span class="o">=</span> <span class="n">LogisticRegression</span><span class="p">(</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">rnd_clf</span> <span class="o">=</span> <span class="n">RandomForestClassifier</span><span class="p">(</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">svm_clf</span> <span class="o">=</span> <span class="n">SVC</span><span class="p">(</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">voting_clf</span> <span class="o">=</span> <span class="n">VotingClassifier</span><span class="p">(</span>
<span class="n">estimators</span><span class="o">=</span><span class="p">[(</span><span class="s1">&#39;lr&#39;</span><span class="p">,</span> <span class="n">log_clf</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;rf&#39;</span><span class="p">,</span> <span class="n">rnd_clf</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;svc&#39;</span><span class="p">,</span> <span class="n">svm_clf</span><span class="p">)],</span>
<span class="n">voting</span><span class="o">=</span><span class="s1">&#39;hard&#39;</span><span class="p">)</span>
<span class="n">voting_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>VotingClassifier(estimators=[(&#39;lr&#39;, LogisticRegression(random_state=42)),
(&#39;rf&#39;, RandomForestClassifier(random_state=42)),
(&#39;svc&#39;, SVC(random_state=42))])
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.metrics</span> <span class="kn">import</span> <span class="n">accuracy_score</span>
<span class="k">for</span> <span class="n">clf</span> <span class="ow">in</span> <span class="p">(</span><span class="n">log_clf</span><span class="p">,</span> <span class="n">rnd_clf</span><span class="p">,</span> <span class="n">svm_clf</span><span class="p">,</span> <span class="n">voting_clf</span><span class="p">):</span>
<span class="n">clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span> <span class="n">accuracy_score</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_pred</span><span class="p">))</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>LogisticRegression 0.864
RandomForestClassifier 0.896
SVC 0.896
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>VotingClassifier 0.912
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">log_clf</span> <span class="o">=</span> <span class="n">LogisticRegression</span><span class="p">(</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">rnd_clf</span> <span class="o">=</span> <span class="n">RandomForestClassifier</span><span class="p">(</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">svm_clf</span> <span class="o">=</span> <span class="n">SVC</span><span class="p">(</span><span class="n">probability</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">voting_clf</span> <span class="o">=</span> <span class="n">VotingClassifier</span><span class="p">(</span>
<span class="n">estimators</span><span class="o">=</span><span class="p">[(</span><span class="s1">&#39;lr&#39;</span><span class="p">,</span> <span class="n">log_clf</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;rf&#39;</span><span class="p">,</span> <span class="n">rnd_clf</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;svc&#39;</span><span class="p">,</span> <span class="n">svm_clf</span><span class="p">)],</span>
<span class="n">voting</span><span class="o">=</span><span class="s1">&#39;soft&#39;</span><span class="p">)</span>
<span class="n">voting_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>VotingClassifier(estimators=[(&#39;lr&#39;, LogisticRegression(random_state=42)),
(&#39;rf&#39;, RandomForestClassifier(random_state=42)),
(&#39;svc&#39;, SVC(probability=True, random_state=42))],
voting=&#39;soft&#39;)
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.metrics</span> <span class="kn">import</span> <span class="n">accuracy_score</span>
<span class="k">for</span> <span class="n">clf</span> <span class="ow">in</span> <span class="p">(</span><span class="n">log_clf</span><span class="p">,</span> <span class="n">rnd_clf</span><span class="p">,</span> <span class="n">svm_clf</span><span class="p">,</span> <span class="n">voting_clf</span><span class="p">):</span>
<span class="n">clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span> <span class="n">accuracy_score</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_pred</span><span class="p">))</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>LogisticRegression 0.864
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>RandomForestClassifier 0.896
SVC 0.896
VotingClassifier 0.92
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="bagging">
<h2>Bagging<a class="headerlink" href="#bagging" title="Permalink to this headline"></a></h2>
<p>The <strong>plain</strong> decision trees suffer from high
variance. This means that if we split the training data into two parts
at random, and fit a decision tree to both halves, the results that we
get could be quite different. In contrast, a procedure with low
variance will yield similar results if applied repeatedly to distinct
data sets; linear regression tends to have low variance, if the ratio
of <span class="math notranslate nohighlight">\(n\)</span> to <span class="math notranslate nohighlight">\(p\)</span> is moderately large.</p>
<p><strong>Bootstrap aggregation</strong>, or just <strong>bagging</strong>, is a
general-purpose procedure for reducing the variance of a statistical
learning method.</p>
</div>
<div class="section" id="more-bagging">
<h2>More bagging<a class="headerlink" href="#more-bagging" title="Permalink to this headline"></a></h2>
<p>Bagging typically results in improved accuracy
over prediction using a single tree. Unfortunately, however, it can be
difficult to interpret the resulting model. Recall that one of the
advantages of decision trees is the attractive and easily interpreted
diagram that results.</p>
<p>However, when we bag a large number of trees, it is no longer
possible to represent the resulting statistical learning procedure
using a single tree, and it is no longer clear which variables are
most important to the procedure. Thus, bagging improves prediction
accuracy at the expense of interpretability. Although the collection
of bagged trees is much more difficult to interpret than a single
tree, one can obtain an overall summary of the importance of each
predictor using the MSE (for bagging regression trees) or the Gini
index (for bagging classification trees). In the case of bagging
regression trees, we can record the total amount that the MSE is
decreased due to splits over a given predictor, averaged over all <span class="math notranslate nohighlight">\(B\)</span> possible
trees. A large value indicates an important predictor. Similarly, in
the context of bagging classification trees, we can add up the total
amount that the Gini index is decreased by splits over a given
predictor, averaged over all <span class="math notranslate nohighlight">\(B\)</span> trees.</p>
</div>
<div class="section" id="making-your-own-bootstrap-changing-the-level-of-the-decision-tree">
<h2>Making your own Bootstrap: Changing the Level of the Decision Tree<a class="headerlink" href="#making-your-own-bootstrap-changing-the-level-of-the-decision-tree" title="Permalink to this headline"></a></h2>
<p>Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with
a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points <span class="math notranslate nohighlight">\(n\)</span>).</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.pipeline</span> <span class="kn">import</span> <span class="n">make_pipeline</span>
<span class="kn">from</span> <span class="nn">sklearn.utils</span> <span class="kn">import</span> <span class="n">resample</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">DecisionTreeRegressor</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">n_boostraps</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">maxdepth</span> <span class="o">=</span> <span class="mi">8</span>
<span class="c1"># Make data set.</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linspace</span><span class="p">(</span><span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="n">n</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="mf">1.5</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
<span class="n">error</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">maxdepth</span><span class="p">)</span>
<span class="n">bias</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">maxdepth</span><span class="p">)</span>
<span class="n">variance</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">maxdepth</span><span class="p">)</span>
<span class="n">polydegree</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">maxdepth</span><span class="p">)</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">StandardScaler</span>
<span class="n">scaler</span> <span class="o">=</span> <span class="n">StandardScaler</span><span class="p">()</span>
<span class="n">scaler</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
<span class="n">X_train_scaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
<span class="n">X_test_scaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="c1"># we produce a simple tree first as benchmark</span>
<span class="n">simpletree</span> <span class="o">=</span> <span class="n">DecisionTreeRegressor</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
<span class="n">simpletree</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train_scaled</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="n">simpleprediction</span> <span class="o">=</span> <span class="n">simpletree</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">)</span>
<span class="k">for</span> <span class="n">degree</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="n">maxdepth</span><span class="p">):</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">DecisionTreeRegressor</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="n">degree</span><span class="p">)</span>
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">empty</span><span class="p">((</span><span class="n">y_test</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">n_boostraps</span><span class="p">))</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_boostraps</span><span class="p">):</span>
<span class="n">x_</span><span class="p">,</span> <span class="n">y_</span> <span class="o">=</span> <span class="n">resample</span><span class="p">(</span><span class="n">X_train_scaled</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">x_</span><span class="p">,</span> <span class="n">y_</span><span class="p">)</span>
<span class="n">y_pred</span><span class="p">[:,</span> <span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">)</span><span class="c1">#.ravel()</span>
<span class="n">polydegree</span><span class="p">[</span><span class="n">degree</span><span class="p">]</span> <span class="o">=</span> <span class="n">degree</span>
<span class="n">error</span><span class="p">[</span><span class="n">degree</span><span class="p">]</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">((</span><span class="n">y_test</span> <span class="o">-</span> <span class="n">y_pred</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdims</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="p">)</span>
<span class="n">bias</span><span class="p">[</span><span class="n">degree</span><span class="p">]</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span> <span class="p">(</span><span class="n">y_test</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y_pred</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdims</span><span class="o">=</span><span class="kc">True</span><span class="p">))</span><span class="o">**</span><span class="mi">2</span> <span class="p">)</span>
<span class="n">variance</span><span class="p">[</span><span class="n">degree</span><span class="p">]</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span> <span class="n">np</span><span class="o">.</span><span class="n">var</span><span class="p">(</span><span class="n">y_pred</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdims</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Polynomial degree:&#39;</span><span class="p">,</span> <span class="n">degree</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Error:&#39;</span><span class="p">,</span> <span class="n">error</span><span class="p">[</span><span class="n">degree</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Bias^2:&#39;</span><span class="p">,</span> <span class="n">bias</span><span class="p">[</span><span class="n">degree</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Var:&#39;</span><span class="p">,</span> <span class="n">variance</span><span class="p">[</span><span class="n">degree</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;</span><span class="si">{}</span><span class="s1"> &gt;= </span><span class="si">{}</span><span class="s1"> + </span><span class="si">{}</span><span class="s1"> = </span><span class="si">{}</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">error</span><span class="p">[</span><span class="n">degree</span><span class="p">],</span> <span class="n">bias</span><span class="p">[</span><span class="n">degree</span><span class="p">],</span> <span class="n">variance</span><span class="p">[</span><span class="n">degree</span><span class="p">],</span> <span class="n">bias</span><span class="p">[</span><span class="n">degree</span><span class="p">]</span><span class="o">+</span><span class="n">variance</span><span class="p">[</span><span class="n">degree</span><span class="p">]))</span>
<span class="n">mse_simpletree</span><span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">((</span><span class="n">y_test</span> <span class="o">-</span> <span class="n">simpleprediction</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Simple tree:&quot;</span><span class="p">,</span><span class="n">mse_simpletree</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlim</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="n">maxdepth</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">polydegree</span><span class="p">,</span> <span class="n">error</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s1">&#39;MSE&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">polydegree</span><span class="p">,</span> <span class="n">bias</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s1">&#39;bias&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">polydegree</span><span class="p">,</span> <span class="n">variance</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s1">&#39;Variance&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">save_fig</span><span class="p">(</span><span class="s2">&quot;baggingboot&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 1
Error: 0.06380941468319971
Bias^2: 0.05160313473529168
Var: 0.01220627994790804
0.06380941468319971 &gt;= 0.05160313473529168 + 0.01220627994790804 = 0.06380941468319971
Polynomial degree: 2
Error: 0.043464037468677004
Bias^2: 0.02659851591375224
Var: 0.01686552155492476
0.043464037468677004 &gt;= 0.02659851591375224 + 0.01686552155492476 = 0.043464037468677
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 3
Error: 0.020716391693769383
Bias^2: 0.01159033914386312
Var: 0.00912605254990626
0.020716391693769383 &gt;= 0.01159033914386312 + 0.00912605254990626 = 0.02071639169376938
Polynomial degree: 4
Error: 0.02063627410934057
Bias^2: 0.0117496656370668
Var: 0.008886608472273775
0.02063627410934057 &gt;= 0.0117496656370668 + 0.008886608472273775 = 0.020636274109340574
Polynomial degree: 5
Error: 0.02087627881701288
Bias^2: 0.01349183949256158
Var: 0.007384439324451296
0.02087627881701288 &gt;= 0.01349183949256158 + 0.007384439324451296 = 0.020876278817012876
Polynomial degree: 6
Error: 0.02069601123831537
Bias^2: 0.013918526350129823
Var: 0.0067774848881855445
0.02069601123831537 &gt;= 0.013918526350129823 + 0.0067774848881855445 = 0.020696011238315368
Polynomial degree: 7
Error: 0.022964339924731444
Bias^2: 0.01550381208433455
Var: 0.007460527840396904
0.022964339924731444 &gt;= 0.01550381208433455 + 0.007460527840396904 = 0.022964339924731455
Simple tree: 0.5148389267750961
</pre></div>
</div>
<img alt="_images/week46_94_2.png" src="_images/week46_94_2.png" />
</div>
</div>
</div>
<div class="section" id="random-forests">
<h2>Random forests<a class="headerlink" href="#random-forests" title="Permalink to this headline"></a></h2>
<p>Random forests provide an improvement over bagged trees by way of a
small tweak that decorrelates the trees.</p>
<p>As in bagging, we build a
number of decision trees on bootstrapped training samples. But when
building these decision trees, each time a split in a tree is
considered, a random sample of <span class="math notranslate nohighlight">\(m\)</span> predictors is chosen as split
candidates from the full set of <span class="math notranslate nohighlight">\(p\)</span> predictors. The split is allowed to
use only one of those <span class="math notranslate nohighlight">\(m\)</span> predictors.</p>
<p>A fresh sample of <span class="math notranslate nohighlight">\(m\)</span> predictors is
taken at each split, and typically we choose</p>
<div class="math notranslate nohighlight">
\[
m\approx \sqrt{p}.
\]</div>
<p>In building a random forest, at
each split in the tree, the algorithm is not even allowed to consider
a majority of the available predictors.</p>
<p>The reason for this is rather clever. Suppose that there is one very
strong predictor in the data set, along with a number of other
moderately strong predictors. Then in the collection of bagged
variable importance random forest trees, most or all of the trees will
use this strong predictor in the top split. Consequently, all of the
bagged trees will look quite similar to each other. Hence the
predictions from the bagged trees will be highly correlated.
Unfortunately, averaging many highly correlated quantities does not
lead to as large of a reduction in variance as averaging many
uncorrelated quantities. In particular, this means that bagging will
not lead to a substantial reduction in variance over a single tree in
this setting.</p>
</div>
<div class="section" id="random-forest-algorithm">
<h2>Random Forest Algorithm<a class="headerlink" href="#random-forest-algorithm" title="Permalink to this headline"></a></h2>
<p>The algorithm described here can be applied to both classification and regression problems.</p>
<p>We will grow of forest of say <span class="math notranslate nohighlight">\(B\)</span> trees.</p>
<ol class="simple">
<li><p>For <span class="math notranslate nohighlight">\(b=1:B\)</span></p></li>
</ol>
<ul class="simple">
<li><p>Draw a bootstrap sample from the training data organized in our <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> matrix.</p></li>
<li><p>We grow then a random forest tree <span class="math notranslate nohighlight">\(T_b\)</span> based on the bootstrapped data by repeating the steps outlined till we reach the maximum node size is reached</p></li>
</ul>
<ol class="simple">
<li><p>we select <span class="math notranslate nohighlight">\(m \le p\)</span> variables at random from the <span class="math notranslate nohighlight">\(p\)</span> predictors/features</p></li>
<li><p>pick the best split point among the <span class="math notranslate nohighlight">\(m\)</span> features using for example the CART algorithm and create a new node</p></li>
<li><p>split the node into daughter nodes</p></li>
<li><p>Output then the ensemble of trees <span class="math notranslate nohighlight">\(\{T_b\}_1^{B}\)</span> and make predictions for either a regression type of problem or a classification type of problem.</p></li>
</ol>
</div>
<div class="section" id="random-forests-compared-with-other-methods-on-the-cancer-data">
<h2>Random Forests Compared with other Methods on the Cancer Data<a class="headerlink" href="#random-forests-compared-with-other-methods-on-the-cancer-data" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="kn">import</span> <span class="n">load_breast_cancer</span>
<span class="kn">from</span> <span class="nn">sklearn.svm</span> <span class="kn">import</span> <span class="n">SVC</span>
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LogisticRegression</span>
<span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="kn">import</span> <span class="n">DecisionTreeClassifier</span>
<span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="kn">import</span> <span class="n">BaggingClassifier</span>
<span class="c1"># Load the data</span>
<span class="n">cancer</span> <span class="o">=</span> <span class="n">load_breast_cancer</span><span class="p">()</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">cancer</span><span class="o">.</span><span class="n">data</span><span class="p">,</span><span class="n">cancer</span><span class="o">.</span><span class="n">target</span><span class="p">,</span><span class="n">random_state</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X_test</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
<span class="c1">#define methods</span>
<span class="c1"># Logistic Regression</span>
<span class="n">logreg</span> <span class="o">=</span> <span class="n">LogisticRegression</span><span class="p">(</span><span class="n">solver</span><span class="o">=</span><span class="s1">&#39;lbfgs&#39;</span><span class="p">)</span>
<span class="c1"># Support vector machine</span>
<span class="n">svm</span> <span class="o">=</span> <span class="n">SVC</span><span class="p">(</span><span class="n">gamma</span><span class="o">=</span><span class="s1">&#39;auto&#39;</span><span class="p">,</span> <span class="n">C</span><span class="o">=</span><span class="mi">100</span><span class="p">)</span>
<span class="c1"># Decision Trees</span>
<span class="n">deep_tree_clf</span> <span class="o">=</span> <span class="n">DecisionTreeClassifier</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
<span class="c1">#Scale the data</span>
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">StandardScaler</span>
<span class="n">scaler</span> <span class="o">=</span> <span class="n">StandardScaler</span><span class="p">()</span>
<span class="n">scaler</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
<span class="n">X_train_scaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
<span class="n">X_test_scaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="c1"># Logistic Regression</span>
<span class="n">logreg</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train_scaled</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Test set accuracy Logistic Regression with scaled data: </span><span class="si">{:.2f}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">logreg</span><span class="o">.</span><span class="n">score</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">,</span><span class="n">y_test</span><span class="p">)))</span>
<span class="c1"># Support Vector Machine</span>
<span class="n">svm</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train_scaled</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Test set accuracy SVM with scaled data: </span><span class="si">{:.2f}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">logreg</span><span class="o">.</span><span class="n">score</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">,</span><span class="n">y_test</span><span class="p">)))</span>
<span class="c1"># Decision Trees</span>
<span class="n">deep_tree_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train_scaled</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Test set accuracy with Decision Trees and scaled data: </span><span class="si">{:.2f}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">deep_tree_clf</span><span class="o">.</span><span class="n">score</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">,</span><span class="n">y_test</span><span class="p">)))</span>
<span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="kn">import</span> <span class="n">RandomForestClassifier</span>
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">LabelEncoder</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">cross_validate</span>
<span class="c1"># Data set not specificied</span>
<span class="c1">#Instantiate the model with 500 trees and entropy as splitting criteria</span>
<span class="n">Random_Forest_model</span> <span class="o">=</span> <span class="n">RandomForestClassifier</span><span class="p">(</span><span class="n">n_estimators</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span><span class="n">criterion</span><span class="o">=</span><span class="s2">&quot;entropy&quot;</span><span class="p">)</span>
<span class="n">Random_Forest_model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train_scaled</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="c1">#Cross validation</span>
<span class="n">accuracy</span> <span class="o">=</span> <span class="n">cross_validate</span><span class="p">(</span><span class="n">Random_Forest_model</span><span class="p">,</span><span class="n">X_test_scaled</span><span class="p">,</span><span class="n">y_test</span><span class="p">,</span><span class="n">cv</span><span class="o">=</span><span class="mi">10</span><span class="p">)[</span><span class="s1">&#39;test_score&#39;</span><span class="p">]</span>
<span class="nb">print</span><span class="p">(</span><span class="n">accuracy</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Test set accuracy with Random Forests and scaled data: </span><span class="si">{:.2f}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">Random_Forest_model</span><span class="o">.</span><span class="n">score</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">,</span><span class="n">y_test</span><span class="p">)))</span>
<span class="kn">import</span> <span class="nn">scikitplot</span> <span class="k">as</span> <span class="nn">skplt</span>
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">Random_Forest_model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">)</span>
<span class="n">skplt</span><span class="o">.</span><span class="n">metrics</span><span class="o">.</span><span class="n">plot_confusion_matrix</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_pred</span><span class="p">,</span> <span class="n">normalize</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="n">y_probas</span> <span class="o">=</span> <span class="n">Random_Forest_model</span><span class="o">.</span><span class="n">predict_proba</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">)</span>
<span class="n">skplt</span><span class="o">.</span><span class="n">metrics</span><span class="o">.</span><span class="n">plot_roc</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_probas</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="n">skplt</span><span class="o">.</span><span class="n">metrics</span><span class="o">.</span><span class="n">plot_cumulative_gain</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_probas</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>(426, 30)
(143, 30)
Test set accuracy Logistic Regression with scaled data: 0.96
Test set accuracy SVM with scaled data: 0.96
Test set accuracy with Decision Trees and scaled data: 0.87
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.93333333 0.73333333 0.93333333 1. 1. 0.92857143
1. 0.92857143 0.92857143 0.92857143]
Test set accuracy with Random Forests and scaled data: 0.98
</pre></div>
</div>
<img alt="_images/week46_100_2.png" src="_images/week46_100_2.png" />
<img alt="_images/week46_100_3.png" src="_images/week46_100_3.png" />
<img alt="_images/week46_100_4.png" src="_images/week46_100_4.png" />
</div>
</div>
<p>Recall that the cumulative gains curve shows the percentage of the
overall number of cases in a given category <em>gained</em> by targeting a
percentage of the total number of cases.</p>
<p>Similarly, the receiver operating characteristic curve, or ROC curve,
displays the diagnostic ability of a binary classifier system as its
discrimination threshold is varied. It plots the true positive rate against the false positive rate.</p>
</div>
<div class="section" id="compare-bagging-on-trees-with-random-forests">
<h2>Compare Bagging on Trees with Random Forests<a class="headerlink" href="#compare-bagging-on-trees-with-random-forests" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">bag_clf</span> <span class="o">=</span> <span class="n">BaggingClassifier</span><span class="p">(</span>
<span class="n">DecisionTreeClassifier</span><span class="p">(</span><span class="n">splitter</span><span class="o">=</span><span class="s2">&quot;random&quot;</span><span class="p">,</span> <span class="n">max_leaf_nodes</span><span class="o">=</span><span class="mi">16</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">),</span>
<span class="n">n_estimators</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span> <span class="n">max_samples</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">bootstrap</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">n_jobs</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">bag_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">bag_clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="kn">import</span> <span class="n">RandomForestClassifier</span>
<span class="n">rnd_clf</span> <span class="o">=</span> <span class="n">RandomForestClassifier</span><span class="p">(</span><span class="n">n_estimators</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span> <span class="n">max_leaf_nodes</span><span class="o">=</span><span class="mi">16</span><span class="p">,</span> <span class="n">n_jobs</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">rnd_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="n">y_pred_rf</span> <span class="o">=</span> <span class="n">rnd_clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">y_pred</span> <span class="o">==</span> <span class="n">y_pred_rf</span><span class="p">)</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">y_pred</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.9790209790209791
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="boosting-a-bird-s-eye-view">
<h2>Boosting, a Bird’s Eye View<a class="headerlink" href="#boosting-a-bird-s-eye-view" title="Permalink to this headline"></a></h2>
<p>The basic idea is to combine weak classifiers in order to create a good
classifier. With a weak classifier we often intend a classifier which
produces results which are only slightly better than we would get by
random guesses.</p>
<p>This is done by applying in an iterative way a weak (or a standard
classifier like decision trees) to modify the data. In each iteration
we emphasize those observations which are misclassified by weighting
them with a factor.</p>
</div>
<div class="section" id="what-is-boosting-additive-modelling-iterative-fitting">
<h2>What is boosting? Additive Modelling/Iterative Fitting<a class="headerlink" href="#what-is-boosting-additive-modelling-iterative-fitting" title="Permalink to this headline"></a></h2>
<p>Boosting is a way of fitting an additive expansion in a set of
elementary basis functions like for example some simple polynomials.
Assume for example that we have a function</p>
<div class="math notranslate nohighlight">
\[
f_M(x) = \sum_{i=1}^M \beta_m b(x;\gamma_m),
\]</div>
<p>where <span class="math notranslate nohighlight">\(\beta_m\)</span> are the expansion parameters to be determined in a
minimization process and <span class="math notranslate nohighlight">\(b(x;\gamma_m)\)</span> are some simple functions of
the multivariable parameter <span class="math notranslate nohighlight">\(x\)</span> which is characterized by the
parameters <span class="math notranslate nohighlight">\(\gamma_m\)</span>.</p>
<p>As an example, consider the Sigmoid function we used in logistic
regression. In that case, we can translate the function
<span class="math notranslate nohighlight">\(b(x;\gamma_m)\)</span> into the Sigmoid function</p>
<div class="math notranslate nohighlight">
\[
\sigma(t) = \frac{1}{1+\exp{(-t)}},
\]</div>
<p>where <span class="math notranslate nohighlight">\(t=\gamma_0+\gamma_1 x\)</span> and the parameters <span class="math notranslate nohighlight">\(\gamma_0\)</span> and
<span class="math notranslate nohighlight">\(\gamma_1\)</span> were determined by the Logistic Regression fitting
algorithm.</p>
<p>As another example, consider the cost function we defined for linear regression</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{y},\boldsymbol{f}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
\]</div>
<p>In this case the function <span class="math notranslate nohighlight">\(f(x)\)</span> was replaced by the design matrix
<span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> and the unknown linear regression parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>,
that is <span class="math notranslate nohighlight">\(\boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta}\)</span>. In linear regression we can
simply invert a matrix and obtain the parameters <span class="math notranslate nohighlight">\(\beta\)</span> by</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\beta}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>In iterative fitting or additive modeling, we minimize the cost function with respect to the parameters <span class="math notranslate nohighlight">\(\beta_m\)</span> and <span class="math notranslate nohighlight">\(\gamma_m\)</span>.</p>
</div>
<div class="section" id="iterative-fitting-regression-and-squared-error-cost-function">
<h2>Iterative Fitting, Regression and Squared-error Cost Function<a class="headerlink" href="#iterative-fitting-regression-and-squared-error-cost-function" title="Permalink to this headline"></a></h2>
<p>The way we proceed is as follows (here we specialize to the squared-error cost function)</p>
<ol class="simple">
<li><p>Establish a cost function, here <span class="math notranslate nohighlight">\(C(\boldsymbol{y},\boldsymbol{f}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-f_M(x_i))^2\)</span> with <span class="math notranslate nohighlight">\(f_M(x) = \sum_{i=1}^M \beta_m b(x;\gamma_m)\)</span>.</p></li>
<li><p>Initialize with a guess <span class="math notranslate nohighlight">\(f_0(x)\)</span>. It could be one or even zero or some random numbers.</p></li>
<li><p>For <span class="math notranslate nohighlight">\(m=1:M\)</span></p></li>
</ol>
<p>a. minimize <span class="math notranslate nohighlight">\(\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2\)</span> wrt <span class="math notranslate nohighlight">\(\gamma\)</span> and <span class="math notranslate nohighlight">\(\beta\)</span></p>
<p>b. This gives the optimal values <span class="math notranslate nohighlight">\(\beta_m\)</span> and <span class="math notranslate nohighlight">\(\gamma_m\)</span></p>
<p>c. Determine then the new values <span class="math notranslate nohighlight">\(f_m(x)=f_{m-1}(x) +\beta_m b(x;\gamma_m)\)</span></p>
<p>We could use any of the algorithms we have discussed till now. If we
use trees, <span class="math notranslate nohighlight">\(\gamma\)</span> parameterizes the split variables and split points
at the internal nodes, and the predictions at the terminal nodes.</p>
</div>
<div class="section" id="squared-error-example-and-iterative-fitting">
<h2>Squared-Error Example and Iterative Fitting<a class="headerlink" href="#squared-error-example-and-iterative-fitting" title="Permalink to this headline"></a></h2>
<p>To better understand what happens, let us develop the steps for the iterative fitting using the above squared error function.</p>
<p>For simplicity we assume also that our functions <span class="math notranslate nohighlight">\(b(x;\gamma)=1+\gamma x\)</span>.</p>
<p>This means that for every iteration <span class="math notranslate nohighlight">\(m\)</span>, we need to optimize</p>
<div class="math notranslate nohighlight">
\[
(\beta_m,\gamma_m) = \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
\]</div>
<p>We start our iteration by simply setting <span class="math notranslate nohighlight">\(f_0(x)=0\)</span>.
Taking the derivatives with respect to <span class="math notranslate nohighlight">\(\beta\)</span> and <span class="math notranslate nohighlight">\(\gamma\)</span> we obtain</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial {\cal C}}{\partial \beta} = -2\sum_{i}(1+\gamma x_i)(y_i-\beta(1+\gamma x_i))=0,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial {\cal C}}{\partial \gamma} =-2\sum_{i}\beta x_i(y_i-\beta(1+\gamma x_i))=0.
\]</div>
<p>We can then rewrite these equations as (defining <span class="math notranslate nohighlight">\(\boldsymbol{w}=\boldsymbol{e}+\gamma \boldsymbol{x})\)</span> with <span class="math notranslate nohighlight">\(\boldsymbol{e}\)</span> being the unit vector)</p>
<div class="math notranslate nohighlight">
\[
\gamma \boldsymbol{w}^T(\boldsymbol{y}-\beta\gamma \boldsymbol{w})=0,
\]</div>
<p>which gives us <span class="math notranslate nohighlight">\(\beta = \boldsymbol{w}^T\boldsymbol{y}/(\boldsymbol{w}^T\boldsymbol{w})\)</span>. Similarly we have</p>
<div class="math notranslate nohighlight">
\[
\beta\gamma \boldsymbol{x}^T(\boldsymbol{y}-\beta(1+\gamma \boldsymbol{x}))=0,
\]</div>
<p>which leads to <span class="math notranslate nohighlight">\(\gamma =(\boldsymbol{x}^T\boldsymbol{y}-\beta\boldsymbol{x}^T\boldsymbol{e})/(\beta\boldsymbol{x}^T\boldsymbol{x})\)</span>. Inserting
for <span class="math notranslate nohighlight">\(\beta\)</span> gives us an equation for <span class="math notranslate nohighlight">\(\gamma\)</span>. This is a non-linear equation in the unknown <span class="math notranslate nohighlight">\(\gamma\)</span> and has to be solved numerically.</p>
<p>The solution to these two equations gives us in turn <span class="math notranslate nohighlight">\(\beta_1\)</span> and <span class="math notranslate nohighlight">\(\gamma_1\)</span> leading to the new expression for <span class="math notranslate nohighlight">\(f_1(x)\)</span> as
<span class="math notranslate nohighlight">\(f_1(x) = \beta_1(1+\gamma_1x)\)</span>. Doing this <span class="math notranslate nohighlight">\(M\)</span> times results in our final estimate for the function <span class="math notranslate nohighlight">\(f\)</span>.</p>
</div>
<div class="section" id="iterative-fitting-classification-and-adaboost">
<h2>Iterative Fitting, Classification and AdaBoost<a class="headerlink" href="#iterative-fitting-classification-and-adaboost" title="Permalink to this headline"></a></h2>
<p>Let us consider a binary classification problem with two outcomes <span class="math notranslate nohighlight">\(y_i \in \{-1,1\}\)</span> and <span class="math notranslate nohighlight">\(i=0,1,2,\dots,n-1\)</span> as our set of
observations. We define a classification function <span class="math notranslate nohighlight">\(G(x)\)</span> which produces a prediction taking one or the other of the two values
<span class="math notranslate nohighlight">\(\{-1,1\}\)</span>.</p>
<p>The error rate of the training sample is then</p>
<div class="math notranslate nohighlight">
\[
\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i)).
\]</div>
<p>The iterative procedure starts with defining a weak classifier whose
error rate is barely better than random guessing. The iterative
procedure in boosting is to sequentially apply a weak
classification algorithm to repeatedly modified versions of the data
producing a sequence of weak classifiers <span class="math notranslate nohighlight">\(G_m(x)\)</span>.</p>
<p>Here we will express our function <span class="math notranslate nohighlight">\(f(x)\)</span> in terms of <span class="math notranslate nohighlight">\(G(x)\)</span>. That is</p>
<div class="math notranslate nohighlight">
\[
f_M(x) = \sum_{i=1}^M \beta_m b(x;\gamma_m),
\]</div>
<p>will be a function of</p>
<div class="math notranslate nohighlight">
\[
G_M(x) = \mathrm{sign} \sum_{i=1}^M \alpha_m G_m(x).
\]</div>
</div>
<div class="section" id="adaptive-boosting-adaboost">
<h2>Adaptive Boosting, AdaBoost<a class="headerlink" href="#adaptive-boosting-adaboost" title="Permalink to this headline"></a></h2>
<p>In our iterative procedure we define thus</p>
<div class="math notranslate nohighlight">
\[
f_m(x) = f_{m-1}(x)+\beta_mG_m(x).
\]</div>
<p>The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the
exponential cost/loss function defined as</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{(-y_i(f_{m-1}(x_i)+\beta G(x_i))}.
\]</div>
<p>We optimize <span class="math notranslate nohighlight">\(\beta\)</span> and <span class="math notranslate nohighlight">\(G\)</span> for each value of <span class="math notranslate nohighlight">\(m=1:M\)</span> as we did in the regression case.
This is normally done in two steps. Let us however first rewrite the cost function as</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{(-y_i\beta G(x_i))},
\]</div>
<p>where we have defined <span class="math notranslate nohighlight">\(w_i^m= \exp{(-y_if_{m-1}(x_i))}\)</span>.</p>
</div>
<div class="section" id="building-up-adaboost">
<h2>Building up AdaBoost<a class="headerlink" href="#building-up-adaboost" title="Permalink to this headline"></a></h2>
<p>First, for any <span class="math notranslate nohighlight">\(\beta &gt; 0\)</span>, we optimize <span class="math notranslate nohighlight">\(G\)</span> by setting</p>
<div class="math notranslate nohighlight">
\[
G_m(x) = \mathrm{sign} \sum_{i=0}^{n-1} w_i^m I(y_i \ne G_(x_i)),
\]</div>
<p>which is the classifier that minimizes the weighted error rate in predicting <span class="math notranslate nohighlight">\(y\)</span>.</p>
<p>We can do this by rewriting</p>
<div class="math notranslate nohighlight">
\[
\exp{-(\beta)}\sum_{y_i=G(x_i)}w_i^m+\exp{(\beta)}\sum_{y_i\ne G(x_i)}w_i^m,
\]</div>
<p>which can be rewritten as</p>
<div class="math notranslate nohighlight">
\[
(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{(-\beta)}\sum_{i=0}^{n-1}w_i^m=0,
\]</div>
<p>which leads to</p>
<div class="math notranslate nohighlight">
\[
\beta_m = \frac{1}{2}\log{\frac{1-\mathrm{\overline{err}}}{\mathrm{\overline{err}}}},
\]</div>
<p>where we have redefined the error as</p>
<div class="math notranslate nohighlight">
\[
\mathrm{\overline{err}}_m=\frac{1}{n}\frac{\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i)}{\sum_{i=0}^{n-1}w_i^m},
\]</div>
<p>which leads to an update of</p>
<div class="math notranslate nohighlight">
\[
f_m(x) = f_{m-1}(x) +\beta_m G_m(x).
\]</div>
<p>This leads to the new weights</p>
<div class="math notranslate nohighlight">
\[
w_i^{m+1} = w_i^m \exp{(-y_i\beta_m G_m(x_i))}
\]</div>
</div>
<div class="section" id="adaptive-boosting-adaboost-basic-algorithm">
<h2>Adaptive boosting: AdaBoost, Basic Algorithm<a class="headerlink" href="#adaptive-boosting-adaboost-basic-algorithm" title="Permalink to this headline"></a></h2>
<p>The algorithm here is rather straightforward. Assume that our weak
classifier is a decision tree and we consider a binary set of outputs
with <span class="math notranslate nohighlight">\(y_i \in \{-1,1\}\)</span> and <span class="math notranslate nohighlight">\(i=0,1,2,\dots,n-1\)</span> as our set of
observations. Our design matrix is given in terms of the
feature/predictor vectors
<span class="math notranslate nohighlight">\(\boldsymbol{X}=[\boldsymbol{x}_0\boldsymbol{x}_1\dots\boldsymbol{x}_{p-1}]\)</span>. Finally, we define also a
classifier determined by our data via a function <span class="math notranslate nohighlight">\(G(x)\)</span>. This function tells us how well we are able to classify our outputs/targets <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>.</p>
<p>We have already defined the misclassification error <span class="math notranslate nohighlight">\(\mathrm{err}\)</span> as</p>
<div class="math notranslate nohighlight">
\[
\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i)),
\]</div>
<p>where the function <span class="math notranslate nohighlight">\(I()\)</span> is one if we misclassify and zero if we classify correctly.</p>
</div>
<div class="section" id="basic-steps-of-adaboost">
<h2>Basic Steps of AdaBoost<a class="headerlink" href="#basic-steps-of-adaboost" title="Permalink to this headline"></a></h2>
<p>With the above definitions we are now ready to set up the algorithm for AdaBoost.
The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.</p>
<ol class="simple">
<li><p>We start by initializing all weights to <span class="math notranslate nohighlight">\(w_i = 1/n\)</span>, with <span class="math notranslate nohighlight">\(i=0,1,2,\dots n-1\)</span>. It is easy to see that we must have <span class="math notranslate nohighlight">\(\sum_{i=0}^{n-1}w_i = 1\)</span>.</p></li>
<li><p>We rewrite the misclassification error as</p></li>
</ol>
<div class="math notranslate nohighlight">
\[
\mathrm{\overline{err}}_m=\frac{\sum_{i=0}^{n-1}w_i^m I(y_i\ne G(x_i))}{\sum_{i=0}^{n-1}w_i},
\]</div>
<ol class="simple">
<li><p>Then we start looping over all attempts at classifying, namely we start an iterative process for <span class="math notranslate nohighlight">\(m=1:M\)</span>, where <span class="math notranslate nohighlight">\(M\)</span> is the final number of classifications. Our given classifier could for example be a plain decision tree.</p></li>
</ol>
<p>a. Fit then a given classifier to the training set using the weights <span class="math notranslate nohighlight">\(w_i\)</span>.</p>
<p>b. Compute then <span class="math notranslate nohighlight">\(\mathrm{err}\)</span> and figure out which events are classified properly and which are classified wrongly.</p>
<p>c. Define a quantity <span class="math notranslate nohighlight">\(\alpha_{m} = \log{(1-\mathrm{\overline{err}}_m)/\mathrm{\overline{err}}_m}\)</span></p>
<p>d. Set the new weights to <span class="math notranslate nohighlight">\(w_i = w_i\times \exp{(\alpha_m I(y_i\ne G(x_i)}\)</span>.</p>
<ol class="simple">
<li><p>Compute the new classifier <span class="math notranslate nohighlight">\(G(x)= \sum_{i=0}^{n-1}\alpha_m I(y_i\ne G(x_i)\)</span>.</p></li>
</ol>
<p>For the iterations with <span class="math notranslate nohighlight">\(m \le 2\)</span> the weights are modified
individually at each steps. The observations which were misclassified
at iteration <span class="math notranslate nohighlight">\(m-1\)</span> have a weight which is larger than those which were
classified properly. As this proceeds, the observations which were
difficult to classifiy correctly are given a larger influence. Each
new classification step <span class="math notranslate nohighlight">\(m\)</span> is then forced to concentrate on those
observations that are missed in the previous iterations.</p>
</div>
<div class="section" id="adaboost-examples">
<h2>AdaBoost Examples<a class="headerlink" href="#adaboost-examples" title="Permalink to this headline"></a></h2>
<p>Using <strong>Scikit-Learn</strong> it is easy to apply the adaptive boosting algorithm, as done here.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="kn">import</span> <span class="n">AdaBoostClassifier</span>
<span class="n">ada_clf</span> <span class="o">=</span> <span class="n">AdaBoostClassifier</span><span class="p">(</span>
<span class="n">DecisionTreeClassifier</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="mi">1</span><span class="p">),</span> <span class="n">n_estimators</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span>
<span class="n">algorithm</span><span class="o">=</span><span class="s2">&quot;SAMME.R&quot;</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.5</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">ada_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="kn">import</span> <span class="n">AdaBoostClassifier</span>
<span class="n">ada_clf</span> <span class="o">=</span> <span class="n">AdaBoostClassifier</span><span class="p">(</span>
<span class="n">DecisionTreeClassifier</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="mi">1</span><span class="p">),</span> <span class="n">n_estimators</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span>
<span class="n">algorithm</span><span class="o">=</span><span class="s2">&quot;SAMME.R&quot;</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.5</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">ada_clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train_scaled</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">ada_clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">)</span>
<span class="n">skplt</span><span class="o">.</span><span class="n">metrics</span><span class="o">.</span><span class="n">plot_confusion_matrix</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_pred</span><span class="p">,</span> <span class="n">normalize</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="n">y_probas</span> <span class="o">=</span> <span class="n">ada_clf</span><span class="o">.</span><span class="n">predict_proba</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">)</span>
<span class="n">skplt</span><span class="o">.</span><span class="n">metrics</span><span class="o">.</span><span class="n">plot_roc</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_probas</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="n">skplt</span><span class="o">.</span><span class="n">metrics</span><span class="o">.</span><span class="n">plot_cumulative_gain</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_probas</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/week46_161_0.png" src="_images/week46_161_0.png" />
<img alt="_images/week46_161_1.png" src="_images/week46_161_1.png" />
<img alt="_images/week46_161_2.png" src="_images/week46_161_2.png" />
</div>
</div>
</div>
</div>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
kernelName: "python3",
path: "./."
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</div>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
<a class='left-prev' id="prev-link" href="week45.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
<p class="prev-next-title">Week 45, Recurrent Neural Networks</p>
</div>
</a>
<a class='right-next' id="next-link" href="week47.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title">Week 47: From Decision Trees to Ensemble Methods, Random Forests and Boosting Methods and Summary of Course</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
</div>
</div>
</div>
<footer class="footer">
<p>
By Morten Hjorth-Jensen<br/>
&copy; Copyright 2021.<br/>
</p>
</footer>
</main>
</div>
</div>
<script src="_static/js/index.be7d3bbb2ef33a8344ce.js"></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink