From 2c1f27b5193bfffa4cabeef3cc941c1272af8132 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Thu, 16 Nov 2023 06:33:54 +0100 Subject: [PATCH] update --- doc/pub/week46/html/._week46-bs000.html | 130 ++++--- doc/pub/week46/html/._week46-bs001.html | 130 ++++--- doc/pub/week46/html/._week46-bs002.html | 130 ++++--- doc/pub/week46/html/._week46-bs003.html | 130 ++++--- doc/pub/week46/html/._week46-bs004.html | 140 ++++---- doc/pub/week46/html/._week46-bs005.html | 145 ++++---- doc/pub/week46/html/._week46-bs006.html | 148 ++++---- doc/pub/week46/html/._week46-bs007.html | 250 ++++++++----- doc/pub/week46/html/._week46-bs008.html | 159 +++++---- doc/pub/week46/html/._week46-bs009.html | 252 ++++--------- doc/pub/week46/html/._week46-bs010.html | 189 +++++----- doc/pub/week46/html/._week46-bs011.html | 156 ++++----- doc/pub/week46/html/._week46-bs012.html | 188 +++++----- doc/pub/week46/html/._week46-bs013.html | 163 +++++---- doc/pub/week46/html/._week46-bs014.html | 170 ++++----- doc/pub/week46/html/._week46-bs015.html | 166 +++++---- doc/pub/week46/html/._week46-bs016.html | 183 +++++----- doc/pub/week46/html/._week46-bs017.html | 202 ++++++----- doc/pub/week46/html/._week46-bs018.html | 211 ++++++----- doc/pub/week46/html/._week46-bs019.html | 175 ++++----- doc/pub/week46/html/._week46-bs020.html | 167 ++++----- doc/pub/week46/html/._week46-bs021.html | 173 ++++----- doc/pub/week46/html/._week46-bs022.html | 180 +++++----- doc/pub/week46/html/._week46-bs023.html | 162 ++++----- doc/pub/week46/html/._week46-bs024.html | 157 ++++----- doc/pub/week46/html/._week46-bs025.html | 166 ++++----- doc/pub/week46/html/._week46-bs026.html | 158 +++++---- doc/pub/week46/html/._week46-bs027.html | 155 ++++---- doc/pub/week46/html/._week46-bs028.html | 160 ++++----- doc/pub/week46/html/._week46-bs029.html | 138 ++++---- doc/pub/week46/html/week46-bs.html | 130 ++++--- doc/pub/week46/html/week46-reveal.html | 15 - doc/pub/week46/html/week46-solarized.html | 21 -- doc/pub/week46/html/week46.html | 21 -- doc/pub/week46/ipynb/DataFiles/cancer.dot | 58 +-- doc/pub/week46/ipynb/ipynb-week46-src.tar.gz | Bin 294343 -> 294343 bytes doc/pub/week46/ipynb/week46.ipynb | 351 +++++++++---------- doc/src/week46/week46.do.txt | 14 - 38 files changed, 2623 insertions(+), 3020 deletions(-) diff --git a/doc/pub/week46/html/._week46-bs000.html b/doc/pub/week46/html/._week46-bs000.html index 39a24a971..dc7a4c9a6 100644 --- a/doc/pub/week46/html/._week46-bs000.html +++ b/doc/pub/week46/html/._week46-bs000.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -372,7 +362,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs001.html b/doc/pub/week46/html/._week46-bs001.html index 2eb5a8a1c..b361e0d1e 100644 --- a/doc/pub/week46/html/._week46-bs001.html +++ b/doc/pub/week46/html/._week46-bs001.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -375,7 +365,7 @@ MathJax.Hub.Config({
  • 10
  • 11
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs002.html b/doc/pub/week46/html/._week46-bs002.html index 01c341185..ae86b0e98 100644 --- a/doc/pub/week46/html/._week46-bs002.html +++ b/doc/pub/week46/html/._week46-bs002.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -373,7 +363,7 @@ where we then finally end up in so called leaf nodes.
  • 11
  • 12
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs003.html b/doc/pub/week46/html/._week46-bs003.html index 439dec21c..d775698da 100644 --- a/doc/pub/week46/html/._week46-bs003.html +++ b/doc/pub/week46/html/._week46-bs003.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -361,7 +351,7 @@ given some assumptions, make predictions about the target feature value
  • 12
  • 13
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs004.html b/doc/pub/week46/html/._week46-bs004.html index 5fb656ca1..c957f9153 100644 --- a/doc/pub/week46/html/._week46-bs004.html +++ b/doc/pub/week46/html/._week46-bs004.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,11 +319,15 @@ MathJax.Hub.Config({

     

     

     

    -

    A Sketch of a Tree, Regression problem

    +

    A typical Decision Tree with its pertinent Jargon, Classification Problem

    -See handwritten notes November 3 +

    +
    +

    +
    +

    - +

    This tree was produced using the Wisconsin cancer data (discussed here as well, see code examples below) using Scikit-Learn's decision tree classifier. Here we have used the so-called gini index (see below) to split the various branches.

    @@ -354,7 +348,7 @@ MathJax.Hub.Config({

  • 13
  • 14
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs005.html b/doc/pub/week46/html/._week46-bs005.html index 7cf82a557..72c21a62a 100644 --- a/doc/pub/week46/html/._week46-bs005.html +++ b/doc/pub/week46/html/._week46-bs005.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,10 +319,19 @@ MathJax.Hub.Config({

     

     

     

    -

    A Sketch of a Tree, Classification problem

    +

    General Features

    -See handwritten notes November 3 - +

    The overarching approach to decision trees is a top-down approach.

    + + +

    This process is then repeated for the subtree rooted at the new +node. +

    @@ -354,7 +353,7 @@ MathJax.Hub.Config({

  • 14
  • 15
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs006.html b/doc/pub/week46/html/._week46-bs006.html index 584e8a6ab..768d40188 100644 --- a/doc/pub/week46/html/._week46-bs006.html +++ b/doc/pub/week46/html/._week46-bs006.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,15 +319,19 @@ MathJax.Hub.Config({

     

     

     

    -

    A typical Decision Tree with its pertinent Jargon, Classification Problem

    +

    How do we set it up?

    -

    -
    -

    -
    -

    +

    In simplified terms, the process of training a decision tree and +predicting the target features of query instances is as follows: +

    -

    This tree was produced using the Wisconsin cancer data (discussed here as well, see code examples below) using Scikit-Learn's decision tree classifier. Here we have used the so-called gini index (see below) to split the various branches.

    +
      +
    1. Present a dataset containing of a number of training instances characterized by a number of descriptive features and a target feature
      2. +
    2. Train the decision tree model by continuously splitting the target feature along the values of the descriptive features using a measure of information gain during the training process
      3. +
    3. Grow the tree until we accomplish a stopping criteria create leaf nodes which represent the predictions we want to make for new query instances
      4. +
    4. Show query instances to the tree and run down the tree until we arrive at leaf nodes
      5. +
    +

    Then we are essentially done!

    @@ -360,7 +354,7 @@ MathJax.Hub.Config({

  • 15
  • 16
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs007.html b/doc/pub/week46/html/._week46-bs007.html index 42c8de6ec..e70877b65 100644 --- a/doc/pub/week46/html/._week46-bs007.html +++ b/doc/pub/week46/html/._week46-bs007.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,19 +319,117 @@ MathJax.Hub.Config({

     

     

     

    -

    General Features

    +

    Decision trees and Regression

    -

    The overarching approach to decision trees is a top-down approach.

    + +
    +
    +
    +
    +
    + 
    import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.linear_model import LinearRegression
+
+steps=250
+
+distance=0
+x=0
+distance_list=[]
+steps_list=[]
+while x<steps:
+    distance+=np.random.randint(-1,2)
+    distance_list.append(distance)
+    x+=1
+    steps_list.append(x)
+plt.plot(steps_list,distance_list, color='green', label="Random Walk Data")
+
+steps_list=np.asarray(steps_list)
+distance_list=np.asarray(distance_list)
+
+X=steps_list[:,np.newaxis]
+
+#Polynomial fits
+
+#Degree 2
+poly_features=PolynomialFeatures(degree=2, include_bias=False)
+X_poly=poly_features.fit_transform(X)
+
+lin_reg=LinearRegression()
+poly_fit=lin_reg.fit(X_poly,distance_list)
+b=lin_reg.coef_
+c=lin_reg.intercept_
+print ("2nd degree coefficients:")
+print ("zero power: ",c)
+print ("first power: ", b[0])
+print ("second power: ",b[1])
+
+z = np.arange(0, steps, .01)
+z_mod=b[1]*z**2+b[0]*z+c
+
+fit_mod=b[1]*X**2+b[0]*X+c
+plt.plot(z, z_mod, color='r', label="2nd Degree Fit")
+plt.title("Polynomial Regression")
+
+plt.xlabel("Steps")
+plt.ylabel("Distance")
+
+#Degree 10
+poly_features10=PolynomialFeatures(degree=10, include_bias=False)
+X_poly10=poly_features10.fit_transform(X)
+
+poly_fit10=lin_reg.fit(X_poly10,distance_list)
+
+y_plot=poly_fit10.predict(X_poly10)
+plt.plot(X, y_plot, color='black', label="10th Degree Fit")
+
+plt.legend()
+plt.show()
+
+
+#Decision Tree Regression
+from sklearn.tree import DecisionTreeRegressor
+regr_1=DecisionTreeRegressor(max_depth=2)
+regr_2=DecisionTreeRegressor(max_depth=5)
+regr_3=DecisionTreeRegressor(max_depth=7)
+regr_1.fit(X, distance_list)
+regr_2.fit(X, distance_list)
+regr_3.fit(X, distance_list)
+
+X_test = np.arange(0.0, steps, 0.01)[:, np.newaxis]
+y_1 = regr_1.predict(X_test)
+y_2 = regr_2.predict(X_test)
+y_3=regr_3.predict(X_test)
+
+# Plot the results
+plt.figure()
+plt.scatter(X, distance_list, s=2.5, c="black", label="data")
+plt.plot(X_test, y_1, color="red",
+         label="max_depth=2", linewidth=2)
+plt.plot(X_test, y_2, color="green", label="max_depth=5", linewidth=2)
+plt.plot(X_test, y_3, color="m", label="max_depth=7", linewidth=2)
+
+plt.xlabel("Data")
+plt.ylabel("Darget")
+plt.title("Decision Tree Regression")
+plt.legend()
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    - -

    This process is then repeated for the subtree rooted at the new -node. -

    @@ -365,7 +453,7 @@ node.

  • 16
  • 17
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs008.html b/doc/pub/week46/html/._week46-bs008.html index 6abaa583f..8e74a2761 100644 --- a/doc/pub/week46/html/._week46-bs008.html +++ b/doc/pub/week46/html/._week46-bs008.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,19 +319,28 @@ MathJax.Hub.Config({

     

     

     

    -

    How do we set it up?

    +

    Building a tree, regression

    -

    In simplified terms, the process of training a decision tree and -predicting the target features of query instances is as follows: +

    There are mainly two steps

    +
      +
    1. We split the predictor space (the set of possible values \( x_1,x_2,\dots, x_p \)) into \( J \) distinct and non-non-overlapping regions, \( R_1,R_2,\dots,R_J \).
      2. +
    2. For every observation that falls into the region \( R_j \) , we make the same prediction, which is simply the mean of the response values for the training observations in \( R_j \).
      3. +
    +

    How do we construct the regions \( R_1,\dots,R_J \)? In theory, the +regions could have any shape. However, we choose to divide the +predictor space into high-dimensional rectangles, or boxes, for +simplicity and for ease of interpretation of the resulting predictive +model. The goal is to find boxes \( R_1,\dots,R_J \) that minimize the +MSE, given by

    -
      -
    1. Present a dataset containing of a number of training instances characterized by a number of descriptive features and a target feature
      2. -
    2. Train the decision tree model by continuously splitting the target feature along the values of the descriptive features using a measure of information gain during the training process
      3. -
    3. Grow the tree until we accomplish a stopping criteria create leaf nodes which represent the predictions we want to make for new query instances
      4. -
    4. Show query instances to the tree and run down the tree until we arrive at leaf nodes
      5. -
    -

    Then we are essentially done!

    +$$ +\sum_{j=1}^J\sum_{i\in R_j}(y_i-\overline{y}_{R_j})^2, +$$ + +

    where \( \overline{y}_{R_j} \) is the mean response for the training observations +within box \( j \). +

    @@ -366,7 +365,7 @@ predicting the target features of query instances is as follows:

  • 17
  • 18
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs009.html b/doc/pub/week46/html/._week46-bs009.html index 1afb983f3..0aadbd71a 100644 --- a/doc/pub/week46/html/._week46-bs009.html +++ b/doc/pub/week46/html/._week46-bs009.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,117 +319,21 @@ MathJax.Hub.Config({

     

     

     

    -

    Decision trees and Regression

    +

    A top-down approach, recursive binary splitting

    - -
    -
    -
    -
    -
    - 
    import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.preprocessing import PolynomialFeatures
-from sklearn.linear_model import LinearRegression
-
-steps=250
-
-distance=0
-x=0
-distance_list=[]
-steps_list=[]
-while x<steps:
-    distance+=np.random.randint(-1,2)
-    distance_list.append(distance)
-    x+=1
-    steps_list.append(x)
-plt.plot(steps_list,distance_list, color='green', label="Random Walk Data")
-
-steps_list=np.asarray(steps_list)
-distance_list=np.asarray(distance_list)
-
-X=steps_list[:,np.newaxis]
-
-#Polynomial fits
-
-#Degree 2
-poly_features=PolynomialFeatures(degree=2, include_bias=False)
-X_poly=poly_features.fit_transform(X)
-
-lin_reg=LinearRegression()
-poly_fit=lin_reg.fit(X_poly,distance_list)
-b=lin_reg.coef_
-c=lin_reg.intercept_
-print ("2nd degree coefficients:")
-print ("zero power: ",c)
-print ("first power: ", b[0])
-print ("second power: ",b[1])
-
-z = np.arange(0, steps, .01)
-z_mod=b[1]*z**2+b[0]*z+c
-
-fit_mod=b[1]*X**2+b[0]*X+c
-plt.plot(z, z_mod, color='r', label="2nd Degree Fit")
-plt.title("Polynomial Regression")
-
-plt.xlabel("Steps")
-plt.ylabel("Distance")
-
-#Degree 10
-poly_features10=PolynomialFeatures(degree=10, include_bias=False)
-X_poly10=poly_features10.fit_transform(X)
-
-poly_fit10=lin_reg.fit(X_poly10,distance_list)
-
-y_plot=poly_fit10.predict(X_poly10)
-plt.plot(X, y_plot, color='black', label="10th Degree Fit")
-
-plt.legend()
-plt.show()
-
-
-#Decision Tree Regression
-from sklearn.tree import DecisionTreeRegressor
-regr_1=DecisionTreeRegressor(max_depth=2)
-regr_2=DecisionTreeRegressor(max_depth=5)
-regr_3=DecisionTreeRegressor(max_depth=7)
-regr_1.fit(X, distance_list)
-regr_2.fit(X, distance_list)
-regr_3.fit(X, distance_list)
-
-X_test = np.arange(0.0, steps, 0.01)[:, np.newaxis]
-y_1 = regr_1.predict(X_test)
-y_2 = regr_2.predict(X_test)
-y_3=regr_3.predict(X_test)
-
-# Plot the results
-plt.figure()
-plt.scatter(X, distance_list, s=2.5, c="black", label="data")
-plt.plot(X_test, y_1, color="red",
-         label="max_depth=2", linewidth=2)
-plt.plot(X_test, y_2, color="green", label="max_depth=5", linewidth=2)
-plt.plot(X_test, y_3, color="m", label="max_depth=7", linewidth=2)
-
-plt.xlabel("Data")
-plt.ylabel("Darget")
-plt.title("Decision Tree Regression")
-plt.legend()
-plt.show()
-
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    +

    Unfortunately, it is computationally infeasible to consider every +possible partition of the feature space into \( J \) boxes. The common +strategy is to take a top-down approach +

    +

    The approach is top-down because it begins at the top of the tree (all +observations belong to a single region) and then successively splits +the predictor space; each split is indicated via two new branches +further down on the tree. It is greedy because at each step of the +tree-building process, the best split is made at that particular step, +rather than looking ahead and picking a split that will lead to a +better tree in some future step. +

    @@ -465,7 +359,7 @@ plt.show()

  • 18
  • 19
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs010.html b/doc/pub/week46/html/._week46-bs010.html index 1748a8188..3c53febc7 100644 --- a/doc/pub/week46/html/._week46-bs010.html +++ b/doc/pub/week46/html/._week46-bs010.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,27 +319,52 @@ MathJax.Hub.Config({

     

     

     

    -

    Building a tree, regression

    +

    Making a tree

    -

    There are mainly two steps

    -
      -
    1. We split the predictor space (the set of possible values \( x_1,x_2,\dots, x_p \)) into \( J \) distinct and non-non-overlapping regions, \( R_1,R_2,\dots,R_J \).
      2. -
    2. For every observation that falls into the region \( R_j \) , we make the same prediction, which is simply the mean of the response values for the training observations in \( R_j \).
      3. -
    -

    How do we construct the regions \( R_1,\dots,R_J \)? In theory, the -regions could have any shape. However, we choose to divide the -predictor space into high-dimensional rectangles, or boxes, for -simplicity and for ease of interpretation of the resulting predictive -model. The goal is to find boxes \( R_1,\dots,R_J \) that minimize the -MSE, given by +

    In order to implement the recursive binary splitting we start by selecting +the predictor \( x_j \) and a cutpoint \( s \) that splits the predictor space into two regions \( R_1 \) and \( R_2 \) +

    +$$ +\left\{X\vert x_j < s\right\}, +$$ + +

    and

    +$$ +\left\{X\vert x_j \geq s\right\}, +$$ + +

    so that we obtain the lowest MSE, that is

    +$$ +\sum_{i:x_i\in R_j}(y_i-\overline{y}_{R_1})^2+\sum_{i:x_i\in R_2}(y_i-\overline{y}_{R_2})^2, +$$ + +

    which we want to minimize by considering all predictors +\( x_1,x_2,\dots,x_p \). We consider also all possible values of \( s \) for +each predictor. These values could be determined by randomly assigned +numbers or by starting at the midpoint and then proceed till we find +an optimal value.

    -$$ -\sum_{j=1}^J\sum_{i\in R_j}(y_i-\overline{y}_{R_j})^2, -$$ +

    For any \( j \) and \( s \), we define the pair of half-planes where +\( \overline{y}_{R_1} \) is the mean response for the training +observations in \( R_1(j,s) \), and \( \overline{y}_{R_2} \) is the mean +response for the training observations in \( R_2(j,s) \). +

    -

    where \( \overline{y}_{R_j} \) is the mean response for the training observations -within box \( j \). +

    Finding the values of \( j \) and \( s \) that minimize the above equation can be +done quite quickly, especially when the number of features \( p \) is not +too large. +

    + +

    Next, we repeat the process, looking +for the best predictor and best cutpoint in order to split the data +further so as to minimize the MSE within each of the resulting +regions. However, this time, instead of splitting the entire predictor +space, we split one of the two previously identified regions. We now +have three regions. Again, we look to split one of these three regions +further, so as to minimize the MSE. The process continues until a +stopping criterion is reached; for instance, we may continue until no +region contains more than five observations.

    @@ -377,7 +392,7 @@ within box \( j \).

  • 19
  • 20
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs011.html b/doc/pub/week46/html/._week46-bs011.html index 1f7b790d4..449efd1fa 100644 --- a/doc/pub/week46/html/._week46-bs011.html +++ b/doc/pub/week46/html/._week46-bs011.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -328,23 +318,25 @@ MathJax.Hub.Config({

     

     

     

    - -

    A top-down approach, recursive binary splitting

    + +

    Pruning the tree

    -

    Unfortunately, it is computationally infeasible to consider every -possible partition of the feature space into \( J \) boxes. The common -strategy is to take a top-down approach +

    The above procedure is rather straightforward, but leads often to +overfitting and unnecessarily large and complicated trees. The basic +idea is to grow a large tree \( T_0 \) and then prune it back in order to +obtain a subtree. A smaller tree with fewer splits (fewer regions) can +lead to smaller variance and better interpretation at the cost of a +little more bias.

    -

    The approach is top-down because it begins at the top of the tree (all -observations belong to a single region) and then successively splits -the predictor space; each split is indicated via two new branches -further down on the tree. It is greedy because at each step of the -tree-building process, the best split is made at that particular step, -rather than looking ahead and picking a split that will lead to a -better tree in some future step. +

    The so-called Cost complexity pruning algorithm gives us a +way to do just this. Rather than considering every possible subtree, +we consider a sequence of trees indexed by a nonnegative tuning +parameter \( \alpha \).

    +

    Read more at the following Scikit-Learn link on pruning.

    +

    diff --git a/doc/pub/week46/html/._week46-bs012.html b/doc/pub/week46/html/._week46-bs012.html index 7d9a465e7..d3190b2f9 100644 --- a/doc/pub/week46/html/._week46-bs012.html +++ b/doc/pub/week46/html/._week46-bs012.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,52 +319,34 @@ MathJax.Hub.Config({

     

     

     

    -

    Making a tree

    +

    Cost complexity pruning

    -

    In order to implement the recursive binary splitting we start by selecting -the predictor \( x_j \) and a cutpoint \( s \) that splits the predictor space into two regions \( R_1 \) and \( R_2 \) -

    +

    For each value of \( \alpha \) there corresponds a subtree \( T \in T_0 \) such that

    $$ -\left\{X\vert x_j < s\right\}, +\sum_{m=1}^{\overline{T}}\sum_{i:x_i\in R_m}(y_i-\overline{y}_{R_m})^2+\alpha\overline{T}, $$ -

    and

    -$$ -\left\{X\vert x_j \geq s\right\}, -$$ - -

    so that we obtain the lowest MSE, that is

    -$$ -\sum_{i:x_i\in R_j}(y_i-\overline{y}_{R_1})^2+\sum_{i:x_i\in R_2}(y_i-\overline{y}_{R_2})^2, -$$ - -

    which we want to minimize by considering all predictors -\( x_1,x_2,\dots,x_p \). We consider also all possible values of \( s \) for -each predictor. These values could be determined by randomly assigned -numbers or by starting at the midpoint and then proceed till we find -an optimal value. +

    is as small as possible. Here \( \overline{T} \) is +the number of terminal nodes of the tree \( T \) , \( R_m \) is the +rectangle (i.e. the subset of predictor space) corresponding to the \( m \)-th terminal node.

    -

    For any \( j \) and \( s \), we define the pair of half-planes where -\( \overline{y}_{R_1} \) is the mean response for the training -observations in \( R_1(j,s) \), and \( \overline{y}_{R_2} \) is the mean -response for the training observations in \( R_2(j,s) \). +

    The tuning parameter \( \alpha \) controls a trade-off between the subtree’s +complexity and its fit to the training data. When \( \alpha = 0 \), then the +subtree \( T \) will simply equal \( T_0 \), +because then the above equation just measures the +training error. +However, as \( \alpha \) increases, there is a price to pay for +having a tree with many terminal nodes. The above equation will +tend to be minimized for a smaller subtree.

    -

    Finding the values of \( j \) and \( s \) that minimize the above equation can be -done quite quickly, especially when the number of features \( p \) is not -too large. -

    - -

    Next, we repeat the process, looking -for the best predictor and best cutpoint in order to split the data -further so as to minimize the MSE within each of the resulting -regions. However, this time, instead of splitting the entire predictor -space, we split one of the two previously identified regions. We now -have three regions. Again, we look to split one of these three regions -further, so as to minimize the MSE. The process continues until a -stopping criterion is reached; for instance, we may continue until no -region contains more than five observations. +

    It turns out that as we increase \( \alpha \) from zero +branches get pruned from the tree in a nested and predictable fashion, +so obtaining the whole sequence of subtrees as a function of \( \alpha \) is +easy. We can select a value of \( \alpha \) using a validation set or using +cross-validation. We then return to the full data set and obtain the +subtree corresponding to \( \alpha \).

    @@ -402,7 +374,7 @@ region contains more than five observations.

  • 21
  • 22
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs013.html b/doc/pub/week46/html/._week46-bs013.html index 040ddab41..39f0958b7 100644 --- a/doc/pub/week46/html/._week46-bs013.html +++ b/doc/pub/week46/html/._week46-bs013.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -328,24 +318,27 @@ MathJax.Hub.Config({

     

     

     

    - -

    Pruning the tree

    + +

    Schematic Regression Procedure

    -

    The above procedure is rather straightforward, but leads often to -overfitting and unnecessarily large and complicated trees. The basic -idea is to grow a large tree \( T_0 \) and then prune it back in order to -obtain a subtree. A smaller tree with fewer splits (fewer regions) can -lead to smaller variance and better interpretation at the cost of a -little more bias. -

    +
    +
    + -

    The so-called Cost complexity pruning algorithm gives us a -way to do just this. Rather than considering every possible subtree, -we consider a sequence of trees indexed by a nonnegative tuning -parameter \( \alpha \). -

    +
      +
    1. Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
      2. +
    2. Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of \( \alpha \).
      3. +
    3. Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we:
      4. +
        +
      • repeat steps 1 and 2 on all but the \( k \)-th fold of the training data.
        • +
      • Then we valuate the mean squared prediction error on the data in the left-out \( k \)-th fold, as a function of \( \alpha \).
        • +
      • Finally we average the results for each value of \( \alpha \), and pick \( \alpha \) to minimize the average error.
        • +
      +
    4. Return the subtree from Step 2 that corresponds to the chosen value of \( \alpha \).
      5. +
    +
    +
    -

    Read more at the following Scikit-Learn link on pruning.

    @@ -372,7 +365,7 @@ parameter \( \alpha \).

  • 22
  • 23
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs014.html b/doc/pub/week46/html/._week46-bs014.html index 746560848..a37a43930 100644 --- a/doc/pub/week46/html/._week46-bs014.html +++ b/doc/pub/week46/html/._week46-bs014.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,34 +319,20 @@ MathJax.Hub.Config({

     

     

     

    -

    Cost complexity pruning

    +

    A Classification Tree

    -

    For each value of \( \alpha \) there corresponds a subtree \( T \in T_0 \) such that

    -$$ -\sum_{m=1}^{\overline{T}}\sum_{i:x_i\in R_m}(y_i-\overline{y}_{R_m})^2+\alpha\overline{T}, -$$ - -

    is as small as possible. Here \( \overline{T} \) is -the number of terminal nodes of the tree \( T \) , \( R_m \) is the -rectangle (i.e. the subset of predictor space) corresponding to the \( m \)-th terminal node. -

    - -

    The tuning parameter \( \alpha \) controls a trade-off between the subtree’s -complexity and its fit to the training data. When \( \alpha = 0 \), then the -subtree \( T \) will simply equal \( T_0 \), -because then the above equation just measures the -training error. -However, as \( \alpha \) increases, there is a price to pay for -having a tree with many terminal nodes. The above equation will -tend to be minimized for a smaller subtree. -

    - -

    It turns out that as we increase \( \alpha \) from zero -branches get pruned from the tree in a nested and predictable fashion, -so obtaining the whole sequence of subtrees as a function of \( \alpha \) is -easy. We can select a value of \( \alpha \) using a validation set or using -cross-validation. We then return to the full data set and obtain the -subtree corresponding to \( \alpha \). +

    A classification tree is very similar to a regression tree, except +that it is used to predict a qualitative response rather than a +quantitative one. Recall that for a regression tree, the predicted +response for an observation is given by the mean response of the +training observations that belong to the same terminal node. In +contrast, for a classification tree, we predict that each observation +belongs to the most commonly occurring class of training observations +in the region to which it belongs. In interpreting the results of a +classification tree, we are often interested not only in the class +prediction corresponding to a particular terminal node region, but +also in the class proportions among the training observations that +fall into that region.

    @@ -384,7 +360,7 @@ subtree corresponding to \( \alpha \).

  • 23
  • 24
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs015.html b/doc/pub/week46/html/._week46-bs015.html index 167fa006c..75da7b1c1 100644 --- a/doc/pub/week46/html/._week46-bs015.html +++ b/doc/pub/week46/html/._week46-bs015.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,26 +319,26 @@ MathJax.Hub.Config({

     

     

     

    -

    Schematic Regression Procedure

    +

    Growing a classification tree

    -
    -
    - - -
      -
    1. Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
      2. -
    2. Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of \( \alpha \).
      3. -
    3. Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we:
      4. -
        -
      • repeat steps 1 and 2 on all but the \( k \)-th fold of the training data.
        • -
      • Then we valuate the mean squared prediction error on the data in the left-out \( k \)-th fold, as a function of \( \alpha \).
        • -
      • Finally we average the results for each value of \( \alpha \), and pick \( \alpha \) to minimize the average error.
        • -
      -
    4. Return the subtree from Step 2 that corresponds to the chosen value of \( \alpha \).
      5. -
    -
    -
    +

    The task of growing a +classification tree is quite similar to the task of growing a +regression tree. Just as in the regression setting, we use recursive +binary splitting to grow a classification tree. However, in the +classification setting, the MSE cannot be used as a criterion for making +the binary splits. A natural alternative to MSE is the classification +error rate. Since we plan to assign an observation in a given region +to the most commonly occurring error rate class of training +observations in that region, the classification error rate is simply +the fraction of the training observations in that region that do not +belong to the most common class. +

    +

    When building a classification tree, either the Gini index or the +entropy are typically used to evaluate the quality of a particular +split, since these two approaches are more sensitive to node purity +than is the classification error rate. +

    @@ -375,7 +365,7 @@ MathJax.Hub.Config({

  • 24
  • 25
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs016.html b/doc/pub/week46/html/._week46-bs016.html index 041c404a1..71f0e8fbf 100644 --- a/doc/pub/week46/html/._week46-bs016.html +++ b/doc/pub/week46/html/._week46-bs016.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,22 +319,49 @@ MathJax.Hub.Config({

     

     

     

    -

    A Classification Tree

    +

    Classification tree, how to split nodes

    -

    A classification tree is very similar to a regression tree, except -that it is used to predict a qualitative response rather than a -quantitative one. Recall that for a regression tree, the predicted -response for an observation is given by the mean response of the -training observations that belong to the same terminal node. In -contrast, for a classification tree, we predict that each observation -belongs to the most commonly occurring class of training observations -in the region to which it belongs. In interpreting the results of a -classification tree, we are often interested not only in the class -prediction corresponding to a particular terminal node region, but -also in the class proportions among the training observations that -fall into that region. +

    If our targets are the outcome of a classification process that takes +for example \( k=1,2,\dots,K \) values, the only thing we need to think of +is to set up the splitting criteria for each node.

    +

    We define a PDF \( p_{mk} \) that represents the number of observations of +a class \( k \) in a region \( R_m \) with \( N_m \) observations. We represent +this likelihood function in terms of the proportion \( I(y_i=k) \) of +observations of this class in the region \( R_m \) as +

    + +$$ +p_{mk} = \frac{1}{N_m}\sum_{x_i\in R_m}I(y_i=k). +$$ + +

    We let \( p_{mk} \) represent the majority class of observations in region +\( m \). The three most common ways of splitting a node are given by +

    + +
      +
    • Misclassification error
      • +
    +$$ +p_{mk} = \frac{1}{N_m}\sum_{x_i\in R_m}I(y_i\ne k) = 1-p_{mk}. +$$ + +
      +
    • Gini index \( g \)
      • +
    +$$ +g = \sum_{k=1}^K p_{mk}(1-p_{mk}). +$$ + +
      +
    • Information entropy or just entropy \( s \)
      • +
    +$$ +s = -\sum_{k=1}^K p_{mk}\log{p_{mk}}. +$$ + +

      @@ -370,7 +387,7 @@ fall into that region.
    • 25
    • 26
    • ...
      • -
    • 65
      • +
    • 63
    • »
    diff --git a/doc/pub/week46/html/._week46-bs017.html b/doc/pub/week46/html/._week46-bs017.html index 0f27d8e2e..9f724738d 100644 --- a/doc/pub/week46/html/._week46-bs017.html +++ b/doc/pub/week46/html/._week46-bs017.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,26 +319,62 @@ MathJax.Hub.Config({

     

     

     

    -

    Growing a classification tree

    +

    Visualizing the Tree, Classification

    -

    The task of growing a -classification tree is quite similar to the task of growing a -regression tree. Just as in the regression setting, we use recursive -binary splitting to grow a classification tree. However, in the -classification setting, the MSE cannot be used as a criterion for making -the binary splits. A natural alternative to MSE is the classification -error rate. Since we plan to assign an observation in a given region -to the most commonly occurring error rate class of training -observations in that region, the classification error rate is simply -the fraction of the training observations in that region that do not -belong to the most common class. -

    + +
    +
    +
    +
    +
    + 
    import os
+from sklearn.datasets import load_breast_cancer
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import confusion_matrix
+from sklearn.tree import export_graphviz
+
+from IPython.display import Image 
+from pydot import graph_from_dot_data
+import pandas as pd
+import numpy as np
+
+
+cancer = load_breast_cancer()
+X = pd.DataFrame(cancer.data, columns=cancer.feature_names)
+print(X)
+y = pd.Categorical.from_codes(cancer.target, cancer.target_names)
+y = pd.get_dummies(y)
+print(y)
+X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)
+tree_clf = DecisionTreeClassifier(max_depth=5)
+tree_clf.fit(X_train, y_train)
+
+export_graphviz(
+    tree_clf,
+    out_file="DataFiles/cancer.dot",
+    feature_names=cancer.feature_names,
+    class_names=cancer.target_names,
+    rounded=True,
+    filled=True
+)
+cmd = 'dot -Tpng DataFiles/cancer.dot -o DataFiles/cancer.png'
+os.system(cmd)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    -

    When building a classification tree, either the Gini index or the -entropy are typically used to evaluate the quality of a particular -split, since these two approaches are more sensitive to node purity -than is the classification error rate. -

    @@ -375,7 +401,7 @@ than is the classification error rate.

  • 26
  • 27
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs018.html b/doc/pub/week46/html/._week46-bs018.html index 862c9be13..8a0dd6278 100644 --- a/doc/pub/week46/html/._week46-bs018.html +++ b/doc/pub/week46/html/._week46-bs018.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,47 +319,52 @@ MathJax.Hub.Config({

     

     

     

    -

    Classification tree, how to split nodes

    +

    Visualizing the Tree, The Moons

    -

    If our targets are the outcome of a classification process that takes -for example \( k=1,2,\dots,K \) values, the only thing we need to think of -is to set up the splitting criteria for each node. -

    + +
    +
    +
    +
    +
    + 
    # Common imports
+import numpy as np
+from sklearn.model_selection import  train_test_split 
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.datasets import make_moons
+from sklearn.tree import export_graphviz
+from pydot import graph_from_dot_data
+import pandas as pd
+import os
 
-
    We define a PDF \( p_{mk} \) that represents the number of observations of
-a class \( k \) in a region \( R_m \) with \( N_m \) observations. We represent
-this likelihood function in terms of the proportion \( I(y_i=k) \) of
-observations of this class in the region \( R_m \) as
-
    
+np.random.seed(42)
+X, y = make_moons(n_samples=100, noise=0.25, random_state=53)
+X_train, X_test, y_train, y_test = train_test_split(X,y,random_state=0)
+tree_clf = DecisionTreeClassifier(max_depth=5)
+tree_clf.fit(X_train, y_train)
 
-$$
-p_{mk} = \frac{1}{N_m}\sum_{x_i\in R_m}I(y_i=k).
-$$
-
-
    We let \( p_{mk} \) represent the majority class of observations in region
-\( m \). The three most common ways of splitting a node are given by
-
    
-
-
      
-
    •  Misclassification error
      •  
-
    
-$$
-p_{mk} = \frac{1}{N_m}\sum_{x_i\in R_m}I(y_i\ne k) = 1-p_{mk}.
-$$
-
-
      
-
    •  Gini index \( g \)
      • 
-
    
-$$
-g = \sum_{k=1}^K p_{mk}(1-p_{mk}).
-$$
-
-
      
-
    •  Information entropy or just entropy \( s \)
      • 
-
    
-$$
-s = -\sum_{k=1}^K p_{mk}\log{p_{mk}}.
-$$
+export_graphviz(
+    tree_clf,
+    out_file="DataFiles/moons.dot",
+    rounded=True,
+    filled=True
+)
+cmd = 'dot -Tpng DataFiles/moons.dot -o DataFiles/moons.png'
+os.system(cmd)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +

    @@ -397,7 +392,7 @@ $$

  • 27
  • 28
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs019.html b/doc/pub/week46/html/._week46-bs019.html index 2847e9a98..40b191e9e 100644 --- a/doc/pub/week46/html/._week46-bs019.html +++ b/doc/pub/week46/html/._week46-bs019.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,7 +319,10 @@ MathJax.Hub.Config({

     

     

     

    -

    Visualizing the Tree, Classification

    +

    Other ways of visualizing the trees

    + +

    Scikit-Learn has also another way to visualize the trees which is very useful, here with the Iris data.

    +
    @@ -337,39 +330,13 @@ MathJax.Hub.Config({
    - 
    import os
-from sklearn.datasets import load_breast_cancer
-from sklearn.tree import DecisionTreeClassifier
-from sklearn.model_selection import train_test_split
-from sklearn.metrics import confusion_matrix
-from sklearn.tree import export_graphviz
-
-from IPython.display import Image 
-from pydot import graph_from_dot_data
-import pandas as pd
-import numpy as np
-
-
-cancer = load_breast_cancer()
-X = pd.DataFrame(cancer.data, columns=cancer.feature_names)
-print(X)
-y = pd.Categorical.from_codes(cancer.target, cancer.target_names)
-y = pd.get_dummies(y)
-print(y)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)
-tree_clf = DecisionTreeClassifier(max_depth=5)
-tree_clf.fit(X_train, y_train)
-
-export_graphviz(
-    tree_clf,
-    out_file="DataFiles/cancer.dot",
-    feature_names=cancer.feature_names,
-    class_names=cancer.target_names,
-    rounded=True,
-    filled=True
-)
-cmd = 'dot -Tpng DataFiles/cancer.dot -o DataFiles/cancer.png'
-os.system(cmd)
+  
    from sklearn.datasets import load_iris
+from sklearn import tree
+X, y = load_iris(return_X_y=True)
+tree_clf = tree.DecisionTreeClassifier()
+tree_clf = tree_clf.fit(X, y)
+# and then plot the tree
+tree.plot_tree(tree_clf)
    @@ -411,7 +378,7 @@ os.system(cmd)
  • 28
  • 29
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs020.html b/doc/pub/week46/html/._week46-bs020.html index 07343c15a..273f54bc4 100644 --- a/doc/pub/week46/html/._week46-bs020.html +++ b/doc/pub/week46/html/._week46-bs020.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,7 +319,12 @@ MathJax.Hub.Config({

     

     

     

    -

    Visualizing the Tree, The Moons

    +

    Printing out as text

    + +

    Alternatively, the tree can also be exported in textual format with the function exporttext. +This method doesn’t require the installation of external libraries and is more compact: +

    +
    @@ -337,30 +332,14 @@ MathJax.Hub.Config({
    - 
    # Common imports
-import numpy as np
-from sklearn.model_selection import  train_test_split 
+  
    from sklearn.datasets import load_iris
 from sklearn.tree import DecisionTreeClassifier
-from sklearn.datasets import make_moons
-from sklearn.tree import export_graphviz
-from pydot import graph_from_dot_data
-import pandas as pd
-import os
-
-np.random.seed(42)
-X, y = make_moons(n_samples=100, noise=0.25, random_state=53)
-X_train, X_test, y_train, y_test = train_test_split(X,y,random_state=0)
-tree_clf = DecisionTreeClassifier(max_depth=5)
-tree_clf.fit(X_train, y_train)
-
-export_graphviz(
-    tree_clf,
-    out_file="DataFiles/moons.dot",
-    rounded=True,
-    filled=True
-)
-cmd = 'dot -Tpng DataFiles/moons.dot -o DataFiles/moons.png'
-os.system(cmd)
+from sklearn.tree import export_text
+iris = load_iris()
+decision_tree = DecisionTreeClassifier(random_state=0, max_depth=2)
+decision_tree = decision_tree.fit(iris.data, iris.target)
+r = export_text(decision_tree, feature_names=iris['feature_names'])
+print(r)
    @@ -402,7 +381,7 @@ os.system(cmd)
  • 29
  • 30
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs021.html b/doc/pub/week46/html/._week46-bs021.html index f24c735bc..52b5197c3 100644 --- a/doc/pub/week46/html/._week46-bs021.html +++ b/doc/pub/week46/html/._week46-bs021.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,39 +319,18 @@ MathJax.Hub.Config({

     

     

     

    -

    Other ways of visualizing the trees

    - -

    Scikit-Learn has also another way to visualize the trees which is very useful, here with the Iris data.

    - - - -
    -
    -
    -
    -
    - 
    from sklearn.datasets import load_iris
-from sklearn import tree
-X, y = load_iris(return_X_y=True)
-tree_clf = tree.DecisionTreeClassifier()
-tree_clf = tree_clf.fit(X, y)
-# and then plot the tree
-tree.plot_tree(tree_clf) 
-
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    +

    Algorithms for Setting up Decision Trees

    +

    Two algorithms stand out in the set up of decision trees:

    +
      +
    1. The CART (Classification And Regression Tree) algorithm for both classification and regression
      2. +
    2. The ID3 algorithm based on the computation of the information gain for classification
      3. +
    +

    We discuss both algorithms with applications here. The popular library +Scikit-Learn uses the CART algorithm. For classification problems +you can use either the gini index or the entropy to split a tree +in two branches. +

    @@ -388,7 +357,7 @@ tree.plot_tree(tree_clf)

  • 30
  • 31
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs022.html b/doc/pub/week46/html/._week46-bs022.html index d638d6cb1..eb7c29b32 100644 --- a/doc/pub/week46/html/._week46-bs022.html +++ b/doc/pub/week46/html/._week46-bs022.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,42 +319,30 @@ MathJax.Hub.Config({

     

     

     

    -

    Printing out as text

    +

    The CART algorithm for Classification

    -

    Alternatively, the tree can also be exported in textual format with the function exporttext. -This method doesn’t require the installation of external libraries and is more compact: +

    For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \). +This could be for example a threshold set by a number below a certain circumference of a malign tumor.

    +

    How do we find these two quantities? +We search for the pair \( (k,t_k) \) that produces the purest subset using for example the gini factor \( G \). +The cost function it tries to minimize is then +

    +$$ +C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}}, +$$ - -
    -
    -
    -
    -
    - 
    from sklearn.datasets import load_iris
-from sklearn.tree import DecisionTreeClassifier
-from sklearn.tree import export_text
-iris = load_iris()
-decision_tree = DecisionTreeClassifier(random_state=0, max_depth=2)
-decision_tree = decision_tree.fit(iris.data, iris.target)
-r = export_text(decision_tree, feature_names=iris['feature_names'])
-print(r)
-
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    +

    where \( G_{\mathrm{left/right}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \) + is the number of instances in the left/right subset +

    +

    Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets +and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the +\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other +hyperparameters control additional stopping conditions such as the \( min\_samples\_split \), +\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \). +

    @@ -391,7 +369,7 @@ r = export_text(decision_tree, feature_names

  • 31
  • 32
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs023.html b/doc/pub/week46/html/._week46-bs023.html index 32687de86..6eb3c1c01 100644 --- a/doc/pub/week46/html/._week46-bs023.html +++ b/doc/pub/week46/html/._week46-bs023.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,17 +319,29 @@ MathJax.Hub.Config({

     

     

     

    -

    Algorithms for Setting up Decision Trees

    +

    The CART algorithm for Regression

    -

    Two algorithms stand out in the set up of decision trees:

    -
      -
    1. The CART (Classification And Regression Tree) algorithm for both classification and regression
      2. -
    2. The ID3 algorithm based on the computation of the information gain for classification
      3. -
    -

    We discuss both algorithms with applications here. The popular library -Scikit-Learn uses the CART algorithm. For classification problems -you can use either the gini index or the entropy to split a tree -in two branches. +

    The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the +training set in a way that minimizes say the gini or entropy impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now +

    +$$ +C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}. +$$ + +

    Here the MSE for a specific node is defined as

    +$$ +\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2, +$$ + +

    with

    +$$ +\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i, +$$ + +

    the mean value of all observations in a specific node.

    + +

    Without any regularization, the regression task for decision trees, +just like for classification tasks, is prone to overfitting.

    @@ -367,7 +369,7 @@ in two branches.

  • 32
  • 33
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs024.html b/doc/pub/week46/html/._week46-bs024.html index 089f86464..227cee7c4 100644 --- a/doc/pub/week46/html/._week46-bs024.html +++ b/doc/pub/week46/html/._week46-bs024.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,29 +319,12 @@ MathJax.Hub.Config({

     

     

     

    -

    The CART algorithm for Classification

    +

    Why binary splits?

    -

    For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \). -This could be for example a threshold set by a number below a certain circumference of a malign tumor. -

    - -

    How do we find these two quantities? -We search for the pair \( (k,t_k) \) that produces the purest subset using for example the gini factor \( G \). -The cost function it tries to minimize is then -

    -$$ -C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}}, -$$ - -

    where \( G_{\mathrm{left/right}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \) - is the number of instances in the left/right subset -

    - -

    Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets -and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the -\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other -hyperparameters control additional stopping conditions such as the \( min\_samples\_split \), -\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \). +

    It is custom to split to a tree uising binary splits. The reason is +that multiway splits fragment the data too quickly, leaving +insufficient data at the next level down. Multiway splits can be +achieved by a series of binary split and this is normally preferred.

    @@ -379,7 +352,7 @@ hyperparameters control additional stopping conditions such as the \( min\_sampl

  • 33
  • 34
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs025.html b/doc/pub/week46/html/._week46-bs025.html index 1c6fd8a01..9597c2365 100644 --- a/doc/pub/week46/html/._week46-bs025.html +++ b/doc/pub/week46/html/._week46-bs025.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,31 +319,23 @@ MathJax.Hub.Config({

     

     

     

    -

    The CART algorithm for Regression

    +

    Computing a Tree using the Gini Index

    -

    The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the -training set in a way that minimizes say the gini or entropy impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now -

    -$$ -C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}. -$$ - -

    Here the MSE for a specific node is defined as

    -$$ -\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2, -$$ - -

    with

    -$$ -\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i, -$$ - -

    the mean value of all observations in a specific node.

    - -

    Without any regularization, the regression task for decision trees, -just like for classification tasks, is prone to overfitting. +

    Consider the following example with attributes/features and two +possible outcomes (classes) for each attribute. Assume we wish to find some +correlations between the average grade of a student as function of the +number of hours studied and hours slept. We want also to correlate the +grade in a given course with the general trend, whether the students +recently has gotten grades below average or above.

    +

    We have three features/attributes

    +
      +
    1. Trend of average grades before present course, classified as either below or above the average grade of the whole class
      2. +
    2. The number of hours studies, classified again as either higher (more than 3 hours per day) or lower . Here we have used a standard for one \( ECTS \) which is scaled to 25-30 hours of work for a semester which lasts 18 weeks, with 15 weeks of lectures and 3 weeks for exams, assuming a total of 30 ECTS per semester.
      3. +
    3. The number of hours slept as high for more than \( 8 \) hours and below for less than 8 hours of sleep, classified again as either high or low
      4. +
    4. The final grade whether it is above or below average
      5. +

      @@ -379,7 +361,7 @@ just like for classification tasks, is prone to overfitting.
    • 34
    • 35
    • ...
      • -
    • 65
      • +
    • 63
    • »
    diff --git a/doc/pub/week46/html/._week46-bs026.html b/doc/pub/week46/html/._week46-bs026.html index 50a6296c6..1f1f403bb 100644 --- a/doc/pub/week46/html/._week46-bs026.html +++ b/doc/pub/week46/html/._week46-bs026.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,13 +319,29 @@ MathJax.Hub.Config({

     

     

     

    -

    Why binary splits?

    +

    The Table

    -

    It is custom to split to a tree uising binary splits. The reason is -that multiway splits fragment the data too quickly, leaving -insufficient data at the next level down. Multiway splits can be -achieved by a series of binary split and this is normally preferred. -

    +
    +
    + + + + + + + + + + + + + + + + +
    Grade Trend Hours slept Hours Studied Grade
    Above Low High Above
    Below High Low Below
    Above Low High Above
    Above High High Above
    Below Low High Below
    Above Low Low Below
    Below High High Below
    Below Low High Below
    Above Low Low Below
    Above High High Above
    +
    +

    @@ -362,7 +368,7 @@ achieved by a series of binary split and this is normally preferred.

  • 35
  • 36
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs027.html b/doc/pub/week46/html/._week46-bs027.html index 30f9e315f..da8264712 100644 --- a/doc/pub/week46/html/._week46-bs027.html +++ b/doc/pub/week46/html/._week46-bs027.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,23 +319,20 @@ MathJax.Hub.Config({

     

     

     

    -

    Computing a Tree using the Gini Index

    +

    Computing the various Gini Indices

    -

    Consider the following example with attributes/features and two -possible outcomes (classes) for each attribute. Assume we wish to find some -correlations between the average grade of a student as function of the -number of hours studied and hours slept. We want also to correlate the -grade in a given course with the general trend, whether the students -recently has gotten grades below average or above. +

    In computations we will translate all classes into numbers. Being +these binary classes, they can easily be split into ones and zeros.

    -

    We have three features/attributes

    -
      -
    1. Trend of average grades before present course, classified as either below or above the average grade of the whole class
      2. -
    2. The number of hours studies, classified again as either higher (more than 3 hours per day) or lower . Here we have used a standard for one \( ECTS \) which is scaled to 25-30 hours of work for a semester which lasts 18 weeks, with 15 weeks of lectures and 3 weeks for exams, assuming a total of 30 ECTS per semester.
      3. -
    3. The number of hours slept as high for more than \( 8 \) hours and below for less than 8 hours of sleep, classified again as either high or low
      4. -
    4. The final grade whether it is above or below average
      5. -
    +
    +
    + +See handwritten notes November 3 +
    +
    + +

      @@ -371,7 +358,7 @@ recently has gotten grades below average or above.
    • 36
    • 37
    • ...
      • -
    • 65
      • +
    • 63
    • »
    diff --git a/doc/pub/week46/html/._week46-bs028.html b/doc/pub/week46/html/._week46-bs028.html index 230585a1f..ef73b2e90 100644 --- a/doc/pub/week46/html/._week46-bs028.html +++ b/doc/pub/week46/html/._week46-bs028.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,29 +319,15 @@ MathJax.Hub.Config({

     

     

     

    -

    The Table

    +

    Computing the various Gini Indices, Hours slept

    + +
    +
    + +See handwritten notes November 3 +
    +
    -
    -
    - - - - - - - - - - - - - - - - -
    Grade Trend Hours slept Hours Studied Grade
    Above Low High Above
    Below High Low Below
    Above Low High Above
    Above High High Above
    Below Low High Below
    Above Low Low Below
    Below High High Below
    Below Low High Below
    Above Low Low Below
    Above High High Above
    -
    -

    @@ -378,7 +354,7 @@ MathJax.Hub.Config({

  • 37
  • 38
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs029.html b/doc/pub/week46/html/._week46-bs029.html index 602ec4d5c..b80039d4c 100644 --- a/doc/pub/week46/html/._week46-bs029.html +++ b/doc/pub/week46/html/._week46-bs029.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -329,11 +319,7 @@ MathJax.Hub.Config({

     

     

     

    -

    Computing the various Gini Indices

    - -

    In computations we will translate all classes into numbers. Being -these binary classes, they can easily be split into ones and zeros. -

    +

    Computing the various Gini Indices, Hours studied

    @@ -343,6 +329,8 @@ these binary classes, they can easily be split into ones and zeros.
    +

    For final tree, see the above handwritten notes

    +

      @@ -368,7 +356,7 @@ these binary classes, they can easily be split into ones and zeros.
    • 38
    • 39
    • ...
      • -
    • 65
      • +
    • 63
    • »
    diff --git a/doc/pub/week46/html/week46-bs.html b/doc/pub/week46/html/week46-bs.html index 39a24a971..dc7a4c9a6 100644 --- a/doc/pub/week46/html/week46-bs.html +++ b/doc/pub/week46/html/week46-bs.html @@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -257,67 +249,65 @@ MathJax.Hub.Config({
  • Plan for week 46
  • Decision trees, overarching aims
  • Basics of a tree
    • -
  • A Sketch of a Tree, Regression problem
    • -
  • A Sketch of a Tree, Classification problem
    • -
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • -
  • General Features
    • -
  • How do we set it up?
    • -
  • Decision trees and Regression
    • -
  • Building a tree, regression
    • -
  • A top-down approach, recursive binary splitting
    • -
  • Making a tree
    • -
  • Pruning the tree
    • -
  • Cost complexity pruning
    • -
  • Schematic Regression Procedure
    • -
  • A Classification Tree
    • -
  • Growing a classification tree
    • -
  • Classification tree, how to split nodes
    • -
  • Visualizing the Tree, Classification
    • -
  • Visualizing the Tree, The Moons
    • -
  • Other ways of visualizing the trees
    • -
  • Printing out as text
    • -
  • Algorithms for Setting up Decision Trees
    • -
  • The CART algorithm for Classification
    • -
  • The CART algorithm for Regression
    • -
  • Why binary splits?
    • -
  • Computing a Tree using the Gini Index
    • -
  • The Table
    • -
  • Computing the various Gini Indices
    • -
  • Computing the various Gini Indices, Hours slept
    • -
  • Computing the various Gini Indices, Hours studied
    • -
  • A possible code using Scikit-Learn
    • -
  • Further example: Computing the Gini index
    • -
  • Simple Python Code to read in Data and perform Classification
    • -
  • Computing the Gini Factor
    • -
  • Regression trees
    • -
  • Final regressor code
    • -
  • Pros and cons of trees, pros
    • -
  • Disadvantages
    • -
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • -
  • An Overview of Ensemble Methods
    • -
  • Why Voting?
    • -
  • Tossing coins
    • -
  • Standard imports first
    • -
  • Simple Voting Example, head or tail
    • -
  • Using the Voting Classifier
    • -
  • Voting and Bagging
    • -
  • Bagging
    • -
  • More bagging
    • -
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • -
  • Random forests
    • -
  • Random Forest Algorithm
    • -
  • Random Forests Compared with other Methods on the Cancer Data
    • -
  • Compare Bagging on Trees with Random Forests
    • -
  • Boosting, a Bird's Eye View
    • -
  • What is boosting? Additive Modelling/Iterative Fitting
    • -
  • Iterative Fitting, Regression and Squared-error Cost Function
    • -
  • Squared-Error Example and Iterative Fitting
    • -
  • Iterative Fitting, Classification and AdaBoost
    • -
  • Adaptive Boosting, AdaBoost
    • -
  • Building up AdaBoost
    • -
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • -
  • Basic Steps of AdaBoost
    • -
  • AdaBoost Examples
    • +
  • A typical Decision Tree with its pertinent Jargon, Classification Problem
    • +
  • General Features
    • +
  • How do we set it up?
    • +
  • Decision trees and Regression
    • +
  • Building a tree, regression
    • +
  • A top-down approach, recursive binary splitting
    • +
  • Making a tree
    • +
  • Pruning the tree
    • +
  • Cost complexity pruning
    • +
  • Schematic Regression Procedure
    • +
  • A Classification Tree
    • +
  • Growing a classification tree
    • +
  • Classification tree, how to split nodes
    • +
  • Visualizing the Tree, Classification
    • +
  • Visualizing the Tree, The Moons
    • +
  • Other ways of visualizing the trees
    • +
  • Printing out as text
    • +
  • Algorithms for Setting up Decision Trees
    • +
  • The CART algorithm for Classification
    • +
  • The CART algorithm for Regression
    • +
  • Why binary splits?
    • +
  • Computing a Tree using the Gini Index
    • +
  • The Table
    • +
  • Computing the various Gini Indices
    • +
  • Computing the various Gini Indices, Hours slept
    • +
  • Computing the various Gini Indices, Hours studied
    • +
  • A possible code using Scikit-Learn
    • +
  • Further example: Computing the Gini index
    • +
  • Simple Python Code to read in Data and perform Classification
    • +
  • Computing the Gini Factor
    • +
  • Regression trees
    • +
  • Final regressor code
    • +
  • Pros and cons of trees, pros
    • +
  • Disadvantages
    • +
  • Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
    • +
  • An Overview of Ensemble Methods
    • +
  • Why Voting?
    • +
  • Tossing coins
    • +
  • Standard imports first
    • +
  • Simple Voting Example, head or tail
    • +
  • Using the Voting Classifier
    • +
  • Voting and Bagging
    • +
  • Bagging
    • +
  • More bagging
    • +
  • Making your own Bootstrap: Changing the Level of the Decision Tree
    • +
  • Random forests
    • +
  • Random Forest Algorithm
    • +
  • Random Forests Compared with other Methods on the Cancer Data
    • +
  • Compare Bagging on Trees with Random Forests
    • +
  • Boosting, a Bird's Eye View
    • +
  • What is boosting? Additive Modelling/Iterative Fitting
    • +
  • Iterative Fitting, Regression and Squared-error Cost Function
    • +
  • Squared-Error Example and Iterative Fitting
    • +
  • Iterative Fitting, Classification and AdaBoost
    • +
  • Adaptive Boosting, AdaBoost
    • +
  • Building up AdaBoost
    • +
  • Adaptive boosting: AdaBoost, Basic Algorithm
    • +
  • Basic Steps of AdaBoost
    • +
  • AdaBoost Examples
    • @@ -372,7 +362,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 65
    • +
  • 63
  • »
    • diff --git a/doc/pub/week46/html/week46-reveal.html b/doc/pub/week46/html/week46-reveal.html index 5a792f4c7..ac658a1d3 100644 --- a/doc/pub/week46/html/week46-reveal.html +++ b/doc/pub/week46/html/week46-reveal.html @@ -277,21 +277,6 @@ given some assumptions, make predictions about the target feature value

    -
    -

    A Sketch of a Tree, Regression problem

    - -See handwritten notes November 3 - - -
    - -
    -

    A Sketch of a Tree, Classification problem

    - -See handwritten notes November 3 - -
    -

    A typical Decision Tree with its pertinent Jargon, Classification Problem

    diff --git a/doc/pub/week46/html/week46-solarized.html b/doc/pub/week46/html/week46-solarized.html index ee7edc6df..cf8b82013 100644 --- a/doc/pub/week46/html/week46-solarized.html +++ b/doc/pub/week46/html/week46-solarized.html @@ -69,14 +69,6 @@ div.toc p,a { None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -360,19 +352,6 @@ given some assumptions, make predictions about the target feature value (class) of unseen query instances.

    -









    -

    A Sketch of a Tree, Regression problem

    - -See handwritten notes November 3 - - - -









    -

    A Sketch of a Tree, Classification problem

    - -See handwritten notes November 3 - -









    A typical Decision Tree with its pertinent Jargon, Classification Problem

    diff --git a/doc/pub/week46/html/week46.html b/doc/pub/week46/html/week46.html index c55285389..a2d5c9cbc 100644 --- a/doc/pub/week46/html/week46.html +++ b/doc/pub/week46/html/week46.html @@ -146,14 +146,6 @@ div.toc p,a { None, 'decision-trees-overarching-aims'), ('Basics of a tree', 2, None, 'basics-of-a-tree'), - ('A Sketch of a Tree, Regression problem', - 2, - None, - 'a-sketch-of-a-tree-regression-problem'), - ('A Sketch of a Tree, Classification problem', - 2, - None, - 'a-sketch-of-a-tree-classification-problem'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, @@ -437,19 +429,6 @@ given some assumptions, make predictions about the target feature value (class) of unseen query instances.

    -









    -

    A Sketch of a Tree, Regression problem

    - -See handwritten notes November 3 - - - -









    -

    A Sketch of a Tree, Classification problem

    - -See handwritten notes November 3 - -









    A typical Decision Tree with its pertinent Jargon, Classification Problem

    diff --git a/doc/pub/week46/ipynb/DataFiles/cancer.dot b/doc/pub/week46/ipynb/DataFiles/cancer.dot index 5b4b48a9b..40010a184 100644 --- a/doc/pub/week46/ipynb/DataFiles/cancer.dot +++ b/doc/pub/week46/ipynb/DataFiles/cancer.dot @@ -1,57 +1,57 @@ digraph Tree { -node [shape=box, style="filled, rounded", color="black", fontname=helvetica] ; -edge [fontname=helvetica] ; -0 [label="worst perimeter <= 106.05\ngini = 0.465\nsamples = 426\nvalue = [[269, 157]\n[157, 269]]", fillcolor="#e5813908"] ; -1 [label="worst concave points <= 0.159\ngini = 0.067\nsamples = 259\nvalue = [[250, 9]\n[9, 250]]", fillcolor="#e58139db"] ; +node [shape=box, style="filled, rounded", color="black", fontname="helvetica"] ; +edge [fontname="helvetica"] ; +0 [label="worst perimeter <= 106.05\ngini = 0.465\nsamples = 426\nvalue = [[269, 157]\n[157, 269]]", fillcolor="#fefbf9"] ; +1 [label="worst concave points <= 0.159\ngini = 0.067\nsamples = 259\nvalue = [[250, 9]\n[9, 250]]", fillcolor="#e99355"] ; 0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ; -2 [label="worst concave points <= 0.135\ngini = 0.031\nsamples = 253\nvalue = [[249, 4]\n[4, 249]]", fillcolor="#e58139ee"] ; +2 [label="worst concave points <= 0.135\ngini = 0.031\nsamples = 253\nvalue = [[249, 4]\n[4, 249]]", fillcolor="#e78946"] ; 1 -> 2 ; -3 [label="radius error <= 0.643\ngini = 0.008\nsamples = 242\nvalue = [[241, 1]\n[1, 241]]", fillcolor="#e58139fb"] ; +3 [label="area error <= 48.975\ngini = 0.008\nsamples = 242\nvalue = [[241, 1]\n[1, 241]]", fillcolor="#e5833c"] ; 2 -> 3 ; -4 [label="gini = 0.0\nsamples = 239\nvalue = [[239, 0]\n[0, 239]]", fillcolor="#e58139ff"] ; +4 [label="gini = 0.0\nsamples = 239\nvalue = [[239, 0]\n[0, 239]]", fillcolor="#e58139"] ; 3 -> 4 ; -5 [label="worst symmetry <= 0.208\ngini = 0.444\nsamples = 3\nvalue = [[2, 1]\n[1, 2]]", fillcolor="#e5813913"] ; +5 [label="concave points error <= 0.016\ngini = 0.444\nsamples = 3\nvalue = [[2, 1]\n[1, 2]]", fillcolor="#fdf6f0"] ; 3 -> 5 ; -6 [label="gini = 0.0\nsamples = 1\nvalue = [[0, 1]\n[1, 0]]", fillcolor="#e58139ff"] ; +6 [label="gini = 0.0\nsamples = 1\nvalue = [[0, 1]\n[1, 0]]", fillcolor="#e58139"] ; 5 -> 6 ; -7 [label="gini = 0.0\nsamples = 2\nvalue = [[2, 0]\n[0, 2]]", fillcolor="#e58139ff"] ; +7 [label="gini = 0.0\nsamples = 2\nvalue = [[2, 0]\n[0, 2]]", fillcolor="#e58139"] ; 5 -> 7 ; -8 [label="worst texture <= 29.455\ngini = 0.397\nsamples = 11\nvalue = [[8, 3]\n[3, 8]]", fillcolor="#e581392c"] ; +8 [label="worst texture <= 29.455\ngini = 0.397\nsamples = 11\nvalue = [[8, 3]\n[3, 8]]", fillcolor="#fae9dd"] ; 2 -> 8 ; -9 [label="gini = 0.0\nsamples = 8\nvalue = [[8, 0]\n[0, 8]]", fillcolor="#e58139ff"] ; +9 [label="gini = 0.0\nsamples = 8\nvalue = [[8, 0]\n[0, 8]]", fillcolor="#e58139"] ; 8 -> 9 ; -10 [label="gini = 0.0\nsamples = 3\nvalue = [[0, 3]\n[3, 0]]", fillcolor="#e58139ff"] ; +10 [label="gini = 0.0\nsamples = 3\nvalue = [[0, 3]\n[3, 0]]", fillcolor="#e58139"] ; 8 -> 10 ; -11 [label="mean texture <= 16.22\ngini = 0.278\nsamples = 6\nvalue = [[1, 5]\n[5, 1]]", fillcolor="#e581396b"] ; +11 [label="area error <= 13.475\ngini = 0.278\nsamples = 6\nvalue = [[1, 5]\n[5, 1]]", fillcolor="#f4caac"] ; 1 -> 11 ; -12 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139ff"] ; +12 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139"] ; 11 -> 12 ; -13 [label="gini = 0.0\nsamples = 5\nvalue = [[0, 5]\n[5, 0]]", fillcolor="#e58139ff"] ; +13 [label="gini = 0.0\nsamples = 5\nvalue = [[0, 5]\n[5, 0]]", fillcolor="#e58139"] ; 11 -> 13 ; -14 [label="worst texture <= 20.645\ngini = 0.202\nsamples = 167\nvalue = [[19, 148]\n[148, 19]]", fillcolor="#e5813994"] ; +14 [label="worst texture <= 20.645\ngini = 0.202\nsamples = 167\nvalue = [[19, 148]\n[148, 19]]", fillcolor="#f0b68c"] ; 0 -> 14 [labeldistance=2.5, labelangle=-45, headlabel="False"] ; -15 [label="worst radius <= 17.74\ngini = 0.375\nsamples = 16\nvalue = [[12, 4]\n[4, 12]]", fillcolor="#e5813938"] ; +15 [label="worst concavity <= 0.318\ngini = 0.375\nsamples = 16\nvalue = [[12, 4]\n[4, 12]]", fillcolor="#f9e3d4"] ; 14 -> 15 ; -16 [label="gini = 0.0\nsamples = 11\nvalue = [[11, 0]\n[0, 11]]", fillcolor="#e58139ff"] ; +16 [label="gini = 0.0\nsamples = 11\nvalue = [[11, 0]\n[0, 11]]", fillcolor="#e58139"] ; 15 -> 16 ; -17 [label="mean texture <= 13.745\ngini = 0.32\nsamples = 5\nvalue = [[1, 4]\n[4, 1]]", fillcolor="#e5813955"] ; +17 [label="worst perimeter <= 115.95\ngini = 0.32\nsamples = 5\nvalue = [[1, 4]\n[4, 1]]", fillcolor="#f6d5bd"] ; 15 -> 17 ; -18 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139ff"] ; +18 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139"] ; 17 -> 18 ; -19 [label="gini = 0.0\nsamples = 4\nvalue = [[0, 4]\n[4, 0]]", fillcolor="#e58139ff"] ; +19 [label="gini = 0.0\nsamples = 4\nvalue = [[0, 4]\n[4, 0]]", fillcolor="#e58139"] ; 17 -> 19 ; -20 [label="mean concave points <= 0.049\ngini = 0.088\nsamples = 151\nvalue = [[7, 144]\n[144, 7]]", fillcolor="#e58139d0"] ; +20 [label="mean concave points <= 0.049\ngini = 0.088\nsamples = 151\nvalue = [[7, 144]\n[144, 7]]", fillcolor="#ea985d"] ; 14 -> 20 ; -21 [label="concave points error <= 0.01\ngini = 0.48\nsamples = 15\nvalue = [[6, 9]\n[9, 6]]", fillcolor="#e5813900"] ; +21 [label="concave points error <= 0.01\ngini = 0.48\nsamples = 15\nvalue = [[6, 9]\n[9, 6]]", fillcolor="#ffffff"] ; 20 -> 21 ; -22 [label="gini = 0.0\nsamples = 9\nvalue = [[0, 9]\n[9, 0]]", fillcolor="#e58139ff"] ; +22 [label="gini = 0.0\nsamples = 9\nvalue = [[0, 9]\n[9, 0]]", fillcolor="#e58139"] ; 21 -> 22 ; -23 [label="gini = 0.0\nsamples = 6\nvalue = [[6, 0]\n[0, 6]]", fillcolor="#e58139ff"] ; +23 [label="gini = 0.0\nsamples = 6\nvalue = [[6, 0]\n[0, 6]]", fillcolor="#e58139"] ; 21 -> 23 ; -24 [label="worst smoothness <= 0.096\ngini = 0.015\nsamples = 136\nvalue = [[1, 135]\n[135, 1]]", fillcolor="#e58139f7"] ; +24 [label="mean smoothness <= 0.079\ngini = 0.015\nsamples = 136\nvalue = [[1, 135]\n[135, 1]]", fillcolor="#e6853f"] ; 20 -> 24 ; -25 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139ff"] ; +25 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139"] ; 24 -> 25 ; -26 [label="gini = 0.0\nsamples = 135\nvalue = [[0, 135]\n[135, 0]]", fillcolor="#e58139ff"] ; +26 [label="gini = 0.0\nsamples = 135\nvalue = [[0, 135]\n[135, 0]]", fillcolor="#e58139"] ; 24 -> 26 ; } \ No newline at end of file diff --git a/doc/pub/week46/ipynb/ipynb-week46-src.tar.gz b/doc/pub/week46/ipynb/ipynb-week46-src.tar.gz index 3de4fc51af02206dc5749342a0192e41d74d9b6a..952a9b04f4be2e0fac81b227465c90856dbd3ca7 100644 GIT binary patch delta 30 mcmX^9Tk!aAL3a6W4u-8OLmSy!*%@2enOfPIx3aSws09GEE($^b delta 30 lcmX^9Tk!aAL3a6W4u&4Lz()2~cE(nArdD?5t?VoZY5}V039" - ] - }, - { - "cell_type": "markdown", - "id": "9deb4609", - "metadata": { - "editable": true - }, - "source": [ - "## A Sketch of a Tree, Classification problem\n", - "\n", - "[See handwritten notes November 3](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2022/NotesNov32022.pdf)\n", - "" - ] - }, - { - "cell_type": "markdown", - "id": "971fbe6d", + "id": "89b0fc63", "metadata": { "editable": true }, @@ -155,7 +128,7 @@ }, { "cell_type": "markdown", - "id": "3c3fee0c", + "id": "d4354730", "metadata": { "editable": true }, @@ -178,7 +151,7 @@ }, { "cell_type": "markdown", - "id": "5d7a417e", + "id": "9db7330a", "metadata": { "editable": true }, @@ -201,7 +174,7 @@ }, { "cell_type": "markdown", - "id": "98739ae8", + "id": "331ebf1d", "metadata": { "editable": true }, @@ -212,7 +185,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "2e75ab9e", + "id": "122986df", "metadata": { "collapsed": false, "editable": true @@ -313,7 +286,7 @@ }, { "cell_type": "markdown", - "id": "402d8154", + "id": "967f86f4", "metadata": { "editable": true }, @@ -335,7 +308,7 @@ }, { "cell_type": "markdown", - "id": "87283414", + "id": "58c89f85", "metadata": { "editable": true }, @@ -347,7 +320,7 @@ }, { "cell_type": "markdown", - "id": "eeb2be0a", + "id": "8d28defb", "metadata": { "editable": true }, @@ -358,7 +331,7 @@ }, { "cell_type": "markdown", - "id": "a5e0c38b", + "id": "3f6c36d6", "metadata": { "editable": true }, @@ -380,7 +353,7 @@ }, { "cell_type": "markdown", - "id": "5ade0c12", + "id": "a89a52db", "metadata": { "editable": true }, @@ -393,7 +366,7 @@ }, { "cell_type": "markdown", - "id": "71c1603e", + "id": "5feb6c75", "metadata": { "editable": true }, @@ -405,7 +378,7 @@ }, { "cell_type": "markdown", - "id": "d6925e9f", + "id": "8de30cae", "metadata": { "editable": true }, @@ -415,7 +388,7 @@ }, { "cell_type": "markdown", - "id": "9af60147", + "id": "95d5c167", "metadata": { "editable": true }, @@ -427,7 +400,7 @@ }, { "cell_type": "markdown", - "id": "1bc203e3", + "id": "c5f10599", "metadata": { "editable": true }, @@ -437,7 +410,7 @@ }, { "cell_type": "markdown", - "id": "76c77683", + "id": "ff6f03cb", "metadata": { "editable": true }, @@ -449,7 +422,7 @@ }, { "cell_type": "markdown", - "id": "118ed843", + "id": "9f031abb", "metadata": { "editable": true }, @@ -482,7 +455,7 @@ }, { "cell_type": "markdown", - "id": "f14ea816", + "id": "83f9c272", "metadata": { "editable": true }, @@ -506,7 +479,7 @@ }, { "cell_type": "markdown", - "id": "4640b5b4", + "id": "9f23f5ac", "metadata": { "editable": true }, @@ -518,7 +491,7 @@ }, { "cell_type": "markdown", - "id": "41242ad5", + "id": "a9b2646e", "metadata": { "editable": true }, @@ -530,7 +503,7 @@ }, { "cell_type": "markdown", - "id": "0805b924", + "id": "8c9f1038", "metadata": { "editable": true }, @@ -558,7 +531,7 @@ }, { "cell_type": "markdown", - "id": "4d6bda4f", + "id": "1e4cf9ca", "metadata": { "editable": true }, @@ -584,7 +557,7 @@ }, { "cell_type": "markdown", - "id": "f4ae76f5", + "id": "328198af", "metadata": { "editable": true }, @@ -607,7 +580,7 @@ }, { "cell_type": "markdown", - "id": "14644af5", + "id": "338052bc", "metadata": { "editable": true }, @@ -634,7 +607,7 @@ }, { "cell_type": "markdown", - "id": "5b850330", + "id": "c96ca06b", "metadata": { "editable": true }, @@ -653,7 +626,7 @@ }, { "cell_type": "markdown", - "id": "ad83d169", + "id": "970d1672", "metadata": { "editable": true }, @@ -665,7 +638,7 @@ }, { "cell_type": "markdown", - "id": "21982bff", + "id": "e5a9ac3c", "metadata": { "editable": true }, @@ -678,7 +651,7 @@ }, { "cell_type": "markdown", - "id": "8c9ebead", + "id": "a0e10726", "metadata": { "editable": true }, @@ -690,7 +663,7 @@ }, { "cell_type": "markdown", - "id": "56e01c7f", + "id": "f73a5a66", "metadata": { "editable": true }, @@ -700,7 +673,7 @@ }, { "cell_type": "markdown", - "id": "9c656bf3", + "id": "c6e5ec5f", "metadata": { "editable": true }, @@ -712,7 +685,7 @@ }, { "cell_type": "markdown", - "id": "91267d63", + "id": "c50e4c55", "metadata": { "editable": true }, @@ -722,7 +695,7 @@ }, { "cell_type": "markdown", - "id": "78e120b8", + "id": "34f4deed", "metadata": { "editable": true }, @@ -734,7 +707,7 @@ }, { "cell_type": "markdown", - "id": "aeac9581", + "id": "b2f791b8", "metadata": { "editable": true }, @@ -745,7 +718,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "ea1f9010", + "id": "b373a31c", "metadata": { "collapsed": false, "editable": true @@ -789,7 +762,7 @@ }, { "cell_type": "markdown", - "id": "8235025b", + "id": "f1649fd9", "metadata": { "editable": true }, @@ -800,7 +773,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "34b62194", + "id": "63e625c0", "metadata": { "collapsed": false, "editable": true @@ -835,7 +808,7 @@ }, { "cell_type": "markdown", - "id": "d89175d8", + "id": "119ea0ae", "metadata": { "editable": true }, @@ -848,7 +821,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "41d0475a", + "id": "711bdf8d", "metadata": { "collapsed": false, "editable": true @@ -866,7 +839,7 @@ }, { "cell_type": "markdown", - "id": "c1b460bb", + "id": "a80a7f27", "metadata": { "editable": true }, @@ -880,7 +853,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "6f103d31", + "id": "3aa4b27b", "metadata": { "collapsed": false, "editable": true @@ -899,7 +872,7 @@ }, { "cell_type": "markdown", - "id": "573c8314", + "id": "e155d65a", "metadata": { "editable": true }, @@ -919,7 +892,7 @@ }, { "cell_type": "markdown", - "id": "090434b9", + "id": "9f64d255", "metadata": { "editable": true }, @@ -936,7 +909,7 @@ }, { "cell_type": "markdown", - "id": "c9a39ce1", + "id": "c67ea6bd", "metadata": { "editable": true }, @@ -948,7 +921,7 @@ }, { "cell_type": "markdown", - "id": "2d631ba5", + "id": "60be0c2f", "metadata": { "editable": true }, @@ -965,7 +938,7 @@ }, { "cell_type": "markdown", - "id": "ffd719b4", + "id": "68adc691", "metadata": { "editable": true }, @@ -978,7 +951,7 @@ }, { "cell_type": "markdown", - "id": "90f1d88b", + "id": "3aa84faa", "metadata": { "editable": true }, @@ -990,7 +963,7 @@ }, { "cell_type": "markdown", - "id": "92376037", + "id": "05821fe6", "metadata": { "editable": true }, @@ -1000,7 +973,7 @@ }, { "cell_type": "markdown", - "id": "24d1cf8d", + "id": "321fb878", "metadata": { "editable": true }, @@ -1012,7 +985,7 @@ }, { "cell_type": "markdown", - "id": "a4216389", + "id": "5703ea44", "metadata": { "editable": true }, @@ -1022,7 +995,7 @@ }, { "cell_type": "markdown", - "id": "ac393440", + "id": "6b6cf145", "metadata": { "editable": true }, @@ -1034,7 +1007,7 @@ }, { "cell_type": "markdown", - "id": "d251764b", + "id": "57f159ee", "metadata": { "editable": true }, @@ -1047,7 +1020,7 @@ }, { "cell_type": "markdown", - "id": "1ab6171a", + "id": "8dba6c9f", "metadata": { "editable": true }, @@ -1062,7 +1035,7 @@ }, { "cell_type": "markdown", - "id": "d0e81ec2", + "id": "54686dd2", "metadata": { "editable": true }, @@ -1088,7 +1061,7 @@ }, { "cell_type": "markdown", - "id": "c2c41db9", + "id": "226714bc", "metadata": { "editable": true }, @@ -1116,7 +1089,7 @@ }, { "cell_type": "markdown", - "id": "8ba76cfd", + "id": "132a6df7", "metadata": { "editable": true }, @@ -1133,7 +1106,7 @@ }, { "cell_type": "markdown", - "id": "a7a4d204", + "id": "75ab3e53", "metadata": { "editable": true }, @@ -1147,7 +1120,7 @@ }, { "cell_type": "markdown", - "id": "51de5c69", + "id": "be9d82ec", "metadata": { "editable": true }, @@ -1163,7 +1136,7 @@ }, { "cell_type": "markdown", - "id": "9f4bf856", + "id": "b502bb89", "metadata": { "editable": true }, @@ -1174,7 +1147,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "020402b2", + "id": "2e5fc857", "metadata": { "collapsed": false, "editable": true @@ -1245,7 +1218,7 @@ }, { "cell_type": "markdown", - "id": "082c7f06", + "id": "fe2aa246", "metadata": { "editable": true }, @@ -1289,7 +1262,7 @@ }, { "cell_type": "markdown", - "id": "b742d7cc", + "id": "46f289da", "metadata": { "editable": true }, @@ -1300,7 +1273,7 @@ { "cell_type": "code", "execution_count": 7, - "id": "1f114a5a", + "id": "38aedbca", "metadata": { "collapsed": false, "editable": true @@ -1378,7 +1351,7 @@ }, { "cell_type": "markdown", - "id": "3ae90db1", + "id": "a6f5da59", "metadata": { "editable": true }, @@ -1396,7 +1369,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "22320738", + "id": "e51855f9", "metadata": { "collapsed": false, "editable": true @@ -1467,7 +1440,7 @@ }, { "cell_type": "markdown", - "id": "ebd0ac9f", + "id": "f6add3e5", "metadata": { "editable": true }, @@ -1478,7 +1451,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "bfdc5109", + "id": "74ecc649", "metadata": { "collapsed": false, "editable": true @@ -1496,7 +1469,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "ab8bb0be", + "id": "04024d89", "metadata": { "collapsed": false, "editable": true @@ -1511,7 +1484,7 @@ }, { "cell_type": "markdown", - "id": "b102de57", + "id": "878b4d23", "metadata": { "editable": true }, @@ -1522,7 +1495,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "3aeec95d", + "id": "3c96bff5", "metadata": { "collapsed": false, "editable": true @@ -1572,7 +1545,7 @@ { "cell_type": "code", "execution_count": 12, - "id": "638f8ac8", + "id": "527b27ca", "metadata": { "collapsed": false, "editable": true @@ -1611,7 +1584,7 @@ }, { "cell_type": "markdown", - "id": "939c2f5c", + "id": "f2a0dd48", "metadata": { "editable": true }, @@ -1635,7 +1608,7 @@ }, { "cell_type": "markdown", - "id": "121bdc13", + "id": "9f896560", "metadata": { "editable": true }, @@ -1663,7 +1636,7 @@ }, { "cell_type": "markdown", - "id": "f58b9924", + "id": "3f6f50e2", "metadata": { "editable": true }, @@ -1694,7 +1667,7 @@ }, { "cell_type": "markdown", - "id": "3369dc37", + "id": "24509012", "metadata": { "editable": true }, @@ -1710,7 +1683,7 @@ }, { "cell_type": "markdown", - "id": "247d8e1f", + "id": "15a871bc", "metadata": { "editable": true }, @@ -1734,7 +1707,7 @@ }, { "cell_type": "markdown", - "id": "418dee56", + "id": "e6d75533", "metadata": { "editable": true }, @@ -1765,7 +1738,7 @@ }, { "cell_type": "markdown", - "id": "81c7b4f6", + "id": "8ecb23d0", "metadata": { "editable": true }, @@ -1776,7 +1749,7 @@ { "cell_type": "code", "execution_count": 13, - "id": "c8635354", + "id": "b42d0a08", "metadata": { "collapsed": false, "editable": true @@ -1824,7 +1797,7 @@ }, { "cell_type": "markdown", - "id": "6945cdc1", + "id": "e3060cfd", "metadata": { "editable": true }, @@ -1835,7 +1808,7 @@ { "cell_type": "code", "execution_count": 14, - "id": "dbe486ef", + "id": "59d25264", "metadata": { "collapsed": false, "editable": true @@ -1869,7 +1842,7 @@ }, { "cell_type": "markdown", - "id": "31814f98", + "id": "f8cf0e6e", "metadata": { "editable": true }, @@ -1882,7 +1855,7 @@ { "cell_type": "code", "execution_count": 15, - "id": "20858a25", + "id": "76fd4c2d", "metadata": { "collapsed": false, "editable": true @@ -1935,7 +1908,7 @@ }, { "cell_type": "markdown", - "id": "69bde19e", + "id": "eacefe6c", "metadata": { "editable": true }, @@ -1946,7 +1919,7 @@ { "cell_type": "code", "execution_count": 16, - "id": "579fcaba", + "id": "796dfa6b", "metadata": { "collapsed": false, "editable": true @@ -1976,7 +1949,7 @@ { "cell_type": "code", "execution_count": 17, - "id": "e19aed80", + "id": "90ec162f", "metadata": { "collapsed": false, "editable": true @@ -1994,7 +1967,7 @@ { "cell_type": "code", "execution_count": 18, - "id": "5e7fff36", + "id": "e46dcb77", "metadata": { "collapsed": false, "editable": true @@ -2014,7 +1987,7 @@ { "cell_type": "code", "execution_count": 19, - "id": "12a367f0", + "id": "67a3b080", "metadata": { "collapsed": false, "editable": true @@ -2031,7 +2004,7 @@ }, { "cell_type": "markdown", - "id": "70753458", + "id": "f0d51672", "metadata": { "editable": true }, @@ -2053,7 +2026,7 @@ }, { "cell_type": "markdown", - "id": "d37bbe68", + "id": "ab182ea8", "metadata": { "editable": true }, @@ -2085,7 +2058,7 @@ }, { "cell_type": "markdown", - "id": "498956cc", + "id": "998512be", "metadata": { "editable": true }, @@ -2099,7 +2072,7 @@ { "cell_type": "code", "execution_count": 20, - "id": "452de3bb", + "id": "6ac20f8b", "metadata": { "collapsed": false, "editable": true @@ -2168,7 +2141,7 @@ }, { "cell_type": "markdown", - "id": "b7a39b9f", + "id": "c9a44ff4", "metadata": { "editable": true }, @@ -2191,7 +2164,7 @@ }, { "cell_type": "markdown", - "id": "d39a6e13", + "id": "74f9056f", "metadata": { "editable": true }, @@ -2203,7 +2176,7 @@ }, { "cell_type": "markdown", - "id": "42f8693b", + "id": "9a0166e1", "metadata": { "editable": true }, @@ -2228,7 +2201,7 @@ }, { "cell_type": "markdown", - "id": "a25fd53e", + "id": "7e5dd3c9", "metadata": { "editable": true }, @@ -2254,7 +2227,7 @@ }, { "cell_type": "markdown", - "id": "c72f80ca", + "id": "b2476d94", "metadata": { "editable": true }, @@ -2265,7 +2238,7 @@ { "cell_type": "code", "execution_count": 21, - "id": "9b20111b", + "id": "0a56d5de", "metadata": { "collapsed": false, "editable": true @@ -2337,7 +2310,7 @@ }, { "cell_type": "markdown", - "id": "a2a69156", + "id": "ef32420e", "metadata": { "editable": true }, @@ -2353,7 +2326,7 @@ }, { "cell_type": "markdown", - "id": "935e5ca1", + "id": "5820ebfd", "metadata": { "editable": true }, @@ -2364,7 +2337,7 @@ { "cell_type": "code", "execution_count": 22, - "id": "2b6604c5", + "id": "bb5bea62", "metadata": { "collapsed": false, "editable": true @@ -2379,7 +2352,7 @@ { "cell_type": "code", "execution_count": 23, - "id": "b6412285", + "id": "b879f3ce", "metadata": { "collapsed": false, "editable": true @@ -2397,7 +2370,7 @@ }, { "cell_type": "markdown", - "id": "c8adc4f4", + "id": "60160f97", "metadata": { "editable": true }, @@ -2417,7 +2390,7 @@ }, { "cell_type": "markdown", - "id": "e0bd3a34", + "id": "b351b1bd", "metadata": { "editable": true }, @@ -2431,7 +2404,7 @@ }, { "cell_type": "markdown", - "id": "b93c398e", + "id": "6e9174ef", "metadata": { "editable": true }, @@ -2443,7 +2416,7 @@ }, { "cell_type": "markdown", - "id": "7fc3ae93", + "id": "fc319721", "metadata": { "editable": true }, @@ -2460,7 +2433,7 @@ }, { "cell_type": "markdown", - "id": "e5c67b21", + "id": "da4ba861", "metadata": { "editable": true }, @@ -2472,7 +2445,7 @@ }, { "cell_type": "markdown", - "id": "2d573725", + "id": "f444a5a4", "metadata": { "editable": true }, @@ -2486,7 +2459,7 @@ }, { "cell_type": "markdown", - "id": "f8cf15d3", + "id": "8a4d8175", "metadata": { "editable": true }, @@ -2498,7 +2471,7 @@ }, { "cell_type": "markdown", - "id": "e734d8d4", + "id": "de12bc14", "metadata": { "editable": true }, @@ -2511,7 +2484,7 @@ }, { "cell_type": "markdown", - "id": "0585b995", + "id": "735bf417", "metadata": { "editable": true }, @@ -2523,7 +2496,7 @@ }, { "cell_type": "markdown", - "id": "b983e555", + "id": "22b8d82f", "metadata": { "editable": true }, @@ -2533,7 +2506,7 @@ }, { "cell_type": "markdown", - "id": "203a7a1c", + "id": "661db2e1", "metadata": { "editable": true }, @@ -2561,7 +2534,7 @@ }, { "cell_type": "markdown", - "id": "6d391d0c", + "id": "2b14c81e", "metadata": { "editable": true }, @@ -2577,7 +2550,7 @@ }, { "cell_type": "markdown", - "id": "3b98bb46", + "id": "64c44231", "metadata": { "editable": true }, @@ -2589,7 +2562,7 @@ }, { "cell_type": "markdown", - "id": "3d2a99d8", + "id": "1040bdaf", "metadata": { "editable": true }, @@ -2600,7 +2573,7 @@ }, { "cell_type": "markdown", - "id": "3af9f679", + "id": "de59d269", "metadata": { "editable": true }, @@ -2612,7 +2585,7 @@ }, { "cell_type": "markdown", - "id": "368371bc", + "id": "5f87e844", "metadata": { "editable": true }, @@ -2622,7 +2595,7 @@ }, { "cell_type": "markdown", - "id": "ebc23cc6", + "id": "a2f9215c", "metadata": { "editable": true }, @@ -2634,7 +2607,7 @@ }, { "cell_type": "markdown", - "id": "3285cd00", + "id": "67f71f90", "metadata": { "editable": true }, @@ -2644,7 +2617,7 @@ }, { "cell_type": "markdown", - "id": "420c5c35", + "id": "5410f260", "metadata": { "editable": true }, @@ -2656,7 +2629,7 @@ }, { "cell_type": "markdown", - "id": "2a386bd3", + "id": "0485a1f5", "metadata": { "editable": true }, @@ -2666,7 +2639,7 @@ }, { "cell_type": "markdown", - "id": "cb75d383", + "id": "3a256711", "metadata": { "editable": true }, @@ -2678,7 +2651,7 @@ }, { "cell_type": "markdown", - "id": "d7aac8e6", + "id": "fc8cd2ae", "metadata": { "editable": true }, @@ -2692,7 +2665,7 @@ }, { "cell_type": "markdown", - "id": "8211e34f", + "id": "0a9ecf4b", "metadata": { "editable": true }, @@ -2708,7 +2681,7 @@ }, { "cell_type": "markdown", - "id": "0759bace", + "id": "ac605ae7", "metadata": { "editable": true }, @@ -2720,7 +2693,7 @@ }, { "cell_type": "markdown", - "id": "6a08f8a3", + "id": "b4d530db", "metadata": { "editable": true }, @@ -2736,7 +2709,7 @@ }, { "cell_type": "markdown", - "id": "ea71963f", + "id": "0f9fce0f", "metadata": { "editable": true }, @@ -2748,7 +2721,7 @@ }, { "cell_type": "markdown", - "id": "42c5df3b", + "id": "73471c17", "metadata": { "editable": true }, @@ -2758,7 +2731,7 @@ }, { "cell_type": "markdown", - "id": "237180f9", + "id": "d8244842", "metadata": { "editable": true }, @@ -2770,7 +2743,7 @@ }, { "cell_type": "markdown", - "id": "2374907b", + "id": "be09fe99", "metadata": { "editable": true }, @@ -2782,7 +2755,7 @@ }, { "cell_type": "markdown", - "id": "04048caf", + "id": "a547cf77", "metadata": { "editable": true }, @@ -2794,7 +2767,7 @@ }, { "cell_type": "markdown", - "id": "1a77afe0", + "id": "67b1198a", "metadata": { "editable": true }, @@ -2805,7 +2778,7 @@ }, { "cell_type": "markdown", - "id": "981883bd", + "id": "f0a75e83", "metadata": { "editable": true }, @@ -2817,7 +2790,7 @@ }, { "cell_type": "markdown", - "id": "3d8d3830", + "id": "9d2d96dc", "metadata": { "editable": true }, @@ -2828,7 +2801,7 @@ }, { "cell_type": "markdown", - "id": "fe3d598c", + "id": "a6c2a558", "metadata": { "editable": true }, @@ -2840,7 +2813,7 @@ }, { "cell_type": "markdown", - "id": "762ac6b8", + "id": "5a582df6", "metadata": { "editable": true }, @@ -2850,7 +2823,7 @@ }, { "cell_type": "markdown", - "id": "396805d8", + "id": "654c5f13", "metadata": { "editable": true }, @@ -2862,7 +2835,7 @@ }, { "cell_type": "markdown", - "id": "19ca93fe", + "id": "efecb2bc", "metadata": { "editable": true }, @@ -2874,7 +2847,7 @@ }, { "cell_type": "markdown", - "id": "5b78c347", + "id": "da78bb27", "metadata": { "editable": true }, @@ -2886,7 +2859,7 @@ }, { "cell_type": "markdown", - "id": "742c335a", + "id": "dc1c118f", "metadata": { "editable": true }, @@ -2898,7 +2871,7 @@ }, { "cell_type": "markdown", - "id": "766b1fdd", + "id": "1b77640d", "metadata": { "editable": true }, @@ -2908,7 +2881,7 @@ }, { "cell_type": "markdown", - "id": "d9698851", + "id": "d7944742", "metadata": { "editable": true }, @@ -2920,7 +2893,7 @@ }, { "cell_type": "markdown", - "id": "8b9dc891", + "id": "94ffa0c4", "metadata": { "editable": true }, @@ -2930,7 +2903,7 @@ }, { "cell_type": "markdown", - "id": "32e16984", + "id": "eae46622", "metadata": { "editable": true }, @@ -2942,7 +2915,7 @@ }, { "cell_type": "markdown", - "id": "85898ea6", + "id": "099f71b5", "metadata": { "editable": true }, @@ -2952,7 +2925,7 @@ }, { "cell_type": "markdown", - "id": "e9c70102", + "id": "11e5f200", "metadata": { "editable": true }, @@ -2964,7 +2937,7 @@ }, { "cell_type": "markdown", - "id": "a00653d9", + "id": "77b52ed2", "metadata": { "editable": true }, @@ -2974,7 +2947,7 @@ }, { "cell_type": "markdown", - "id": "e041fdeb", + "id": "e8fe5df6", "metadata": { "editable": true }, @@ -2986,7 +2959,7 @@ }, { "cell_type": "markdown", - "id": "5f07c0b1", + "id": "4c1ea9b7", "metadata": { "editable": true }, @@ -2996,7 +2969,7 @@ }, { "cell_type": "markdown", - "id": "d13323ee", + "id": "a61b875a", "metadata": { "editable": true }, @@ -3008,7 +2981,7 @@ }, { "cell_type": "markdown", - "id": "c7237587", + "id": "a0df6e36", "metadata": { "editable": true }, @@ -3028,7 +3001,7 @@ }, { "cell_type": "markdown", - "id": "5eabf8df", + "id": "862806de", "metadata": { "editable": true }, @@ -3040,7 +3013,7 @@ }, { "cell_type": "markdown", - "id": "309fe485", + "id": "60c6b96e", "metadata": { "editable": true }, @@ -3050,7 +3023,7 @@ }, { "cell_type": "markdown", - "id": "344c2fc2", + "id": "d4cf16bb", "metadata": { "editable": true }, @@ -3066,7 +3039,7 @@ }, { "cell_type": "markdown", - "id": "0af051ff", + "id": "91e907b9", "metadata": { "editable": true }, @@ -3078,7 +3051,7 @@ }, { "cell_type": "markdown", - "id": "ebf99e70", + "id": "cc913a38", "metadata": { "editable": true }, @@ -3106,7 +3079,7 @@ }, { "cell_type": "markdown", - "id": "38533606", + "id": "87e49535", "metadata": { "editable": true }, @@ -3119,7 +3092,7 @@ { "cell_type": "code", "execution_count": 24, - "id": "58b8da05", + "id": "a48ac6a2", "metadata": { "collapsed": false, "editable": true diff --git a/doc/src/week46/week46.do.txt b/doc/src/week46/week46.do.txt index f3f7b5662..4401f9a49 100644 --- a/doc/src/week46/week46.do.txt +++ b/doc/src/week46/week46.do.txt @@ -61,20 +61,6 @@ learned the underlying structure of the training data and hence can, given some assumptions, make predictions about the target feature value (class) of unseen query instances. -!split -===== A Sketch of a Tree, Regression problem ===== - -"See handwritten notes November 3":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2022/NotesNov32022.pdf" - -#FIGURE: [DataFiles/Regsimpletree.png, width=600 frac=0.8] - -!split -===== A Sketch of a Tree, Classification problem ===== - -"See handwritten notes November 3":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2022/NotesNov32022.pdf" -#FIGURE: [DataFiles/Classimpletree.png, width=600 frac=0.8] - - !split ===== A typical Decision Tree with its pertinent Jargon, Classification Problem =====