Files
FYS-STK4155/doc/pub/week46/html/._week46-bs015.html
Morten Hjorth-Jensen 2c1f27b519 update
2023-11-16 06:33:54 +01:00

387 lines
23 KiB
HTML

<!--
HTML file automatically generated from DocOnce source
(https://github.com/doconce/doconce/)
doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=default --html_admon=bootstrap_panel --html_output=week46-bs --no_mako
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 46: Decision Trees, Ensemble methods and Random Forests">
<title>Week 46: Decision Trees, Ensemble methods and Random Forests</title>
<!-- Bootstrap style: bootstrap -->
<!-- doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=default --html_admon=bootstrap_panel --html_output=week46-bs --no_mako -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 46', 2, None, 'plan-for-week-46'),
('Decision trees, overarching aims',
2,
None,
'decision-trees-overarching-aims'),
('Basics of a tree', 2, None, 'basics-of-a-tree'),
('A typical Decision Tree with its pertinent Jargon, '
'Classification Problem',
2,
None,
'a-typical-decision-tree-with-its-pertinent-jargon-classification-problem'),
('General Features', 2, None, 'general-features'),
('How do we set it up?', 2, None, 'how-do-we-set-it-up'),
('Decision trees and Regression',
2,
None,
'decision-trees-and-regression'),
('Building a tree, regression',
2,
None,
'building-a-tree-regression'),
('A top-down approach, recursive binary splitting',
2,
None,
'a-top-down-approach-recursive-binary-splitting'),
('Making a tree', 2, None, 'making-a-tree'),
('Pruning the tree', 2, None, 'pruning-the-tree'),
('Cost complexity pruning', 2, None, 'cost-complexity-pruning'),
('Schematic Regression Procedure',
2,
None,
'schematic-regression-procedure'),
('A Classification Tree', 2, None, 'a-classification-tree'),
('Growing a classification tree',
2,
None,
'growing-a-classification-tree'),
('Classification tree, how to split nodes',
2,
None,
'classification-tree-how-to-split-nodes'),
('Visualizing the Tree, Classification',
2,
None,
'visualizing-the-tree-classification'),
('Visualizing the Tree, The Moons',
2,
None,
'visualizing-the-tree-the-moons'),
('Other ways of visualizing the trees',
2,
None,
'other-ways-of-visualizing-the-trees'),
('Printing out as text', 2, None, 'printing-out-as-text'),
('Algorithms for Setting up Decision Trees',
2,
None,
'algorithms-for-setting-up-decision-trees'),
('The CART algorithm for Classification',
2,
None,
'the-cart-algorithm-for-classification'),
('The CART algorithm for Regression',
2,
None,
'the-cart-algorithm-for-regression'),
('Why binary splits?', 2, None, 'why-binary-splits'),
('Computing a Tree using the Gini Index',
2,
None,
'computing-a-tree-using-the-gini-index'),
('The Table', 2, None, 'the-table'),
('Computing the various Gini Indices',
2,
None,
'computing-the-various-gini-indices'),
('Computing the various Gini Indices, Hours slept',
2,
None,
'computing-the-various-gini-indices-hours-slept'),
('Computing the various Gini Indices, Hours studied',
2,
None,
'computing-the-various-gini-indices-hours-studied'),
('A possible code using Scikit-Learn',
2,
None,
'a-possible-code-using-scikit-learn'),
('Further example: Computing the Gini index',
2,
None,
'further-example-computing-the-gini-index'),
('Simple Python Code to read in Data and perform Classification',
2,
None,
'simple-python-code-to-read-in-data-and-perform-classification'),
('Computing the Gini Factor',
2,
None,
'computing-the-gini-factor'),
('Regression trees', 2, None, 'regression-trees'),
('Final regressor code', 2, None, 'final-regressor-code'),
('Pros and cons of trees, pros',
2,
None,
'pros-and-cons-of-trees-pros'),
('Disadvantages', 2, None, 'disadvantages'),
('Ensemble Methods: From a Single Tree to Many Trees and Extreme '
'Boosting, Meet the Jungle of Methods',
2,
None,
'ensemble-methods-from-a-single-tree-to-many-trees-and-extreme-boosting-meet-the-jungle-of-methods'),
('An Overview of Ensemble Methods',
2,
None,
'an-overview-of-ensemble-methods'),
('Why Voting?', 2, None, 'why-voting'),
('Tossing coins', 2, None, 'tossing-coins'),
('Standard imports first', 2, None, 'standard-imports-first'),
('Simple Voting Example, head or tail',
2,
None,
'simple-voting-example-head-or-tail'),
('Using the Voting Classifier',
2,
None,
'using-the-voting-classifier'),
('Voting and Bagging', 2, None, 'voting-and-bagging'),
('Bagging', 2, None, 'bagging'),
('More bagging', 2, None, 'more-bagging'),
('Making your own Bootstrap: Changing the Level of the Decision '
'Tree',
2,
None,
'making-your-own-bootstrap-changing-the-level-of-the-decision-tree'),
('Random forests', 2, None, 'random-forests'),
('Random Forest Algorithm', 2, None, 'random-forest-algorithm'),
('Random Forests Compared with other Methods on the Cancer Data',
2,
None,
'random-forests-compared-with-other-methods-on-the-cancer-data'),
('Compare Bagging on Trees with Random Forests',
2,
None,
'compare-bagging-on-trees-with-random-forests'),
("Boosting, a Bird's Eye View",
2,
None,
'boosting-a-bird-s-eye-view'),
('What is boosting? Additive Modelling/Iterative Fitting',
2,
None,
'what-is-boosting-additive-modelling-iterative-fitting'),
('Iterative Fitting, Regression and Squared-error Cost Function',
2,
None,
'iterative-fitting-regression-and-squared-error-cost-function'),
('Squared-Error Example and Iterative Fitting',
2,
None,
'squared-error-example-and-iterative-fitting'),
('Iterative Fitting, Classification and AdaBoost',
2,
None,
'iterative-fitting-classification-and-adaboost'),
('Adaptive Boosting, AdaBoost',
2,
None,
'adaptive-boosting-adaboost'),
('Building up AdaBoost', 2, None, 'building-up-adaboost'),
('Adaptive boosting: AdaBoost, Basic Algorithm',
2,
None,
'adaptive-boosting-adaboost-basic-algorithm'),
('Basic Steps of AdaBoost', 2, None, 'basic-steps-of-adaboost'),
('AdaBoost Examples', 2, None, 'adaboost-examples')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="week46-bs.html">Week 46: Decision Trees, Ensemble methods and Random Forests</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week46-bs001.html#plan-for-week-46" style="font-size: 80%;">Plan for week 46</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs002.html#decision-trees-overarching-aims" style="font-size: 80%;">Decision trees, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs003.html#basics-of-a-tree" style="font-size: 80%;">Basics of a tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs004.html#a-typical-decision-tree-with-its-pertinent-jargon-classification-problem" style="font-size: 80%;">A typical Decision Tree with its pertinent Jargon, Classification Problem</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs005.html#general-features" style="font-size: 80%;">General Features</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs006.html#how-do-we-set-it-up" style="font-size: 80%;">How do we set it up?</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs007.html#decision-trees-and-regression" style="font-size: 80%;">Decision trees and Regression</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs008.html#building-a-tree-regression" style="font-size: 80%;">Building a tree, regression</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs009.html#a-top-down-approach-recursive-binary-splitting" style="font-size: 80%;">A top-down approach, recursive binary splitting</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs010.html#making-a-tree" style="font-size: 80%;">Making a tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs011.html#pruning-the-tree" style="font-size: 80%;">Pruning the tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs012.html#cost-complexity-pruning" style="font-size: 80%;">Cost complexity pruning</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs013.html#schematic-regression-procedure" style="font-size: 80%;">Schematic Regression Procedure</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs014.html#a-classification-tree" style="font-size: 80%;">A Classification Tree</a></li>
<!-- navigation toc: --> <li><a href="#growing-a-classification-tree" style="font-size: 80%;">Growing a classification tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs016.html#classification-tree-how-to-split-nodes" style="font-size: 80%;">Classification tree, how to split nodes</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs017.html#visualizing-the-tree-classification" style="font-size: 80%;">Visualizing the Tree, Classification</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs018.html#visualizing-the-tree-the-moons" style="font-size: 80%;">Visualizing the Tree, The Moons</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs019.html#other-ways-of-visualizing-the-trees" style="font-size: 80%;">Other ways of visualizing the trees</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs020.html#printing-out-as-text" style="font-size: 80%;">Printing out as text</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs021.html#algorithms-for-setting-up-decision-trees" style="font-size: 80%;">Algorithms for Setting up Decision Trees</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs022.html#the-cart-algorithm-for-classification" style="font-size: 80%;">The CART algorithm for Classification</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs023.html#the-cart-algorithm-for-regression" style="font-size: 80%;">The CART algorithm for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs024.html#why-binary-splits" style="font-size: 80%;">Why binary splits?</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs025.html#computing-a-tree-using-the-gini-index" style="font-size: 80%;">Computing a Tree using the Gini Index</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs026.html#the-table" style="font-size: 80%;">The Table</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs027.html#computing-the-various-gini-indices" style="font-size: 80%;">Computing the various Gini Indices</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs028.html#computing-the-various-gini-indices-hours-slept" style="font-size: 80%;">Computing the various Gini Indices, Hours slept</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs029.html#computing-the-various-gini-indices-hours-studied" style="font-size: 80%;">Computing the various Gini Indices, Hours studied</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs030.html#a-possible-code-using-scikit-learn" style="font-size: 80%;">A possible code using Scikit-Learn</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs031.html#further-example-computing-the-gini-index" style="font-size: 80%;">Further example: Computing the Gini index</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs032.html#simple-python-code-to-read-in-data-and-perform-classification" style="font-size: 80%;">Simple Python Code to read in Data and perform Classification</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs033.html#computing-the-gini-factor" style="font-size: 80%;">Computing the Gini Factor</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs034.html#regression-trees" style="font-size: 80%;">Regression trees</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs035.html#final-regressor-code" style="font-size: 80%;">Final regressor code</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs036.html#pros-and-cons-of-trees-pros" style="font-size: 80%;">Pros and cons of trees, pros</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs037.html#disadvantages" style="font-size: 80%;">Disadvantages</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs038.html#ensemble-methods-from-a-single-tree-to-many-trees-and-extreme-boosting-meet-the-jungle-of-methods" style="font-size: 80%;">Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs039.html#an-overview-of-ensemble-methods" style="font-size: 80%;">An Overview of Ensemble Methods</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs040.html#why-voting" style="font-size: 80%;">Why Voting?</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs041.html#tossing-coins" style="font-size: 80%;">Tossing coins</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs042.html#standard-imports-first" style="font-size: 80%;">Standard imports first</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs043.html#simple-voting-example-head-or-tail" style="font-size: 80%;">Simple Voting Example, head or tail</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs044.html#using-the-voting-classifier" style="font-size: 80%;">Using the Voting Classifier</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs045.html#voting-and-bagging" style="font-size: 80%;">Voting and Bagging</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs046.html#bagging" style="font-size: 80%;">Bagging</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs047.html#more-bagging" style="font-size: 80%;">More bagging</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs048.html#making-your-own-bootstrap-changing-the-level-of-the-decision-tree" style="font-size: 80%;">Making your own Bootstrap: Changing the Level of the Decision Tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs049.html#random-forests" style="font-size: 80%;">Random forests</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs050.html#random-forest-algorithm" style="font-size: 80%;">Random Forest Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs051.html#random-forests-compared-with-other-methods-on-the-cancer-data" style="font-size: 80%;">Random Forests Compared with other Methods on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs052.html#compare-bagging-on-trees-with-random-forests" style="font-size: 80%;">Compare Bagging on Trees with Random Forests</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs053.html#boosting-a-bird-s-eye-view" style="font-size: 80%;">Boosting, a Bird's Eye View</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs054.html#what-is-boosting-additive-modelling-iterative-fitting" style="font-size: 80%;">What is boosting? Additive Modelling/Iterative Fitting</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs055.html#iterative-fitting-regression-and-squared-error-cost-function" style="font-size: 80%;">Iterative Fitting, Regression and Squared-error Cost Function</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs056.html#squared-error-example-and-iterative-fitting" style="font-size: 80%;">Squared-Error Example and Iterative Fitting</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs057.html#iterative-fitting-classification-and-adaboost" style="font-size: 80%;">Iterative Fitting, Classification and AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs058.html#adaptive-boosting-adaboost" style="font-size: 80%;">Adaptive Boosting, AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs059.html#building-up-adaboost" style="font-size: 80%;">Building up AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs060.html#adaptive-boosting-adaboost-basic-algorithm" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs061.html#basic-steps-of-adaboost" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs062.html#adaboost-examples" style="font-size: 80%;">AdaBoost Examples</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0015"></a>
<!-- !split -->
<h2 id="growing-a-classification-tree" class="anchor">Growing a classification tree </h2>
<p>The task of growing a
classification tree is quite similar to the task of growing a
regression tree. Just as in the regression setting, we use recursive
binary splitting to grow a classification tree. However, in the
classification setting, the MSE cannot be used as a criterion for making
the binary splits. A natural alternative to MSE is the <b>classification
error rate</b>. Since we plan to assign an observation in a given region
to the most commonly occurring error rate class of training
observations in that region, the classification error rate is simply
the fraction of the training observations in that region that do not
belong to the most common class.
</p>
<p>When building a classification tree, either the Gini index or the
entropy are typically used to evaluate the quality of a particular
split, since these two approaches are more sensitive to node purity
than is the classification error rate.
</p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._week46-bs014.html">&laquo;</a></li>
<li><a href="._week46-bs000.html">1</a></li>
<li><a href="">...</a></li>
<li><a href="._week46-bs007.html">8</a></li>
<li><a href="._week46-bs008.html">9</a></li>
<li><a href="._week46-bs009.html">10</a></li>
<li><a href="._week46-bs010.html">11</a></li>
<li><a href="._week46-bs011.html">12</a></li>
<li><a href="._week46-bs012.html">13</a></li>
<li><a href="._week46-bs013.html">14</a></li>
<li><a href="._week46-bs014.html">15</a></li>
<li class="active"><a href="._week46-bs015.html">16</a></li>
<li><a href="._week46-bs016.html">17</a></li>
<li><a href="._week46-bs017.html">18</a></li>
<li><a href="._week46-bs018.html">19</a></li>
<li><a href="._week46-bs019.html">20</a></li>
<li><a href="._week46-bs020.html">21</a></li>
<li><a href="._week46-bs021.html">22</a></li>
<li><a href="._week46-bs022.html">23</a></li>
<li><a href="._week46-bs023.html">24</a></li>
<li><a href="._week46-bs024.html">25</a></li>
<li><a href="">...</a></li>
<li><a href="._week46-bs062.html">63</a></li>
<li><a href="._week46-bs016.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="https://..."><img width="250" align=right src="https://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>