lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on decision trees

Browse Source
This commit is contained in:
mhjensen
2018-11-09 05:04:05 +01:00
parent 6a18aa51b4
commit a99e8da478
10 changed files with 1988 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/DecisionTrees/html/DecisionTrees-bs.html
+398 -3
View File
@@ -44,11 +44,43 @@ Automatically generated HTML file from DocOnce source
('Nodes, leafs, roots and branches', 2, None, '___sec1'),
('How do we set it up?', 2, None, '___sec2'),
('Decision trees and Regression', 2, None, '___sec3'),
('Maxwell-Boltzmann velocity distribution', 2, None, '___sec4')]}
('Maxwell-Boltzmann velocity distribution', 2, None, '___sec4'),
('Building a tree, regression', 2, None, '___sec5'),
('A top-down approach, recursive binary splitting',
2,
None,
'___sec6'),
('Making a tree', 2, None, '___sec7'),
('Pruning the tree', 2, None, '___sec8'),
('Cost complexity pruning', 2, None, '___sec9'),
('A schematic procedure', 2, None, '___sec10'),
('A classification tree', 2, None, '___sec11'),
('Growing a classification tree', 2, None, '___sec12'),
('Pros and cons of trees, pros', 2, None, '___sec13'),
('Disadvantages', 2, None, '___sec14'),
('Bagging', 2, None, '___sec15'),
('Random forests', 2, None, '___sec16'),
('A simple scikit-learn example', 2, None, '___sec17')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "AMS" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
@@ -71,6 +103,19 @@ end of tocinfo -->
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">How do we set it up?</a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">Decision trees and Regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">Maxwell-Boltzmann velocity distribution</a></li>
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;">Building a tree, regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;">A top-down approach, recursive binary splitting</a></li>
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;">Making a tree</a></li>
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;">Pruning the tree</a></li>
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;">Cost complexity pruning</a></li>
<!-- navigation toc: --> <li><a href="#___sec10" style="font-size: 80%;">A schematic procedure</a></li>
<!-- navigation toc: --> <li><a href="#___sec11" style="font-size: 80%;">A classification tree</a></li>
<!-- navigation toc: --> <li><a href="#___sec12" style="font-size: 80%;">Growing a classification tree</a></li>
<!-- navigation toc: --> <li><a href="#___sec13" style="font-size: 80%;">Pros and cons of trees, pros</a></li>
<!-- navigation toc: --> <li><a href="#___sec14" style="font-size: 80%;">Disadvantages</a></li>
<!-- navigation toc: --> <li><a href="#___sec15" style="font-size: 80%;">Bagging</a></li>
<!-- navigation toc: --> <li><a href="#___sec16" style="font-size: 80%;">Random forests</a></li>
<!-- navigation toc: --> <li><a href="#___sec17" style="font-size: 80%;">A simple scikit-learn example</a></li>
</ul>
</li>
@@ -104,7 +149,7 @@ end of tocinfo -->
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<center><h4>Nov 9, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- potential-jumbotron-button -->
@@ -141,7 +186,7 @@ The main idea of decision trees
is to find those descriptive features which contain the most
<b>information</b> regarding the target feature and then split the dataset
along the values of these features such that the target feature values
for the resulting sub\_datasets are as pure as possible.
for the resulting sub datasets are as pure as possible.
<p>
The descriptive feature which leaves the target feature most purely is said
@@ -380,6 +425,356 @@ plt<span style="color: #666666">.</span>show()
<span style="color: #408080; font-style: italic">#i.e. frequency of all velocities in range -5 to -4.9, -4.9 to -4.8, etc...</span>
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec5" class="anchor">Building a tree, regression </h2>
<p>
There are mainly two steps
<ol>
<li> We split the predictor space (the set of possible values \( x_1,x_2,\dots, x_p \)) into \( J \)</li>
</ol>
distinct and non-non-overlapping regions, \( R_1,R_2,\dots,R_J \).
<ol>
<li> For every observation that falls into the region \( R_j \) , we make the same prediction, which is simply the mean of the response values for the training observations in \( R_j \).</li>
</ol>
How do we construct the regions \( R_1,\dots,R_J \)?
In theory, the regions could have any shape. However, we
choose to divide the predictor space into high-dimensional rectangles,
or boxes, for simplicity and for ease of interpretation of the
resulting predic- tive model. The goal is to find boxes \( R_1,\dots,R_J \)
that minimize the MSE, given by
$$
\sum_{j=1}^J\sum_{i\in R_j}(y_i-\overline{y}_{R_j})^2,
$$
where \( \overline{y}_{R_j} \) is the mean response for the training observations
within the $j$th
box.
<p>
<!-- !split -->
<h2 id="___sec6" class="anchor">A top-down approach, recursive binary splitting </h2>
<p>
Unfortunately, it is computationally infeasible to consider every
possible partition of the feature space into \( J \) boxes.
The common strategy is to take a top-down approach
<p>
The approach is top-down because it begins at the top of the tree (all
observations belong to a single region) and then successively splits
the predictor space; each split is indicated via two new branches
further down on the tree. It is greedy because at each step of the
tree-building process, the best split is made at that particular step,
rather than looking ahead and picking a split that will lead to a
better tree in some future step.
<p>
<!-- !split -->
<h2 id="___sec7" class="anchor">Making a tree </h2>
<p>
In order to implement the recursive binary splitting we start by selecting
the predictor \( x_j \) and a cutpoint \( s \) that splits the predictor space into two regions \( R_1 \) and \( R_2 \)
$$
\left\{X\vert x_j < s\right\},
$$
and
$$
\left\{X\vert x_j \geq s\right\},
$$
so that we obtain the lowest MSE, that is
$$
\sum_{i:x_i\in R_j}(y_i-\overline{y}_{R_1})^2+\sum_{i:x_i\in R_2}(y_i-\overline{y}_{R_2})^2,
$$
which we want to minimize by considering all predictors \( x_1,x_2,\dots,x_p \).
We consider also all possible values of \( s \) for each predictor. These values could be determined by randomly assigned numbers or by starting at the midpoint and then proceed till we find an optimal value.
<p>
For any \( j \) and \( s \), we define the pair of
half-planes where \( \overline{y}_{R_1} \) is the mean response for the training
observations in \( R_1(j,s) \), and \( \overline{y}_{R_2} \) is the mean response for the
training observations in \( R_2(j,s) \).
<p>
Finding the values of j and s that
minimize the above equation can be done quite quickly, especially when the number
of features \( p \) is not too large.
<p>
Next, we repeat the process, looking
for the best predictor and best cutpoint in order to split the data
further so as to minimize the RSS within each of the resulting
regions. However, this time, instead of splitting the entire predictor
space, we split one of the two previously identified regions. We now
have three regions. Again, we look to split one of these three regions
further, so as to minimize the MSE. The process continues until a
stopping criterion is reached; for instance, we may continue until no
region contains more than five observations.
<p>
<!-- !split -->
<h2 id="___sec8" class="anchor">Pruning the tree </h2>
<p>
The above procedure is rather straightforward, but leads often to
overfitting and unnecessarily large and complicated trees. The basic
idea is to grow a large tree \( T_0 \) and then prune it back in order to
obtain a subtree. A smaller tree with fewer splits (fewer regions) can
lead to smaller variance and better interpretation at the cost of a
little more bias.
<p>
The so-called Cost complexity pruning algorithm gives us a
way to do just this. Rather than considering every possible subtree,
we consider a sequence of trees indexed by a nonnegative tuning
parameter \( \alpha \).
<p>
<!-- !split -->
<h2 id="___sec9" class="anchor">Cost complexity pruning </h2>
For each value of \( \alpha \) there corresponds a subtree \( T \in T_0 \) such that
$$
\sum_{m=1}^{\overline{T}}\sum_{i:x_i\in R_m}(y_i-\overline{y}_{R_m})^2+\alpha\overline{T},
$$
is as small as possible. Here \( \overline{T} \) is
the number of terminal nodes of the tree \( T \) , \( R_m \) is the
rectangle (i.e. the subset of predictor space) corresponding to the $m$th terminal node.
<p>
The tuning parameter \( \alpha \) controls a trade-off between the subtree&#8217;s
com- plexity and its fit to the training data. When \( \alpha = 0 \), then the
subtree \( T \) will simply equal \( T_0 \),
because then the above equation just measures the
training error.
However, as \( \alpha \) increases, there is a price to pay for
having a tree with many terminal nodes. The above equation will
tend to be minimized for a smaller subtree.
<p>
It turns out that as we increase \( \alpha \) from zero
branches get pruned from the tree in a nested and predictable fashion,
so obtaining the whole sequence of subtrees as a function of \( \alpha \) is
easy. We can select a value of \( \alpha \) using a validation set or using
cross-validation. We then return to the full data set and obtain the
subtree corresponding to \( \alpha \).
<p>
<!-- !split -->
<h2 id="___sec10" class="anchor">A schematic procedure </h2>
<p>
<div class="panel panel-default">
<div class="panel-body">
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
<ol>
<li> Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
<ol type="a"></li>
<li> Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of \( \alpha \).</li>
</ol>
<li> Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we repeat Steps 1 and 2 on all but the $k$th fold of the training data. Then we valuate the mean squared prediction error on the data in the left-out $k$th fold, as a function of \( \alpha \).</li>
<li> Then we average the results for each value of \( alpha \), and pick \( \alpha \) to minimize the average error.</li>
<li> Return the subtree from Step 2 that corresponds to the chosen value of \( \alpha \).</li>
</ol>
</div>
</div>
<p>
<!-- !split -->
<h2 id="___sec11" class="anchor">A classification tree </h2>
<p>
A classification tree is very similar to a regression tree, except
that it is used to predict a qualitative response rather than a
quantitative one. Recall that for a regression tree, the predicted
response for an observation is given by the mean response of the
training observations that belong to the same terminal node. In
contrast, for a classification tree, we predict that each observation
belongs to the most commonly occurring class of training observations
in the region to which it belongs. In interpreting the results of a
classification tree, we are often interested not only in the class
prediction corresponding to a particular terminal node region, but
also in the class proportions among the training observations that
fall into that region.
<p>
<!-- !split -->
<h2 id="___sec12" class="anchor">Growing a classification tree </h2>
<p>
The task of growing a
classification tree is quite similar to the task of growing a
regression tree. Just as in the regression setting, we use recursive
binary splitting to grow a classification tree. However, in the
classification setting, the MSE cannot be used as a criterion for making
the binary splits. A natural alternative to MSE is the <b>classification
error rate</b>. Since we plan to assign an observation in a given region
to the most commonly occurring error rate class of training
observations in that region, the classification error rate is simply
the fraction of the training observations in that region that do not
belong to the most common class.
<p>
When building a classification tree, either the Gini index or the
entropy are typically used to evaluate the quality of a particular
split, since these two approaches are more sensitive to node purity
than is the classification error rate.
<p>
<!-- !split -->
<h2 id="___sec13" class="anchor">Pros and cons of trees, pros </h2>
<ul>
<li> White box, easy to interpret model. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches discussed earlier (think of support vector machines)</li>
<li> Trees are very easy to explain to people. In fact, they are even easier to explain than linear regression!</li>
<li> No feature normalization needed</li>
<li> Tree models can handle both continuous and categorical data (Classification and Regression Trees)</li>
<li> Can model nonlinear relationships</li>
<li> Can model interactions between the different descriptive features</li>
<li> Trees can be displayed graphically, and are easily interpreted even by a non-expert (especially if they are small)</li>
</ul>
<!-- !split -->
<h2 id="___sec14" class="anchor">Disadvantages </h2>
<ul>
<li> Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches</li>
<li> If continuous features are used the tree may become quite large and hence less interpretable</li>
<li> Decision trees are prone to overfit the training data and hence do not well generalize the data if no stopping criteria or improvements like pruning, boosting or bagging are implemented</li>
<li> Small changes in the data may lead to a completely different tree. This issue can be addressed by using ensemble methods like bagging, boosting or random forests</li>
<li> Unbalanced datasets where some target feature values occur much more frequently than others may lead to biased trees since the frequently occurring feature values are preferred over the less frequently occurring ones.</li>
<li> If the number of features is relatively large (high dimensional) and the number of instances is relatively low, the tree might overfit the data</li>
<li> Features with many levels may be preferred over features with less levels since for them it is <em>more easy</em> to split the dataset such that the sub datasets only contain pure target feature values. This issue can be addressed by preferring for instance the information gain ratio as splitting criteria over information gain</li>
</ul>
However, by aggregating many decision trees, using methods like bagging, random forests, and boosting, the predictive performance of trees can be substantially improved.
<p>
<!-- !split -->
<h2 id="___sec15" class="anchor">Bagging </h2>
<p>
The <b>plain</b> decision trees suffer from high
variance. This means that if we split the training data into two parts
at random, and fit a decision tree to both halves, the results that we
get could be quite different. In contrast, a procedure with low
variance will yield similar results if applied repeatedly to distinct
data sets; linear regression tends to have low variance, if the ratio
of \( n \) to \( p \) is moderately large.
<p>
<b>Bootstrap aggregation</b>, or just <b>bagging</b>, is a
general-purpose procedure for reducing the variance of a statistical
learning method.
<p>
Bagging typically results in improved accuracy
over prediction using a single tree. Unfortunately, however, it can be
difficult to interpret the resulting model. Recall that one of the
advantages of decision trees is the attractive and easily interpreted
diagram that results.
<p>
However, when we bag a large number of trees, it is no longer
possible to represent the resulting statistical learning procedure
using a single tree, and it is no longer clear which variables are
most important to the procedure. Thus, bagging improves prediction
accuracy at the expense of interpretability. Although the collection
of bagged trees is much more difficult to interpret than a single
tree, one can obtain an overall summary of the importance of each
predictor using the MSE (for bagging regression trees) or the Gini
index (for bagging classification trees). In the case of bagging
regression trees, we can record the total amount that the MSE is
decreased due to splits over a given predictor, averaged over all \( B \) possible
trees. A large value indicates an important predictor. Similarly, in
the context of bagging classification trees, we can add up the total
amount that the Gini index is decreased by splits over a given
predictor, averaged over all \( B \) trees.
<p>
<!-- !split -->
<h2 id="___sec16" class="anchor">Random forests </h2>
<p>
Random forests provide an improvement over bagged trees by way of a
small tweak that decorrelates the trees.
<p>
As in bagging, we build a
number of decision trees on bootstrapped training samples. But when
building these decision trees, each time a split in a tree is
considered, a random sample of \( m \) predictors is chosen as split
candidates from the full set of \( p \) predictors. The split is allowed to
use only one of those \( m \) predictors.
<p>
A fresh sample of \( m \) predictors is
taken at each split, and typically we choose
$$
m\approx \sqrt{p}.
$$
In building a random forest, at
each split in the tree, the algorithm is not even allowed to consider
a majority of the available predictors.
<p>
The reason for this is rather clever. Suppose that there is one very
strong predictor in the data set, along with a number of other
moderately strong predictors. Then in the collection of bagged
variable importance random forest trees, most or all of the trees will
use this strong predictor in the top split. Consequently, all of the
bagged trees will look quite similar to each other. Hence the
predictions from the bagged trees will be highly correlated.
Unfortunately, averaging many highly correlated quantities does not lead
to as large of a reduction in variance as averaging many uncorrelated
quanti- ties. In particular, this means that bagging will not lead to
a substantial reduction in variance over a single tree in this
setting.
<p>
<!-- !split -->
<h2 id="___sec17" class="anchor">A simple scikit-learn example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.ensemble</span> <span style="color: #008000; font-weight: bold">import</span> RandomForestClassifier
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> LabelEncoder
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> cross_validate
<span style="color: #408080; font-style: italic"># Data set not specificied</span>
X <span style="color: #666666">=</span> dataset<span style="color: #666666">.</span>XXX
Y <span style="color: #666666">=</span> dataset<span style="color: #666666">.</span>YYY
<span style="color: #408080; font-style: italic">#Instantiate the model with 100 trees and entropy as splitting criteria</span>
Random_Forest_model <span style="color: #666666">=</span> RandomForestClassifier(n_estimators<span style="color: #666666">=100</span>,criterion<span style="color: #666666">=</span><span style="color: #BA2121">&quot;entropy&quot;</span>)
<span style="color: #408080; font-style: italic">#Cross validation</span>
accuracy <span style="color: #666666">=</span> cross_validate(Random_Forest_model,X,Y,cv<span style="color: #666666">=10</span>)[<span style="color: #BA2121">&#39;test_score&#39;</span>]
</pre></div>
<p>
<!-- ------------------- end of main content --------------- -->
doc/pub/DecisionTrees/html/DecisionTrees-reveal.html
+380 -2
View File
@@ -107,6 +107,22 @@ td.padding {
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
@@ -132,7 +148,7 @@ td.padding {
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<center><h4>Nov 9, 2018</h4></center> <!-- date -->
<br>
<p>
@@ -168,7 +184,7 @@ The main idea of decision trees
is to find those descriptive features which contain the most
<b>information</b> regarding the target feature and then split the dataset
along the values of these features such that the target feature values
for the resulting sub\_datasets are as pure as possible.
for the resulting sub datasets are as pure as possible.
<p>
The descriptive feature which leaves the target feature most purely is said
@@ -411,6 +427,368 @@ plt.show()
</section>
<section>
<h2 id="___sec5">Building a tree, regression </h2>
<p>
There are mainly two steps
<ol>
<p><li> We split the predictor space (the set of possible values \( x_1,x_2,\dots, x_p \)) into \( J \)</li>
</ol>
<p>
distinct and non-non-overlapping regions, \( R_1,R_2,\dots,R_J \).
<ol>
<p><li> For every observation that falls into the region \( R_j \) , we make the same prediction, which is simply the mean of the response values for the training observations in \( R_j \).</li>
</ol>
<p>
How do we construct the regions \( R_1,\dots,R_J \)?
In theory, the regions could have any shape. However, we
choose to divide the predictor space into high-dimensional rectangles,
or boxes, for simplicity and for ease of interpretation of the
resulting predic- tive model. The goal is to find boxes \( R_1,\dots,R_J \)
that minimize the MSE, given by
<p>&nbsp;<br>
$$
\sum_{j=1}^J\sum_{i\in R_j}(y_i-\overline{y}_{R_j})^2,
$$
<p>&nbsp;<br>
where \( \overline{y}_{R_j} \) is the mean response for the training observations
within the $j$th
box.
</section>
<section>
<h2 id="___sec6">A top-down approach, recursive binary splitting </h2>
<p>
Unfortunately, it is computationally infeasible to consider every
possible partition of the feature space into \( J \) boxes.
The common strategy is to take a top-down approach
<p>
The approach is top-down because it begins at the top of the tree (all
observations belong to a single region) and then successively splits
the predictor space; each split is indicated via two new branches
further down on the tree. It is greedy because at each step of the
tree-building process, the best split is made at that particular step,
rather than looking ahead and picking a split that will lead to a
better tree in some future step.
</section>
<section>
<h2 id="___sec7">Making a tree </h2>
<p>
In order to implement the recursive binary splitting we start by selecting
the predictor \( x_j \) and a cutpoint \( s \) that splits the predictor space into two regions \( R_1 \) and \( R_2 \)
<p>&nbsp;<br>
$$
\left\{X\vert x_j < s\right\},
$$
<p>&nbsp;<br>
and
<p>&nbsp;<br>
$$
\left\{X\vert x_j \geq s\right\},
$$
<p>&nbsp;<br>
so that we obtain the lowest MSE, that is
<p>&nbsp;<br>
$$
\sum_{i:x_i\in R_j}(y_i-\overline{y}_{R_1})^2+\sum_{i:x_i\in R_2}(y_i-\overline{y}_{R_2})^2,
$$
<p>&nbsp;<br>
which we want to minimize by considering all predictors \( x_1,x_2,\dots,x_p \).
We consider also all possible values of \( s \) for each predictor. These values could be determined by randomly assigned numbers or by starting at the midpoint and then proceed till we find an optimal value.
<p>
For any \( j \) and \( s \), we define the pair of
half-planes where \( \overline{y}_{R_1} \) is the mean response for the training
observations in \( R_1(j,s) \), and \( \overline{y}_{R_2} \) is the mean response for the
training observations in \( R_2(j,s) \).
<p>
Finding the values of j and s that
minimize the above equation can be done quite quickly, especially when the number
of features \( p \) is not too large.
<p>
Next, we repeat the process, looking
for the best predictor and best cutpoint in order to split the data
further so as to minimize the RSS within each of the resulting
regions. However, this time, instead of splitting the entire predictor
space, we split one of the two previously identified regions. We now
have three regions. Again, we look to split one of these three regions
further, so as to minimize the MSE. The process continues until a
stopping criterion is reached; for instance, we may continue until no
region contains more than five observations.
</section>
<section>
<h2 id="___sec8">Pruning the tree </h2>
<p>
The above procedure is rather straightforward, but leads often to
overfitting and unnecessarily large and complicated trees. The basic
idea is to grow a large tree \( T_0 \) and then prune it back in order to
obtain a subtree. A smaller tree with fewer splits (fewer regions) can
lead to smaller variance and better interpretation at the cost of a
little more bias.
<p>
The so-called Cost complexity pruning algorithm gives us a
way to do just this. Rather than considering every possible subtree,
we consider a sequence of trees indexed by a nonnegative tuning
parameter \( \alpha \).
</section>
<section>
<h2 id="___sec9">Cost complexity pruning </h2>
For each value of \( \alpha \) there corresponds a subtree \( T \in T_0 \) such that
<p>&nbsp;<br>
$$
\sum_{m=1}^{\overline{T}}\sum_{i:x_i\in R_m}(y_i-\overline{y}_{R_m})^2+\alpha\overline{T},
$$
<p>&nbsp;<br>
is as small as possible. Here \( \overline{T} \) is
the number of terminal nodes of the tree \( T \) , \( R_m \) is the
rectangle (i.e. the subset of predictor space) corresponding to the $m$th terminal node.
<p>
The tuning parameter \( \alpha \) controls a trade-off between the subtree&#8217;s
com- plexity and its fit to the training data. When \( \alpha = 0 \), then the
subtree \( T \) will simply equal \( T_0 \),
because then the above equation just measures the
training error.
However, as \( \alpha \) increases, there is a price to pay for
having a tree with many terminal nodes. The above equation will
tend to be minimized for a smaller subtree.
<p>
It turns out that as we increase \( \alpha \) from zero
branches get pruned from the tree in a nested and predictable fashion,
so obtaining the whole sequence of subtrees as a function of \( \alpha \) is
easy. We can select a value of \( \alpha \) using a validation set or using
cross-validation. We then return to the full data set and obtain the
subtree corresponding to \( \alpha \).
</section>
<section>
<h2 id="___sec10">A schematic procedure </h2>
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>Building a Regression Tree.</b>
<ol>
<p><li> Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
<ol type="a"></li>
<p><li> Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of \( \alpha \).</li>
</ol>
<p><li> Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we repeat Steps 1 and 2 on all but the $k$th fold of the training data. Then we valuate the mean squared prediction error on the data in the left-out $k$th fold, as a function of \( \alpha \).</li>
<p><li> Then we average the results for each value of \( alpha \), and pick \( \alpha \) to minimize the average error.</li>
<p><li> Return the subtree from Step 2 that corresponds to the chosen value of \( \alpha \).</li>
</ol>
</div>
</section>
<section>
<h2 id="___sec11">A classification tree </h2>
<p>
A classification tree is very similar to a regression tree, except
that it is used to predict a qualitative response rather than a
quantitative one. Recall that for a regression tree, the predicted
response for an observation is given by the mean response of the
training observations that belong to the same terminal node. In
contrast, for a classification tree, we predict that each observation
belongs to the most commonly occurring class of training observations
in the region to which it belongs. In interpreting the results of a
classification tree, we are often interested not only in the class
prediction corresponding to a particular terminal node region, but
also in the class proportions among the training observations that
fall into that region.
</section>
<section>
<h2 id="___sec12">Growing a classification tree </h2>
<p>
The task of growing a
classification tree is quite similar to the task of growing a
regression tree. Just as in the regression setting, we use recursive
binary splitting to grow a classification tree. However, in the
classification setting, the MSE cannot be used as a criterion for making
the binary splits. A natural alternative to MSE is the <b>classification
error rate</b>. Since we plan to assign an observation in a given region
to the most commonly occurring error rate class of training
observations in that region, the classification error rate is simply
the fraction of the training observations in that region that do not
belong to the most common class.
<p>
When building a classification tree, either the Gini index or the
entropy are typically used to evaluate the quality of a particular
split, since these two approaches are more sensitive to node purity
than is the classification error rate.
</section>
<section>
<h2 id="___sec13">Pros and cons of trees, pros </h2>
<ul>
<p><li> White box, easy to interpret model. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches discussed earlier (think of support vector machines)</li>
<p><li> Trees are very easy to explain to people. In fact, they are even easier to explain than linear regression!</li>
<p><li> No feature normalization needed</li>
<p><li> Tree models can handle both continuous and categorical data (Classification and Regression Trees)</li>
<p><li> Can model nonlinear relationships</li>
<p><li> Can model interactions between the different descriptive features</li>
<p><li> Trees can be displayed graphically, and are easily interpreted even by a non-expert (especially if they are small)</li>
</ul>
</section>
<section>
<h2 id="___sec14">Disadvantages </h2>
<ul>
<p><li> Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches</li>
<p><li> If continuous features are used the tree may become quite large and hence less interpretable</li>
<p><li> Decision trees are prone to overfit the training data and hence do not well generalize the data if no stopping criteria or improvements like pruning, boosting or bagging are implemented</li>
<p><li> Small changes in the data may lead to a completely different tree. This issue can be addressed by using ensemble methods like bagging, boosting or random forests</li>
<p><li> Unbalanced datasets where some target feature values occur much more frequently than others may lead to biased trees since the frequently occurring feature values are preferred over the less frequently occurring ones.</li>
<p><li> If the number of features is relatively large (high dimensional) and the number of instances is relatively low, the tree might overfit the data</li>
<p><li> Features with many levels may be preferred over features with less levels since for them it is <em>more easy</em> to split the dataset such that the sub datasets only contain pure target feature values. This issue can be addressed by preferring for instance the information gain ratio as splitting criteria over information gain</li>
</ul>
<p>
However, by aggregating many decision trees, using methods like bagging, random forests, and boosting, the predictive performance of trees can be substantially improved.
</section>
<section>
<h2 id="___sec15">Bagging </h2>
<p>
The <b>plain</b> decision trees suffer from high
variance. This means that if we split the training data into two parts
at random, and fit a decision tree to both halves, the results that we
get could be quite different. In contrast, a procedure with low
variance will yield similar results if applied repeatedly to distinct
data sets; linear regression tends to have low variance, if the ratio
of \( n \) to \( p \) is moderately large.
<p>
<b>Bootstrap aggregation</b>, or just <b>bagging</b>, is a
general-purpose procedure for reducing the variance of a statistical
learning method.
<p>
Bagging typically results in improved accuracy
over prediction using a single tree. Unfortunately, however, it can be
difficult to interpret the resulting model. Recall that one of the
advantages of decision trees is the attractive and easily interpreted
diagram that results.
<p>
However, when we bag a large number of trees, it is no longer
possible to represent the resulting statistical learning procedure
using a single tree, and it is no longer clear which variables are
most important to the procedure. Thus, bagging improves prediction
accuracy at the expense of interpretability. Although the collection
of bagged trees is much more difficult to interpret than a single
tree, one can obtain an overall summary of the importance of each
predictor using the MSE (for bagging regression trees) or the Gini
index (for bagging classification trees). In the case of bagging
regression trees, we can record the total amount that the MSE is
decreased due to splits over a given predictor, averaged over all \( B \) possible
trees. A large value indicates an important predictor. Similarly, in
the context of bagging classification trees, we can add up the total
amount that the Gini index is decreased by splits over a given
predictor, averaged over all \( B \) trees.
</section>
<section>
<h2 id="___sec16">Random forests </h2>
<p>
Random forests provide an improvement over bagged trees by way of a
small tweak that decorrelates the trees.
<p>
As in bagging, we build a
number of decision trees on bootstrapped training samples. But when
building these decision trees, each time a split in a tree is
considered, a random sample of \( m \) predictors is chosen as split
candidates from the full set of \( p \) predictors. The split is allowed to
use only one of those \( m \) predictors.
<p>
A fresh sample of \( m \) predictors is
taken at each split, and typically we choose
<p>&nbsp;<br>
$$
m\approx \sqrt{p}.
$$
<p>&nbsp;<br>
In building a random forest, at
each split in the tree, the algorithm is not even allowed to consider
a majority of the available predictors.
<p>
The reason for this is rather clever. Suppose that there is one very
strong predictor in the data set, along with a number of other
moderately strong predictors. Then in the collection of bagged
variable importance random forest trees, most or all of the trees will
use this strong predictor in the top split. Consequently, all of the
bagged trees will look quite similar to each other. Hence the
predictions from the bagged trees will be highly correlated.
Unfortunately, averaging many highly correlated quantities does not lead
to as large of a reduction in variance as averaging many uncorrelated
quanti- ties. In particular, this means that bagging will not lead to
a substantial reduction in variance over a single tree in this
setting.
</section>
<section>
<h2 id="___sec17">A simple scikit-learn example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.ensemble</span> <span style="color: #8B008B; font-weight: bold">import</span> RandomForestClassifier
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> LabelEncoder
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.model_selection</span> <span style="color: #8B008B; font-weight: bold">import</span> cross_validate
<span style="color: #228B22"># Data set not specificied</span>
X = dataset.XXX
Y = dataset.YYY
<span style="color: #228B22">#Instantiate the model with 100 trees and entropy as splitting criteria</span>
Random_Forest_model = RandomForestClassifier(n_estimators=<span style="color: #B452CD">100</span>,criterion=<span style="color: #CD5555">&quot;entropy&quot;</span>)
<span style="color: #228B22">#Cross validation</span>
accuracy = cross_validate(Random_Forest_model,X,Y,cv=<span style="color: #B452CD">10</span>)[<span style="color: #CD5555">&#39;test_score&#39;</span>]
</pre></div>
</section>
</div> <!-- class="slides" -->
</div> <!-- class="reveal" -->
doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
+384 -3
View File
@@ -64,11 +64,43 @@ div { text-align: justify; text-justify: inter-word; }
('Nodes, leafs, roots and branches', 2, None, '___sec1'),
('How do we set it up?', 2, None, '___sec2'),
('Decision trees and Regression', 2, None, '___sec3'),
('Maxwell-Boltzmann velocity distribution', 2, None, '___sec4')]}
('Maxwell-Boltzmann velocity distribution', 2, None, '___sec4'),
('Building a tree, regression', 2, None, '___sec5'),
('A top-down approach, recursive binary splitting',
2,
None,
'___sec6'),
('Making a tree', 2, None, '___sec7'),
('Pruning the tree', 2, None, '___sec8'),
('Cost complexity pruning', 2, None, '___sec9'),
('A schematic procedure', 2, None, '___sec10'),
('A classification tree', 2, None, '___sec11'),
('Growing a classification tree', 2, None, '___sec12'),
('Pros and cons of trees, pros', 2, None, '___sec13'),
('Disadvantages', 2, None, '___sec14'),
('Bagging', 2, None, '___sec15'),
('Random forests', 2, None, '___sec16'),
('A simple scikit-learn example', 2, None, '___sec17')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "AMS" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- ------------------- main content ---------------------- -->
@@ -90,7 +122,7 @@ end of tocinfo -->
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<center><h4>Nov 9, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -123,7 +155,7 @@ The main idea of decision trees
is to find those descriptive features which contain the most
<b>information</b> regarding the target feature and then split the dataset
along the values of these features such that the target feature values
for the resulting sub\_datasets are as pure as possible.
for the resulting sub datasets are as pure as possible.
<p>
The descriptive feature which leaves the target feature most purely is said
@@ -362,6 +394,355 @@ plt.show()
<span style="color: #228B22">#i.e. frequency of all velocities in range -5 to -4.9, -4.9 to -4.8, etc...</span>
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Building a tree, regression </h2>
<p>
There are mainly two steps
<ol>
<li> We split the predictor space (the set of possible values \( x_1,x_2,\dots, x_p \)) into \( J \)</li>
</ol>
distinct and non-non-overlapping regions, \( R_1,R_2,\dots,R_J \).
<ol>
<li> For every observation that falls into the region \( R_j \) , we make the same prediction, which is simply the mean of the response values for the training observations in \( R_j \).</li>
</ol>
How do we construct the regions \( R_1,\dots,R_J \)?
In theory, the regions could have any shape. However, we
choose to divide the predictor space into high-dimensional rectangles,
or boxes, for simplicity and for ease of interpretation of the
resulting predic- tive model. The goal is to find boxes \( R_1,\dots,R_J \)
that minimize the MSE, given by
$$
\sum_{j=1}^J\sum_{i\in R_j}(y_i-\overline{y}_{R_j})^2,
$$
where \( \overline{y}_{R_j} \) is the mean response for the training observations
within the $j$th
box.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">A top-down approach, recursive binary splitting </h2>
<p>
Unfortunately, it is computationally infeasible to consider every
possible partition of the feature space into \( J \) boxes.
The common strategy is to take a top-down approach
<p>
The approach is top-down because it begins at the top of the tree (all
observations belong to a single region) and then successively splits
the predictor space; each split is indicated via two new branches
further down on the tree. It is greedy because at each step of the
tree-building process, the best split is made at that particular step,
rather than looking ahead and picking a split that will lead to a
better tree in some future step.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Making a tree </h2>
<p>
In order to implement the recursive binary splitting we start by selecting
the predictor \( x_j \) and a cutpoint \( s \) that splits the predictor space into two regions \( R_1 \) and \( R_2 \)
$$
\left\{X\vert x_j < s\right\},
$$
and
$$
\left\{X\vert x_j \geq s\right\},
$$
so that we obtain the lowest MSE, that is
$$
\sum_{i:x_i\in R_j}(y_i-\overline{y}_{R_1})^2+\sum_{i:x_i\in R_2}(y_i-\overline{y}_{R_2})^2,
$$
which we want to minimize by considering all predictors \( x_1,x_2,\dots,x_p \).
We consider also all possible values of \( s \) for each predictor. These values could be determined by randomly assigned numbers or by starting at the midpoint and then proceed till we find an optimal value.
<p>
For any \( j \) and \( s \), we define the pair of
half-planes where \( \overline{y}_{R_1} \) is the mean response for the training
observations in \( R_1(j,s) \), and \( \overline{y}_{R_2} \) is the mean response for the
training observations in \( R_2(j,s) \).
<p>
Finding the values of j and s that
minimize the above equation can be done quite quickly, especially when the number
of features \( p \) is not too large.
<p>
Next, we repeat the process, looking
for the best predictor and best cutpoint in order to split the data
further so as to minimize the RSS within each of the resulting
regions. However, this time, instead of splitting the entire predictor
space, we split one of the two previously identified regions. We now
have three regions. Again, we look to split one of these three regions
further, so as to minimize the MSE. The process continues until a
stopping criterion is reached; for instance, we may continue until no
region contains more than five observations.
<p>
<!-- !split -->
<h2 id="___sec8">Pruning the tree </h2>
<p>
The above procedure is rather straightforward, but leads often to
overfitting and unnecessarily large and complicated trees. The basic
idea is to grow a large tree \( T_0 \) and then prune it back in order to
obtain a subtree. A smaller tree with fewer splits (fewer regions) can
lead to smaller variance and better interpretation at the cost of a
little more bias.
<p>
The so-called Cost complexity pruning algorithm gives us a
way to do just this. Rather than considering every possible subtree,
we consider a sequence of trees indexed by a nonnegative tuning
parameter \( \alpha \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Cost complexity pruning </h2>
For each value of \( \alpha \) there corresponds a subtree \( T \in T_0 \) such that
$$
\sum_{m=1}^{\overline{T}}\sum_{i:x_i\in R_m}(y_i-\overline{y}_{R_m})^2+\alpha\overline{T},
$$
is as small as possible. Here \( \overline{T} \) is
the number of terminal nodes of the tree \( T \) , \( R_m \) is the
rectangle (i.e. the subset of predictor space) corresponding to the $m$th terminal node.
<p>
The tuning parameter \( \alpha \) controls a trade-off between the subtree&#8217;s
com- plexity and its fit to the training data. When \( \alpha = 0 \), then the
subtree \( T \) will simply equal \( T_0 \),
because then the above equation just measures the
training error.
However, as \( \alpha \) increases, there is a price to pay for
having a tree with many terminal nodes. The above equation will
tend to be minimized for a smaller subtree.
<p>
It turns out that as we increase \( \alpha \) from zero
branches get pruned from the tree in a nested and predictable fashion,
so obtaining the whole sequence of subtrees as a function of \( \alpha \) is
easy. We can select a value of \( \alpha \) using a validation set or using
cross-validation. We then return to the full data set and obtain the
subtree corresponding to \( \alpha \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">A schematic procedure </h2>
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>Building a Regression Tree.</b>
<p>
<ol>
<li> Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
<ol type="a"></li>
<li> Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of \( \alpha \).</li>
</ol>
<li> Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we repeat Steps 1 and 2 on all but the $k$th fold of the training data. Then we valuate the mean squared prediction error on the data in the left-out $k$th fold, as a function of \( \alpha \).</li>
<li> Then we average the results for each value of \( alpha \), and pick \( \alpha \) to minimize the average error.</li>
<li> Return the subtree from Step 2 that corresponds to the chosen value of \( \alpha \).</li>
</ol>
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">A classification tree </h2>
<p>
A classification tree is very similar to a regression tree, except
that it is used to predict a qualitative response rather than a
quantitative one. Recall that for a regression tree, the predicted
response for an observation is given by the mean response of the
training observations that belong to the same terminal node. In
contrast, for a classification tree, we predict that each observation
belongs to the most commonly occurring class of training observations
in the region to which it belongs. In interpreting the results of a
classification tree, we are often interested not only in the class
prediction corresponding to a particular terminal node region, but
also in the class proportions among the training observations that
fall into that region.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Growing a classification tree </h2>
<p>
The task of growing a
classification tree is quite similar to the task of growing a
regression tree. Just as in the regression setting, we use recursive
binary splitting to grow a classification tree. However, in the
classification setting, the MSE cannot be used as a criterion for making
the binary splits. A natural alternative to MSE is the <b>classification
error rate</b>. Since we plan to assign an observation in a given region
to the most commonly occurring error rate class of training
observations in that region, the classification error rate is simply
the fraction of the training observations in that region that do not
belong to the most common class.
<p>
When building a classification tree, either the Gini index or the
entropy are typically used to evaluate the quality of a particular
split, since these two approaches are more sensitive to node purity
than is the classification error rate.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Pros and cons of trees, pros </h2>
<ul>
<li> White box, easy to interpret model. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches discussed earlier (think of support vector machines)</li>
<li> Trees are very easy to explain to people. In fact, they are even easier to explain than linear regression!</li>
<li> No feature normalization needed</li>
<li> Tree models can handle both continuous and categorical data (Classification and Regression Trees)</li>
<li> Can model nonlinear relationships</li>
<li> Can model interactions between the different descriptive features</li>
<li> Trees can be displayed graphically, and are easily interpreted even by a non-expert (especially if they are small)</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">Disadvantages </h2>
<ul>
<li> Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches</li>
<li> If continuous features are used the tree may become quite large and hence less interpretable</li>
<li> Decision trees are prone to overfit the training data and hence do not well generalize the data if no stopping criteria or improvements like pruning, boosting or bagging are implemented</li>
<li> Small changes in the data may lead to a completely different tree. This issue can be addressed by using ensemble methods like bagging, boosting or random forests</li>
<li> Unbalanced datasets where some target feature values occur much more frequently than others may lead to biased trees since the frequently occurring feature values are preferred over the less frequently occurring ones.</li>
<li> If the number of features is relatively large (high dimensional) and the number of instances is relatively low, the tree might overfit the data</li>
<li> Features with many levels may be preferred over features with less levels since for them it is <em>more easy</em> to split the dataset such that the sub datasets only contain pure target feature values. This issue can be addressed by preferring for instance the information gain ratio as splitting criteria over information gain</li>
</ul>
However, by aggregating many decision trees, using methods like bagging, random forests, and boosting, the predictive performance of trees can be substantially improved.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Bagging </h2>
<p>
The <b>plain</b> decision trees suffer from high
variance. This means that if we split the training data into two parts
at random, and fit a decision tree to both halves, the results that we
get could be quite different. In contrast, a procedure with low
variance will yield similar results if applied repeatedly to distinct
data sets; linear regression tends to have low variance, if the ratio
of \( n \) to \( p \) is moderately large.
<p>
<b>Bootstrap aggregation</b>, or just <b>bagging</b>, is a
general-purpose procedure for reducing the variance of a statistical
learning method.
<p>
Bagging typically results in improved accuracy
over prediction using a single tree. Unfortunately, however, it can be
difficult to interpret the resulting model. Recall that one of the
advantages of decision trees is the attractive and easily interpreted
diagram that results.
<p>
However, when we bag a large number of trees, it is no longer
possible to represent the resulting statistical learning procedure
using a single tree, and it is no longer clear which variables are
most important to the procedure. Thus, bagging improves prediction
accuracy at the expense of interpretability. Although the collection
of bagged trees is much more difficult to interpret than a single
tree, one can obtain an overall summary of the importance of each
predictor using the MSE (for bagging regression trees) or the Gini
index (for bagging classification trees). In the case of bagging
regression trees, we can record the total amount that the MSE is
decreased due to splits over a given predictor, averaged over all \( B \) possible
trees. A large value indicates an important predictor. Similarly, in
the context of bagging classification trees, we can add up the total
amount that the Gini index is decreased by splits over a given
predictor, averaged over all \( B \) trees.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">Random forests </h2>
<p>
Random forests provide an improvement over bagged trees by way of a
small tweak that decorrelates the trees.
<p>
As in bagging, we build a
number of decision trees on bootstrapped training samples. But when
building these decision trees, each time a split in a tree is
considered, a random sample of \( m \) predictors is chosen as split
candidates from the full set of \( p \) predictors. The split is allowed to
use only one of those \( m \) predictors.
<p>
A fresh sample of \( m \) predictors is
taken at each split, and typically we choose
$$
m\approx \sqrt{p}.
$$
In building a random forest, at
each split in the tree, the algorithm is not even allowed to consider
a majority of the available predictors.
<p>
The reason for this is rather clever. Suppose that there is one very
strong predictor in the data set, along with a number of other
moderately strong predictors. Then in the collection of bagged
variable importance random forest trees, most or all of the trees will
use this strong predictor in the top split. Consequently, all of the
bagged trees will look quite similar to each other. Hence the
predictions from the bagged trees will be highly correlated.
Unfortunately, averaging many highly correlated quantities does not lead
to as large of a reduction in variance as averaging many uncorrelated
quanti- ties. In particular, this means that bagging will not lead to
a substantial reduction in variance over a single tree in this
setting.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">A simple scikit-learn example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.ensemble</span> <span style="color: #8B008B; font-weight: bold">import</span> RandomForestClassifier
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> LabelEncoder
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.model_selection</span> <span style="color: #8B008B; font-weight: bold">import</span> cross_validate
<span style="color: #228B22"># Data set not specificied</span>
X = dataset.XXX
Y = dataset.YYY
<span style="color: #228B22">#Instantiate the model with 100 trees and entropy as splitting criteria</span>
Random_Forest_model = RandomForestClassifier(n_estimators=<span style="color: #B452CD">100</span>,criterion=<span style="color: #CD5555">&quot;entropy&quot;</span>)
<span style="color: #228B22">#Cross validation</span>
accuracy = cross_validate(Random_Forest_model,X,Y,cv=<span style="color: #B452CD">10</span>)[<span style="color: #CD5555">&#39;test_score&#39;</span>]
</pre></div>
<p>
<!-- ------------------- end of main content --------------- -->
doc/pub/DecisionTrees/html/DecisionTrees.html
+384 -3
View File
@@ -69,11 +69,43 @@ div { text-align: justify; text-justify: inter-word; }
('Nodes, leafs, roots and branches', 2, None, '___sec1'),
('How do we set it up?', 2, None, '___sec2'),
('Decision trees and Regression', 2, None, '___sec3'),
('Maxwell-Boltzmann velocity distribution', 2, None, '___sec4')]}
('Maxwell-Boltzmann velocity distribution', 2, None, '___sec4'),
('Building a tree, regression', 2, None, '___sec5'),
('A top-down approach, recursive binary splitting',
2,
None,
'___sec6'),
('Making a tree', 2, None, '___sec7'),
('Pruning the tree', 2, None, '___sec8'),
('Cost complexity pruning', 2, None, '___sec9'),
('A schematic procedure', 2, None, '___sec10'),
('A classification tree', 2, None, '___sec11'),
('Growing a classification tree', 2, None, '___sec12'),
('Pros and cons of trees, pros', 2, None, '___sec13'),
('Disadvantages', 2, None, '___sec14'),
('Bagging', 2, None, '___sec15'),
('Random forests', 2, None, '___sec16'),
('A simple scikit-learn example', 2, None, '___sec17')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "AMS" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- ------------------- main content ---------------------- -->
@@ -95,7 +127,7 @@ end of tocinfo -->
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<center><h4>Nov 9, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -128,7 +160,7 @@ The main idea of decision trees
is to find those descriptive features which contain the most
<b>information</b> regarding the target feature and then split the dataset
along the values of these features such that the target feature values
for the resulting sub\_datasets are as pure as possible.
for the resulting sub datasets are as pure as possible.
<p>
The descriptive feature which leaves the target feature most purely is said
@@ -367,6 +399,355 @@ plt<span style="color: #666666">.</span>show()
<span style="color: #408080; font-style: italic">#i.e. frequency of all velocities in range -5 to -4.9, -4.9 to -4.8, etc...</span>
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Building a tree, regression </h2>
<p>
There are mainly two steps
<ol>
<li> We split the predictor space (the set of possible values \( x_1,x_2,\dots, x_p \)) into \( J \)</li>
</ol>
distinct and non-non-overlapping regions, \( R_1,R_2,\dots,R_J \).
<ol>
<li> For every observation that falls into the region \( R_j \) , we make the same prediction, which is simply the mean of the response values for the training observations in \( R_j \).</li>
</ol>
How do we construct the regions \( R_1,\dots,R_J \)?
In theory, the regions could have any shape. However, we
choose to divide the predictor space into high-dimensional rectangles,
or boxes, for simplicity and for ease of interpretation of the
resulting predic- tive model. The goal is to find boxes \( R_1,\dots,R_J \)
that minimize the MSE, given by
$$
\sum_{j=1}^J\sum_{i\in R_j}(y_i-\overline{y}_{R_j})^2,
$$
where \( \overline{y}_{R_j} \) is the mean response for the training observations
within the $j$th
box.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">A top-down approach, recursive binary splitting </h2>
<p>
Unfortunately, it is computationally infeasible to consider every
possible partition of the feature space into \( J \) boxes.
The common strategy is to take a top-down approach
<p>
The approach is top-down because it begins at the top of the tree (all
observations belong to a single region) and then successively splits
the predictor space; each split is indicated via two new branches
further down on the tree. It is greedy because at each step of the
tree-building process, the best split is made at that particular step,
rather than looking ahead and picking a split that will lead to a
better tree in some future step.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Making a tree </h2>
<p>
In order to implement the recursive binary splitting we start by selecting
the predictor \( x_j \) and a cutpoint \( s \) that splits the predictor space into two regions \( R_1 \) and \( R_2 \)
$$
\left\{X\vert x_j < s\right\},
$$
and
$$
\left\{X\vert x_j \geq s\right\},
$$
so that we obtain the lowest MSE, that is
$$
\sum_{i:x_i\in R_j}(y_i-\overline{y}_{R_1})^2+\sum_{i:x_i\in R_2}(y_i-\overline{y}_{R_2})^2,
$$
which we want to minimize by considering all predictors \( x_1,x_2,\dots,x_p \).
We consider also all possible values of \( s \) for each predictor. These values could be determined by randomly assigned numbers or by starting at the midpoint and then proceed till we find an optimal value.
<p>
For any \( j \) and \( s \), we define the pair of
half-planes where \( \overline{y}_{R_1} \) is the mean response for the training
observations in \( R_1(j,s) \), and \( \overline{y}_{R_2} \) is the mean response for the
training observations in \( R_2(j,s) \).
<p>
Finding the values of j and s that
minimize the above equation can be done quite quickly, especially when the number
of features \( p \) is not too large.
<p>
Next, we repeat the process, looking
for the best predictor and best cutpoint in order to split the data
further so as to minimize the RSS within each of the resulting
regions. However, this time, instead of splitting the entire predictor
space, we split one of the two previously identified regions. We now
have three regions. Again, we look to split one of these three regions
further, so as to minimize the MSE. The process continues until a
stopping criterion is reached; for instance, we may continue until no
region contains more than five observations.
<p>
<!-- !split -->
<h2 id="___sec8">Pruning the tree </h2>
<p>
The above procedure is rather straightforward, but leads often to
overfitting and unnecessarily large and complicated trees. The basic
idea is to grow a large tree \( T_0 \) and then prune it back in order to
obtain a subtree. A smaller tree with fewer splits (fewer regions) can
lead to smaller variance and better interpretation at the cost of a
little more bias.
<p>
The so-called Cost complexity pruning algorithm gives us a
way to do just this. Rather than considering every possible subtree,
we consider a sequence of trees indexed by a nonnegative tuning
parameter \( \alpha \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Cost complexity pruning </h2>
For each value of \( \alpha \) there corresponds a subtree \( T \in T_0 \) such that
$$
\sum_{m=1}^{\overline{T}}\sum_{i:x_i\in R_m}(y_i-\overline{y}_{R_m})^2+\alpha\overline{T},
$$
is as small as possible. Here \( \overline{T} \) is
the number of terminal nodes of the tree \( T \) , \( R_m \) is the
rectangle (i.e. the subset of predictor space) corresponding to the $m$th terminal node.
<p>
The tuning parameter \( \alpha \) controls a trade-off between the subtree&#8217;s
com- plexity and its fit to the training data. When \( \alpha = 0 \), then the
subtree \( T \) will simply equal \( T_0 \),
because then the above equation just measures the
training error.
However, as \( \alpha \) increases, there is a price to pay for
having a tree with many terminal nodes. The above equation will
tend to be minimized for a smaller subtree.
<p>
It turns out that as we increase \( \alpha \) from zero
branches get pruned from the tree in a nested and predictable fashion,
so obtaining the whole sequence of subtrees as a function of \( \alpha \) is
easy. We can select a value of \( \alpha \) using a validation set or using
cross-validation. We then return to the full data set and obtain the
subtree corresponding to \( \alpha \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">A schematic procedure </h2>
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>Building a Regression Tree.</b>
<p>
<ol>
<li> Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
<ol type="a"></li>
<li> Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of \( \alpha \).</li>
</ol>
<li> Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we repeat Steps 1 and 2 on all but the $k$th fold of the training data. Then we valuate the mean squared prediction error on the data in the left-out $k$th fold, as a function of \( \alpha \).</li>
<li> Then we average the results for each value of \( alpha \), and pick \( \alpha \) to minimize the average error.</li>
<li> Return the subtree from Step 2 that corresponds to the chosen value of \( \alpha \).</li>
</ol>
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">A classification tree </h2>
<p>
A classification tree is very similar to a regression tree, except
that it is used to predict a qualitative response rather than a
quantitative one. Recall that for a regression tree, the predicted
response for an observation is given by the mean response of the
training observations that belong to the same terminal node. In
contrast, for a classification tree, we predict that each observation
belongs to the most commonly occurring class of training observations
in the region to which it belongs. In interpreting the results of a
classification tree, we are often interested not only in the class
prediction corresponding to a particular terminal node region, but
also in the class proportions among the training observations that
fall into that region.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Growing a classification tree </h2>
<p>
The task of growing a
classification tree is quite similar to the task of growing a
regression tree. Just as in the regression setting, we use recursive
binary splitting to grow a classification tree. However, in the
classification setting, the MSE cannot be used as a criterion for making
the binary splits. A natural alternative to MSE is the <b>classification
error rate</b>. Since we plan to assign an observation in a given region
to the most commonly occurring error rate class of training
observations in that region, the classification error rate is simply
the fraction of the training observations in that region that do not
belong to the most common class.
<p>
When building a classification tree, either the Gini index or the
entropy are typically used to evaluate the quality of a particular
split, since these two approaches are more sensitive to node purity
than is the classification error rate.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Pros and cons of trees, pros </h2>
<ul>
<li> White box, easy to interpret model. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches discussed earlier (think of support vector machines)</li>
<li> Trees are very easy to explain to people. In fact, they are even easier to explain than linear regression!</li>
<li> No feature normalization needed</li>
<li> Tree models can handle both continuous and categorical data (Classification and Regression Trees)</li>
<li> Can model nonlinear relationships</li>
<li> Can model interactions between the different descriptive features</li>
<li> Trees can be displayed graphically, and are easily interpreted even by a non-expert (especially if they are small)</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">Disadvantages </h2>
<ul>
<li> Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches</li>
<li> If continuous features are used the tree may become quite large and hence less interpretable</li>
<li> Decision trees are prone to overfit the training data and hence do not well generalize the data if no stopping criteria or improvements like pruning, boosting or bagging are implemented</li>
<li> Small changes in the data may lead to a completely different tree. This issue can be addressed by using ensemble methods like bagging, boosting or random forests</li>
<li> Unbalanced datasets where some target feature values occur much more frequently than others may lead to biased trees since the frequently occurring feature values are preferred over the less frequently occurring ones.</li>
<li> If the number of features is relatively large (high dimensional) and the number of instances is relatively low, the tree might overfit the data</li>
<li> Features with many levels may be preferred over features with less levels since for them it is <em>more easy</em> to split the dataset such that the sub datasets only contain pure target feature values. This issue can be addressed by preferring for instance the information gain ratio as splitting criteria over information gain</li>
</ul>
However, by aggregating many decision trees, using methods like bagging, random forests, and boosting, the predictive performance of trees can be substantially improved.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Bagging </h2>
<p>
The <b>plain</b> decision trees suffer from high
variance. This means that if we split the training data into two parts
at random, and fit a decision tree to both halves, the results that we
get could be quite different. In contrast, a procedure with low
variance will yield similar results if applied repeatedly to distinct
data sets; linear regression tends to have low variance, if the ratio
of \( n \) to \( p \) is moderately large.
<p>
<b>Bootstrap aggregation</b>, or just <b>bagging</b>, is a
general-purpose procedure for reducing the variance of a statistical
learning method.
<p>
Bagging typically results in improved accuracy
over prediction using a single tree. Unfortunately, however, it can be
difficult to interpret the resulting model. Recall that one of the
advantages of decision trees is the attractive and easily interpreted
diagram that results.
<p>
However, when we bag a large number of trees, it is no longer
possible to represent the resulting statistical learning procedure
using a single tree, and it is no longer clear which variables are
most important to the procedure. Thus, bagging improves prediction
accuracy at the expense of interpretability. Although the collection
of bagged trees is much more difficult to interpret than a single
tree, one can obtain an overall summary of the importance of each
predictor using the MSE (for bagging regression trees) or the Gini
index (for bagging classification trees). In the case of bagging
regression trees, we can record the total amount that the MSE is
decreased due to splits over a given predictor, averaged over all \( B \) possible
trees. A large value indicates an important predictor. Similarly, in
the context of bagging classification trees, we can add up the total
amount that the Gini index is decreased by splits over a given
predictor, averaged over all \( B \) trees.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">Random forests </h2>
<p>
Random forests provide an improvement over bagged trees by way of a
small tweak that decorrelates the trees.
<p>
As in bagging, we build a
number of decision trees on bootstrapped training samples. But when
building these decision trees, each time a split in a tree is
considered, a random sample of \( m \) predictors is chosen as split
candidates from the full set of \( p \) predictors. The split is allowed to
use only one of those \( m \) predictors.
<p>
A fresh sample of \( m \) predictors is
taken at each split, and typically we choose
$$
m\approx \sqrt{p}.
$$
In building a random forest, at
each split in the tree, the algorithm is not even allowed to consider
a majority of the available predictors.
<p>
The reason for this is rather clever. Suppose that there is one very
strong predictor in the data set, along with a number of other
moderately strong predictors. Then in the collection of bagged
variable importance random forest trees, most or all of the trees will
use this strong predictor in the top split. Consequently, all of the
bagged trees will look quite similar to each other. Hence the
predictions from the bagged trees will be highly correlated.
Unfortunately, averaging many highly correlated quantities does not lead
to as large of a reduction in variance as averaging many uncorrelated
quanti- ties. In particular, this means that bagging will not lead to
a substantial reduction in variance over a single tree in this
setting.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">A simple scikit-learn example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.ensemble</span> <span style="color: #008000; font-weight: bold">import</span> RandomForestClassifier
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> LabelEncoder
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> cross_validate
<span style="color: #408080; font-style: italic"># Data set not specificied</span>
X <span style="color: #666666">=</span> dataset<span style="color: #666666">.</span>XXX
Y <span style="color: #666666">=</span> dataset<span style="color: #666666">.</span>YYY
<span style="color: #408080; font-style: italic">#Instantiate the model with 100 trees and entropy as splitting criteria</span>
Random_Forest_model <span style="color: #666666">=</span> RandomForestClassifier(n_estimators<span style="color: #666666">=100</span>,criterion<span style="color: #666666">=</span><span style="color: #BA2121">&quot;entropy&quot;</span>)
<span style="color: #408080; font-style: italic">#Cross validation</span>
accuracy <span style="color: #666666">=</span> cross_validate(Random_Forest_model,X,Y,cv<span style="color: #666666">=10</span>)[<span style="color: #BA2121">&#39;test_score&#39;</span>]
</pre></div>
<p>
<!-- ------------------- end of main content --------------- -->
doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/DecisionTrees/pdf/DecisionTrees-beamer-handouts2x3.pdf
BIN
View File
Binary file not shown.
doc/pub/DecisionTrees/pdf/DecisionTrees-beamer.pdf
BIN
View File
Binary file not shown.
doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf
BIN
View File
Binary file not shown.
doc/pub/svm/ipynb/svm.ipynb
+153 -27
View File
File diff suppressed because one or more lines are too long
doc/src/DecisionTrees/DecisionTrees.do.txt
+289 -1
View File
@@ -26,7 +26,7 @@ The main idea of decision trees
is to find those descriptive features which contain the most
_information_ regarding the target feature and then split the dataset
along the values of these features such that the target feature values
for the resulting sub\_datasets are as pure as possible.
for the resulting sub datasets are as pure as possible.
The descriptive feature which leaves the target feature most purely is said
to be the most informative one. This process of finding the _most
@@ -65,6 +65,7 @@ Then we are essentially done!
!split
===== Decision trees and Regression =====
!bc pycod
@@ -262,6 +263,293 @@ plt.show()
!ec
!split
===== Building a tree, regression =====
There are mainly two steps
o We split the predictor space (the set of possible values $x_1,x_2,\dots, x_p$) into $J$
distinct and non-non-overlapping regions, $R_1,R_2,\dots,R_J$.
o For every observation that falls into the region $R_j$ , we make the same prediction, which is simply the mean of the response values for the training observations in $R_j$.
How do we construct the regions $R_1,\dots,R_J$?
In theory, the regions could have any shape. However, we
choose to divide the predictor space into high-dimensional rectangles,
or boxes, for simplicity and for ease of interpretation of the
resulting predic- tive model. The goal is to find boxes $R_1,\dots,R_J$
that minimize the MSE, given by
!bt
\[
\sum_{j=1}^J\sum_{i\in R_j}(y_i-\overline{y}_{R_j})^2,
\]
!et
where $\overline{y}_{R_j}$ is the mean response for the training observations
within the $j$th
box.
!split
===== A top-down approach, recursive binary splitting =====
Unfortunately, it is computationally infeasible to consider every
possible partition of the feature space into $J$ boxes.
The common strategy is to take a top-down approach
The approach is top-down because it begins at the top of the tree (all
observations belong to a single region) and then successively splits
the predictor space; each split is indicated via two new branches
further down on the tree. It is greedy because at each step of the
tree-building process, the best split is made at that particular step,
rather than looking ahead and picking a split that will lead to a
better tree in some future step.
!split
===== Making a tree =====
In order to implement the recursive binary splitting we start by selecting
the predictor $x_j$ and a cutpoint $s$ that splits the predictor space into two regions $R_1$ and $R_2$
!bt
\[
\left\{X\vert x_j < s\right\},
\]
!et
and
!bt
\[
\left\{X\vert x_j \geq s\right\},
\]
!et
so that we obtain the lowest MSE, that is
!bt
\[
\sum_{i:x_i\in R_j}(y_i-\overline{y}_{R_1})^2+\sum_{i:x_i\in R_2}(y_i-\overline{y}_{R_2})^2,
\]
!et
which we want to minimize by considering all predictors $x_1,x_2,\dots,x_p$.
We consider also all possible values of $s$ for each predictor. These values could be determined by randomly assigned numbers or by starting at the midpoint and then proceed till we find an optimal value.
For any $j$ and $s$, we define the pair of
half-planes where $\overline{y}_{R_1}$ is the mean response for the training
observations in $R_1(j,s)$, and $\overline{y}_{R_2}$ is the mean response for the
training observations in $R_2(j,s)$.
Finding the values of j and s that
minimize the above equation can be done quite quickly, especially when the number
of features $p$ is not too large.
Next, we repeat the process, looking
for the best predictor and best cutpoint in order to split the data
further so as to minimize the RSS within each of the resulting
regions. However, this time, instead of splitting the entire predictor
space, we split one of the two previously identified regions. We now
have three regions. Again, we look to split one of these three regions
further, so as to minimize the MSE. The process continues until a
stopping criterion is reached; for instance, we may continue until no
region contains more than five observations.
!split
===== Pruning the tree =====
The above procedure is rather straightforward, but leads often to
overfitting and unnecessarily large and complicated trees. The basic
idea is to grow a large tree $T_0$ and then prune it back in order to
obtain a subtree. A smaller tree with fewer splits (fewer regions) can
lead to smaller variance and better interpretation at the cost of a
little more bias.
The so-called Cost complexity pruning algorithm gives us a
way to do just this. Rather than considering every possible subtree,
we consider a sequence of trees indexed by a nonnegative tuning
parameter $\alpha$.
!split
===== Cost complexity pruning =====
For each value of $\alpha$ there corresponds a subtree $T \in T_0$ such that
!bt
\[
\sum_{m=1}^{\overline{T}}\sum_{i:x_i\in R_m}(y_i-\overline{y}_{R_m})^2+\alpha\overline{T},
\]
!et
is as small as possible. Here $\overline{T}$ is
the number of terminal nodes of the tree $T$ , $R_m$ is the
rectangle (i.e. the subset of predictor space) corresponding to the $m$th terminal node.
The tuning parameter $\alpha$ controls a trade-off between the subtree’s
com- plexity and its fit to the training data. When $\alpha = 0$, then the
subtree $T$ will simply equal $T_0$,
because then the above equation just measures the
training error.
However, as $\alpha$ increases, there is a price to pay for
having a tree with many terminal nodes. The above equation will
tend to be minimized for a smaller subtree.
It turns out that as we increase $\alpha$ from zero
branches get pruned from the tree in a nested and predictable fashion,
so obtaining the whole sequence of subtrees as a function of $\alpha$ is
easy. We can select a value of $\alpha$ using a validation set or using
cross-validation. We then return to the full data set and obtain the
subtree corresponding to $\alpha$.
!split
===== A schematic procedure =====
!bblock Building a Regression Tree
o Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
o Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of $\alpha$.
o Use for example $K$-fold cross-validation to choose $\alpha$. Divide the training observations into $K$ folds. For each $k=1,2,\dots,K$ we repeat Steps 1 and 2 on all but the $k$th fold of the training data. Then we valuate the mean squared prediction error on the data in the left-out $k$th fold, as a function of $\alpha$.
o Then we average the results for each value of $alpha$, and pick $\alpha$ to minimize the average error.
o Return the subtree from Step 2 that corresponds to the chosen value of $\alpha$.
!eblock
!split
===== A classification tree =====
A classification tree is very similar to a regression tree, except
that it is used to predict a qualitative response rather than a
quantitative one. Recall that for a regression tree, the predicted
response for an observation is given by the mean response of the
training observations that belong to the same terminal node. In
contrast, for a classification tree, we predict that each observation
belongs to the most commonly occurring class of training observations
in the region to which it belongs. In interpreting the results of a
classification tree, we are often interested not only in the class
prediction corresponding to a particular terminal node region, but
also in the class proportions among the training observations that
fall into that region.
!split
===== Growing a classification tree =====
The task of growing a
classification tree is quite similar to the task of growing a
regression tree. Just as in the regression setting, we use recursive
binary splitting to grow a classification tree. However, in the
classification setting, the MSE cannot be used as a criterion for making
the binary splits. A natural alternative to MSE is the _classification
error rate_. Since we plan to assign an observation in a given region
to the most commonly occurring error rate class of training
observations in that region, the classification error rate is simply
the fraction of the training observations in that region that do not
belong to the most common class.
When building a classification tree, either the Gini index or the
entropy are typically used to evaluate the quality of a particular
split, since these two approaches are more sensitive to node purity
than is the classification error rate.
!split
===== Pros and cons of trees, pros =====
* White box, easy to interpret model. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches discussed earlier (think of support vector machines)
* Trees are very easy to explain to people. In fact, they are even easier to explain than linear regression!
* No feature normalization needed
* Tree models can handle both continuous and categorical data (Classification and Regression Trees)
* Can model nonlinear relationships
* Can model interactions between the different descriptive features
* Trees can be displayed graphically, and are easily interpreted even by a non-expert (especially if they are small)
!split
===== Disadvantages =====
* Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches
* If continuous features are used the tree may become quite large and hence less interpretable
* Decision trees are prone to overfit the training data and hence do not well generalize the data if no stopping criteria or improvements like pruning, boosting or bagging are implemented
* Small changes in the data may lead to a completely different tree. This issue can be addressed by using ensemble methods like bagging, boosting or random forests
* Unbalanced datasets where some target feature values occur much more frequently than others may lead to biased trees since the frequently occurring feature values are preferred over the less frequently occurring ones.
* If the number of features is relatively large (high dimensional) and the number of instances is relatively low, the tree might overfit the data
* Features with many levels may be preferred over features with less levels since for them it is *more easy* to split the dataset such that the sub datasets only contain pure target feature values. This issue can be addressed by preferring for instance the information gain ratio as splitting criteria over information gain
However, by aggregating many decision trees, using methods like bagging, random forests, and boosting, the predictive performance of trees can be substantially improved.
!split
===== Bagging =====
The _plain_ decision trees suffer from high
variance. This means that if we split the training data into two parts
at random, and fit a decision tree to both halves, the results that we
get could be quite different. In contrast, a procedure with low
variance will yield similar results if applied repeatedly to distinct
data sets; linear regression tends to have low variance, if the ratio
of $n$ to $p$ is moderately large.
_Bootstrap aggregation_, or just _bagging_, is a
general-purpose procedure for reducing the variance of a statistical
learning method.
Bagging typically results in improved accuracy
over prediction using a single tree. Unfortunately, however, it can be
difficult to interpret the resulting model. Recall that one of the
advantages of decision trees is the attractive and easily interpreted
diagram that results.
However, when we bag a large number of trees, it is no longer
possible to represent the resulting statistical learning procedure
using a single tree, and it is no longer clear which variables are
most important to the procedure. Thus, bagging improves prediction
accuracy at the expense of interpretability. Although the collection
of bagged trees is much more difficult to interpret than a single
tree, one can obtain an overall summary of the importance of each
predictor using the MSE (for bagging regression trees) or the Gini
index (for bagging classification trees). In the case of bagging
regression trees, we can record the total amount that the MSE is
decreased due to splits over a given predictor, averaged over all $B$ possible
trees. A large value indicates an important predictor. Similarly, in
the context of bagging classification trees, we can add up the total
amount that the Gini index is decreased by splits over a given
predictor, averaged over all $B$ trees.
!split
===== Random forests =====
Random forests provide an improvement over bagged trees by way of a
small tweak that decorrelates the trees.
As in bagging, we build a
number of decision trees on bootstrapped training samples. But when
building these decision trees, each time a split in a tree is
considered, a random sample of $m$ predictors is chosen as split
candidates from the full set of $p$ predictors. The split is allowed to
use only one of those $m$ predictors.
A fresh sample of $m$ predictors is
taken at each split, and typically we choose
!bt
\[
m\approx \sqrt{p}.
\]
!et
In building a random forest, at
each split in the tree, the algorithm is not even allowed to consider
a majority of the available predictors.
The reason for this is rather clever. Suppose that there is one very
strong predictor in the data set, along with a number of other
moderately strong predictors. Then in the collection of bagged
variable importance random forest trees, most or all of the trees will
use this strong predictor in the top split. Consequently, all of the
bagged trees will look quite similar to each other. Hence the
predictions from the bagged trees will be highly correlated.
Unfortunately, averaging many highly correlated quantities does not lead
to as large of a reduction in variance as averaging many uncorrelated
quanti- ties. In particular, this means that bagging will not lead to
a substantial reduction in variance over a single tree in this
setting.
!split
===== A simple scikit-learn example =====
!bc pycod
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import cross_validate
# Data set not specificied
X = dataset.XXX
Y = dataset.YYY
#Instantiate the model with 100 trees and entropy as splitting criteria
Random_Forest_model = RandomForestClassifier(n_estimators=100,criterion="entropy")
#Cross validation
accuracy = cross_validate(Random_Forest_model,X,Y,cv=10)['test_score']
!ec