update
This commit is contained in:
Morten Hjorth-Jensen
@@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'decision-trees-overarching-aims'),
|
||||
('Basics of a tree', 2, None, 'basics-of-a-tree'),
|
||||
('A Sketch of a Tree, Regression problem',
|
||||
2,
|
||||
None,
|
||||
'a-sketch-of-a-tree-regression-problem'),
|
||||
('A Sketch of a Tree, Classification problem',
|
||||
2,
|
||||
None,
|
||||
'a-sketch-of-a-tree-classification-problem'),
|
||||
('A typical Decision Tree with its pertinent Jargon, '
|
||||
'Classification Problem',
|
||||
2,
|
||||
@@ -257,67 +249,65 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs001.html#plan-for-week-46" style="font-size: 80%;">Plan for week 46</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs002.html#decision-trees-overarching-aims" style="font-size: 80%;">Decision trees, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs003.html#basics-of-a-tree" style="font-size: 80%;">Basics of a tree</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs004.html#a-sketch-of-a-tree-regression-problem" style="font-size: 80%;">A Sketch of a Tree, Regression problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs005.html#a-sketch-of-a-tree-classification-problem" style="font-size: 80%;">A Sketch of a Tree, Classification problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs006.html#a-typical-decision-tree-with-its-pertinent-jargon-classification-problem" style="font-size: 80%;">A typical Decision Tree with its pertinent Jargon, Classification Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs007.html#general-features" style="font-size: 80%;">General Features</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs008.html#how-do-we-set-it-up" style="font-size: 80%;">How do we set it up?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs009.html#decision-trees-and-regression" style="font-size: 80%;">Decision trees and Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs010.html#building-a-tree-regression" style="font-size: 80%;">Building a tree, regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs011.html#a-top-down-approach-recursive-binary-splitting" style="font-size: 80%;">A top-down approach, recursive binary splitting</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#making-a-tree" style="font-size: 80%;">Making a tree</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs013.html#pruning-the-tree" style="font-size: 80%;">Pruning the tree</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs014.html#cost-complexity-pruning" style="font-size: 80%;">Cost complexity pruning</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs015.html#schematic-regression-procedure" style="font-size: 80%;">Schematic Regression Procedure</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs016.html#a-classification-tree" style="font-size: 80%;">A Classification Tree</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs017.html#growing-a-classification-tree" style="font-size: 80%;">Growing a classification tree</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs018.html#classification-tree-how-to-split-nodes" style="font-size: 80%;">Classification tree, how to split nodes</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs019.html#visualizing-the-tree-classification" style="font-size: 80%;">Visualizing the Tree, Classification</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs020.html#visualizing-the-tree-the-moons" style="font-size: 80%;">Visualizing the Tree, The Moons</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs021.html#other-ways-of-visualizing-the-trees" style="font-size: 80%;">Other ways of visualizing the trees</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs022.html#printing-out-as-text" style="font-size: 80%;">Printing out as text</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs023.html#algorithms-for-setting-up-decision-trees" style="font-size: 80%;">Algorithms for Setting up Decision Trees</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs024.html#the-cart-algorithm-for-classification" style="font-size: 80%;">The CART algorithm for Classification</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs025.html#the-cart-algorithm-for-regression" style="font-size: 80%;">The CART algorithm for Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs026.html#why-binary-splits" style="font-size: 80%;">Why binary splits?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs027.html#computing-a-tree-using-the-gini-index" style="font-size: 80%;">Computing a Tree using the Gini Index</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs028.html#the-table" style="font-size: 80%;">The Table</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs029.html#computing-the-various-gini-indices" style="font-size: 80%;">Computing the various Gini Indices</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs030.html#computing-the-various-gini-indices-hours-slept" style="font-size: 80%;">Computing the various Gini Indices, Hours slept</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs031.html#computing-the-various-gini-indices-hours-studied" style="font-size: 80%;">Computing the various Gini Indices, Hours studied</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs032.html#a-possible-code-using-scikit-learn" style="font-size: 80%;">A possible code using Scikit-Learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs033.html#further-example-computing-the-gini-index" style="font-size: 80%;">Further example: Computing the Gini index</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs034.html#simple-python-code-to-read-in-data-and-perform-classification" style="font-size: 80%;">Simple Python Code to read in Data and perform Classification</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs035.html#computing-the-gini-factor" style="font-size: 80%;">Computing the Gini Factor</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs036.html#regression-trees" style="font-size: 80%;">Regression trees</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs037.html#final-regressor-code" style="font-size: 80%;">Final regressor code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs038.html#pros-and-cons-of-trees-pros" style="font-size: 80%;">Pros and cons of trees, pros</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs039.html#disadvantages" style="font-size: 80%;">Disadvantages</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs040.html#ensemble-methods-from-a-single-tree-to-many-trees-and-extreme-boosting-meet-the-jungle-of-methods" style="font-size: 80%;">Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs041.html#an-overview-of-ensemble-methods" style="font-size: 80%;">An Overview of Ensemble Methods</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs042.html#why-voting" style="font-size: 80%;">Why Voting?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs043.html#tossing-coins" style="font-size: 80%;">Tossing coins</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs044.html#standard-imports-first" style="font-size: 80%;">Standard imports first</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs045.html#simple-voting-example-head-or-tail" style="font-size: 80%;">Simple Voting Example, head or tail</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs046.html#using-the-voting-classifier" style="font-size: 80%;">Using the Voting Classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs047.html#voting-and-bagging" style="font-size: 80%;">Voting and Bagging</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs048.html#bagging" style="font-size: 80%;">Bagging</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs049.html#more-bagging" style="font-size: 80%;">More bagging</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs050.html#making-your-own-bootstrap-changing-the-level-of-the-decision-tree" style="font-size: 80%;">Making your own Bootstrap: Changing the Level of the Decision Tree</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs051.html#random-forests" style="font-size: 80%;">Random forests</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs052.html#random-forest-algorithm" style="font-size: 80%;">Random Forest Algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs053.html#random-forests-compared-with-other-methods-on-the-cancer-data" style="font-size: 80%;">Random Forests Compared with other Methods on the Cancer Data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs054.html#compare-bagging-on-trees-with-random-forests" style="font-size: 80%;">Compare Bagging on Trees with Random Forests</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs055.html#boosting-a-bird-s-eye-view" style="font-size: 80%;">Boosting, a Bird's Eye View</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs056.html#what-is-boosting-additive-modelling-iterative-fitting" style="font-size: 80%;">What is boosting? Additive Modelling/Iterative Fitting</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs057.html#iterative-fitting-regression-and-squared-error-cost-function" style="font-size: 80%;">Iterative Fitting, Regression and Squared-error Cost Function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs058.html#squared-error-example-and-iterative-fitting" style="font-size: 80%;">Squared-Error Example and Iterative Fitting</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs059.html#iterative-fitting-classification-and-adaboost" style="font-size: 80%;">Iterative Fitting, Classification and AdaBoost</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs060.html#adaptive-boosting-adaboost" style="font-size: 80%;">Adaptive Boosting, AdaBoost</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs061.html#building-up-adaboost" style="font-size: 80%;">Building up AdaBoost</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs062.html#adaptive-boosting-adaboost-basic-algorithm" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs063.html#basic-steps-of-adaboost" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs064.html#adaboost-examples" style="font-size: 80%;">AdaBoost Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs004.html#a-typical-decision-tree-with-its-pertinent-jargon-classification-problem" style="font-size: 80%;">A typical Decision Tree with its pertinent Jargon, Classification Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs005.html#general-features" style="font-size: 80%;">General Features</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs006.html#how-do-we-set-it-up" style="font-size: 80%;">How do we set it up?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs007.html#decision-trees-and-regression" style="font-size: 80%;">Decision trees and Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs008.html#building-a-tree-regression" style="font-size: 80%;">Building a tree, regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs009.html#a-top-down-approach-recursive-binary-splitting" style="font-size: 80%;">A top-down approach, recursive binary splitting</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs010.html#making-a-tree" style="font-size: 80%;">Making a tree</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs011.html#pruning-the-tree" style="font-size: 80%;">Pruning the tree</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#cost-complexity-pruning" style="font-size: 80%;">Cost complexity pruning</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs013.html#schematic-regression-procedure" style="font-size: 80%;">Schematic Regression Procedure</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs014.html#a-classification-tree" style="font-size: 80%;">A Classification Tree</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs015.html#growing-a-classification-tree" style="font-size: 80%;">Growing a classification tree</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs016.html#classification-tree-how-to-split-nodes" style="font-size: 80%;">Classification tree, how to split nodes</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs017.html#visualizing-the-tree-classification" style="font-size: 80%;">Visualizing the Tree, Classification</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs018.html#visualizing-the-tree-the-moons" style="font-size: 80%;">Visualizing the Tree, The Moons</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs019.html#other-ways-of-visualizing-the-trees" style="font-size: 80%;">Other ways of visualizing the trees</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs020.html#printing-out-as-text" style="font-size: 80%;">Printing out as text</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs021.html#algorithms-for-setting-up-decision-trees" style="font-size: 80%;">Algorithms for Setting up Decision Trees</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs022.html#the-cart-algorithm-for-classification" style="font-size: 80%;">The CART algorithm for Classification</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs023.html#the-cart-algorithm-for-regression" style="font-size: 80%;">The CART algorithm for Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs024.html#why-binary-splits" style="font-size: 80%;">Why binary splits?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs025.html#computing-a-tree-using-the-gini-index" style="font-size: 80%;">Computing a Tree using the Gini Index</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs026.html#the-table" style="font-size: 80%;">The Table</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs027.html#computing-the-various-gini-indices" style="font-size: 80%;">Computing the various Gini Indices</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs028.html#computing-the-various-gini-indices-hours-slept" style="font-size: 80%;">Computing the various Gini Indices, Hours slept</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs029.html#computing-the-various-gini-indices-hours-studied" style="font-size: 80%;">Computing the various Gini Indices, Hours studied</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs030.html#a-possible-code-using-scikit-learn" style="font-size: 80%;">A possible code using Scikit-Learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs031.html#further-example-computing-the-gini-index" style="font-size: 80%;">Further example: Computing the Gini index</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs032.html#simple-python-code-to-read-in-data-and-perform-classification" style="font-size: 80%;">Simple Python Code to read in Data and perform Classification</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs033.html#computing-the-gini-factor" style="font-size: 80%;">Computing the Gini Factor</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs034.html#regression-trees" style="font-size: 80%;">Regression trees</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs035.html#final-regressor-code" style="font-size: 80%;">Final regressor code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs036.html#pros-and-cons-of-trees-pros" style="font-size: 80%;">Pros and cons of trees, pros</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs037.html#disadvantages" style="font-size: 80%;">Disadvantages</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs038.html#ensemble-methods-from-a-single-tree-to-many-trees-and-extreme-boosting-meet-the-jungle-of-methods" style="font-size: 80%;">Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs039.html#an-overview-of-ensemble-methods" style="font-size: 80%;">An Overview of Ensemble Methods</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs040.html#why-voting" style="font-size: 80%;">Why Voting?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs041.html#tossing-coins" style="font-size: 80%;">Tossing coins</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs042.html#standard-imports-first" style="font-size: 80%;">Standard imports first</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs043.html#simple-voting-example-head-or-tail" style="font-size: 80%;">Simple Voting Example, head or tail</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs044.html#using-the-voting-classifier" style="font-size: 80%;">Using the Voting Classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs045.html#voting-and-bagging" style="font-size: 80%;">Voting and Bagging</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs046.html#bagging" style="font-size: 80%;">Bagging</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs047.html#more-bagging" style="font-size: 80%;">More bagging</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs048.html#making-your-own-bootstrap-changing-the-level-of-the-decision-tree" style="font-size: 80%;">Making your own Bootstrap: Changing the Level of the Decision Tree</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs049.html#random-forests" style="font-size: 80%;">Random forests</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs050.html#random-forest-algorithm" style="font-size: 80%;">Random Forest Algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs051.html#random-forests-compared-with-other-methods-on-the-cancer-data" style="font-size: 80%;">Random Forests Compared with other Methods on the Cancer Data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs052.html#compare-bagging-on-trees-with-random-forests" style="font-size: 80%;">Compare Bagging on Trees with Random Forests</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs053.html#boosting-a-bird-s-eye-view" style="font-size: 80%;">Boosting, a Bird's Eye View</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs054.html#what-is-boosting-additive-modelling-iterative-fitting" style="font-size: 80%;">What is boosting? Additive Modelling/Iterative Fitting</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs055.html#iterative-fitting-regression-and-squared-error-cost-function" style="font-size: 80%;">Iterative Fitting, Regression and Squared-error Cost Function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs056.html#squared-error-example-and-iterative-fitting" style="font-size: 80%;">Squared-Error Example and Iterative Fitting</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs057.html#iterative-fitting-classification-and-adaboost" style="font-size: 80%;">Iterative Fitting, Classification and AdaBoost</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs058.html#adaptive-boosting-adaboost" style="font-size: 80%;">Adaptive Boosting, AdaBoost</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs059.html#building-up-adaboost" style="font-size: 80%;">Building up AdaBoost</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs060.html#adaptive-boosting-adaboost-basic-algorithm" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs061.html#basic-steps-of-adaboost" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week46-bs062.html#adaboost-examples" style="font-size: 80%;">AdaBoost Examples</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -329,52 +319,34 @@ MathJax.Hub.Config({
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
<a name="part0012"></a>
|
||||
<!-- !split -->
|
||||
<h2 id="making-a-tree" class="anchor">Making a tree </h2>
|
||||
<h2 id="cost-complexity-pruning" class="anchor">Cost complexity pruning </h2>
|
||||
|
||||
<p>In order to implement the recursive binary splitting we start by selecting
|
||||
the predictor \( x_j \) and a cutpoint \( s \) that splits the predictor space into two regions \( R_1 \) and \( R_2 \)
|
||||
</p>
|
||||
<p>For each value of \( \alpha \) there corresponds a subtree \( T \in T_0 \) such that</p>
|
||||
$$
|
||||
\left\{X\vert x_j < s\right\},
|
||||
\sum_{m=1}^{\overline{T}}\sum_{i:x_i\in R_m}(y_i-\overline{y}_{R_m})^2+\alpha\overline{T},
|
||||
$$
|
||||
|
||||
<p>and</p>
|
||||
$$
|
||||
\left\{X\vert x_j \geq s\right\},
|
||||
$$
|
||||
|
||||
<p>so that we obtain the lowest MSE, that is</p>
|
||||
$$
|
||||
\sum_{i:x_i\in R_j}(y_i-\overline{y}_{R_1})^2+\sum_{i:x_i\in R_2}(y_i-\overline{y}_{R_2})^2,
|
||||
$$
|
||||
|
||||
<p>which we want to minimize by considering all predictors
|
||||
\( x_1,x_2,\dots,x_p \). We consider also all possible values of \( s \) for
|
||||
each predictor. These values could be determined by randomly assigned
|
||||
numbers or by starting at the midpoint and then proceed till we find
|
||||
an optimal value.
|
||||
<p>is as small as possible. Here \( \overline{T} \) is
|
||||
the number of terminal nodes of the tree \( T \) , \( R_m \) is the
|
||||
rectangle (i.e. the subset of predictor space) corresponding to the \( m \)-th terminal node.
|
||||
</p>
|
||||
|
||||
<p>For any \( j \) and \( s \), we define the pair of half-planes where
|
||||
\( \overline{y}_{R_1} \) is the mean response for the training
|
||||
observations in \( R_1(j,s) \), and \( \overline{y}_{R_2} \) is the mean
|
||||
response for the training observations in \( R_2(j,s) \).
|
||||
<p>The tuning parameter \( \alpha \) controls a trade-off between the subtree’s
|
||||
complexity and its fit to the training data. When \( \alpha = 0 \), then the
|
||||
subtree \( T \) will simply equal \( T_0 \),
|
||||
because then the above equation just measures the
|
||||
training error.
|
||||
However, as \( \alpha \) increases, there is a price to pay for
|
||||
having a tree with many terminal nodes. The above equation will
|
||||
tend to be minimized for a smaller subtree.
|
||||
</p>
|
||||
|
||||
<p>Finding the values of \( j \) and \( s \) that minimize the above equation can be
|
||||
done quite quickly, especially when the number of features \( p \) is not
|
||||
too large.
|
||||
</p>
|
||||
|
||||
<p>Next, we repeat the process, looking
|
||||
for the best predictor and best cutpoint in order to split the data
|
||||
further so as to minimize the MSE within each of the resulting
|
||||
regions. However, this time, instead of splitting the entire predictor
|
||||
space, we split one of the two previously identified regions. We now
|
||||
have three regions. Again, we look to split one of these three regions
|
||||
further, so as to minimize the MSE. The process continues until a
|
||||
stopping criterion is reached; for instance, we may continue until no
|
||||
region contains more than five observations.
|
||||
<p>It turns out that as we increase \( \alpha \) from zero
|
||||
branches get pruned from the tree in a nested and predictable fashion,
|
||||
so obtaining the whole sequence of subtrees as a function of \( \alpha \) is
|
||||
easy. We can select a value of \( \alpha \) using a validation set or using
|
||||
cross-validation. We then return to the full data set and obtain the
|
||||
subtree corresponding to \( \alpha \).
|
||||
</p>
|
||||
|
||||
<p>
|
||||
@@ -402,7 +374,7 @@ region contains more than five observations.
|
||||
<li><a href="._week46-bs020.html">21</a></li>
|
||||
<li><a href="._week46-bs021.html">22</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week46-bs064.html">65</a></li>
|
||||
<li><a href="._week46-bs062.html">63</a></li>
|
||||
<li><a href="._week46-bs013.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
Reference in New Issue
Block a user