lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-11-16 06:33:54 +01:00
parent 73e82cd8dd
commit 2c1f27b519
38 changed files with 2623 additions and 3020 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week46/html/._week46-bs024.html
+65 -92
View File
@@ -42,14 +42,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
None,
'decision-trees-overarching-aims'),
('Basics of a tree', 2, None, 'basics-of-a-tree'),
('A Sketch of a Tree, Regression problem',
2,
None,
'a-sketch-of-a-tree-regression-problem'),
('A Sketch of a Tree, Classification problem',
2,
None,
'a-sketch-of-a-tree-classification-problem'),
('A typical Decision Tree with its pertinent Jargon, '
'Classification Problem',
2,
@@ -257,67 +249,65 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week46-bs001.html#plan-for-week-46" style="font-size: 80%;">Plan for week 46</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs002.html#decision-trees-overarching-aims" style="font-size: 80%;">Decision trees, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs003.html#basics-of-a-tree" style="font-size: 80%;">Basics of a tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs004.html#a-sketch-of-a-tree-regression-problem" style="font-size: 80%;">A Sketch of a Tree, Regression problem</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs005.html#a-sketch-of-a-tree-classification-problem" style="font-size: 80%;">A Sketch of a Tree, Classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs006.html#a-typical-decision-tree-with-its-pertinent-jargon-classification-problem" style="font-size: 80%;">A typical Decision Tree with its pertinent Jargon, Classification Problem</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs007.html#general-features" style="font-size: 80%;">General Features</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs008.html#how-do-we-set-it-up" style="font-size: 80%;">How do we set it up?</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs009.html#decision-trees-and-regression" style="font-size: 80%;">Decision trees and Regression</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs010.html#building-a-tree-regression" style="font-size: 80%;">Building a tree, regression</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs011.html#a-top-down-approach-recursive-binary-splitting" style="font-size: 80%;">A top-down approach, recursive binary splitting</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs012.html#making-a-tree" style="font-size: 80%;">Making a tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs013.html#pruning-the-tree" style="font-size: 80%;">Pruning the tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs014.html#cost-complexity-pruning" style="font-size: 80%;">Cost complexity pruning</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs015.html#schematic-regression-procedure" style="font-size: 80%;">Schematic Regression Procedure</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs016.html#a-classification-tree" style="font-size: 80%;">A Classification Tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs017.html#growing-a-classification-tree" style="font-size: 80%;">Growing a classification tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs018.html#classification-tree-how-to-split-nodes" style="font-size: 80%;">Classification tree, how to split nodes</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs019.html#visualizing-the-tree-classification" style="font-size: 80%;">Visualizing the Tree, Classification</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs020.html#visualizing-the-tree-the-moons" style="font-size: 80%;">Visualizing the Tree, The Moons</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs021.html#other-ways-of-visualizing-the-trees" style="font-size: 80%;">Other ways of visualizing the trees</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs022.html#printing-out-as-text" style="font-size: 80%;">Printing out as text</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs023.html#algorithms-for-setting-up-decision-trees" style="font-size: 80%;">Algorithms for Setting up Decision Trees</a></li>
<!-- navigation toc: --> <li><a href="#the-cart-algorithm-for-classification" style="font-size: 80%;">The CART algorithm for Classification</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs025.html#the-cart-algorithm-for-regression" style="font-size: 80%;">The CART algorithm for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs026.html#why-binary-splits" style="font-size: 80%;">Why binary splits?</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs027.html#computing-a-tree-using-the-gini-index" style="font-size: 80%;">Computing a Tree using the Gini Index</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs028.html#the-table" style="font-size: 80%;">The Table</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs029.html#computing-the-various-gini-indices" style="font-size: 80%;">Computing the various Gini Indices</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs030.html#computing-the-various-gini-indices-hours-slept" style="font-size: 80%;">Computing the various Gini Indices, Hours slept</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs031.html#computing-the-various-gini-indices-hours-studied" style="font-size: 80%;">Computing the various Gini Indices, Hours studied</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs032.html#a-possible-code-using-scikit-learn" style="font-size: 80%;">A possible code using Scikit-Learn</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs033.html#further-example-computing-the-gini-index" style="font-size: 80%;">Further example: Computing the Gini index</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs034.html#simple-python-code-to-read-in-data-and-perform-classification" style="font-size: 80%;">Simple Python Code to read in Data and perform Classification</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs035.html#computing-the-gini-factor" style="font-size: 80%;">Computing the Gini Factor</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs036.html#regression-trees" style="font-size: 80%;">Regression trees</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs037.html#final-regressor-code" style="font-size: 80%;">Final regressor code</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs038.html#pros-and-cons-of-trees-pros" style="font-size: 80%;">Pros and cons of trees, pros</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs039.html#disadvantages" style="font-size: 80%;">Disadvantages</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs040.html#ensemble-methods-from-a-single-tree-to-many-trees-and-extreme-boosting-meet-the-jungle-of-methods" style="font-size: 80%;">Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs041.html#an-overview-of-ensemble-methods" style="font-size: 80%;">An Overview of Ensemble Methods</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs042.html#why-voting" style="font-size: 80%;">Why Voting?</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs043.html#tossing-coins" style="font-size: 80%;">Tossing coins</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs044.html#standard-imports-first" style="font-size: 80%;">Standard imports first</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs045.html#simple-voting-example-head-or-tail" style="font-size: 80%;">Simple Voting Example, head or tail</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs046.html#using-the-voting-classifier" style="font-size: 80%;">Using the Voting Classifier</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs047.html#voting-and-bagging" style="font-size: 80%;">Voting and Bagging</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs048.html#bagging" style="font-size: 80%;">Bagging</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs049.html#more-bagging" style="font-size: 80%;">More bagging</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs050.html#making-your-own-bootstrap-changing-the-level-of-the-decision-tree" style="font-size: 80%;">Making your own Bootstrap: Changing the Level of the Decision Tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs051.html#random-forests" style="font-size: 80%;">Random forests</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs052.html#random-forest-algorithm" style="font-size: 80%;">Random Forest Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs053.html#random-forests-compared-with-other-methods-on-the-cancer-data" style="font-size: 80%;">Random Forests Compared with other Methods on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs054.html#compare-bagging-on-trees-with-random-forests" style="font-size: 80%;">Compare Bagging on Trees with Random Forests</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs055.html#boosting-a-bird-s-eye-view" style="font-size: 80%;">Boosting, a Bird's Eye View</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs056.html#what-is-boosting-additive-modelling-iterative-fitting" style="font-size: 80%;">What is boosting? Additive Modelling/Iterative Fitting</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs057.html#iterative-fitting-regression-and-squared-error-cost-function" style="font-size: 80%;">Iterative Fitting, Regression and Squared-error Cost Function</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs058.html#squared-error-example-and-iterative-fitting" style="font-size: 80%;">Squared-Error Example and Iterative Fitting</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs059.html#iterative-fitting-classification-and-adaboost" style="font-size: 80%;">Iterative Fitting, Classification and AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs060.html#adaptive-boosting-adaboost" style="font-size: 80%;">Adaptive Boosting, AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs061.html#building-up-adaboost" style="font-size: 80%;">Building up AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs062.html#adaptive-boosting-adaboost-basic-algorithm" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs063.html#basic-steps-of-adaboost" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs064.html#adaboost-examples" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs004.html#a-typical-decision-tree-with-its-pertinent-jargon-classification-problem" style="font-size: 80%;">A typical Decision Tree with its pertinent Jargon, Classification Problem</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs005.html#general-features" style="font-size: 80%;">General Features</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs006.html#how-do-we-set-it-up" style="font-size: 80%;">How do we set it up?</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs007.html#decision-trees-and-regression" style="font-size: 80%;">Decision trees and Regression</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs008.html#building-a-tree-regression" style="font-size: 80%;">Building a tree, regression</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs009.html#a-top-down-approach-recursive-binary-splitting" style="font-size: 80%;">A top-down approach, recursive binary splitting</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs010.html#making-a-tree" style="font-size: 80%;">Making a tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs011.html#pruning-the-tree" style="font-size: 80%;">Pruning the tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs012.html#cost-complexity-pruning" style="font-size: 80%;">Cost complexity pruning</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs013.html#schematic-regression-procedure" style="font-size: 80%;">Schematic Regression Procedure</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs014.html#a-classification-tree" style="font-size: 80%;">A Classification Tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs015.html#growing-a-classification-tree" style="font-size: 80%;">Growing a classification tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs016.html#classification-tree-how-to-split-nodes" style="font-size: 80%;">Classification tree, how to split nodes</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs017.html#visualizing-the-tree-classification" style="font-size: 80%;">Visualizing the Tree, Classification</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs018.html#visualizing-the-tree-the-moons" style="font-size: 80%;">Visualizing the Tree, The Moons</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs019.html#other-ways-of-visualizing-the-trees" style="font-size: 80%;">Other ways of visualizing the trees</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs020.html#printing-out-as-text" style="font-size: 80%;">Printing out as text</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs021.html#algorithms-for-setting-up-decision-trees" style="font-size: 80%;">Algorithms for Setting up Decision Trees</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs022.html#the-cart-algorithm-for-classification" style="font-size: 80%;">The CART algorithm for Classification</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs023.html#the-cart-algorithm-for-regression" style="font-size: 80%;">The CART algorithm for Regression</a></li>
<!-- navigation toc: --> <li><a href="#why-binary-splits" style="font-size: 80%;">Why binary splits?</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs025.html#computing-a-tree-using-the-gini-index" style="font-size: 80%;">Computing a Tree using the Gini Index</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs026.html#the-table" style="font-size: 80%;">The Table</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs027.html#computing-the-various-gini-indices" style="font-size: 80%;">Computing the various Gini Indices</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs028.html#computing-the-various-gini-indices-hours-slept" style="font-size: 80%;">Computing the various Gini Indices, Hours slept</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs029.html#computing-the-various-gini-indices-hours-studied" style="font-size: 80%;">Computing the various Gini Indices, Hours studied</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs030.html#a-possible-code-using-scikit-learn" style="font-size: 80%;">A possible code using Scikit-Learn</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs031.html#further-example-computing-the-gini-index" style="font-size: 80%;">Further example: Computing the Gini index</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs032.html#simple-python-code-to-read-in-data-and-perform-classification" style="font-size: 80%;">Simple Python Code to read in Data and perform Classification</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs033.html#computing-the-gini-factor" style="font-size: 80%;">Computing the Gini Factor</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs034.html#regression-trees" style="font-size: 80%;">Regression trees</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs035.html#final-regressor-code" style="font-size: 80%;">Final regressor code</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs036.html#pros-and-cons-of-trees-pros" style="font-size: 80%;">Pros and cons of trees, pros</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs037.html#disadvantages" style="font-size: 80%;">Disadvantages</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs038.html#ensemble-methods-from-a-single-tree-to-many-trees-and-extreme-boosting-meet-the-jungle-of-methods" style="font-size: 80%;">Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs039.html#an-overview-of-ensemble-methods" style="font-size: 80%;">An Overview of Ensemble Methods</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs040.html#why-voting" style="font-size: 80%;">Why Voting?</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs041.html#tossing-coins" style="font-size: 80%;">Tossing coins</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs042.html#standard-imports-first" style="font-size: 80%;">Standard imports first</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs043.html#simple-voting-example-head-or-tail" style="font-size: 80%;">Simple Voting Example, head or tail</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs044.html#using-the-voting-classifier" style="font-size: 80%;">Using the Voting Classifier</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs045.html#voting-and-bagging" style="font-size: 80%;">Voting and Bagging</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs046.html#bagging" style="font-size: 80%;">Bagging</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs047.html#more-bagging" style="font-size: 80%;">More bagging</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs048.html#making-your-own-bootstrap-changing-the-level-of-the-decision-tree" style="font-size: 80%;">Making your own Bootstrap: Changing the Level of the Decision Tree</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs049.html#random-forests" style="font-size: 80%;">Random forests</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs050.html#random-forest-algorithm" style="font-size: 80%;">Random Forest Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs051.html#random-forests-compared-with-other-methods-on-the-cancer-data" style="font-size: 80%;">Random Forests Compared with other Methods on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs052.html#compare-bagging-on-trees-with-random-forests" style="font-size: 80%;">Compare Bagging on Trees with Random Forests</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs053.html#boosting-a-bird-s-eye-view" style="font-size: 80%;">Boosting, a Bird's Eye View</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs054.html#what-is-boosting-additive-modelling-iterative-fitting" style="font-size: 80%;">What is boosting? Additive Modelling/Iterative Fitting</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs055.html#iterative-fitting-regression-and-squared-error-cost-function" style="font-size: 80%;">Iterative Fitting, Regression and Squared-error Cost Function</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs056.html#squared-error-example-and-iterative-fitting" style="font-size: 80%;">Squared-Error Example and Iterative Fitting</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs057.html#iterative-fitting-classification-and-adaboost" style="font-size: 80%;">Iterative Fitting, Classification and AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs058.html#adaptive-boosting-adaboost" style="font-size: 80%;">Adaptive Boosting, AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs059.html#building-up-adaboost" style="font-size: 80%;">Building up AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs060.html#adaptive-boosting-adaboost-basic-algorithm" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs061.html#basic-steps-of-adaboost" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs062.html#adaboost-examples" style="font-size: 80%;">AdaBoost Examples</a></li>
</ul>
</li>
@@ -329,29 +319,12 @@ MathJax.Hub.Config({
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0024"></a>
<!-- !split -->
<h2 id="the-cart-algorithm-for-classification" class="anchor">The CART algorithm for Classification </h2>
<h2 id="why-binary-splits" class="anchor">Why binary splits? </h2>
<p>For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \).
This could be for example a threshold set by a number below a certain circumference of a malign tumor.
</p>
<p>How do we find these two quantities?
We search for the pair \( (k,t_k) \) that produces the purest subset using for example the <b>gini</b> factor \( G \).
The cost function it tries to minimize is then
</p>
$$
C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}},
$$
<p>where \( G_{\mathrm{left/right}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \)
is the number of instances in the left/right subset
</p>
<p>Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets
and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the
\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other
hyperparameters control additional stopping conditions such as the \( min\_samples\_split \),
\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \).
<p>It is custom to split to a tree uising binary splits. The reason is
that multiway splits fragment the data too quickly, leaving
insufficient data at the next level down. Multiway splits can be
achieved by a series of binary split and this is normally preferred.
</p>
<p>
@@ -379,7 +352,7 @@ hyperparameters control additional stopping conditions such as the \( min\_sampl
<li><a href="._week46-bs032.html">33</a></li>
<li><a href="._week46-bs033.html">34</a></li>
<li><a href="">...</a></li>
<li><a href="._week46-bs064.html">65</a></li>
<li><a href="._week46-bs062.html">63</a></li>
<li><a href="._week46-bs025.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->