lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

new update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-13 14:05:16 +02:00
parent 30d664a82b
commit ad92592adb
76 changed files with 5646 additions and 2878 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter6.html
+52 -52
View File
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>8. Decision trees, overarching aims &#8212; Applied Data Analysis and Machine Learning</title>
<title>9. Decision trees, overarching aims &#8212; Applied Data Analysis and Machine Learning</title>
<link href="_static/css/theme.css" rel="stylesheet" />
<link href="_static/css/index.c5995385ac14fb8791e8eb36b4908be2.css" rel="stylesheet" />
@@ -54,8 +54,8 @@
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="9. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods" href="chapter7.html" />
<link rel="prev" title="7. Support Vector Machines, overarching aims" href="chapter5.html" />
<link rel="next" title="10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods" href="chapter7.html" />
<link rel="prev" title="8. Support Vector Machines, overarching aims" href="chapter5.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en" />
@@ -139,17 +139,17 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter1.html">
3. Linear Regression, basic Elements
3. Linear Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html">
4. Resampling Methods
4. Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
5. Ridge and Lasso Regression
5. Resampling Methods
</a>
</li>
<li class="toctree-l1">
@@ -159,7 +159,7 @@
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
7. Support Vector Machines, overarching aims
8. Support Vector Machines, overarching aims
</a>
</li>
</ul>
@@ -171,12 +171,12 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
8. Decision trees, overarching aims
9. Decision trees, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
9. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
</a>
</li>
</ul>
@@ -188,12 +188,12 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
10. Basic ideas of the Principal Component Analysis (PCA)
11. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="Clustering.html">
11. Clustering Analysis
12. Clustering Analysis
</a>
</li>
</ul>
@@ -205,12 +205,12 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter9.html">
12. Neural networks
13. Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter10.html">
13. Building a Feed Forward Neural Network
14. Building a Feed Forward Neural Network
</a>
</li>
</ul>
@@ -286,100 +286,100 @@
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basics-of-a-tree">
8.1. Basics of a tree
9.1. Basics of a tree
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#general-features">
8.2. General Features
9.2. General Features
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#building-a-tree-regression">
8.3. Building a tree, regression
9.3. Building a tree, regression
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#making-a-tree">
8.3.1. Making a tree
9.3.1. Making a tree
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#schematic-regression-procedure">
8.3.2. Schematic Regression Procedure
9.3.2. Schematic Regression Procedure
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-classification-tree">
8.4. A Classification Tree
9.4. A Classification Tree
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#visualizing-the-tree-classification">
8.4.1. Visualizing the Tree, Classification
9.4.1. Visualizing the Tree, Classification
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-ways-of-visualizing-the-trees">
8.4.2. Other ways of visualizing the trees
9.4.2. Other ways of visualizing the trees
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#algorithms-for-setting-up-decision-trees">
8.5. Algorithms for Setting up Decision Trees
9.5. Algorithms for Setting up Decision Trees
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-cart-algorithm-for-classification">
8.5.1. The CART algorithm for Classification
9.5.1. The CART algorithm for Classification
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-cart-algorithm-for-regression">
8.5.2. The CART algorithm for Regression
9.5.2. The CART algorithm for Regression
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#computing-the-gini-index">
8.5.3. Computing the Gini index
9.5.3. Computing the Gini index
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#simple-python-code-to-read-in-data-and-perform-classification">
8.5.4. Simple Python Code to read in Data and perform Classification
9.5.4. Simple Python Code to read in Data and perform Classification
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#entropy-and-the-id3-algorithm">
8.6. Entropy and the ID3 algorithm
9.6. Entropy and the ID3 algorithm
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#cancer-data-again-now-with-decision-trees-and-other-methods">
8.6.1. Cancer Data again now with Decision Trees and other Methods
9.6.1. Cancer Data again now with Decision Trees and other Methods
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#another-example-the-moons-again">
8.6.2. Another example, the moons again
9.6.2. Another example, the moons again
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#pros-and-cons-of-trees-pros">
8.7. Pros and cons of trees, pros
9.7. Pros and cons of trees, pros
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#disadvantages">
8.7.1. Disadvantages
9.7.1. Disadvantages
</a>
</li>
</ul>
@@ -396,7 +396,7 @@
<div>
<div class="section" id="decision-trees-overarching-aims">
<h1><span class="section-number">8. </span>Decision trees, overarching aims<a class="headerlink" href="#decision-trees-overarching-aims" title="Permalink to this headline"></a></h1>
<h1><span class="section-number">9. </span>Decision trees, overarching aims<a class="headerlink" href="#decision-trees-overarching-aims" title="Permalink to this headline"></a></h1>
<p>We start here with the most basic algorithm, the so-called decision
tree. With this basic algorithm we can in turn build more complex
networks, spanning from homogeneous and heterogenous forests (bagging,
@@ -415,7 +415,7 @@ to be the most informative ones. The process of finding the <strong>most
informative</strong> feature is done until we accomplish a stopping criteria
where we then finally end up in so called <strong>leaf nodes</strong>.</p>
<div class="section" id="basics-of-a-tree">
<h2><span class="section-number">8.1. </span>Basics of a tree<a class="headerlink" href="#basics-of-a-tree" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">9.1. </span>Basics of a tree<a class="headerlink" href="#basics-of-a-tree" title="Permalink to this headline"></a></h2>
<p>A decision tree is typically divided into a <strong>root node</strong>, the <strong>interior nodes</strong>,
and the final <strong>leaf nodes</strong> or just <strong>leaves</strong>. These entities are then connected by so-called <strong>branches</strong>.</p>
<p>The leaf nodes
@@ -426,7 +426,7 @@ given some assumptions, make predictions about the target feature value
(class) of unseen query instances.</p>
</div>
<div class="section" id="general-features">
<h2><span class="section-number">8.2. </span>General Features<a class="headerlink" href="#general-features" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">9.2. </span>General Features<a class="headerlink" href="#general-features" title="Permalink to this headline"></a></h2>
<p>The overarching approach to decision trees is a top-down approach.</p>
<ul class="simple">
<li><p>A leaf provides the classification of a given instance.</p></li>
@@ -553,7 +553,7 @@ second power: 0.00015921913736794912
</div>
</div>
<div class="section" id="building-a-tree-regression">
<h2><span class="section-number">8.3. </span>Building a tree, regression<a class="headerlink" href="#building-a-tree-regression" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">9.3. </span>Building a tree, regression<a class="headerlink" href="#building-a-tree-regression" title="Permalink to this headline"></a></h2>
<p>There are mainly two steps</p>
<ol class="simple">
<li><p>We split the predictor space (the set of possible values <span class="math notranslate nohighlight">\(x_1,x_2,\dots, x_p\)</span>) into <span class="math notranslate nohighlight">\(J\)</span> distinct and non-non-overlapping regions, <span class="math notranslate nohighlight">\(R_1,R_2,\dots,R_J\)</span>.</p></li>
@@ -582,7 +582,7 @@ tree-building process, the best split is made at that particular step,
rather than looking ahead and picking a split that will lead to a
better tree in some future step.</p>
<div class="section" id="making-a-tree">
<h3><span class="section-number">8.3.1. </span>Making a tree<a class="headerlink" href="#making-a-tree" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">9.3.1. </span>Making a tree<a class="headerlink" href="#making-a-tree" title="Permalink to this headline"></a></h3>
<p>In order to implement the recursive binary splitting we start by selecting
the predictor <span class="math notranslate nohighlight">\(x_j\)</span> and a cutpoint <span class="math notranslate nohighlight">\(s\)</span> that splits the predictor space into two regions <span class="math notranslate nohighlight">\(R_1\)</span> and <span class="math notranslate nohighlight">\(R_2\)</span></p>
<div class="math notranslate nohighlight">
@@ -655,7 +655,7 @@ cross-validation. We then return to the full data set and obtain the
subtree corresponding to <span class="math notranslate nohighlight">\(\alpha\)</span>.</p>
</div>
<div class="section" id="schematic-regression-procedure">
<h3><span class="section-number">8.3.2. </span>Schematic Regression Procedure<a class="headerlink" href="#schematic-regression-procedure" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">9.3.2. </span>Schematic Regression Procedure<a class="headerlink" href="#schematic-regression-procedure" title="Permalink to this headline"></a></h3>
<p>Building a Regression Tree</p>
<ol class="simple">
<li><p>Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.</p></li>
@@ -674,7 +674,7 @@ subtree corresponding to <span class="math notranslate nohighlight">\(\alpha\)</
</div>
</div>
<div class="section" id="a-classification-tree">
<h2><span class="section-number">8.4. </span>A Classification Tree<a class="headerlink" href="#a-classification-tree" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">9.4. </span>A Classification Tree<a class="headerlink" href="#a-classification-tree" title="Permalink to this headline"></a></h2>
<p>A classification tree is very similar to a regression tree, except
that it is used to predict a qualitative response rather than a
quantitative one. Recall that for a regression tree, the predicted
@@ -737,7 +737,7 @@ g = \sum_{k=1}^K p_{mk}(1-p_{mk}).
s = -\sum_{k=1}^K p_{mk}\log{p_{mk}}.
\]</div>
<div class="section" id="visualizing-the-tree-classification">
<h3><span class="section-number">8.4.1. </span>Visualizing the Tree, Classification<a class="headerlink" href="#visualizing-the-tree-classification" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">9.4.1. </span>Visualizing the Tree, Classification<a class="headerlink" href="#visualizing-the-tree-classification" title="Permalink to this headline"></a></h3>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
@@ -823,7 +823,7 @@ s = -\sum_{k=1}^K p_{mk}\log{p_{mk}}.
</div>
</div>
<div class="section" id="other-ways-of-visualizing-the-trees">
<h3><span class="section-number">8.4.2. </span>Other ways of visualizing the trees<a class="headerlink" href="#other-ways-of-visualizing-the-trees" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">9.4.2. </span>Other ways of visualizing the trees<a class="headerlink" href="#other-ways-of-visualizing-the-trees" title="Permalink to this headline"></a></h3>
<p><strong>Scikit-Learn</strong> has also another way to visualize the trees which is very useful, here with the Iris data.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -857,7 +857,7 @@ This method doesn’t require the installation of external libraries and is more
</div>
</div>
<div class="section" id="algorithms-for-setting-up-decision-trees">
<h2><span class="section-number">8.5. </span>Algorithms for Setting up Decision Trees<a class="headerlink" href="#algorithms-for-setting-up-decision-trees" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">9.5. </span>Algorithms for Setting up Decision Trees<a class="headerlink" href="#algorithms-for-setting-up-decision-trees" title="Permalink to this headline"></a></h2>
<p>Two algorithms stand out in the set up of decision trees:</p>
<ol class="simple">
<li><p>The CART (Classification And Regression Tree) algorithm for both classification and regression</p></li>
@@ -868,7 +868,7 @@ This method doesn’t require the installation of external libraries and is more
you can use either the <strong>gini</strong> index or the <strong>entropy</strong> to split a tree
in two branches.</p>
<div class="section" id="the-cart-algorithm-for-classification">
<h3><span class="section-number">8.5.1. </span>The CART algorithm for Classification<a class="headerlink" href="#the-cart-algorithm-for-classification" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">9.5.1. </span>The CART algorithm for Classification<a class="headerlink" href="#the-cart-algorithm-for-classification" title="Permalink to this headline"></a></h3>
<p>For classification, the CART algorithm splits the data set in two subsets using a single feature <span class="math notranslate nohighlight">\(k\)</span> and a threshold <span class="math notranslate nohighlight">\(t_k\)</span>.
This could be for example a threshold set by a number below a certain circumference of a malign tumor.</p>
<p>How do we find these two quantities?
@@ -887,7 +887,7 @@ hyperparameters control additional stopping conditions such as the <span class="
<span class="math notranslate nohighlight">\(min\_samples\_leaf\)</span>, <span class="math notranslate nohighlight">\(min\_weight\_fraction\_leaf\)</span>, and <span class="math notranslate nohighlight">\(max\_leaf\_nodes\)</span>.</p>
</div>
<div class="section" id="the-cart-algorithm-for-regression">
<h3><span class="section-number">8.5.2. </span>The CART algorithm for Regression<a class="headerlink" href="#the-cart-algorithm-for-regression" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">9.5.2. </span>The CART algorithm for Regression<a class="headerlink" href="#the-cart-algorithm-for-regression" title="Permalink to this headline"></a></h3>
<p>The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the
training set in a way that minimizes say the <strong>gini</strong> or <strong>entropy</strong> impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now</p>
<div class="math notranslate nohighlight">
@@ -909,7 +909,7 @@ C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\ma
just like for classification tasks, is prone to overfitting.</p>
</div>
<div class="section" id="computing-the-gini-index">
<h3><span class="section-number">8.5.3. </span>Computing the Gini index<a class="headerlink" href="#computing-the-gini-index" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">9.5.3. </span>Computing the Gini index<a class="headerlink" href="#computing-the-gini-index" title="Permalink to this headline"></a></h3>
<p>The example we will look at is a classical one in many Machine
Learning applications. Based on various meteorological features, we
have several so-called attributes which decide whether we at the end
@@ -944,7 +944,7 @@ humidity and weak and strong for wind.</p>
</table>
</div>
<div class="section" id="simple-python-code-to-read-in-data-and-perform-classification">
<h3><span class="section-number">8.5.4. </span>Simple Python Code to read in Data and perform Classification<a class="headerlink" href="#simple-python-code-to-read-in-data-and-perform-classification" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">9.5.4. </span>Simple Python Code to read in Data and perform Classification<a class="headerlink" href="#simple-python-code-to-read-in-data-and-perform-classification" title="Permalink to this headline"></a></h3>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Common imports</span>
@@ -1092,7 +1092,7 @@ algorithm ID3.</p>
</div>
</div>
<div class="section" id="entropy-and-the-id3-algorithm">
<h2><span class="section-number">8.6. </span>Entropy and the ID3 algorithm<a class="headerlink" href="#entropy-and-the-id3-algorithm" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">9.6. </span>Entropy and the ID3 algorithm<a class="headerlink" href="#entropy-and-the-id3-algorithm" title="Permalink to this headline"></a></h2>
<p>The ID3 algorithm learns decision trees by constructing
them in a top down way, beginning with the question <strong>which attribute should be tested at the root of the tree</strong>?</p>
<ol class="simple">
@@ -1113,7 +1113,7 @@ training examples according to their target classification.</p>
<p>The ID3 algorithm uses this information gain measure to select among the candidate
attributes at each step while growing the tree.</p>
<div class="section" id="cancer-data-again-now-with-decision-trees-and-other-methods">
<h3><span class="section-number">8.6.1. </span>Cancer Data again now with Decision Trees and other Methods<a class="headerlink" href="#cancer-data-again-now-with-decision-trees-and-other-methods" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">9.6.1. </span>Cancer Data again now with Decision Trees and other Methods<a class="headerlink" href="#cancer-data-again-now-with-decision-trees-and-other-methods" title="Permalink to this headline"></a></h3>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
@@ -1163,7 +1163,7 @@ attributes at each step while growing the tree.</p>
</div>
</div>
<div class="section" id="another-example-the-moons-again">
<h3><span class="section-number">8.6.2. </span>Another example, the moons again<a class="headerlink" href="#another-example-the-moons-again" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">9.6.2. </span>Another example, the moons again<a class="headerlink" href="#another-example-the-moons-again" title="Permalink to this headline"></a></h3>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">division</span><span class="p">,</span> <span class="n">print_function</span><span class="p">,</span> <span class="n">unicode_literals</span>
@@ -1363,7 +1363,7 @@ attributes at each step while growing the tree.</p>
</div>
</div>
<div class="section" id="pros-and-cons-of-trees-pros">
<h2><span class="section-number">8.7. </span>Pros and cons of trees, pros<a class="headerlink" href="#pros-and-cons-of-trees-pros" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">9.7. </span>Pros and cons of trees, pros<a class="headerlink" href="#pros-and-cons-of-trees-pros" title="Permalink to this headline"></a></h2>
<ul class="simple">
<li><p>White box, easy to interpret model. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches discussed earlier (think of support vector machines)</p></li>
<li><p>Trees are very easy to explain to people. In fact, they are even easier to explain than linear regression!</p></li>
@@ -1374,7 +1374,7 @@ attributes at each step while growing the tree.</p>
<li><p>Trees can be displayed graphically, and are easily interpreted even by a non-expert (especially if they are small)</p></li>
</ul>
<div class="section" id="disadvantages">
<h3><span class="section-number">8.7.1. </span>Disadvantages<a class="headerlink" href="#disadvantages" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">9.7.1. </span>Disadvantages<a class="headerlink" href="#disadvantages" title="Permalink to this headline"></a></h3>
<ul class="simple">
<li><p>Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches</p></li>
<li><p>If continuous features are used the tree may become quite large and hence less interpretable</p></li>
@@ -1416,8 +1416,8 @@ trees can be substantially improved.</p>
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="chapter5.html" title="previous page"><span class="section-number">7. </span>Support Vector Machines, overarching aims</a>
<a class='right-next' id="next-link" href="chapter7.html" title="next page"><span class="section-number">9. </span>Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods</a>
<a class='left-prev' id="prev-link" href="chapter5.html" title="previous page"><span class="section-number">8. </span>Support Vector Machines, overarching aims</a>
<a class='right-next' id="next-link" href="chapter7.html" title="next page"><span class="section-number">10. </span>Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods</a>
</div>