lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

new update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-13 14:05:16 +02:00
parent 30d664a82b
commit ad92592adb
76 changed files with 5646 additions and 2878 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter7.html
+46 -46
View File
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>9. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods &#8212; Applied Data Analysis and Machine Learning</title>
<title>10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods &#8212; Applied Data Analysis and Machine Learning</title>
<link href="_static/css/theme.css" rel="stylesheet" />
<link href="_static/css/index.c5995385ac14fb8791e8eb36b4908be2.css" rel="stylesheet" />
@@ -54,8 +54,8 @@
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="10. Basic ideas of the Principal Component Analysis (PCA)" href="chapter8.html" />
<link rel="prev" title="8. Decision trees, overarching aims" href="chapter6.html" />
<link rel="next" title="11. Basic ideas of the Principal Component Analysis (PCA)" href="chapter8.html" />
<link rel="prev" title="9. Decision trees, overarching aims" href="chapter6.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en" />
@@ -139,17 +139,17 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter1.html">
3. Linear Regression, basic Elements
3. Linear Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html">
4. Resampling Methods
4. Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
5. Ridge and Lasso Regression
5. Resampling Methods
</a>
</li>
<li class="toctree-l1">
@@ -159,7 +159,7 @@
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
7. Support Vector Machines, overarching aims
8. Support Vector Machines, overarching aims
</a>
</li>
</ul>
@@ -171,12 +171,12 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
8. Decision trees, overarching aims
9. Decision trees, overarching aims
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
9. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
</a>
</li>
</ul>
@@ -188,12 +188,12 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
10. Basic ideas of the Principal Component Analysis (PCA)
11. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="Clustering.html">
11. Clustering Analysis
12. Clustering Analysis
</a>
</li>
</ul>
@@ -205,12 +205,12 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter9.html">
12. Neural networks
13. Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter10.html">
13. Building a Feed Forward Neural Network
14. Building a Feed Forward Neural Network
</a>
</li>
</ul>
@@ -286,83 +286,83 @@
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#an-overview-of-ensemble-methods">
9.1. An Overview of Ensemble Methods
10.1. An Overview of Ensemble Methods
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#bagging">
9.2. Bagging
10.2. Bagging
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#bagging-examples">
9.3. Bagging Examples
10.3. Bagging Examples
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#making-your-own-bootstrap-changing-the-level-of-the-decision-tree">
9.3.1. Making your own Bootstrap: Changing the Level of the Decision Tree
10.3.1. Making your own Bootstrap: Changing the Level of the Decision Tree
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-forests">
9.4. Random forests
10.4. Random forests
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#compare-bagging-on-trees-with-random-forests">
9.4.1. Compare Bagging on Trees with Random Forests
10.4.1. Compare Bagging on Trees with Random Forests
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#boosting-a-bird-s-eye-view">
9.5. Boosting, a Bird’s Eye View
10.5. Boosting, a Bird’s Eye View
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#iterative-fitting-regression-and-squared-error-cost-function">
9.5.1. Iterative Fitting, Regression and Squared-error Cost Function
10.5.1. Iterative Fitting, Regression and Squared-error Cost Function
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#iterative-fitting-classification-and-adaboost">
9.5.2. Iterative Fitting, Classification and AdaBoost
10.5.2. Iterative Fitting, Classification and AdaBoost
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#adaptive-boosting-adaboost-basic-algorithm">
9.5.3. Adaptive boosting: AdaBoost, Basic Algorithm
10.5.3. Adaptive boosting: AdaBoost, Basic Algorithm
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#gradient-boosting-basics-with-steepest-descent-functional-gradient-descent">
9.6. Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent
10.6. Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#gradient-boosting-examples-of-regression">
9.7. Gradient Boosting, Examples of Regression
10.7. Gradient Boosting, Examples of Regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#gradient-boosting-classification-example">
9.8. Gradient Boosting, Classification Example
10.8. Gradient Boosting, Classification Example
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#xgboost-extreme-gradient-boosting">
9.9. XGBoost: Extreme Gradient Boosting
10.9. XGBoost: Extreme Gradient Boosting
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#regression-case">
9.10. Regression Case
10.10. Regression Case
</a>
</li>
</ul>
@@ -377,7 +377,7 @@
<div>
<div class="section" id="ensemble-methods-from-a-single-tree-to-many-trees-and-extreme-boosting-meet-the-jungle-of-methods">
<h1><span class="section-number">9. </span>Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods<a class="headerlink" href="#ensemble-methods-from-a-single-tree-to-many-trees-and-extreme-boosting-meet-the-jungle-of-methods" title="Permalink to this headline"></a></h1>
<h1><span class="section-number">10. </span>Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods<a class="headerlink" href="#ensemble-methods-from-a-single-tree-to-many-trees-and-extreme-boosting-meet-the-jungle-of-methods" title="Permalink to this headline"></a></h1>
<p>As stated previously and seen in many of the examples discussed in the previous chapter about
a single decision tree, we often end up overfitting our training
data. This normally means that we have a high variance. Can we reduce
@@ -395,11 +395,11 @@ try to explain here. These are</p>
</ol>
<p>We discuss these methods here.</p>
<div class="section" id="an-overview-of-ensemble-methods">
<h2><span class="section-number">9.1. </span>An Overview of Ensemble Methods<a class="headerlink" href="#an-overview-of-ensemble-methods" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">10.1. </span>An Overview of Ensemble Methods<a class="headerlink" href="#an-overview-of-ensemble-methods" title="Permalink to this headline"></a></h2>
<!-- FIGURE: [DataFiles/ensembleoverview.png, width=600 frac=0.8] -->
</div>
<div class="section" id="bagging">
<h2><span class="section-number">9.2. </span>Bagging<a class="headerlink" href="#bagging" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">10.2. </span>Bagging<a class="headerlink" href="#bagging" title="Permalink to this headline"></a></h2>
<p>The <strong>plain</strong> decision trees suffer from high
variance. This means that if we split the training data into two parts
at random, and fit a decision tree to both halves, the results that we
@@ -576,7 +576,7 @@ predictor, averaged over all <span class="math notranslate nohighlight">\(B\)</s
</div>
</div>
<div class="section" id="bagging-examples">
<h2><span class="section-number">9.3. </span>Bagging Examples<a class="headerlink" href="#bagging-examples" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">10.3. </span>Bagging Examples<a class="headerlink" href="#bagging-examples" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="kn">import</span> <span class="n">BaggingClassifier</span>
@@ -645,7 +645,7 @@ predictor, averaged over all <span class="math notranslate nohighlight">\(B\)</s
</div>
</div>
<div class="section" id="making-your-own-bootstrap-changing-the-level-of-the-decision-tree">
<h3><span class="section-number">9.3.1. </span>Making your own Bootstrap: Changing the Level of the Decision Tree<a class="headerlink" href="#making-your-own-bootstrap-changing-the-level-of-the-decision-tree" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">10.3.1. </span>Making your own Bootstrap: Changing the Level of the Decision Tree<a class="headerlink" href="#making-your-own-bootstrap-changing-the-level-of-the-decision-tree" title="Permalink to this headline"></a></h3>
<p>Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with
a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points <span class="math notranslate nohighlight">\(n\)</span>).</p>
<div class="cell docutils container">
@@ -714,7 +714,7 @@ a decision tree wth different depths and perform a bootstrap aggregate (in this
</div>
</div>
<div class="section" id="random-forests">
<h2><span class="section-number">9.4. </span>Random forests<a class="headerlink" href="#random-forests" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">10.4. </span>Random forests<a class="headerlink" href="#random-forests" title="Permalink to this headline"></a></h2>
<p>Random forests provide an improvement over bagged trees by way of a
small tweak that decorrelates the trees.</p>
<p>As in bagging, we build a
@@ -838,7 +838,7 @@ percentage of the total number of cases.</p>
displays the diagnostic ability of a binary classifier system as its
discrimination threshold is varied. It plots the true positive rate against the false positive rate.</p>
<div class="section" id="compare-bagging-on-trees-with-random-forests">
<h3><span class="section-number">9.4.1. </span>Compare Bagging on Trees with Random Forests<a class="headerlink" href="#compare-bagging-on-trees-with-random-forests" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">10.4.1. </span>Compare Bagging on Trees with Random Forests<a class="headerlink" href="#compare-bagging-on-trees-with-random-forests" title="Permalink to this headline"></a></h3>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">bag_clf</span> <span class="o">=</span> <span class="n">BaggingClassifier</span><span class="p">(</span>
@@ -864,7 +864,7 @@ discrimination threshold is varied. It plots the true positive rate against the
</div>
</div>
<div class="section" id="boosting-a-bird-s-eye-view">
<h2><span class="section-number">9.5. </span>Boosting, a Bird’s Eye View<a class="headerlink" href="#boosting-a-bird-s-eye-view" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">10.5. </span>Boosting, a Bird’s Eye View<a class="headerlink" href="#boosting-a-bird-s-eye-view" title="Permalink to this headline"></a></h2>
<p>The basic idea is to combine weak classifiers in order to create a good
classifier. With a weak classifier we often intend a classifier which
produces results which are only slightly better than we would get by
@@ -909,7 +909,7 @@ simply invert a matrix and obtain the parameters <span class="math notranslate n
\]</div>
<p>In iterative fitting or additive modeling, we minimize the cost function with respect to the parameters <span class="math notranslate nohighlight">\(\beta_m\)</span> and <span class="math notranslate nohighlight">\(\gamma_m\)</span>.</p>
<div class="section" id="iterative-fitting-regression-and-squared-error-cost-function">
<h3><span class="section-number">9.5.1. </span>Iterative Fitting, Regression and Squared-error Cost Function<a class="headerlink" href="#iterative-fitting-regression-and-squared-error-cost-function" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">10.5.1. </span>Iterative Fitting, Regression and Squared-error Cost Function<a class="headerlink" href="#iterative-fitting-regression-and-squared-error-cost-function" title="Permalink to this headline"></a></h3>
<p>The way we proceed is as follows (here we specialize to the squared-error cost function)</p>
<ol class="simple">
<li><p>Establish a cost function, here <span class="math notranslate nohighlight">\(\cal{C}(\boldsymbol{y},\boldsymbol{f}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-f_M(x_i))^2\)</span> with <span class="math notranslate nohighlight">\(f_M(x) = \sum_{i=1}^M \beta_m b(x;\gamma_m)\)</span>.</p></li>
@@ -956,7 +956,7 @@ for <span class="math notranslate nohighlight">\(\beta\)</span> gives us an equa
<span class="math notranslate nohighlight">\(f_1(x) = \beta_1(1+\gamma_1x)\)</span>. Doing this <span class="math notranslate nohighlight">\(M\)</span> times results in our final estimate for the function <span class="math notranslate nohighlight">\(f\)</span>.</p>
</div>
<div class="section" id="iterative-fitting-classification-and-adaboost">
<h3><span class="section-number">9.5.2. </span>Iterative Fitting, Classification and AdaBoost<a class="headerlink" href="#iterative-fitting-classification-and-adaboost" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">10.5.2. </span>Iterative Fitting, Classification and AdaBoost<a class="headerlink" href="#iterative-fitting-classification-and-adaboost" title="Permalink to this headline"></a></h3>
<p>Let us consider a binary classification problem with two outcomes <span class="math notranslate nohighlight">\(y_i \in \{-1,1\}\)</span> and <span class="math notranslate nohighlight">\(i=0,1,2,\dots,n-1\)</span> as our set of
observations. We define a classification function <span class="math notranslate nohighlight">\(G(x)\)</span> which produces a prediction taking one or the other of the two values
<span class="math notranslate nohighlight">\(\{-1,1\}\)</span>.</p>
@@ -1036,7 +1036,7 @@ w_i^{m+1} = w_i^m \exp{(-y_i\beta_m G_m(x_i))}
\]</div>
</div>
<div class="section" id="adaptive-boosting-adaboost-basic-algorithm">
<h3><span class="section-number">9.5.3. </span>Adaptive boosting: AdaBoost, Basic Algorithm<a class="headerlink" href="#adaptive-boosting-adaboost-basic-algorithm" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">10.5.3. </span>Adaptive boosting: AdaBoost, Basic Algorithm<a class="headerlink" href="#adaptive-boosting-adaboost-basic-algorithm" title="Permalink to this headline"></a></h3>
<p>The algorithm here is rather straightforward. Assume that our weak
classifier is a decision tree and we consider a binary set of outputs
with <span class="math notranslate nohighlight">\(y_i \in \{-1,1\}\)</span> and <span class="math notranslate nohighlight">\(i=0,1,2,\dots,n-1\)</span> as our set of
@@ -1108,7 +1108,7 @@ observations that are missed in the previous iterations.</p>
</div>
</div>
<div class="section" id="gradient-boosting-basics-with-steepest-descent-functional-gradient-descent">
<h2><span class="section-number">9.6. </span>Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent<a class="headerlink" href="#gradient-boosting-basics-with-steepest-descent-functional-gradient-descent" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">10.6. </span>Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent<a class="headerlink" href="#gradient-boosting-basics-with-steepest-descent-functional-gradient-descent" title="Permalink to this headline"></a></h2>
<p>Gradient boosting is again a similar technique to Adaptive boosting,
it combines so-called weak classifiers or regressors into a strong
method via a series of iterations.</p>
@@ -1170,7 +1170,7 @@ C(\boldsymbol{y},\boldsymbol{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2.
</ol>
</div>
<div class="section" id="gradient-boosting-examples-of-regression">
<h2><span class="section-number">9.7. </span>Gradient Boosting, Examples of Regression<a class="headerlink" href="#gradient-boosting-examples-of-regression" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">10.7. </span>Gradient Boosting, Examples of Regression<a class="headerlink" href="#gradient-boosting-examples-of-regression" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
@@ -1225,7 +1225,7 @@ C(\boldsymbol{y},\boldsymbol{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2.
</div>
</div>
<div class="section" id="gradient-boosting-classification-example">
<h2><span class="section-number">9.8. </span>Gradient Boosting, Classification Example<a class="headerlink" href="#gradient-boosting-classification-example" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">10.8. </span>Gradient Boosting, Classification Example<a class="headerlink" href="#gradient-boosting-classification-example" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
@@ -1274,7 +1274,7 @@ C(\boldsymbol{y},\boldsymbol{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2.
</div>
</div>
<div class="section" id="xgboost-extreme-gradient-boosting">
<h2><span class="section-number">9.9. </span>XGBoost: Extreme Gradient Boosting<a class="headerlink" href="#xgboost-extreme-gradient-boosting" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">10.9. </span>XGBoost: Extreme Gradient Boosting<a class="headerlink" href="#xgboost-extreme-gradient-boosting" title="Permalink to this headline"></a></h2>
<p><a class="reference external" href="https://github.com/dmlc/xgboost">XGBoost</a> or Extreme Gradient
Boosting, is an optimized distributed gradient boosting library
designed to be highly efficient, flexible and portable. It implements
@@ -1287,7 +1287,7 @@ sketch for efficient proposal calculation. It introduces a novel sparsity-aware
<p>It is now the algorithm which wins essentially all ML competitions!!!</p>
</div>
<div class="section" id="regression-case">
<h2><span class="section-number">9.10. </span>Regression Case<a class="headerlink" href="#regression-case" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">10.10. </span>Regression Case<a class="headerlink" href="#regression-case" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
@@ -1426,8 +1426,8 @@ sketch for efficient proposal calculation. It introduces a novel sparsity-aware
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="chapter6.html" title="previous page"><span class="section-number">8. </span>Decision trees, overarching aims</a>
<a class='right-next' id="next-link" href="chapter8.html" title="next page"><span class="section-number">10. </span>Basic ideas of the Principal Component Analysis (PCA)</a>
<a class='left-prev' id="prev-link" href="chapter6.html" title="previous page"><span class="section-number">9. </span>Decision trees, overarching aims</a>
<a class='right-next' id="next-link" href="chapter8.html" title="next page"><span class="section-number">11. </span>Basic ideas of the Principal Component Analysis (PCA)</a>
</div>