lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week 45

Browse Source
This commit is contained in:
mhjensen
2021-11-10 16:37:21 +01:00
parent 902ce81b35
commit 0e5076dfbf
79 changed files with 5458 additions and 7367 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week45/html/._week45-bs064.html
+68 -106
View File
@@ -101,81 +101,74 @@ Automatically generated HTML file from DocOnce source
('An Overview of Ensemble Methods', 2, None, '___sec37'),
('Bagging', 2, None, '___sec38'),
('More bagging', 2, None, '___sec39'),
('Simple Voting Example, head or tail', 2, None, '___sec40'),
('Using the Voting Classifier', 2, None, '___sec41'),
('Please, not the moons again! Voting and Bagging',
2,
None,
'___sec42'),
('Bagging Examples', 2, None, '___sec43'),
('Making your own Bootstrap: Changing the Level of the Decision '
'Tree',
2,
None,
'___sec44'),
('Why Voting?', 2, None, '___sec45'),
('Tossing coins', 2, None, '___sec46'),
('Standard imports first', 2, None, '___sec47'),
('Simple Voting Example, head or tail', 2, None, '___sec48'),
('Using the Voting Classifier', 2, None, '___sec49'),
('Voting and Bagging', 2, None, '___sec50'),
('Random forests', 2, None, '___sec51'),
('Random Forest Algorithm', 2, None, '___sec52'),
'___sec40'),
('Why Voting?', 2, None, '___sec41'),
('Tossing coins', 2, None, '___sec42'),
('Standard imports first', 2, None, '___sec43'),
('Simple Voting Example, head or tail', 2, None, '___sec44'),
('Using the Voting Classifier', 2, None, '___sec45'),
('Voting and Bagging', 2, None, '___sec46'),
('Random forests', 2, None, '___sec47'),
('Random Forest Algorithm', 2, None, '___sec48'),
('Random Forests Compared with other Methods on the Cancer Data',
2,
None,
'___sec53'),
'___sec49'),
('Compare Bagging on Trees with Random Forests',
2,
None,
'___sec54'),
("Boosting, a Bird's Eye View", 2, None, '___sec55'),
'___sec50'),
("Boosting, a Bird's Eye View", 2, None, '___sec51'),
('What is boosting? Additive Modelling/Iterative Fitting',
2,
None,
'___sec56'),
'___sec52'),
('Iterative Fitting, Regression and Squared-error Cost Function',
2,
None,
'___sec57'),
'___sec53'),
('Squared-Error Example and Iterative Fitting',
2,
None,
'___sec58'),
'___sec54'),
('Iterative Fitting, Classification and AdaBoost',
2,
None,
'___sec59'),
('Adaptive Boosting, AdaBoost', 2, None, '___sec60'),
('Building up AdaBoost', 2, None, '___sec61'),
'___sec55'),
('Adaptive Boosting, AdaBoost', 2, None, '___sec56'),
('Building up AdaBoost', 2, None, '___sec57'),
('Adaptive boosting: AdaBoost, Basic Algorithm',
2,
None,
'___sec62'),
('Basic Steps of AdaBoost', 2, None, '___sec63'),
('AdaBoost Examples', 2, None, '___sec64'),
'___sec58'),
('Basic Steps of AdaBoost', 2, None, '___sec59'),
('AdaBoost Examples', 2, None, '___sec60'),
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec65'),
'___sec61'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec66'),
('Steepest Descent Example', 2, None, '___sec67'),
('Gradient Boosting, algorithm', 2, None, '___sec68'),
'___sec62'),
('Steepest Descent Example', 2, None, '___sec63'),
('Gradient Boosting, algorithm', 2, None, '___sec64'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec69'),
'___sec65'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec70'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec71'),
('Regression Case', 2, None, '___sec72'),
('Xgboost on the Cancer Data', 2, None, '___sec73')]}
'___sec66'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec67'),
('Regression Case', 2, None, '___sec68'),
('Xgboost on the Cancer Data', 2, None, '___sec69')]}
end of tocinfo -->
<body>
@@ -253,40 +246,36 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs038.html#___sec37" style="font-size: 80%;">An Overview of Ensemble Methods</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs039.html#___sec38" style="font-size: 80%;">Bagging</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs040.html#___sec39" style="font-size: 80%;">More bagging</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs041.html#___sec40" style="font-size: 80%;">Simple Voting Example, head or tail</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs042.html#___sec41" style="font-size: 80%;">Using the Voting Classifier</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs043.html#___sec42" style="font-size: 80%;">Please, not the moons again! Voting and Bagging</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs044.html#___sec43" style="font-size: 80%;">Bagging Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs045.html#___sec44" style="font-size: 80%;">Making your own Bootstrap: Changing the Level of the Decision Tree</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs046.html#___sec45" style="font-size: 80%;">Why Voting?</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs047.html#___sec46" style="font-size: 80%;">Tossing coins</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs048.html#___sec47" style="font-size: 80%;">Standard imports first</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs049.html#___sec48" style="font-size: 80%;">Simple Voting Example, head or tail</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs050.html#___sec49" style="font-size: 80%;">Using the Voting Classifier</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs051.html#___sec50" style="font-size: 80%;">Voting and Bagging</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs052.html#___sec51" style="font-size: 80%;">Random forests</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs053.html#___sec52" style="font-size: 80%;">Random Forest Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs054.html#___sec53" style="font-size: 80%;">Random Forests Compared with other Methods on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs055.html#___sec54" style="font-size: 80%;">Compare Bagging on Trees with Random Forests</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs056.html#___sec55" style="font-size: 80%;">Boosting, a Bird's Eye View</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs057.html#___sec56" style="font-size: 80%;">What is boosting? Additive Modelling/Iterative Fitting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs058.html#___sec57" style="font-size: 80%;">Iterative Fitting, Regression and Squared-error Cost Function</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs059.html#___sec58" style="font-size: 80%;">Squared-Error Example and Iterative Fitting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs060.html#___sec59" style="font-size: 80%;">Iterative Fitting, Classification and AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs061.html#___sec60" style="font-size: 80%;">Adaptive Boosting, AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs062.html#___sec61" style="font-size: 80%;">Building up AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs063.html#___sec62" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="#___sec63" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs065.html#___sec64" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs066.html#___sec65" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs067.html#___sec66" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs068.html#___sec67" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs069.html#___sec68" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs070.html#___sec69" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs071.html#___sec70" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs072.html#___sec71" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs073.html#___sec72" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs074.html#___sec73" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs041.html#___sec40" style="font-size: 80%;">Making your own Bootstrap: Changing the Level of the Decision Tree</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs042.html#___sec41" style="font-size: 80%;">Why Voting?</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs043.html#___sec42" style="font-size: 80%;">Tossing coins</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs044.html#___sec43" style="font-size: 80%;">Standard imports first</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs045.html#___sec44" style="font-size: 80%;">Simple Voting Example, head or tail</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs046.html#___sec45" style="font-size: 80%;">Using the Voting Classifier</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs047.html#___sec46" style="font-size: 80%;">Voting and Bagging</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs048.html#___sec47" style="font-size: 80%;">Random forests</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs049.html#___sec48" style="font-size: 80%;">Random Forest Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs050.html#___sec49" style="font-size: 80%;">Random Forests Compared with other Methods on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs051.html#___sec50" style="font-size: 80%;">Compare Bagging on Trees with Random Forests</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs052.html#___sec51" style="font-size: 80%;">Boosting, a Bird's Eye View</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs053.html#___sec52" style="font-size: 80%;">What is boosting? Additive Modelling/Iterative Fitting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs054.html#___sec53" style="font-size: 80%;">Iterative Fitting, Regression and Squared-error Cost Function</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs055.html#___sec54" style="font-size: 80%;">Squared-Error Example and Iterative Fitting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs056.html#___sec55" style="font-size: 80%;">Iterative Fitting, Classification and AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs057.html#___sec56" style="font-size: 80%;">Adaptive Boosting, AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs058.html#___sec57" style="font-size: 80%;">Building up AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs059.html#___sec58" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs060.html#___sec59" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs061.html#___sec60" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs062.html#___sec61" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs063.html#___sec62" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="#___sec63" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs065.html#___sec64" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs066.html#___sec65" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs067.html#___sec66" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs068.html#___sec67" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs069.html#___sec68" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs070.html#___sec69" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -302,42 +291,20 @@ MathJax.Hub.Config({
<a name="part0064"></a>
<!-- !split -->
<h2 id="___sec63" class="anchor">Basic Steps of AdaBoost </h2>
<h2 id="___sec63" class="anchor">Steepest Descent Example </h2>
<p>
With the above definitions we are now ready to set up the algorithm for AdaBoost.
The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.
<ol>
<li> We start by initializing all weights to \( w_i = 1/n \), with \( i=0,1,2,\dots n-1 \). It is easy to see that we must have \( \sum_{i=0}^{n-1}w_i = 1 \).</li>
<li> We rewrite the misclassification error as</li>
</ol>
Optimizing with respect to \( \rho \) we obtain (taking the derivative) that \( \rho_1 = -1/2 \). We have then that
$$
\mathrm{\overline{err}}_m=\frac{\sum_{i=0}^{n-1}w_i^m I(y_i\ne G(x_i))}{\sum_{i=0}^{n-1}w_i},
f_1(x) = f_{0}(x) -\rho_1 g_1(x)=-y_i.
$$
We can then proceed and compute
$$
g_2(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{1}(x_i)=y_i}=-4y_i,
$$
<ol>
<li> Then we start looping over all attempts at classifying, namely we start an iterative process for \( m=1:M \), where \( M \) is the final number of classifications. Our given classifier could for example be a plain decision tree.
<ol type="a"></li>
<li> Fit then a given classifier to the training set using the weights \( w_i \).</li>
<li> Compute then \( \mathrm{err} \) and figure out which events are classified properly and which are classified wrongly.</li>
<li> Define a quantity \( \alpha_{m} = \log{(1-\mathrm{\overline{err}}_m)/\mathrm{\overline{err}}_m} \)</li>
<li> Set the new weights to \( w_i = w_i\times \exp{(\alpha_m I(y_i\ne G(x_i)} \).</li>
</ol>
<li> Compute the new classifier \( G(x)= \sum_{i=0}^{n-1}\alpha_m I(y_i\ne G(x_i) \).</li>
</ol>
For the iterations with \( m \le 2 \) the weights are modified
individually at each steps. The observations which were misclassified
at iteration \( m-1 \) have a weight which is larger than those which were
classified properly. As this proceeds, the observations which were
difficult to classifiy correctly are given a larger influence. Each
new classification step \( m \) is then forced to concentrate on those
observations that are missed in the previous iterations.
and find a new value for \( \rho_2=-1/2 \) and continue till we have reached \( m=M \). We can modify the steepest descent method, or steepest boosting, by introducing what is called <b>gradient boosting</b>.
<p>
<p>
@@ -361,11 +328,6 @@ observations that are missed in the previous iterations.
<li><a href="._week45-bs068.html">69</a></li>
<li><a href="._week45-bs069.html">70</a></li>
<li><a href="._week45-bs070.html">71</a></li>
<li><a href="._week45-bs071.html">72</a></li>
<li><a href="._week45-bs072.html">73</a></li>
<li><a href="._week45-bs073.html">74</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs074.html">75</a></li>
<li><a href="._week45-bs065.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->