lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

typos in week 45

Browse Source
This commit is contained in:
mhjensen
2020-11-12 11:24:09 +01:00
parent 587951f8d4
commit 92b2f672f3
39 changed files with 1058 additions and 1671 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week45/html/._week45-bs000.html
+22 -23
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -209,7 +208,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 6, 2020</h4></center> <!-- date -->
<center><h4>Nov 12, 2020</h4></center> <!-- date -->
<br>
<p>
@@ -233,7 +232,7 @@ MathJax.Hub.Config({
<li><a href="._week45-bs008.html">9</a></li>
<li><a href="._week45-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs001.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -219,7 +218,7 @@ Geron's chapter 7. See also lecture from <a href="https://www.uio.no/studier/emn
<li><a href="._week45-bs009.html">10</a></li>
<li><a href="._week45-bs010.html">11</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs002.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs002.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -215,7 +214,7 @@ We repeat here the voting approach since this will serve as a motivation for boo
<li><a href="._week45-bs010.html">11</a></li>
<li><a href="._week45-bs011.html">12</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs003.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs003.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -229,7 +228,7 @@ Decision trees play an important role as our weak classifier. They serve as the
<li><a href="._week45-bs011.html">12</a></li>
<li><a href="._week45-bs012.html">13</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs004.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs004.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -237,7 +236,7 @@ numbers kicking in.
<li><a href="._week45-bs012.html">13</a></li>
<li><a href="._week45-bs013.html">14</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs005.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs005.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -254,7 +253,7 @@ DATA_ID <span style="color: #666666">=</span> <span style="color: #BA2121">&quot
<li><a href="._week45-bs013.html">14</a></li>
<li><a href="._week45-bs014.html">15</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs006.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs006.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -239,7 +238,7 @@ plt<span style="color: #666666">.</span>show()
<li><a href="._week45-bs014.html">15</a></li>
<li><a href="._week45-bs015.html">16</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs007.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs007.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -263,7 +262,7 @@ voting_clf<span style="color: #666666">.</span>fit(X_train, y_train)
<li><a href="._week45-bs015.html">16</a></li>
<li><a href="._week45-bs016.html">17</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs008.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs008.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -270,7 +269,7 @@ voting_clf<span style="color: #666666">.</span>fit(X_train, y_train)
<li><a href="._week45-bs016.html">17</a></li>
<li><a href="._week45-bs017.html">18</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs009.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs009.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -256,7 +255,7 @@ this setting.
<li><a href="._week45-bs017.html">18</a></li>
<li><a href="._week45-bs018.html">19</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs010.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs010.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -239,7 +238,7 @@ We will grow of forest of say \( B \) trees.
<li><a href="._week45-bs018.html">19</a></li>
<li><a href="._week45-bs019.html">20</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs011.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs011.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -297,7 +296,7 @@ discrimination threshold is varied. It plots the true positive rate against the
<li><a href="._week45-bs019.html">20</a></li>
<li><a href="._week45-bs020.html">21</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs012.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs012.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -235,7 +234,7 @@ np<span style="color: #666666">.</span>sum(y_pred <span style="color: #666666">=
<li><a href="._week45-bs020.html">21</a></li>
<li><a href="._week45-bs021.html">22</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs013.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs013.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -230,7 +229,7 @@ them with a factor.
<li><a href="._week45-bs021.html">22</a></li>
<li><a href="._week45-bs022.html">23</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs014.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs014.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -265,7 +264,7 @@ In iterative fitting or additive modeling, we minimize the cost function with re
<li><a href="._week45-bs022.html">23</a></li>
<li><a href="._week45-bs023.html">24</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs015.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs015.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -238,7 +237,7 @@ at the internal nodes, and the predictions at the terminal nodes.
<li><a href="._week45-bs023.html">24</a></li>
<li><a href="._week45-bs024.html">25</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs016.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs016.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -261,7 +260,7 @@ The solution to these two equations gives us in turn \( \beta_1 \) and \( \gamma
<li><a href="._week45-bs024.html">25</a></li>
<li><a href="._week45-bs025.html">26</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs017.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs017.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -248,7 +247,7 @@ $$
<li><a href="._week45-bs025.html">26</a></li>
<li><a href="._week45-bs026.html">27</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs018.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs018.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -241,7 +240,7 @@ where we have defined \( w_i^m= \exp{(-y_if_{m-1}(x_i))} \).
<li><a href="._week45-bs026.html">27</a></li>
<li><a href="._week45-bs027.html">28</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs019.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs019.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -257,7 +256,7 @@ $$
<li><a href="._week45-bs027.html">28</a></li>
<li><a href="._week45-bs028.html">29</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs020.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs020.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -235,7 +234,7 @@ where the function \( I() \) is one if we misclassify and zero if we classify co
<li><a href="._week45-bs028.html">29</a></li>
<li><a href="._week45-bs029.html">30</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs021.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs021.html
+21 -22
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -253,7 +252,7 @@ observations that are missed in the previous iterations.
<li><a href="._week45-bs029.html">30</a></li>
<li><a href="._week45-bs030.html">31</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs022.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs022.html
+20 -23
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -245,8 +244,6 @@ plt<span style="color: #666666">.</span>show()
<li><a href="._week45-bs029.html">30</a></li>
<li><a href="._week45-bs030.html">31</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs023.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs023.html
+27 -42
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -190,30 +189,17 @@ MathJax.Hub.Config({
<a name="part0023"></a>
<!-- !split -->
<h2 id="___sec22" class="anchor">Additive boosting for Regression </h2>
<h2 id="___sec22" class="anchor">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent </h2>
<p>
Here we present <a href="https://pdfs.semanticscholar.org/8d49/e2dedb817f2c3330e74b63c5fc86d2399ce3.pdf" target="_self">Drucker's AdaBoost</a> tailored for regression.
Gradient boosting is again a similar technique to Adaptive boosting,
it combines so-called weak classifiers or regressors into a strong
method via a series of iterations.
<p>
In bagging, each training example is equally likely to be
picked. In boosting, the probability of a particular
example being in the training set of a particular machine
depends on the performance of the prior machines on
that example. The following is a modification of
Adaboost by Drucker.
<p>
Start by selecting a set of training data \( n \) and assign to each entry a weight \( w_i=1 \) for \( i=1,2,\dots,n \). As we have done earlier, we could pick say \( 80\% \) of the data set for training. The algorithm runs as follows:
<ol>
<li> We define the probability that the training sample \( i \) is in the set by \( p_i = w_i/\sum_iw_i \). We pick \( n \) samples (with replacement) to form our training set. We pick a number uniformly in the range \( [0,\sum_iw_i] \).</li>
<li> We choose then a regression machine (for example plain linear regression or a simple decision tree). A given regression machine makes then a hypothesis.</li>
<li> Using every member of the training set with the chosen regression machine we obtain then a prediction \( \tilde{y}_i \).</li>
<li> We calculate then the loss function \( L_i \) for each training sample. We can use various types of loss function as long as we have a value</li>
</ol>
\( L_i\in [0,1] \).
In order to understand the method, let us illustrate its basics by
bringing back the essential steps in linear regression, where our cost
function was the least squares function.
<p>
<p>
@@ -239,7 +225,6 @@ Start by selecting a set of training data \( n \) and assign to each entry a wei
<li><a href="._week45-bs029.html">30</a></li>
<li><a href="._week45-bs030.html">31</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs024.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs024.html
+47 -29
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -190,17 +189,37 @@ MathJax.Hub.Config({
<a name="part0024"></a>
<!-- !split -->
<h2 id="___sec23" class="anchor">Gradient boosting: Basics with Steepest Descent </h2>
<h2 id="___sec23" class="anchor">The Squared-Error again! Steepest Descent </h2>
<p>
Gradient boosting is again a similar technique to Adaptive boosting,
it combines so-called weak classifiers or regressors into a strong
method via a series of iterations.
We start again with our cost function \( {\cal C}(\boldsymbol{y}m\boldsymbol{f})=\sum_{i=0}^{n-1}{\cal L}(y_i, f(x_i)) \) where we want to minimize
This means that for every iteration, we need to optimize
$$
(\hat{\boldsymbol{f}}) = \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
$$
<p>
In order to understand the method, let us illustrate its basics by
bringing back the essential steps in linear regression, where our cost
function was the least squares function.
We define a real function \( h_m(x) \) that defines our final function \( f_M(x) \) as
$$
f_M(x) = \sum_{m=0}^M h_m(x).
$$
<p>
In the steepest decent approach we approximate \( h_m(x) = -\rho_m g_m(x) \), where \( \rho_m \) is a scalar and \( g_m(x) \) the gradient defined as
$$
g_m(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{m-1}(x_i)}.
$$
<p>
With the new gradient we can update \( f_m(x) = f_{m-1}(x) -\rho_m g_m(x) \). Using the above squared-error function we see that
the gradient is \( g_m(x_i) = -2(y_i-f(x_i)) \).
<p>
Choosing \( f_0(x)=0 \) we obtain \( g_m(x) = -2y_i \) and inserting this into the minimization problem for the cost function we have
$$
(\rho_1) = \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
$$
<p>
<p>
@@ -225,7 +244,6 @@ function was the least squares function.
<li><a href="._week45-bs029.html">30</a></li>
<li><a href="._week45-bs030.html">31</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs025.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs025.html
+26 -45
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -190,37 +189,20 @@ MathJax.Hub.Config({
<a name="part0025"></a>
<!-- !split -->
<h2 id="___sec24" class="anchor">The Squared-Error again! Steepest Descent </h2>
<h2 id="___sec24" class="anchor">Steepest Descent Example </h2>
<p>
We start again with our cost function \( {\cal C}(\boldsymbol{y}m\boldsymbol{f})=\sum_{i=0}^{n-1}{\cal L}(y_i, f(x_i)) \) where we want to minimize
This means that for every iteration, we need to optimize
Optimizing with respect to \( \rho \) we obtain (taking the derivative) that \( \rho_1 = -1/2 \). We have then that
$$
(\hat{\boldsymbol{f}}) = \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
f_1(x) = f_{0}(x) -\rho_1 g_1(x)=-y_i.
$$
<p>
We define a real function \( h_m(x) \) that defines our final function \( f_M(x) \) as
We can then proceed and compute
$$
f_M(x) = \sum_{m=0}^M h_m(x).
g_2(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{1}(x_i)=y_i}=-4y_i,
$$
<p>
In the steepest decent approach we approximate \( h_m(x) = -\rho_m g_m(x) \), where \( \rho_m \) is a scalar and \( g_m(x) \) the gradient defined as
$$
g_m(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{m-1}(x_i)}.
$$
<p>
With the new gradient we can update \( f_m(x) = f_{m-1}(x) -\rho_m g_m(x) \). Using the above squared-error function we see that
the gradient is \( g_m(x_i) = -2(y_i-f(x_i)) \).
<p>
Choosing \( f_0(x)=0 \) we obtain \( g_m(x) = -2y_i \) and inserting this into the minimization problem for the cost function we have
$$
(\rho_1) = \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
$$
and find a new value for \( \rho_2=-1/2 \) and continue till we have reached \( m=M \). We can modify the steepest descent method, or steepest boosting, by introducing what is called <b>gradient boosting</b>.
<p>
<p>
@@ -244,7 +226,6 @@ $$
<li><a href="._week45-bs029.html">30</a></li>
<li><a href="._week45-bs030.html">31</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs026.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs026.html
+45 -34
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -190,22 +189,35 @@ MathJax.Hub.Config({
<a name="part0026"></a>
<!-- !split -->
<h2 id="___sec25" class="anchor">Steepest Descent Example </h2>
<h2 id="___sec25" class="anchor">Gradient Boosting, algorithm </h2>
<p>
Optimizing with respect to \( \rho \) we obtain (taking the derivative) that \( \rho_1 = -1/2 \). We have then that
$$
f_1(x) = f_{0}(x) -\rho_1 g_1(x)=-y_i.
$$
We can then proceed and compute
$$
g_2(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{1}(x_i)=y_i}=-4y_i,
$$
and find a new value for \( \rho_2=-1/2 \) and continue till we have reached \( m=M \). We can modify the steepest descent method, or steepest boosting, by introducing what is called <b>gradient boosting</b>.
Steepest descent is however not much used, since it only optimizes \( f \) at a fixed set of \( n \) points,
so we do not learn a function that can generalize. However, we can modify the algorithm by
fitting a weak learner to approximate the negative gradient signal.
<p>
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function
$$
C(\boldsymbol{y},\boldsymbol{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2.
$$
<p>
The way we proceed in an iterative fashion is to
<ol>
<li> Initialize our estimate \( f_0(x) \).</li>
<li> For \( m=1:M \), we
<ol type="a"></li>
<li> compute the negative gradient vector \( \boldsymbol{u}_m = -\partial C(\boldsymbol{y},\boldsymbol{f})/\partial \boldsymbol{f}(x) \) at \( f(x) = f_{m-1}(x) \);</li>
<li> fit the so-called base-learner to the negative gradient \( h_m(u_m,x) \);</li>
<li> update the estimate \( f_m(x) = f_{m-1}(x)+h_m(u_m,x) \);</li>
</ol>
<li> The final estimate is then \( f_M(x) = \sum_{m=1}^M h_m(u_m,x) \).</li>
</ol>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
@@ -226,7 +238,6 @@ and find a new value for \( \rho_2=-1/2 \) and continue till we have reached \(
<li><a href="._week45-bs029.html">30</a></li>
<li><a href="._week45-bs030.html">31</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs027.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs027.html
+69 -48
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -190,35 +189,58 @@ MathJax.Hub.Config({
<a name="part0027"></a>
<!-- !split -->
<h2 id="___sec26" class="anchor">Gradient Boosting, algorithm </h2>
<h2 id="___sec26" class="anchor">Gradient Boosting, Examples of Regression </h2>
<p>
Steepest descent is however not much used, since it only optimizes \( f \) at a fixed set of \( n \) points,
so we do not learn a function that can generalize. However, we can modify the algorithm by
fitting a weak learner to approximate the negative gradient signal.
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.ensemble</span> <span style="color: #008000; font-weight: bold">import</span> GradientBoostingRegressor
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scikitplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">skplt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.metrics</span> <span style="color: #008000; font-weight: bold">import</span> mean_squared_error
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
maxdegree <span style="color: #666666">=</span> <span style="color: #666666">6</span>
<span style="color: #408080; font-style: italic"># Make data set.</span>
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">-3</span>, <span style="color: #666666">3</span>, n)<span style="color: #666666">.</span>reshape(<span style="color: #666666">-1</span>, <span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>x<span style="color: #666666">**2</span>) <span style="color: #666666">+</span> <span style="color: #666666">1.5</span> <span style="color: #666666">*</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(x<span style="color: #666666">-2</span>)<span style="color: #666666">**2</span>)<span style="color: #666666">+</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>normal(<span style="color: #666666">0</span>, <span style="color: #666666">0.1</span>, x<span style="color: #666666">.</span>shape)
error <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
bias <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
variance <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
polydegree <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(x, y, test_size<span style="color: #666666">=0.2</span>)
scaler <span style="color: #666666">=</span> StandardScaler()
scaler<span style="color: #666666">.</span>fit(X_train)
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
<span style="color: #008000; font-weight: bold">for</span> degree <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">1</span>,maxdegree):
model <span style="color: #666666">=</span> GradientBoostingRegressor(max_depth<span style="color: #666666">=</span>degree, n_estimators<span style="color: #666666">=100</span>, learning_rate<span style="color: #666666">=1.0</span>)
model<span style="color: #666666">.</span>fit(X_train_scaled,y_train)
y_pred <span style="color: #666666">=</span> model<span style="color: #666666">.</span>predict(X_test_scaled)
polydegree[degree] <span style="color: #666666">=</span> degree
error[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>mean((y_test <span style="color: #666666">-</span> y_pred)<span style="color: #666666">**2</span>) )
bias[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( (y_test <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(y_pred))<span style="color: #666666">**2</span> )
variance[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>var(y_pred) )
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Max depth:&#39;</span>, degree)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Error:&#39;</span>, error[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Bias^2:&#39;</span>, bias[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Var:&#39;</span>, variance[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;</span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> &gt;= </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> + </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">&#39;</span><span style="color: #666666">.</span>format(error[degree], bias[degree], variance[degree], bias[degree]<span style="color: #666666">+</span>variance[degree]))
plt<span style="color: #666666">.</span>xlim(<span style="color: #666666">1</span>,maxdegree<span style="color: #666666">-1</span>)
plt<span style="color: #666666">.</span>plot(polydegree, error, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Error&#39;</span>)
plt<span style="color: #666666">.</span>plot(polydegree, bias, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;bias&#39;</span>)
plt<span style="color: #666666">.</span>plot(polydegree, variance, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Variance&#39;</span>)
plt<span style="color: #666666">.</span>legend()
save_fig(<span style="color: #BA2121">&quot;gdregression&quot;</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function
$$
C(\boldsymbol{y},\boldsymbol{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2.
$$
<p>
The way we proceed in an iterative fashion is to
<ol>
<li> Initialize our estimate \( f_0(x) \).</li>
<li> For \( m=1:M \), we
<ol type="a"></li>
<li> compute the negative gradient vector \( \boldsymbol{u}_m = -\partial C(\boldsymbol{y},\boldsymbol{f})/\partial \boldsymbol{f}(x) \) at \( f(x) = f_{m-1}(x) \);</li>
<li> fit the so-called base-learner to the negative gradient \( h_m(u_m,x) \);</li>
<li> update the estimate \( f_m(x) = f_{m-1}(x)+\nu h_m(u_m,x) \);</li>
</ol>
<li> The final estimate is then \( f_M(x) = \sum_{m=1}^M\nu h_m(u_m,x) \).</li>
</ol>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
@@ -238,7 +260,6 @@ The way we proceed in an iterative fashion is to
<li><a href="._week45-bs029.html">30</a></li>
<li><a href="._week45-bs030.html">31</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs028.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs028.html
+49 -57
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -190,55 +189,49 @@ MathJax.Hub.Config({
<a name="part0028"></a>
<!-- !split -->
<h2 id="___sec27" class="anchor">Gradient Boosting, Examples of Regression </h2>
<h2 id="___sec27" class="anchor">Gradient Boosting, Classification Example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.ensemble</span> <span style="color: #008000; font-weight: bold">import</span> GradientBoostingRegressor
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> load_breast_cancer
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scikitplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">skplt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.metrics</span> <span style="color: #008000; font-weight: bold">import</span> mean_squared_error
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.ensemble</span> <span style="color: #008000; font-weight: bold">import</span> GradientBoostingClassifier
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> cross_validate
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
maxdegree <span style="color: #666666">=</span> <span style="color: #666666">6</span>
<span style="color: #408080; font-style: italic"># Load the data</span>
cancer <span style="color: #666666">=</span> load_breast_cancer()
<span style="color: #408080; font-style: italic"># Make data set.</span>
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">-3</span>, <span style="color: #666666">3</span>, n)<span style="color: #666666">.</span>reshape(<span style="color: #666666">-1</span>, <span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>x<span style="color: #666666">**2</span>) <span style="color: #666666">+</span> <span style="color: #666666">1.5</span> <span style="color: #666666">*</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(x<span style="color: #666666">-2</span>)<span style="color: #666666">**2</span>)<span style="color: #666666">+</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>normal(<span style="color: #666666">0</span>, <span style="color: #666666">0.1</span>, x<span style="color: #666666">.</span>shape)
error <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
bias <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
variance <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
polydegree <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(x, y, test_size<span style="color: #666666">=0.2</span>)
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(cancer<span style="color: #666666">.</span>data,cancer<span style="color: #666666">.</span>target,random_state<span style="color: #666666">=0</span>)
<span style="color: #008000">print</span>(X_train<span style="color: #666666">.</span>shape)
<span style="color: #008000">print</span>(X_test<span style="color: #666666">.</span>shape)
<span style="color: #408080; font-style: italic">#now scale the data</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
scaler <span style="color: #666666">=</span> StandardScaler()
scaler<span style="color: #666666">.</span>fit(X_train)
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
<span style="color: #008000; font-weight: bold">for</span> degree <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">1</span>,maxdegree):
model <span style="color: #666666">=</span> GradientBoostingRegressor(max_depth<span style="color: #666666">=</span>degree, n_estimators<span style="color: #666666">=100</span>, learning_rate<span style="color: #666666">=1.0</span>)
model<span style="color: #666666">.</span>fit(X_train_scaled,y_train)
y_pred <span style="color: #666666">=</span> model<span style="color: #666666">.</span>predict(X_test_scaled)
polydegree[degree] <span style="color: #666666">=</span> degree
error[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>mean((y_test <span style="color: #666666">-</span> y_pred)<span style="color: #666666">**2</span>) )
bias[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( (y_test <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(y_pred))<span style="color: #666666">**2</span> )
variance[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>var(y_pred) )
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Max depth:&#39;</span>, degree)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Error:&#39;</span>, error[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Bias^2:&#39;</span>, bias[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Var:&#39;</span>, variance[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;</span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> &gt;= </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> + </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">&#39;</span><span style="color: #666666">.</span>format(error[degree], bias[degree], variance[degree], bias[degree]<span style="color: #666666">+</span>variance[degree]))
gd_clf <span style="color: #666666">=</span> GradientBoostingClassifier(max_depth<span style="color: #666666">=3</span>, n_estimators<span style="color: #666666">=100</span>, learning_rate<span style="color: #666666">=1.0</span>)
gd_clf<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<span style="color: #408080; font-style: italic">#Cross validation</span>
accuracy <span style="color: #666666">=</span> cross_validate(gd_clf,X_test_scaled,y_test,cv<span style="color: #666666">=10</span>)[<span style="color: #BA2121">&#39;test_score&#39;</span>]
<span style="color: #008000">print</span>(accuracy)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Test set accuracy with Random Forests and scaled data: </span><span style="color: #BB6688; font-weight: bold">{:.2f}</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">.</span>format(gd_clf<span style="color: #666666">.</span>score(X_test_scaled,y_test)))
plt<span style="color: #666666">.</span>xlim(<span style="color: #666666">1</span>,maxdegree<span style="color: #666666">-1</span>)
plt<span style="color: #666666">.</span>plot(polydegree, error, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Error&#39;</span>)
plt<span style="color: #666666">.</span>plot(polydegree, bias, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;bias&#39;</span>)
plt<span style="color: #666666">.</span>plot(polydegree, variance, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Variance&#39;</span>)
plt<span style="color: #666666">.</span>legend()
save_fig(<span style="color: #BA2121">&quot;gdregression&quot;</span>)
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scikitplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">skplt</span>
y_pred <span style="color: #666666">=</span> gd_clf<span style="color: #666666">.</span>predict(X_test_scaled)
skplt<span style="color: #666666">.</span>metrics<span style="color: #666666">.</span>plot_confusion_matrix(y_test, y_pred, normalize<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>)
save_fig(<span style="color: #BA2121">&quot;gdclassiffierconfusion&quot;</span>)
plt<span style="color: #666666">.</span>show()
y_probas <span style="color: #666666">=</span> gd_clf<span style="color: #666666">.</span>predict_proba(X_test_scaled)
skplt<span style="color: #666666">.</span>metrics<span style="color: #666666">.</span>plot_roc(y_test, y_probas)
save_fig(<span style="color: #BA2121">&quot;gdclassiffierroc&quot;</span>)
plt<span style="color: #666666">.</span>show()
skplt<span style="color: #666666">.</span>metrics<span style="color: #666666">.</span>plot_cumulative_gain(y_test, y_probas)
save_fig(<span style="color: #BA2121">&quot;gdclassiffiercgain&quot;</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
@@ -260,7 +253,6 @@ plt<span style="color: #666666">.</span>show()
<li><a href="._week45-bs029.html">30</a></li>
<li><a href="._week45-bs030.html">31</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs029.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs029.html
+34 -63
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -190,51 +189,24 @@ MathJax.Hub.Config({
<a name="part0029"></a>
<!-- !split -->
<h2 id="___sec28" class="anchor">Gradient Boosting, Classification Example </h2>
<h2 id="___sec28" class="anchor">XGBoost: Extreme Gradient Boosting </h2>
<p>
<a href="https://github.com/dmlc/xgboost" target="_self">XGBoost</a> or Extreme Gradient
Boosting, is an optimized distributed gradient boosting library
designed to be highly efficient, flexible and portable. It implements
machine learning algorithms under the Gradient Boosting
framework. XGBoost provides a parallel tree boosting that solve many
data science problems in a fast and accurate way. See the <a href="https://arxiv.org/abs/1603.02754" target="_self">article by Chen and Guestrin</a>.
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> load_breast_cancer
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scikitplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">skplt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.ensemble</span> <span style="color: #008000; font-weight: bold">import</span> GradientBoostingClassifier
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> cross_validate
<p>
The authors design and build a highly scalable end-to-end tree
boosting system. It has a theoretically justified weighted quantile
sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning.
<span style="color: #408080; font-style: italic"># Load the data</span>
cancer <span style="color: #666666">=</span> load_breast_cancer()
<p>
It is now the algorithm which wins essentially all ML competitions!!!
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(cancer<span style="color: #666666">.</span>data,cancer<span style="color: #666666">.</span>target,random_state<span style="color: #666666">=0</span>)
<span style="color: #008000">print</span>(X_train<span style="color: #666666">.</span>shape)
<span style="color: #008000">print</span>(X_test<span style="color: #666666">.</span>shape)
<span style="color: #408080; font-style: italic">#now scale the data</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
scaler <span style="color: #666666">=</span> StandardScaler()
scaler<span style="color: #666666">.</span>fit(X_train)
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
gd_clf <span style="color: #666666">=</span> GradientBoostingClassifier(max_depth<span style="color: #666666">=3</span>, n_estimators<span style="color: #666666">=100</span>, learning_rate<span style="color: #666666">=1.0</span>)
gd_clf<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<span style="color: #408080; font-style: italic">#Cross validation</span>
accuracy <span style="color: #666666">=</span> cross_validate(gd_clf,X_test_scaled,y_test,cv<span style="color: #666666">=10</span>)[<span style="color: #BA2121">&#39;test_score&#39;</span>]
<span style="color: #008000">print</span>(accuracy)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Test set accuracy with Random Forests and scaled data: </span><span style="color: #BB6688; font-weight: bold">{:.2f}</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">.</span>format(gd_clf<span style="color: #666666">.</span>score(X_test_scaled,y_test)))
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scikitplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">skplt</span>
y_pred <span style="color: #666666">=</span> gd_clf<span style="color: #666666">.</span>predict(X_test_scaled)
skplt<span style="color: #666666">.</span>metrics<span style="color: #666666">.</span>plot_confusion_matrix(y_test, y_pred, normalize<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>)
save_fig(<span style="color: #BA2121">&quot;gdclassiffierconfusion&quot;</span>)
plt<span style="color: #666666">.</span>show()
y_probas <span style="color: #666666">=</span> gd_clf<span style="color: #666666">.</span>predict_proba(X_test_scaled)
skplt<span style="color: #666666">.</span>metrics<span style="color: #666666">.</span>plot_roc(y_test, y_probas)
save_fig(<span style="color: #BA2121">&quot;gdclassiffierroc&quot;</span>)
plt<span style="color: #666666">.</span>show()
skplt<span style="color: #666666">.</span>metrics<span style="color: #666666">.</span>plot_cumulative_gain(y_test, y_probas)
save_fig(<span style="color: #BA2121">&quot;gdclassiffiercgain&quot;</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
@@ -253,7 +225,6 @@ plt<span style="color: #666666">.</span>show()
<li class="active"><a href="._week45-bs029.html">30</a></li>
<li><a href="._week45-bs030.html">31</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs030.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs030.html
+67 -35
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -190,24 +189,58 @@ MathJax.Hub.Config({
<a name="part0030"></a>
<!-- !split -->
<h2 id="___sec29" class="anchor">XGBoost: Extreme Gradient Boosting </h2>
<h2 id="___sec29" class="anchor">Regression Case </h2>
<p>
<a href="https://github.com/dmlc/xgboost" target="_self">XGBoost</a> or Extreme Gradient
Boosting, is an optimized distributed gradient boosting library
designed to be highly efficient, flexible and portable. It implements
machine learning algorithms under the Gradient Boosting
framework. XGBoost provides a parallel tree boosting that solve many
data science problems in a fast and accurate way. See the <a href="https://arxiv.org/abs/1603.02754" target="_self">article by Chen and Guestrin</a>.
<p>
The authors design and build a highly scalable end-to-end tree
boosting system. It has a theoretically justified weighted quantile
sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning.
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">xgboost</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">xgb</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scikitplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">skplt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.metrics</span> <span style="color: #008000; font-weight: bold">import</span> mean_squared_error
<p>
It is now the algorithm which wins essentially all ML competitions!!!
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
maxdegree <span style="color: #666666">=</span> <span style="color: #666666">6</span>
<span style="color: #408080; font-style: italic"># Make data set.</span>
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">-3</span>, <span style="color: #666666">3</span>, n)<span style="color: #666666">.</span>reshape(<span style="color: #666666">-1</span>, <span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>x<span style="color: #666666">**2</span>) <span style="color: #666666">+</span> <span style="color: #666666">1.5</span> <span style="color: #666666">*</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(x<span style="color: #666666">-2</span>)<span style="color: #666666">**2</span>)<span style="color: #666666">+</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>normal(<span style="color: #666666">0</span>, <span style="color: #666666">0.1</span>, x<span style="color: #666666">.</span>shape)
error <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
bias <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
variance <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
polydegree <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(x, y, test_size<span style="color: #666666">=0.2</span>)
scaler <span style="color: #666666">=</span> StandardScaler()
scaler<span style="color: #666666">.</span>fit(X_train)
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
<span style="color: #008000; font-weight: bold">for</span> degree <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(maxdegree):
model <span style="color: #666666">=</span> xgb<span style="color: #666666">.</span>XGBRegressor(objective <span style="color: #666666">=</span><span style="color: #BA2121">&#39;reg:squarederror&#39;</span>, colsaobjective <span style="color: #666666">=</span><span style="color: #BA2121">&#39;reg:squarederror&#39;</span>, colsample_bytree <span style="color: #666666">=</span> <span style="color: #666666">0.3</span>, learning_rate <span style="color: #666666">=</span> <span style="color: #666666">0.1</span>,max_depth <span style="color: #666666">=</span> degree, alpha <span style="color: #666666">=</span> <span style="color: #666666">10</span>, n_estimators <span style="color: #666666">=</span> <span style="color: #666666">200</span>)
model<span style="color: #666666">.</span>fit(X_train_scaled,y_train)
y_pred <span style="color: #666666">=</span> model<span style="color: #666666">.</span>predict(X_test_scaled)
polydegree[degree] <span style="color: #666666">=</span> degree
error[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>mean((y_test <span style="color: #666666">-</span> y_pred)<span style="color: #666666">**2</span>) )
bias[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( (y_test <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(y_pred))<span style="color: #666666">**2</span> )
variance[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>var(y_pred) )
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Max depth:&#39;</span>, degree)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Error:&#39;</span>, error[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Bias^2:&#39;</span>, bias[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Var:&#39;</span>, variance[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;</span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> &gt;= </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> + </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">&#39;</span><span style="color: #666666">.</span>format(error[degree], bias[degree], variance[degree], bias[degree]<span style="color: #666666">+</span>variance[degree]))
plt<span style="color: #666666">.</span>xlim(<span style="color: #666666">1</span>,maxdegree<span style="color: #666666">-1</span>)
plt<span style="color: #666666">.</span>plot(polydegree, error, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Error&#39;</span>)
plt<span style="color: #666666">.</span>plot(polydegree, bias, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;bias&#39;</span>)
plt<span style="color: #666666">.</span>plot(polydegree, variance, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Variance&#39;</span>)
plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
@@ -225,7 +258,6 @@ It is now the algorithm which wins essentially all ML competitions!!!
<li><a href="._week45-bs029.html">30</a></li>
<li class="active"><a href="._week45-bs030.html">31</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/._week45-bs031.html
+63 -58
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -190,59 +189,67 @@ MathJax.Hub.Config({
<a name="part0031"></a>
<!-- !split -->
<h2 id="___sec30" class="anchor">Regression Case </h2>
<h2 id="___sec30" class="anchor">Xgboost on the Cancer Data </h2>
<p>
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">xgboost</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">xgb</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> load_breast_cancer
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> LabelEncoder
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> cross_validate
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scikitplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">skplt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.metrics</span> <span style="color: #008000; font-weight: bold">import</span> mean_squared_error
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">xgboost</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">xgb</span>
<span style="color: #408080; font-style: italic"># Load the data</span>
cancer <span style="color: #666666">=</span> load_breast_cancer()
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
maxdegree <span style="color: #666666">=</span> <span style="color: #666666">6</span>
<span style="color: #408080; font-style: italic"># Make data set.</span>
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">-3</span>, <span style="color: #666666">3</span>, n)<span style="color: #666666">.</span>reshape(<span style="color: #666666">-1</span>, <span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>x<span style="color: #666666">**2</span>) <span style="color: #666666">+</span> <span style="color: #666666">1.5</span> <span style="color: #666666">*</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(x<span style="color: #666666">-2</span>)<span style="color: #666666">**2</span>)<span style="color: #666666">+</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>normal(<span style="color: #666666">0</span>, <span style="color: #666666">0.1</span>, x<span style="color: #666666">.</span>shape)
error <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
bias <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
variance <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
polydegree <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(maxdegree)
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(x, y, test_size<span style="color: #666666">=0.2</span>)
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(cancer<span style="color: #666666">.</span>data,cancer<span style="color: #666666">.</span>target,random_state<span style="color: #666666">=0</span>)
<span style="color: #008000">print</span>(X_train<span style="color: #666666">.</span>shape)
<span style="color: #008000">print</span>(X_test<span style="color: #666666">.</span>shape)
<span style="color: #408080; font-style: italic">#now scale the data</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
scaler <span style="color: #666666">=</span> StandardScaler()
scaler<span style="color: #666666">.</span>fit(X_train)
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
<span style="color: #008000; font-weight: bold">for</span> degree <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(maxdegree):
model <span style="color: #666666">=</span> xgb<span style="color: #666666">.</span>XGBRegressor(objective <span style="color: #666666">=</span><span style="color: #BA2121">&#39;reg:squarederror&#39;</span>, colsaobjective <span style="color: #666666">=</span><span style="color: #BA2121">&#39;reg:squarederror&#39;</span>, colsample_bytree <span style="color: #666666">=</span> <span style="color: #666666">0.3</span>, learning_rate <span style="color: #666666">=</span> <span style="color: #666666">0.1</span>,max_depth <span style="color: #666666">=</span> degree, alpha <span style="color: #666666">=</span> <span style="color: #666666">10</span>, n_estimators <span style="color: #666666">=</span> <span style="color: #666666">200</span>)
xg_clf <span style="color: #666666">=</span> xgb<span style="color: #666666">.</span>XGBClassifier()
xg_clf<span style="color: #666666">.</span>fit(X_train_scaled,y_train)
model<span style="color: #666666">.</span>fit(X_train_scaled,y_train)
y_pred <span style="color: #666666">=</span> model<span style="color: #666666">.</span>predict(X_test_scaled)
polydegree[degree] <span style="color: #666666">=</span> degree
error[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>mean((y_test <span style="color: #666666">-</span> y_pred)<span style="color: #666666">**2</span>) )
bias[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( (y_test <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(y_pred))<span style="color: #666666">**2</span> )
variance[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>var(y_pred) )
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Max depth:&#39;</span>, degree)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Error:&#39;</span>, error[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Bias^2:&#39;</span>, bias[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Var:&#39;</span>, variance[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;</span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> &gt;= </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> + </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">&#39;</span><span style="color: #666666">.</span>format(error[degree], bias[degree], variance[degree], bias[degree]<span style="color: #666666">+</span>variance[degree]))
y_test <span style="color: #666666">=</span> xg_clf<span style="color: #666666">.</span>predict(X_test_scaled)
plt<span style="color: #666666">.</span>xlim(<span style="color: #666666">1</span>,maxdegree<span style="color: #666666">-1</span>)
plt<span style="color: #666666">.</span>plot(polydegree, error, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Error&#39;</span>)
plt<span style="color: #666666">.</span>plot(polydegree, bias, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;bias&#39;</span>)
plt<span style="color: #666666">.</span>plot(polydegree, variance, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Variance&#39;</span>)
plt<span style="color: #666666">.</span>legend()
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Test set accuracy with Random Forests and scaled data: </span><span style="color: #BB6688; font-weight: bold">{:.2f}</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">.</span>format(xg_clf<span style="color: #666666">.</span>score(X_test_scaled,y_test)))
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scikitplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">skplt</span>
y_pred <span style="color: #666666">=</span> xg_clf<span style="color: #666666">.</span>predict(X_test_scaled)
skplt<span style="color: #666666">.</span>metrics<span style="color: #666666">.</span>plot_confusion_matrix(y_test, y_pred, normalize<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>)
save_fig(<span style="color: #BA2121">&quot;xdclassiffierconfusion&quot;</span>)
plt<span style="color: #666666">.</span>show()
y_probas <span style="color: #666666">=</span> xg_clf<span style="color: #666666">.</span>predict_proba(X_test_scaled)
skplt<span style="color: #666666">.</span>metrics<span style="color: #666666">.</span>plot_roc(y_test, y_probas)
save_fig(<span style="color: #BA2121">&quot;xdclassiffierroc&quot;</span>)
plt<span style="color: #666666">.</span>show()
skplt<span style="color: #666666">.</span>metrics<span style="color: #666666">.</span>plot_cumulative_gain(y_test, y_probas)
save_fig(<span style="color: #BA2121">&quot;gdclassiffiercgain&quot;</span>)
plt<span style="color: #666666">.</span>show()
xgb<span style="color: #666666">.</span>plot_tree(xg_clf,num_trees<span style="color: #666666">=0</span>)
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">&#39;figure.figsize&#39;</span>] <span style="color: #666666">=</span> [<span style="color: #666666">50</span>, <span style="color: #666666">10</span>]
save_fig(<span style="color: #BA2121">&quot;xgtree&quot;</span>)
plt<span style="color: #666666">.</span>show()
xgb<span style="color: #666666">.</span>plot_importance(xg_clf)
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">&#39;figure.figsize&#39;</span>] <span style="color: #666666">=</span> [<span style="color: #666666">5</span>, <span style="color: #666666">5</span>]
save_fig(<span style="color: #BA2121">&quot;xgparams&quot;</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
@@ -258,8 +265,6 @@ plt<span style="color: #666666">.</span>show()
<li><a href="._week45-bs029.html">30</a></li>
<li><a href="._week45-bs030.html">31</a></li>
<li class="active"><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs032.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/week45-bs.html
+22 -23
View File
@@ -84,28 +84,28 @@ Automatically generated HTML file from DocOnce source
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -165,16 +165,15 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week45-bs020.html#___sec19" style="font-size: 80%;">Adaptive boosting: AdaBoost, Basic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs021.html#___sec20" style="font-size: 80%;">Basic Steps of AdaBoost</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs022.html#___sec21" style="font-size: 80%;">AdaBoost Examples</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Additive boosting for Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs032.html#___sec31" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs023.html#___sec22" style="font-size: 80%;">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs024.html#___sec23" style="font-size: 80%;">The Squared-Error again! Steepest Descent</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs025.html#___sec24" style="font-size: 80%;">Steepest Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs026.html#___sec25" style="font-size: 80%;">Gradient Boosting, algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs027.html#___sec26" style="font-size: 80%;">Gradient Boosting, Examples of Regression</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs028.html#___sec27" style="font-size: 80%;">Gradient Boosting, Classification Example</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs029.html#___sec28" style="font-size: 80%;">XGBoost: Extreme Gradient Boosting</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs030.html#___sec29" style="font-size: 80%;">Regression Case</a></li>
<!-- navigation toc: --> <li><a href="._week45-bs031.html#___sec30" style="font-size: 80%;">Xgboost on the Cancer Data</a></li>
</ul>
</li>
@@ -209,7 +208,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 6, 2020</h4></center> <!-- date -->
<center><h4>Nov 12, 2020</h4></center> <!-- date -->
<br>
<p>
@@ -233,7 +232,7 @@ MathJax.Hub.Config({
<li><a href="._week45-bs008.html">9</a></li>
<li><a href="._week45-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week45-bs032.html">33</a></li>
<li><a href="._week45-bs031.html">32</a></li>
<li><a href="._week45-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week45/html/week45-reveal.html
+12 -41
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Nov 6, 2020</h4></center> <!-- date -->
<center><h4>Nov 12, 2020</h4></center> <!-- date -->
<br>
<p>
@@ -1007,36 +1007,7 @@ plt.show()
<section>
<h2 id="___sec22">Additive boosting for Regression </h2>
<p>
Here we present <a href="https://pdfs.semanticscholar.org/8d49/e2dedb817f2c3330e74b63c5fc86d2399ce3.pdf" target="_blank">Drucker's AdaBoost</a> tailored for regression.
<p>
In bagging, each training example is equally likely to be
picked. In boosting, the probability of a particular
example being in the training set of a particular machine
depends on the performance of the prior machines on
that example. The following is a modification of
Adaboost by Drucker.
<p>
Start by selecting a set of training data \( n \) and assign to each entry a weight \( w_i=1 \) for \( i=1,2,\dots,n \). As we have done earlier, we could pick say \( 80\% \) of the data set for training. The algorithm runs as follows:
<ol>
<p><li> We define the probability that the training sample \( i \) is in the set by \( p_i = w_i/\sum_iw_i \). We pick \( n \) samples (with replacement) to form our training set. We pick a number uniformly in the range \( [0,\sum_iw_i] \).</li>
<p><li> We choose then a regression machine (for example plain linear regression or a simple decision tree). A given regression machine makes then a hypothesis.</li>
<p><li> Using every member of the training set with the chosen regression machine we obtain then a prediction \( \tilde{y}_i \).</li>
<p><li> We calculate then the loss function \( L_i \) for each training sample. We can use various types of loss function as long as we have a value</li>
</ol>
<p>
\( L_i\in [0,1] \).
</section>
<section>
<h2 id="___sec23">Gradient boosting: Basics with Steepest Descent </h2>
<h2 id="___sec22">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent </h2>
<p>
Gradient boosting is again a similar technique to Adaptive boosting,
@@ -1051,7 +1022,7 @@ function was the least squares function.
<section>
<h2 id="___sec24">The Squared-Error again! Steepest Descent </h2>
<h2 id="___sec23">The Squared-Error again! Steepest Descent </h2>
<p>
We start again with our cost function \( {\cal C}(\boldsymbol{y}m\boldsymbol{f})=\sum_{i=0}^{n-1}{\cal L}(y_i, f(x_i)) \) where we want to minimize
@@ -1094,7 +1065,7 @@ $$
<section>
<h2 id="___sec25">Steepest Descent Example </h2>
<h2 id="___sec24">Steepest Descent Example </h2>
<p>
Optimizing with respect to \( \rho \) we obtain (taking the derivative) that \( \rho_1 = -1/2 \). We have then that
@@ -1116,7 +1087,7 @@ and find a new value for \( \rho_2=-1/2 \) and continue till we have reached \(
<section>
<h2 id="___sec26">Gradient Boosting, algorithm </h2>
<h2 id="___sec25">Gradient Boosting, algorithm </h2>
<p>
Steepest descent is however not much used, since it only optimizes \( f \) at a fixed set of \( n \) points,
@@ -1141,15 +1112,15 @@ The way we proceed in an iterative fashion is to
<ol type="a"></li>
<p><li> compute the negative gradient vector \( \boldsymbol{u}_m = -\partial C(\boldsymbol{y},\boldsymbol{f})/\partial \boldsymbol{f}(x) \) at \( f(x) = f_{m-1}(x) \);</li>
<p><li> fit the so-called base-learner to the negative gradient \( h_m(u_m,x) \);</li>
<p><li> update the estimate \( f_m(x) = f_{m-1}(x)+\nu h_m(u_m,x) \);</li>
<p><li> update the estimate \( f_m(x) = f_{m-1}(x)+h_m(u_m,x) \);</li>
</ol>
<p><li> The final estimate is then \( f_M(x) = \sum_{m=1}^M\nu h_m(u_m,x) \).</li>
<p><li> The final estimate is then \( f_M(x) = \sum_{m=1}^M h_m(u_m,x) \).</li>
</ol>
</section>
<section>
<h2 id="___sec27">Gradient Boosting, Examples of Regression </h2>
<h2 id="___sec26">Gradient Boosting, Examples of Regression </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -1204,7 +1175,7 @@ plt.show()
<section>
<h2 id="___sec28">Gradient Boosting, Classification Example </h2>
<h2 id="___sec27">Gradient Boosting, Classification Example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -1253,7 +1224,7 @@ plt.show()
<section>
<h2 id="___sec29">XGBoost: Extreme Gradient Boosting </h2>
<h2 id="___sec28">XGBoost: Extreme Gradient Boosting </h2>
<p>
<a href="https://github.com/dmlc/xgboost" target="_blank">XGBoost</a> or Extreme Gradient
@@ -1274,7 +1245,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
<section>
<h2 id="___sec30">Regression Case </h2>
<h2 id="___sec29">Regression Case </h2>
<p>
@@ -1330,7 +1301,7 @@ plt.show()
<section>
<h2 id="___sec31">Xgboost on the Cancer Data </h2>
<h2 id="___sec30">Xgboost on the Cancer Data </h2>
<p>
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
doc/pub/week45/html/week45-solarized.html
+23 -51
View File
@@ -78,28 +78,28 @@ div { text-align: justify; text-justify: inter-word; }
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -141,7 +141,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 6, 2020</h4></center> <!-- date -->
<center><h4>Nov 12, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -932,35 +932,7 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec22">Additive boosting for Regression </h2>
<p>
Here we present <a href="https://pdfs.semanticscholar.org/8d49/e2dedb817f2c3330e74b63c5fc86d2399ce3.pdf" target="_blank">Drucker's AdaBoost</a> tailored for regression.
<p>
In bagging, each training example is equally likely to be
picked. In boosting, the probability of a particular
example being in the training set of a particular machine
depends on the performance of the prior machines on
that example. The following is a modification of
Adaboost by Drucker.
<p>
Start by selecting a set of training data \( n \) and assign to each entry a weight \( w_i=1 \) for \( i=1,2,\dots,n \). As we have done earlier, we could pick say \( 80\% \) of the data set for training. The algorithm runs as follows:
<ol>
<li> We define the probability that the training sample \( i \) is in the set by \( p_i = w_i/\sum_iw_i \). We pick \( n \) samples (with replacement) to form our training set. We pick a number uniformly in the range \( [0,\sum_iw_i] \).</li>
<li> We choose then a regression machine (for example plain linear regression or a simple decision tree). A given regression machine makes then a hypothesis.</li>
<li> Using every member of the training set with the chosen regression machine we obtain then a prediction \( \tilde{y}_i \).</li>
<li> We calculate then the loss function \( L_i \) for each training sample. We can use various types of loss function as long as we have a value</li>
</ol>
\( L_i\in [0,1] \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">Gradient boosting: Basics with Steepest Descent </h2>
<h2 id="___sec22">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent </h2>
<p>
Gradient boosting is again a similar technique to Adaptive boosting,
@@ -975,7 +947,7 @@ function was the least squares function.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">The Squared-Error again! Steepest Descent </h2>
<h2 id="___sec23">The Squared-Error again! Steepest Descent </h2>
<p>
We start again with our cost function \( {\cal C}(\boldsymbol{y}m\boldsymbol{f})=\sum_{i=0}^{n-1}{\cal L}(y_i, f(x_i)) \) where we want to minimize
@@ -1010,7 +982,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">Steepest Descent Example </h2>
<h2 id="___sec24">Steepest Descent Example </h2>
<p>
Optimizing with respect to \( \rho \) we obtain (taking the derivative) that \( \rho_1 = -1/2 \). We have then that
@@ -1028,7 +1000,7 @@ and find a new value for \( \rho_2=-1/2 \) and continue till we have reached \(
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">Gradient Boosting, algorithm </h2>
<h2 id="___sec25">Gradient Boosting, algorithm </h2>
<p>
Steepest descent is however not much used, since it only optimizes \( f \) at a fixed set of \( n \) points,
@@ -1051,15 +1023,15 @@ The way we proceed in an iterative fashion is to
<ol type="a"></li>
<li> compute the negative gradient vector \( \boldsymbol{u}_m = -\partial C(\boldsymbol{y},\boldsymbol{f})/\partial \boldsymbol{f}(x) \) at \( f(x) = f_{m-1}(x) \);</li>
<li> fit the so-called base-learner to the negative gradient \( h_m(u_m,x) \);</li>
<li> update the estimate \( f_m(x) = f_{m-1}(x)+\nu h_m(u_m,x) \);</li>
<li> update the estimate \( f_m(x) = f_{m-1}(x)+h_m(u_m,x) \);</li>
</ol>
<li> The final estimate is then \( f_M(x) = \sum_{m=1}^M\nu h_m(u_m,x) \).</li>
<li> The final estimate is then \( f_M(x) = \sum_{m=1}^M h_m(u_m,x) \).</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec27">Gradient Boosting, Examples of Regression </h2>
<h2 id="___sec26">Gradient Boosting, Examples of Regression </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -1113,7 +1085,7 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec28">Gradient Boosting, Classification Example </h2>
<h2 id="___sec27">Gradient Boosting, Classification Example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -1161,7 +1133,7 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec29">XGBoost: Extreme Gradient Boosting </h2>
<h2 id="___sec28">XGBoost: Extreme Gradient Boosting </h2>
<p>
<a href="https://github.com/dmlc/xgboost" target="_blank">XGBoost</a> or Extreme Gradient
@@ -1182,7 +1154,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec30">Regression Case </h2>
<h2 id="___sec29">Regression Case </h2>
<p>
@@ -1237,7 +1209,7 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec31">Xgboost on the Cancer Data </h2>
<h2 id="___sec30">Xgboost on the Cancer Data </h2>
<p>
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
doc/pub/week45/html/week45.html
+23 -51
View File
@@ -83,28 +83,28 @@ div { text-align: justify; text-justify: inter-word; }
'___sec19'),
('Basic Steps of AdaBoost', 2, None, '___sec20'),
('AdaBoost Examples', 2, None, '___sec21'),
('Additive boosting for Regression', 2, None, '___sec22'),
('Gradient boosting: Basics with Steepest Descent',
('Gradient boosting: Basics with Steepest Descent/Functional '
'Gradient Descent',
2,
None,
'___sec23'),
'___sec22'),
('The Squared-Error again! Steepest Descent',
2,
None,
'___sec24'),
('Steepest Descent Example', 2, None, '___sec25'),
('Gradient Boosting, algorithm', 2, None, '___sec26'),
'___sec23'),
('Steepest Descent Example', 2, None, '___sec24'),
('Gradient Boosting, algorithm', 2, None, '___sec25'),
('Gradient Boosting, Examples of Regression',
2,
None,
'___sec27'),
'___sec26'),
('Gradient Boosting, Classification Example',
2,
None,
'___sec28'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec29'),
('Regression Case', 2, None, '___sec30'),
('Xgboost on the Cancer Data', 2, None, '___sec31')]}
'___sec27'),
('XGBoost: Extreme Gradient Boosting', 2, None, '___sec28'),
('Regression Case', 2, None, '___sec29'),
('Xgboost on the Cancer Data', 2, None, '___sec30')]}
end of tocinfo -->
<body>
@@ -146,7 +146,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 6, 2020</h4></center> <!-- date -->
<center><h4>Nov 12, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -937,35 +937,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec22">Additive boosting for Regression </h2>
<p>
Here we present <a href="https://pdfs.semanticscholar.org/8d49/e2dedb817f2c3330e74b63c5fc86d2399ce3.pdf" target="_blank">Drucker's AdaBoost</a> tailored for regression.
<p>
In bagging, each training example is equally likely to be
picked. In boosting, the probability of a particular
example being in the training set of a particular machine
depends on the performance of the prior machines on
that example. The following is a modification of
Adaboost by Drucker.
<p>
Start by selecting a set of training data \( n \) and assign to each entry a weight \( w_i=1 \) for \( i=1,2,\dots,n \). As we have done earlier, we could pick say \( 80\% \) of the data set for training. The algorithm runs as follows:
<ol>
<li> We define the probability that the training sample \( i \) is in the set by \( p_i = w_i/\sum_iw_i \). We pick \( n \) samples (with replacement) to form our training set. We pick a number uniformly in the range \( [0,\sum_iw_i] \).</li>
<li> We choose then a regression machine (for example plain linear regression or a simple decision tree). A given regression machine makes then a hypothesis.</li>
<li> Using every member of the training set with the chosen regression machine we obtain then a prediction \( \tilde{y}_i \).</li>
<li> We calculate then the loss function \( L_i \) for each training sample. We can use various types of loss function as long as we have a value</li>
</ol>
\( L_i\in [0,1] \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">Gradient boosting: Basics with Steepest Descent </h2>
<h2 id="___sec22">Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent </h2>
<p>
Gradient boosting is again a similar technique to Adaptive boosting,
@@ -980,7 +952,7 @@ function was the least squares function.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">The Squared-Error again! Steepest Descent </h2>
<h2 id="___sec23">The Squared-Error again! Steepest Descent </h2>
<p>
We start again with our cost function \( {\cal C}(\boldsymbol{y}m\boldsymbol{f})=\sum_{i=0}^{n-1}{\cal L}(y_i, f(x_i)) \) where we want to minimize
@@ -1015,7 +987,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">Steepest Descent Example </h2>
<h2 id="___sec24">Steepest Descent Example </h2>
<p>
Optimizing with respect to \( \rho \) we obtain (taking the derivative) that \( \rho_1 = -1/2 \). We have then that
@@ -1033,7 +1005,7 @@ and find a new value for \( \rho_2=-1/2 \) and continue till we have reached \(
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">Gradient Boosting, algorithm </h2>
<h2 id="___sec25">Gradient Boosting, algorithm </h2>
<p>
Steepest descent is however not much used, since it only optimizes \( f \) at a fixed set of \( n \) points,
@@ -1056,15 +1028,15 @@ The way we proceed in an iterative fashion is to
<ol type="a"></li>
<li> compute the negative gradient vector \( \boldsymbol{u}_m = -\partial C(\boldsymbol{y},\boldsymbol{f})/\partial \boldsymbol{f}(x) \) at \( f(x) = f_{m-1}(x) \);</li>
<li> fit the so-called base-learner to the negative gradient \( h_m(u_m,x) \);</li>
<li> update the estimate \( f_m(x) = f_{m-1}(x)+\nu h_m(u_m,x) \);</li>
<li> update the estimate \( f_m(x) = f_{m-1}(x)+h_m(u_m,x) \);</li>
</ol>
<li> The final estimate is then \( f_M(x) = \sum_{m=1}^M\nu h_m(u_m,x) \).</li>
<li> The final estimate is then \( f_M(x) = \sum_{m=1}^M h_m(u_m,x) \).</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec27">Gradient Boosting, Examples of Regression </h2>
<h2 id="___sec26">Gradient Boosting, Examples of Regression </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -1118,7 +1090,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec28">Gradient Boosting, Classification Example </h2>
<h2 id="___sec27">Gradient Boosting, Classification Example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -1166,7 +1138,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec29">XGBoost: Extreme Gradient Boosting </h2>
<h2 id="___sec28">XGBoost: Extreme Gradient Boosting </h2>
<p>
<a href="https://github.com/dmlc/xgboost" target="_blank">XGBoost</a> or Extreme Gradient
@@ -1187,7 +1159,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec30">Regression Case </h2>
<h2 id="___sec29">Regression Case </h2>
<p>
@@ -1242,7 +1214,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec31">Xgboost on the Cancer Data </h2>
<h2 id="___sec30">Xgboost on the Cancer Data </h2>
<p>
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
doc/pub/week45/ipynb/ipynb-week45-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week45/ipynb/week45.ipynb
+65 -565
View File
File diff suppressed because one or more lines are too long
doc/src/week45/week45.do.txt
+3 -21
View File
@@ -734,27 +734,9 @@ plt.show()
!ec
!split
===== Additive boosting for Regression =====
Here we present "Drucker's AdaBoost":"https://pdfs.semanticscholar.org/8d49/e2dedb817f2c3330e74b63c5fc86d2399ce3.pdf" tailored for regression.
In bagging, each training example is equally likely to be
picked. In boosting, the probability of a particular
example being in the training set of a particular machine
depends on the performance of the prior machines on
that example. The following is a modification of
Adaboost by Drucker.
Start by selecting a set of training data $n$ and assign to each entry a weight $w_i=1$ for $i=1,2,\dots,n$. As we have done earlier, we could pick say $80\%$ of the data set for training. The algorithm runs as follows:
o We define the probability that the training sample $i$ is in the set by $p_i = w_i/\sum_iw_i$. We pick $n$ samples (with replacement) to form our training set. We pick a number uniformly in the range $[0,\sum_iw_i]$.
o We choose then a regression machine (for example plain linear regression or a simple decision tree). A given regression machine makes then a hypothesis.
o Using every member of the training set with the chosen regression machine we obtain then a prediction $\tilde{y}_i$.
o We calculate then the loss function $L_i$ for each training sample. We can use various types of loss function as long as we have a value
$L_i\in [0,1]$.
!split
===== Gradient boosting: Basics with Steepest Descent =====
===== Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent =====
Gradient boosting is again a similar technique to Adaptive boosting,
it combines so-called weak classifiers or regressors into a strong
@@ -836,8 +818,8 @@ o Initialize our estimate $f_0(x)$.
o For $m=1:M$, we
o compute the negative gradient vector $\bm{u}_m = -\partial C(\bm{y},\bm{f})/\partial \bm{f}(x)$ at $f(x) = f_{m-1}(x)$;
o fit the so-called base-learner to the negative gradient $h_m(u_m,x)$;
o update the estimate $f_m(x) = f_{m-1}(x)+\nu h_m(u_m,x)$;
o The final estimate is then $f_M(x) = \sum_{m=1}^M\nu h_m(u_m,x)$.
o update the estimate $f_m(x) = f_{m-1}(x)+h_m(u_m,x)$;
o The final estimate is then $f_M(x) = \sum_{m=1}^M h_m(u_m,x)$.