+We optimize \( \beta \) and \( G \) for each value of \( m=1:M \) as we did in the regression case. +This is normally done in two steps. Let us however first rewrite the cost function as + +$$ +C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{-(y_i\beta G(x_i))}, +$$ + +where we have defined \( w_i^m= \exp{-(y_if_{m-1}(x_i))} \). +
@@ -294,7 +306,7 @@ $$
-The algorithm here is rather straightforward. Assume that our weak -classifier is a decision tree and we consider a binary set of outputs -with \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of -observations. Our design matrix is given in terms of the -feature/predictor vectors -\( \boldsymbol{X}=[\boldsymbol{x}_0\boldsymbol{x}_1\dots\boldsymbol{x}_{p-1} \). Finally, we define also a -classifier determined by our data via a function \( G(\boldsymbol{X}) \). This function tells us how well we are able to classify our outputs/targets \( \boldsymbol{y} \). - -
-We can then define the misclassification error \( \mathrm{err} \) as +First, for any \( \beta > 0 \), we optimize \( G \) by setting $$ -\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(\boldsymbol{X}_{i*}), +G_m(x) = \mathrm{sign} \sum_{i=0}^{n-1} w_i^m I(y_i \ne G_(x_i)), $$ -where the function \( I() \) is one if we misclassify and zero if we classify correctly. +which is the classifier that minimizes the weighted error rate in predicting \( y \).
+We can do this by rewriting +$$ +\exp{-\beta}\sum_{y_i=G(x_i)}w_i^m+\exp{\beta}\sum_{y_i\ne G(x_i)}w_i^m, +$$ + +which can be rewritten as +$$ +(\exp{\beta}-\exp{-\beta})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{-\beta}\sum_{i=0}^{n-1}w_i^m=0, +$$ + +which leads to +$$ +\beta_m = frac{1}{2}\log{\frac{1-\mathrm{\overline{err}}}{\mathrm{\overline{err}}}}, +$$ +
-With the above definitions we are now ready to set up the algorithm for AdaBoost. -The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases. - -
+We can then define the misclassification error \( \mathrm{err} \) as $$ -\mathrm{err}=\frac{\sum_{i=0}^{n-1}w_iI(y_i\ne G(\boldsymbol{X}_{i*})}{\sum_{i=0}^{n-1}w_i}, +\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(\boldsymbol{X}_{i*}), $$ - -
@@ -314,6 +298,7 @@ observations that are missed in the previous iterations.
-Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here. +With the above definitions we are now ready to set up the algorithm for AdaBoost. +The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases. -
+
from sklearn.ensemble import AdaBoostClassifier
+$$
+\mathrm{err}=\frac{\sum_{i=0}^{n-1}w_iI(y_i\ne G(\boldsymbol{X}_{i*})}{\sum_{i=0}^{n-1}w_i},
+$$
-ada_clf = AdaBoostClassifier(
- DecisionTreeClassifier(max_depth=1), n_estimators=200,
- algorithm="SAMME.R", learning_rate=0.5, random_state=42)
-ada_clf.fit(X_train, y_train)
-from sklearn.ensemble import AdaBoostClassifier
+
+- Then we start looping over all attempts at classifying, namely we start an iterative process for \( m=1:M \), where \( M \) is the final number of classifications. Our given classifier could for example be a plain decision tree.
+
+
+ - Fit then a given classifier to the training using the weights \( w_i \).
+ - Compute then \( \mathrm{err} \) and figure out which events are classified properly and which are classified wrongly.
+ - Define a quantity \( \alpha_{m} = \log{(1-\mathrm{err})/\mathrm{err}} \)
+ - Set the new weights to \( w_i = w_i\times \exp{(\alpha_m I(y_i\ne G(\boldsymbol{X}_{i*})} \).
+
+
+
@@ -306,6 +315,7 @@ plt.show()
-Gradient boosting is again a similar technique to Adapative boosting, -it combines so-called weak classifiers or regressors into a strong -method via a series of iterations. +Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here.
-In order to understand the method, let us illustrate its basics by -bringing back the essential steps in linear regression, where our cost -function was the least squares function. + +
from sklearn.ensemble import AdaBoostClassifier
+
+ada_clf = AdaBoostClassifier(
+ DecisionTreeClassifier(max_depth=1), n_estimators=200,
+ algorithm="SAMME.R", learning_rate=0.5, random_state=42)
+ada_clf.fit(X_train, y_train)
+
+from sklearn.ensemble import AdaBoostClassifier
+
+ada_clf = AdaBoostClassifier(
+ DecisionTreeClassifier(max_depth=1), n_estimators=200,
+ algorithm="SAMME.R", learning_rate=0.5, random_state=42)
+ada_clf.fit(X_train_scaled, y_train)
+y_pred = ada_clf.predict(X_test_scaled)
+skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
+plt.show()
+y_probas = ada_clf.predict_proba(X_test_scaled)
+skplt.metrics.plot_roc(y_test, y_probas)
+plt.show()
+skplt.metrics.plot_cumulative_gain(y_test, y_probas)
+plt.show()
+
@@ -287,6 +307,7 @@ function was the least squares function.
-Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard least squares function -$$ -C(\boldsymbol{y},\boldsymbol{f})=\frac{1}{n}\sum_{i=0}^{n-1}(y_i-f(x_i))^2. -$$ +Gradient boosting is again a similar technique to Adapative boosting, +it combines so-called weak classifiers or regressors into a strong +method via a series of iterations.
-The way we proceed in an iterative fashion is to - -
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs055.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs055.html index d807e85cf..ea2668395 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs055.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs055.html @@ -129,19 +129,20 @@ Automatically generated HTML file from DocOnce source None, '___sec47'), ('Adaptive Boosting, AdaBoost', 2, None, '___sec48'), + ('Building up AdaBoost', 2, None, '___sec49'), ('Adaptive boosting: AdaBoost, Basic Algorithm', 2, None, - '___sec49'), - ('Basic Steps of AdaBoost', 2, None, '___sec50'), - ('AdaBoost Examples', 2, None, '___sec51'), - ('Gradient boosting: Basics', 2, None, '___sec52'), - ('Gradient Boosting, algorithm', 2, None, '___sec53'), - ('Gradient Boosting, Examples', 2, None, '___sec54'), - ('Gradient Boots with Early Stopping', 2, None, '___sec55'), - ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec56'), - ('Regression Case', 2, None, '___sec57'), - ('Xgboost on the Cancer Data', 2, None, '___sec58')]} + '___sec50'), + ('Basic Steps of AdaBoost', 2, None, '___sec51'), + ('AdaBoost Examples', 2, None, '___sec52'), + ('Gradient boosting: Basics', 2, None, '___sec53'), + ('Gradient Boosting, algorithm', 2, None, '___sec54'), + ('Gradient Boosting, Examples', 2, None, '___sec55'), + ('Gradient Boots with Early Stopping', 2, None, '___sec56'), + ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec57'), + ('Regression Case', 2, None, '___sec58'), + ('Xgboost on the Cancer Data', 2, None, '___sec59')]} end of tocinfo --> @@ -228,16 +229,17 @@ MathJax.Hub.Config({
+Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard least squares function +$$ +C(\boldsymbol{y},\boldsymbol{f})=\frac{1}{n}\sum_{i=0}^{n-1}(y_i-f(x_i))^2. +$$ - -
np.random.seed(42)
-X = np.random.rand(100, 1) - 0.5
-y = 3*X[:, 0]**2 + 0.05 * np.random.randn(100)
-
-from sklearn.tree import DecisionTreeRegressor
-
-tree_reg1 = DecisionTreeRegressor(max_depth=2, random_state=42)
-tree_reg1.fit(X, y)
-
-y2 = y - tree_reg1.predict(X)
-tree_reg2 = DecisionTreeRegressor(max_depth=2, random_state=42)
-tree_reg2.fit(X, y2)
-
-y3 = y2 - tree_reg2.predict(X)
-tree_reg3 = DecisionTreeRegressor(max_depth=2, random_state=42)
-tree_reg3.fit(X, y3)
-
-X_new = np.array([[0.8]])
-y_pred = sum(tree.predict(X_new) for tree in (tree_reg1, tree_reg2, tree_reg3))
-
-def plot_predictions(regressors, X, y, axes, label=None, style="r-", data_style="b.", data_label=None):
- x1 = np.linspace(axes[0], axes[1], 500)
- y_pred = sum(regressor.predict(x1.reshape(-1, 1)) for regressor in regressors)
- plt.plot(X[:, 0], y, data_style, label=data_label)
- plt.plot(x1, y_pred, style, linewidth=2, label=label)
- if label or data_label:
- plt.legend(loc="upper center", fontsize=16)
- plt.axis(axes)
-
-plt.figure(figsize=(11,11))
-
-plt.subplot(321)
-plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h_1(x_1)$", style="g-", data_label="Training set")
-plt.ylabel("$y$", fontsize=16, rotation=0)
-plt.title("Residuals and tree predictions", fontsize=16)
-
-plt.subplot(322)
-plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h(x_1) = h_1(x_1)$", data_label="Training set")
-plt.ylabel("$y$", fontsize=16, rotation=0)
-plt.title("Ensemble predictions", fontsize=16)
-
-plt.subplot(323)
-plot_predictions([tree_reg2], X, y2, axes=[-0.5, 0.5, -0.5, 0.5], label="$h_2(x_1)$", style="g-", data_style="k+", data_label="Residuals")
-plt.ylabel("$y - h_1(x_1)$", fontsize=16)
-
-plt.subplot(324)
-plot_predictions([tree_reg1, tree_reg2], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h(x_1) = h_1(x_1) + h_2(x_1)$")
-plt.ylabel("$y$", fontsize=16, rotation=0)
-
-plt.subplot(325)
-plot_predictions([tree_reg3], X, y3, axes=[-0.5, 0.5, -0.5, 0.5], label="$h_3(x_1)$", style="g-", data_style="k+")
-plt.ylabel("$y - h_1(x_1) - h_2(x_1)$", fontsize=16)
-plt.xlabel("$x_1$", fontsize=16)
-
-plt.subplot(326)
-plot_predictions([tree_reg1, tree_reg2, tree_reg3], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h(x_1) = h_1(x_1) + h_2(x_1) + h_3(x_1)$")
-plt.xlabel("$x_1$", fontsize=16)
-plt.ylabel("$y$", fontsize=16, rotation=0)
-
-save_fig("gradient_boosting_plot")
-plt.show()
-
-from sklearn.ensemble import GradientBoostingRegressor
-
-gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=3, learning_rate=1.0, random_state=42)
-gbrt.fit(X, y)
-
-gbrt_slow = GradientBoostingRegressor(max_depth=2, n_estimators=200, learning_rate=0.1, random_state=42)
-gbrt_slow.fit(X, y)
-
-plt.figure(figsize=(11,4))
-
-plt.subplot(121)
-plot_predictions([gbrt], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="Ensemble predictions")
-plt.title("learning_rate={}, n_estimators={}".format(gbrt.learning_rate, gbrt.n_estimators), fontsize=14)
-
-plt.subplot(122)
-plot_predictions([gbrt_slow], X, y, axes=[-0.5, 0.5, -0.1, 0.8])
-plt.title("learning_rate={}, n_estimators={}".format(gbrt_slow.learning_rate, gbrt_slow.n_estimators), fontsize=14)
-
-save_fig("gbrt_learning_rate_plot")
-plt.show()
-+The way we proceed in an iterative fashion is to + +
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs056.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs056.html index fd4fc52b9..665e91b5d 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs056.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs056.html @@ -129,19 +129,20 @@ Automatically generated HTML file from DocOnce source None, '___sec47'), ('Adaptive Boosting, AdaBoost', 2, None, '___sec48'), + ('Building up AdaBoost', 2, None, '___sec49'), ('Adaptive boosting: AdaBoost, Basic Algorithm', 2, None, - '___sec49'), - ('Basic Steps of AdaBoost', 2, None, '___sec50'), - ('AdaBoost Examples', 2, None, '___sec51'), - ('Gradient boosting: Basics', 2, None, '___sec52'), - ('Gradient Boosting, algorithm', 2, None, '___sec53'), - ('Gradient Boosting, Examples', 2, None, '___sec54'), - ('Gradient Boots with Early Stopping', 2, None, '___sec55'), - ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec56'), - ('Regression Case', 2, None, '___sec57'), - ('Xgboost on the Cancer Data', 2, None, '___sec58')]} + '___sec50'), + ('Basic Steps of AdaBoost', 2, None, '___sec51'), + ('AdaBoost Examples', 2, None, '___sec52'), + ('Gradient boosting: Basics', 2, None, '___sec53'), + ('Gradient Boosting, algorithm', 2, None, '___sec54'), + ('Gradient Boosting, Examples', 2, None, '___sec55'), + ('Gradient Boots with Early Stopping', 2, None, '___sec56'), + ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec57'), + ('Regression Case', 2, None, '___sec58'), + ('Xgboost on the Cancer Data', 2, None, '___sec59')]} end of tocinfo --> @@ -228,16 +229,17 @@ MathJax.Hub.Config({
-
from sklearn.model_selection import train_test_split
-from sklearn.metrics import mean_squared_error
+np.random.seed(42)
+X = np.random.rand(100, 1) - 0.5
+y = 3*X[:, 0]**2 + 0.05 * np.random.randn(100)
-X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=49)
+from sklearn.tree import DecisionTreeRegressor
-gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=120, random_state=42)
-gbrt.fit(X_train, y_train)
+tree_reg1 = DecisionTreeRegressor(max_depth=2, random_state=42)
+tree_reg1.fit(X, y)
-errors = [mean_squared_error(y_val, y_pred)
- for y_pred in gbrt.staged_predict(X_val)]
-bst_n_estimators = np.argmin(errors) + 1
+y2 = y - tree_reg1.predict(X)
+tree_reg2 = DecisionTreeRegressor(max_depth=2, random_state=42)
+tree_reg2.fit(X, y2)
-gbrt_best = GradientBoostingRegressor(max_depth=2,n_estimators=bst_n_estimators, random_state=42)
-gbrt_best.fit(X_train, y_train)
+y3 = y2 - tree_reg2.predict(X)
+tree_reg3 = DecisionTreeRegressor(max_depth=2, random_state=42)
+tree_reg3.fit(X, y3)
-min_error = np.min(errors)
-plt.figure(figsize=(11, 4))
+X_new = np.array([[0.8]])
+y_pred = sum(tree.predict(X_new) for tree in (tree_reg1, tree_reg2, tree_reg3))
-plt.subplot(121)
-plt.plot(errors, "b.-")
-plt.plot([bst_n_estimators, bst_n_estimators], [0, min_error], "k--")
-plt.plot([0, 120], [min_error, min_error], "k--")
-plt.plot(bst_n_estimators, min_error, "ko")
-plt.text(bst_n_estimators, min_error*1.2, "Minimum", ha="center", fontsize=14)
-plt.axis([0, 120, 0, 0.01])
-plt.xlabel("Number of trees")
-plt.title("Validation error", fontsize=14)
+def plot_predictions(regressors, X, y, axes, label=None, style="r-", data_style="b.", data_label=None):
+ x1 = np.linspace(axes[0], axes[1], 500)
+ y_pred = sum(regressor.predict(x1.reshape(-1, 1)) for regressor in regressors)
+ plt.plot(X[:, 0], y, data_style, label=data_label)
+ plt.plot(x1, y_pred, style, linewidth=2, label=label)
+ if label or data_label:
+ plt.legend(loc="upper center", fontsize=16)
+ plt.axis(axes)
-plt.subplot(122)
-plot_predictions([gbrt_best], X, y, axes=[-0.5, 0.5, -0.1, 0.8])
-plt.title("Best model (%d trees)" % bst_n_estimators, fontsize=14)
+plt.figure(figsize=(11,11))
-save_fig("early_stopping_gbrt_plot")
+plt.subplot(321)
+plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h_1(x_1)$", style="g-", data_label="Training set")
+plt.ylabel("$y$", fontsize=16, rotation=0)
+plt.title("Residuals and tree predictions", fontsize=16)
+
+plt.subplot(322)
+plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h(x_1) = h_1(x_1)$", data_label="Training set")
+plt.ylabel("$y$", fontsize=16, rotation=0)
+plt.title("Ensemble predictions", fontsize=16)
+
+plt.subplot(323)
+plot_predictions([tree_reg2], X, y2, axes=[-0.5, 0.5, -0.5, 0.5], label="$h_2(x_1)$", style="g-", data_style="k+", data_label="Residuals")
+plt.ylabel("$y - h_1(x_1)$", fontsize=16)
+
+plt.subplot(324)
+plot_predictions([tree_reg1, tree_reg2], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h(x_1) = h_1(x_1) + h_2(x_1)$")
+plt.ylabel("$y$", fontsize=16, rotation=0)
+
+plt.subplot(325)
+plot_predictions([tree_reg3], X, y3, axes=[-0.5, 0.5, -0.5, 0.5], label="$h_3(x_1)$", style="g-", data_style="k+")
+plt.ylabel("$y - h_1(x_1) - h_2(x_1)$", fontsize=16)
+plt.xlabel("$x_1$", fontsize=16)
+
+plt.subplot(326)
+plot_predictions([tree_reg1, tree_reg2, tree_reg3], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h(x_1) = h_1(x_1) + h_2(x_1) + h_3(x_1)$")
+plt.xlabel("$x_1$", fontsize=16)
+plt.ylabel("$y$", fontsize=16, rotation=0)
+
+save_fig("gradient_boosting_plot")
plt.show()
+from sklearn.ensemble import GradientBoostingRegressor
-gbrt = GradientBoostingRegressor(max_depth=2, warm_start=True, random_state=42)
+gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=3, learning_rate=1.0, random_state=42)
+gbrt.fit(X, y)
-min_val_error = float("inf")
-error_going_up = 0
-for n_estimators in range(1, 120):
- gbrt.n_estimators = n_estimators
- gbrt.fit(X_train, y_train)
- y_pred = gbrt.predict(X_val)
- val_error = mean_squared_error(y_val, y_pred)
- if val_error < min_val_error:
- min_val_error = val_error
- error_going_up = 0
- else:
- error_going_up += 1
- if error_going_up == 5:
- break # early stopping
+gbrt_slow = GradientBoostingRegressor(max_depth=2, n_estimators=200, learning_rate=0.1, random_state=42)
+gbrt_slow.fit(X, y)
+plt.figure(figsize=(11,4))
-print(gbrt.n_estimators)
-print("Minimum validation MSE:", min_val_error)
+plt.subplot(121)
+plot_predictions([gbrt], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="Ensemble predictions")
+plt.title("learning_rate={}, n_estimators={}".format(gbrt.learning_rate, gbrt.n_estimators), fontsize=14)
+
+plt.subplot(122)
+plot_predictions([gbrt_slow], X, y, axes=[-0.5, 0.5, -0.1, 0.8])
+plt.title("learning_rate={}, n_estimators={}".format(gbrt_slow.learning_rate, gbrt_slow.n_estimators), fontsize=14)
+
+save_fig("gbrt_learning_rate_plot")
+plt.show()
@@ -333,6 +361,7 @@ error_going_up = 58
59
60
+ 61
»
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html
index 14c41a09d..203990494 100644
--- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html
+++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html
@@ -129,19 +129,20 @@ Automatically generated HTML file from DocOnce source
None,
'___sec47'),
('Adaptive Boosting, AdaBoost', 2, None, '___sec48'),
+ ('Building up AdaBoost', 2, None, '___sec49'),
('Adaptive boosting: AdaBoost, Basic Algorithm',
2,
None,
- '___sec49'),
- ('Basic Steps of AdaBoost', 2, None, '___sec50'),
- ('AdaBoost Examples', 2, None, '___sec51'),
- ('Gradient boosting: Basics', 2, None, '___sec52'),
- ('Gradient Boosting, algorithm', 2, None, '___sec53'),
- ('Gradient Boosting, Examples', 2, None, '___sec54'),
- ('Gradient Boots with Early Stopping', 2, None, '___sec55'),
- ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec56'),
- ('Regression Case', 2, None, '___sec57'),
- ('Xgboost on the Cancer Data', 2, None, '___sec58')]}
+ '___sec50'),
+ ('Basic Steps of AdaBoost', 2, None, '___sec51'),
+ ('AdaBoost Examples', 2, None, '___sec52'),
+ ('Gradient boosting: Basics', 2, None, '___sec53'),
+ ('Gradient Boosting, algorithm', 2, None, '___sec54'),
+ ('Gradient Boosting, Examples', 2, None, '___sec55'),
+ ('Gradient Boots with Early Stopping', 2, None, '___sec56'),
+ ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec57'),
+ ('Regression Case', 2, None, '___sec58'),
+ ('Xgboost on the Cancer Data', 2, None, '___sec59')]}
end of tocinfo -->
@@ -228,16 +229,17 @@ MathJax.Hub.Config({
Iterative Fitting, Regression and Squared-error Cost Function
Iterative Fitting, Classification, AdaBoost
Adaptive Boosting, AdaBoost
- Adaptive boosting: AdaBoost, Basic Algorithm
- Basic Steps of AdaBoost
- AdaBoost Examples
- Gradient boosting: Basics
- Gradient Boosting, algorithm
- Gradient Boosting, Examples
- Gradient Boots with Early Stopping
- XGBoost: Extreme Gradient Boosting
- Regression Case
- Xgboost on the Cancer Data
+ Building up AdaBoost
+ Adaptive boosting: AdaBoost, Basic Algorithm
+ Basic Steps of AdaBoost
+ AdaBoost Examples
+ Gradient boosting: Basics
+ Gradient Boosting, algorithm
+ Gradient Boosting, Examples
+ Gradient Boots with Early Stopping
+ XGBoost: Extreme Gradient Boosting
+ Regression Case
+ Xgboost on the Cancer Data
@@ -253,24 +255,67 @@ MathJax.Hub.Config({
-XGBoost: Extreme Gradient Boosting
-
+Gradient Boots with Early Stopping
-XGBoost or Extreme Gradient
-Boosting, is an optimized distributed gradient boosting library
-designed to be highly efficient, flexible and portable. It implements
-machine learning algorithms under the Gradient Boosting
-framework. XGBoost provides a parallel tree boosting that solve many
-data science problems in a fast and accurate way. See the article by Chen and Guestrin.
-
-The authors design and build a highly scalable end-to-end tree
-boosting system. It has a theoretically justified weighted quantile
-sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning.
+
+
from sklearn.model_selection import train_test_split
+from sklearn.metrics import mean_squared_error
-
-It is now the algorithm which wins essentially all ML competitions!!!
+X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=49)
+gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=120, random_state=42)
+gbrt.fit(X_train, y_train)
+
+errors = [mean_squared_error(y_val, y_pred)
+ for y_pred in gbrt.staged_predict(X_val)]
+bst_n_estimators = np.argmin(errors) + 1
+
+gbrt_best = GradientBoostingRegressor(max_depth=2,n_estimators=bst_n_estimators, random_state=42)
+gbrt_best.fit(X_train, y_train)
+
+min_error = np.min(errors)
+plt.figure(figsize=(11, 4))
+
+plt.subplot(121)
+plt.plot(errors, "b.-")
+plt.plot([bst_n_estimators, bst_n_estimators], [0, min_error], "k--")
+plt.plot([0, 120], [min_error, min_error], "k--")
+plt.plot(bst_n_estimators, min_error, "ko")
+plt.text(bst_n_estimators, min_error*1.2, "Minimum", ha="center", fontsize=14)
+plt.axis([0, 120, 0, 0.01])
+plt.xlabel("Number of trees")
+plt.title("Validation error", fontsize=14)
+
+plt.subplot(122)
+plot_predictions([gbrt_best], X, y, axes=[-0.5, 0.5, -0.1, 0.8])
+plt.title("Best model (%d trees)" % bst_n_estimators, fontsize=14)
+
+save_fig("early_stopping_gbrt_plot")
+plt.show()
+
+
+gbrt = GradientBoostingRegressor(max_depth=2, warm_start=True, random_state=42)
+
+min_val_error = float("inf")
+error_going_up = 0
+for n_estimators in range(1, 120):
+ gbrt.n_estimators = n_estimators
+ gbrt.fit(X_train, y_train)
+ y_pred = gbrt.predict(X_val)
+ val_error = mean_squared_error(y_val, y_pred)
+ if val_error < min_val_error:
+ min_val_error = val_error
+ error_going_up = 0
+ else:
+ error_going_up += 1
+ if error_going_up == 5:
+ break # early stopping
+
+
+print(gbrt.n_estimators)
+print("Minimum validation MSE:", min_val_error)
+
@@ -289,6 +334,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
58
59
60
+ 61
»
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees-bs.html b/doc/pub/DecisionTrees/html/DecisionTrees-bs.html
index 2c5b1c41f..4ad69c7e8 100644
--- a/doc/pub/DecisionTrees/html/DecisionTrees-bs.html
+++ b/doc/pub/DecisionTrees/html/DecisionTrees-bs.html
@@ -129,19 +129,20 @@ Automatically generated HTML file from DocOnce source
None,
'___sec47'),
('Adaptive Boosting, AdaBoost', 2, None, '___sec48'),
+ ('Building up AdaBoost', 2, None, '___sec49'),
('Adaptive boosting: AdaBoost, Basic Algorithm',
2,
None,
- '___sec49'),
- ('Basic Steps of AdaBoost', 2, None, '___sec50'),
- ('AdaBoost Examples', 2, None, '___sec51'),
- ('Gradient boosting: Basics', 2, None, '___sec52'),
- ('Gradient Boosting, algorithm', 2, None, '___sec53'),
- ('Gradient Boosting, Examples', 2, None, '___sec54'),
- ('Gradient Boots with Early Stopping', 2, None, '___sec55'),
- ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec56'),
- ('Regression Case', 2, None, '___sec57'),
- ('Xgboost on the Cancer Data', 2, None, '___sec58')]}
+ '___sec50'),
+ ('Basic Steps of AdaBoost', 2, None, '___sec51'),
+ ('AdaBoost Examples', 2, None, '___sec52'),
+ ('Gradient boosting: Basics', 2, None, '___sec53'),
+ ('Gradient Boosting, algorithm', 2, None, '___sec54'),
+ ('Gradient Boosting, Examples', 2, None, '___sec55'),
+ ('Gradient Boots with Early Stopping', 2, None, '___sec56'),
+ ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec57'),
+ ('Regression Case', 2, None, '___sec58'),
+ ('Xgboost on the Cancer Data', 2, None, '___sec59')]}
end of tocinfo -->
@@ -228,16 +229,17 @@ MathJax.Hub.Config({
Iterative Fitting, Regression and Squared-error Cost Function
Iterative Fitting, Classification, AdaBoost
Adaptive Boosting, AdaBoost
- Adaptive boosting: AdaBoost, Basic Algorithm
- Basic Steps of AdaBoost
- AdaBoost Examples
- Gradient boosting: Basics
- Gradient Boosting, algorithm
- Gradient Boosting, Examples
- Gradient Boots with Early Stopping
- XGBoost: Extreme Gradient Boosting
- Regression Case
- Xgboost on the Cancer Data
+ Building up AdaBoost
+ Adaptive boosting: AdaBoost, Basic Algorithm
+ Basic Steps of AdaBoost
+ AdaBoost Examples
+ Gradient boosting: Basics
+ Gradient Boosting, algorithm
+ Gradient Boosting, Examples
+ Gradient Boots with Early Stopping
+ XGBoost: Extreme Gradient Boosting
+ Regression Case
+ Xgboost on the Cancer Data
@@ -296,7 +298,7 @@ MathJax.Hub.Config({
9
10
...
- 60
+ 61
»
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees-reveal.html b/doc/pub/DecisionTrees/html/DecisionTrees-reveal.html
index b36ba49b9..e9b1c112e 100644
--- a/doc/pub/DecisionTrees/html/DecisionTrees-reveal.html
+++ b/doc/pub/DecisionTrees/html/DecisionTrees-reveal.html
@@ -2026,9 +2026,9 @@ The way we proceed is as follows (here we specialize to the squared-error cost f
For \( m=1:M \)
- minmize $\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2 wrt \( \gamma \) and $\beta$$
+ minimize \( \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2 \) wrt \( \gamma \) and \( \beta \)
This gives the optimial values \( \beta_m \) and \( \gamma_m \)
- Determine then the new values $f_m(x)=f_{m-1}(x) +\beta_m b(x;\gamma_m)
+ Determine then the new values \( f_m(x)=f_{m-1}(x) +\beta_m b(x;\gamma_m) \)
@@ -2094,14 +2094,63 @@ The simplest possible cost function which leads (also simple from a computationa
exponential cost/loss function defined as
$$
-C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1(x_i)+\beta G(x_i})}
+C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1}(x_i)+\beta G(x_i))}.
+$$
+
+
+
+We optimize \( \beta \) and \( G \) for each value of \( m=1:M \) as we did in the regression case.
+This is normally done in two steps. Let us however first rewrite the cost function as
+
+
+$$
+C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{-(y_i\beta G(x_i))},
+$$
+
+
+where we have defined \( w_i^m= \exp{-(y_if_{m-1}(x_i))} \).
+
+
+
+
+Building up AdaBoost
+
+
+First, for any \( \beta > 0 \), we optimize \( G \) by setting
+
+$$
+G_m(x) = \mathrm{sign} \sum_{i=0}^{n-1} w_i^m I(y_i \ne G_(x_i)),
+$$
+
+
+which is the classifier that minimizes the weighted error rate in predicting \( y \).
+
+
+We can do this by rewriting
+
+$$
+\exp{-\beta}\sum_{y_i=G(x_i)}w_i^m+\exp{\beta}\sum_{y_i\ne G(x_i)}w_i^m,
+$$
+
+
+which can be rewritten as
+
+$$
+(\exp{\beta}-\exp{-\beta})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{-\beta}\sum_{i=0}^{n-1}w_i^m=0,
+$$
+
+
+which leads to
+
+$$
+\beta_m = frac{1}{2}\log{\frac{1-\mathrm{\overline{err}}}{\mathrm{\overline{err}}}},
$$
-Adaptive boosting: AdaBoost, Basic Algorithm
+Adaptive boosting: AdaBoost, Basic Algorithm
The algorithm here is rather straightforward. Assume that our weak
@@ -2125,7 +2174,7 @@ where the function \( I() \) is one if we misclassify and zero if we classify co
-Basic Steps of AdaBoost
+Basic Steps of AdaBoost
With the above definitions we are now ready to set up the algorithm for AdaBoost.
@@ -2166,7 +2215,7 @@ observations that are missed in the previous iterations.
-AdaBoost Examples
+AdaBoost Examples
Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here.
@@ -2200,7 +2249,7 @@ plt.show()
-Gradient boosting: Basics
+Gradient boosting: Basics
Gradient boosting is again a similar technique to Adapative boosting,
@@ -2215,7 +2264,7 @@ function was the least squares function.
-Gradient Boosting, algorithm
+Gradient Boosting, algorithm
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard least squares function
@@ -2243,7 +2292,7 @@ The way we proceed in an iterative fashion is to
-Gradient Boosting, Examples
+Gradient Boosting, Examples
@@ -2334,7 +2383,7 @@ plt.show()
-Gradient Boots with Early Stopping
+Gradient Boots with Early Stopping
@@ -2399,7 +2448,7 @@ error_going_up = 0
-XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
XGBoost or Extreme Gradient
@@ -2420,7 +2469,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
-Regression Case
+Regression Case
@@ -2476,7 +2525,7 @@ plt.show()
-Xgboost on the Cancer Data
+Xgboost on the Cancer Data
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html b/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
index 9b84cf8d4..8a40a0fa1 100644
--- a/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
+++ b/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
@@ -149,19 +149,20 @@ div { text-align: justify; text-justify: inter-word; }
None,
'___sec47'),
('Adaptive Boosting, AdaBoost', 2, None, '___sec48'),
+ ('Building up AdaBoost', 2, None, '___sec49'),
('Adaptive boosting: AdaBoost, Basic Algorithm',
2,
None,
- '___sec49'),
- ('Basic Steps of AdaBoost', 2, None, '___sec50'),
- ('AdaBoost Examples', 2, None, '___sec51'),
- ('Gradient boosting: Basics', 2, None, '___sec52'),
- ('Gradient Boosting, algorithm', 2, None, '___sec53'),
- ('Gradient Boosting, Examples', 2, None, '___sec54'),
- ('Gradient Boots with Early Stopping', 2, None, '___sec55'),
- ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec56'),
- ('Regression Case', 2, None, '___sec57'),
- ('Xgboost on the Cancer Data', 2, None, '___sec58')]}
+ '___sec50'),
+ ('Basic Steps of AdaBoost', 2, None, '___sec51'),
+ ('AdaBoost Examples', 2, None, '___sec52'),
+ ('Gradient boosting: Basics', 2, None, '___sec53'),
+ ('Gradient Boosting, algorithm', 2, None, '___sec54'),
+ ('Gradient Boosting, Examples', 2, None, '___sec55'),
+ ('Gradient Boots with Early Stopping', 2, None, '___sec56'),
+ ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec57'),
+ ('Regression Case', 2, None, '___sec58'),
+ ('Xgboost on the Cancer Data', 2, None, '___sec59')]}
end of tocinfo -->
@@ -2018,9 +2019,9 @@ The way we proceed is as follows (here we specialize to the squared-error cost f
For \( m=1:M \)
- minmize $\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2 wrt \( \gamma \) and $\beta$$
+ minimize \( \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2 \) wrt \( \gamma \) and \( \beta \)
This gives the optimial values \( \beta_m \) and \( \gamma_m \)
- Determine then the new values $f_m(x)=f_{m-1}(x) +\beta_m b(x;\gamma_m)
+ Determine then the new values \( f_m(x)=f_{m-1}(x) +\beta_m b(x;\gamma_m) \)
@@ -2076,13 +2077,51 @@ $$
The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the
exponential cost/loss function defined as
$$
-C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1(x_i)+\beta G(x_i})}
+C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1}(x_i)+\beta G(x_i))}.
$$
+
+We optimize \( \beta \) and \( G \) for each value of \( m=1:M \) as we did in the regression case.
+This is normally done in two steps. Let us however first rewrite the cost function as
+
+$$
+C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{-(y_i\beta G(x_i))},
+$$
+
+where we have defined \( w_i^m= \exp{-(y_if_{m-1}(x_i))} \).
+
-
Adaptive boosting: AdaBoost, Basic Algorithm
+Building up AdaBoost
+
+
+First, for any \( \beta > 0 \), we optimize \( G \) by setting
+$$
+G_m(x) = \mathrm{sign} \sum_{i=0}^{n-1} w_i^m I(y_i \ne G_(x_i)),
+$$
+
+which is the classifier that minimizes the weighted error rate in predicting \( y \).
+
+
+We can do this by rewriting
+$$
+\exp{-\beta}\sum_{y_i=G(x_i)}w_i^m+\exp{\beta}\sum_{y_i\ne G(x_i)}w_i^m,
+$$
+
+which can be rewritten as
+$$
+(\exp{\beta}-\exp{-\beta})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{-\beta}\sum_{i=0}^{n-1}w_i^m=0,
+$$
+
+which leads to
+$$
+\beta_m = frac{1}{2}\log{\frac{1-\mathrm{\overline{err}}}{\mathrm{\overline{err}}}},
+$$
+
+
+
+
Adaptive boosting: AdaBoost, Basic Algorithm
The algorithm here is rather straightforward. Assume that our weak
@@ -2104,7 +2143,7 @@ where the function \( I() \) is one if we misclassify and zero if we classify co
-
Basic Steps of AdaBoost
+Basic Steps of AdaBoost
With the above definitions we are now ready to set up the algorithm for AdaBoost.
@@ -2144,7 +2183,7 @@ observations that are missed in the previous iterations.
-
AdaBoost Examples
+AdaBoost Examples
Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here.
@@ -2177,7 +2216,7 @@ plt.show()
-
Gradient boosting: Basics
+Gradient boosting: Basics
Gradient boosting is again a similar technique to Adapative boosting,
@@ -2192,7 +2231,7 @@ function was the least squares function.
-
Gradient Boosting, algorithm
+Gradient Boosting, algorithm
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard least squares function
@@ -2218,7 +2257,7 @@ The way we proceed in an iterative fashion is to
-
Gradient Boosting, Examples
+Gradient Boosting, Examples
@@ -2308,7 +2347,7 @@ plt.show()
-
Gradient Boots with Early Stopping
+Gradient Boots with Early Stopping
@@ -2372,7 +2411,7 @@ error_going_up = 0
-
XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
XGBoost or Extreme Gradient
@@ -2393,7 +2432,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
-
Regression Case
+Regression Case
@@ -2448,7 +2487,7 @@ plt.show()
-
Xgboost on the Cancer Data
+Xgboost on the Cancer Data
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees.html b/doc/pub/DecisionTrees/html/DecisionTrees.html
index 42acee4ec..98c64a7c6 100644
--- a/doc/pub/DecisionTrees/html/DecisionTrees.html
+++ b/doc/pub/DecisionTrees/html/DecisionTrees.html
@@ -154,19 +154,20 @@ div { text-align: justify; text-justify: inter-word; }
None,
'___sec47'),
('Adaptive Boosting, AdaBoost', 2, None, '___sec48'),
+ ('Building up AdaBoost', 2, None, '___sec49'),
('Adaptive boosting: AdaBoost, Basic Algorithm',
2,
None,
- '___sec49'),
- ('Basic Steps of AdaBoost', 2, None, '___sec50'),
- ('AdaBoost Examples', 2, None, '___sec51'),
- ('Gradient boosting: Basics', 2, None, '___sec52'),
- ('Gradient Boosting, algorithm', 2, None, '___sec53'),
- ('Gradient Boosting, Examples', 2, None, '___sec54'),
- ('Gradient Boots with Early Stopping', 2, None, '___sec55'),
- ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec56'),
- ('Regression Case', 2, None, '___sec57'),
- ('Xgboost on the Cancer Data', 2, None, '___sec58')]}
+ '___sec50'),
+ ('Basic Steps of AdaBoost', 2, None, '___sec51'),
+ ('AdaBoost Examples', 2, None, '___sec52'),
+ ('Gradient boosting: Basics', 2, None, '___sec53'),
+ ('Gradient Boosting, algorithm', 2, None, '___sec54'),
+ ('Gradient Boosting, Examples', 2, None, '___sec55'),
+ ('Gradient Boots with Early Stopping', 2, None, '___sec56'),
+ ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec57'),
+ ('Regression Case', 2, None, '___sec58'),
+ ('Xgboost on the Cancer Data', 2, None, '___sec59')]}
end of tocinfo -->
@@ -2023,9 +2024,9 @@ The way we proceed is as follows (here we specialize to the squared-error cost f
For \( m=1:M \)
- minmize $\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2 wrt \( \gamma \) and $\beta$$
+ minimize \( \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2 \) wrt \( \gamma \) and \( \beta \)
This gives the optimial values \( \beta_m \) and \( \gamma_m \)
- Determine then the new values $f_m(x)=f_{m-1}(x) +\beta_m b(x;\gamma_m)
+ Determine then the new values \( f_m(x)=f_{m-1}(x) +\beta_m b(x;\gamma_m) \)
@@ -2081,13 +2082,51 @@ $$
The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the
exponential cost/loss function defined as
$$
-C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1(x_i)+\beta G(x_i})}
+C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1}(x_i)+\beta G(x_i))}.
$$
+
+We optimize \( \beta \) and \( G \) for each value of \( m=1:M \) as we did in the regression case.
+This is normally done in two steps. Let us however first rewrite the cost function as
+
+$$
+C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{-(y_i\beta G(x_i))},
+$$
+
+where we have defined \( w_i^m= \exp{-(y_if_{m-1}(x_i))} \).
+
-
Adaptive boosting: AdaBoost, Basic Algorithm
+Building up AdaBoost
+
+
+First, for any \( \beta > 0 \), we optimize \( G \) by setting
+$$
+G_m(x) = \mathrm{sign} \sum_{i=0}^{n-1} w_i^m I(y_i \ne G_(x_i)),
+$$
+
+which is the classifier that minimizes the weighted error rate in predicting \( y \).
+
+
+We can do this by rewriting
+$$
+\exp{-\beta}\sum_{y_i=G(x_i)}w_i^m+\exp{\beta}\sum_{y_i\ne G(x_i)}w_i^m,
+$$
+
+which can be rewritten as
+$$
+(\exp{\beta}-\exp{-\beta})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{-\beta}\sum_{i=0}^{n-1}w_i^m=0,
+$$
+
+which leads to
+$$
+\beta_m = frac{1}{2}\log{\frac{1-\mathrm{\overline{err}}}{\mathrm{\overline{err}}}},
+$$
+
+
+
+
Adaptive boosting: AdaBoost, Basic Algorithm
The algorithm here is rather straightforward. Assume that our weak
@@ -2109,7 +2148,7 @@ where the function \( I() \) is one if we misclassify and zero if we classify co
-
Basic Steps of AdaBoost
+Basic Steps of AdaBoost
With the above definitions we are now ready to set up the algorithm for AdaBoost.
@@ -2149,7 +2188,7 @@ observations that are missed in the previous iterations.
-
AdaBoost Examples
+AdaBoost Examples
Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here.
@@ -2182,7 +2221,7 @@ plt.show()
-
Gradient boosting: Basics
+Gradient boosting: Basics
Gradient boosting is again a similar technique to Adapative boosting,
@@ -2197,7 +2236,7 @@ function was the least squares function.
-
Gradient Boosting, algorithm
+Gradient Boosting, algorithm
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard least squares function
@@ -2223,7 +2262,7 @@ The way we proceed in an iterative fashion is to
-
Gradient Boosting, Examples
+Gradient Boosting, Examples
@@ -2313,7 +2352,7 @@ plt.show()
-
Gradient Boots with Early Stopping
+Gradient Boots with Early Stopping
@@ -2377,7 +2416,7 @@ error_going_up = XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
XGBoost or Extreme Gradient
@@ -2398,7 +2437,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
-
Regression Case
+Regression Case
@@ -2453,7 +2492,7 @@ plt.show()
-
Xgboost on the Cancer Data
+Xgboost on the Cancer Data
diff --git a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
index 018a5f714..4891455ed 100644
--- a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
+++ b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
@@ -2084,11 +2084,11 @@
"\n",
"3. For $m=1:M$\n",
"\n",
- "a. minmize $\\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta b(x;\\gamma))^2 wrt $\\gamma$ and $\\beta$$\n",
+ "a. minimize $\\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta b(x;\\gamma))^2$ wrt $\\gamma$ and $\\beta$\n",
"\n",
"b. This gives the optimial values $\\beta_m$ and $\\gamma_m$\n",
"\n",
- "c. Determine then the new values $f_m(x)=f_{m-1}(x) +\\beta_m b(x;\\gamma_m)\n",
+ "c. Determine then the new values $f_m(x)=f_{m-1}(x) +\\beta_m b(x;\\gamma_m)$\n",
"\n",
"\n",
"We could use any of the algorithms we have discussed till now. If we use trees, $\\gamma$ parameterizes the split variables and split points at the internal nodes, and the predictions at the terminal nodes. \n",
@@ -2180,7 +2180,94 @@
"metadata": {},
"source": [
"$$\n",
- "C(\\boldsymbol{y},\\boldsymbol{f}) = \\sum_{i=0}^{n-1}\\exp{-(y_i(f_{m-1(x_i)+\\beta G(x_i})}\n",
+ "C(\\boldsymbol{y},\\boldsymbol{f}) = \\sum_{i=0}^{n-1}\\exp{-(y_i(f_{m-1}(x_i)+\\beta G(x_i))}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We optimize $\\beta$ and $G$ for each value of $m=1:M$ as we did in the regression case.\n",
+ "This is normally done in two steps. Let us however first rewrite the cost function as"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "C(\\boldsymbol{y},\\boldsymbol{f}) = \\sum_{i=0}^{n-1}w_i^{m}\\exp{-(y_i\\beta G(x_i))},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "where we have defined $w_i^m= \\exp{-(y_if_{m-1}(x_i))}$.\n",
+ "\n",
+ "## Building up AdaBoost\n",
+ "\n",
+ "First, for any $\\beta > 0$, we optimize $G$ by setting"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "G_m(x) = \\mathrm{sign} \\sum_{i=0}^{n-1} w_i^m I(y_i \\ne G_(x_i)),\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "which is the classifier that minimizes the weighted error rate in predicting $y$.\n",
+ "\n",
+ "We can do this by rewriting"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\exp{-\\beta}\\sum_{y_i=G(x_i)}w_i^m+\\exp{\\beta}\\sum_{y_i\\ne G(x_i)}w_i^m,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "which can be rewritten as"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "(\\exp{\\beta}-\\exp{-\\beta})\\sum_{i=0}^{n-1}w_i^mI(y_i\\ne G(x_i))+\\exp{-\\beta}\\sum_{i=0}^{n-1}w_i^m=0,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "which leads to"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\beta_m = frac{1}{2}\\log{\\frac{1-\\mathrm{\\overline{err}}}{\\mathrm{\\overline{err}}}},\n",
"$$"
]
},
diff --git a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz
index b9e443560..b4cba0fa7 100644
Binary files a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz and b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz differ
diff --git a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf
index 67c0dcc5d..2e4ad3013 100644
Binary files a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf and b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf differ
diff --git a/doc/src/DecisionTrees/DecisionTrees.do.txt b/doc/src/DecisionTrees/DecisionTrees.do.txt
index 4faf6a288..06b0ae844 100644
--- a/doc/src/DecisionTrees/DecisionTrees.do.txt
+++ b/doc/src/DecisionTrees/DecisionTrees.do.txt
@@ -1664,9 +1664,9 @@ The way we proceed is as follows (here we specialize to the squared-error cost f
o Establish a cost function, here $C(\bm{y},\bm{f}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-f_M(x_i))^2$ with $f_M(x) = \sum_{i=1}^M \beta_m b(x;\gamma_m)$.
o Initialize with a guess $f_0(x)$. It could be one or even zero or some random numbers.
o For $m=1:M$
- o minmize $\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2 wrt $\gamma$ and $\beta$$
+ o minimize $\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2$ wrt $\gamma$ and $\beta$
o This gives the optimial values $\beta_m$ and $\gamma_m$
- o Determine then the new values $f_m(x)=f_{m-1}(x) +\beta_m b(x;\gamma_m)
+ o Determine then the new values $f_m(x)=f_{m-1}(x) +\beta_m b(x;\gamma_m)$
We could use any of the algorithms we have discussed till now. If we use trees, $\gamma$ parameterizes the split variables and split points at the internal nodes, and the predictions at the terminal nodes.
@@ -1720,10 +1720,49 @@ The simplest possible cost function which leads (also simple from a computationa
exponential cost/loss function defined as
!bt
\[
-C(\bm{y},\bm{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1(x_i)+\beta G(x_i})}
+C(\bm{y},\bm{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1}(x_i)+\beta G(x_i))}.
\]
!et
+We optimize $\beta$ and $G$ for each value of $m=1:M$ as we did in the regression case.
+This is normally done in two steps. Let us however first rewrite the cost function as
+
+!bt
+\[
+C(\bm{y},\bm{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{-(y_i\beta G(x_i))},
+\]
+!et
+where we have defined $w_i^m= \exp{-(y_if_{m-1}(x_i))}$.
+
+!split
+===== Building up AdaBoost =====
+
+First, for any $\beta > 0$, we optimize $G$ by setting
+!bt
+\[
+G_m(x) = \mathrm{sign} \sum_{i=0}^{n-1} w_i^m I(y_i \ne G_(x_i)),
+\]
+!et
+which is the classifier that minimizes the weighted error rate in predicting $y$.
+
+We can do this by rewriting
+!bt
+\[
+\exp{-\beta}\sum_{y_i=G(x_i)}w_i^m+\exp{\beta}\sum_{y_i\ne G(x_i)}w_i^m,
+\]
+!et
+which can be rewritten as
+!bt
+\[
+(\exp{\beta}-\exp{-\beta})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{-\beta}\sum_{i=0}^{n-1}w_i^m=0,
+\]
+!et
+which leads to
+!bt
+\[
+\beta_m = frac{1}{2}\log{\frac{1-\mathrm{\overline{err}}}{\mathrm{\overline{err}}}},
+\]
+!et
!split
===== Adaptive boosting: AdaBoost, Basic Algorithm =====