@@ -325,7 +327,7 @@ $$
-Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of -observations. We define a classification function \( G(x) \) which produces a prediction taking any of the two values -\( \{-1,1\} \). - -
-The error rate of the training sample is then - -$$ -\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i). -$$ - -
-The iterative procedure starts with defining a weak classifier whose -error rate is barely better than random guessing. Teh iterative -procedure in boosting is to sequentially apply a the weak -classification algorithm to repeatedly modified versions of the data producing a sequence of weak classifiers \( G_m(x) \). - -
-Here we will express our function \( f(x) \) in terms of \( G(x) \). That is -$$ -f_M(x) = \sum_{i=1}^M \beta_m b(x;\gamma_m), -$$ - -will be a function of -$$ -G_M(x) = \mathrm{sign} \sum_{i=1}^M \alpha_m G_m(x). -$$ +With these equations we can then in turn find the parameters \( \beta_1 \) and \( \gamma_0^{1} \) and \( \gamma_1^1 \) as
@@ -323,7 +299,7 @@ $$
-In our iterative procedure we define thus +Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of +observations. We define a classification function \( G(x) \) which produces a prediction taking any of the two values +\( \{-1,1\} \). + +
+The error rate of the training sample is then + $$ -f_m(x) = f_{m-1}(x)+\beta_mG_m(x). +\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i)). $$
-The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the -exponential cost/loss function defined as -$$ -C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1}(x_i)+\beta G(x_i))}. -$$ +The iterative procedure starts with defining a weak classifier whose +error rate is barely better than random guessing. The iterative +procedure in boosting is to sequentially apply a the weak +classification algorithm to repeatedly modified versions of the data +producing a sequence of weak classifiers \( G_m(x) \).
-We optimize \( \beta \) and \( G \) for each value of \( m=1:M \) as we did in the regression case. -This is normally done in two steps. Let us however first rewrite the cost function as - +Here we will express our function \( f(x) \) in terms of \( G(x) \). That is $$ -C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{-(y_i\beta G(x_i))}, +f_M(x) = \sum_{i=1}^M \beta_m b(x;\gamma_m), $$ -where we have defined \( w_i^m= \exp{-(y_if_{m-1}(x_i))} \). +will be a function of +$$ +G_M(x) = \mathrm{sign} \sum_{i=1}^M \alpha_m G_m(x). +$$
@@ -317,7 +326,7 @@ where we have defined \( w_i^m= \exp{-(y_if_{m-1}(x_i))} \).
-First, for any \( \beta > 0 \), we optimize \( G \) by setting +In our iterative procedure we define thus $$ -G_m(x) = \mathrm{sign} \sum_{i=0}^{n-1} w_i^m I(y_i \ne G_(x_i)), +f_m(x) = f_{m-1}(x)+\beta_mG_m(x). $$ -which is the classifier that minimizes the weighted error rate in predicting \( y \). -
-We can do this by rewriting +The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the +exponential cost/loss function defined as $$ -\exp{-(\beta)}\sum_{y_i=G(x_i)}w_i^m+\exp{(\beta)}\sum_{y_i\ne G(x_i)}w_i^m, +C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{(-y_i(f_{m-1}(x_i)+\beta G(x_i))}. $$ -which can be rewritten as +
+We optimize \( \beta \) and \( G \) for each value of \( m=1:M \) as we did in the regression case. +This is normally done in two steps. Let us however first rewrite the cost function as + $$ -(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{-(\beta)}\sum_{i=0}^{n-1}w_i^m=0, +C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{(-y_i\beta G(x_i))}, $$ -which leads to -$$ -\beta_m = \frac{1}{2}\log{\frac{1-\mathrm{\overline{err}}}{\mathrm{\overline{err}}}}, -$$ - -where we have redefined the error as -$$ -\mathrm{\overline{err}}_m=\frac{1}{n}\frac{\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i)}{\sum_{i=0}^{n-1}w_i^m}, -$$ - -which leads to an update of -$$ -f_m(x) = f_{m-1}(x) +\beta_m G_m(x). -$$ - -This leads to the new weights -$$ -w_i^{m+1} = w_i^m \exp{-(y_i\beta_m G_m(x_i))} -$$ +where we have defined \( w_i^m= \exp{(-y_if_{m-1}(x_i))} \). +
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs052.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs052.html index 69e2a0f44..9cfb5899f 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs052.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs052.html @@ -128,31 +128,32 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec47'), + ('Finding the Optimal Parameters', 2, None, '___sec48'), ('Iterative Fitting, Classification, AdaBoost', 2, None, - '___sec48'), - ('Adaptive Boosting, AdaBoost', 2, None, '___sec49'), - ('Building up AdaBoost', 2, None, '___sec50'), + '___sec49'), + ('Adaptive Boosting, AdaBoost', 2, None, '___sec50'), + ('Building up AdaBoost', 2, None, '___sec51'), ('Adaptive boosting: AdaBoost, Basic Algorithm', 2, None, - '___sec51'), - ('Basic Steps of AdaBoost', 2, None, '___sec52'), - ('AdaBoost Examples', 2, None, '___sec53'), - ('Gradient boosting: Basics', 2, None, '___sec54'), - ('Gradient Boosting, algorithm', 2, None, '___sec55'), + '___sec52'), + ('Basic Steps of AdaBoost', 2, None, '___sec53'), + ('AdaBoost Examples', 2, None, '___sec54'), + ('Gradient boosting: Basics', 2, None, '___sec55'), + ('Gradient Boosting, algorithm', 2, None, '___sec56'), ('Gradient Boosting, Examples of Regression', 2, None, - '___sec56'), - ('Gradient Boosting, Examples of Classification', + '___sec57'), + ('Gradient Boosting, Classification Example', 2, None, - '___sec57'), - ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'), - ('Regression Case', 2, None, '___sec59'), - ('Xgboost on the Cancer Data', 2, None, '___sec60')]} + '___sec58'), + ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'), + ('Regression Case', 2, None, '___sec60'), + ('Xgboost on the Cancer Data', 2, None, '___sec61')]} end of tocinfo --> @@ -238,19 +239,20 @@ MathJax.Hub.Config({
-The algorithm here is rather straightforward. Assume that our weak -classifier is a decision tree and we consider a binary set of outputs -with \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of -observations. Our design matrix is given in terms of the -feature/predictor vectors -\( \boldsymbol{X}=[\boldsymbol{x}_0\boldsymbol{x}_1\dots\boldsymbol{x}_{p-1}] \). Finally, we define also a -classifier determined by our data via a function \( G(x) \). This function tells us how well we are able to classify our outputs/targets \( \boldsymbol{y} \). - -
-We can then define the misclassification error \( \mathrm{err} \) as +First, for any \( \beta > 0 \), we optimize \( G \) by setting $$ -\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i), +G_m(x) = \mathrm{sign} \sum_{i=0}^{n-1} w_i^m I(y_i \ne G_(x_i)), $$ -where the function \( I() \) is one if we misclassify and zero if we classify correctly. +which is the classifier that minimizes the weighted error rate in predicting \( y \).
+We can do this by rewriting +$$ +\exp{-(\beta)}\sum_{y_i=G(x_i)}w_i^m+\exp{(\beta)}\sum_{y_i\ne G(x_i)}w_i^m, +$$ + +which can be rewritten as +$$ +(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{(-\beta)}\sum_{i=0}^{n-1}w_i^m=0, +$$ + +which leads to +$$ +\beta_m = \frac{1}{2}\log{\frac{1-\mathrm{\overline{err}}}{\mathrm{\overline{err}}}}, +$$ + +where we have redefined the error as +$$ +\mathrm{\overline{err}}_m=\frac{1}{n}\frac{\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i)}{\sum_{i=0}^{n-1}w_i^m}, +$$ + +which leads to an update of +$$ +f_m(x) = f_{m-1}(x) +\beta_m G_m(x). +$$ + +This leads to the new weights +$$ +w_i^{m+1} = w_i^m \exp{(-y_i\beta_m G_m(x_i))} +$$ +
-With the above definitions we are now ready to set up the algorithm for AdaBoost. -The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases. - -
+We have already defined the misclassification error \( \mathrm{err} \) as $$ -\mathrm{\overline{err}}_m=\frac{\sum_{i=0}^{n-1}w_i^m I(y_i\ne G(x_i))}{\sum_{i=0}^{n-1}w_i}, +\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i)), $$ - -
@@ -327,6 +311,7 @@ observations that are missed in the previous iterations.
-Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here. +With the above definitions we are now ready to set up the algorithm for AdaBoost. +The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases. -
+
from sklearn.ensemble import AdaBoostClassifier
+$$
+\mathrm{\overline{err}}_m=\frac{\sum_{i=0}^{n-1}w_i^m I(y_i\ne G(x_i))}{\sum_{i=0}^{n-1}w_i},
+$$
-ada_clf = AdaBoostClassifier(
- DecisionTreeClassifier(max_depth=1), n_estimators=200,
- algorithm="SAMME.R", learning_rate=0.5, random_state=42)
-ada_clf.fit(X_train, y_train)
-from sklearn.ensemble import AdaBoostClassifier
+
+- Then we start looping over all attempts at classifying, namely we start an iterative process for \( m=1:M \), where \( M \) is the final number of classifications. Our given classifier could for example be a plain decision tree.
+
+
+ - Fit then a given classifier to the training using the weights \( w_i \).
+ - Compute then \( \mathrm{err} \) and figure out which events are classified properly and which are classified wrongly.
+ - Define a quantity \( \alpha_{m} = \log{(1-\mathrm{\overline{err}}_m)/\mathrm{\overline{err}}_m} \)
+ - Set the new weights to \( w_i = w_i\times \exp{(\alpha_m I(y_i\ne G(x_i)} \).
+
+
+
@@ -319,6 +328,7 @@ plt.show()
-Gradient boosting is again a similar technique to Adapative boosting, -it combines so-called weak classifiers or regressors into a strong -method via a series of iterations. +Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here.
-In order to understand the method, let us illustrate its basics by -bringing back the essential steps in linear regression, where our cost -function was the least squares function. -
-See discussion during lecture November 8. + +
from sklearn.ensemble import AdaBoostClassifier
+ada_clf = AdaBoostClassifier(
+ DecisionTreeClassifier(max_depth=1), n_estimators=200,
+ algorithm="SAMME.R", learning_rate=0.5, random_state=42)
+ada_clf.fit(X_train, y_train)
+
+from sklearn.ensemble import AdaBoostClassifier
+
+ada_clf = AdaBoostClassifier(
+ DecisionTreeClassifier(max_depth=1), n_estimators=200,
+ algorithm="SAMME.R", learning_rate=0.5, random_state=42)
+ada_clf.fit(X_train_scaled, y_train)
+y_pred = ada_clf.predict(X_test_scaled)
+skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
+plt.show()
+y_probas = ada_clf.predict_proba(X_test_scaled)
+skplt.metrics.plot_roc(y_test, y_probas)
+plt.show()
+skplt.metrics.plot_cumulative_gain(y_test, y_probas)
+plt.show()
+
@@ -303,6 +320,7 @@ See discussion during lecture November 8.
-Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard square-error function -$$ -C(\boldsymbol{y},\boldsymbol{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2. -$$ +Gradient boosting is again a similar technique to Adapative boosting, +it combines so-called weak classifiers or regressors into a strong +method via a series of iterations.
-The way we proceed in an iterative fashion is to +In order to understand the method, let us illustrate its basics by +bringing back the essential steps in linear regression, where our cost +function was the least squares function. -
+See discussion during lecture November 8. +
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html index 37c8e5a53..8975c7810 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html @@ -128,31 +128,32 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec47'), + ('Finding the Optimal Parameters', 2, None, '___sec48'), ('Iterative Fitting, Classification, AdaBoost', 2, None, - '___sec48'), - ('Adaptive Boosting, AdaBoost', 2, None, '___sec49'), - ('Building up AdaBoost', 2, None, '___sec50'), + '___sec49'), + ('Adaptive Boosting, AdaBoost', 2, None, '___sec50'), + ('Building up AdaBoost', 2, None, '___sec51'), ('Adaptive boosting: AdaBoost, Basic Algorithm', 2, None, - '___sec51'), - ('Basic Steps of AdaBoost', 2, None, '___sec52'), - ('AdaBoost Examples', 2, None, '___sec53'), - ('Gradient boosting: Basics', 2, None, '___sec54'), - ('Gradient Boosting, algorithm', 2, None, '___sec55'), + '___sec52'), + ('Basic Steps of AdaBoost', 2, None, '___sec53'), + ('AdaBoost Examples', 2, None, '___sec54'), + ('Gradient boosting: Basics', 2, None, '___sec55'), + ('Gradient Boosting, algorithm', 2, None, '___sec56'), ('Gradient Boosting, Examples of Regression', 2, None, - '___sec56'), - ('Gradient Boosting, Examples of Classification', + '___sec57'), + ('Gradient Boosting, Classification Example', 2, None, - '___sec57'), - ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'), - ('Regression Case', 2, None, '___sec59'), - ('Xgboost on the Cancer Data', 2, None, '___sec60')]} + '___sec58'), + ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'), + ('Regression Case', 2, None, '___sec60'), + ('Xgboost on the Cancer Data', 2, None, '___sec61')]} end of tocinfo --> @@ -238,19 +239,20 @@ MathJax.Hub.Config({
+Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard square-error function +$$ +C(\boldsymbol{y},\boldsymbol{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2. +$$ - -
import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.ensemble import GradientBoostingRegressor
-from sklearn.preprocessing import StandardScaler
-import scikitplot as skplt
-from sklearn.metrics import mean_squared_error
-
-n = 100
-maxdegree = 6
-
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-
-error = np.zeros(maxdegree)
-bias = np.zeros(maxdegree)
-variance = np.zeros(maxdegree)
-polydegree = np.zeros(maxdegree)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-for degree in range(1,maxdegree):
- model = GradientBoostingRegressor(max_depth=degree, n_estimators=100, learning_rate=1.0)
- model.fit(X_train_scaled,y_train)
- y_pred = model.predict(X_test_scaled)
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
- variance[degree] = np.mean( np.var(y_pred) )
- print('Max depth:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-
-plt.xlim(1,maxdegree-1)
-plt.plot(polydegree, error, label='Error')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
-save_fig("gdregression")
-plt.show()
-+The way we proceed in an iterative fashion is to + +
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html index c52637bec..ddda5b9cf 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html @@ -128,31 +128,32 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec47'), + ('Finding the Optimal Parameters', 2, None, '___sec48'), ('Iterative Fitting, Classification, AdaBoost', 2, None, - '___sec48'), - ('Adaptive Boosting, AdaBoost', 2, None, '___sec49'), - ('Building up AdaBoost', 2, None, '___sec50'), + '___sec49'), + ('Adaptive Boosting, AdaBoost', 2, None, '___sec50'), + ('Building up AdaBoost', 2, None, '___sec51'), ('Adaptive boosting: AdaBoost, Basic Algorithm', 2, None, - '___sec51'), - ('Basic Steps of AdaBoost', 2, None, '___sec52'), - ('AdaBoost Examples', 2, None, '___sec53'), - ('Gradient boosting: Basics', 2, None, '___sec54'), - ('Gradient Boosting, algorithm', 2, None, '___sec55'), + '___sec52'), + ('Basic Steps of AdaBoost', 2, None, '___sec53'), + ('AdaBoost Examples', 2, None, '___sec54'), + ('Gradient boosting: Basics', 2, None, '___sec55'), + ('Gradient Boosting, algorithm', 2, None, '___sec56'), ('Gradient Boosting, Examples of Regression', 2, None, - '___sec56'), - ('Gradient Boosting, Examples of Classification', + '___sec57'), + ('Gradient Boosting, Classification Example', 2, None, - '___sec57'), - ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'), - ('Regression Case', 2, None, '___sec59'), - ('Xgboost on the Cancer Data', 2, None, '___sec60')]} + '___sec58'), + ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'), + ('Regression Case', 2, None, '___sec60'), + ('Xgboost on the Cancer Data', 2, None, '___sec61')]} end of tocinfo --> @@ -238,19 +239,20 @@ MathJax.Hub.Config({
import matplotlib.pyplot as plt
import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import load_breast_cancer
-import scikitplot as skplt
-from sklearn.ensemble import GradientBoostingClassifier
-from sklearn.model_selection import cross_validate
-
-# Load the data
-cancer = load_breast_cancer()
-
-X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
-print(X_train.shape)
-print(X_test.shape)
-#now scale the data
+from sklearn.model_selection import train_test_split
+from sklearn.ensemble import GradientBoostingRegressor
from sklearn.preprocessing import StandardScaler
+import scikitplot as skplt
+from sklearn.metrics import mean_squared_error
+
+n = 100
+maxdegree = 6
+
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+
+error = np.zeros(maxdegree)
+bias = np.zeros(maxdegree)
+variance = np.zeros(maxdegree)
+polydegree = np.zeros(maxdegree)
+X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
-gd_clf = GradientBoostingClassifier(max_depth=3, n_estimators=100, learning_rate=1.0)
-gd_clf.fit(X_train_scaled, y_train)
-#Cross validation
-accuracy = cross_validate(gd_clf,X_test_scaled,y_test,cv=10)['test_score']
-print(accuracy)
-print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(gd_clf.score(X_test_scaled,y_test)))
+for degree in range(1,maxdegree):
+ model = GradientBoostingRegressor(max_depth=degree, n_estimators=100, learning_rate=1.0)
+ model.fit(X_train_scaled,y_train)
+ y_pred = model.predict(X_test_scaled)
+ polydegree[degree] = degree
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
+ variance[degree] = np.mean( np.var(y_pred) )
+ print('Max depth:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-import scikitplot as skplt
-y_pred = gd_clf.predict(X_test_scaled)
-skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
-save_fig("gdclassiffierconfusion")
-plt.show()
-y_probas = gd_clf.predict_proba(X_test_scaled)
-skplt.metrics.plot_roc(y_test, y_probas)
-save_fig("gdclassiffierroc")
-plt.show()
-skplt.metrics.plot_cumulative_gain(y_test, y_probas)
-save_fig("gdclassiffiercgain")
+plt.xlim(1,maxdegree-1)
+plt.plot(polydegree, error, label='Error')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+save_fig("gdregression")
plt.show()
@@ -330,6 +338,7 @@ plt.show()
-XGBoost or Extreme Gradient -Boosting, is an optimized distributed gradient boosting library -designed to be highly efficient, flexible and portable. It implements -machine learning algorithms under the Gradient Boosting -framework. XGBoost provides a parallel tree boosting that solve many -data science problems in a fast and accurate way. See the article by Chen and Guestrin. -
-The authors design and build a highly scalable end-to-end tree -boosting system. It has a theoretically justified weighted quantile -sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning. + +
import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import load_breast_cancer
+import scikitplot as skplt
+from sklearn.ensemble import GradientBoostingClassifier
+from sklearn.model_selection import cross_validate
-
-It is now the algorithm which wins essentially all ML competitions!!!
+# Load the data
+cancer = load_breast_cancer()
+X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
+print(X_train.shape)
+print(X_test.shape)
+#now scale the data
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+
+gd_clf = GradientBoostingClassifier(max_depth=3, n_estimators=100, learning_rate=1.0)
+gd_clf.fit(X_train_scaled, y_train)
+#Cross validation
+accuracy = cross_validate(gd_clf,X_test_scaled,y_test,cv=10)['test_score']
+print(accuracy)
+print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(gd_clf.score(X_test_scaled,y_test)))
+
+import scikitplot as skplt
+y_pred = gd_clf.predict(X_test_scaled)
+skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
+save_fig("gdclassiffierconfusion")
+plt.show()
+y_probas = gd_clf.predict_proba(X_test_scaled)
+skplt.metrics.plot_roc(y_test, y_probas)
+save_fig("gdclassiffierroc")
+plt.show()
+skplt.metrics.plot_cumulative_gain(y_test, y_probas)
+save_fig("gdclassiffiercgain")
+plt.show()
+
@@ -302,6 +331,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
+XGBoost or Extreme Gradient +Boosting, is an optimized distributed gradient boosting library +designed to be highly efficient, flexible and portable. It implements +machine learning algorithms under the Gradient Boosting +framework. XGBoost provides a parallel tree boosting that solve many +data science problems in a fast and accurate way. See the article by Chen and Guestrin. - -
import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-import xgboost as xgb
-from sklearn.preprocessing import StandardScaler
-import scikitplot as skplt
-from sklearn.metrics import mean_squared_error
+
+The authors design and build a highly scalable end-to-end tree
+boosting system. It has a theoretically justified weighted quantile
+sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning.
-n = 100
-maxdegree = 6
+
+It is now the algorithm which wins essentially all ML competitions!!!
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-
-error = np.zeros(maxdegree)
-bias = np.zeros(maxdegree)
-variance = np.zeros(maxdegree)
-polydegree = np.zeros(maxdegree)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-for degree in range(maxdegree):
- model = xgb.XGBRegressor(objective ='reg:squarederror', colsaobjective ='reg:squarederror', colsample_bytree = 0.3, learning_rate = 0.1,max_depth = degree, alpha = 10, n_estimators = 200)
-
- model.fit(X_train_scaled,y_train)
- y_pred = model.predict(X_test_scaled)
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
- variance[degree] = np.mean( np.var(y_pred) )
- print('Max depth:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-
-plt.xlim(1,maxdegree-1)
-plt.plot(polydegree, error, label='Error')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
-plt.show()
-
@@ -335,6 +303,7 @@ plt.show()
$$
-(\beta_m,\gamma_m) \mathrm{armmin}_{\beta,\gambda}\hspace{0.2cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(\gamma_0+\gamma_1 x_i))^2.
+(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\gambda}\hspace{0.2cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(\gamma_0+\gamma_1 x_i))^2.
\[
$$
@@ -2095,21 +2095,29 @@ $$
and
$$
-\frac{\partial {\cal C}}{\partial \gamma_0} =
+\frac{\partial {\cal C}}{\partial \gamma_0} =-2\sum_{i}\beta(y_i-\beta(\gamma_0+\gamma_1 x_i))=0,
$$
and
$$
-\frac{\partial {\cal C}}{\partial \gamma_1} =
+\frac{\partial {\cal C}}{\partial \gamma_1} = =-2\sum_{i}\beta x_i(y_i-\beta(\gamma_0+\gamma_1 x_i))=0.
$$
+With these equations we can then in turn find the parameters \( \beta_1 \) and \( \gamma_0^{1} \) and \( \gamma_1^1 \) as
+
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
@@ -2121,15 +2129,16 @@ The error rate of the training sample is then
The iterative procedure starts with defining a weak classifier whose
-error rate is barely better than random guessing. Teh iterative
+error rate is barely better than random guessing. The iterative
procedure in boosting is to sequentially apply a the weak
-classification algorithm to repeatedly modified versions of the data producing a sequence of weak classifiers \( G_m(x) \).
+classification algorithm to repeatedly modified versions of the data
+producing a sequence of weak classifiers \( G_m(x) \).
Here we will express our function \( f(x) \) in terms of \( G(x) \). That is
@@ -2149,7 +2158,7 @@ $$
In our iterative procedure we define thus
@@ -2164,7 +2173,7 @@ The simplest possible cost function which leads (also simple from a computationa
exponential cost/loss function defined as
First, for any \( \beta > 0 \), we optimize \( G \) by setting
@@ -2206,7 +2215,7 @@ $$
which can be rewritten as
The algorithm here is rather straightforward. Assume that our weak
@@ -2253,10 +2262,10 @@ feature/predictor vectors
classifier determined by our data via a function \( G(x) \). This function tells us how well we are able to classify our outputs/targets \( \boldsymbol{y} \).
-We can then define the misclassification error \( \mathrm{err} \) as
+We have already defined the misclassification error \( \mathrm{err} \) as
With the above definitions we are now ready to set up the algorithm for AdaBoost.
@@ -2306,7 +2315,7 @@ observations that are missed in the previous iterations.
Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here.
@@ -2340,7 +2349,7 @@ plt.show()
Gradient boosting is again a similar technique to Adapative boosting,
@@ -2358,7 +2367,7 @@ See discussion during lecture November 8.
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard square-error function
@@ -2386,7 +2395,7 @@ The way we proceed in an iterative fashion is to
@@ -2441,7 +2450,7 @@ plt.show()
@@ -2490,7 +2499,7 @@ plt.show()
XGBoost or Extreme Gradient
@@ -2511,7 +2520,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
@@ -2567,7 +2576,10 @@ plt.show()
+As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html b/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
index 9d9bf9af6..e0c9995ac 100644
--- a/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
+++ b/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
@@ -148,31 +148,32 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec47'),
+ ('Finding the Optimal Parameters', 2, None, '___sec48'),
('Iterative Fitting, Classification, AdaBoost',
2,
None,
- '___sec48'),
- ('Adaptive Boosting, AdaBoost', 2, None, '___sec49'),
- ('Building up AdaBoost', 2, None, '___sec50'),
+ '___sec49'),
+ ('Adaptive Boosting, AdaBoost', 2, None, '___sec50'),
+ ('Building up AdaBoost', 2, None, '___sec51'),
('Adaptive boosting: AdaBoost, Basic Algorithm',
2,
None,
- '___sec51'),
- ('Basic Steps of AdaBoost', 2, None, '___sec52'),
- ('AdaBoost Examples', 2, None, '___sec53'),
- ('Gradient boosting: Basics', 2, None, '___sec54'),
- ('Gradient Boosting, algorithm', 2, None, '___sec55'),
+ '___sec52'),
+ ('Basic Steps of AdaBoost', 2, None, '___sec53'),
+ ('AdaBoost Examples', 2, None, '___sec54'),
+ ('Gradient boosting: Basics', 2, None, '___sec55'),
+ ('Gradient Boosting, algorithm', 2, None, '___sec56'),
('Gradient Boosting, Examples of Regression',
2,
None,
- '___sec56'),
- ('Gradient Boosting, Examples of Classification',
+ '___sec57'),
+ ('Gradient Boosting, Classification Example',
2,
None,
- '___sec57'),
- ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'),
- ('Regression Case', 2, None, '___sec59'),
- ('Xgboost on the Cancer Data', 2, None, '___sec60')]}
+ '___sec58'),
+ ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'),
+ ('Regression Case', 2, None, '___sec60'),
+ ('Xgboost on the Cancer Data', 2, None, '___sec61')]}
end of tocinfo -->
+With these equations we can then in turn find the parameters \( \beta_1 \) and \( \gamma_0^{1} \) and \( \gamma_1^1 \) as
+
+
+
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
@@ -2114,14 +2123,15 @@ observations. We define a classification function \( G(x) \) which produces a pr
The error rate of the training sample is then
$$
-\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i).
+\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i)).
$$
The iterative procedure starts with defining a weak classifier whose
-error rate is barely better than random guessing. Teh iterative
+error rate is barely better than random guessing. The iterative
procedure in boosting is to sequentially apply a the weak
-classification algorithm to repeatedly modified versions of the data producing a sequence of weak classifiers \( G_m(x) \).
+classification algorithm to repeatedly modified versions of the data
+producing a sequence of weak classifiers \( G_m(x) \).
Here we will express our function \( f(x) \) in terms of \( G(x) \). That is
@@ -2137,7 +2147,7 @@ $$
In our iterative procedure we define thus
@@ -2149,7 +2159,7 @@ $$
The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the
exponential cost/loss function defined as
$$
-C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1}(x_i)+\beta G(x_i))}.
+C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{(-y_i(f_{m-1}(x_i)+\beta G(x_i))}.
$$
@@ -2157,15 +2167,15 @@ We optimize \( \beta \) and \( G \) for each value of \( m=1:M \) as we did in t
This is normally done in two steps. Let us however first rewrite the cost function as
$$
-C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{-(y_i\beta G(x_i))},
+C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{(-y_i\beta G(x_i))},
$$
-where we have defined \( w_i^m= \exp{-(y_if_{m-1}(x_i))} \).
+where we have defined \( w_i^m= \exp{(-y_if_{m-1}(x_i))} \).
First, for any \( \beta > 0 \), we optimize \( G \) by setting
@@ -2183,7 +2193,7 @@ $$
which can be rewritten as
$$
-(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{-(\beta)}\sum_{i=0}^{n-1}w_i^m=0,
+(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{(-\beta)}\sum_{i=0}^{n-1}w_i^m=0,
$$
which leads to
@@ -2203,12 +2213,12 @@ $$
This leads to the new weights
$$
-w_i^{m+1} = w_i^m \exp{-(y_i\beta_m G_m(x_i))}
+w_i^{m+1} = w_i^m \exp{(-y_i\beta_m G_m(x_i))}
$$
The algorithm here is rather straightforward. Assume that our weak
@@ -2220,9 +2230,9 @@ feature/predictor vectors
classifier determined by our data via a function \( G(x) \). This function tells us how well we are able to classify our outputs/targets \( \boldsymbol{y} \).
-We can then define the misclassification error \( \mathrm{err} \) as
+We have already defined the misclassification error \( \mathrm{err} \) as
$$
-\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i),
+\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i)),
$$
where the function \( I() \) is one if we misclassify and zero if we classify correctly.
@@ -2230,7 +2240,7 @@ where the function \( I() \) is one if we misclassify and zero if we classify co
With the above definitions we are now ready to set up the algorithm for AdaBoost.
@@ -2270,7 +2280,7 @@ observations that are missed in the previous iterations.
Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here.
@@ -2303,7 +2313,7 @@ plt.show()
Gradient boosting is again a similar technique to Adapative boosting,
@@ -2321,7 +2331,7 @@ See discussion during lecture November 8.
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard square-error function
@@ -2347,7 +2357,7 @@ The way we proceed in an iterative fashion is to
@@ -2401,7 +2411,7 @@ plt.show()
@@ -2449,7 +2459,7 @@ plt.show()
XGBoost or Extreme Gradient
@@ -2470,7 +2480,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
@@ -2525,7 +2535,10 @@ plt.show()
+As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees.html b/doc/pub/DecisionTrees/html/DecisionTrees.html
index 054a1ea4b..d038bafd1 100644
--- a/doc/pub/DecisionTrees/html/DecisionTrees.html
+++ b/doc/pub/DecisionTrees/html/DecisionTrees.html
@@ -153,31 +153,32 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec47'),
+ ('Finding the Optimal Parameters', 2, None, '___sec48'),
('Iterative Fitting, Classification, AdaBoost',
2,
None,
- '___sec48'),
- ('Adaptive Boosting, AdaBoost', 2, None, '___sec49'),
- ('Building up AdaBoost', 2, None, '___sec50'),
+ '___sec49'),
+ ('Adaptive Boosting, AdaBoost', 2, None, '___sec50'),
+ ('Building up AdaBoost', 2, None, '___sec51'),
('Adaptive boosting: AdaBoost, Basic Algorithm',
2,
None,
- '___sec51'),
- ('Basic Steps of AdaBoost', 2, None, '___sec52'),
- ('AdaBoost Examples', 2, None, '___sec53'),
- ('Gradient boosting: Basics', 2, None, '___sec54'),
- ('Gradient Boosting, algorithm', 2, None, '___sec55'),
+ '___sec52'),
+ ('Basic Steps of AdaBoost', 2, None, '___sec53'),
+ ('AdaBoost Examples', 2, None, '___sec54'),
+ ('Gradient boosting: Basics', 2, None, '___sec55'),
+ ('Gradient Boosting, algorithm', 2, None, '___sec56'),
('Gradient Boosting, Examples of Regression',
2,
None,
- '___sec56'),
- ('Gradient Boosting, Examples of Classification',
+ '___sec57'),
+ ('Gradient Boosting, Classification Example',
2,
None,
- '___sec57'),
- ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'),
- ('Regression Case', 2, None, '___sec59'),
- ('Xgboost on the Cancer Data', 2, None, '___sec60')]}
+ '___sec58'),
+ ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'),
+ ('Regression Case', 2, None, '___sec60'),
+ ('Xgboost on the Cancer Data', 2, None, '___sec61')]}
end of tocinfo -->
+With these equations we can then in turn find the parameters \( \beta_1 \) and \( \gamma_0^{1} \) and \( \gamma_1^1 \) as
+
+
+
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
@@ -2119,14 +2128,15 @@ observations. We define a classification function \( G(x) \) which produces a pr
The error rate of the training sample is then
$$
-\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i).
+\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i)).
$$
The iterative procedure starts with defining a weak classifier whose
-error rate is barely better than random guessing. Teh iterative
+error rate is barely better than random guessing. The iterative
procedure in boosting is to sequentially apply a the weak
-classification algorithm to repeatedly modified versions of the data producing a sequence of weak classifiers \( G_m(x) \).
+classification algorithm to repeatedly modified versions of the data
+producing a sequence of weak classifiers \( G_m(x) \).
Here we will express our function \( f(x) \) in terms of \( G(x) \). That is
@@ -2142,7 +2152,7 @@ $$
In our iterative procedure we define thus
@@ -2154,7 +2164,7 @@ $$
The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the
exponential cost/loss function defined as
$$
-C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1}(x_i)+\beta G(x_i))}.
+C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{(-y_i(f_{m-1}(x_i)+\beta G(x_i))}.
$$
@@ -2162,15 +2172,15 @@ We optimize \( \beta \) and \( G \) for each value of \( m=1:M \) as we did in t
This is normally done in two steps. Let us however first rewrite the cost function as
$$
-C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{-(y_i\beta G(x_i))},
+C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{(-y_i\beta G(x_i))},
$$
-where we have defined \( w_i^m= \exp{-(y_if_{m-1}(x_i))} \).
+where we have defined \( w_i^m= \exp{(-y_if_{m-1}(x_i))} \).
First, for any \( \beta > 0 \), we optimize \( G \) by setting
@@ -2188,7 +2198,7 @@ $$
which can be rewritten as
$$
-(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{-(\beta)}\sum_{i=0}^{n-1}w_i^m=0,
+(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{(-\beta)}\sum_{i=0}^{n-1}w_i^m=0,
$$
which leads to
@@ -2208,12 +2218,12 @@ $$
This leads to the new weights
$$
-w_i^{m+1} = w_i^m \exp{-(y_i\beta_m G_m(x_i))}
+w_i^{m+1} = w_i^m \exp{(-y_i\beta_m G_m(x_i))}
$$
The algorithm here is rather straightforward. Assume that our weak
@@ -2225,9 +2235,9 @@ feature/predictor vectors
classifier determined by our data via a function \( G(x) \). This function tells us how well we are able to classify our outputs/targets \( \boldsymbol{y} \).
-We can then define the misclassification error \( \mathrm{err} \) as
+We have already defined the misclassification error \( \mathrm{err} \) as
$$
-\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i),
+\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i)),
$$
where the function \( I() \) is one if we misclassify and zero if we classify correctly.
@@ -2235,7 +2245,7 @@ where the function \( I() \) is one if we misclassify and zero if we classify co
With the above definitions we are now ready to set up the algorithm for AdaBoost.
@@ -2275,7 +2285,7 @@ observations that are missed in the previous iterations.
Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here.
@@ -2308,7 +2318,7 @@ plt.show()
Gradient boosting is again a similar technique to Adapative boosting,
@@ -2326,7 +2336,7 @@ See discussion during lecture November 8.
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard square-error function
@@ -2352,7 +2362,7 @@ The way we proceed in an iterative fashion is to
@@ -2406,7 +2416,7 @@ plt.show()
@@ -2454,7 +2464,7 @@ plt.show()
XGBoost or Extreme Gradient
@@ -2475,7 +2485,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
@@ -2530,7 +2540,10 @@ plt.show()
+As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
diff --git a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
index 1c70f99be..752b3f244 100644
--- a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
+++ b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
@@ -2131,7 +2131,7 @@
"metadata": {},
"source": [
"$$\n",
- "(\\beta_m,\\gamma_m) \\mathrm{armmin}_{\\beta,\\gambda}\\hspace{0.2cm} \\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta b(x;\\gamma))^2\\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta(\\gamma_0+\\gamma_1 x_i))^2.\n",
+ "(\\beta_m,\\gamma_m) \\mathrm{argmin}_{\\beta,\\gambda}\\hspace{0.2cm} \\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta b(x;\\gamma))^2\\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta(\\gamma_0+\\gamma_1 x_i))^2.\n",
"$$"
]
},
@@ -2164,7 +2164,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\frac{\\partial {\\cal C}}{\\partial \\gamma_0} =\n",
+ "\\frac{\\partial {\\cal C}}{\\partial \\gamma_0} =-2\\sum_{i}\\beta(y_i-\\beta(\\gamma_0+\\gamma_1 x_i))=0,\n",
"$$"
]
},
@@ -2180,7 +2180,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\frac{\\partial {\\cal C}}{\\partial \\gamma_1} =\n",
+ "\\frac{\\partial {\\cal C}}{\\partial \\gamma_1} = =-2\\sum_{i}\\beta x_i(y_i-\\beta(\\gamma_0+\\gamma_1 x_i))=0.\n",
"$$"
]
},
@@ -2188,6 +2188,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
+ "## Finding the Optimal Parameters\n",
+ "\n",
+ "With these equations we can then in turn find the parameters $\\beta_1$ and $\\gamma_0^{1}$ and $\\gamma_1^1$ as\n",
+ "\n",
+ "\n",
+ "\n",
"## Iterative Fitting, Classification, AdaBoost\n",
"\n",
"Let us consider a binary classification problem with two outcomes $y_i \\in \\{-1,1\\}$ and $i=0,1,2,\\dots,n-1$ as our set of\n",
@@ -2202,7 +2208,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\mathrm{\\overline{err}}=\\frac{1}{n} \\sum_{i=0}^{n-1} I(y_i\\ne G(x_i).\n",
+ "\\mathrm{\\overline{err}}=\\frac{1}{n} \\sum_{i=0}^{n-1} I(y_i\\ne G(x_i)).\n",
"$$"
]
},
@@ -2211,9 +2217,10 @@
"metadata": {},
"source": [
"The iterative procedure starts with defining a weak classifier whose\n",
- "error rate is barely better than random guessing. Teh iterative\n",
+ "error rate is barely better than random guessing. The iterative\n",
"procedure in boosting is to sequentially apply a the weak\n",
- "classification algorithm to repeatedly modified versions of the data producing a sequence of weak classifiers $G_m(x)$.\n",
+ "classification algorithm to repeatedly modified versions of the data\n",
+ "producing a sequence of weak classifiers $G_m(x)$.\n",
"\n",
"Here we will express our function $f(x)$ in terms of $G(x)$. That is"
]
@@ -2274,7 +2281,7 @@
"metadata": {},
"source": [
"$$\n",
- "C(\\boldsymbol{y},\\boldsymbol{f}) = \\sum_{i=0}^{n-1}\\exp{-(y_i(f_{m-1}(x_i)+\\beta G(x_i))}.\n",
+ "C(\\boldsymbol{y},\\boldsymbol{f}) = \\sum_{i=0}^{n-1}\\exp{(-y_i(f_{m-1}(x_i)+\\beta G(x_i))}.\n",
"$$"
]
},
@@ -2291,7 +2298,7 @@
"metadata": {},
"source": [
"$$\n",
- "C(\\boldsymbol{y},\\boldsymbol{f}) = \\sum_{i=0}^{n-1}w_i^{m}\\exp{-(y_i\\beta G(x_i))},\n",
+ "C(\\boldsymbol{y},\\boldsymbol{f}) = \\sum_{i=0}^{n-1}w_i^{m}\\exp{(-y_i\\beta G(x_i))},\n",
"$$"
]
},
@@ -2299,7 +2306,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "where we have defined $w_i^m= \\exp{-(y_if_{m-1}(x_i))}$.\n",
+ "where we have defined $w_i^m= \\exp{(-y_if_{m-1}(x_i))}$.\n",
"\n",
"## Building up AdaBoost\n",
"\n",
@@ -2345,7 +2352,7 @@
"metadata": {},
"source": [
"$$\n",
- "(\\exp{(\\beta)}-\\exp{-(\\beta)})\\sum_{i=0}^{n-1}w_i^mI(y_i\\ne G(x_i))+\\exp{-(\\beta)}\\sum_{i=0}^{n-1}w_i^m=0,\n",
+ "(\\exp{(\\beta)}-\\exp{-(\\beta)})\\sum_{i=0}^{n-1}w_i^mI(y_i\\ne G(x_i))+\\exp{(-\\beta)}\\sum_{i=0}^{n-1}w_i^m=0,\n",
"$$"
]
},
@@ -2409,7 +2416,7 @@
"metadata": {},
"source": [
"$$\n",
- "w_i^{m+1} = w_i^m \\exp{-(y_i\\beta_m G_m(x_i))}\n",
+ "w_i^{m+1} = w_i^m \\exp{(-y_i\\beta_m G_m(x_i))}\n",
"$$"
]
},
@@ -2427,7 +2434,7 @@
"$\\boldsymbol{X}=[\\boldsymbol{x}_0\\boldsymbol{x}_1\\dots\\boldsymbol{x}_{p-1}]$. Finally, we define also a\n",
"classifier determined by our data via a function $G(x)$. This function tells us how well we are able to classify our outputs/targets $\\boldsymbol{y}$. \n",
"\n",
- "We can then define the misclassification error $\\mathrm{err}$ as"
+ "We have already defined the misclassification error $\\mathrm{err}$ as"
]
},
{
@@ -2435,7 +2442,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\mathrm{err}=\\frac{1}{n}\\sum_{i=0}^{n-1}I(y_i\\ne G(x_i),\n",
+ "\\mathrm{err}=\\frac{1}{n}\\sum_{i=0}^{n-1}I(y_i\\ne G(x_i)),\n",
"$$"
]
},
@@ -2637,7 +2644,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Gradient Boosting, Examples of Classification"
+ "## Gradient Boosting, Classification Example"
]
},
{
@@ -2773,7 +2780,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Xgboost on the Cancer Data"
+ "## Xgboost on the Cancer Data\n",
+ "\n",
+ "As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now."
]
},
{
diff --git a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz
index 6de7d4efd..d5bddb5be 100644
Binary files a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz and b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz differ
diff --git a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf
index 4ccea0d81..7ab42a659 100644
Binary files a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf and b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf differ
diff --git a/doc/src/DecisionTrees/DecisionTrees.do.txt b/doc/src/DecisionTrees/DecisionTrees.do.txt
index 0dd7b2336..fd1221da7 100644
--- a/doc/src/DecisionTrees/DecisionTrees.do.txt
+++ b/doc/src/DecisionTrees/DecisionTrees.do.txt
@@ -1710,7 +1710,7 @@ This means that for every iteration, we need to optmize
!bt
\[
-(\beta_m,\gamma_m) \mathrm{armmin}_{\beta,\gambda}\hspace{0.2cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(\gamma_0+\gamma_1 x_i))^2.
+(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\gambda}\hspace{0.2cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(\gamma_0+\gamma_1 x_i))^2.
\[
!et
@@ -1724,17 +1724,23 @@ Taking the derivatives with respect to $\beta$ and $\gamma$ we obtain
and
!bt
\[
-\frac{\partial {\cal C}}{\partial \gamma_0} =
+\frac{\partial {\cal C}}{\partial \gamma_0} =-2\sum_{i}\beta(y_i-\beta(\gamma_0+\gamma_1 x_i))=0,
\]
!et
and
!bt
\[
-\frac{\partial {\cal C}}{\partial \gamma_1} =
+\frac{\partial {\cal C}}{\partial \gamma_1} = =-2\sum_{i}\beta x_i(y_i-\beta(\gamma_0+\gamma_1 x_i))=0.
\]
!et
+!split
+===== Finding the Optimal Parameters =====
+
+With these equations we can then in turn find the parameters $\beta_1$ and $\gamma_0^{1}$ and $\gamma_1^1$ as
+
+
!split
===== Iterative Fitting, Classification, AdaBoost =====
@@ -1747,14 +1753,15 @@ The error rate of the training sample is then
!bt
\[
-\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i).
+\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i)).
\]
!et
The iterative procedure starts with defining a weak classifier whose
-error rate is barely better than random guessing. Teh iterative
+error rate is barely better than random guessing. The iterative
procedure in boosting is to sequentially apply a the weak
-classification algorithm to repeatedly modified versions of the data producing a sequence of weak classifiers $G_m(x)$.
+classification algorithm to repeatedly modified versions of the data
+producing a sequence of weak classifiers $G_m(x)$.
Here we will express our function $f(x)$ in terms of $G(x)$. That is
!bt
@@ -1785,7 +1792,7 @@ The simplest possible cost function which leads (also simple from a computationa
exponential cost/loss function defined as
!bt
\[
-C(\bm{y},\bm{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1}(x_i)+\beta G(x_i))}.
+C(\bm{y},\bm{f}) = \sum_{i=0}^{n-1}\exp{(-y_i(f_{m-1}(x_i)+\beta G(x_i))}.
\]
!et
@@ -1794,10 +1801,10 @@ This is normally done in two steps. Let us however first rewrite the cost functi
!bt
\[
-C(\bm{y},\bm{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{-(y_i\beta G(x_i))},
+C(\bm{y},\bm{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{(-y_i\beta G(x_i))},
\]
!et
-where we have defined $w_i^m= \exp{-(y_if_{m-1}(x_i))}$.
+where we have defined $w_i^m= \exp{(-y_if_{m-1}(x_i))}$.
!split
===== Building up AdaBoost =====
@@ -1819,7 +1826,7 @@ We can do this by rewriting
which can be rewritten as
!bt
\[
-(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{-(\beta)}\sum_{i=0}^{n-1}w_i^m=0,
+(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{(-\beta)}\sum_{i=0}^{n-1}w_i^m=0,
\]
!et
which leads to
@@ -1843,7 +1850,7 @@ f_m(x) = f_{m-1}(x) +\beta_m G_m(x).
This leads to the new weights
!bt
\[
-w_i^{m+1} = w_i^m \exp{-(y_i\beta_m G_m(x_i))}
+w_i^{m+1} = w_i^m \exp{(-y_i\beta_m G_m(x_i))}
\]
!et
!split
@@ -1857,10 +1864,10 @@ feature/predictor vectors
$\bm{X}=[\bm{x}_0\bm{x}_1\dots\bm{x}_{p-1}]$. Finally, we define also a
classifier determined by our data via a function $G(x)$. This function tells us how well we are able to classify our outputs/targets $\bm{y}$.
-We can then define the misclassification error $\mathrm{err}$ as
+We have already defined the misclassification error $\mathrm{err}$ as
!bt
\[
-\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i),
+\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i)),
\]
!et
where the function $I()$ is one if we misclassify and zero if we classify correctly.
@@ -2011,7 +2018,7 @@ plt.show()
!split
-===== Gradient Boosting, Examples of Classification =====
+===== Gradient Boosting, Classification Example =====
!bc pycod
import matplotlib.pyplot as plt
import numpy as np
@@ -2128,6 +2135,8 @@ plt.show()
!split
===== Xgboost on the Cancer Data =====
+
+As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
!bc pycod
import matplotlib.pyplot as plt
Iterative Fitting, Classification, AdaBoost
+Finding the Optimal Parameters
+
+Iterative Fitting, Classification, AdaBoost
$$
-\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i).
+\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i)).
$$
Adaptive Boosting, AdaBoost
+Adaptive Boosting, AdaBoost
$$
-C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1}(x_i)+\beta G(x_i))}.
+C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{(-y_i(f_{m-1}(x_i)+\beta G(x_i))}.
$$
@@ -2174,16 +2183,16 @@ This is normally done in two steps. Let us however first rewrite the cost functi
$$
-C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{-(y_i\beta G(x_i))},
+C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{(-y_i\beta G(x_i))},
$$
-where we have defined \( w_i^m= \exp{-(y_if_{m-1}(x_i))} \).
+where we have defined \( w_i^m= \exp{(-y_if_{m-1}(x_i))} \).
Building up AdaBoost
+Building up AdaBoost
$$
-(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{-(\beta)}\sum_{i=0}^{n-1}w_i^m=0,
+(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{(-\beta)}\sum_{i=0}^{n-1}w_i^m=0,
$$
@@ -2234,14 +2243,14 @@ $$
This leads to the new weights
$$
-w_i^{m+1} = w_i^m \exp{-(y_i\beta_m G_m(x_i))}
+w_i^{m+1} = w_i^m \exp{(-y_i\beta_m G_m(x_i))}
$$
Adaptive boosting: AdaBoost, Basic Algorithm
+Adaptive boosting: AdaBoost, Basic Algorithm
$$
-\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i),
+\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i)),
$$
@@ -2265,7 +2274,7 @@ where the function \( I() \) is one if we misclassify and zero if we classify co
Basic Steps of AdaBoost
+Basic Steps of AdaBoost
AdaBoost Examples
+AdaBoost Examples
Gradient boosting: Basics
+Gradient boosting: Basics
Gradient Boosting, algorithm
+Gradient Boosting, algorithm
Gradient Boosting, Examples of Regression
+Gradient Boosting, Examples of Regression
Gradient Boosting, Examples of Classification
+Gradient Boosting, Classification Example
XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
Regression Case
+Regression Case
Xgboost on the Cancer Data
+Xgboost on the Cancer Data
+
+
-Iterative Fitting, Classification, AdaBoost
+Finding the Optimal Parameters
+
+
+
+Iterative Fitting, Classification, AdaBoost
-Adaptive Boosting, AdaBoost
+Adaptive Boosting, AdaBoost
-Building up AdaBoost
+Building up AdaBoost
-Adaptive boosting: AdaBoost, Basic Algorithm
+Adaptive boosting: AdaBoost, Basic Algorithm
-Basic Steps of AdaBoost
+Basic Steps of AdaBoost
-AdaBoost Examples
+AdaBoost Examples
-Gradient boosting: Basics
+Gradient boosting: Basics
-Gradient Boosting, algorithm
+Gradient Boosting, algorithm
-Gradient Boosting, Examples of Regression
+Gradient Boosting, Examples of Regression
-Gradient Boosting, Examples of Classification
+Gradient Boosting, Classification Example
-XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
-Regression Case
+Regression Case
-Xgboost on the Cancer Data
+Xgboost on the Cancer Data
+
+
-Iterative Fitting, Classification, AdaBoost
+Finding the Optimal Parameters
+
+
+
+Iterative Fitting, Classification, AdaBoost
-Adaptive Boosting, AdaBoost
+Adaptive Boosting, AdaBoost
-Building up AdaBoost
+Building up AdaBoost
-Adaptive boosting: AdaBoost, Basic Algorithm
+Adaptive boosting: AdaBoost, Basic Algorithm
-Basic Steps of AdaBoost
+Basic Steps of AdaBoost
-AdaBoost Examples
+AdaBoost Examples
-Gradient boosting: Basics
+Gradient boosting: Basics
-Gradient Boosting, algorithm
+Gradient Boosting, algorithm
-Gradient Boosting, Examples of Regression
+Gradient Boosting, Examples of Regression
-Gradient Boosting, Examples of Classification
+Gradient Boosting, Classification Example
-XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
-Regression Case
+Regression Case
-Xgboost on the Cancer Data
+Xgboost on the Cancer Data
+
+