-For simplicity we assume also that our functions \( b(x;\gamma)=\gamma_0+\gamma_1 x \), with \( \gamma_0 \) and \( \gamma_1 \) as the parameters to be determined. +For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \).
This means that for every iteration, we need to optmize $$ -(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\gambda}\hspace{0.2cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(\gamma_0+\gamma_1 x_i))^2. -\[ +(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2. $$
-We start our iteration by simply setting \( \f_0(x)=0 \). +We start our iteration by simply setting \( f_0(x)=0 \). Taking the derivatives with respect to \( \beta \) and \( \gamma \) we obtain $$ -\frac{\partial {\cal C}}{\partial \beta} = -2\sum_{i}(\gamma_0+\gamma_1 x_i)(y_i-\beta(\gamma_0+\gamma_1 x_i))=0, +\frac{\partial {\cal C}}{\partial \beta} = -2\sum_{i}(1+\gamma x_i)(y_i-\beta(1+\gamma x_i))=0, $$ and $$ -\frac{\partial {\cal C}}{\partial \gamma_0} =-2\sum_{i}\beta(y_i-\beta(\gamma_0+\gamma_1 x_i))=0, +\frac{\partial {\cal C}}{\partial \gamma} =-2\sum_{i}\beta x_i(y_i-\beta(1+\gamma x_i))=0. $$ -and +We can then rewrite these equations as (defining \( w_i=1+\gamma x_i) \) $$ -\frac{\partial {\cal C}}{\partial \gamma_1} = =-2\sum_{i}\beta x_i(y_i-\beta(\gamma_0+\gamma_1 x_i))=0. +\gamma \boldsymbol{w}^T(\boldsymbol{y}-\beta\gamma \boldsymbol{w})=0, $$ +which gives us \( \beta = \boldsymbol{w}^T\boldsymbol{y}/(\boldsymbol{w}^T\boldsymbol{w}) \). Similarly we have +$$ +\beta\gamma \boldsymbol{x}^T(\boldsymbol{y}-\beta(1+\gamma \boldsymbol{x}))=0, +$$ + +
+which leads \( \gamma =(\boldsymbol{x}^T\boldsymbol{y}-\beta\boldsymbol{x}^T\em{e})/(\beta\boldsymbol{x}^T\boldsymbol{x}) \). Inserting +for \( \beta \) gives us an equation for \( \gamma \). + +
+The solution to these two equations gives us in turn \( \beta_1 \) and \( \gamma_1 \) leading to the new expression for \( f_1(x) \) as +\( f_1(x) = \beta_1(1+\gamma_1x) \). Doing this \( M \) times results in our final estimate for the function \( f \). +
@@ -327,7 +337,7 @@ $$
-With these equations we can then in turn find the parameters \( \beta_1 \) and \( \gamma_0^{1} \) and \( \gamma_1^1 \) as +Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of +observations. We define a classification function \( G(x) \) which produces a prediction taking any of the two values +\( \{-1,1\} \). + +
+The error rate of the training sample is then + +$$ +\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i)). +$$ + +
+The iterative procedure starts with defining a weak classifier whose +error rate is barely better than random guessing. The iterative +procedure in boosting is to sequentially apply a the weak +classification algorithm to repeatedly modified versions of the data +producing a sequence of weak classifiers \( G_m(x) \). + +
+Here we will express our function \( f(x) \) in terms of \( G(x) \). That is +$$ +f_M(x) = \sum_{i=1}^M \beta_m b(x;\gamma_m), +$$ + +will be a function of +$$ +G_M(x) = \mathrm{sign} \sum_{i=1}^M \alpha_m G_m(x). +$$
@@ -299,7 +324,7 @@ With these equations we can then in turn find the parameters \( \beta_1 \) and \
-Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of -observations. We define a classification function \( G(x) \) which produces a prediction taking any of the two values -\( \{-1,1\} \). - -
-The error rate of the training sample is then - +In our iterative procedure we define thus $$ -\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i)). +f_m(x) = f_{m-1}(x)+\beta_mG_m(x). $$
-The iterative procedure starts with defining a weak classifier whose -error rate is barely better than random guessing. The iterative -procedure in boosting is to sequentially apply a the weak -classification algorithm to repeatedly modified versions of the data -producing a sequence of weak classifiers \( G_m(x) \). +The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the +exponential cost/loss function defined as +$$ +C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{(-y_i(f_{m-1}(x_i)+\beta G(x_i))}. +$$
-Here we will express our function \( f(x) \) in terms of \( G(x) \). That is +We optimize \( \beta \) and \( G \) for each value of \( m=1:M \) as we did in the regression case. +This is normally done in two steps. Let us however first rewrite the cost function as + $$ -f_M(x) = \sum_{i=1}^M \beta_m b(x;\gamma_m), +C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{(-y_i\beta G(x_i))}, $$ -will be a function of -$$ -G_M(x) = \mathrm{sign} \sum_{i=1}^M \alpha_m G_m(x). -$$ +where we have defined \( w_i^m= \exp{(-y_if_{m-1}(x_i))} \).
@@ -326,7 +317,7 @@ $$
-In our iterative procedure we define thus +First, for any \( \beta > 0 \), we optimize \( G \) by setting $$ -f_m(x) = f_{m-1}(x)+\beta_mG_m(x). +G_m(x) = \mathrm{sign} \sum_{i=0}^{n-1} w_i^m I(y_i \ne G_(x_i)), $$ +which is the classifier that minimizes the weighted error rate in predicting \( y \). +
-The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the -exponential cost/loss function defined as +We can do this by rewriting $$ -C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}\exp{(-y_i(f_{m-1}(x_i)+\beta G(x_i))}. +\exp{-(\beta)}\sum_{y_i=G(x_i)}w_i^m+\exp{(\beta)}\sum_{y_i\ne G(x_i)}w_i^m, $$ -
-We optimize \( \beta \) and \( G \) for each value of \( m=1:M \) as we did in the regression case. -This is normally done in two steps. Let us however first rewrite the cost function as - +which can be rewritten as $$ -C(\boldsymbol{y},\boldsymbol{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{(-y_i\beta G(x_i))}, +(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{(-\beta)}\sum_{i=0}^{n-1}w_i^m=0, $$ -where we have defined \( w_i^m= \exp{(-y_if_{m-1}(x_i))} \). +which leads to +$$ +\beta_m = \frac{1}{2}\log{\frac{1-\mathrm{\overline{err}}}{\mathrm{\overline{err}}}}, +$$ + +where we have redefined the error as +$$ +\mathrm{\overline{err}}_m=\frac{1}{n}\frac{\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i)}{\sum_{i=0}^{n-1}w_i^m}, +$$ + +which leads to an update of +$$ +f_m(x) = f_{m-1}(x) +\beta_m G_m(x). +$$ + +This leads to the new weights +$$ +w_i^{m+1} = w_i^m \exp{(-y_i\beta_m G_m(x_i))} +$$ -
-First, for any \( \beta > 0 \), we optimize \( G \) by setting -$$ -G_m(x) = \mathrm{sign} \sum_{i=0}^{n-1} w_i^m I(y_i \ne G_(x_i)), -$$ - -which is the classifier that minimizes the weighted error rate in predicting \( y \). +The algorithm here is rather straightforward. Assume that our weak +classifier is a decision tree and we consider a binary set of outputs +with \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of +observations. Our design matrix is given in terms of the +feature/predictor vectors +\( \boldsymbol{X}=[\boldsymbol{x}_0\boldsymbol{x}_1\dots\boldsymbol{x}_{p-1}] \). Finally, we define also a +classifier determined by our data via a function \( G(x) \). This function tells us how well we are able to classify our outputs/targets \( \boldsymbol{y} \).
-We can do this by rewriting +We have already defined the misclassification error \( \mathrm{err} \) as $$ -\exp{-(\beta)}\sum_{y_i=G(x_i)}w_i^m+\exp{(\beta)}\sum_{y_i\ne G(x_i)}w_i^m, +\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i)), $$ -which can be rewritten as -$$ -(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{(-\beta)}\sum_{i=0}^{n-1}w_i^m=0, -$$ - -which leads to -$$ -\beta_m = \frac{1}{2}\log{\frac{1-\mathrm{\overline{err}}}{\mathrm{\overline{err}}}}, -$$ - -where we have redefined the error as -$$ -\mathrm{\overline{err}}_m=\frac{1}{n}\frac{\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i)}{\sum_{i=0}^{n-1}w_i^m}, -$$ - -which leads to an update of -$$ -f_m(x) = f_{m-1}(x) +\beta_m G_m(x). -$$ - -This leads to the new weights -$$ -w_i^{m+1} = w_i^m \exp{(-y_i\beta_m G_m(x_i))} -$$ +where the function \( I() \) is one if we misclassify and zero if we classify correctly. +
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs053.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs053.html index 040d28dd3..bd612b14e 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs053.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs053.html @@ -128,32 +128,31 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec47'), - ('Finding the Optimal Parameters', 2, None, '___sec48'), ('Iterative Fitting, Classification, AdaBoost', 2, None, - '___sec49'), - ('Adaptive Boosting, AdaBoost', 2, None, '___sec50'), - ('Building up AdaBoost', 2, None, '___sec51'), + '___sec48'), + ('Adaptive Boosting, AdaBoost', 2, None, '___sec49'), + ('Building up AdaBoost', 2, None, '___sec50'), ('Adaptive boosting: AdaBoost, Basic Algorithm', 2, None, - '___sec52'), - ('Basic Steps of AdaBoost', 2, None, '___sec53'), - ('AdaBoost Examples', 2, None, '___sec54'), - ('Gradient boosting: Basics', 2, None, '___sec55'), - ('Gradient Boosting, algorithm', 2, None, '___sec56'), + '___sec51'), + ('Basic Steps of AdaBoost', 2, None, '___sec52'), + ('AdaBoost Examples', 2, None, '___sec53'), + ('Gradient boosting: Basics', 2, None, '___sec54'), + ('Gradient Boosting, algorithm', 2, None, '___sec55'), ('Gradient Boosting, Examples of Regression', 2, None, - '___sec57'), + '___sec56'), ('Gradient Boosting, Classification Example', 2, None, - '___sec58'), - ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'), - ('Regression Case', 2, None, '___sec60'), - ('Xgboost on the Cancer Data', 2, None, '___sec61')]} + '___sec57'), + ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'), + ('Regression Case', 2, None, '___sec59'), + ('Xgboost on the Cancer Data', 2, None, '___sec60')]} end of tocinfo --> @@ -239,20 +238,19 @@ MathJax.Hub.Config({
-The algorithm here is rather straightforward. Assume that our weak -classifier is a decision tree and we consider a binary set of outputs -with \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of -observations. Our design matrix is given in terms of the -feature/predictor vectors -\( \boldsymbol{X}=[\boldsymbol{x}_0\boldsymbol{x}_1\dots\boldsymbol{x}_{p-1}] \). Finally, we define also a -classifier determined by our data via a function \( G(x) \). This function tells us how well we are able to classify our outputs/targets \( \boldsymbol{y} \). +With the above definitions we are now ready to set up the algorithm for AdaBoost. +The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases. + +
-We have already defined the misclassification error \( \mathrm{err} \) as $$ -\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i)), +\mathrm{\overline{err}}_m=\frac{\sum_{i=0}^{n-1}w_i^m I(y_i\ne G(x_i))}{\sum_{i=0}^{n-1}w_i}, $$ -where the function \( I() \) is one if we misclassify and zero if we classify correctly. + +
@@ -311,7 +327,6 @@ where the function \( I() \) is one if we misclassify and zero if we classify co
-With the above definitions we are now ready to set up the algorithm for AdaBoost. -The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases. +Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here. -
-$$ -\mathrm{\overline{err}}_m=\frac{\sum_{i=0}^{n-1}w_i^m I(y_i\ne G(x_i))}{\sum_{i=0}^{n-1}w_i}, -$$ + +
from sklearn.ensemble import AdaBoostClassifier
+ada_clf = AdaBoostClassifier(
+ DecisionTreeClassifier(max_depth=1), n_estimators=200,
+ algorithm="SAMME.R", learning_rate=0.5, random_state=42)
+ada_clf.fit(X_train, y_train)
-
-- Then we start looping over all attempts at classifying, namely we start an iterative process for \( m=1:M \), where \( M \) is the final number of classifications. Our given classifier could for example be a plain decision tree.
-
-
- - Fit then a given classifier to the training using the weights \( w_i \).
- - Compute then \( \mathrm{err} \) and figure out which events are classified properly and which are classified wrongly.
- - Define a quantity \( \alpha_{m} = \log{(1-\mathrm{\overline{err}}_m)/\mathrm{\overline{err}}_m} \)
- - Set the new weights to \( w_i = w_i\times \exp{(\alpha_m I(y_i\ne G(x_i)} \).
-
-
-
@@ -328,7 +319,6 @@ observations that are missed in the previous iterations.
-Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here. +Gradient boosting is again a similar technique to Adapative boosting, +it combines so-called weak classifiers or regressors into a strong +method via a series of iterations.
+In order to understand the method, let us illustrate its basics by +bringing back the essential steps in linear regression, where our cost +function was the least squares function. - -
from sklearn.ensemble import AdaBoostClassifier
+
+See discussion during lecture November 8.
-ada_clf = AdaBoostClassifier(
- DecisionTreeClassifier(max_depth=1), n_estimators=200,
- algorithm="SAMME.R", learning_rate=0.5, random_state=42)
-ada_clf.fit(X_train, y_train)
-
-from sklearn.ensemble import AdaBoostClassifier
-
-ada_clf = AdaBoostClassifier(
- DecisionTreeClassifier(max_depth=1), n_estimators=200,
- algorithm="SAMME.R", learning_rate=0.5, random_state=42)
-ada_clf.fit(X_train_scaled, y_train)
-y_pred = ada_clf.predict(X_test_scaled)
-skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
-plt.show()
-y_probas = ada_clf.predict_proba(X_test_scaled)
-skplt.metrics.plot_roc(y_test, y_probas)
-plt.show()
-skplt.metrics.plot_cumulative_gain(y_test, y_probas)
-plt.show()
-
@@ -320,7 +303,6 @@ plt.show()
-Gradient boosting is again a similar technique to Adapative boosting, -it combines so-called weak classifiers or regressors into a strong -method via a series of iterations. +Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard square-error function +$$ +C(\boldsymbol{y},\boldsymbol{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2. +$$
-In order to understand the method, let us illustrate its basics by -bringing back the essential steps in linear regression, where our cost -function was the least squares function. +The way we proceed in an iterative fashion is to -
-See discussion during lecture November 8. +
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html index 8975c7810..e35744ad7 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html @@ -128,32 +128,31 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec47'), - ('Finding the Optimal Parameters', 2, None, '___sec48'), ('Iterative Fitting, Classification, AdaBoost', 2, None, - '___sec49'), - ('Adaptive Boosting, AdaBoost', 2, None, '___sec50'), - ('Building up AdaBoost', 2, None, '___sec51'), + '___sec48'), + ('Adaptive Boosting, AdaBoost', 2, None, '___sec49'), + ('Building up AdaBoost', 2, None, '___sec50'), ('Adaptive boosting: AdaBoost, Basic Algorithm', 2, None, - '___sec52'), - ('Basic Steps of AdaBoost', 2, None, '___sec53'), - ('AdaBoost Examples', 2, None, '___sec54'), - ('Gradient boosting: Basics', 2, None, '___sec55'), - ('Gradient Boosting, algorithm', 2, None, '___sec56'), + '___sec51'), + ('Basic Steps of AdaBoost', 2, None, '___sec52'), + ('AdaBoost Examples', 2, None, '___sec53'), + ('Gradient boosting: Basics', 2, None, '___sec54'), + ('Gradient Boosting, algorithm', 2, None, '___sec55'), ('Gradient Boosting, Examples of Regression', 2, None, - '___sec57'), + '___sec56'), ('Gradient Boosting, Classification Example', 2, None, - '___sec58'), - ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'), - ('Regression Case', 2, None, '___sec60'), - ('Xgboost on the Cancer Data', 2, None, '___sec61')]} + '___sec57'), + ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'), + ('Regression Case', 2, None, '___sec59'), + ('Xgboost on the Cancer Data', 2, None, '___sec60')]} end of tocinfo --> @@ -239,20 +238,19 @@ MathJax.Hub.Config({
-Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard square-error function -$$ -C(\boldsymbol{y},\boldsymbol{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2. -$$ + +
import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.ensemble import GradientBoostingRegressor
+from sklearn.preprocessing import StandardScaler
+import scikitplot as skplt
+from sklearn.metrics import mean_squared_error
+
+n = 100
+maxdegree = 6
+
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+
+error = np.zeros(maxdegree)
+bias = np.zeros(maxdegree)
+variance = np.zeros(maxdegree)
+polydegree = np.zeros(maxdegree)
+X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+
+for degree in range(1,maxdegree):
+ model = GradientBoostingRegressor(max_depth=degree, n_estimators=100, learning_rate=1.0)
+ model.fit(X_train_scaled,y_train)
+ y_pred = model.predict(X_test_scaled)
+ polydegree[degree] = degree
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
+ variance[degree] = np.mean( np.var(y_pred) )
+ print('Max depth:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+
+plt.xlim(1,maxdegree-1)
+plt.plot(polydegree, error, label='Error')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+save_fig("gdregression")
+plt.show()
+-The way we proceed in an iterative fashion is to - -
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html index ddda5b9cf..2cda0d1a6 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html @@ -128,32 +128,31 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec47'), - ('Finding the Optimal Parameters', 2, None, '___sec48'), ('Iterative Fitting, Classification, AdaBoost', 2, None, - '___sec49'), - ('Adaptive Boosting, AdaBoost', 2, None, '___sec50'), - ('Building up AdaBoost', 2, None, '___sec51'), + '___sec48'), + ('Adaptive Boosting, AdaBoost', 2, None, '___sec49'), + ('Building up AdaBoost', 2, None, '___sec50'), ('Adaptive boosting: AdaBoost, Basic Algorithm', 2, None, - '___sec52'), - ('Basic Steps of AdaBoost', 2, None, '___sec53'), - ('AdaBoost Examples', 2, None, '___sec54'), - ('Gradient boosting: Basics', 2, None, '___sec55'), - ('Gradient Boosting, algorithm', 2, None, '___sec56'), + '___sec51'), + ('Basic Steps of AdaBoost', 2, None, '___sec52'), + ('AdaBoost Examples', 2, None, '___sec53'), + ('Gradient boosting: Basics', 2, None, '___sec54'), + ('Gradient Boosting, algorithm', 2, None, '___sec55'), ('Gradient Boosting, Examples of Regression', 2, None, - '___sec57'), + '___sec56'), ('Gradient Boosting, Classification Example', 2, None, - '___sec58'), - ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'), - ('Regression Case', 2, None, '___sec60'), - ('Xgboost on the Cancer Data', 2, None, '___sec61')]} + '___sec57'), + ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'), + ('Regression Case', 2, None, '___sec59'), + ('Xgboost on the Cancer Data', 2, None, '___sec60')]} end of tocinfo --> @@ -239,20 +238,19 @@ MathJax.Hub.Config({
import matplotlib.pyplot as plt
import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.ensemble import GradientBoostingRegressor
-from sklearn.preprocessing import StandardScaler
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import load_breast_cancer
import scikitplot as skplt
-from sklearn.metrics import mean_squared_error
+from sklearn.ensemble import GradientBoostingClassifier
+from sklearn.model_selection import cross_validate
-n = 100
-maxdegree = 6
+# Load the data
+cancer = load_breast_cancer()
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-
-error = np.zeros(maxdegree)
-bias = np.zeros(maxdegree)
-variance = np.zeros(maxdegree)
-polydegree = np.zeros(maxdegree)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
+print(X_train.shape)
+print(X_test.shape)
+#now scale the data
+from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
-for degree in range(1,maxdegree):
- model = GradientBoostingRegressor(max_depth=degree, n_estimators=100, learning_rate=1.0)
- model.fit(X_train_scaled,y_train)
- y_pred = model.predict(X_test_scaled)
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
- variance[degree] = np.mean( np.var(y_pred) )
- print('Max depth:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+gd_clf = GradientBoostingClassifier(max_depth=3, n_estimators=100, learning_rate=1.0)
+gd_clf.fit(X_train_scaled, y_train)
+#Cross validation
+accuracy = cross_validate(gd_clf,X_test_scaled,y_test,cv=10)['test_score']
+print(accuracy)
+print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(gd_clf.score(X_test_scaled,y_test)))
-plt.xlim(1,maxdegree-1)
-plt.plot(polydegree, error, label='Error')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
-save_fig("gdregression")
+import scikitplot as skplt
+y_pred = gd_clf.predict(X_test_scaled)
+skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
+save_fig("gdclassiffierconfusion")
+plt.show()
+y_probas = gd_clf.predict_proba(X_test_scaled)
+skplt.metrics.plot_roc(y_test, y_probas)
+save_fig("gdclassiffierroc")
+plt.show()
+skplt.metrics.plot_cumulative_gain(y_test, y_probas)
+save_fig("gdclassiffiercgain")
plt.show()
@@ -338,7 +330,6 @@ plt.show()
+XGBoost or Extreme Gradient +Boosting, is an optimized distributed gradient boosting library +designed to be highly efficient, flexible and portable. It implements +machine learning algorithms under the Gradient Boosting +framework. XGBoost provides a parallel tree boosting that solve many +data science problems in a fast and accurate way. See the article by Chen and Guestrin. - -
import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import load_breast_cancer
-import scikitplot as skplt
-from sklearn.ensemble import GradientBoostingClassifier
-from sklearn.model_selection import cross_validate
+
+The authors design and build a highly scalable end-to-end tree
+boosting system. It has a theoretically justified weighted quantile
+sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning.
-# Load the data
-cancer = load_breast_cancer()
+
+It is now the algorithm which wins essentially all ML competitions!!!
-X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
-print(X_train.shape)
-print(X_test.shape)
-#now scale the data
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-gd_clf = GradientBoostingClassifier(max_depth=3, n_estimators=100, learning_rate=1.0)
-gd_clf.fit(X_train_scaled, y_train)
-#Cross validation
-accuracy = cross_validate(gd_clf,X_test_scaled,y_test,cv=10)['test_score']
-print(accuracy)
-print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(gd_clf.score(X_test_scaled,y_test)))
-
-import scikitplot as skplt
-y_pred = gd_clf.predict(X_test_scaled)
-skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
-save_fig("gdclassiffierconfusion")
-plt.show()
-y_probas = gd_clf.predict_proba(X_test_scaled)
-skplt.metrics.plot_roc(y_test, y_probas)
-save_fig("gdclassiffierroc")
-plt.show()
-skplt.metrics.plot_cumulative_gain(y_test, y_probas)
-save_fig("gdclassiffiercgain")
-plt.show()
-
@@ -331,7 +302,6 @@ plt.show()
-XGBoost or Extreme Gradient -Boosting, is an optimized distributed gradient boosting library -designed to be highly efficient, flexible and portable. It implements -machine learning algorithms under the Gradient Boosting -framework. XGBoost provides a parallel tree boosting that solve many -data science problems in a fast and accurate way. See the article by Chen and Guestrin. -
-The authors design and build a highly scalable end-to-end tree -boosting system. It has a theoretically justified weighted quantile -sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning. + +
import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+import xgboost as xgb
+from sklearn.preprocessing import StandardScaler
+import scikitplot as skplt
+from sklearn.metrics import mean_squared_error
-
-It is now the algorithm which wins essentially all ML competitions!!!
+n = 100
+maxdegree = 6
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+
+error = np.zeros(maxdegree)
+bias = np.zeros(maxdegree)
+variance = np.zeros(maxdegree)
+polydegree = np.zeros(maxdegree)
+X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+
+for degree in range(maxdegree):
+ model = xgb.XGBRegressor(objective ='reg:squarederror', colsaobjective ='reg:squarederror', colsample_bytree = 0.3, learning_rate = 0.1,max_depth = degree, alpha = 10, n_estimators = 200)
+
+ model.fit(X_train_scaled,y_train)
+ y_pred = model.predict(X_test_scaled)
+ polydegree[degree] = degree
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
+ variance[degree] = np.mean( np.var(y_pred) )
+ print('Max depth:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+
+plt.xlim(1,maxdegree-1)
+plt.plot(polydegree, error, label='Error')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+plt.show()
+
@@ -303,7 +335,6 @@ It is now the algorithm which wins essentially all ML competitions!!!
-For simplicity we assume also that our functions \( b(x;\gamma)=\gamma_0+\gamma_1 x \), with \( \gamma_0 \) and \( \gamma_1 \) as the parameters to be determined. +For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \).
This means that for every iteration, we need to optmize
$$
-(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\gambda}\hspace{0.2cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(\gamma_0+\gamma_1 x_i))^2.
-\[
+(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
$$
-We start our iteration by simply setting \( \f_0(x)=0 \). +We start our iteration by simply setting \( f_0(x)=0 \). Taking the derivatives with respect to \( \beta \) and \( \gamma \) we obtain
$$
-\frac{\partial {\cal C}}{\partial \beta} = -2\sum_{i}(\gamma_0+\gamma_1 x_i)(y_i-\beta(\gamma_0+\gamma_1 x_i))=0,
+\frac{\partial {\cal C}}{\partial \beta} = -2\sum_{i}(1+\gamma x_i)(y_i-\beta(1+\gamma x_i))=0,
$$
and
$$
-\frac{\partial {\cal C}}{\partial \gamma_0} =-2\sum_{i}\beta(y_i-\beta(\gamma_0+\gamma_1 x_i))=0,
+\frac{\partial {\cal C}}{\partial \gamma} =-2\sum_{i}\beta x_i(y_i-\beta(1+\gamma x_i))=0.
$$
-and
+We can then rewrite these equations as (defining \( w_i=1+\gamma x_i) \)
$$
-\frac{\partial {\cal C}}{\partial \gamma_1} = =-2\sum_{i}\beta x_i(y_i-\beta(\gamma_0+\gamma_1 x_i))=0.
+\gamma \boldsymbol{w}^T(\boldsymbol{y}-\beta\gamma \boldsymbol{w})=0,
$$
-With these equations we can then in turn find the parameters \( \beta_1 \) and \( \gamma_0^{1} \) and \( \gamma_1^1 \) as
+which leads \( \gamma =(\boldsymbol{x}^T\boldsymbol{y}-\beta\boldsymbol{x}^T\em{e})/(\beta\boldsymbol{x}^T\boldsymbol{x}) \). Inserting
+for \( \beta \) gives us an equation for \( \gamma \).
+
+
+The solution to these two equations gives us in turn \( \beta_1 \) and \( \gamma_1 \) leading to the new expression for \( f_1(x) \) as
+\( f_1(x) = \beta_1(1+\gamma_1x) \). Doing this \( M \) times results in our final estimate for the function \( f \).
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
@@ -2158,7 +2164,7 @@ $$
In our iterative procedure we define thus
@@ -2192,7 +2198,7 @@ where we have defined \( w_i^m= \exp{(-y_if_{m-1}(x_i))} \).
First, for any \( \beta > 0 \), we optimize \( G \) by setting
@@ -2250,7 +2256,7 @@ $$
The algorithm here is rather straightforward. Assume that our weak
@@ -2274,7 +2280,7 @@ where the function \( I() \) is one if we misclassify and zero if we classify co
With the above definitions we are now ready to set up the algorithm for AdaBoost.
@@ -2315,7 +2321,7 @@ observations that are missed in the previous iterations.
Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here.
@@ -2349,7 +2355,7 @@ plt.show()
Gradient boosting is again a similar technique to Adapative boosting,
@@ -2367,7 +2373,7 @@ See discussion during lecture November 8.
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard square-error function
@@ -2395,7 +2401,7 @@ The way we proceed in an iterative fashion is to
@@ -2450,7 +2456,7 @@ plt.show()
@@ -2499,7 +2505,7 @@ plt.show()
XGBoost or Extreme Gradient
@@ -2520,7 +2526,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
@@ -2576,7 +2582,7 @@ plt.show()
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html b/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
index e0c9995ac..acfe40fa5 100644
--- a/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
+++ b/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
@@ -148,32 +148,31 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec47'),
- ('Finding the Optimal Parameters', 2, None, '___sec48'),
('Iterative Fitting, Classification, AdaBoost',
2,
None,
- '___sec49'),
- ('Adaptive Boosting, AdaBoost', 2, None, '___sec50'),
- ('Building up AdaBoost', 2, None, '___sec51'),
+ '___sec48'),
+ ('Adaptive Boosting, AdaBoost', 2, None, '___sec49'),
+ ('Building up AdaBoost', 2, None, '___sec50'),
('Adaptive boosting: AdaBoost, Basic Algorithm',
2,
None,
- '___sec52'),
- ('Basic Steps of AdaBoost', 2, None, '___sec53'),
- ('AdaBoost Examples', 2, None, '___sec54'),
- ('Gradient boosting: Basics', 2, None, '___sec55'),
- ('Gradient Boosting, algorithm', 2, None, '___sec56'),
+ '___sec51'),
+ ('Basic Steps of AdaBoost', 2, None, '___sec52'),
+ ('AdaBoost Examples', 2, None, '___sec53'),
+ ('Gradient boosting: Basics', 2, None, '___sec54'),
+ ('Gradient Boosting, algorithm', 2, None, '___sec55'),
('Gradient Boosting, Examples of Regression',
2,
None,
- '___sec57'),
+ '___sec56'),
('Gradient Boosting, Classification Example',
2,
None,
- '___sec58'),
- ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'),
- ('Regression Case', 2, None, '___sec60'),
- ('Xgboost on the Cancer Data', 2, None, '___sec61')]}
+ '___sec57'),
+ ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'),
+ ('Regression Case', 2, None, '___sec59'),
+ ('Xgboost on the Cancer Data', 2, None, '___sec60')]}
end of tocinfo -->
-For simplicity we assume also that our functions \( b(x;\gamma)=\gamma_0+\gamma_1 x \), with \( \gamma_0 \) and \( \gamma_1 \) as the parameters to be determined.
+For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \).
This means that for every iteration, we need to optmize
$$
-(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\gambda}\hspace{0.2cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(\gamma_0+\gamma_1 x_i))^2.
-\[
+(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
$$
-We start our iteration by simply setting \( \f_0(x)=0 \).
+We start our iteration by simply setting \( f_0(x)=0 \).
Taking the derivatives with respect to \( \beta \) and \( \gamma \) we obtain
$$
-\frac{\partial {\cal C}}{\partial \beta} = -2\sum_{i}(\gamma_0+\gamma_1 x_i)(y_i-\beta(\gamma_0+\gamma_1 x_i))=0,
+\frac{\partial {\cal C}}{\partial \beta} = -2\sum_{i}(1+\gamma x_i)(y_i-\beta(1+\gamma x_i))=0,
$$
and
$$
-\frac{\partial {\cal C}}{\partial \gamma_0} =-2\sum_{i}\beta(y_i-\beta(\gamma_0+\gamma_1 x_i))=0,
+\frac{\partial {\cal C}}{\partial \gamma} =-2\sum_{i}\beta x_i(y_i-\beta(1+\gamma x_i))=0.
$$
-and
+We can then rewrite these equations as (defining \( w_i=1+\gamma x_i) \)
$$
-\frac{\partial {\cal C}}{\partial \gamma_1} = =-2\sum_{i}\beta x_i(y_i-\beta(\gamma_0+\gamma_1 x_i))=0.
+\gamma \boldsymbol{w}^T(\boldsymbol{y}-\beta\gamma \boldsymbol{w})=0,
$$
+which gives us \( \beta = \boldsymbol{w}^T\boldsymbol{y}/(\boldsymbol{w}^T\boldsymbol{w}) \). Similarly we have
+$$
+\beta\gamma \boldsymbol{x}^T(\boldsymbol{y}-\beta(1+\gamma \boldsymbol{x}))=0,
+$$
+
+
+which leads \( \gamma =(\boldsymbol{x}^T\boldsymbol{y}-\beta\boldsymbol{x}^T\em{e})/(\beta\boldsymbol{x}^T\boldsymbol{x}) \). Inserting
+for \( \beta \) gives us an equation for \( \gamma \).
+
+
+The solution to these two equations gives us in turn \( \beta_1 \) and \( \gamma_1 \) leading to the new expression for \( f_1(x) \) as
+\( f_1(x) = \beta_1(1+\gamma_1x) \). Doing this \( M \) times results in our final estimate for the function \( f \).
+
-With these equations we can then in turn find the parameters \( \beta_1 \) and \( \gamma_0^{1} \) and \( \gamma_1^1 \) as
-
-
-
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
@@ -2147,7 +2150,7 @@ $$
In our iterative procedure we define thus
@@ -2175,7 +2178,7 @@ where we have defined \( w_i^m= \exp{(-y_if_{m-1}(x_i))} \).
First, for any \( \beta > 0 \), we optimize \( G \) by setting
@@ -2218,7 +2221,7 @@ $$
The algorithm here is rather straightforward. Assume that our weak
@@ -2240,7 +2243,7 @@ where the function \( I() \) is one if we misclassify and zero if we classify co
With the above definitions we are now ready to set up the algorithm for AdaBoost.
@@ -2280,7 +2283,7 @@ observations that are missed in the previous iterations.
Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here.
@@ -2313,7 +2316,7 @@ plt.show()
Gradient boosting is again a similar technique to Adapative boosting,
@@ -2331,7 +2334,7 @@ See discussion during lecture November 8.
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard square-error function
@@ -2357,7 +2360,7 @@ The way we proceed in an iterative fashion is to
@@ -2411,7 +2414,7 @@ plt.show()
@@ -2459,7 +2462,7 @@ plt.show()
XGBoost or Extreme Gradient
@@ -2480,7 +2483,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
@@ -2535,7 +2538,7 @@ plt.show()
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees.html b/doc/pub/DecisionTrees/html/DecisionTrees.html
index d038bafd1..1c1b06da5 100644
--- a/doc/pub/DecisionTrees/html/DecisionTrees.html
+++ b/doc/pub/DecisionTrees/html/DecisionTrees.html
@@ -153,32 +153,31 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec47'),
- ('Finding the Optimal Parameters', 2, None, '___sec48'),
('Iterative Fitting, Classification, AdaBoost',
2,
None,
- '___sec49'),
- ('Adaptive Boosting, AdaBoost', 2, None, '___sec50'),
- ('Building up AdaBoost', 2, None, '___sec51'),
+ '___sec48'),
+ ('Adaptive Boosting, AdaBoost', 2, None, '___sec49'),
+ ('Building up AdaBoost', 2, None, '___sec50'),
('Adaptive boosting: AdaBoost, Basic Algorithm',
2,
None,
- '___sec52'),
- ('Basic Steps of AdaBoost', 2, None, '___sec53'),
- ('AdaBoost Examples', 2, None, '___sec54'),
- ('Gradient boosting: Basics', 2, None, '___sec55'),
- ('Gradient Boosting, algorithm', 2, None, '___sec56'),
+ '___sec51'),
+ ('Basic Steps of AdaBoost', 2, None, '___sec52'),
+ ('AdaBoost Examples', 2, None, '___sec53'),
+ ('Gradient boosting: Basics', 2, None, '___sec54'),
+ ('Gradient Boosting, algorithm', 2, None, '___sec55'),
('Gradient Boosting, Examples of Regression',
2,
None,
- '___sec57'),
+ '___sec56'),
('Gradient Boosting, Classification Example',
2,
None,
- '___sec58'),
- ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'),
- ('Regression Case', 2, None, '___sec60'),
- ('Xgboost on the Cancer Data', 2, None, '___sec61')]}
+ '___sec57'),
+ ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'),
+ ('Regression Case', 2, None, '___sec59'),
+ ('Xgboost on the Cancer Data', 2, None, '___sec60')]}
end of tocinfo -->
-For simplicity we assume also that our functions \( b(x;\gamma)=\gamma_0+\gamma_1 x \), with \( \gamma_0 \) and \( \gamma_1 \) as the parameters to be determined.
+For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \).
This means that for every iteration, we need to optmize
$$
-(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\gambda}\hspace{0.2cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(\gamma_0+\gamma_1 x_i))^2.
-\[
+(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
$$
-We start our iteration by simply setting \( \f_0(x)=0 \).
+We start our iteration by simply setting \( f_0(x)=0 \).
Taking the derivatives with respect to \( \beta \) and \( \gamma \) we obtain
$$
-\frac{\partial {\cal C}}{\partial \beta} = -2\sum_{i}(\gamma_0+\gamma_1 x_i)(y_i-\beta(\gamma_0+\gamma_1 x_i))=0,
+\frac{\partial {\cal C}}{\partial \beta} = -2\sum_{i}(1+\gamma x_i)(y_i-\beta(1+\gamma x_i))=0,
$$
and
$$
-\frac{\partial {\cal C}}{\partial \gamma_0} =-2\sum_{i}\beta(y_i-\beta(\gamma_0+\gamma_1 x_i))=0,
+\frac{\partial {\cal C}}{\partial \gamma} =-2\sum_{i}\beta x_i(y_i-\beta(1+\gamma x_i))=0.
$$
-and
+We can then rewrite these equations as (defining \( w_i=1+\gamma x_i) \)
$$
-\frac{\partial {\cal C}}{\partial \gamma_1} = =-2\sum_{i}\beta x_i(y_i-\beta(\gamma_0+\gamma_1 x_i))=0.
+\gamma \boldsymbol{w}^T(\boldsymbol{y}-\beta\gamma \boldsymbol{w})=0,
$$
+which gives us \( \beta = \boldsymbol{w}^T\boldsymbol{y}/(\boldsymbol{w}^T\boldsymbol{w}) \). Similarly we have
+$$
+\beta\gamma \boldsymbol{x}^T(\boldsymbol{y}-\beta(1+\gamma \boldsymbol{x}))=0,
+$$
+
+
+which leads \( \gamma =(\boldsymbol{x}^T\boldsymbol{y}-\beta\boldsymbol{x}^T\em{e})/(\beta\boldsymbol{x}^T\boldsymbol{x}) \). Inserting
+for \( \beta \) gives us an equation for \( \gamma \).
+
+
+The solution to these two equations gives us in turn \( \beta_1 \) and \( \gamma_1 \) leading to the new expression for \( f_1(x) \) as
+\( f_1(x) = \beta_1(1+\gamma_1x) \). Doing this \( M \) times results in our final estimate for the function \( f \).
+
-With these equations we can then in turn find the parameters \( \beta_1 \) and \( \gamma_0^{1} \) and \( \gamma_1^1 \) as
-
-
-
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
@@ -2152,7 +2155,7 @@ $$
In our iterative procedure we define thus
@@ -2180,7 +2183,7 @@ where we have defined \( w_i^m= \exp{(-y_if_{m-1}(x_i))} \).
First, for any \( \beta > 0 \), we optimize \( G \) by setting
@@ -2223,7 +2226,7 @@ $$
The algorithm here is rather straightforward. Assume that our weak
@@ -2245,7 +2248,7 @@ where the function \( I() \) is one if we misclassify and zero if we classify co
With the above definitions we are now ready to set up the algorithm for AdaBoost.
@@ -2285,7 +2288,7 @@ observations that are missed in the previous iterations.
Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here.
@@ -2318,7 +2321,7 @@ plt.show()
Gradient boosting is again a similar technique to Adapative boosting,
@@ -2336,7 +2339,7 @@ See discussion during lecture November 8.
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard square-error function
@@ -2362,7 +2365,7 @@ The way we proceed in an iterative fashion is to
@@ -2416,7 +2419,7 @@ plt.show()
@@ -2464,7 +2467,7 @@ plt.show()
XGBoost or Extreme Gradient
@@ -2485,7 +2488,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
@@ -2540,7 +2543,7 @@ plt.show()
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
diff --git a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
index 752b3f244..a1baa3fae 100644
--- a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
+++ b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
@@ -2121,7 +2121,7 @@
"\n",
"To better understand what happens, let us develop the steps for the iterative fitting using the above squared error function.\n",
"\n",
- "For simplicity we assume also that our functions $b(x;\\gamma)=\\gamma_0+\\gamma_1 x$, with $\\gamma_0$ and $\\gamma_1$ as the parameters to be determined.\n",
+ "For simplicity we assume also that our functions $b(x;\\gamma)=1+\\gamma x$. \n",
"\n",
"This means that for every iteration, we need to optmize"
]
@@ -2131,7 +2131,7 @@
"metadata": {},
"source": [
"$$\n",
- "(\\beta_m,\\gamma_m) \\mathrm{argmin}_{\\beta,\\gambda}\\hspace{0.2cm} \\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta b(x;\\gamma))^2\\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta(\\gamma_0+\\gamma_1 x_i))^2.\n",
+ "(\\beta_m,\\gamma_m) \\mathrm{argmin}_{\\beta,\\lambda}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta b(x;\\gamma))^2=\\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta(1+\\gamma x_i))^2.\n",
"$$"
]
},
@@ -2139,7 +2139,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "We start our iteration by simply setting $\\f_0(x)=0$. \n",
+ "We start our iteration by simply setting $f_0(x)=0$. \n",
"Taking the derivatives with respect to $\\beta$ and $\\gamma$ we obtain"
]
},
@@ -2148,7 +2148,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\frac{\\partial {\\cal C}}{\\partial \\beta} = -2\\sum_{i}(\\gamma_0+\\gamma_1 x_i)(y_i-\\beta(\\gamma_0+\\gamma_1 x_i))=0,\n",
+ "\\frac{\\partial {\\cal C}}{\\partial \\beta} = -2\\sum_{i}(1+\\gamma x_i)(y_i-\\beta(1+\\gamma x_i))=0,\n",
"$$"
]
},
@@ -2164,7 +2164,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\frac{\\partial {\\cal C}}{\\partial \\gamma_0} =-2\\sum_{i}\\beta(y_i-\\beta(\\gamma_0+\\gamma_1 x_i))=0,\n",
+ "\\frac{\\partial {\\cal C}}{\\partial \\gamma} =-2\\sum_{i}\\beta x_i(y_i-\\beta(1+\\gamma x_i))=0.\n",
"$$"
]
},
@@ -2172,7 +2172,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "and"
+ "We can then rewrite these equations as (defining $w_i=1+\\gamma x_i)$"
]
},
{
@@ -2180,7 +2180,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\frac{\\partial {\\cal C}}{\\partial \\gamma_1} = =-2\\sum_{i}\\beta x_i(y_i-\\beta(\\gamma_0+\\gamma_1 x_i))=0.\n",
+ "\\gamma \\boldsymbol{w}^T(\\boldsymbol{y}-\\beta\\gamma \\boldsymbol{w})=0,\n",
"$$"
]
},
@@ -2188,9 +2188,27 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Finding the Optimal Parameters\n",
+ "which gives us $\\beta = \\boldsymbol{w}^T\\boldsymbol{y}/(\\boldsymbol{w}^T\\boldsymbol{w})$. Similarly we have"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\beta\\gamma \\boldsymbol{x}^T(\\boldsymbol{y}-\\beta(1+\\gamma \\boldsymbol{x}))=0,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "which leads $\\gamma =(\\boldsymbol{x}^T\\boldsymbol{y}-\\beta\\boldsymbol{x}^T\\em{e})/(\\beta\\boldsymbol{x}^T\\boldsymbol{x})$. Inserting\n",
+ "for $\\beta$ gives us an equation for $\\gamma$.\n",
"\n",
- "With these equations we can then in turn find the parameters $\\beta_1$ and $\\gamma_0^{1}$ and $\\gamma_1^1$ as\n",
+ "The solution to these two equations gives us in turn $\\beta_1$ and $\\gamma_1$ leading to the new expression for $f_1(x)$ as\n",
+ "$f_1(x) = \\beta_1(1+\\gamma_1x)$. Doing this $M$ times results in our final estimate for the function $f$. \n",
"\n",
"\n",
"\n",
diff --git a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz
index d5bddb5be..2a8f44d75 100644
Binary files a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz and b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz differ
diff --git a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf
index 7ab42a659..bfe73b849 100644
Binary files a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf and b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf differ
diff --git a/doc/src/DecisionTrees/DecisionTrees.do.txt b/doc/src/DecisionTrees/DecisionTrees.do.txt
index fd1221da7..583e64200 100644
--- a/doc/src/DecisionTrees/DecisionTrees.do.txt
+++ b/doc/src/DecisionTrees/DecisionTrees.do.txt
@@ -1704,41 +1704,47 @@ at the internal nodes, and the predictions at the terminal nodes.
To better understand what happens, let us develop the steps for the iterative fitting using the above squared error function.
-For simplicity we assume also that our functions $b(x;\gamma)=\gamma_0+\gamma_1 x$, with $\gamma_0$ and $\gamma_1$ as the parameters to be determined.
+For simplicity we assume also that our functions $b(x;\gamma)=1+\gamma x$.
This means that for every iteration, we need to optmize
!bt
\[
-(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\gambda}\hspace{0.2cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(\gamma_0+\gamma_1 x_i))^2.
-\[
+(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
+\]
!et
-We start our iteration by simply setting $\f_0(x)=0$.
+We start our iteration by simply setting $f_0(x)=0$.
Taking the derivatives with respect to $\beta$ and $\gamma$ we obtain
!bt
\[
-\frac{\partial {\cal C}}{\partial \beta} = -2\sum_{i}(\gamma_0+\gamma_1 x_i)(y_i-\beta(\gamma_0+\gamma_1 x_i))=0,
+\frac{\partial {\cal C}}{\partial \beta} = -2\sum_{i}(1+\gamma x_i)(y_i-\beta(1+\gamma x_i))=0,
\]
!et
and
!bt
\[
-\frac{\partial {\cal C}}{\partial \gamma_0} =-2\sum_{i}\beta(y_i-\beta(\gamma_0+\gamma_1 x_i))=0,
+\frac{\partial {\cal C}}{\partial \gamma} =-2\sum_{i}\beta x_i(y_i-\beta(1+\gamma x_i))=0.
\]
!et
-and
+We can then rewrite these equations as (defining $w_i=1+\gamma x_i)$
!bt
\[
-\frac{\partial {\cal C}}{\partial \gamma_1} = =-2\sum_{i}\beta x_i(y_i-\beta(\gamma_0+\gamma_1 x_i))=0.
+\gamma \bm{w}^T(\bm{y}-\beta\gamma \bm{w})=0,
+\]
+!et
+which gives us $\beta = \bm{w}^T\bm{y}/(\bm{w}^T\bm{w})$. Similarly we have
+!bt
+\[
+\beta\gamma \bm{x}^T(\bm{y}-\beta(1+\gamma \bm{x}))=0,
\]
!et
+which leads $\gamma =(\bm{x}^T\bm{y}-\beta\bm{x}^T\em{e})/(\beta\bm{x}^T\bm{x})$. Inserting
+for $\beta$ gives us an equation for $\gamma$.
-!split
-===== Finding the Optimal Parameters =====
-
-With these equations we can then in turn find the parameters $\beta_1$ and $\gamma_0^{1}$ and $\gamma_1^1$ as
+The solution to these two equations gives us in turn $\beta_1$ and $\gamma_1$ leading to the new expression for $f_1(x)$ as
+$f_1(x) = \beta_1(1+\gamma_1x)$. Doing this $M$ times results in our final estimate for the function $f$.
-
-
-Finding the Optimal Parameters
+which gives us \( \beta = \boldsymbol{w}^T\boldsymbol{y}/(\boldsymbol{w}^T\boldsymbol{w}) \). Similarly we have
+
+$$
+\beta\gamma \boldsymbol{x}^T(\boldsymbol{y}-\beta(1+\gamma \boldsymbol{x}))=0,
+$$
+
Iterative Fitting, Classification, AdaBoost
+Iterative Fitting, Classification, AdaBoost
Adaptive Boosting, AdaBoost
+Adaptive Boosting, AdaBoost
Building up AdaBoost
+Building up AdaBoost
Adaptive boosting: AdaBoost, Basic Algorithm
+Adaptive boosting: AdaBoost, Basic Algorithm
Basic Steps of AdaBoost
+Basic Steps of AdaBoost
AdaBoost Examples
+AdaBoost Examples
Gradient boosting: Basics
+Gradient boosting: Basics
Gradient Boosting, algorithm
+Gradient Boosting, algorithm
Gradient Boosting, Examples of Regression
+Gradient Boosting, Examples of Regression
Gradient Boosting, Classification Example
+Gradient Boosting, Classification Example
XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
Regression Case
+Regression Case
Xgboost on the Cancer Data
+Xgboost on the Cancer Data
-Finding the Optimal Parameters
-
-
-
-Iterative Fitting, Classification, AdaBoost
+Iterative Fitting, Classification, AdaBoost
-Adaptive Boosting, AdaBoost
+Adaptive Boosting, AdaBoost
-Building up AdaBoost
+Building up AdaBoost
-Adaptive boosting: AdaBoost, Basic Algorithm
+Adaptive boosting: AdaBoost, Basic Algorithm
-Basic Steps of AdaBoost
+Basic Steps of AdaBoost
-AdaBoost Examples
+AdaBoost Examples
-Gradient boosting: Basics
+Gradient boosting: Basics
-Gradient Boosting, algorithm
+Gradient Boosting, algorithm
-Gradient Boosting, Examples of Regression
+Gradient Boosting, Examples of Regression
-Gradient Boosting, Classification Example
+Gradient Boosting, Classification Example
-XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
-Regression Case
+Regression Case
-Xgboost on the Cancer Data
+Xgboost on the Cancer Data
-Finding the Optimal Parameters
-
-
-
-Iterative Fitting, Classification, AdaBoost
+Iterative Fitting, Classification, AdaBoost
-Adaptive Boosting, AdaBoost
+Adaptive Boosting, AdaBoost
-Building up AdaBoost
+Building up AdaBoost
-Adaptive boosting: AdaBoost, Basic Algorithm
+Adaptive boosting: AdaBoost, Basic Algorithm
-Basic Steps of AdaBoost
+Basic Steps of AdaBoost
-AdaBoost Examples
+AdaBoost Examples
-Gradient boosting: Basics
+Gradient boosting: Basics
-Gradient Boosting, algorithm
+Gradient Boosting, algorithm
-Gradient Boosting, Examples of Regression
+Gradient Boosting, Examples of Regression
-Gradient Boosting, Classification Example
+Gradient Boosting, Classification Example
-XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
-Regression Case
+Regression Case
-Xgboost on the Cancer Data
+Xgboost on the Cancer Data