-
@@ -317,7 +321,7 @@ MathJax.Hub.Config({
@@ -303,7 +307,7 @@ the gradient is \( g_m(x_i) = -2(y_i-f(x_i)) \).
Choosing \( f_0(x)=0 \) we obtain \( g_m(x) = -2y_i \) and inserting this into the minimization problem for the cost function we have $$ -(\rho_1) \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2. +(\rho_1) = \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2. $$
@@ -328,6 +332,8 @@ $$
-Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function +Optimizing with respect to \( \rho \) we obtain (taking the derivative) that \( \rho_1 = -1/2 \). We have then that $$ -C(\boldsymbol{y},\boldsymbol{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2. +f_1(x) = f_{0}(x) -\rho_1 g_1(x)=-y_i. $$ +We can then proceed and compute +$$ +g_2(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{1}(x_i)=y_i}=-4y_i, +$$ + +and find a new value for \( \rho_2=-1/2 \) and continue till we have reached \( m=M \). We can modify the steepest descent method, or steepest boosting, by introducing what is called gradient boosting. +
-The way we proceed in an iterative fashion is to - -
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html index 53fb8cb7a..5758ef294 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html @@ -148,18 +148,20 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec55'), - ('Gradient Boosting, algorithm', 2, None, '___sec56'), + ('Steepest Descent Example', 2, None, '___sec56'), + ('Gradient Boosting, algorithm', 2, None, '___sec57'), + ('Gradient Boosting Example, Regression', 2, None, '___sec58'), ('Gradient Boosting, Examples of Regression', 2, None, - '___sec57'), + '___sec59'), ('Gradient Boosting, Classification Example', 2, None, - '___sec58'), - ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'), - ('Regression Case', 2, None, '___sec60'), - ('Xgboost on the Cancer Data', 2, None, '___sec61')]} + '___sec60'), + ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec61'), + ('Regression Case', 2, None, '___sec62'), + ('Xgboost on the Cancer Data', 2, None, '___sec63')]} end of tocinfo --> @@ -253,12 +255,14 @@ MathJax.Hub.Config({
+Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function +$$ +C(\boldsymbol{y},\boldsymbol{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2. +$$ - -
import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.ensemble import GradientBoostingRegressor
-from sklearn.preprocessing import StandardScaler
-import scikitplot as skplt
-from sklearn.metrics import mean_squared_error
-
-n = 100
-maxdegree = 6
-
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-
-error = np.zeros(maxdegree)
-bias = np.zeros(maxdegree)
-variance = np.zeros(maxdegree)
-polydegree = np.zeros(maxdegree)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-for degree in range(1,maxdegree):
- model = GradientBoostingRegressor(max_depth=degree, n_estimators=100, learning_rate=1.0)
- model.fit(X_train_scaled,y_train)
- y_pred = model.predict(X_test_scaled)
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
- variance[degree] = np.mean( np.var(y_pred) )
- print('Max depth:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-
-plt.xlim(1,maxdegree-1)
-plt.plot(polydegree, error, label='Error')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
-save_fig("gdregression")
-plt.show()
-+The way we proceed in an iterative fashion is to + +
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs059.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs059.html index 95c63fc44..98e82e736 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs059.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs059.html @@ -148,18 +148,20 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec55'), - ('Gradient Boosting, algorithm', 2, None, '___sec56'), + ('Steepest Descent Example', 2, None, '___sec56'), + ('Gradient Boosting, algorithm', 2, None, '___sec57'), + ('Gradient Boosting Example, Regression', 2, None, '___sec58'), ('Gradient Boosting, Examples of Regression', 2, None, - '___sec57'), + '___sec59'), ('Gradient Boosting, Classification Example', 2, None, - '___sec58'), - ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'), - ('Regression Case', 2, None, '___sec60'), - ('Xgboost on the Cancer Data', 2, None, '___sec61')]} + '___sec60'), + ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec61'), + ('Regression Case', 2, None, '___sec62'), + ('Xgboost on the Cancer Data', 2, None, '___sec63')]} end of tocinfo --> @@ -253,12 +255,14 @@ MathJax.Hub.Config({
+We discuss here the difference between the steepest descent approach and gradient boosting by repeating our simple regression example above. - -
import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import load_breast_cancer
-import scikitplot as skplt
-from sklearn.ensemble import GradientBoostingClassifier
-from sklearn.model_selection import cross_validate
-
-# Load the data
-cancer = load_breast_cancer()
-
-X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
-print(X_train.shape)
-print(X_test.shape)
-#now scale the data
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-gd_clf = GradientBoostingClassifier(max_depth=3, n_estimators=100, learning_rate=1.0)
-gd_clf.fit(X_train_scaled, y_train)
-#Cross validation
-accuracy = cross_validate(gd_clf,X_test_scaled,y_test,cv=10)['test_score']
-print(accuracy)
-print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(gd_clf.score(X_test_scaled,y_test)))
-
-import scikitplot as skplt
-y_pred = gd_clf.predict(X_test_scaled)
-skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
-save_fig("gdclassiffierconfusion")
-plt.show()
-y_probas = gd_clf.predict_proba(X_test_scaled)
-skplt.metrics.plot_roc(y_test, y_probas)
-save_fig("gdclassiffierroc")
-plt.show()
-skplt.metrics.plot_cumulative_gain(y_test, y_probas)
-save_fig("gdclassiffiercgain")
-plt.show()
-
@@ -338,6 +302,8 @@ plt.show()
-XGBoost or Extreme Gradient -Boosting, is an optimized distributed gradient boosting library -designed to be highly efficient, flexible and portable. It implements -machine learning algorithms under the Gradient Boosting -framework. XGBoost provides a parallel tree boosting that solve many -data science problems in a fast and accurate way. See the article by Chen and Guestrin. -
-The authors design and build a highly scalable end-to-end tree -boosting system. It has a theoretically justified weighted quantile -sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning. + +
import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.ensemble import GradientBoostingRegressor
+from sklearn.preprocessing import StandardScaler
+import scikitplot as skplt
+from sklearn.metrics import mean_squared_error
-
-It is now the algorithm which wins essentially all ML competitions!!!
+n = 100
+maxdegree = 6
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+
+error = np.zeros(maxdegree)
+bias = np.zeros(maxdegree)
+variance = np.zeros(maxdegree)
+polydegree = np.zeros(maxdegree)
+X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+
+for degree in range(1,maxdegree):
+ model = GradientBoostingRegressor(max_depth=degree, n_estimators=100, learning_rate=1.0)
+ model.fit(X_train_scaled,y_train)
+ y_pred = model.predict(X_test_scaled)
+ polydegree[degree] = degree
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
+ variance[degree] = np.mean( np.var(y_pred) )
+ print('Max depth:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+
+plt.xlim(1,maxdegree-1)
+plt.plot(polydegree, error, label='Error')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+save_fig("gdregression")
+plt.show()
+
@@ -310,6 +347,8 @@ It is now the algorithm which wins essentially all ML competitions!!!
-
@@ -317,7 +321,7 @@ MathJax.Hub.Config({
-
@@ -1758,6 +1758,10 @@ scaler.fit(X_train) X_train_scaled = scaler.transform(X_train) X_test_scaled = scaler.transform(X_test) +# we produce a simple tree first as benchmark +simpletree = DecisionTreeRegressor(max_depth=3) +simpletree.fit(X_train_scaled, y_train) +simpleprediction = simpletree.predict(X_test_scaled) for degree in range(1,maxdepth): model = DecisionTreeRegressor(max_depth=degree) y_pred = np.empty((y_test.shape[0], n_boostraps)) @@ -1776,8 +1780,10 @@ X_test_scaled = scaler.transform(X_test) print('Var:', variance[degree]) print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree])) +mse_simpletree = np.mean( np.mean((y_test - simpleprediction)**2) plt.xlim(1,maxdepth) -plt.plot(polydegree, error, label='Error') +plt.plot(polydegree, error, label='MSE simple tree') +plt.plot(polydegree, mse_simpletree, label='MSE for Bootstrap') plt.plot(polydegree, bias, label='bias') plt.plot(polydegree, variance, label='Variance') plt.legend() @@ -2378,7 +2384,7 @@ This means that for every iteration, we need to optimize
$$
-(\hat{\boldsymbol{f}}) \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
+(\hat{\boldsymbol{f}}) = \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
$$
@@ -2406,14 +2412,36 @@ the gradient is \( g_m(x_i) = -2(y_i-f(x_i)) \).
Choosing \( f_0(x)=0 \) we obtain \( g_m(x) = -2y_i \) and inserting this into the minimization problem for the cost function we have
$$
-(\rho_1) \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
+(\rho_1) = \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
$$
+Optimizing with respect to \( \rho \) we obtain (taking the derivative) that \( \rho_1 = -1/2 \). We have then that
+
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function
@@ -2441,7 +2469,15 @@ The way we proceed in an iterative fashion is to
+We discuss here the difference between the steepest descent approach and gradient boosting by repeating our simple regression example above.
+
@@ -2496,7 +2532,7 @@ plt.show()
@@ -2545,7 +2581,7 @@ plt.show()
XGBoost or Extreme Gradient
@@ -2566,7 +2602,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
@@ -2622,7 +2658,7 @@ plt.show()
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html b/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
index 3045d1183..3285766a2 100644
--- a/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
+++ b/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
@@ -168,18 +168,20 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec55'),
- ('Gradient Boosting, algorithm', 2, None, '___sec56'),
+ ('Steepest Descent Example', 2, None, '___sec56'),
+ ('Gradient Boosting, algorithm', 2, None, '___sec57'),
+ ('Gradient Boosting Example, Regression', 2, None, '___sec58'),
('Gradient Boosting, Examples of Regression',
2,
None,
- '___sec57'),
+ '___sec59'),
('Gradient Boosting, Classification Example',
2,
None,
- '___sec58'),
- ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'),
- ('Regression Case', 2, None, '___sec60'),
- ('Xgboost on the Cancer Data', 2, None, '___sec61')]}
+ '___sec60'),
+ ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec61'),
+ ('Regression Case', 2, None, '___sec62'),
+ ('Xgboost on the Cancer Data', 2, None, '___sec63')]}
end of tocinfo -->
-
@@ -2367,13 +2375,31 @@ the gradient is \( g_m(x_i) = -2(y_i-f(x_i)) \).
Choosing \( f_0(x)=0 \) we obtain \( g_m(x) = -2y_i \) and inserting this into the minimization problem for the cost function we have
$$
-(\rho_1) \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
+(\rho_1) = \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
$$
+Optimizing with respect to \( \rho \) we obtain (taking the derivative) that \( \rho_1 = -1/2 \). We have then that
+$$
+f_1(x) = f_{0}(x) -\rho_1 g_1(x)=-y_i.
+$$
+
+We can then proceed and compute
+$$
+g_2(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{1}(x_i)=y_i}=-4y_i,
+$$
+
+and find a new value for \( \rho_2=-1/2 \) and continue till we have reached \( m=M \). We can modify the steepest descent method, or steepest boosting, by introducing what is called gradient boosting.
+
+
+
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function
@@ -2399,7 +2425,15 @@ The way we proceed in an iterative fashion is to
+We discuss here the difference between the steepest descent approach and gradient boosting by repeating our simple regression example above.
+
+
+
@@ -2453,7 +2487,7 @@ plt.show()
@@ -2501,7 +2535,7 @@ plt.show()
XGBoost or Extreme Gradient
@@ -2522,7 +2556,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
@@ -2577,7 +2611,7 @@ plt.show()
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees.html b/doc/pub/DecisionTrees/html/DecisionTrees.html
index ab637865b..61f9eb3dd 100644
--- a/doc/pub/DecisionTrees/html/DecisionTrees.html
+++ b/doc/pub/DecisionTrees/html/DecisionTrees.html
@@ -173,18 +173,20 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec55'),
- ('Gradient Boosting, algorithm', 2, None, '___sec56'),
+ ('Steepest Descent Example', 2, None, '___sec56'),
+ ('Gradient Boosting, algorithm', 2, None, '___sec57'),
+ ('Gradient Boosting Example, Regression', 2, None, '___sec58'),
('Gradient Boosting, Examples of Regression',
2,
None,
- '___sec57'),
+ '___sec59'),
('Gradient Boosting, Classification Example',
2,
None,
- '___sec58'),
- ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'),
- ('Regression Case', 2, None, '___sec60'),
- ('Xgboost on the Cancer Data', 2, None, '___sec61')]}
+ '___sec60'),
+ ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec61'),
+ ('Regression Case', 2, None, '___sec62'),
+ ('Xgboost on the Cancer Data', 2, None, '___sec63')]}
end of tocinfo -->
-
@@ -2372,13 +2380,31 @@ the gradient is \( g_m(x_i) = -2(y_i-f(x_i)) \).
Choosing \( f_0(x)=0 \) we obtain \( g_m(x) = -2y_i \) and inserting this into the minimization problem for the cost function we have
$$
-(\rho_1) \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
+(\rho_1) = \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
$$
+Optimizing with respect to \( \rho \) we obtain (taking the derivative) that \( \rho_1 = -1/2 \). We have then that
+$$
+f_1(x) = f_{0}(x) -\rho_1 g_1(x)=-y_i.
+$$
+
+We can then proceed and compute
+$$
+g_2(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{1}(x_i)=y_i}=-4y_i,
+$$
+
+and find a new value for \( \rho_2=-1/2 \) and continue till we have reached \( m=M \). We can modify the steepest descent method, or steepest boosting, by introducing what is called gradient boosting.
+
+
+
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function
@@ -2404,7 +2430,15 @@ The way we proceed in an iterative fashion is to
+We discuss here the difference between the steepest descent approach and gradient boosting by repeating our simple regression example above.
+
+
+
@@ -2458,7 +2492,7 @@ plt.show()
@@ -2506,7 +2540,7 @@ plt.show()
XGBoost or Extreme Gradient
@@ -2527,7 +2561,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
@@ -2582,7 +2616,7 @@ plt.show()
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
diff --git a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
index 333396a8e..5f72bcd04 100644
--- a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
+++ b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Nov 12, 2019**\n",
+ "Date: **Nov 13, 2019**\n",
"\n",
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -1780,6 +1780,10 @@
"X_train_scaled = scaler.transform(X_train)\n",
"X_test_scaled = scaler.transform(X_test)\n",
"\n",
+ "# we produce a simple tree first as benchmark\n",
+ "simpletree = DecisionTreeRegressor(max_depth=3) \n",
+ "simpletree.fit(X_train_scaled, y_train)\n",
+ "simpleprediction = simpletree.predict(X_test_scaled)\n",
"for degree in range(1,maxdepth):\n",
" model = DecisionTreeRegressor(max_depth=degree) \n",
" y_pred = np.empty((y_test.shape[0], n_boostraps))\n",
@@ -1798,8 +1802,10 @@
" print('Var:', variance[degree])\n",
" print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))\n",
"\n",
+ "mse_simpletree = np.mean( np.mean((y_test - simpleprediction)**2)\n",
"plt.xlim(1,maxdepth)\n",
- "plt.plot(polydegree, error, label='Error')\n",
+ "plt.plot(polydegree, error, label='MSE simple tree')\n",
+ "plt.plot(polydegree, mse_simpletree, label='MSE for Bootstrap')\n",
"plt.plot(polydegree, bias, label='bias')\n",
"plt.plot(polydegree, variance, label='Variance')\n",
"plt.legend()\n",
@@ -2576,7 +2582,7 @@
"metadata": {},
"source": [
"$$\n",
- "(\\hat{\\boldsymbol{f}}) \\mathrm{argmin}_{\\boldsymbol{f}}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i-f(x_i))^2.\n",
+ "(\\hat{\\boldsymbol{f}}) = \\mathrm{argmin}_{\\boldsymbol{f}}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i-f(x_i))^2.\n",
"$$"
]
},
@@ -2627,7 +2633,7 @@
"metadata": {},
"source": [
"$$\n",
- "(\\rho_1) \\mathrm{argmin}_{\\rho}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i+2\\rho y_i)^2.\n",
+ "(\\rho_1) = \\mathrm{argmin}_{\\rho}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i+2\\rho y_i)^2.\n",
"$$"
]
},
@@ -2635,6 +2641,42 @@
"cell_type": "markdown",
"metadata": {},
"source": [
+ "## Steepest Descent Example\n",
+ "\n",
+ "Optimizing with respect to $\\rho$ we obtain (taking the derivative) that $\\rho_1 = -1/2$. We have then that"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "f_1(x) = f_{0}(x) -\\rho_1 g_1(x)=-y_i.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can then proceed and compute"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "g_2(x_i) = \\left[ \\frac{\\partial {\\cal L}(y_i, f(x_i))}{\\partial f(x_i)}\\right]_{f(x_i)=f_{1}(x_i)=y_i}=-4y_i,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and find a new value for $\\rho_2=-1/2$ and continue till we have reached $m=M$. We can modify the steepest descent method, or steepest boosting, by introducing what is called **gradient boosting**. \n",
+ "\n",
"## Gradient Boosting, algorithm\n",
"\n",
"Suppose we have a cost function $C(f)=\\sum_{i=0}^{n-1}L(y_i, f(x_i))$ where $y_i$ is our target and $f(x_i)$ the function which is meant to model $y_i$. The above cost function could be our standard squared-error function"
@@ -2667,6 +2709,11 @@
"\n",
"4. The final estimate is then $f_M(x) = \\sum_{m=1}^M\\nu h_m(u_m,x)$.\n",
"\n",
+ "## Gradient Boosting Example, Regression\n",
+ "\n",
+ "We discuss here the difference between the steepest descent approach and gradient boosting by repeating our simple regression example above. \n",
+ "\n",
+ "\n",
"## Gradient Boosting, Examples of Regression"
]
},
diff --git a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz
index 65c88d19d..43c2508b6 100644
Binary files a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz and b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz differ
diff --git a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf
index 42f87b2c2..519b8268c 100644
Binary files a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf and b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf differ
diff --git a/doc/src/DecisionTrees/DecisionTrees.do.txt b/doc/src/DecisionTrees/DecisionTrees.do.txt
index d728c55d8..1b2dd2ae9 100644
--- a/doc/src/DecisionTrees/DecisionTrees.do.txt
+++ b/doc/src/DecisionTrees/DecisionTrees.do.txt
@@ -1433,6 +1433,10 @@ scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
+# we produce a simple tree first as benchmark
+simpletree = DecisionTreeRegressor(max_depth=3)
+simpletree.fit(X_train_scaled, y_train)
+simpleprediction = simpletree.predict(X_test_scaled)
for degree in range(1,maxdepth):
model = DecisionTreeRegressor(max_depth=degree)
y_pred = np.empty((y_test.shape[0], n_boostraps))
@@ -1451,8 +1455,10 @@ for degree in range(1,maxdepth):
print('Var:', variance[degree])
print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+mse_simpletree = np.mean( np.mean((y_test - simpleprediction)**2)
plt.xlim(1,maxdepth)
-plt.plot(polydegree, error, label='Error')
+plt.plot(polydegree, error, label='MSE simple tree')
+plt.plot(polydegree, mse_simpletree, label='MSE for Bootstrap')
plt.plot(polydegree, bias, label='bias')
plt.plot(polydegree, variance, label='Variance')
plt.legend()
@@ -1956,7 +1962,7 @@ This means that for every iteration, we need to optimize
!bt
\[
-(\hat{\bm{f}}) \mathrm{argmin}_{\bm{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
+(\hat{\bm{f}}) = \mathrm{argmin}_{\bm{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
\]
!et
@@ -1980,10 +1986,26 @@ the gradient is $g_m(x_i) = -2(y_i-f(x_i))$.
Choosing $f_0(x)=0$ we obtain $g_m(x) = -2y_i$ and inserting this into the minimization problem for the cost function we have
!bt
\[
-(\rho_1) \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
+(\rho_1) = \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
\]
!et
+!split
+===== Steepest Descent Example =====
+
+Optimizing with respect to $\rho$ we obtain (taking the derivative) that $\rho_1 = -1/2$. We have then that
+!bt
+\[
+f_1(x) = f_{0}(x) -\rho_1 g_1(x)=-y_i.
+\]
+!et
+We can then proceed and compute
+!bt
+\[
+g_2(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{1}(x_i)=y_i}=-4y_i,
+\]
+!et
+and find a new value for $\rho_2=-1/2$ and continue till we have reached $m=M$. We can modify the steepest descent method, or steepest boosting, by introducing what is called _gradient boosting_.
!split
===== Gradient Boosting, algorithm =====
@@ -2003,7 +2025,10 @@ o For $m=1:M$, we
o update the estimate $f_m(x) = f_{m-1}(x)+\nu h_m(u_m,x)$;
o The final estimate is then $f_M(x) = \sum_{m=1}^M\nu h_m(u_m,x)$.
+!split
+===== Gradient Boosting Example, Regression =====
+We discuss here the difference between the steepest descent approach and gradient boosting by repeating our simple regression example above.
!split
Gradient Boosting, algorithm
+Steepest Descent Example
+
+
+$$
+f_1(x) = f_{0}(x) -\rho_1 g_1(x)=-y_i.
+$$
+
+
+We can then proceed and compute
+
+$$
+g_2(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{1}(x_i)=y_i}=-4y_i,
+$$
+
+
+and find a new value for \( \rho_2=-1/2 \) and continue till we have reached \( m=M \). We can modify the steepest descent method, or steepest boosting, by introducing what is called gradient boosting.
+Gradient Boosting, algorithm
Gradient Boosting, Examples of Regression
+Gradient Boosting Example, Regression
+
+Gradient Boosting, Examples of Regression
Gradient Boosting, Classification Example
+Gradient Boosting, Classification Example
XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
Regression Case
+Regression Case
Xgboost on the Cancer Data
+Xgboost on the Cancer Data
Nov 12, 2019
Nov 13, 2019
@@ -1786,6 +1788,10 @@ scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
+# we produce a simple tree first as benchmark
+simpletree = DecisionTreeRegressor(max_depth=3)
+simpletree.fit(X_train_scaled, y_train)
+simpleprediction = simpletree.predict(X_test_scaled)
for degree in range(1,maxdepth):
model = DecisionTreeRegressor(max_depth=degree)
y_pred = np.empty((y_test.shape[0], n_boostraps))
@@ -1804,8 +1810,10 @@ X_test_scaled = scaler.transform(X_test)
print('Var:', variance[degree])
print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+mse_simpletree = np.mean( np.mean((y_test - simpleprediction)**2)
plt.xlim(1,maxdepth)
-plt.plot(polydegree, error, label='Error')
+plt.plot(polydegree, error, label='MSE simple tree')
+plt.plot(polydegree, mse_simpletree, label='MSE for Bootstrap')
plt.plot(polydegree, bias, label='bias')
plt.plot(polydegree, variance, label='Variance')
plt.legend()
@@ -2345,7 +2353,7 @@ We start again with our cost function \( {\cal C}(\boldsymbol{y}m\boldsymbol{f})
This means that for every iteration, we need to optimize
$$
-(\hat{\boldsymbol{f}}) \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
+(\hat{\boldsymbol{f}}) = \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
$$
-Gradient Boosting, algorithm
+Steepest Descent Example
+
+
+
+Gradient Boosting, algorithm
-Gradient Boosting, Examples of Regression
+Gradient Boosting Example, Regression
+
+
+
+Gradient Boosting, Examples of Regression
-Gradient Boosting, Classification Example
+Gradient Boosting, Classification Example
-XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
-Regression Case
+Regression Case
-Xgboost on the Cancer Data
+Xgboost on the Cancer Data
Nov 12, 2019
Nov 13, 2019
@@ -1791,6 +1793,10 @@ scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
+# we produce a simple tree first as benchmark
+simpletree = DecisionTreeRegressor(max_depth=3)
+simpletree.fit(X_train_scaled, y_train)
+simpleprediction = simpletree.predict(X_test_scaled)
for degree in range(1,maxdepth):
model = DecisionTreeRegressor(max_depth=degree)
y_pred = np.empty((y_test.shape[0], n_boostraps))
@@ -1809,8 +1815,10 @@ X_test_scaled = scalerprint('Var:', variance[degree])
print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+mse_simpletree = np.mean( np.mean((y_test - simpleprediction)**2)
plt.xlim(1,maxdepth)
-plt.plot(polydegree, error, label='Error')
+plt.plot(polydegree, error, label='MSE simple tree')
+plt.plot(polydegree, mse_simpletree, label='MSE for Bootstrap')
plt.plot(polydegree, bias, label='bias')
plt.plot(polydegree, variance, label='Variance')
plt.legend()
@@ -2350,7 +2358,7 @@ We start again with our cost function \( {\cal C}(\boldsymbol{y}m\boldsymbol{f})
This means that for every iteration, we need to optimize
$$
-(\hat{\boldsymbol{f}}) \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
+(\hat{\boldsymbol{f}}) = \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
$$
-Gradient Boosting, algorithm
+Steepest Descent Example
+
+
+
+Gradient Boosting, algorithm
-Gradient Boosting, Examples of Regression
+Gradient Boosting Example, Regression
+
+
+
+Gradient Boosting, Examples of Regression
-Gradient Boosting, Classification Example
+Gradient Boosting, Classification Example
-XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
-Regression Case
+Regression Case
-Xgboost on the Cancer Data
+Xgboost on the Cancer Data