Gradient boosting is again a similar technique to Adaptive boosting, @@ -278,9 +286,6 @@ In order to understand the method, let us illustrate its basics by bringing back the essential steps in linear regression, where our cost function was the least squares function. -
-See discussion during lecture November 8. -
@@ -303,6 +308,7 @@ See discussion during lecture November 8.
-Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function +We start again with our cost function \( {\cal C}(\boldsymbol{y}m\boldsymbol{f})=\sum_{i=0}^{n-1}{\cal L}(y_i, f(x_i)) \) where we want to minimize +This means that for every iteration, we need to optimize + $$ -C(\boldsymbol{y},\boldsymbol{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2. +(\hat{\boldsymbol{f}}) \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2. $$
-The way we proceed in an iterative fashion is to - -
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html index f4d69394d..59ffb9c1c 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs057.html @@ -140,19 +140,26 @@ Automatically generated HTML file from DocOnce source '___sec51'), ('Basic Steps of AdaBoost', 2, None, '___sec52'), ('AdaBoost Examples', 2, None, '___sec53'), - ('Gradient boosting: Basics', 2, None, '___sec54'), - ('Gradient Boosting, algorithm', 2, None, '___sec55'), + ('Gradient boosting: Basics with Steepest Descent', + 2, + None, + '___sec54'), + ('The Squared-Error again! Steepest Descent', + 2, + None, + '___sec55'), + ('Gradient Boosting, algorithm', 2, None, '___sec56'), ('Gradient Boosting, Examples of Regression', 2, None, - '___sec56'), + '___sec57'), ('Gradient Boosting, Classification Example', 2, None, - '___sec57'), - ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'), - ('Regression Case', 2, None, '___sec59'), - ('Xgboost on the Cancer Data', 2, None, '___sec60')]} + '___sec58'), + ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'), + ('Regression Case', 2, None, '___sec60'), + ('Xgboost on the Cancer Data', 2, None, '___sec61')]} end of tocinfo --> @@ -244,13 +251,14 @@ MathJax.Hub.Config({
+Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function +$$ +C(\boldsymbol{y},\boldsymbol{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2. +$$ - -
import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.ensemble import GradientBoostingRegressor
-from sklearn.preprocessing import StandardScaler
-import scikitplot as skplt
-from sklearn.metrics import mean_squared_error
-
-n = 100
-maxdegree = 6
-
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-
-error = np.zeros(maxdegree)
-bias = np.zeros(maxdegree)
-variance = np.zeros(maxdegree)
-polydegree = np.zeros(maxdegree)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-for degree in range(1,maxdegree):
- model = GradientBoostingRegressor(max_depth=degree, n_estimators=100, learning_rate=1.0)
- model.fit(X_train_scaled,y_train)
- y_pred = model.predict(X_test_scaled)
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
- variance[degree] = np.mean( np.var(y_pred) )
- print('Max depth:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-
-plt.xlim(1,maxdegree-1)
-plt.plot(polydegree, error, label='Error')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
-save_fig("gdregression")
-plt.show()
-+The way we proceed in an iterative fashion is to + +
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html index 9f20d329f..53fb8cb7a 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs058.html @@ -140,19 +140,26 @@ Automatically generated HTML file from DocOnce source '___sec51'), ('Basic Steps of AdaBoost', 2, None, '___sec52'), ('AdaBoost Examples', 2, None, '___sec53'), - ('Gradient boosting: Basics', 2, None, '___sec54'), - ('Gradient Boosting, algorithm', 2, None, '___sec55'), + ('Gradient boosting: Basics with Steepest Descent', + 2, + None, + '___sec54'), + ('The Squared-Error again! Steepest Descent', + 2, + None, + '___sec55'), + ('Gradient Boosting, algorithm', 2, None, '___sec56'), ('Gradient Boosting, Examples of Regression', 2, None, - '___sec56'), + '___sec57'), ('Gradient Boosting, Classification Example', 2, None, - '___sec57'), - ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'), - ('Regression Case', 2, None, '___sec59'), - ('Xgboost on the Cancer Data', 2, None, '___sec60')]} + '___sec58'), + ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'), + ('Regression Case', 2, None, '___sec60'), + ('Xgboost on the Cancer Data', 2, None, '___sec61')]} end of tocinfo --> @@ -244,13 +251,14 @@ MathJax.Hub.Config({
import matplotlib.pyplot as plt
import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import load_breast_cancer
-import scikitplot as skplt
-from sklearn.ensemble import GradientBoostingClassifier
-from sklearn.model_selection import cross_validate
-
-# Load the data
-cancer = load_breast_cancer()
-
-X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
-print(X_train.shape)
-print(X_test.shape)
-#now scale the data
+from sklearn.model_selection import train_test_split
+from sklearn.ensemble import GradientBoostingRegressor
from sklearn.preprocessing import StandardScaler
+import scikitplot as skplt
+from sklearn.metrics import mean_squared_error
+
+n = 100
+maxdegree = 6
+
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+
+error = np.zeros(maxdegree)
+bias = np.zeros(maxdegree)
+variance = np.zeros(maxdegree)
+polydegree = np.zeros(maxdegree)
+X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
-gd_clf = GradientBoostingClassifier(max_depth=3, n_estimators=100, learning_rate=1.0)
-gd_clf.fit(X_train_scaled, y_train)
-#Cross validation
-accuracy = cross_validate(gd_clf,X_test_scaled,y_test,cv=10)['test_score']
-print(accuracy)
-print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(gd_clf.score(X_test_scaled,y_test)))
+for degree in range(1,maxdegree):
+ model = GradientBoostingRegressor(max_depth=degree, n_estimators=100, learning_rate=1.0)
+ model.fit(X_train_scaled,y_train)
+ y_pred = model.predict(X_test_scaled)
+ polydegree[degree] = degree
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
+ variance[degree] = np.mean( np.var(y_pred) )
+ print('Max depth:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-import scikitplot as skplt
-y_pred = gd_clf.predict(X_test_scaled)
-skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
-save_fig("gdclassiffierconfusion")
-plt.show()
-y_probas = gd_clf.predict_proba(X_test_scaled)
-skplt.metrics.plot_roc(y_test, y_probas)
-save_fig("gdclassiffierroc")
-plt.show()
-skplt.metrics.plot_cumulative_gain(y_test, y_probas)
-save_fig("gdclassiffiercgain")
+plt.xlim(1,maxdegree-1)
+plt.plot(polydegree, error, label='Error')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+save_fig("gdregression")
plt.show()
@@ -330,6 +344,7 @@ plt.show()
-XGBoost or Extreme Gradient -Boosting, is an optimized distributed gradient boosting library -designed to be highly efficient, flexible and portable. It implements -machine learning algorithms under the Gradient Boosting -framework. XGBoost provides a parallel tree boosting that solve many -data science problems in a fast and accurate way. See the article by Chen and Guestrin. -
-The authors design and build a highly scalable end-to-end tree -boosting system. It has a theoretically justified weighted quantile -sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning. + +
import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import load_breast_cancer
+import scikitplot as skplt
+from sklearn.ensemble import GradientBoostingClassifier
+from sklearn.model_selection import cross_validate
-
-It is now the algorithm which wins essentially all ML competitions!!!
+# Load the data
+cancer = load_breast_cancer()
+X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
+print(X_train.shape)
+print(X_test.shape)
+#now scale the data
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+
+gd_clf = GradientBoostingClassifier(max_depth=3, n_estimators=100, learning_rate=1.0)
+gd_clf.fit(X_train_scaled, y_train)
+#Cross validation
+accuracy = cross_validate(gd_clf,X_test_scaled,y_test,cv=10)['test_score']
+print(accuracy)
+print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(gd_clf.score(X_test_scaled,y_test)))
+
+import scikitplot as skplt
+y_pred = gd_clf.predict(X_test_scaled)
+skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
+save_fig("gdclassiffierconfusion")
+plt.show()
+y_probas = gd_clf.predict_proba(X_test_scaled)
+skplt.metrics.plot_roc(y_test, y_probas)
+save_fig("gdclassiffierroc")
+plt.show()
+skplt.metrics.plot_cumulative_gain(y_test, y_probas)
+save_fig("gdclassiffiercgain")
+plt.show()
+
@@ -302,6 +337,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
+XGBoost or Extreme Gradient +Boosting, is an optimized distributed gradient boosting library +designed to be highly efficient, flexible and portable. It implements +machine learning algorithms under the Gradient Boosting +framework. XGBoost provides a parallel tree boosting that solve many +data science problems in a fast and accurate way. See the article by Chen and Guestrin. - -
import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-import xgboost as xgb
-from sklearn.preprocessing import StandardScaler
-import scikitplot as skplt
-from sklearn.metrics import mean_squared_error
+
+The authors design and build a highly scalable end-to-end tree
+boosting system. It has a theoretically justified weighted quantile
+sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning.
-n = 100
-maxdegree = 6
+
+It is now the algorithm which wins essentially all ML competitions!!!
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-
-error = np.zeros(maxdegree)
-bias = np.zeros(maxdegree)
-variance = np.zeros(maxdegree)
-polydegree = np.zeros(maxdegree)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-for degree in range(maxdegree):
- model = xgb.XGBRegressor(objective ='reg:squarederror', colsaobjective ='reg:squarederror', colsample_bytree = 0.3, learning_rate = 0.1,max_depth = degree, alpha = 10, n_estimators = 200)
-
- model.fit(X_train_scaled,y_train)
- y_pred = model.predict(X_test_scaled)
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
- variance[degree] = np.mean( np.var(y_pred) )
- print('Max depth:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-
-plt.xlim(1,maxdegree-1)
-plt.plot(polydegree, error, label='Error')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
-plt.show()
-
@@ -335,6 +309,7 @@ plt.show()
Gradient boosting is again a similar technique to Adaptive boosting, @@ -2366,14 +2366,26 @@ method via a series of iterations. In order to understand the method, let us illustrate its basics by bringing back the essential steps in linear regression, where our cost function was the least squares function. - -
-See discussion during lecture November 8.
+We start again with our cost function \( {\cal C}(\boldsymbol{y}m\boldsymbol{f})=\sum_{i=0}^{n-1}{\cal L}(y_i, f(x_i)) \) where we want to minimize +This means that for every iteration, we need to optimize + +
+$$
+(\hat{\boldsymbol{f}}) \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
+$$
+
+
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function
@@ -2401,7 +2413,7 @@ The way we proceed in an iterative fashion is to
@@ -2456,7 +2468,7 @@ plt.show()
@@ -2505,7 +2517,7 @@ plt.show()
XGBoost or Extreme Gradient
@@ -2526,7 +2538,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
@@ -2582,7 +2594,7 @@ plt.show()
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html b/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
index d0156f2db..c9df08769 100644
--- a/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
+++ b/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
@@ -160,19 +160,26 @@ div { text-align: justify; text-justify: inter-word; }
'___sec51'),
('Basic Steps of AdaBoost', 2, None, '___sec52'),
('AdaBoost Examples', 2, None, '___sec53'),
- ('Gradient boosting: Basics', 2, None, '___sec54'),
- ('Gradient Boosting, algorithm', 2, None, '___sec55'),
+ ('Gradient boosting: Basics with Steepest Descent',
+ 2,
+ None,
+ '___sec54'),
+ ('The Squared-Error again! Steepest Descent',
+ 2,
+ None,
+ '___sec55'),
+ ('Gradient Boosting, algorithm', 2, None, '___sec56'),
('Gradient Boosting, Examples of Regression',
2,
None,
- '___sec56'),
+ '___sec57'),
('Gradient Boosting, Classification Example',
2,
None,
- '___sec57'),
- ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'),
- ('Regression Case', 2, None, '___sec59'),
- ('Xgboost on the Cancer Data', 2, None, '___sec60')]}
+ '___sec58'),
+ ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'),
+ ('Regression Case', 2, None, '___sec60'),
+ ('Xgboost on the Cancer Data', 2, None, '___sec61')]}
end of tocinfo -->
Gradient boosting is again a similar technique to Adaptive boosting,
@@ -2329,12 +2336,22 @@ bringing back the essential steps in linear regression, where our cost
function was the least squares function.
-See discussion during lecture November 8.
+
+We start again with our cost function \( {\cal C}(\boldsymbol{y}m\boldsymbol{f})=\sum_{i=0}^{n-1}{\cal L}(y_i, f(x_i)) \) where we want to minimize
+This means that for every iteration, we need to optimize
+
+$$
+(\hat{\boldsymbol{f}}) \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
+$$
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function
@@ -2360,7 +2377,7 @@ The way we proceed in an iterative fashion is to
@@ -2414,7 +2431,7 @@ plt.show()
@@ -2462,7 +2479,7 @@ plt.show()
XGBoost or Extreme Gradient
@@ -2483,7 +2500,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
@@ -2538,7 +2555,7 @@ plt.show()
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees.html b/doc/pub/DecisionTrees/html/DecisionTrees.html
index 941aaf5f3..3ac58f8f9 100644
--- a/doc/pub/DecisionTrees/html/DecisionTrees.html
+++ b/doc/pub/DecisionTrees/html/DecisionTrees.html
@@ -165,19 +165,26 @@ div { text-align: justify; text-justify: inter-word; }
'___sec51'),
('Basic Steps of AdaBoost', 2, None, '___sec52'),
('AdaBoost Examples', 2, None, '___sec53'),
- ('Gradient boosting: Basics', 2, None, '___sec54'),
- ('Gradient Boosting, algorithm', 2, None, '___sec55'),
+ ('Gradient boosting: Basics with Steepest Descent',
+ 2,
+ None,
+ '___sec54'),
+ ('The Squared-Error again! Steepest Descent',
+ 2,
+ None,
+ '___sec55'),
+ ('Gradient Boosting, algorithm', 2, None, '___sec56'),
('Gradient Boosting, Examples of Regression',
2,
None,
- '___sec56'),
+ '___sec57'),
('Gradient Boosting, Classification Example',
2,
None,
- '___sec57'),
- ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec58'),
- ('Regression Case', 2, None, '___sec59'),
- ('Xgboost on the Cancer Data', 2, None, '___sec60')]}
+ '___sec58'),
+ ('XGBoost: Extreme Gradient Boosting', 2, None, '___sec59'),
+ ('Regression Case', 2, None, '___sec60'),
+ ('Xgboost on the Cancer Data', 2, None, '___sec61')]}
end of tocinfo -->
Gradient boosting is again a similar technique to Adaptive boosting,
@@ -2334,12 +2341,22 @@ bringing back the essential steps in linear regression, where our cost
function was the least squares function.
-See discussion during lecture November 8.
+
+We start again with our cost function \( {\cal C}(\boldsymbol{y}m\boldsymbol{f})=\sum_{i=0}^{n-1}{\cal L}(y_i, f(x_i)) \) where we want to minimize
+This means that for every iteration, we need to optimize
+
+$$
+(\hat{\boldsymbol{f}}) \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
+$$
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function
@@ -2365,7 +2382,7 @@ The way we proceed in an iterative fashion is to
@@ -2419,7 +2436,7 @@ plt.show()
@@ -2467,7 +2484,7 @@ plt.show()
XGBoost or Extreme Gradient
@@ -2488,7 +2505,7 @@ It is now the algorithm which wins essentially all ML competitions!!!
@@ -2543,7 +2560,7 @@ plt.show()
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
diff --git a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
index 2aeaf36de..835164d95 100644
--- a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
+++ b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
@@ -2555,7 +2555,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Gradient boosting: Basics\n",
+ "## Gradient boosting: Basics with Steepest Descent\n",
"\n",
"Gradient boosting is again a similar technique to Adaptive boosting,\n",
"it combines so-called weak classifiers or regressors into a strong\n",
@@ -2565,8 +2565,25 @@
"bringing back the essential steps in linear regression, where our cost\n",
"function was the least squares function.\n",
"\n",
- "See discussion during lecture November 8.\n",
+ "## The Squared-Error again! Steepest Descent\n",
"\n",
+ "We start again with our cost function ${\\cal C}(\\boldsymbol{y}m\\boldsymbol{f})=\\sum_{i=0}^{n-1}{\\cal L}(y_i, f(x_i))$ where we want to minimize\n",
+ "This means that for every iteration, we need to optimize"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "(\\hat{\\boldsymbol{f}}) \\mathrm{argmin}_{\\boldsymbol{f}}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i-f(x_i))^2.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
"## Gradient Boosting, algorithm\n",
"\n",
"Suppose we have a cost function $C(f)=\\sum_{i=0}^{n-1}L(y_i, f(x_i))$ where $y_i$ is our target and $f(x_i)$ the function which is meant to model $y_i$. The above cost function could be our standard squared-error function"
diff --git a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz
index 246b4020e..19258d47e 100644
Binary files a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz and b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz differ
diff --git a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf
index 21d19f2c7..be995b115 100644
Binary files a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf and b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf differ
diff --git a/doc/src/DecisionTrees/DecisionTrees.do.txt b/doc/src/DecisionTrees/DecisionTrees.do.txt
index 0dfa6de41..b019ed3ec 100644
--- a/doc/src/DecisionTrees/DecisionTrees.do.txt
+++ b/doc/src/DecisionTrees/DecisionTrees.do.txt
@@ -1938,7 +1938,7 @@ plt.show()
!split
-===== Gradient boosting: Basics =====
+===== Gradient boosting: Basics with Steepest Descent =====
Gradient boosting is again a similar technique to Adaptive boosting,
it combines so-called weak classifiers or regressors into a strong
@@ -1948,7 +1948,20 @@ In order to understand the method, let us illustrate its basics by
bringing back the essential steps in linear regression, where our cost
function was the least squares function.
-See discussion during lecture November 8.
+!split
+===== The Squared-Error again! Steepest Descent =====
+
+We start again with our cost function ${\cal C}(\bm{y}m\bm{f})=\sum_{i=0}^{n-1}{\cal L}(y_i, f(x_i))$ where we want to minimize
+This means that for every iteration, we need to optimize
+
+!bt
+\[
+(\hat{\bm{f}}) \mathrm{argmin}_{\bm{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
+\]
+!et
+
+
+
!split
===== Gradient Boosting, algorithm =====
Gradient Boosting, Examples of Regression
+Gradient Boosting, Examples of Regression
Gradient Boosting, Classification Example
+Gradient Boosting, Classification Example
XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
Regression Case
+Regression Case
Xgboost on the Cancer Data
+Xgboost on the Cancer Data
-Gradient boosting: Basics
+Gradient boosting: Basics with Steepest Descent
+
+The Squared-Error again! Steepest Descent
+
+
-Gradient Boosting, algorithm
+Gradient Boosting, algorithm
-Gradient Boosting, Examples of Regression
+Gradient Boosting, Examples of Regression
-Gradient Boosting, Classification Example
+Gradient Boosting, Classification Example
-XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
-Regression Case
+Regression Case
-Xgboost on the Cancer Data
+Xgboost on the Cancer Data
-Gradient boosting: Basics
+Gradient boosting: Basics with Steepest Descent
+
+The Squared-Error again! Steepest Descent
+
+
-Gradient Boosting, algorithm
+Gradient Boosting, algorithm
-Gradient Boosting, Examples of Regression
+Gradient Boosting, Examples of Regression
-Gradient Boosting, Classification Example
+Gradient Boosting, Classification Example
-XGBoost: Extreme Gradient Boosting
+XGBoost: Extreme Gradient Boosting
-Regression Case
+Regression Case
-Xgboost on the Cancer Data
+Xgboost on the Cancer Data