Updating decision trees

doc/src/DecisionTrees/DecisionTrees.do.txt

@@ -35,6 +35,7 @@ given some assumptions, make predictions about the target feature value
 !split
 ===== A typical Decision Tree with its pertinent Jargon, Classification Problem =====
 
+Figure to come here.
 
 
 
@@ -462,6 +463,23 @@ os.system(cmd)
 !ec
 
 
+!split
+===== Algorithms for Setting up Decision Trees =====
+Two algorithms stand out in the set up of decision trees:
+o The CART (Classification And Regression Tree) algorithm for both classification and regression
+o The ID3 algorithm based on the computation of the information gain for classification
+
+We discuss both algorithms with applications here. The popular library -Scikit-Learn_ uses the CART algorithm. For classification problems you can use either the _gini_ index or the _entropy_ to split a tree in two branches.
+
+!split
+===== The CART algorithm for Classification =====
+
+
+!split
+===== The CART algorithm for Regression =====
+
+
+
 !split
 ===== Computing the Gini index =====
 
@@ -1127,6 +1145,24 @@ plt.show()
 
 However, by aggregating many decision trees, using methods like bagging, random forests, and boosting, the predictive performance of trees can be substantially improved. 
 
+
+!split
+===== From a Single Tree to Many Trees, that is meet the Jungle of Methods =====
+
+As stated above and seen in many of the examples discussed here about
+a single decision tree, we often end up overfitting our training
+data. This normally means that we have a high variance. Can we reduce
+the variance of a statistical learning method?
+
+This leads us to a set of different methods that can combine different
+machine learning algorithms or just use one of them to construct forests and jungles of trees, homogeneous ones or heterogenous ones. These methods are recognized by different names which we will try to explain here. These are
+o Votign classifiers
+o Bagging and Pasting
+o Random forests
+o Boosting methods
+
+We discuss these methods here.
+
 !split
 ===== Bagging =====
 
@@ -1426,9 +1462,19 @@ np.sum(y_pred == y_pred_rf) / len(y_pred)
 
 Example will be added here.
 
-!split
-===== Boosting: AdaBoost =====
 
+!split
+===== Boosting, a Bird'e Eye =====
+
+
+
+
+!split
+===== Adaptive boosting: AdaBoost, Basic Algorithm =====
+
+
+!split
+===== AdaBoost Examples =====
 !bc pycod
 from sklearn.ensemble import AdaBoostClassifier
 
@@ -1461,14 +1507,17 @@ for subplot, learning_rate in ((121, 1), (122, 0.5)):
 
 save_fig("boosting_plot")
 plt.show()
-
-
-
 !ec
 
 
 !split
-===== Gradient Boosting =====
+===== Gradient boosting: Basic Algorithm =====
+
+
+
+
+!split
+===== Gradient Boosting, Examples =====
 !bc pycod
 np.random.seed(42)
 X = np.random.rand(100, 1) - 0.5
@@ -1617,3 +1666,7 @@ for n_estimators in range(1, 120):
 print(gbrt.n_estimators)
 print("Minimum validation MSE:", min_val_error)
 !ec
+
+!split
+===== XGBoost: Extreme Gradient Boosting =====
+