-
@@ -244,7 +253,7 @@ MathJax.Hub.Config({
-A decision tree is typically divided into a root node, the interior nodes, -and the final leaf nodes or just leaves. These entities are then connected by so-called branches. - -
-The leaf nodes -contain the predictions we will make for new query instances presented -to our trained model. This is possible since the model has -learned the underlying structure of the training data and hence can, -given some assumptions, make predictions about the target feature value -(class) of unseen query instances. -
@@ -259,7 +256,7 @@ given some assumptions, make predictions about the target feature value
-
-This tree was produced using the Wisconsin cancer data (discussed here as well, see code examples below) using Scikit-Learn's decision tree classifier. Here we have used the so-called gini index (see below) to split the various branches. +The leaf nodes +contain the predictions we will make for new query instances presented +to our trained model. This is possible since the model has +learned the underlying structure of the training data and hence can, +given some assumptions, make predictions about the target feature value +(class) of unseen query instances.
@@ -229,7 +244,7 @@ This tree was produced using the Wisconsin cancer data (discussed here as well,
-The overarching approach to decision trees is a top-down approach. - -
@@ -237,7 +236,7 @@ node.
-In simplified terms, the process of training a decision tree and -predicting the target features of query instances is as follows: - -
@@ -238,7 +237,7 @@ Then we are essentially done!
+
import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.preprocessing import PolynomialFeatures
-from sklearn.linear_model import LinearRegression
+
+This tree was produced using the Wisconsin cancer data (discussed here as well, see code examples below) using Scikit-Learn's decision tree classifier. Here we have used the so-called gini index (see below) to split the various branches.
-steps=250
-
-distance=0
-x=0
-distance_list=[]
-steps_list=[]
-while x<steps:
- distance+=np.random.randint(-1,2)
- distance_list.append(distance)
- x+=1
- steps_list.append(x)
-plt.plot(steps_list,distance_list, color='green', label="Random Walk Data")
-
-steps_list=np.asarray(steps_list)
-distance_list=np.asarray(distance_list)
-
-X=steps_list[:,np.newaxis]
-
-#Polynomial fits
-
-#Degree 2
-poly_features=PolynomialFeatures(degree=2, include_bias=False)
-X_poly=poly_features.fit_transform(X)
-
-lin_reg=LinearRegression()
-poly_fit=lin_reg.fit(X_poly,distance_list)
-b=lin_reg.coef_
-c=lin_reg.intercept_
-print ("2nd degree coefficients:")
-print ("zero power: ",c)
-print ("first power: ", b[0])
-print ("second power: ",b[1])
-
-z = np.arange(0, steps, .01)
-z_mod=b[1]*z**2+b[0]*z+c
-
-fit_mod=b[1]*X**2+b[0]*X+c
-plt.plot(z, z_mod, color='r', label="2nd Degree Fit")
-plt.title("Polynomial Regression")
-
-plt.xlabel("Steps")
-plt.ylabel("Distance")
-
-#Degree 10
-poly_features10=PolynomialFeatures(degree=10, include_bias=False)
-X_poly10=poly_features10.fit_transform(X)
-
-poly_fit10=lin_reg.fit(X_poly10,distance_list)
-
-y_plot=poly_fit10.predict(X_poly10)
-plt.plot(X, y_plot, color='black', label="10th Degree Fit")
-
-plt.legend()
-plt.show()
-
-
-#Decision Tree Regression
-from sklearn.tree import DecisionTreeRegressor
-regr_1=DecisionTreeRegressor(max_depth=2)
-regr_2=DecisionTreeRegressor(max_depth=5)
-regr_3=DecisionTreeRegressor(max_depth=7)
-regr_1.fit(X, distance_list)
-regr_2.fit(X, distance_list)
-regr_3.fit(X, distance_list)
-
-X_test = np.arange(0.0, steps, 0.01)[:, np.newaxis]
-y_1 = regr_1.predict(X_test)
-y_2 = regr_2.predict(X_test)
-y_3=regr_3.predict(X_test)
-
-# Plot the results
-plt.figure()
-plt.scatter(X, distance_list, s=2.5, c="black", label="data")
-plt.plot(X_test, y_1, color="red",
- label="max_depth=2", linewidth=2)
-plt.plot(X_test, y_2, color="green", label="max_depth=5", linewidth=2)
-plt.plot(X_test, y_3, color="m", label="max_depth=7", linewidth=2)
-
-plt.xlabel("Data")
-plt.ylabel("Darget")
-plt.title("Decision Tree Regression")
-plt.legend()
-plt.show()
-
@@ -317,7 +241,7 @@ plt.show()
-There are mainly two steps +The overarching approach to decision trees is a top-down approach. -
-where \( \overline{y}_{R_j} \) is the mean response for the training observations -within box \( j \). +This process is then repeated for the subtree rooted at the new +node.
@@ -250,7 +249,7 @@ within box \( j \).
-Unfortunately, it is computationally infeasible to consider every -possible partition of the feature space into \( J \) boxes. The common -strategy is to take a top-down approach +In simplified terms, the process of training a decision tree and +predicting the target features of query instances is as follows: -
-The approach is top-down because it begins at the top of the tree (all -observations belong to a single region) and then successively splits -the predictor space; each split is indicated via two new branches -further down on the tree. It is greedy because at each step of the -tree-building process, the best split is made at that particular step, -rather than looking ahead and picking a split that will lead to a -better tree in some future step. +
@@ -242,7 +250,7 @@ better tree in some future step.
-In order to implement the recursive binary splitting we start by selecting -the predictor \( x_j \) and a cutpoint \( s \) that splits the predictor space into two regions \( R_1 \) and \( R_2 \) -$$ -\left\{X\vert x_j < s\right\}, -$$ -and -$$ -\left\{X\vert x_j \geq s\right\}, -$$ + +
import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.linear_model import LinearRegression
-so that we obtain the lowest MSE, that is
-$$
-\sum_{i:x_i\in R_j}(y_i-\overline{y}_{R_1})^2+\sum_{i:x_i\in R_2}(y_i-\overline{y}_{R_2})^2,
-$$
+steps=250
-
-which we want to minimize by considering all predictors
-\( x_1,x_2,\dots,x_p \). We consider also all possible values of \( s \) for
-each predictor. These values could be determined by randomly assigned
-numbers or by starting at the midpoint and then proceed till we find
-an optimal value.
+distance=0
+x=0
+distance_list=[]
+steps_list=[]
+while x<steps:
+ distance+=np.random.randint(-1,2)
+ distance_list.append(distance)
+ x+=1
+ steps_list.append(x)
+plt.plot(steps_list,distance_list, color='green', label="Random Walk Data")
-
-For any \( j \) and \( s \), we define the pair of half-planes where
-\( \overline{y}_{R_1} \) is the mean response for the training
-observations in \( R_1(j,s) \), and \( \overline{y}_{R_2} \) is the mean
-response for the training observations in \( R_2(j,s) \).
+steps_list=np.asarray(steps_list)
+distance_list=np.asarray(distance_list)
-
-Finding the values of \( j \) and \( s \) that minimize the above equation can be
-done quite quickly, especially when the number of features \( p \) is not
-too large.
+X=steps_list[:,np.newaxis]
-
-Next, we repeat the process, looking
-for the best predictor and best cutpoint in order to split the data
-further so as to minimize the MSE within each of the resulting
-regions. However, this time, instead of splitting the entire predictor
-space, we split one of the two previously identified regions. We now
-have three regions. Again, we look to split one of these three regions
-further, so as to minimize the MSE. The process continues until a
-stopping criterion is reached; for instance, we may continue until no
-region contains more than five observations.
+#Polynomial fits
+#Degree 2
+poly_features=PolynomialFeatures(degree=2, include_bias=False)
+X_poly=poly_features.fit_transform(X)
+
+lin_reg=LinearRegression()
+poly_fit=lin_reg.fit(X_poly,distance_list)
+b=lin_reg.coef_
+c=lin_reg.intercept_
+print ("2nd degree coefficients:")
+print ("zero power: ",c)
+print ("first power: ", b[0])
+print ("second power: ",b[1])
+
+z = np.arange(0, steps, .01)
+z_mod=b[1]*z**2+b[0]*z+c
+
+fit_mod=b[1]*X**2+b[0]*X+c
+plt.plot(z, z_mod, color='r', label="2nd Degree Fit")
+plt.title("Polynomial Regression")
+
+plt.xlabel("Steps")
+plt.ylabel("Distance")
+
+#Degree 10
+poly_features10=PolynomialFeatures(degree=10, include_bias=False)
+X_poly10=poly_features10.fit_transform(X)
+
+poly_fit10=lin_reg.fit(X_poly10,distance_list)
+
+y_plot=poly_fit10.predict(X_poly10)
+plt.plot(X, y_plot, color='black', label="10th Degree Fit")
+
+plt.legend()
+plt.show()
+
+
+#Decision Tree Regression
+from sklearn.tree import DecisionTreeRegressor
+regr_1=DecisionTreeRegressor(max_depth=2)
+regr_2=DecisionTreeRegressor(max_depth=5)
+regr_3=DecisionTreeRegressor(max_depth=7)
+regr_1.fit(X, distance_list)
+regr_2.fit(X, distance_list)
+regr_3.fit(X, distance_list)
+
+X_test = np.arange(0.0, steps, 0.01)[:, np.newaxis]
+y_1 = regr_1.predict(X_test)
+y_2 = regr_2.predict(X_test)
+y_3=regr_3.predict(X_test)
+
+# Plot the results
+plt.figure()
+plt.scatter(X, distance_list, s=2.5, c="black", label="data")
+plt.plot(X_test, y_1, color="red",
+ label="max_depth=2", linewidth=2)
+plt.plot(X_test, y_2, color="green", label="max_depth=5", linewidth=2)
+plt.plot(X_test, y_3, color="m", label="max_depth=7", linewidth=2)
+
+plt.xlabel("Data")
+plt.ylabel("Darget")
+plt.title("Decision Tree Regression")
+plt.legend()
+plt.show()
+
@@ -275,7 +329,7 @@ region contains more than five observations.
- + -
-The above procedure is rather straightforward, but leads often to -overfitting and unnecessarily large and complicated trees. The basic -idea is to grow a large tree \( T_0 \) and then prune it back in order to -obtain a subtree. A smaller tree with fewer splits (fewer regions) can -lead to smaller variance and better interpretation at the cost of a -little more bias. +There are mainly two steps + +
-The so-called Cost complexity pruning algorithm gives us a -way to do just this. Rather than considering every possible subtree, -we consider a sequence of trees indexed by a nonnegative tuning -parameter \( \alpha \). +where \( \overline{y}_{R_j} \) is the mean response for the training observations +within box \( j \).
@@ -243,7 +261,7 @@ parameter \( \alpha \).
-The tuning parameter \( \alpha \) controls a trade-off between the subtree’s -com- plexity and its fit to the training data. When \( \alpha = 0 \), then the -subtree \( T \) will simply equal \( T_0 \), -because then the above equation just measures the -training error. -However, as \( \alpha \) increases, there is a price to pay for -having a tree with many terminal nodes. The above equation will -tend to be minimized for a smaller subtree. +Unfortunately, it is computationally infeasible to consider every +possible partition of the feature space into \( J \) boxes. The common +strategy is to take a top-down approach
-It turns out that as we increase \( \alpha \) from zero -branches get pruned from the tree in a nested and predictable fashion, -so obtaining the whole sequence of subtrees as a function of \( \alpha \) is -easy. We can select a value of \( \alpha \) using a validation set or using -cross-validation. We then return to the full data set and obtain the -subtree corresponding to \( \alpha \). +The approach is top-down because it begins at the top of the tree (all +observations belong to a single region) and then successively splits +the predictor space; each split is indicated via two new branches +further down on the tree. It is greedy because at each step of the +tree-building process, the best split is made at that particular step, +rather than looking ahead and picking a split that will lead to a +better tree in some future step.
@@ -255,7 +252,7 @@ subtree corresponding to \( \alpha \).
-
+In order to implement the recursive binary splitting we start by selecting +the predictor \( x_j \) and a cutpoint \( s \) that splits the predictor space into two regions \( R_1 \) and \( R_2 \) +$$ +\left\{X\vert x_j < s\right\}, +$$ -
+which we want to minimize by considering all predictors +\( x_1,x_2,\dots,x_p \). We consider also all possible values of \( s \) for +each predictor. These values could be determined by randomly assigned +numbers or by starting at the midpoint and then proceed till we find +an optimal value. +
+For any \( j \) and \( s \), we define the pair of half-planes where +\( \overline{y}_{R_1} \) is the mean response for the training +observations in \( R_1(j,s) \), and \( \overline{y}_{R_2} \) is the mean +response for the training observations in \( R_2(j,s) \). + +
+Finding the values of \( j \) and \( s \) that minimize the above equation can be +done quite quickly, especially when the number of features \( p \) is not +too large. + +
+Next, we repeat the process, looking +for the best predictor and best cutpoint in order to split the data +further so as to minimize the MSE within each of the resulting +regions. However, this time, instead of splitting the entire predictor +space, we split one of the two previously identified regions. We now +have three regions. Again, we look to split one of these three regions +further, so as to minimize the MSE. The process continues until a +stopping criterion is reached; for instance, we may continue until no +region contains more than five observations.
@@ -251,7 +284,7 @@ MathJax.Hub.Config({
- + -
-A classification tree is very similar to a regression tree, except -that it is used to predict a qualitative response rather than a -quantitative one. Recall that for a regression tree, the predicted -response for an observation is given by the mean response of the -training observations that belong to the same terminal node. In -contrast, for a classification tree, we predict that each observation -belongs to the most commonly occurring class of training observations -in the region to which it belongs. In interpreting the results of a -classification tree, we are often interested not only in the class -prediction corresponding to a particular terminal node region, but -also in the class proportions among the training observations that -fall into that region. +The above procedure is rather straightforward, but leads often to +overfitting and unnecessarily large and complicated trees. The basic +idea is to grow a large tree \( T_0 \) and then prune it back in order to +obtain a subtree. A smaller tree with fewer splits (fewer regions) can +lead to smaller variance and better interpretation at the cost of a +little more bias. + +
+The so-called Cost complexity pruning algorithm gives us a +way to do just this. Rather than considering every possible subtree, +we consider a sequence of trees indexed by a nonnegative tuning +parameter \( \alpha \).
@@ -243,7 +252,7 @@ fall into that region.
-The task of growing a -classification tree is quite similar to the task of growing a -regression tree. Just as in the regression setting, we use recursive -binary splitting to grow a classification tree. However, in the -classification setting, the MSE cannot be used as a criterion for making -the binary splits. A natural alternative to MSE is the classification -error rate. Since we plan to assign an observation in a given region -to the most commonly occurring error rate class of training -observations in that region, the classification error rate is simply -the fraction of the training observations in that region that do not -belong to the most common class. +The tuning parameter \( \alpha \) controls a trade-off between the subtree’s +com- plexity and its fit to the training data. When \( \alpha = 0 \), then the +subtree \( T \) will simply equal \( T_0 \), +because then the above equation just measures the +training error. +However, as \( \alpha \) increases, there is a price to pay for +having a tree with many terminal nodes. The above equation will +tend to be minimized for a smaller subtree.
-When building a classification tree, either the Gini index or the -entropy are typically used to evaluate the quality of a particular -split, since these two approaches are more sensitive to node purity -than is the classification error rate. +It turns out that as we increase \( \alpha \) from zero +branches get pruned from the tree in a nested and predictable fashion, +so obtaining the whole sequence of subtrees as a function of \( \alpha \) is +easy. We can select a value of \( \alpha \) using a validation set or using +cross-validation. We then return to the full data set and obtain the +subtree corresponding to \( \alpha \).
@@ -248,7 +264,7 @@ than is the classification error rate.
-If our targets are the outcome of a classification process that takes -for example \( k=1,2,\dots,K \) values, the only thing we need to think of -is to set up the splitting criteria for each node. +
-
-We define a PDF \( p_{mk} \) that represents the number of observations of -a class \( k \) in a region \( R_m \) with \( N_m \) observations. We represent -this likelihood function in terms of the proportion \( I(y_i=k) \) of -observations of this class in the region \( R_m \) as - -$$ -p_{mk} = \frac{1}{N_m}\sum_{x_i\in R_m}I(y_i=k). -$$ - -
-We let \( p_{mk} \) represent the majority class of observations in region -\( m \). The three most common ways of splitting a node are given by +
@@ -274,7 +260,7 @@ $$
+A classification tree is very similar to a regression tree, except +that it is used to predict a qualitative response rather than a +quantitative one. Recall that for a regression tree, the predicted +response for an observation is given by the mean response of the +training observations that belong to the same terminal node. In +contrast, for a classification tree, we predict that each observation +belongs to the most commonly occurring class of training observations +in the region to which it belongs. In interpreting the results of a +classification tree, we are often interested not only in the class +prediction corresponding to a particular terminal node region, but +also in the class proportions among the training observations that +fall into that region. - -
import os
-from sklearn.datasets import load_breast_cancer
-from sklearn.tree import DecisionTreeClassifier
-from sklearn.model_selection import train_test_split
-from sklearn.metrics import confusion_matrix
-from sklearn.tree import export_graphviz
-
-from IPython.display import Image
-from pydot import graph_from_dot_data
-import pandas as pd
-import numpy as np
-
-
-cancer = load_breast_cancer()
-X = pd.DataFrame(cancer.data, columns=cancer.feature_names)
-print(X)
-y = pd.Categorical.from_codes(cancer.target, cancer.target_names)
-y = pd.get_dummies(y)
-print(y)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)
-tree_clf = DecisionTreeClassifier(max_depth=5)
-tree_clf.fit(X_train, y_train)
-
-export_graphviz(
- tree_clf,
- out_file="DataFiles/cancer.dot",
- feature_names=cancer.feature_names,
- class_names=cancer.target_names,
- rounded=True,
- filled=True
-)
-cmd = 'dot -Tpng DataFiles/cancer.dot -o DataFiles/cancer.png'
-os.system(cmd)
-
@@ -265,7 +252,7 @@ os.system(cmd)
+The task of growing a +classification tree is quite similar to the task of growing a +regression tree. Just as in the regression setting, we use recursive +binary splitting to grow a classification tree. However, in the +classification setting, the MSE cannot be used as a criterion for making +the binary splits. A natural alternative to MSE is the classification +error rate. Since we plan to assign an observation in a given region +to the most commonly occurring error rate class of training +observations in that region, the classification error rate is simply +the fraction of the training observations in that region that do not +belong to the most common class. - -
# Common imports
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.tree import DecisionTreeClassifier
-from sklearn.datasets import make_moons
-from sklearn.tree import export_graphviz
-from pydot import graph_from_dot_data
-import pandas as pd
-import os
+
+When building a classification tree, either the Gini index or the
+entropy are typically used to evaluate the quality of a particular
+split, since these two approaches are more sensitive to node purity
+than is the classification error rate.
-np.random.seed(42)
-X, y = make_moons(n_samples=100, noise=0.25, random_state=53)
-X_train, X_test, y_train, y_test = train_test_split(X,y,random_state=0)
-tree_clf = DecisionTreeClassifier(max_depth=5)
-tree_clf.fit(X_train, y_train)
-
-export_graphviz(
- tree_clf,
- out_file="DataFiles/moons.dot",
- rounded=True,
- filled=True
-)
-cmd = 'dot -Tpng DataFiles/moons.dot -o DataFiles/moons.png'
-os.system(cmd)
-
@@ -256,7 +257,7 @@ os.system(cmd)
-Two algorithms stand out in the set up of decision trees: +If our targets are the outcome of a classification process that takes +for example \( k=1,2,\dots,K \) values, the only thing we need to think of +is to set up the splitting criteria for each node. -
+We define a PDF \( p_{mk} \) that represents the number of observations of +a class \( k \) in a region \( R_m \) with \( N_m \) observations. We represent +this likelihood function in terms of the proportion \( I(y_i=k) \) of +observations of this class in the region \( R_m \) as -We discuss both algorithms with applications here. The popular library -Scikit-Learn uses the CART algorithm. For classification problems -you can use either the gini index or the entropy to split a tree -in two branches. +$$ +p_{mk} = \frac{1}{N_m}\sum_{x_i\in R_m}I(y_i=k). +$$ + +
+We let \( p_{mk} \) represent the majority class of observations in region +\( m \). The three most common ways of splitting a node are given by + +
@@ -242,7 +283,7 @@ in two branches.
-For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \). -This could be for example a threshold set by a number below a certain circumference of a malign tumor. -
-How do we find these two quantities? -We search for the pair \( (k,t_k) \) that produces the purest subset using for example the gini factor \( G \). -The cost function it tries to minimize is then -$$ -C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}}, -$$ + +
import os
+from sklearn.datasets import load_breast_cancer
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import confusion_matrix
+from sklearn.tree import export_graphviz
-where \( G_{\mathrm{left/right}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \)
- is the number of instances in the left/right subset
+from IPython.display import Image
+from pydot import graph_from_dot_data
+import pandas as pd
+import numpy as np
-
-Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets
-and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the
-\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other
-hyperparameters control additional stopping conditions such as the \( min\_samples\_split \),
-\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \).
+cancer = load_breast_cancer()
+X = pd.DataFrame(cancer.data, columns=cancer.feature_names)
+print(X)
+y = pd.Categorical.from_codes(cancer.target, cancer.target_names)
+y = pd.get_dummies(y)
+print(y)
+X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)
+tree_clf = DecisionTreeClassifier(max_depth=5)
+tree_clf.fit(X_train, y_train)
+
+export_graphviz(
+ tree_clf,
+ out_file="DataFiles/cancer.dot",
+ feature_names=cancer.feature_names,
+ class_names=cancer.target_names,
+ rounded=True,
+ filled=True
+)
+cmd = 'dot -Tpng DataFiles/cancer.dot -o DataFiles/cancer.png'
+os.system(cmd)
+
@@ -251,7 +274,7 @@ hyperparameters control additional stopping conditions such as the \( min\_sampl
-The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the -training set in a way that minimizes say the gini or entropy impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now -$$ -C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}. -$$ -Here the MSE for a specific node is defined as -$$ -\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2, -$$ + +
# Common imports
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.datasets import make_moons
+from sklearn.tree import export_graphviz
+from pydot import graph_from_dot_data
+import pandas as pd
+import os
-with
-$$
-\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i,
-$$
-
-the mean value of all observations in a specific node.
-
-
-Without any regularization, the regression task for decision trees,
-just like for classification tasks, is prone to overfitting.
+np.random.seed(42)
+X, y = make_moons(n_samples=100, noise=0.25, random_state=53)
+X_train, X_test, y_train, y_test = train_test_split(X,y,random_state=0)
+tree_clf = DecisionTreeClassifier(max_depth=5)
+tree_clf.fit(X_train, y_train)
+export_graphviz(
+ tree_clf,
+ out_file="DataFiles/moons.dot",
+ rounded=True,
+ filled=True
+)
+cmd = 'dot -Tpng DataFiles/moons.dot -o DataFiles/moons.png'
+os.system(cmd)
+
@@ -252,7 +265,7 @@ just like for classification tasks, is prone to overfitting.
-The example we will look at is a classical one in many Machine -Learning applications. Based on various meteorological features, we -have several so-called attributes which decide whether we at the end -will do some outdoor activity like skiing, going for a bike ride etc -etc. The table here contains the feautures outlook, temperature, -humidity and wind. The target or output is whether we ride -(True=1) or whether we do something else that day (False=0). The -attributes for each feature are then sunny, overcast and rain for the -outlook, hot, cold and mild for temperature, high and normal for -humidity and weak and strong for wind. +Two algorithms stand out in the set up of decision trees: -
-The table here summarizes the various attributes and +
| Day | Outlook | Temperature | Humidity | Wind | Ride |
| 1 | Sunny | Hot | High | Weak | 0 |
| 2 | Sunny | Hot | High | Strong | 1 |
| 3 | Overcast | Hot | High | Weak | 1 |
| 4 | Rain | Mild | High | Weak | 1 |
| 5 | Rain | Cool | Normal | Weak | 1 |
| 6 | Rain | Cool | Normal | Strong | 0 |
| 7 | Overcast | Cool | Normal | Strong | 1 |
| 8 | Sunny | Mild | High | Weak | 0 |
| 9 | Sunny | Cool | Normal | Weak | 1 |
| 10 | Rain | Mild | Normal | Weak | 1 |
| 11 | Sunny | Mild | Normal | Strong | 1 |
| 12 | Overcast | Mild | High | Strong | 1 |
| 13 | Overcast | Hot | Normal | Weak | 1 |
| 14 | Rain | Mild | High | Strong | 0 |
@@ -269,7 +251,7 @@ The table here summarizes the various attributes and
+For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \). +This could be for example a threshold set by a number below a certain circumference of a malign tumor. - -
# Common imports
-import numpy as np
-import pandas as pd
-import matplotlib.pyplot as plt
-from sklearn.tree import DecisionTreeClassifier
-from sklearn.model_selection import train_test_split
-from sklearn.tree import export_graphviz
-from sklearn.preprocessing import StandardScaler, OneHotEncoder
-from sklearn.compose import ColumnTransformer
-from IPython.display import Image
-from pydot import graph_from_dot_data
-import os
+
+How do we find these two quantities?
+We search for the pair \( (k,t_k) \) that produces the purest subset using for example the gini factor \( G \).
+The cost function it tries to minimize is then
+$$
+C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}},
+$$
-# Where to save the figures and data files
-PROJECT_ROOT_DIR = "Results"
-FIGURE_ID = "Results/FigureFiles"
-DATA_ID = "DataFiles/"
+where \( G_{\mathrm{left/right}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \)
+ is the number of instances in the left/right subset
-if not os.path.exists(PROJECT_ROOT_DIR):
- os.mkdir(PROJECT_ROOT_DIR)
+
+Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets
+and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the
+\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other
+hyperparameters control additional stopping conditions such as the \( min\_samples\_split \),
+\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \).
-if not os.path.exists(FIGURE_ID):
- os.makedirs(FIGURE_ID)
-
-if not os.path.exists(DATA_ID):
- os.makedirs(DATA_ID)
-
-def image_path(fig_id):
- return os.path.join(FIGURE_ID, fig_id)
-
-def data_path(dat_id):
- return os.path.join(DATA_ID, dat_id)
-
-def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
-
-infile = open(data_path("rideclass.csv"),'r')
-
-# Read the experimental data with Pandas
-from IPython.display import display
-ridedata = pd.read_csv(infile,names = ('Outlook','Temperature','Humidity','Wind','Ride'))
-ridedata = pd.DataFrame(ridedata)
-
-# Features and targets
-X = ridedata.loc[:, ridedata.columns != 'Ride'].values
-y = ridedata.loc[:, ridedata.columns == 'Ride'].values
-
-# Create the encoder.
-encoder = OneHotEncoder(handle_unknown="ignore")
-# Assume for simplicity all features are categorical.
-encoder.fit(X)
-# Apply the encoder.
-X = encoder.transform(X)
-print(X)
-# Then do a Classification tree
-tree_clf = DecisionTreeClassifier(max_depth=2)
-tree_clf.fit(X, y)
-print("Train set accuracy with Decision Tree: {:.2f}".format(tree_clf.score(X,y)))
-#transfer to a decision tree graph
-export_graphviz(
- tree_clf,
- out_file="DataFiles/ride.dot",
- rounded=True,
- filled=True
-)
-cmd = 'dot -Tpng DataFiles/cancer.dot -o DataFiles/cancer.png'
-os.system(cmd)
-
@@ -300,7 +260,7 @@ os.system(cmd)
-The above functions (gini, entropy and misclassification error) are -important components of the so-called CART algorithm. We will discuss -this algorithm below after we have discussed the information gain -algorithm ID3. +The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the +training set in a way that minimizes say the gini or entropy impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now +$$ +C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}. +$$ + +Here the MSE for a specific node is defined as +$$ +\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2, +$$ + +with +$$ +\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i, +$$ + +the mean value of all observations in a specific node.
-In the example here we have converted all our attributes into numerical values \( 0,1,2 \) etc. +Without any regularization, the regression task for decision trees, +just like for classification tasks, is prone to overfitting. -
- - -
# Split a dataset based on an attribute and an attribute value
-def test_split(index, value, dataset):
- left, right = list(), list()
- for row in dataset:
- if row[index] < value:
- left.append(row)
- else:
- right.append(row)
- return left, right
-
-# Calculate the Gini index for a split dataset
-def gini_index(groups, classes):
- # count all samples at split point
- n_instances = float(sum([len(group) for group in groups]))
- # sum weighted Gini index for each group
- gini = 0.0
- for group in groups:
- size = float(len(group))
- # avoid divide by zero
- if size == 0:
- continue
- score = 0.0
- # score the group based on the score for each class
- for class_val in classes:
- p = [row[-1] for row in group].count(class_val) / size
- score += p * p
- # weight the group score by its relative size
- gini += (1.0 - score) * (size / n_instances)
- return gini
-
-# Select the best split point for a dataset
-def get_split(dataset):
- class_values = list(set(row[-1] for row in dataset))
- b_index, b_value, b_score, b_groups = 999, 999, 999, None
- for index in range(len(dataset[0])-1):
- for row in dataset:
- groups = test_split(index, row[index], dataset)
- gini = gini_index(groups, class_values)
- print('X%d < %.3f Gini=%.3f' % ((index+1), row[index], gini))
- if gini < b_score:
- b_index, b_value, b_score, b_groups = index, row[index], gini, groups
- return {'index':b_index, 'value':b_value, 'groups':b_groups}
-
-dataset = [[0,0,0,0,0],
- [0,0,0,1,1],
- [1,0,0,0,1],
- [2,1,0,0,1],
- [2,2,1,0,1],
- [2,2,1,1,0],
- [1,2,1,1,1],
- [0,1,0,0,0],
- [0,2,1,0,1],
- [2,1,1,0,1],
- [0,1,1,1,1],
- [1,1,0,1,1],
- [1,0,1,0,1],
- [2,1,0,1,0]]
-
-split = get_split(dataset)
-print('Split: [X%d < %.3f]' % ((split['index']+1), split['value']))
-
@@ -302,7 +261,7 @@ split = get_split(dataset)
-ID3, learns decision trees by constructing -them topdown, beginning with the question which attribute should be tested at the root of the tree? - -
-We would like to select the attribute that is most useful for classifying -examples. - -
-What is a good quantitative measure of the worth of an attribute? - -
-Information gain measures how well a given attribute separates the -training examples according to their target classification. - -
-The ID3 algorithm uses this information gain measure to select among the candidate -attributes at each step while growing the tree. +The table here summarizes the various attributes and +
| Day | Outlook | Temperature | Humidity | Wind | Ride |
| 1 | Sunny | Hot | High | Weak | 0 |
| 2 | Sunny | Hot | High | Strong | 1 |
| 3 | Overcast | Hot | High | Weak | 1 |
| 4 | Rain | Mild | High | Weak | 1 |
| 5 | Rain | Cool | Normal | Weak | 1 |
| 6 | Rain | Cool | Normal | Strong | 0 |
| 7 | Overcast | Cool | Normal | Strong | 1 |
| 8 | Sunny | Mild | High | Weak | 0 |
| 9 | Sunny | Cool | Normal | Weak | 1 |
| 10 | Rain | Mild | Normal | Weak | 1 |
| 11 | Sunny | Mild | Normal | Strong | 1 |
| 12 | Overcast | Mild | High | Strong | 1 |
| 13 | Overcast | Hot | Normal | Weak | 1 |
| 14 | Rain | Mild | High | Strong | 0 |
@@ -260,7 +278,7 @@ attributes at each step while growing the tree.
-
import re
-import math
-from collections import deque
+# Common imports
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.model_selection import train_test_split
+from sklearn.tree import export_graphviz
+from sklearn.preprocessing import StandardScaler, OneHotEncoder
+from sklearn.compose import ColumnTransformer
+from IPython.display import Image
+from pydot import graph_from_dot_data
+import os
-# x is examples in training set
-# y is set of targets
-# label is target attributes
-# Node is a class which has properties values, childs, and next
-# root is top node in the decision tree
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
-class Node(object):
- def __init__(self):
- self.value = None
- self.next = None
- self.childs = None
+if not os.path.exists(PROJECT_ROOT_DIR):
+ os.mkdir(PROJECT_ROOT_DIR)
-# Simple class of Decision Tree
-# Aimed for who want to learn Decision Tree, so it is not optimized
-class DecisionTree(object):
- def __init__(self, sample, attributes, labels):
- self.sample = sample
- self.attributes = attributes
- self.labels = labels
- self.labelCodes = None
- self.labelCodesCount = None
- self.initLabelCodes()
- # print(self.labelCodes)
- self.root = None
- self.entropy = self.getEntropy([x for x in range(len(self.labels))])
+if not os.path.exists(FIGURE_ID):
+ os.makedirs(FIGURE_ID)
- def initLabelCodes(self):
- self.labelCodes = []
- self.labelCodesCount = []
- for l in self.labels:
- if l not in self.labelCodes:
- self.labelCodes.append(l)
- self.labelCodesCount.append(0)
- self.labelCodesCount[self.labelCodes.index(l)] += 1
+if not os.path.exists(DATA_ID):
+ os.makedirs(DATA_ID)
- def getLabelCodeId(self, sampleId):
- return self.labelCodes.index(self.labels[sampleId])
+def image_path(fig_id):
+ return os.path.join(FIGURE_ID, fig_id)
- def getAttributeValues(self, sampleIds, attributeId):
- vals = []
- for sid in sampleIds:
- val = self.sample[sid][attributeId]
- if val not in vals:
- vals.append(val)
- # print(vals)
- return vals
+def data_path(dat_id):
+ return os.path.join(DATA_ID, dat_id)
- def getEntropy(self, sampleIds):
- entropy = 0
- labelCount = [0] * len(self.labelCodes)
- for sid in sampleIds:
- labelCount[self.getLabelCodeId(sid)] += 1
- # print("-ge", labelCount)
- for lv in labelCount:
- # print(lv)
- if lv != 0:
- entropy += -lv/len(sampleIds) * math.log(lv/len(sampleIds), 2)
- else:
- entropy += 0
- return entropy
+def save_fig(fig_id):
+ plt.savefig(image_path(fig_id) + ".png", format='png')
- def getDominantLabel(self, sampleIds):
- labelCodesCount = [0] * len(self.labelCodes)
- for sid in sampleIds:
- labelCodesCount[self.labelCodes.index(self.labels[sid])] += 1
- return self.labelCodes[labelCodesCount.index(max(labelCodesCount))]
+infile = open(data_path("rideclass.csv"),'r')
- def getInformationGain(self, sampleIds, attributeId):
- gain = self.getEntropy(sampleIds)
- attributeVals = []
- attributeValsCount = []
- attributeValsIds = []
- for sid in sampleIds:
- val = self.sample[sid][attributeId]
- if val not in attributeVals:
- attributeVals.append(val)
- attributeValsCount.append(0)
- attributeValsIds.append([])
- vid = attributeVals.index(val)
- attributeValsCount[vid] += 1
- attributeValsIds[vid].append(sid)
- # print("-gig", self.attributes[attributeId])
- for vc, vids in zip(attributeValsCount, attributeValsIds):
- # print("-gig", vids)
- gain -= vc/len(sampleIds) * self.getEntropy(vids)
- return gain
+# Read the experimental data with Pandas
+from IPython.display import display
+ridedata = pd.read_csv(infile,names = ('Outlook','Temperature','Humidity','Wind','Ride'))
+ridedata = pd.DataFrame(ridedata)
- def getAttributeMaxInformationGain(self, sampleIds, attributeIds):
- attributesEntropy = [0] * len(attributeIds)
- for i, attId in zip(range(len(attributeIds)), attributeIds):
- attributesEntropy[i] = self.getInformationGain(sampleIds, attId)
- maxId = attributeIds[attributesEntropy.index(max(attributesEntropy))]
- return self.attributes[maxId], maxId
+# Features and targets
+X = ridedata.loc[:, ridedata.columns != 'Ride'].values
+y = ridedata.loc[:, ridedata.columns == 'Ride'].values
- def isSingleLabeled(self, sampleIds):
- label = self.labels[sampleIds[0]]
- for sid in sampleIds:
- if self.labels[sid] != label:
- return False
- return True
-
- def getLabel(self, sampleId):
- return self.labels[sampleId]
-
- def id3(self):
- sampleIds = [x for x in range(len(self.sample))]
- attributeIds = [x for x in range(len(self.attributes))]
- self.root = self.id3Recv(sampleIds, attributeIds, self.root)
-
- def id3Recv(self, sampleIds, attributeIds, root):
- root = Node() # Initialize current root
- if self.isSingleLabeled(sampleIds):
- root.value = self.labels[sampleIds[0]]
- return root
- # print(attributeIds)
- if len(attributeIds) == 0:
- root.value = self.getDominantLabel(sampleIds)
- return root
- bestAttrName, bestAttrId = self.getAttributeMaxInformationGain(
- sampleIds, attributeIds)
- # print(bestAttrName)
- root.value = bestAttrName
- root.childs = [] # Create list of children
- for value in self.getAttributeValues(sampleIds, bestAttrId):
- # print(value)
- child = Node()
- child.value = value
- root.childs.append(child) # Append new child node to current
- # root
- childSampleIds = []
- for sid in sampleIds:
- if self.sample[sid][bestAttrId] == value:
- childSampleIds.append(sid)
- if len(childSampleIds) == 0:
- child.next = self.getDominantLabel(sampleIds)
- else:
- # print(bestAttrName, bestAttrId)
- # print(attributeIds)
- if len(attributeIds) > 0 and bestAttrId in attributeIds:
- toRemove = attributeIds.index(bestAttrId)
- attributeIds.pop(toRemove)
- child.next = self.id3Recv(
- childSampleIds, attributeIds, child.next)
- return root
-
- def printTree(self):
- if self.root:
- roots = deque()
- roots.append(self.root)
- while len(roots) > 0:
- root = roots.popleft()
- print(root.value)
- if root.childs:
- for child in root.childs:
- print('({})'.format(child.value))
- roots.append(child.next)
- elif root.next:
- print(root.next)
-
-
-def test():
- f = open('DataFiles/rideclass.csv')
- attributes = f.readline().split(',')
- attributes = attributes[1:len(attributes)-1]
- print(attributes)
- sample = f.readlines()
- f.close()
- for i in range(len(sample)):
- sample[i] = re.sub('\d+,', '', sample[i])
- sample[i] = sample[i].strip().split(',')
- labels = []
- for s in sample:
- labels.append(s.pop())
- # print(sample)
- # print(labels)
- decisionTree = DecisionTree(sample, attributes, labels)
- print("System entropy {}".format(decisionTree.entropy))
- decisionTree.id3()
- decisionTree.printTree()
-
-
-if __name__ == '__main__':
- test()
+# Create the encoder.
+encoder = OneHotEncoder(handle_unknown="ignore")
+# Assume for simplicity all features are categorical.
+encoder.fit(X)
+# Apply the encoder.
+X = encoder.transform(X)
+print(X)
+# Then do a Classification tree
+tree_clf = DecisionTreeClassifier(max_depth=2)
+tree_clf.fit(X, y)
+print("Train set accuracy with Decision Tree: {:.2f}".format(tree_clf.score(X,y)))
+#transfer to a decision tree graph
+export_graphviz(
+ tree_clf,
+ out_file="DataFiles/ride.dot",
+ rounded=True,
+ filled=True
+)
+cmd = 'dot -Tpng DataFiles/cancer.dot -o DataFiles/cancer.png'
+os.system(cmd)
@@ -420,7 +309,7 @@ MathJax.Hub.Config({
+The above functions (gini, entropy and misclassification error) are +important components of the so-called CART algorithm. We will discuss +this algorithm below after we have discussed the information gain +algorithm ID3. + +
+In the example here we have converted all our attributes into numerical values \( 0,1,2 \) etc. +
-
import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import load_breast_cancer
-from sklearn.svm import SVC
-from sklearn.linear_model import LogisticRegression
-from sklearn.tree import DecisionTreeClassifier
+# Split a dataset based on an attribute and an attribute value
+def test_split(index, value, dataset):
+ left, right = list(), list()
+ for row in dataset:
+ if row[index] < value:
+ left.append(row)
+ else:
+ right.append(row)
+ return left, right
+
+# Calculate the Gini index for a split dataset
+def gini_index(groups, classes):
+ # count all samples at split point
+ n_instances = float(sum([len(group) for group in groups]))
+ # sum weighted Gini index for each group
+ gini = 0.0
+ for group in groups:
+ size = float(len(group))
+ # avoid divide by zero
+ if size == 0:
+ continue
+ score = 0.0
+ # score the group based on the score for each class
+ for class_val in classes:
+ p = [row[-1] for row in group].count(class_val) / size
+ score += p * p
+ # weight the group score by its relative size
+ gini += (1.0 - score) * (size / n_instances)
+ return gini
-# Load the data
-cancer = load_breast_cancer()
+# Select the best split point for a dataset
+def get_split(dataset):
+ class_values = list(set(row[-1] for row in dataset))
+ b_index, b_value, b_score, b_groups = 999, 999, 999, None
+ for index in range(len(dataset[0])-1):
+ for row in dataset:
+ groups = test_split(index, row[index], dataset)
+ gini = gini_index(groups, class_values)
+ print('X%d < %.3f Gini=%.3f' % ((index+1), row[index], gini))
+ if gini < b_score:
+ b_index, b_value, b_score, b_groups = index, row[index], gini, groups
+ return {'index':b_index, 'value':b_value, 'groups':b_groups}
+
+dataset = [[0,0,0,0,0],
+ [0,0,0,1,1],
+ [1,0,0,0,1],
+ [2,1,0,0,1],
+ [2,2,1,0,1],
+ [2,2,1,1,0],
+ [1,2,1,1,1],
+ [0,1,0,0,0],
+ [0,2,1,0,1],
+ [2,1,1,0,1],
+ [0,1,1,1,1],
+ [1,1,0,1,1],
+ [1,0,1,0,1],
+ [2,1,0,1,0]]
-X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
-print(X_train.shape)
-print(X_test.shape)
-# Logistic Regression
-logreg = LogisticRegression(solver='lbfgs')
-logreg.fit(X_train, y_train)
-print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
-# Support vector machine
-svm = SVC(gamma='auto', C=100)
-svm.fit(X_train, y_train)
-print("Test set accuracy with SVM: {:.2f}".format(svm.score(X_test,y_test)))
-# Decision Trees
-deep_tree_clf = DecisionTreeClassifier(max_depth=None)
-deep_tree_clf.fit(X_train, y_train)
-print("Test set accuracy with Decision Trees: {:.2f}".format(deep_tree_clf.score(X_test,y_test)))
-#now scale the data
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-# Logistic Regression
-logreg.fit(X_train_scaled, y_train)
-print("Test set accuracy Logistic Regression with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
-# Support Vector Machine
-svm.fit(X_train_scaled, y_train)
-print("Test set accuracy SVM with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
-# Decision Trees
-deep_tree_clf.fit(X_train_scaled, y_train)
-print("Test set accuracy with Decision Trees and scaled data: {:.2f}".format(deep_tree_clf.score(X_test_scaled,y_test)))
+split = get_split(dataset)
+print('Split: [X%d < %.3f]' % ((split['index']+1), split['value']))
@@ -273,7 +311,7 @@ deep_tree_clf.fit(X_train_scaled, y_train)
+ID3, learns decision trees by constructing +them topdown, beginning with the question which attribute should be tested at the root of the tree? - -
from __future__ import division, print_function, unicode_literals
+
+- Each instance attribute is evaluated using a statistical test to determine how well it alone classifies the training examples.
+- The best attribute is selected and used as the test at the root node of the tree.
+- A descendant of the root node is then created for each possible value of this attribute.
+- Training examples are sorted to the appropriate descendant node.
+- The entire process is then repeated using the training examples associated with each descendant node to select the best attribute to test at that point in the tree.
+- This forms a greedy search for an acceptable decision tree, in which the algorithm never backtracks to reconsider earlier choices.
+
-# Common imports
-import numpy as np
-import os
+The ID3 algorithm selects, which attribute to test at each node in the
+tree.
-# to make this notebook's output stable across runs
-np.random.seed(42)
+
+We would like to select the attribute that is most useful for classifying
+examples.
-# To plot pretty figures
-import matplotlib
-import matplotlib.pyplot as plt
-from matplotlib.colors import ListedColormap
-plt.rcParams['axes.labelsize'] = 14
-plt.rcParams['xtick.labelsize'] = 12
-plt.rcParams['ytick.labelsize'] = 12
+
+What is a good quantitative measure of the worth of an attribute?
+
+Information gain measures how well a given attribute separates the
+training examples according to their target classification.
-from sklearn.svm import SVC
-from sklearn import datasets
-from sklearn.tree import DecisionTreeClassifier
-from sklearn.datasets import make_moons
-from sklearn.tree import export_graphviz
+
+The ID3 algorithm uses this information gain measure to select among the candidate
+attributes at each step while growing the tree.
-Xm, ym = make_moons(n_samples=100, noise=0.25, random_state=53)
-
-deep_tree_clf1 = DecisionTreeClassifier(random_state=42)
-deep_tree_clf2 = DecisionTreeClassifier(min_samples_leaf=4, random_state=42)
-deep_tree_clf1.fit(Xm, ym)
-deep_tree_clf2.fit(Xm, ym)
-
-
-def plot_decision_boundary(clf, X, y, axes=[0, 7.5, 0, 3], iris=True, legend=False, plot_training=True):
- x1s = np.linspace(axes[0], axes[1], 100)
- x2s = np.linspace(axes[2], axes[3], 100)
- x1, x2 = np.meshgrid(x1s, x2s)
- X_new = np.c_[x1.ravel(), x2.ravel()]
- y_pred = clf.predict(X_new).reshape(x1.shape)
- custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
- plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
- if not iris:
- custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
- plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
- if plot_training:
- plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", label="Iris-Setosa")
- plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", label="Iris-Versicolor")
- plt.plot(X[:, 0][y==2], X[:, 1][y==2], "g^", label="Iris-Virginica")
- plt.axis(axes)
- if iris:
- plt.xlabel("Petal length", fontsize=14)
- plt.ylabel("Petal width", fontsize=14)
- else:
- plt.xlabel(r"$x_1$", fontsize=18)
- plt.ylabel(r"$x_2$", fontsize=18, rotation=0)
- if legend:
- plt.legend(loc="lower right", fontsize=14)
-plt.figure(figsize=(11, 4))
-plt.subplot(121)
-plot_decision_boundary(deep_tree_clf1, Xm, ym, axes=[-1.5, 2.5, -1, 1.5], iris=False)
-plt.title("No restrictions", fontsize=16)
-plt.subplot(122)
-plot_decision_boundary(deep_tree_clf2, Xm, ym, axes=[-1.5, 2.5, -1, 1.5], iris=False)
-plt.title("min_samples_leaf = {}".format(deep_tree_clf2.min_samples_leaf), fontsize=14)
-plt.show()
-
@@ -296,7 +269,7 @@ plt.show()
-
np.random.seed(6)
-Xs = np.random.rand(100, 2) - 0.5
-ys = (Xs[:, 0] > 0).astype(np.float32) * 2
+import re
+import math
+from collections import deque
-angle = np.pi/4
-rotation_matrix = np.array([[np.cos(angle), -np.sin(angle)], [np.sin(angle), np.cos(angle)]])
-Xsr = Xs.dot(rotation_matrix)
+# x is examples in training set
+# y is set of targets
+# label is target attributes
+# Node is a class which has properties values, childs, and next
+# root is top node in the decision tree
-tree_clf_s = DecisionTreeClassifier(random_state=42)
-tree_clf_s.fit(Xs, ys)
-tree_clf_sr = DecisionTreeClassifier(random_state=42)
-tree_clf_sr.fit(Xsr, ys)
+class Node(object):
+ def __init__(self):
+ self.value = None
+ self.next = None
+ self.childs = None
-plt.figure(figsize=(11, 4))
-plt.subplot(121)
-plot_decision_boundary(tree_clf_s, Xs, ys, axes=[-0.7, 0.7, -0.7, 0.7], iris=False)
-plt.subplot(122)
-plot_decision_boundary(tree_clf_sr, Xsr, ys, axes=[-0.7, 0.7, -0.7, 0.7], iris=False)
+# Simple class of Decision Tree
+# Aimed for who want to learn Decision Tree, so it is not optimized
+class DecisionTree(object):
+ def __init__(self, sample, attributes, labels):
+ self.sample = sample
+ self.attributes = attributes
+ self.labels = labels
+ self.labelCodes = None
+ self.labelCodesCount = None
+ self.initLabelCodes()
+ # print(self.labelCodes)
+ self.root = None
+ self.entropy = self.getEntropy([x for x in range(len(self.labels))])
-plt.show()
+ def initLabelCodes(self):
+ self.labelCodes = []
+ self.labelCodesCount = []
+ for l in self.labels:
+ if l not in self.labelCodes:
+ self.labelCodes.append(l)
+ self.labelCodesCount.append(0)
+ self.labelCodesCount[self.labelCodes.index(l)] += 1
+
+ def getLabelCodeId(self, sampleId):
+ return self.labelCodes.index(self.labels[sampleId])
+
+ def getAttributeValues(self, sampleIds, attributeId):
+ vals = []
+ for sid in sampleIds:
+ val = self.sample[sid][attributeId]
+ if val not in vals:
+ vals.append(val)
+ # print(vals)
+ return vals
+
+ def getEntropy(self, sampleIds):
+ entropy = 0
+ labelCount = [0] * len(self.labelCodes)
+ for sid in sampleIds:
+ labelCount[self.getLabelCodeId(sid)] += 1
+ # print("-ge", labelCount)
+ for lv in labelCount:
+ # print(lv)
+ if lv != 0:
+ entropy += -lv/len(sampleIds) * math.log(lv/len(sampleIds), 2)
+ else:
+ entropy += 0
+ return entropy
+
+ def getDominantLabel(self, sampleIds):
+ labelCodesCount = [0] * len(self.labelCodes)
+ for sid in sampleIds:
+ labelCodesCount[self.labelCodes.index(self.labels[sid])] += 1
+ return self.labelCodes[labelCodesCount.index(max(labelCodesCount))]
+
+ def getInformationGain(self, sampleIds, attributeId):
+ gain = self.getEntropy(sampleIds)
+ attributeVals = []
+ attributeValsCount = []
+ attributeValsIds = []
+ for sid in sampleIds:
+ val = self.sample[sid][attributeId]
+ if val not in attributeVals:
+ attributeVals.append(val)
+ attributeValsCount.append(0)
+ attributeValsIds.append([])
+ vid = attributeVals.index(val)
+ attributeValsCount[vid] += 1
+ attributeValsIds[vid].append(sid)
+ # print("-gig", self.attributes[attributeId])
+ for vc, vids in zip(attributeValsCount, attributeValsIds):
+ # print("-gig", vids)
+ gain -= vc/len(sampleIds) * self.getEntropy(vids)
+ return gain
+
+ def getAttributeMaxInformationGain(self, sampleIds, attributeIds):
+ attributesEntropy = [0] * len(attributeIds)
+ for i, attId in zip(range(len(attributeIds)), attributeIds):
+ attributesEntropy[i] = self.getInformationGain(sampleIds, attId)
+ maxId = attributeIds[attributesEntropy.index(max(attributesEntropy))]
+ return self.attributes[maxId], maxId
+
+ def isSingleLabeled(self, sampleIds):
+ label = self.labels[sampleIds[0]]
+ for sid in sampleIds:
+ if self.labels[sid] != label:
+ return False
+ return True
+
+ def getLabel(self, sampleId):
+ return self.labels[sampleId]
+
+ def id3(self):
+ sampleIds = [x for x in range(len(self.sample))]
+ attributeIds = [x for x in range(len(self.attributes))]
+ self.root = self.id3Recv(sampleIds, attributeIds, self.root)
+
+ def id3Recv(self, sampleIds, attributeIds, root):
+ root = Node() # Initialize current root
+ if self.isSingleLabeled(sampleIds):
+ root.value = self.labels[sampleIds[0]]
+ return root
+ # print(attributeIds)
+ if len(attributeIds) == 0:
+ root.value = self.getDominantLabel(sampleIds)
+ return root
+ bestAttrName, bestAttrId = self.getAttributeMaxInformationGain(
+ sampleIds, attributeIds)
+ # print(bestAttrName)
+ root.value = bestAttrName
+ root.childs = [] # Create list of children
+ for value in self.getAttributeValues(sampleIds, bestAttrId):
+ # print(value)
+ child = Node()
+ child.value = value
+ root.childs.append(child) # Append new child node to current
+ # root
+ childSampleIds = []
+ for sid in sampleIds:
+ if self.sample[sid][bestAttrId] == value:
+ childSampleIds.append(sid)
+ if len(childSampleIds) == 0:
+ child.next = self.getDominantLabel(sampleIds)
+ else:
+ # print(bestAttrName, bestAttrId)
+ # print(attributeIds)
+ if len(attributeIds) > 0 and bestAttrId in attributeIds:
+ toRemove = attributeIds.index(bestAttrId)
+ attributeIds.pop(toRemove)
+ child.next = self.id3Recv(
+ childSampleIds, attributeIds, child.next)
+ return root
+
+ def printTree(self):
+ if self.root:
+ roots = deque()
+ roots.append(self.root)
+ while len(roots) > 0:
+ root = roots.popleft()
+ print(root.value)
+ if root.childs:
+ for child in root.childs:
+ print('({})'.format(child.value))
+ roots.append(child.next)
+ elif root.next:
+ print(root.next)
+
+
+def test():
+ f = open('DataFiles/rideclass.csv')
+ attributes = f.readline().split(',')
+ attributes = attributes[1:len(attributes)-1]
+ print(attributes)
+ sample = f.readlines()
+ f.close()
+ for i in range(len(sample)):
+ sample[i] = re.sub('\d+,', '', sample[i])
+ sample[i] = sample[i].strip().split(',')
+ labels = []
+ for s in sample:
+ labels.append(s.pop())
+ # print(sample)
+ # print(labels)
+ decisionTree = DecisionTree(sample, attributes, labels)
+ print("System entropy {}".format(decisionTree.entropy))
+ decisionTree.id3()
+ decisionTree.printTree()
+
+
+if __name__ == '__main__':
+ test()
@@ -252,7 +429,7 @@ plt.show()
-
# Quadratic training set + noise
-np.random.seed(42)
-m = 200
-X = np.random.rand(m, 1)
-y = 4 * (X - 0.5) ** 2
-y = y + np.random.randn(m, 1) / 10
-+
import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import load_breast_cancer
+from sklearn.svm import SVC
+from sklearn.linear_model import LogisticRegression
+from sklearn.tree import DecisionTreeClassifier
-
-from sklearn.tree import DecisionTreeRegressor
+# Load the data
+cancer = load_breast_cancer()
-tree_reg = DecisionTreeRegressor(max_depth=2, random_state=42)
-tree_reg.fit(X, y)
+X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
+print(X_train.shape)
+print(X_test.shape)
+# Logistic Regression
+logreg = LogisticRegression(solver='lbfgs')
+logreg.fit(X_train, y_train)
+print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
+# Support vector machine
+svm = SVC(gamma='auto', C=100)
+svm.fit(X_train, y_train)
+print("Test set accuracy with SVM: {:.2f}".format(svm.score(X_test,y_test)))
+# Decision Trees
+deep_tree_clf = DecisionTreeClassifier(max_depth=None)
+deep_tree_clf.fit(X_train, y_train)
+print("Test set accuracy with Decision Trees: {:.2f}".format(deep_tree_clf.score(X_test,y_test)))
+#now scale the data
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+# Logistic Regression
+logreg.fit(X_train_scaled, y_train)
+print("Test set accuracy Logistic Regression with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
+# Support Vector Machine
+svm.fit(X_train_scaled, y_train)
+print("Test set accuracy SVM with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
+# Decision Trees
+deep_tree_clf.fit(X_train_scaled, y_train)
+print("Test set accuracy with Decision Trees and scaled data: {:.2f}".format(deep_tree_clf.score(X_test_scaled,y_test)))
@@ -246,7 +282,7 @@ tree_reg.fit(X, y)
-
from sklearn.tree import DecisionTreeRegressor
+from __future__ import division, print_function, unicode_literals
-tree_reg1 = DecisionTreeRegressor(random_state=42, max_depth=2)
-tree_reg2 = DecisionTreeRegressor(random_state=42, max_depth=3)
-tree_reg1.fit(X, y)
-tree_reg2.fit(X, y)
+# Common imports
+import numpy as np
+import os
-def plot_regression_predictions(tree_reg, X, y, axes=[0, 1, -0.2, 1], ylabel="$y$"):
- x1 = np.linspace(axes[0], axes[1], 500).reshape(-1, 1)
- y_pred = tree_reg.predict(x1)
- plt.axis(axes)
- plt.xlabel("$x_1$", fontsize=18)
- if ylabel:
- plt.ylabel(ylabel, fontsize=18, rotation=0)
- plt.plot(X, y, "b.")
- plt.plot(x1, y_pred, "r.-", linewidth=2, label=r"$\hat{y}$")
+# to make this notebook's output stable across runs
+np.random.seed(42)
+# To plot pretty figures
+import matplotlib
+import matplotlib.pyplot as plt
+from matplotlib.colors import ListedColormap
+plt.rcParams['axes.labelsize'] = 14
+plt.rcParams['xtick.labelsize'] = 12
+plt.rcParams['ytick.labelsize'] = 12
+
+
+from sklearn.svm import SVC
+from sklearn import datasets
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.datasets import make_moons
+from sklearn.tree import export_graphviz
+
+Xm, ym = make_moons(n_samples=100, noise=0.25, random_state=53)
+
+deep_tree_clf1 = DecisionTreeClassifier(random_state=42)
+deep_tree_clf2 = DecisionTreeClassifier(min_samples_leaf=4, random_state=42)
+deep_tree_clf1.fit(Xm, ym)
+deep_tree_clf2.fit(Xm, ym)
+
+
+def plot_decision_boundary(clf, X, y, axes=[0, 7.5, 0, 3], iris=True, legend=False, plot_training=True):
+ x1s = np.linspace(axes[0], axes[1], 100)
+ x2s = np.linspace(axes[2], axes[3], 100)
+ x1, x2 = np.meshgrid(x1s, x2s)
+ X_new = np.c_[x1.ravel(), x2.ravel()]
+ y_pred = clf.predict(X_new).reshape(x1.shape)
+ custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
+ plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
+ if not iris:
+ custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
+ plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
+ if plot_training:
+ plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", label="Iris-Setosa")
+ plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", label="Iris-Versicolor")
+ plt.plot(X[:, 0][y==2], X[:, 1][y==2], "g^", label="Iris-Virginica")
+ plt.axis(axes)
+ if iris:
+ plt.xlabel("Petal length", fontsize=14)
+ plt.ylabel("Petal width", fontsize=14)
+ else:
+ plt.xlabel(r"$x_1$", fontsize=18)
+ plt.ylabel(r"$x_2$", fontsize=18, rotation=0)
+ if legend:
+ plt.legend(loc="lower right", fontsize=14)
plt.figure(figsize=(11, 4))
plt.subplot(121)
-plot_regression_predictions(tree_reg1, X, y)
-for split, style in ((0.1973, "k-"), (0.0917, "k--"), (0.7718, "k--")):
- plt.plot([split, split], [-0.2, 1], style, linewidth=2)
-plt.text(0.21, 0.65, "Depth=0", fontsize=15)
-plt.text(0.01, 0.2, "Depth=1", fontsize=13)
-plt.text(0.65, 0.8, "Depth=1", fontsize=13)
-plt.legend(loc="upper center", fontsize=18)
-plt.title("max_depth=2", fontsize=14)
-
+plot_decision_boundary(deep_tree_clf1, Xm, ym, axes=[-1.5, 2.5, -1, 1.5], iris=False)
+plt.title("No restrictions", fontsize=16)
plt.subplot(122)
-plot_regression_predictions(tree_reg2, X, y, ylabel=None)
-for split, style in ((0.1973, "k-"), (0.0917, "k--"), (0.7718, "k--")):
- plt.plot([split, split], [-0.2, 1], style, linewidth=2)
-for split in (0.0458, 0.1298, 0.2873, 0.9040):
- plt.plot([split, split], [-0.2, 1], "k:", linewidth=1)
-plt.text(0.3, 0.5, "Depth=2", fontsize=13)
-plt.title("max_depth=3", fontsize=14)
-
-plt.show()
-
-
-
-
-
tree_reg1 = DecisionTreeRegressor(random_state=42)
-tree_reg2 = DecisionTreeRegressor(random_state=42, min_samples_leaf=10)
-tree_reg1.fit(X, y)
-tree_reg2.fit(X, y)
-
-x1 = np.linspace(0, 1, 500).reshape(-1, 1)
-y_pred1 = tree_reg1.predict(x1)
-y_pred2 = tree_reg2.predict(x1)
-
-plt.figure(figsize=(11, 4))
-
-plt.subplot(121)
-plt.plot(X, y, "b.")
-plt.plot(x1, y_pred1, "r.-", linewidth=2, label=r"$\hat{y}$")
-plt.axis([0, 1, -0.2, 1.1])
-plt.xlabel("$x_1$", fontsize=18)
-plt.ylabel("$y$", fontsize=18, rotation=0)
-plt.legend(loc="upper center", fontsize=18)
-plt.title("No restrictions", fontsize=14)
-
-plt.subplot(122)
-plt.plot(X, y, "b.")
-plt.plot(x1, y_pred2, "r.-", linewidth=2, label=r"$\hat{y}$")
-plt.axis([0, 1, -0.2, 1.1])
-plt.xlabel("$x_1$", fontsize=18)
-plt.title("min_samples_leaf={}".format(tree_reg2.min_samples_leaf), fontsize=14)
-
+plot_decision_boundary(deep_tree_clf2, Xm, ym, axes=[-1.5, 2.5, -1, 1.5], iris=False)
+plt.title("min_samples_leaf = {}".format(deep_tree_clf2.min_samples_leaf), fontsize=14)
plt.show()
@@ -302,7 +305,7 @@ plt.show()
-
np.random.seed(6)
+Xs = np.random.rand(100, 2) - 0.5
+ys = (Xs[:, 0] > 0).astype(np.float32) * 2
+angle = np.pi/4
+rotation_matrix = np.array([[np.cos(angle), -np.sin(angle)], [np.sin(angle), np.cos(angle)]])
+Xsr = Xs.dot(rotation_matrix)
+
+tree_clf_s = DecisionTreeClassifier(random_state=42)
+tree_clf_s.fit(Xs, ys)
+tree_clf_sr = DecisionTreeClassifier(random_state=42)
+tree_clf_sr.fit(Xsr, ys)
+
+plt.figure(figsize=(11, 4))
+plt.subplot(121)
+plot_decision_boundary(tree_clf_s, Xs, ys, axes=[-0.7, 0.7, -0.7, 0.7], iris=False)
+plt.subplot(122)
+plot_decision_boundary(tree_clf_sr, Xsr, ys, axes=[-0.7, 0.7, -0.7, 0.7], iris=False)
+
+plt.show()
+
diff --git a/doc/pub/week44/html/._week44-bs033.html b/doc/pub/week44/html/._week44-bs033.html index 835e0560d..51e7a5c1f 100644 --- a/doc/pub/week44/html/._week44-bs033.html +++ b/doc/pub/week44/html/._week44-bs033.html @@ -44,69 +44,75 @@ Automatically generated HTML file from DocOnce source 'sections': [('Overview of week 44', 2, None, '___sec0'), ('Thursday', 2, None, '___sec1'), ('Decision trees, overarching aims', 2, None, '___sec2'), + ('Basics of a tree', 2, None, '___sec3'), + ('A Sketch of a Tree, Regression problem', 2, None, '___sec4'), + ('A Sketch of a Tree, Classification problem', + 2, + None, + '___sec5'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, None, - '___sec3'), - ('General Features', 2, None, '___sec4'), - ('How do we set it up?', 2, None, '___sec5'), - ('Decision trees and Regression', 2, None, '___sec6'), - ('Building a tree, regression', 2, None, '___sec7'), + '___sec6'), + ('General Features', 2, None, '___sec7'), + ('How do we set it up?', 2, None, '___sec8'), + ('Decision trees and Regression', 2, None, '___sec9'), + ('Building a tree, regression', 2, None, '___sec10'), ('A top-down approach, recursive binary splitting', 2, None, - '___sec8'), - ('Making a tree', 2, None, '___sec9'), - ('Pruning the tree', 2, None, '___sec10'), - ('Cost complexity pruning', 2, None, '___sec11'), - ('Schematic Regression Procedure', 2, None, '___sec12'), - ('A Classification Tree', 2, None, '___sec13'), - ('Growing a classification tree', 2, None, '___sec14'), - ('Classification tree, how to split nodes', 2, None, '___sec15'), - ('Visualizing the Tree, Classification', 2, None, '___sec16'), - ('Visualizing the Tree, The Moons', 2, None, '___sec17'), - ('Algorithms for Setting up Decision Trees', 2, None, '___sec18'), - ('The CART algorithm for Classification', 2, None, '___sec19'), - ('The CART algorithm for Regression', 2, None, '___sec20'), - ('Computing the Gini index', 2, None, '___sec21'), + '___sec11'), + ('Making a tree', 2, None, '___sec12'), + ('Pruning the tree', 2, None, '___sec13'), + ('Cost complexity pruning', 2, None, '___sec14'), + ('Schematic Regression Procedure', 2, None, '___sec15'), + ('A Classification Tree', 2, None, '___sec16'), + ('Growing a classification tree', 2, None, '___sec17'), + ('Classification tree, how to split nodes', 2, None, '___sec18'), + ('Visualizing the Tree, Classification', 2, None, '___sec19'), + ('Visualizing the Tree, The Moons', 2, None, '___sec20'), + ('Algorithms for Setting up Decision Trees', 2, None, '___sec21'), + ('The CART algorithm for Classification', 2, None, '___sec22'), + ('The CART algorithm for Regression', 2, None, '___sec23'), + ('Computing the Gini index', 2, None, '___sec24'), ('Simple Python Code to read in Data and perform Classification', 2, None, - '___sec22'), - ('Computing the Gini Factor', 2, None, '___sec23'), - ('Entropy and the ID3 algorithm', 2, None, '___sec24'), - ('Implementing the ID3 Algorithm', 2, None, '___sec25'), + '___sec25'), + ('Computing the Gini Factor', 2, None, '___sec26'), + ('Entropy and the ID3 algorithm', 2, None, '___sec27'), + ('Implementing the ID3 Algorithm', 2, None, '___sec28'), ('Cancer Data again now with Decision Trees and other Methods', 2, None, - '___sec26'), - ('Another example, the moons again', 2, None, '___sec27'), - ('Playing around with regions', 2, None, '___sec28'), - ('Regression trees', 2, None, '___sec29'), - ('Final regressor code', 2, None, '___sec30'), - ('Pros and cons of trees, pros', 2, None, '___sec31'), - ('Disadvantages', 2, None, '___sec32'), + '___sec29'), + ('Another example, the moons again', 2, None, '___sec30'), + ('Playing around with regions', 2, None, '___sec31'), + ('Regression trees', 2, None, '___sec32'), + ('Final regressor code', 2, None, '___sec33'), + ('Pros and cons of trees, pros', 2, None, '___sec34'), + ('Disadvantages', 2, None, '___sec35'), ('Ensemble Methods: From a Single Tree to Many Trees and Extreme ' 'Boosting, Meet the Jungle of Methods', 2, None, - '___sec33'), - ('An Overview of Ensemble Methods', 2, None, '___sec34'), - ('Bagging', 2, None, '___sec35'), - ('More bagging', 2, None, '___sec36'), - ('Simple Voting Example, head or tail', 2, None, '___sec37'), - ('Using the Voting Classifier', 2, None, '___sec38'), + '___sec36'), + ('An Overview of Ensemble Methods', 2, None, '___sec37'), + ('Bagging', 2, None, '___sec38'), + ('More bagging', 2, None, '___sec39'), + ('Simple Voting Example, head or tail', 2, None, '___sec40'), + ('Using the Voting Classifier', 2, None, '___sec41'), ('Please, not the moons again! Voting and Bagging', 2, None, - '___sec39'), - ('Bagging Examples', 2, None, '___sec40'), + '___sec42'), + ('Bagging Examples', 2, None, '___sec43'), ('Making your own Bootstrap: Changing the Level of the Decision ' 'Tree', 2, None, - '___sec41')]} + '___sec44')]} end of tocinfo --> @@ -147,45 +153,48 @@ MathJax.Hub.Config({
-
# Quadratic training set + noise
+np.random.seed(42)
+m = 200
+X = np.random.rand(m, 1)
+y = 4 * (X - 0.5) ** 2
+y = y + np.random.randn(m, 1) / 10
+-However, by aggregating many decision trees, using methods like -bagging, random forests, and boosting, the predictive performance of -trees can be substantially improved. + +
from sklearn.tree import DecisionTreeRegressor
+tree_reg = DecisionTreeRegressor(max_depth=2, random_state=42)
+tree_reg.fit(X, y)
+
@@ -242,6 +254,8 @@ trees can be substantially improved.
-As stated above and seen in many of the examples discussed here about -a single decision tree, we often end up overfitting our training -data. This normally means that we have a high variance. Can we reduce -the variance of a statistical learning method? + +
from sklearn.tree import DecisionTreeRegressor
+
+tree_reg1 = DecisionTreeRegressor(random_state=42, max_depth=2)
+tree_reg2 = DecisionTreeRegressor(random_state=42, max_depth=3)
+tree_reg1.fit(X, y)
+tree_reg2.fit(X, y)
+
+def plot_regression_predictions(tree_reg, X, y, axes=[0, 1, -0.2, 1], ylabel="$y$"):
+ x1 = np.linspace(axes[0], axes[1], 500).reshape(-1, 1)
+ y_pred = tree_reg.predict(x1)
+ plt.axis(axes)
+ plt.xlabel("$x_1$", fontsize=18)
+ if ylabel:
+ plt.ylabel(ylabel, fontsize=18, rotation=0)
+ plt.plot(X, y, "b.")
+ plt.plot(x1, y_pred, "r.-", linewidth=2, label=r"$\hat{y}$")
+
+plt.figure(figsize=(11, 4))
+plt.subplot(121)
+plot_regression_predictions(tree_reg1, X, y)
+for split, style in ((0.1973, "k-"), (0.0917, "k--"), (0.7718, "k--")):
+ plt.plot([split, split], [-0.2, 1], style, linewidth=2)
+plt.text(0.21, 0.65, "Depth=0", fontsize=15)
+plt.text(0.01, 0.2, "Depth=1", fontsize=13)
+plt.text(0.65, 0.8, "Depth=1", fontsize=13)
+plt.legend(loc="upper center", fontsize=18)
+plt.title("max_depth=2", fontsize=14)
+
+plt.subplot(122)
+plot_regression_predictions(tree_reg2, X, y, ylabel=None)
+for split, style in ((0.1973, "k-"), (0.0917, "k--"), (0.7718, "k--")):
+ plt.plot([split, split], [-0.2, 1], style, linewidth=2)
+for split in (0.0458, 0.1298, 0.2873, 0.9040):
+ plt.plot([split, split], [-0.2, 1], "k:", linewidth=1)
+plt.text(0.3, 0.5, "Depth=2", fontsize=13)
+plt.title("max_depth=3", fontsize=14)
+
+plt.show()
+-This leads us to a set of different methods that can combine different -machine learning algorithms or just use one of them to construct -forests and jungles of trees, homogeneous ones or heterogenous -ones. These methods are recognized by different names which we will -try to explain here. These are -
tree_reg1 = DecisionTreeRegressor(random_state=42)
+tree_reg2 = DecisionTreeRegressor(random_state=42, min_samples_leaf=10)
+tree_reg1.fit(X, y)
+tree_reg2.fit(X, y)
-We discuss these methods here.
+x1 = np.linspace(0, 1, 500).reshape(-1, 1)
+y_pred1 = tree_reg1.predict(x1)
+y_pred2 = tree_reg2.predict(x1)
+plt.figure(figsize=(11, 4))
+
+plt.subplot(121)
+plt.plot(X, y, "b.")
+plt.plot(x1, y_pred1, "r.-", linewidth=2, label=r"$\hat{y}$")
+plt.axis([0, 1, -0.2, 1.1])
+plt.xlabel("$x_1$", fontsize=18)
+plt.ylabel("$y$", fontsize=18, rotation=0)
+plt.legend(loc="upper center", fontsize=18)
+plt.title("No restrictions", fontsize=14)
+
+plt.subplot(122)
+plt.plot(X, y, "b.")
+plt.plot(x1, y_pred2, "r.-", linewidth=2, label=r"$\hat{y}$")
+plt.axis([0, 1, -0.2, 1.1])
+plt.xlabel("$x_1$", fontsize=18)
+plt.title("min_samples_leaf={}".format(tree_reg2.min_samples_leaf), fontsize=14)
+
+plt.show()
+
@@ -249,6 +309,9 @@ We discuss these methods here.
-
diff --git a/doc/pub/week44/html/._week44-bs036.html b/doc/pub/week44/html/._week44-bs036.html index 9f955c8b4..76329009e 100644 --- a/doc/pub/week44/html/._week44-bs036.html +++ b/doc/pub/week44/html/._week44-bs036.html @@ -44,69 +44,75 @@ Automatically generated HTML file from DocOnce source 'sections': [('Overview of week 44', 2, None, '___sec0'), ('Thursday', 2, None, '___sec1'), ('Decision trees, overarching aims', 2, None, '___sec2'), + ('Basics of a tree', 2, None, '___sec3'), + ('A Sketch of a Tree, Regression problem', 2, None, '___sec4'), + ('A Sketch of a Tree, Classification problem', + 2, + None, + '___sec5'), ('A typical Decision Tree with its pertinent Jargon, ' 'Classification Problem', 2, None, - '___sec3'), - ('General Features', 2, None, '___sec4'), - ('How do we set it up?', 2, None, '___sec5'), - ('Decision trees and Regression', 2, None, '___sec6'), - ('Building a tree, regression', 2, None, '___sec7'), + '___sec6'), + ('General Features', 2, None, '___sec7'), + ('How do we set it up?', 2, None, '___sec8'), + ('Decision trees and Regression', 2, None, '___sec9'), + ('Building a tree, regression', 2, None, '___sec10'), ('A top-down approach, recursive binary splitting', 2, None, - '___sec8'), - ('Making a tree', 2, None, '___sec9'), - ('Pruning the tree', 2, None, '___sec10'), - ('Cost complexity pruning', 2, None, '___sec11'), - ('Schematic Regression Procedure', 2, None, '___sec12'), - ('A Classification Tree', 2, None, '___sec13'), - ('Growing a classification tree', 2, None, '___sec14'), - ('Classification tree, how to split nodes', 2, None, '___sec15'), - ('Visualizing the Tree, Classification', 2, None, '___sec16'), - ('Visualizing the Tree, The Moons', 2, None, '___sec17'), - ('Algorithms for Setting up Decision Trees', 2, None, '___sec18'), - ('The CART algorithm for Classification', 2, None, '___sec19'), - ('The CART algorithm for Regression', 2, None, '___sec20'), - ('Computing the Gini index', 2, None, '___sec21'), + '___sec11'), + ('Making a tree', 2, None, '___sec12'), + ('Pruning the tree', 2, None, '___sec13'), + ('Cost complexity pruning', 2, None, '___sec14'), + ('Schematic Regression Procedure', 2, None, '___sec15'), + ('A Classification Tree', 2, None, '___sec16'), + ('Growing a classification tree', 2, None, '___sec17'), + ('Classification tree, how to split nodes', 2, None, '___sec18'), + ('Visualizing the Tree, Classification', 2, None, '___sec19'), + ('Visualizing the Tree, The Moons', 2, None, '___sec20'), + ('Algorithms for Setting up Decision Trees', 2, None, '___sec21'), + ('The CART algorithm for Classification', 2, None, '___sec22'), + ('The CART algorithm for Regression', 2, None, '___sec23'), + ('Computing the Gini index', 2, None, '___sec24'), ('Simple Python Code to read in Data and perform Classification', 2, None, - '___sec22'), - ('Computing the Gini Factor', 2, None, '___sec23'), - ('Entropy and the ID3 algorithm', 2, None, '___sec24'), - ('Implementing the ID3 Algorithm', 2, None, '___sec25'), + '___sec25'), + ('Computing the Gini Factor', 2, None, '___sec26'), + ('Entropy and the ID3 algorithm', 2, None, '___sec27'), + ('Implementing the ID3 Algorithm', 2, None, '___sec28'), ('Cancer Data again now with Decision Trees and other Methods', 2, None, - '___sec26'), - ('Another example, the moons again', 2, None, '___sec27'), - ('Playing around with regions', 2, None, '___sec28'), - ('Regression trees', 2, None, '___sec29'), - ('Final regressor code', 2, None, '___sec30'), - ('Pros and cons of trees, pros', 2, None, '___sec31'), - ('Disadvantages', 2, None, '___sec32'), + '___sec29'), + ('Another example, the moons again', 2, None, '___sec30'), + ('Playing around with regions', 2, None, '___sec31'), + ('Regression trees', 2, None, '___sec32'), + ('Final regressor code', 2, None, '___sec33'), + ('Pros and cons of trees, pros', 2, None, '___sec34'), + ('Disadvantages', 2, None, '___sec35'), ('Ensemble Methods: From a Single Tree to Many Trees and Extreme ' 'Boosting, Meet the Jungle of Methods', 2, None, - '___sec33'), - ('An Overview of Ensemble Methods', 2, None, '___sec34'), - ('Bagging', 2, None, '___sec35'), - ('More bagging', 2, None, '___sec36'), - ('Simple Voting Example, head or tail', 2, None, '___sec37'), - ('Using the Voting Classifier', 2, None, '___sec38'), + '___sec36'), + ('An Overview of Ensemble Methods', 2, None, '___sec37'), + ('Bagging', 2, None, '___sec38'), + ('More bagging', 2, None, '___sec39'), + ('Simple Voting Example, head or tail', 2, None, '___sec40'), + ('Using the Voting Classifier', 2, None, '___sec41'), ('Please, not the moons again! Voting and Bagging', 2, None, - '___sec39'), - ('Bagging Examples', 2, None, '___sec40'), + '___sec42'), + ('Bagging Examples', 2, None, '___sec43'), ('Making your own Bootstrap: Changing the Level of the Decision ' 'Tree', 2, None, - '___sec41')]} + '___sec44')]} end of tocinfo --> @@ -147,45 +153,48 @@ MathJax.Hub.Config({
-The plain decision trees suffer from high -variance. This means that if we split the training data into two parts -at random, and fit a decision tree to both halves, the results that we -get could be quite different. In contrast, a procedure with low -variance will yield similar results if applied repeatedly to distinct -data sets; linear regression tends to have low variance, if the ratio -of \( n \) to \( p \) is moderately large. +
-Bootstrap aggregation, or just bagging, is a -general-purpose procedure for reducing the variance of a statistical -learning method. +However, by aggregating many decision trees, using methods like +bagging, random forests, and boosting, the predictive performance of +trees can be substantially improved.
@@ -239,6 +248,9 @@ learning method.
-Bagging typically results in improved accuracy -over prediction using a single tree. Unfortunately, however, it can be -difficult to interpret the resulting model. Recall that one of the -advantages of decision trees is the attractive and easily interpreted -diagram that results. +As stated above and seen in many of the examples discussed here about +a single decision tree, we often end up overfitting our training +data. This normally means that we have a high variance. Can we reduce +the variance of a statistical learning method?
-However, when we bag a large number of trees, it is no longer -possible to represent the resulting statistical learning procedure -using a single tree, and it is no longer clear which variables are -most important to the procedure. Thus, bagging improves prediction -accuracy at the expense of interpretability. Although the collection -of bagged trees is much more difficult to interpret than a single -tree, one can obtain an overall summary of the importance of each -predictor using the MSE (for bagging regression trees) or the Gini -index (for bagging classification trees). In the case of bagging -regression trees, we can record the total amount that the MSE is -decreased due to splits over a given predictor, averaged over all \( B \) possible -trees. A large value indicates an important predictor. Similarly, in -the context of bagging classification trees, we can add up the total -amount that the Gini index is decreased by splits over a given -predictor, averaged over all \( B \) trees. +This leads us to a set of different methods that can combine different +machine learning algorithms or just use one of them to construct +forests and jungles of trees, homogeneous ones or heterogenous +ones. These methods are recognized by different names which we will +try to explain here. These are + +
@@ -248,6 +255,9 @@ predictor, averaged over all \( B \) trees.
+
+
heads_proba = 0.51
-coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
-cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)
-plt.figure(figsize=(8,3.5))
-plt.plot(cumulative_heads_ratio)
-plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
-plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
-plt.xlabel("Number of coin tosses")
-plt.ylabel("Heads ratio")
-plt.legend(loc="lower right")
-plt.axis([0, 10000, 0.42, 0.58])
-save_fig("votingsimple")
-plt.show()
-
@@ -239,6 +235,9 @@ plt.show()
+The plain decision trees suffer from high +variance. This means that if we split the training data into two parts +at random, and fit a decision tree to both halves, the results that we +get could be quite different. In contrast, a procedure with low +variance will yield similar results if applied repeatedly to distinct +data sets; linear regression tends to have low variance, if the ratio +of \( n \) to \( p \) is moderately large. - -
from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
+
+Bootstrap aggregation, or just bagging, is a
+general-purpose procedure for reducing the variance of a statistical
+learning method.
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
@@ -268,6 +245,9 @@ voting_clf.fit(X_train, y_train)
+Bagging typically results in improved accuracy +over prediction using a single tree. Unfortunately, however, it can be +difficult to interpret the resulting model. Recall that one of the +advantages of decision trees is the attractive and easily interpreted +diagram that results. - -
from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
-
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-voting_clf.fit(X_train, y_train)
-+However, when we bag a large number of trees, it is no longer +possible to represent the resulting statistical learning procedure +using a single tree, and it is no longer clear which variables are +most important to the procedure. Thus, bagging improves prediction +accuracy at the expense of interpretability. Although the collection +of bagged trees is much more difficult to interpret than a single +tree, one can obtain an overall summary of the importance of each +predictor using the MSE (for bagging regression trees) or the Gini +index (for bagging classification trees). In the case of bagging +regression trees, we can record the total amount that the MSE is +decreased due to splits over a given predictor, averaged over all \( B \) possible +trees. A large value indicates an important predictor. Similarly, in +the context of bagging classification trees, we can add up the total +amount that the Gini index is decreased by splits over a given +predictor, averaged over all \( B \) trees. - -
from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-- - -
log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-- - -
from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
@@ -275,6 +254,9 @@ voting_clf.fit(X_train, y_train)
-
from sklearn.ensemble import BaggingClassifier
-from sklearn.tree import DecisionTreeClassifier
-
-bag_clf = BaggingClassifier(
- DecisionTreeClassifier(random_state=42), n_estimators=500,
- max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
-bag_clf.fit(X_train, y_train)
-y_pred = bag_clf.predict(X_test)
-- - -
from sklearn.metrics import accuracy_score
-print(accuracy_score(y_test, y_pred))
-- - -
tree_clf = DecisionTreeClassifier(random_state=42)
-tree_clf.fit(X_train, y_train)
-y_pred_tree = tree_clf.predict(X_test)
-print(accuracy_score(y_test, y_pred_tree))
-- - -
from matplotlib.colors import ListedColormap
-
-def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
- x1s = np.linspace(axes[0], axes[1], 100)
- x2s = np.linspace(axes[2], axes[3], 100)
- x1, x2 = np.meshgrid(x1s, x2s)
- X_new = np.c_[x1.ravel(), x2.ravel()]
- y_pred = clf.predict(X_new).reshape(x1.shape)
- custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
- plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
- if contour:
- custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
- plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
- plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
- plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
- plt.axis(axes)
- plt.xlabel(r"$x_1$", fontsize=18)
- plt.ylabel(r"$x_2$", fontsize=18, rotation=0)
-plt.figure(figsize=(11,4))
-plt.subplot(121)
-plot_decision_boundary(tree_clf, X, y)
-plt.title("Decision Tree", fontsize=14)
-plt.subplot(122)
-plot_decision_boundary(bag_clf, X, y)
-plt.title("Decision Trees with Bagging", fontsize=14)
-save_fig("baggingtree")
+heads_proba = 0.51
+coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
+cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)
+plt.figure(figsize=(8,3.5))
+plt.plot(cumulative_heads_ratio)
+plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
+plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
+plt.xlabel("Number of coin tosses")
+plt.ylabel("Heads ratio")
+plt.legend(loc="lower right")
+plt.axis([0, 10000, 0.42, 0.58])
+save_fig("votingsimple")
plt.show()
@@ -277,6 +245,9 @@ plt.show()
-Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with -a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points \( n \)). +
-
import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.pipeline import make_pipeline
-from sklearn.utils import resample
-from sklearn.tree import DecisionTreeRegressor
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import make_moons
-n = 100
-n_boostraps = 100
-maxdepth = 8
+X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
+X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-error = np.zeros(maxdepth)
-bias = np.zeros(maxdepth)
-variance = np.zeros(maxdepth)
-polydegree = np.zeros(maxdepth)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.ensemble import VotingClassifier
+from sklearn.linear_model import LogisticRegression
+from sklearn.svm import SVC
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
+log_clf = LogisticRegression(solver="liblinear", random_state=42)
+rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
+svm_clf = SVC(gamma="auto", random_state=42)
-# we produce a simple tree first as benchmark
-simpletree = DecisionTreeRegressor(max_depth=3)
-simpletree.fit(X_train_scaled, y_train)
-simpleprediction = simpletree.predict(X_test_scaled)
-for degree in range(1,maxdepth):
- model = DecisionTreeRegressor(max_depth=degree)
- y_pred = np.empty((y_test.shape[0], n_boostraps))
- for i in range(n_boostraps):
- x_, y_ = resample(X_train_scaled, y_train)
- model.fit(x_, y_)
- y_pred[:, i] = model.predict(X_test_scaled)#.ravel()
+voting_clf = VotingClassifier(
+ estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
+ voting='hard')
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
- variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
- print('Polynomial degree:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+voting_clf.fit(X_train, y_train)
-mse_simpletree = np.mean( np.mean((y_test - simpleprediction)**2)
-plt.xlim(1,maxdepth)
-plt.plot(polydegree, error, label='MSE simple tree')
-plt.plot(polydegree, mse_simpletree, label='MSE for Bootstrap')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
-save_fig("baggingboot")
-plt.show()
+from sklearn.metrics import accuracy_score
+
+for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
+ clf.fit(X_train, y_train)
+ y_pred = clf.predict(X_test)
+ print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
+
+log_clf = LogisticRegression(solver="liblinear", random_state=42)
+rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
+svm_clf = SVC(gamma="auto", probability=True, random_state=42)
+
+voting_clf = VotingClassifier(
+ estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
+ voting='soft')
+voting_clf.fit(X_train, y_train)
+
+from sklearn.metrics import accuracy_score
+
+for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
+ clf.fit(X_train, y_train)
+ y_pred = clf.predict(X_test)
+ print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
diff --git a/doc/pub/week44/html/week44-bs.html b/doc/pub/week44/html/week44-bs.html
index c9eceb6e3..1d7a41046 100644
--- a/doc/pub/week44/html/week44-bs.html
+++ b/doc/pub/week44/html/week44-bs.html
@@ -44,69 +44,75 @@ Automatically generated HTML file from DocOnce source
'sections': [('Overview of week 44', 2, None, '___sec0'),
('Thursday', 2, None, '___sec1'),
('Decision trees, overarching aims', 2, None, '___sec2'),
+ ('Basics of a tree', 2, None, '___sec3'),
+ ('A Sketch of a Tree, Regression problem', 2, None, '___sec4'),
+ ('A Sketch of a Tree, Classification problem',
+ 2,
+ None,
+ '___sec5'),
('A typical Decision Tree with its pertinent Jargon, '
'Classification Problem',
2,
None,
- '___sec3'),
- ('General Features', 2, None, '___sec4'),
- ('How do we set it up?', 2, None, '___sec5'),
- ('Decision trees and Regression', 2, None, '___sec6'),
- ('Building a tree, regression', 2, None, '___sec7'),
+ '___sec6'),
+ ('General Features', 2, None, '___sec7'),
+ ('How do we set it up?', 2, None, '___sec8'),
+ ('Decision trees and Regression', 2, None, '___sec9'),
+ ('Building a tree, regression', 2, None, '___sec10'),
('A top-down approach, recursive binary splitting',
2,
None,
- '___sec8'),
- ('Making a tree', 2, None, '___sec9'),
- ('Pruning the tree', 2, None, '___sec10'),
- ('Cost complexity pruning', 2, None, '___sec11'),
- ('Schematic Regression Procedure', 2, None, '___sec12'),
- ('A Classification Tree', 2, None, '___sec13'),
- ('Growing a classification tree', 2, None, '___sec14'),
- ('Classification tree, how to split nodes', 2, None, '___sec15'),
- ('Visualizing the Tree, Classification', 2, None, '___sec16'),
- ('Visualizing the Tree, The Moons', 2, None, '___sec17'),
- ('Algorithms for Setting up Decision Trees', 2, None, '___sec18'),
- ('The CART algorithm for Classification', 2, None, '___sec19'),
- ('The CART algorithm for Regression', 2, None, '___sec20'),
- ('Computing the Gini index', 2, None, '___sec21'),
+ '___sec11'),
+ ('Making a tree', 2, None, '___sec12'),
+ ('Pruning the tree', 2, None, '___sec13'),
+ ('Cost complexity pruning', 2, None, '___sec14'),
+ ('Schematic Regression Procedure', 2, None, '___sec15'),
+ ('A Classification Tree', 2, None, '___sec16'),
+ ('Growing a classification tree', 2, None, '___sec17'),
+ ('Classification tree, how to split nodes', 2, None, '___sec18'),
+ ('Visualizing the Tree, Classification', 2, None, '___sec19'),
+ ('Visualizing the Tree, The Moons', 2, None, '___sec20'),
+ ('Algorithms for Setting up Decision Trees', 2, None, '___sec21'),
+ ('The CART algorithm for Classification', 2, None, '___sec22'),
+ ('The CART algorithm for Regression', 2, None, '___sec23'),
+ ('Computing the Gini index', 2, None, '___sec24'),
('Simple Python Code to read in Data and perform Classification',
2,
None,
- '___sec22'),
- ('Computing the Gini Factor', 2, None, '___sec23'),
- ('Entropy and the ID3 algorithm', 2, None, '___sec24'),
- ('Implementing the ID3 Algorithm', 2, None, '___sec25'),
+ '___sec25'),
+ ('Computing the Gini Factor', 2, None, '___sec26'),
+ ('Entropy and the ID3 algorithm', 2, None, '___sec27'),
+ ('Implementing the ID3 Algorithm', 2, None, '___sec28'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
- '___sec26'),
- ('Another example, the moons again', 2, None, '___sec27'),
- ('Playing around with regions', 2, None, '___sec28'),
- ('Regression trees', 2, None, '___sec29'),
- ('Final regressor code', 2, None, '___sec30'),
- ('Pros and cons of trees, pros', 2, None, '___sec31'),
- ('Disadvantages', 2, None, '___sec32'),
+ '___sec29'),
+ ('Another example, the moons again', 2, None, '___sec30'),
+ ('Playing around with regions', 2, None, '___sec31'),
+ ('Regression trees', 2, None, '___sec32'),
+ ('Final regressor code', 2, None, '___sec33'),
+ ('Pros and cons of trees, pros', 2, None, '___sec34'),
+ ('Disadvantages', 2, None, '___sec35'),
('Ensemble Methods: From a Single Tree to Many Trees and Extreme '
'Boosting, Meet the Jungle of Methods',
2,
None,
- '___sec33'),
- ('An Overview of Ensemble Methods', 2, None, '___sec34'),
- ('Bagging', 2, None, '___sec35'),
- ('More bagging', 2, None, '___sec36'),
- ('Simple Voting Example, head or tail', 2, None, '___sec37'),
- ('Using the Voting Classifier', 2, None, '___sec38'),
+ '___sec36'),
+ ('An Overview of Ensemble Methods', 2, None, '___sec37'),
+ ('Bagging', 2, None, '___sec38'),
+ ('More bagging', 2, None, '___sec39'),
+ ('Simple Voting Example, head or tail', 2, None, '___sec40'),
+ ('Using the Voting Classifier', 2, None, '___sec41'),
('Please, not the moons again! Voting and Bagging',
2,
None,
- '___sec39'),
- ('Bagging Examples', 2, None, '___sec40'),
+ '___sec42'),
+ ('Bagging Examples', 2, None, '___sec43'),
('Making your own Bootstrap: Changing the Level of the Decision '
'Tree',
2,
None,
- '___sec41')]}
+ '___sec44')]}
end of tocinfo -->
@@ -147,45 +153,48 @@ MathJax.Hub.Config({
-
@@ -244,7 +253,7 @@ MathJax.Hub.Config({
-
@@ -206,6 +206,11 @@ The descriptive features which reproduce best the target/output features are nor
to be the most informative ones. The process of finding the most
informative feature is done until we accomplish a stopping criteria
where we then finally end up in so called leaf nodes.
+
+
+
+
A decision tree is typically divided into a root node, the interior nodes,
@@ -222,7 +227,23 @@ given some assumptions, make predictions about the target feature value
+
+
+
+
The overarching approach to decision trees is a top-down approach.
@@ -252,7 +273,7 @@ node.
In simplified terms, the process of training a decision tree and
@@ -271,7 +292,7 @@ Then we are essentially done!
@@ -368,7 +389,7 @@ plt.show()
There are mainly two steps
@@ -400,7 +421,7 @@ within box \( j \).
Unfortunately, it is computationally infeasible to consider every
@@ -419,7 +440,7 @@ better tree in some future step.
In order to implement the recursive binary splitting we start by selecting
@@ -476,7 +497,7 @@ region contains more than five observations.
The above procedure is rather straightforward, but leads often to
@@ -495,7 +516,7 @@ parameter \( \alpha \).
A classification tree is very similar to a regression tree, except
@@ -572,7 +593,7 @@ fall into that region.
The task of growing a
@@ -596,7 +617,7 @@ than is the classification error rate.
If our targets are the outcome of a classification process that takes
@@ -651,7 +672,7 @@ $$
@@ -693,7 +714,7 @@ os.system(cmd)
@@ -726,7 +747,7 @@ os.system(cmd)
Two algorithms stand out in the set up of decision trees:
@@ -745,7 +766,7 @@ in two branches.
For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \).
@@ -774,7 +795,7 @@ hyperparameters control additional stopping conditions such as the \( min\_sampl
The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the
@@ -808,7 +829,7 @@ just like for classification tasks, is prone to overfitting.
The example we will look at is a classical one in many Machine
@@ -849,7 +870,7 @@ The table here summarizes the various attributes and
@@ -926,7 +947,7 @@ os.system(cmd)
The above functions (gini, entropy and misclassification error) are
@@ -1005,7 +1026,7 @@ split = get_split(dataset)
ID3, learns decision trees by constructing
@@ -1042,7 +1063,7 @@ attributes at each step while growing the tree.
@@ -1239,7 +1260,7 @@ attributes at each step while growing the tree.
@@ -1289,7 +1310,7 @@ deep_tree_clf.fit(X_train_scaled, y_train)
@@ -1362,7 +1383,7 @@ plt.show()
@@ -1391,7 +1412,7 @@ plt.show()
@@ -1414,7 +1435,7 @@ tree_reg.fit(X, y)
@@ -1493,7 +1514,7 @@ plt.show()
As stated above and seen in many of the examples discussed here about
@@ -1556,7 +1577,7 @@ We discuss these methods here.
The plain decision trees suffer from high
@@ -1583,7 +1604,7 @@ learning method.
Bagging typically results in improved accuracy
@@ -1612,7 +1633,7 @@ predictor, averaged over all \( B \) trees.
@@ -1634,7 +1655,7 @@ plt.show()
@@ -1686,7 +1707,7 @@ voting_clf.fit(X_train, y_train)
@@ -1746,7 +1767,7 @@ voting_clf.fit(X_train, y_train)
@@ -1809,7 +1830,7 @@ plt.show()
Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with
diff --git a/doc/pub/week44/html/week44-solarized.html b/doc/pub/week44/html/week44-solarized.html
index 84f75cd8f..a41a9488a 100644
--- a/doc/pub/week44/html/week44-solarized.html
+++ b/doc/pub/week44/html/week44-solarized.html
@@ -64,69 +64,75 @@ div { text-align: justify; text-justify: inter-word; }
'sections': [('Overview of week 44', 2, None, '___sec0'),
('Thursday', 2, None, '___sec1'),
('Decision trees, overarching aims', 2, None, '___sec2'),
+ ('Basics of a tree', 2, None, '___sec3'),
+ ('A Sketch of a Tree, Regression problem', 2, None, '___sec4'),
+ ('A Sketch of a Tree, Classification problem',
+ 2,
+ None,
+ '___sec5'),
('A typical Decision Tree with its pertinent Jargon, '
'Classification Problem',
2,
None,
- '___sec3'),
- ('General Features', 2, None, '___sec4'),
- ('How do we set it up?', 2, None, '___sec5'),
- ('Decision trees and Regression', 2, None, '___sec6'),
- ('Building a tree, regression', 2, None, '___sec7'),
+ '___sec6'),
+ ('General Features', 2, None, '___sec7'),
+ ('How do we set it up?', 2, None, '___sec8'),
+ ('Decision trees and Regression', 2, None, '___sec9'),
+ ('Building a tree, regression', 2, None, '___sec10'),
('A top-down approach, recursive binary splitting',
2,
None,
- '___sec8'),
- ('Making a tree', 2, None, '___sec9'),
- ('Pruning the tree', 2, None, '___sec10'),
- ('Cost complexity pruning', 2, None, '___sec11'),
- ('Schematic Regression Procedure', 2, None, '___sec12'),
- ('A Classification Tree', 2, None, '___sec13'),
- ('Growing a classification tree', 2, None, '___sec14'),
- ('Classification tree, how to split nodes', 2, None, '___sec15'),
- ('Visualizing the Tree, Classification', 2, None, '___sec16'),
- ('Visualizing the Tree, The Moons', 2, None, '___sec17'),
- ('Algorithms for Setting up Decision Trees', 2, None, '___sec18'),
- ('The CART algorithm for Classification', 2, None, '___sec19'),
- ('The CART algorithm for Regression', 2, None, '___sec20'),
- ('Computing the Gini index', 2, None, '___sec21'),
+ '___sec11'),
+ ('Making a tree', 2, None, '___sec12'),
+ ('Pruning the tree', 2, None, '___sec13'),
+ ('Cost complexity pruning', 2, None, '___sec14'),
+ ('Schematic Regression Procedure', 2, None, '___sec15'),
+ ('A Classification Tree', 2, None, '___sec16'),
+ ('Growing a classification tree', 2, None, '___sec17'),
+ ('Classification tree, how to split nodes', 2, None, '___sec18'),
+ ('Visualizing the Tree, Classification', 2, None, '___sec19'),
+ ('Visualizing the Tree, The Moons', 2, None, '___sec20'),
+ ('Algorithms for Setting up Decision Trees', 2, None, '___sec21'),
+ ('The CART algorithm for Classification', 2, None, '___sec22'),
+ ('The CART algorithm for Regression', 2, None, '___sec23'),
+ ('Computing the Gini index', 2, None, '___sec24'),
('Simple Python Code to read in Data and perform Classification',
2,
None,
- '___sec22'),
- ('Computing the Gini Factor', 2, None, '___sec23'),
- ('Entropy and the ID3 algorithm', 2, None, '___sec24'),
- ('Implementing the ID3 Algorithm', 2, None, '___sec25'),
+ '___sec25'),
+ ('Computing the Gini Factor', 2, None, '___sec26'),
+ ('Entropy and the ID3 algorithm', 2, None, '___sec27'),
+ ('Implementing the ID3 Algorithm', 2, None, '___sec28'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
- '___sec26'),
- ('Another example, the moons again', 2, None, '___sec27'),
- ('Playing around with regions', 2, None, '___sec28'),
- ('Regression trees', 2, None, '___sec29'),
- ('Final regressor code', 2, None, '___sec30'),
- ('Pros and cons of trees, pros', 2, None, '___sec31'),
- ('Disadvantages', 2, None, '___sec32'),
+ '___sec29'),
+ ('Another example, the moons again', 2, None, '___sec30'),
+ ('Playing around with regions', 2, None, '___sec31'),
+ ('Regression trees', 2, None, '___sec32'),
+ ('Final regressor code', 2, None, '___sec33'),
+ ('Pros and cons of trees, pros', 2, None, '___sec34'),
+ ('Disadvantages', 2, None, '___sec35'),
('Ensemble Methods: From a Single Tree to Many Trees and Extreme '
'Boosting, Meet the Jungle of Methods',
2,
None,
- '___sec33'),
- ('An Overview of Ensemble Methods', 2, None, '___sec34'),
- ('Bagging', 2, None, '___sec35'),
- ('More bagging', 2, None, '___sec36'),
- ('Simple Voting Example, head or tail', 2, None, '___sec37'),
- ('Using the Voting Classifier', 2, None, '___sec38'),
+ '___sec36'),
+ ('An Overview of Ensemble Methods', 2, None, '___sec37'),
+ ('Bagging', 2, None, '___sec38'),
+ ('More bagging', 2, None, '___sec39'),
+ ('Simple Voting Example, head or tail', 2, None, '___sec40'),
+ ('Using the Voting Classifier', 2, None, '___sec41'),
('Please, not the moons again! Voting and Bagging',
2,
None,
- '___sec39'),
- ('Bagging Examples', 2, None, '___sec40'),
+ '___sec42'),
+ ('Bagging Examples', 2, None, '___sec43'),
('Making your own Bootstrap: Changing the Level of the Decision '
'Tree',
2,
None,
- '___sec41')]}
+ '___sec44')]}
end of tocinfo -->
-
+
A decision tree is typically divided into a root node, the interior nodes,
and the final leaf nodes or just leaves. These entities are then connected by so-called branches.
@@ -235,7 +246,23 @@ given some assumptions, make predictions about the target feature value
+
+
+
+
+
+
+
+
The overarching approach to decision trees is a top-down approach.
@@ -264,7 +291,7 @@ node.
In simplified terms, the process of training a decision tree and
@@ -282,7 +309,7 @@ Then we are essentially done!
@@ -378,7 +405,7 @@ plt.show()
There are mainly two steps
@@ -406,7 +433,7 @@ within box \( j \).
Unfortunately, it is computationally infeasible to consider every
@@ -425,7 +452,7 @@ better tree in some future step.
In order to implement the recursive binary splitting we start by selecting
@@ -476,7 +503,7 @@ region contains more than five observations.
-
The above procedure is rather straightforward, but leads often to
@@ -495,7 +522,7 @@ parameter \( \alpha \).
A classification tree is very similar to a regression tree, except
@@ -571,7 +598,7 @@ fall into that region.
The task of growing a
@@ -595,7 +622,7 @@ than is the classification error rate.
If our targets are the outcome of a classification process that takes
@@ -645,7 +672,7 @@ $$
@@ -686,7 +713,7 @@ os.system(cmd)
@@ -718,7 +745,7 @@ os.system(cmd)
Two algorithms stand out in the set up of decision trees:
@@ -736,7 +763,7 @@ in two branches.
For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \).
@@ -763,7 +790,7 @@ hyperparameters control additional stopping conditions such as the \( min\_sampl
The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the
@@ -791,7 +818,7 @@ just like for classification tasks, is prone to overfitting.
The example we will look at is a classical one in many Machine
@@ -831,7 +858,7 @@ The table here summarizes the various attributes and
@@ -907,7 +934,7 @@ os.system(cmd)
The above functions (gini, entropy and misclassification error) are
@@ -985,7 +1012,7 @@ split = get_split(dataset)
ID3, learns decision trees by constructing
@@ -1021,7 +1048,7 @@ attributes at each step while growing the tree.
@@ -1217,7 +1244,7 @@ attributes at each step while growing the tree.
@@ -1266,7 +1293,7 @@ deep_tree_clf.fit(X_train_scaled, y_train)
@@ -1338,7 +1365,7 @@ plt.show()
@@ -1366,7 +1393,7 @@ plt.show()
@@ -1388,7 +1415,7 @@ tree_reg.fit(X, y)
@@ -1466,7 +1493,7 @@ plt.show()
As stated above and seen in many of the examples discussed here about
@@ -1526,7 +1553,7 @@ We discuss these methods here.
The plain decision trees suffer from high
@@ -1553,7 +1580,7 @@ learning method.
Bagging typically results in improved accuracy
@@ -1582,7 +1609,7 @@ predictor, averaged over all \( B \) trees.
@@ -1603,7 +1630,7 @@ plt.show()
@@ -1654,7 +1681,7 @@ voting_clf.fit(X_train, y_train)
@@ -1713,7 +1740,7 @@ voting_clf.fit(X_train, y_train)
@@ -1775,7 +1802,7 @@ plt.show()
Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with
diff --git a/doc/pub/week44/html/week44.html b/doc/pub/week44/html/week44.html
index df25034c3..1c88c5c6d 100644
--- a/doc/pub/week44/html/week44.html
+++ b/doc/pub/week44/html/week44.html
@@ -69,69 +69,75 @@ div { text-align: justify; text-justify: inter-word; }
'sections': [('Overview of week 44', 2, None, '___sec0'),
('Thursday', 2, None, '___sec1'),
('Decision trees, overarching aims', 2, None, '___sec2'),
+ ('Basics of a tree', 2, None, '___sec3'),
+ ('A Sketch of a Tree, Regression problem', 2, None, '___sec4'),
+ ('A Sketch of a Tree, Classification problem',
+ 2,
+ None,
+ '___sec5'),
('A typical Decision Tree with its pertinent Jargon, '
'Classification Problem',
2,
None,
- '___sec3'),
- ('General Features', 2, None, '___sec4'),
- ('How do we set it up?', 2, None, '___sec5'),
- ('Decision trees and Regression', 2, None, '___sec6'),
- ('Building a tree, regression', 2, None, '___sec7'),
+ '___sec6'),
+ ('General Features', 2, None, '___sec7'),
+ ('How do we set it up?', 2, None, '___sec8'),
+ ('Decision trees and Regression', 2, None, '___sec9'),
+ ('Building a tree, regression', 2, None, '___sec10'),
('A top-down approach, recursive binary splitting',
2,
None,
- '___sec8'),
- ('Making a tree', 2, None, '___sec9'),
- ('Pruning the tree', 2, None, '___sec10'),
- ('Cost complexity pruning', 2, None, '___sec11'),
- ('Schematic Regression Procedure', 2, None, '___sec12'),
- ('A Classification Tree', 2, None, '___sec13'),
- ('Growing a classification tree', 2, None, '___sec14'),
- ('Classification tree, how to split nodes', 2, None, '___sec15'),
- ('Visualizing the Tree, Classification', 2, None, '___sec16'),
- ('Visualizing the Tree, The Moons', 2, None, '___sec17'),
- ('Algorithms for Setting up Decision Trees', 2, None, '___sec18'),
- ('The CART algorithm for Classification', 2, None, '___sec19'),
- ('The CART algorithm for Regression', 2, None, '___sec20'),
- ('Computing the Gini index', 2, None, '___sec21'),
+ '___sec11'),
+ ('Making a tree', 2, None, '___sec12'),
+ ('Pruning the tree', 2, None, '___sec13'),
+ ('Cost complexity pruning', 2, None, '___sec14'),
+ ('Schematic Regression Procedure', 2, None, '___sec15'),
+ ('A Classification Tree', 2, None, '___sec16'),
+ ('Growing a classification tree', 2, None, '___sec17'),
+ ('Classification tree, how to split nodes', 2, None, '___sec18'),
+ ('Visualizing the Tree, Classification', 2, None, '___sec19'),
+ ('Visualizing the Tree, The Moons', 2, None, '___sec20'),
+ ('Algorithms for Setting up Decision Trees', 2, None, '___sec21'),
+ ('The CART algorithm for Classification', 2, None, '___sec22'),
+ ('The CART algorithm for Regression', 2, None, '___sec23'),
+ ('Computing the Gini index', 2, None, '___sec24'),
('Simple Python Code to read in Data and perform Classification',
2,
None,
- '___sec22'),
- ('Computing the Gini Factor', 2, None, '___sec23'),
- ('Entropy and the ID3 algorithm', 2, None, '___sec24'),
- ('Implementing the ID3 Algorithm', 2, None, '___sec25'),
+ '___sec25'),
+ ('Computing the Gini Factor', 2, None, '___sec26'),
+ ('Entropy and the ID3 algorithm', 2, None, '___sec27'),
+ ('Implementing the ID3 Algorithm', 2, None, '___sec28'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
- '___sec26'),
- ('Another example, the moons again', 2, None, '___sec27'),
- ('Playing around with regions', 2, None, '___sec28'),
- ('Regression trees', 2, None, '___sec29'),
- ('Final regressor code', 2, None, '___sec30'),
- ('Pros and cons of trees, pros', 2, None, '___sec31'),
- ('Disadvantages', 2, None, '___sec32'),
+ '___sec29'),
+ ('Another example, the moons again', 2, None, '___sec30'),
+ ('Playing around with regions', 2, None, '___sec31'),
+ ('Regression trees', 2, None, '___sec32'),
+ ('Final regressor code', 2, None, '___sec33'),
+ ('Pros and cons of trees, pros', 2, None, '___sec34'),
+ ('Disadvantages', 2, None, '___sec35'),
('Ensemble Methods: From a Single Tree to Many Trees and Extreme '
'Boosting, Meet the Jungle of Methods',
2,
None,
- '___sec33'),
- ('An Overview of Ensemble Methods', 2, None, '___sec34'),
- ('Bagging', 2, None, '___sec35'),
- ('More bagging', 2, None, '___sec36'),
- ('Simple Voting Example, head or tail', 2, None, '___sec37'),
- ('Using the Voting Classifier', 2, None, '___sec38'),
+ '___sec36'),
+ ('An Overview of Ensemble Methods', 2, None, '___sec37'),
+ ('Bagging', 2, None, '___sec38'),
+ ('More bagging', 2, None, '___sec39'),
+ ('Simple Voting Example, head or tail', 2, None, '___sec40'),
+ ('Using the Voting Classifier', 2, None, '___sec41'),
('Please, not the moons again! Voting and Bagging',
2,
None,
- '___sec39'),
- ('Bagging Examples', 2, None, '___sec40'),
+ '___sec42'),
+ ('Bagging Examples', 2, None, '___sec43'),
('Making your own Bootstrap: Changing the Level of the Decision '
'Tree',
2,
None,
- '___sec41')]}
+ '___sec44')]}
end of tocinfo -->
-
+
A decision tree is typically divided into a root node, the interior nodes,
and the final leaf nodes or just leaves. These entities are then connected by so-called branches.
@@ -240,7 +251,23 @@ given some assumptions, make predictions about the target feature value
+
+
+
+
+
+
+
+
The overarching approach to decision trees is a top-down approach.
@@ -269,7 +296,7 @@ node.
In simplified terms, the process of training a decision tree and
@@ -287,7 +314,7 @@ Then we are essentially done!
@@ -383,7 +410,7 @@ plt.show()
There are mainly two steps
@@ -411,7 +438,7 @@ within box \( j \).
Unfortunately, it is computationally infeasible to consider every
@@ -430,7 +457,7 @@ better tree in some future step.
In order to implement the recursive binary splitting we start by selecting
@@ -481,7 +508,7 @@ region contains more than five observations.
-
The above procedure is rather straightforward, but leads often to
@@ -500,7 +527,7 @@ parameter \( \alpha \).
A classification tree is very similar to a regression tree, except
@@ -576,7 +603,7 @@ fall into that region.
The task of growing a
@@ -600,7 +627,7 @@ than is the classification error rate.
If our targets are the outcome of a classification process that takes
@@ -650,7 +677,7 @@ $$
@@ -691,7 +718,7 @@ os.system(cmd)
@@ -723,7 +750,7 @@ os.system(cmd)
Two algorithms stand out in the set up of decision trees:
@@ -741,7 +768,7 @@ in two branches.
For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \).
@@ -768,7 +795,7 @@ hyperparameters control additional stopping conditions such as the \( min\_sampl
The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the
@@ -796,7 +823,7 @@ just like for classification tasks, is prone to overfitting.
The example we will look at is a classical one in many Machine
@@ -836,7 +863,7 @@ The table here summarizes the various attributes and
@@ -912,7 +939,7 @@ os.system(cmd)
The above functions (gini, entropy and misclassification error) are
@@ -990,7 +1017,7 @@ split = get_split(dataset)
ID3, learns decision trees by constructing
@@ -1026,7 +1053,7 @@ attributes at each step while growing the tree.
@@ -1222,7 +1249,7 @@ attributes at each step while growing the tree.
@@ -1271,7 +1298,7 @@ deep_tree_clf.fit(X_train_scaled, y_train)
@@ -1343,7 +1370,7 @@ plt.show()
@@ -1371,7 +1398,7 @@ plt.show()
@@ -1393,7 +1420,7 @@ tree_reg.fit(X, y)
@@ -1471,7 +1498,7 @@ plt.show()
As stated above and seen in many of the examples discussed here about
@@ -1531,7 +1558,7 @@ We discuss these methods here.
The plain decision trees suffer from high
@@ -1558,7 +1585,7 @@ learning method.
Bagging typically results in improved accuracy
@@ -1587,7 +1614,7 @@ predictor, averaged over all \( B \) trees.
@@ -1608,7 +1635,7 @@ plt.show()
@@ -1659,7 +1686,7 @@ voting_clf.fit(X_train, y_train)
@@ -1718,7 +1745,7 @@ voting_clf.fit(X_train, y_train)
@@ -1780,7 +1807,7 @@ plt.show()
Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with
diff --git a/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz b/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz
index 5da6806d7..2bd714a70 100644
Binary files a/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz and b/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz differ
diff --git a/doc/pub/week44/ipynb/week44.ipynb b/doc/pub/week44/ipynb/week44.ipynb
index b3c277422..1107ee413 100644
--- a/doc/pub/week44/ipynb/week44.ipynb
+++ b/doc/pub/week44/ipynb/week44.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Oct 26, 2020**\n",
+ "Date: **Oct 27, 2020**\n",
"\n",
"Copyright 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -55,7 +55,7 @@
"informative** feature is done until we accomplish a stopping criteria\n",
"where we then finally end up in so called **leaf nodes**. \n",
"\n",
- "\n",
+ "## Basics of a tree\n",
"\n",
"A decision tree is typically divided into a **root node**, the **interior nodes**,\n",
"and the final **leaf nodes** or just **leaves**. These entities are then connected by so-called **branches**.\n",
@@ -67,6 +67,16 @@
"given some assumptions, make predictions about the target feature value\n",
"(class) of unseen query instances.\n",
"\n",
+ "## A Sketch of a Tree, Regression problem\n",
+ "\n",
+ "\n",
+ "\n",
+ "## A Sketch of a Tree, Classification problem\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
"## A typical Decision Tree with its pertinent Jargon, Classification Problem\n",
"\n",
"\n",
diff --git a/doc/src/week44/week44.do.txt b/doc/src/week44/week44.do.txt
index 02e5f7ff1..c6a1123b4 100644
--- a/doc/src/week44/week44.do.txt
+++ b/doc/src/week44/week44.do.txt
@@ -44,7 +44,8 @@ to be the most informative ones. The process of finding the _most
informative_ feature is done until we accomplish a stopping criteria
where we then finally end up in so called _leaf nodes_.
-
+!split
+===== Basics of a tree =====
A decision tree is typically divided into a _root node_, the _interior nodes_,
and the final _leaf nodes_ or just _leaves_. These entities are then connected by so-called _branches_.
@@ -56,6 +57,18 @@ learned the underlying structure of the training data and hence can,
given some assumptions, make predictions about the target feature value
(class) of unseen query instances.
+!split
+===== A Sketch of a Tree, Regression problem =====
+
+#FIGURE: [DataFiles/Regsimpletree.png, width=600 frac=0.8]
+
+!split
+===== A Sketch of a Tree, Classification problem =====
+
+#FIGURE: [DataFiles/Classimpletree.png, width=600 frac=0.8]
+
+
+
!split
===== A typical Decision Tree with its pertinent Jargon, Classification Problem =====
Basics of a tree
A typical Decision Tree with its pertinent Jargon, Classification Problem
+A Sketch of a Tree, Regression problem
+
+A Sketch of a Tree, Classification problem
+
+A typical Decision Tree with its pertinent Jargon, Classification Problem
@@ -233,7 +254,7 @@ This tree was produced using the Wisconsin cancer data (discussed here as well,
General Features
+General Features
How do we set it up?
+How do we set it up?
Decision trees and Regression
+Decision trees and Regression
Building a tree, regression
+Building a tree, regression
A top-down approach, recursive binary splitting
+A top-down approach, recursive binary splitting
Making a tree
+Making a tree
Pruning the tree
+Pruning the tree
Cost complexity pruning
+Cost complexity pruning
For each value of \( \alpha \) there corresponds a subtree \( T \in T_0 \) such that
$$
@@ -528,7 +549,7 @@ subtree corresponding to \( \alpha \).
Schematic Regression Procedure
+Schematic Regression Procedure
@@ -553,7 +574,7 @@ subtree corresponding to \( \alpha \).
A Classification Tree
+A Classification Tree
Growing a classification tree
+Growing a classification tree
Classification tree, how to split nodes
+Classification tree, how to split nodes
Visualizing the Tree, Classification
+Visualizing the Tree, Classification
Visualizing the Tree, The Moons
+Visualizing the Tree, The Moons
Algorithms for Setting up Decision Trees
+Algorithms for Setting up Decision Trees
The CART algorithm for Classification
+The CART algorithm for Classification
The CART algorithm for Regression
+The CART algorithm for Regression
Computing the Gini index
+Computing the Gini index
Simple Python Code to read in Data and perform Classification
+Simple Python Code to read in Data and perform Classification
Computing the Gini Factor
+Computing the Gini Factor
Entropy and the ID3 algorithm
+Entropy and the ID3 algorithm
Implementing the ID3 Algorithm
+Implementing the ID3 Algorithm
Cancer Data again now with Decision Trees and other Methods
+Cancer Data again now with Decision Trees and other Methods
Another example, the moons again
+Another example, the moons again
Playing around with regions
+Playing around with regions
Regression trees
+Regression trees
Final regressor code
+Final regressor code
Pros and cons of trees, pros
+Pros and cons of trees, pros
Disadvantages
+Disadvantages
Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
+Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
An Overview of Ensemble Methods
+An Overview of Ensemble Methods
@@ -1564,7 +1585,7 @@ We discuss these methods here.
Bagging
+Bagging
More bagging
+More bagging
Simple Voting Example, head or tail
+Simple Voting Example, head or tail
Using the Voting Classifier
+Using the Voting Classifier
Please, not the moons again! Voting and Bagging
+Please, not the moons again! Voting and Bagging
Bagging Examples
+Bagging Examples
Making your own Bootstrap: Changing the Level of the Decision Tree
+Making your own Bootstrap: Changing the Level of the Decision Tree
Oct 26, 2020
Oct 27, 2020
@@ -220,6 +226,11 @@ to be the most informative ones. The process of finding the most
informative feature is done until we accomplish a stopping criteria
where we then finally end up in so called leaf nodes.
+
+
+Basics of a tree
+
-A typical Decision Tree with its pertinent Jargon, Classification Problem
+A Sketch of a Tree, Regression problem
+
+
+
+A Sketch of a Tree, Classification problem
+
+
+
+A typical Decision Tree with its pertinent Jargon, Classification Problem
@@ -246,7 +273,7 @@ This tree was produced using the Wisconsin cancer data (discussed here as well,
-General Features
+General Features
-How do we set it up?
+How do we set it up?
-Decision trees and Regression
+Decision trees and Regression
-Building a tree, regression
+Building a tree, regression
-A top-down approach, recursive binary splitting
+A top-down approach, recursive binary splitting
-Making a tree
+Making a tree
Pruning the tree
+Pruning the tree
-Cost complexity pruning
+Cost complexity pruning
For each value of \( \alpha \) there corresponds a subtree \( T \in T_0 \) such that
$$
\sum_{m=1}^{\overline{T}}\sum_{i:x_i\in R_m}(y_i-\overline{y}_{R_m})^2+\alpha\overline{T},
@@ -526,7 +553,7 @@ subtree corresponding to \( \alpha \).
-Schematic Regression Procedure
+Schematic Regression Procedure
@@ -552,7 +579,7 @@ subtree corresponding to \( \alpha \).
-A Classification Tree
+A Classification Tree
-Growing a classification tree
+Growing a classification tree
-Classification tree, how to split nodes
+Classification tree, how to split nodes
-Visualizing the Tree, Classification
+Visualizing the Tree, Classification
-Visualizing the Tree, The Moons
+Visualizing the Tree, The Moons
-Algorithms for Setting up Decision Trees
+Algorithms for Setting up Decision Trees
-The CART algorithm for Classification
+The CART algorithm for Classification
-The CART algorithm for Regression
+The CART algorithm for Regression
-Computing the Gini index
+Computing the Gini index
-Simple Python Code to read in Data and perform Classification
+Simple Python Code to read in Data and perform Classification
-Computing the Gini Factor
+Computing the Gini Factor
-Entropy and the ID3 algorithm
+Entropy and the ID3 algorithm
-Implementing the ID3 Algorithm
+Implementing the ID3 Algorithm
-Cancer Data again now with Decision Trees and other Methods
+Cancer Data again now with Decision Trees and other Methods
-Another example, the moons again
+Another example, the moons again
-Playing around with regions
+Playing around with regions
-Regression trees
+Regression trees
-Final regressor code
+Final regressor code
-Pros and cons of trees, pros
+Pros and cons of trees, pros
-Disadvantages
+Disadvantages
-Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
+Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
-An Overview of Ensemble Methods
+An Overview of Ensemble Methods
@@ -1534,7 +1561,7 @@ We discuss these methods here.
-Bagging
+Bagging
-More bagging
+More bagging
-Simple Voting Example, head or tail
+Simple Voting Example, head or tail
-Using the Voting Classifier
+Using the Voting Classifier
-Please, not the moons again! Voting and Bagging
+Please, not the moons again! Voting and Bagging
-Bagging Examples
+Bagging Examples
-Making your own Bootstrap: Changing the Level of the Decision Tree
+Making your own Bootstrap: Changing the Level of the Decision Tree
Oct 26, 2020
Oct 27, 2020
@@ -225,6 +231,11 @@ to be the most informative ones. The process of finding the most
informative feature is done until we accomplish a stopping criteria
where we then finally end up in so called leaf nodes.
+
+
+Basics of a tree
+
-A typical Decision Tree with its pertinent Jargon, Classification Problem
+A Sketch of a Tree, Regression problem
+
+
+
+A Sketch of a Tree, Classification problem
+
+
+
+A typical Decision Tree with its pertinent Jargon, Classification Problem
@@ -251,7 +278,7 @@ This tree was produced using the Wisconsin cancer data (discussed here as well,
-General Features
+General Features
-How do we set it up?
+How do we set it up?
-Decision trees and Regression
+Decision trees and Regression
-Building a tree, regression
+Building a tree, regression
-A top-down approach, recursive binary splitting
+A top-down approach, recursive binary splitting
-Making a tree
+Making a tree
Pruning the tree
+Pruning the tree
-Cost complexity pruning
+Cost complexity pruning
For each value of \( \alpha \) there corresponds a subtree \( T \in T_0 \) such that
$$
\sum_{m=1}^{\overline{T}}\sum_{i:x_i\in R_m}(y_i-\overline{y}_{R_m})^2+\alpha\overline{T},
@@ -531,7 +558,7 @@ subtree corresponding to \( \alpha \).
-Schematic Regression Procedure
+Schematic Regression Procedure
@@ -557,7 +584,7 @@ subtree corresponding to \( \alpha \).
-A Classification Tree
+A Classification Tree
-Growing a classification tree
+Growing a classification tree
-Classification tree, how to split nodes
+Classification tree, how to split nodes
-Visualizing the Tree, Classification
+Visualizing the Tree, Classification
-Visualizing the Tree, The Moons
+Visualizing the Tree, The Moons
-Algorithms for Setting up Decision Trees
+Algorithms for Setting up Decision Trees
-The CART algorithm for Classification
+The CART algorithm for Classification
-The CART algorithm for Regression
+The CART algorithm for Regression
-Computing the Gini index
+Computing the Gini index
-Simple Python Code to read in Data and perform Classification
+Simple Python Code to read in Data and perform Classification
-Computing the Gini Factor
+Computing the Gini Factor
-Entropy and the ID3 algorithm
+Entropy and the ID3 algorithm
-Implementing the ID3 Algorithm
+Implementing the ID3 Algorithm
-Cancer Data again now with Decision Trees and other Methods
+Cancer Data again now with Decision Trees and other Methods
-Another example, the moons again
+Another example, the moons again
-Playing around with regions
+Playing around with regions
-Regression trees
+Regression trees
-Final regressor code
+Final regressor code
-Pros and cons of trees, pros
+Pros and cons of trees, pros
-Disadvantages
+Disadvantages
-Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
+Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
-An Overview of Ensemble Methods
+An Overview of Ensemble Methods
@@ -1539,7 +1566,7 @@ We discuss these methods here.
-Bagging
+Bagging
-More bagging
+More bagging
-Simple Voting Example, head or tail
+Simple Voting Example, head or tail
-Using the Voting Classifier
+Using the Voting Classifier
-Please, not the moons again! Voting and Bagging
+Please, not the moons again! Voting and Bagging
-Bagging Examples
+Bagging Examples
-Making your own Bootstrap: Changing the Level of the Decision Tree
+Making your own Bootstrap: Changing the Level of the Decision Tree