from sklearn.datasets import load_iris
-from sklearn import tree
-X, y = load_iris(return_X_y=True)
-tree_clf = tree.DecisionTreeClassifier()
-tree_clf = tree_clf.fit(X, y)
-# and then plot the tree
-tree.plot_tree(tree_clf)
+ import os
+from sklearn.datasets import load_breast_cancer
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import confusion_matrix
+from sklearn.tree import export_graphviz
+
+from IPython.display import Image
+from pydot import graph_from_dot_data
+import pandas as pd
+import numpy as np
+
+
+cancer = load_breast_cancer()
+X = pd.DataFrame(cancer.data, columns=cancer.feature_names)
+print(X)
+y = pd.Categorical.from_codes(cancer.target, cancer.target_names)
+y = pd.get_dummies(y)
+print(y)
+X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)
+tree_clf = DecisionTreeClassifier(max_depth=5)
+tree_clf.fit(X_train, y_train)
+
+export_graphviz(
+ tree_clf,
+ out_file="DataFiles/cancer.dot",
+ feature_names=cancer.feature_names,
+ class_names=cancer.target_names,
+ rounded=True,
+ filled=True
+)
+cmd = 'dot -Tpng DataFiles/cancer.dot -o DataFiles/cancer.png'
+os.system(cmd)
-
Printing out as text
- -Alternatively, the tree can also be exported in textual format with the function exporttext. -This method doesn’t require the installation of external libraries and is more compact: -
- +Visualizing the Tree, The Moons
from sklearn.datasets import load_iris
+ # Common imports
+import numpy as np
+from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
-from sklearn.tree import export_text
-iris = load_iris()
-decision_tree = DecisionTreeClassifier(random_state=0, max_depth=2)
-decision_tree = decision_tree.fit(iris.data, iris.target)
-r = export_text(decision_tree, feature_names=iris['feature_names'])
-print(r)
+from sklearn.datasets import make_moons
+from sklearn.tree import export_graphviz
+from pydot import graph_from_dot_data
+import pandas as pd
+import os
+
+np.random.seed(42)
+X, y = make_moons(n_samples=100, noise=0.25, random_state=53)
+X_train, X_test, y_train, y_test = train_test_split(X,y,random_state=0)
+tree_clf = DecisionTreeClassifier(max_depth=5)
+tree_clf.fit(X_train, y_train)
+
+export_graphviz(
+ tree_clf,
+ out_file="DataFiles/moons.dot",
+ rounded=True,
+ filled=True
+)
+cmd = 'dot -Tpng DataFiles/moons.dot -o DataFiles/moons.png'
+os.system(cmd)
-
Algorithms for Setting up Decision Trees
+Other ways of visualizing the trees
+ +Scikit-Learn has also another way to visualize the trees which is very useful, here with the Iris data.
+ + + +from sklearn.datasets import load_iris
+from sklearn import tree
+X, y = load_iris(return_X_y=True)
+tree_clf = tree.DecisionTreeClassifier()
+tree_clf = tree_clf.fit(X, y)
+# and then plot the tree
+tree.plot_tree(tree_clf)
+
+Two algorithms stand out in the set up of decision trees:
--
-
- The CART (Classification And Regression Tree) algorithm for both classification and regression -
- The ID3 algorithm based on the computation of the information gain for classification -
We discuss both algorithms with applications here. The popular library -Scikit-Learn uses the CART algorithm. For classification problems -you can use either the gini index or the entropy to split a tree -in two branches. -
@@ -412,7 +440,7 @@ in two branches.
-
The CART algorithm for Classification
+Printing out as text
-For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \). -This could be for example a threshold set by a number below a certain circumference of a malign tumor. +
Alternatively, the tree can also be exported in textual format with the function exporttext. +This method doesn’t require the installation of external libraries and is more compact:
-How do we find these two quantities? -We search for the pair \( (k,t_k) \) that produces the purest subset using for example the gini factor \( G \). -The cost function it tries to minimize is then -
-$$ -C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}}, -$$ -where \( G_{\mathrm{left/right}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \) - is the number of instances in the left/right subset -
+ +from sklearn.datasets import load_iris
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.tree import export_text
+iris = load_iris()
+decision_tree = DecisionTreeClassifier(random_state=0, max_depth=2)
+decision_tree = decision_tree.fit(iris.data, iris.target)
+r = export_text(decision_tree, feature_names=iris['feature_names'])
+print(r)
+
+Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets -and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the -\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other -hyperparameters control additional stopping conditions such as the \( min\_samples\_split \), -\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \). -
@@ -424,7 +443,7 @@ hyperparameters control additional stopping conditions such as the \( min\_sampl
-
The CART algorithm for Regression
+Algorithms for Setting up Decision Trees
-The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the -training set in a way that minimizes say the gini or entropy impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now -
-$$ -C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}. -$$ - -Here the MSE for a specific node is defined as
-$$ -\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2, -$$ - -with
-$$ -\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i, -$$ - -the mean value of all observations in a specific node.
- -Without any regularization, the regression task for decision trees, -just like for classification tasks, is prone to overfitting. +
Two algorithms stand out in the set up of decision trees:
+-
+
- The CART (Classification And Regression Tree) algorithm for both classification and regression +
- The ID3 algorithm based on the computation of the information gain for classification +
We discuss both algorithms with applications here. The popular library +Scikit-Learn uses the CART algorithm. For classification problems +you can use either the gini index or the entropy to split a tree +in two branches.
@@ -424,7 +419,7 @@ just like for classification tasks, is prone to overfitting.
-
Computing the Gini index
+The CART algorithm for Classification
-The example we will look at is a classical one in many Machine -Learning applications. Based on various meteorological features, we -have several so-called attributes which decide whether we at the end -will do some outdoor activity like skiing, going for a bike ride etc -etc. The table here contains the feautures outlook, temperature, -humidity and wind. The target or output is whether we ride -(True=1) or whether we do something else that day (False=0). The -attributes for each feature are then sunny, overcast and rain for the -outlook, hot, cold and mild for temperature, high and normal for -humidity and weak and strong for wind. +
For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \). +This could be for example a threshold set by a number below a certain circumference of a malign tumor.
-The table here summarizes the various attributes and
-| Day | Outlook | Temperature | Humidity | Wind | Ride |
| 1 | Sunny | Hot | High | Weak | 0 |
| 2 | Sunny | Hot | High | Strong | 1 |
| 3 | Overcast | Hot | High | Weak | 1 |
| 4 | Rain | Mild | High | Weak | 1 |
| 5 | Rain | Cool | Normal | Weak | 1 |
| 6 | Rain | Cool | Normal | Strong | 0 |
| 7 | Overcast | Cool | Normal | Strong | 1 |
| 8 | Sunny | Mild | High | Weak | 0 |
| 9 | Sunny | Cool | Normal | Weak | 1 |
| 10 | Rain | Mild | Normal | Weak | 1 |
| 11 | Sunny | Mild | Normal | Strong | 1 |
| 12 | Overcast | Mild | High | Strong | 1 |
| 13 | Overcast | Hot | Normal | Weak | 1 |
| 14 | Rain | Mild | High | Strong | 0 |
How do we find these two quantities? +We search for the pair \( (k,t_k) \) that produces the purest subset using for example the gini factor \( G \). +The cost function it tries to minimize is then +
+$$ +C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}}, +$$ + +where \( G_{\mathrm{left/right}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \) + is the number of instances in the left/right subset +
+ +Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets +and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the +\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other +hyperparameters control additional stopping conditions such as the \( min\_samples\_split \), +\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \). +
@@ -440,7 +431,7 @@ humidity and weak and strong for wind.
-
Simple Python Code to read in Data and perform Classification
+The CART algorithm for Regression
+The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the +training set in a way that minimizes say the gini or entropy impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now +
+$$ +C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}. +$$ - -# Common imports
-import numpy as np
-import pandas as pd
-import matplotlib.pyplot as plt
-from sklearn.tree import DecisionTreeClassifier
-from sklearn.model_selection import train_test_split
-from sklearn.tree import export_graphviz
-from sklearn.preprocessing import StandardScaler, OneHotEncoder
-from sklearn.compose import ColumnTransformer
-from IPython.display import Image
-from pydot import graph_from_dot_data
-import os
+Here the MSE for a specific node is defined as
+$$
+\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2,
+$$
-# Where to save the figures and data files
-PROJECT_ROOT_DIR = "Results"
-FIGURE_ID = "Results/FigureFiles"
-DATA_ID = "DataFiles/"
+with
+$$
+\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i,
+$$
-if not os.path.exists(PROJECT_ROOT_DIR):
- os.mkdir(PROJECT_ROOT_DIR)
-
-if not os.path.exists(FIGURE_ID):
- os.makedirs(FIGURE_ID)
-
-if not os.path.exists(DATA_ID):
- os.makedirs(DATA_ID)
-
-def image_path(fig_id):
- return os.path.join(FIGURE_ID, fig_id)
-
-def data_path(dat_id):
- return os.path.join(DATA_ID, dat_id)
-
-def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
-
-infile = open(data_path("rideclass.csv"),'r')
-
-# Read the experimental data with Pandas
-from IPython.display import display
-ridedata = pd.read_csv(infile,names = ('Outlook','Temperature','Humidity','Wind','Ride'))
-ridedata = pd.DataFrame(ridedata)
-
-# Features and targets
-X = ridedata.loc[:, ridedata.columns != 'Ride'].values
-y = ridedata.loc[:, ridedata.columns == 'Ride'].values
-
-# Create the encoder.
-encoder = OneHotEncoder(handle_unknown="ignore")
-# Assume for simplicity all features are categorical.
-encoder.fit(X)
-# Apply the encoder.
-X = encoder.transform(X)
-print(X)
-# Then do a Classification tree
-tree_clf = DecisionTreeClassifier(max_depth=2)
-tree_clf.fit(X, y)
-print("Train set accuracy with Decision Tree: {:.2f}".format(tree_clf.score(X,y)))
-#transfer to a decision tree graph
-export_graphviz(
- tree_clf,
- out_file="DataFiles/ride.dot",
- rounded=True,
- filled=True
-)
-cmd = 'dot -Tpng DataFiles/cancer.dot -o DataFiles/cancer.png'
-os.system(cmd)
-
-the mean value of all observations in a specific node.
+Without any regularization, the regression task for decision trees, +just like for classification tasks, is prone to overfitting. +
@@ -491,7 +431,7 @@ os.system(cmd)
-
Computing the Gini Factor
+Computing the Gini index
-The above functions (gini, entropy and misclassification error) are -important components of the so-called CART algorithm. We will discuss -this algorithm below after we have discussed the information gain -algorithm ID3. +
The example we will look at is a classical one in many Machine +Learning applications. Based on various meteorological features, we +have several so-called attributes which decide whether we at the end +will do some outdoor activity like skiing, going for a bike ride etc +etc. The table here contains the feautures outlook, temperature, +humidity and wind. The target or output is whether we ride +(True=1) or whether we do something else that day (False=0). The +attributes for each feature are then sunny, overcast and rain for the +outlook, hot, cold and mild for temperature, high and normal for +humidity and weak and strong for wind.
-In the example here we have converted all our attributes into numerical values \( 0,1,2 \) etc.
- - - -# Split a dataset based on an attribute and an attribute value
-def test_split(index, value, dataset):
- left, right = list(), list()
- for row in dataset:
- if row[index] < value:
- left.append(row)
- else:
- right.append(row)
- return left, right
-
-# Calculate the Gini index for a split dataset
-def gini_index(groups, classes):
- # count all samples at split point
- n_instances = float(sum([len(group) for group in groups]))
- # sum weighted Gini index for each group
- gini = 0.0
- for group in groups:
- size = float(len(group))
- # avoid divide by zero
- if size == 0:
- continue
- score = 0.0
- # score the group based on the score for each class
- for class_val in classes:
- p = [row[-1] for row in group].count(class_val) / size
- score += p * p
- # weight the group score by its relative size
- gini += (1.0 - score) * (size / n_instances)
- return gini
-
-# Select the best split point for a dataset
-def get_split(dataset):
- class_values = list(set(row[-1] for row in dataset))
- b_index, b_value, b_score, b_groups = 999, 999, 999, None
- for index in range(len(dataset[0])-1):
- for row in dataset:
- groups = test_split(index, row[index], dataset)
- gini = gini_index(groups, class_values)
- print('X%d < %.3f Gini=%.3f' % ((index+1), row[index], gini))
- if gini < b_score:
- b_index, b_value, b_score, b_groups = index, row[index], gini, groups
- return {'index':b_index, 'value':b_value, 'groups':b_groups}
-
-dataset = [[0,0,0,0,0],
- [0,0,0,1,1],
- [1,0,0,0,1],
- [2,1,0,0,1],
- [2,2,1,0,1],
- [2,2,1,1,0],
- [1,2,1,1,1],
- [0,1,0,0,0],
- [0,2,1,0,1],
- [2,1,1,0,1],
- [0,1,1,1,1],
- [1,1,0,1,1],
- [1,0,1,0,1],
- [2,1,0,1,0]]
-
-split = get_split(dataset)
-print('Split: [X%d < %.3f]' % ((split['index']+1), split['value']))
-
-The table here summarizes the various attributes and
+| Day | Outlook | Temperature | Humidity | Wind | Ride |
| 1 | Sunny | Hot | High | Weak | 0 |
| 2 | Sunny | Hot | High | Strong | 1 |
| 3 | Overcast | Hot | High | Weak | 1 |
| 4 | Rain | Mild | High | Weak | 1 |
| 5 | Rain | Cool | Normal | Weak | 1 |
| 6 | Rain | Cool | Normal | Strong | 0 |
| 7 | Overcast | Cool | Normal | Strong | 1 |
| 8 | Sunny | Mild | High | Weak | 0 |
| 9 | Sunny | Cool | Normal | Weak | 1 |
| 10 | Rain | Mild | Normal | Weak | 1 |
| 11 | Sunny | Mild | Normal | Strong | 1 |
| 12 | Overcast | Mild | High | Strong | 1 |
| 13 | Overcast | Hot | Normal | Weak | 1 |
| 14 | Rain | Mild | High | Strong | 0 |
@@ -492,7 +447,7 @@ split = get_split(dataset)
-
Entropy and the ID3 algorithm
+Simple Python Code to read in Data and perform Classification
-The ID3 algorithm learns decision trees by constructing -them in a top down way, beginning with the question which attribute should be tested at the root of the tree? -
--
-
- Each instance attribute is evaluated using a statistical test to determine how well it alone classifies the training examples. -
- The best attribute is selected and used as the test at the root node of the tree. -
- A descendant of the root node is then created for each possible value of this attribute. -
- Training examples are sorted to the appropriate descendant node. -
- The entire process is then repeated using the training examples associated with each descendant node to select the best attribute to test at that point in the tree. -
- This forms a greedy search for an acceptable decision tree, in which the algorithm never backtracks to reconsider earlier choices. -
The ID3 algorithm selects which attribute to test at each node in the -tree. -
+ +# Common imports
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.model_selection import train_test_split
+from sklearn.tree import export_graphviz
+from sklearn.preprocessing import StandardScaler, OneHotEncoder
+from sklearn.compose import ColumnTransformer
+from IPython.display import Image
+from pydot import graph_from_dot_data
+import os
-We would like to select the attribute that is most useful for classifying
-examples.
-
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
-What is a good quantitative measure of the worth of an attribute?
+if not os.path.exists(PROJECT_ROOT_DIR):
+ os.mkdir(PROJECT_ROOT_DIR)
-Information gain measures how well a given attribute separates the
-training examples according to their target classification.
-
+if not os.path.exists(FIGURE_ID):
+ os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+ os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+ return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+ return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+ plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("rideclass.csv"),'r')
+
+# Read the experimental data with Pandas
+from IPython.display import display
+ridedata = pd.read_csv(infile,names = ('Outlook','Temperature','Humidity','Wind','Ride'))
+ridedata = pd.DataFrame(ridedata)
+
+# Features and targets
+X = ridedata.loc[:, ridedata.columns != 'Ride'].values
+y = ridedata.loc[:, ridedata.columns == 'Ride'].values
+
+# Create the encoder.
+encoder = OneHotEncoder(handle_unknown="ignore")
+# Assume for simplicity all features are categorical.
+encoder.fit(X)
+# Apply the encoder.
+X = encoder.transform(X)
+print(X)
+# Then do a Classification tree
+tree_clf = DecisionTreeClassifier(max_depth=2)
+tree_clf.fit(X, y)
+print("Train set accuracy with Decision Tree: {:.2f}".format(tree_clf.score(X,y)))
+#transfer to a decision tree graph
+export_graphviz(
+ tree_clf,
+ out_file="DataFiles/ride.dot",
+ rounded=True,
+ filled=True
+)
+cmd = 'dot -Tpng DataFiles/cancer.dot -o DataFiles/cancer.png'
+os.system(cmd)
+
+The ID3 algorithm uses this information gain measure to select among the candidate -attributes at each step while growing the tree. -
@@ -431,7 +498,7 @@ attributes at each step while growing the tree.
-
Cancer Data again now with Decision Trees and other Methods
+Computing the Gini Factor
+ +The above functions (gini, entropy and misclassification error) are +important components of the so-called CART algorithm. We will discuss +this algorithm below after we have discussed the information gain +algorithm ID3. +
+ +In the example here we have converted all our attributes into numerical values \( 0,1,2 \) etc.
+import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import load_breast_cancer
-from sklearn.svm import SVC
-from sklearn.linear_model import LogisticRegression
-from sklearn.tree import DecisionTreeClassifier
+ # Split a dataset based on an attribute and an attribute value
+def test_split(index, value, dataset):
+ left, right = list(), list()
+ for row in dataset:
+ if row[index] < value:
+ left.append(row)
+ else:
+ right.append(row)
+ return left, right
+
+# Calculate the Gini index for a split dataset
+def gini_index(groups, classes):
+ # count all samples at split point
+ n_instances = float(sum([len(group) for group in groups]))
+ # sum weighted Gini index for each group
+ gini = 0.0
+ for group in groups:
+ size = float(len(group))
+ # avoid divide by zero
+ if size == 0:
+ continue
+ score = 0.0
+ # score the group based on the score for each class
+ for class_val in classes:
+ p = [row[-1] for row in group].count(class_val) / size
+ score += p * p
+ # weight the group score by its relative size
+ gini += (1.0 - score) * (size / n_instances)
+ return gini
-# Load the data
-cancer = load_breast_cancer()
+# Select the best split point for a dataset
+def get_split(dataset):
+ class_values = list(set(row[-1] for row in dataset))
+ b_index, b_value, b_score, b_groups = 999, 999, 999, None
+ for index in range(len(dataset[0])-1):
+ for row in dataset:
+ groups = test_split(index, row[index], dataset)
+ gini = gini_index(groups, class_values)
+ print('X%d < %.3f Gini=%.3f' % ((index+1), row[index], gini))
+ if gini < b_score:
+ b_index, b_value, b_score, b_groups = index, row[index], gini, groups
+ return {'index':b_index, 'value':b_value, 'groups':b_groups}
+
+dataset = [[0,0,0,0,0],
+ [0,0,0,1,1],
+ [1,0,0,0,1],
+ [2,1,0,0,1],
+ [2,2,1,0,1],
+ [2,2,1,1,0],
+ [1,2,1,1,1],
+ [0,1,0,0,0],
+ [0,2,1,0,1],
+ [2,1,1,0,1],
+ [0,1,1,1,1],
+ [1,1,0,1,1],
+ [1,0,1,0,1],
+ [2,1,0,1,0]]
-X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
-print(X_train.shape)
-print(X_test.shape)
-# Logistic Regression
-logreg = LogisticRegression(solver='lbfgs')
-logreg.fit(X_train, y_train)
-print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
-# Support vector machine
-svm = SVC(gamma='auto', C=100)
-svm.fit(X_train, y_train)
-print("Test set accuracy with SVM: {:.2f}".format(svm.score(X_test,y_test)))
-# Decision Trees
-deep_tree_clf = DecisionTreeClassifier(max_depth=None)
-deep_tree_clf.fit(X_train, y_train)
-print("Test set accuracy with Decision Trees: {:.2f}".format(deep_tree_clf.score(X_test,y_test)))
-#now scale the data
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-# Logistic Regression
-logreg.fit(X_train_scaled, y_train)
-print("Test set accuracy Logistic Regression with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
-# Support Vector Machine
-svm.fit(X_train_scaled, y_train)
-print("Test set accuracy SVM with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
-# Decision Trees
-deep_tree_clf.fit(X_train_scaled, y_train)
-print("Test set accuracy with Decision Trees and scaled data: {:.2f}".format(deep_tree_clf.score(X_test_scaled,y_test)))
+split = get_split(dataset)
+print('Split: [X%d < %.3f]' % ((split['index']+1), split['value']))
-
Another example, the moons again
+Entropy and the ID3 algorithm
- -from __future__ import division, print_function, unicode_literals
+The ID3 algorithm learns decision trees by constructing
+them in a top down way, beginning with the question which attribute should be tested at the root of the tree?
+
-# Common imports
-import numpy as np
-import os
+
+- Each instance attribute is evaluated using a statistical test to determine how well it alone classifies the training examples.
+- The best attribute is selected and used as the test at the root node of the tree.
+- A descendant of the root node is then created for each possible value of this attribute.
+- Training examples are sorted to the appropriate descendant node.
+- The entire process is then repeated using the training examples associated with each descendant node to select the best attribute to test at that point in the tree.
+- This forms a greedy search for an acceptable decision tree, in which the algorithm never backtracks to reconsider earlier choices.
+
+The ID3 algorithm selects which attribute to test at each node in the
+tree.
+
-# to make this notebook's output stable across runs
-np.random.seed(42)
+We would like to select the attribute that is most useful for classifying
+examples.
+
-# To plot pretty figures
-import matplotlib
-import matplotlib.pyplot as plt
-from matplotlib.colors import ListedColormap
-plt.rcParams['axes.labelsize'] = 14
-plt.rcParams['xtick.labelsize'] = 12
-plt.rcParams['ytick.labelsize'] = 12
+What is a good quantitative measure of the worth of an attribute?
+Information gain measures how well a given attribute separates the
+training examples according to their target classification.
+
-from sklearn.svm import SVC
-from sklearn import datasets
-from sklearn.tree import DecisionTreeClassifier
-from sklearn.datasets import make_moons
-from sklearn.tree import export_graphviz
-
-Xm, ym = make_moons(n_samples=100, noise=0.25, random_state=53)
-
-deep_tree_clf1 = DecisionTreeClassifier(random_state=42)
-deep_tree_clf2 = DecisionTreeClassifier(min_samples_leaf=4, random_state=42)
-deep_tree_clf1.fit(Xm, ym)
-deep_tree_clf2.fit(Xm, ym)
-
-
-def plot_decision_boundary(clf, X, y, axes=[0, 7.5, 0, 3], iris=True, legend=False, plot_training=True):
- x1s = np.linspace(axes[0], axes[1], 100)
- x2s = np.linspace(axes[2], axes[3], 100)
- x1, x2 = np.meshgrid(x1s, x2s)
- X_new = np.c_[x1.ravel(), x2.ravel()]
- y_pred = clf.predict(X_new).reshape(x1.shape)
- custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
- plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
- if not iris:
- custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
- plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
- if plot_training:
- plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", label="Iris-Setosa")
- plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", label="Iris-Versicolor")
- plt.plot(X[:, 0][y==2], X[:, 1][y==2], "g^", label="Iris-Virginica")
- plt.axis(axes)
- if iris:
- plt.xlabel("Petal length", fontsize=14)
- plt.ylabel("Petal width", fontsize=14)
- else:
- plt.xlabel(r"$x_1$", fontsize=18)
- plt.ylabel(r"$x_2$", fontsize=18, rotation=0)
- if legend:
- plt.legend(loc="lower right", fontsize=14)
-plt.figure(figsize=(11, 4))
-plt.subplot(121)
-plot_decision_boundary(deep_tree_clf1, Xm, ym, axes=[-1.5, 2.5, -1, 1.5], iris=False)
-plt.title("No restrictions", fontsize=16)
-plt.subplot(122)
-plot_decision_boundary(deep_tree_clf2, Xm, ym, axes=[-1.5, 2.5, -1, 1.5], iris=False)
-plt.title("min_samples_leaf = {}".format(deep_tree_clf2.min_samples_leaf), fontsize=14)
-plt.show()
-
-The ID3 algorithm uses this information gain measure to select among the candidate +attributes at each step while growing the tree. +
@@ -487,7 +438,7 @@ plt.show()
-
Playing around with regions
+Cancer Data again now with Decision Trees and other Methods
np.random.seed(6)
-Xs = np.random.rand(100, 2) - 0.5
-ys = (Xs[:, 0] > 0).astype(np.float32) * 2
+ import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import load_breast_cancer
+from sklearn.svm import SVC
+from sklearn.linear_model import LogisticRegression
+from sklearn.tree import DecisionTreeClassifier
-angle = np.pi/4
-rotation_matrix = np.array([[np.cos(angle), -np.sin(angle)], [np.sin(angle), np.cos(angle)]])
-Xsr = Xs.dot(rotation_matrix)
+# Load the data
+cancer = load_breast_cancer()
-tree_clf_s = DecisionTreeClassifier(random_state=42)
-tree_clf_s.fit(Xs, ys)
-tree_clf_sr = DecisionTreeClassifier(random_state=42)
-tree_clf_sr.fit(Xsr, ys)
-
-plt.figure(figsize=(11, 4))
-plt.subplot(121)
-plot_decision_boundary(tree_clf_s, Xs, ys, axes=[-0.7, 0.7, -0.7, 0.7], iris=False)
-plt.subplot(122)
-plot_decision_boundary(tree_clf_sr, Xsr, ys, axes=[-0.7, 0.7, -0.7, 0.7], iris=False)
-
-plt.show()
+X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
+print(X_train.shape)
+print(X_test.shape)
+# Logistic Regression
+logreg = LogisticRegression(solver='lbfgs')
+logreg.fit(X_train, y_train)
+print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
+# Support vector machine
+svm = SVC(gamma='auto', C=100)
+svm.fit(X_train, y_train)
+print("Test set accuracy with SVM: {:.2f}".format(svm.score(X_test,y_test)))
+# Decision Trees
+deep_tree_clf = DecisionTreeClassifier(max_depth=None)
+deep_tree_clf.fit(X_train, y_train)
+print("Test set accuracy with Decision Trees: {:.2f}".format(deep_tree_clf.score(X_test,y_test)))
+#now scale the data
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+# Logistic Regression
+logreg.fit(X_train_scaled, y_train)
+print("Test set accuracy Logistic Regression with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
+# Support Vector Machine
+svm.fit(X_train_scaled, y_train)
+print("Test set accuracy SVM with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
+# Decision Trees
+deep_tree_clf.fit(X_train_scaled, y_train)
+print("Test set accuracy with Decision Trees and scaled data: {:.2f}".format(deep_tree_clf.score(X_test_scaled,y_test)))
-
Regression trees
+Another example, the moons again
# Quadratic training set + noise
+ from __future__ import division, print_function, unicode_literals
+
+# Common imports
+import numpy as np
+import os
+
+# to make this notebook's output stable across runs
np.random.seed(42)
-m = 200
-X = np.random.rand(m, 1)
-y = 4 * (X - 0.5) ** 2
-y = y + np.random.randn(m, 1) / 10
-
-from sklearn.tree import DecisionTreeRegressor
-tree_reg = DecisionTreeRegressor(max_depth=2, random_state=42)
-tree_reg.fit(X, y)
+# To plot pretty figures
+import matplotlib
+import matplotlib.pyplot as plt
+from matplotlib.colors import ListedColormap
+plt.rcParams['axes.labelsize'] = 14
+plt.rcParams['xtick.labelsize'] = 12
+plt.rcParams['ytick.labelsize'] = 12
+
+
+from sklearn.svm import SVC
+from sklearn import datasets
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.datasets import make_moons
+from sklearn.tree import export_graphviz
+
+Xm, ym = make_moons(n_samples=100, noise=0.25, random_state=53)
+
+deep_tree_clf1 = DecisionTreeClassifier(random_state=42)
+deep_tree_clf2 = DecisionTreeClassifier(min_samples_leaf=4, random_state=42)
+deep_tree_clf1.fit(Xm, ym)
+deep_tree_clf2.fit(Xm, ym)
+
+
+def plot_decision_boundary(clf, X, y, axes=[0, 7.5, 0, 3], iris=True, legend=False, plot_training=True):
+ x1s = np.linspace(axes[0], axes[1], 100)
+ x2s = np.linspace(axes[2], axes[3], 100)
+ x1, x2 = np.meshgrid(x1s, x2s)
+ X_new = np.c_[x1.ravel(), x2.ravel()]
+ y_pred = clf.predict(X_new).reshape(x1.shape)
+ custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
+ plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
+ if not iris:
+ custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
+ plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
+ if plot_training:
+ plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", label="Iris-Setosa")
+ plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", label="Iris-Versicolor")
+ plt.plot(X[:, 0][y==2], X[:, 1][y==2], "g^", label="Iris-Virginica")
+ plt.axis(axes)
+ if iris:
+ plt.xlabel("Petal length", fontsize=14)
+ plt.ylabel("Petal width", fontsize=14)
+ else:
+ plt.xlabel(r"$x_1$", fontsize=18)
+ plt.ylabel(r"$x_2$", fontsize=18, rotation=0)
+ if legend:
+ plt.legend(loc="lower right", fontsize=14)
+plt.figure(figsize=(11, 4))
+plt.subplot(121)
+plot_decision_boundary(deep_tree_clf1, Xm, ym, axes=[-1.5, 2.5, -1, 1.5], iris=False)
+plt.title("No restrictions", fontsize=16)
+plt.subplot(122)
+plot_decision_boundary(deep_tree_clf2, Xm, ym, axes=[-1.5, 2.5, -1, 1.5], iris=False)
+plt.title("min_samples_leaf = {}".format(deep_tree_clf2.min_samples_leaf), fontsize=14)
+plt.show()
-
Final regressor code
+Playing around with regions
from sklearn.tree import DecisionTreeRegressor
+ np.random.seed(6)
+Xs = np.random.rand(100, 2) - 0.5
+ys = (Xs[:, 0] > 0).astype(np.float32) * 2
-tree_reg1 = DecisionTreeRegressor(random_state=42, max_depth=2)
-tree_reg2 = DecisionTreeRegressor(random_state=42, max_depth=3)
-tree_reg1.fit(X, y)
-tree_reg2.fit(X, y)
+angle = np.pi/4
+rotation_matrix = np.array([[np.cos(angle), -np.sin(angle)], [np.sin(angle), np.cos(angle)]])
+Xsr = Xs.dot(rotation_matrix)
-def plot_regression_predictions(tree_reg, X, y, axes=[0, 1, -0.2, 1], ylabel="$y$"):
- x1 = np.linspace(axes[0], axes[1], 500).reshape(-1, 1)
- y_pred = tree_reg.predict(x1)
- plt.axis(axes)
- plt.xlabel("$x_1$", fontsize=18)
- if ylabel:
- plt.ylabel(ylabel, fontsize=18, rotation=0)
- plt.plot(X, y, "b.")
- plt.plot(x1, y_pred, "r.-", linewidth=2, label=r"$\hat{y}$")
+tree_clf_s = DecisionTreeClassifier(random_state=42)
+tree_clf_s.fit(Xs, ys)
+tree_clf_sr = DecisionTreeClassifier(random_state=42)
+tree_clf_sr.fit(Xsr, ys)
plt.figure(figsize=(11, 4))
plt.subplot(121)
-plot_regression_predictions(tree_reg1, X, y)
-for split, style in ((0.1973, "k-"), (0.0917, "k--"), (0.7718, "k--")):
- plt.plot([split, split], [-0.2, 1], style, linewidth=2)
-plt.text(0.21, 0.65, "Depth=0", fontsize=15)
-plt.text(0.01, 0.2, "Depth=1", fontsize=13)
-plt.text(0.65, 0.8, "Depth=1", fontsize=13)
-plt.legend(loc="upper center", fontsize=18)
-plt.title("max_depth=2", fontsize=14)
-
+plot_decision_boundary(tree_clf_s, Xs, ys, axes=[-0.7, 0.7, -0.7, 0.7], iris=False)
plt.subplot(122)
-plot_regression_predictions(tree_reg2, X, y, ylabel=None)
-for split, style in ((0.1973, "k-"), (0.0917, "k--"), (0.7718, "k--")):
- plt.plot([split, split], [-0.2, 1], style, linewidth=2)
-for split in (0.0458, 0.1298, 0.2873, 0.9040):
- plt.plot([split, split], [-0.2, 1], "k:", linewidth=1)
-plt.text(0.3, 0.5, "Depth=2", fontsize=13)
-plt.title("max_depth=3", fontsize=14)
-
-plt.show()
-
-tree_reg1 = DecisionTreeRegressor(random_state=42)
-tree_reg2 = DecisionTreeRegressor(random_state=42, min_samples_leaf=10)
-tree_reg1.fit(X, y)
-tree_reg2.fit(X, y)
-
-x1 = np.linspace(0, 1, 500).reshape(-1, 1)
-y_pred1 = tree_reg1.predict(x1)
-y_pred2 = tree_reg2.predict(x1)
-
-plt.figure(figsize=(11, 4))
-
-plt.subplot(121)
-plt.plot(X, y, "b.")
-plt.plot(x1, y_pred1, "r.-", linewidth=2, label=r"$\hat{y}$")
-plt.axis([0, 1, -0.2, 1.1])
-plt.xlabel("$x_1$", fontsize=18)
-plt.ylabel("$y$", fontsize=18, rotation=0)
-plt.legend(loc="upper center", fontsize=18)
-plt.title("No restrictions", fontsize=14)
-
-plt.subplot(122)
-plt.plot(X, y, "b.")
-plt.plot(x1, y_pred2, "r.-", linewidth=2, label=r"$\hat{y}$")
-plt.axis([0, 1, -0.2, 1.1])
-plt.xlabel("$x_1$", fontsize=18)
-plt.title("min_samples_leaf={}".format(tree_reg2.min_samples_leaf), fontsize=14)
+plot_decision_boundary(tree_clf_sr, Xsr, ys, axes=[-0.7, 0.7, -0.7, 0.7], iris=False)
plt.show()
@@ -508,7 +450,7 @@ plt.show()
-
Pros and cons of trees, pros
+Regression trees
+ + +# Quadratic training set + noise
+np.random.seed(42)
+m = 200
+X = np.random.rand(m, 1)
+y = 4 * (X - 0.5) ** 2
+y = y + np.random.randn(m, 1) / 10
+
+from sklearn.tree import DecisionTreeRegressor
+
+tree_reg = DecisionTreeRegressor(max_depth=2, random_state=42)
+tree_reg.fit(X, y)
+
+-
-
- White box, easy to interpret model. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches discussed earlier (think of support vector machines) -
- Trees are very easy to explain to people. In fact, they are even easier to explain than linear regression! -
- No feature normalization needed -
- Tree models can handle both continuous and categorical data (Classification and Regression Trees) -
- Can model nonlinear relationships -
- Can model interactions between the different descriptive features -
- Trees can be displayed graphically, and are easily interpreted even by a non-expert (especially if they are small) -
diff --git a/doc/pub/week45/html/._week45-bs036.html b/doc/pub/week45/html/._week45-bs036.html index f8d1cd9ef..5f16c62bb 100644 --- a/doc/pub/week45/html/._week45-bs036.html +++ b/doc/pub/week45/html/._week45-bs036.html @@ -85,6 +85,11 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'classification-tree-how-to-split-nodes'), + ('Gini Index?Coefficient/Impurity', + 2, + None, + 'gini-index-coefficient-impurity'), + ('Why binary split?', 2, None, 'why-binary-split'), ('Visualizing the Tree, Classification', 2, None, @@ -307,62 +312,64 @@ MathJax.Hub.Config({
-
Disadvantages
+Final regressor code
+ + +from sklearn.tree import DecisionTreeRegressor
+
+tree_reg1 = DecisionTreeRegressor(random_state=42, max_depth=2)
+tree_reg2 = DecisionTreeRegressor(random_state=42, max_depth=3)
+tree_reg1.fit(X, y)
+tree_reg2.fit(X, y)
+
+def plot_regression_predictions(tree_reg, X, y, axes=[0, 1, -0.2, 1], ylabel="$y$"):
+ x1 = np.linspace(axes[0], axes[1], 500).reshape(-1, 1)
+ y_pred = tree_reg.predict(x1)
+ plt.axis(axes)
+ plt.xlabel("$x_1$", fontsize=18)
+ if ylabel:
+ plt.ylabel(ylabel, fontsize=18, rotation=0)
+ plt.plot(X, y, "b.")
+ plt.plot(x1, y_pred, "r.-", linewidth=2, label=r"$\hat{y}$")
+
+plt.figure(figsize=(11, 4))
+plt.subplot(121)
+plot_regression_predictions(tree_reg1, X, y)
+for split, style in ((0.1973, "k-"), (0.0917, "k--"), (0.7718, "k--")):
+ plt.plot([split, split], [-0.2, 1], style, linewidth=2)
+plt.text(0.21, 0.65, "Depth=0", fontsize=15)
+plt.text(0.01, 0.2, "Depth=1", fontsize=13)
+plt.text(0.65, 0.8, "Depth=1", fontsize=13)
+plt.legend(loc="upper center", fontsize=18)
+plt.title("max_depth=2", fontsize=14)
+
+plt.subplot(122)
+plot_regression_predictions(tree_reg2, X, y, ylabel=None)
+for split, style in ((0.1973, "k-"), (0.0917, "k--"), (0.7718, "k--")):
+ plt.plot([split, split], [-0.2, 1], style, linewidth=2)
+for split in (0.0458, 0.1298, 0.2873, 0.9040):
+ plt.plot([split, split], [-0.2, 1], "k:", linewidth=1)
+plt.text(0.3, 0.5, "Depth=2", fontsize=13)
+plt.title("max_depth=3", fontsize=14)
+
+plt.show()
+
+tree_reg1 = DecisionTreeRegressor(random_state=42)
+tree_reg2 = DecisionTreeRegressor(random_state=42, min_samples_leaf=10)
+tree_reg1.fit(X, y)
+tree_reg2.fit(X, y)
+
+x1 = np.linspace(0, 1, 500).reshape(-1, 1)
+y_pred1 = tree_reg1.predict(x1)
+y_pred2 = tree_reg2.predict(x1)
+
+plt.figure(figsize=(11, 4))
+
+plt.subplot(121)
+plt.plot(X, y, "b.")
+plt.plot(x1, y_pred1, "r.-", linewidth=2, label=r"$\hat{y}$")
+plt.axis([0, 1, -0.2, 1.1])
+plt.xlabel("$x_1$", fontsize=18)
+plt.ylabel("$y$", fontsize=18, rotation=0)
+plt.legend(loc="upper center", fontsize=18)
+plt.title("No restrictions", fontsize=14)
+
+plt.subplot(122)
+plt.plot(X, y, "b.")
+plt.plot(x1, y_pred2, "r.-", linewidth=2, label=r"$\hat{y}$")
+plt.axis([0, 1, -0.2, 1.1])
+plt.xlabel("$x_1$", fontsize=18)
+plt.title("min_samples_leaf={}".format(tree_reg2.min_samples_leaf), fontsize=14)
+
+plt.show()
+
+-
-
- Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches -
- If continuous features are used the tree may become quite large and hence less interpretable -
- Decision trees are prone to overfit the training data and hence do not well generalize the data if no stopping criteria or improvements like pruning, boosting or bagging are implemented -
- Small changes in the data may lead to a completely different tree. This issue can be addressed by using ensemble methods like bagging, boosting or random forests -
- Unbalanced datasets where some target feature values occur much more frequently than others may lead to biased trees since the frequently occurring feature values are preferred over the less frequently occurring ones. -
- If the number of features is relatively large (high dimensional) and the number of instances is relatively low, the tree might overfit the data -
- Features with many levels may be preferred over features with less levels since for them it is more easy to split the dataset such that the sub datasets only contain pure target feature values. This issue can be addressed by preferring for instance the information gain ratio as splitting criteria over information gain -
However, by aggregating many decision trees, using methods like -bagging, random forests, and boosting, the predictive performance of -trees can be substantially improved. -
@@ -415,7 +515,7 @@ trees can be substantially improved.
-
Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
- -As stated above and seen in many of the examples discussed here about -a single decision tree, we often end up overfitting our training -data. This normally means that we have a high variance. Can we reduce -the variance of a statistical learning method? -
- -This leads us to a set of different methods that can combine different -machine learning algorithms or just use one of them to construct -forests and jungles of trees, homogeneous ones or heterogenous -ones. These methods are recognized by different names which we will -try to explain here. These are -
- --
-
- Voting classifiers -
- Bagging and Pasting -
- Random forests -
- Boosting methods, from adaptive to Extreme Gradient Boosting (XGBoost) -
We discuss these methods here.
+Pros and cons of trees, pros
+-
+
- White box, easy to interpret model. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches discussed earlier (think of support vector machines) +
- Trees are very easy to explain to people. In fact, they are even easier to explain than linear regression! +
- No feature normalization needed +
- Tree models can handle both continuous and categorical data (Classification and Regression Trees) +
- Can model nonlinear relationships +
- Can model interactions between the different descriptive features +
- Trees can be displayed graphically, and are easily interpreted even by a non-expert (especially if they are small) +
diff --git a/doc/pub/week45/html/._week45-bs038.html b/doc/pub/week45/html/._week45-bs038.html index 71333f1f8..159a1ae3b 100644 --- a/doc/pub/week45/html/._week45-bs038.html +++ b/doc/pub/week45/html/._week45-bs038.html @@ -85,6 +85,11 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'classification-tree-how-to-split-nodes'), + ('Gini Index?Coefficient/Impurity', + 2, + None, + 'gini-index-coefficient-impurity'), + ('Why binary split?', 2, None, 'why-binary-split'), ('Visualizing the Tree, Classification', 2, None, @@ -307,62 +312,64 @@ MathJax.Hub.Config({
-
An Overview of Ensemble Methods
+Disadvantages
--
+
-
+
- Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches +
- If continuous features are used the tree may become quite large and hence less interpretable +
- Decision trees are prone to overfit the training data and hence do not well generalize the data if no stopping criteria or improvements like pruning, boosting or bagging are implemented +
- Small changes in the data may lead to a completely different tree. This issue can be addressed by using ensemble methods like bagging, boosting or random forests +
- Unbalanced datasets where some target feature values occur much more frequently than others may lead to biased trees since the frequently occurring feature values are preferred over the less frequently occurring ones. +
- If the number of features is relatively large (high dimensional) and the number of instances is relatively low, the tree might overfit the data +
- Features with many levels may be preferred over features with less levels since for them it is more easy to split the dataset such that the sub datasets only contain pure target feature values. This issue can be addressed by preferring for instance the information gain ratio as splitting criteria over information gain +
However, by aggregating many decision trees, using methods like +bagging, random forests, and boosting, the predictive performance of +trees can be substantially improved. +
@@ -407,7 +422,7 @@ MathJax.Hub.Config({
-
Bagging
+Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
-The plain decision trees suffer from high -variance. This means that if we split the training data into two parts -at random, and fit a decision tree to both halves, the results that we -get could be quite different. In contrast, a procedure with low -variance will yield similar results if applied repeatedly to distinct -data sets; linear regression tends to have low variance, if the ratio -of \( n \) to \( p \) is moderately large. +
As stated above and seen in many of the examples discussed here about +a single decision tree, we often end up overfitting our training +data. This normally means that we have a high variance. Can we reduce +the variance of a statistical learning method?
-Bootstrap aggregation, or just bagging, is a -general-purpose procedure for reducing the variance of a statistical -learning method. +
This leads us to a set of different methods that can combine different +machine learning algorithms or just use one of them to construct +forests and jungles of trees, homogeneous ones or heterogenous +ones. These methods are recognized by different names which we will +try to explain here. These are
+-
+
- Voting classifiers +
- Bagging and Pasting +
- Random forests +
- Boosting methods, from adaptive to Extreme Gradient Boosting (XGBoost) +
We discuss these methods here.
+diff --git a/doc/pub/week45/html/._week45-bs040.html b/doc/pub/week45/html/._week45-bs040.html index efa344e0f..9f4f59be1 100644 --- a/doc/pub/week45/html/._week45-bs040.html +++ b/doc/pub/week45/html/._week45-bs040.html @@ -85,6 +85,11 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'classification-tree-how-to-split-nodes'), + ('Gini Index?Coefficient/Impurity', + 2, + None, + 'gini-index-coefficient-impurity'), + ('Why binary split?', 2, None, 'why-binary-split'), ('Visualizing the Tree, Classification', 2, None, @@ -307,62 +312,64 @@ MathJax.Hub.Config({
-
More bagging
+An Overview of Ensemble Methods
-Bagging typically results in improved accuracy -over prediction using a single tree. Unfortunately, however, it can be -difficult to interpret the resulting model. Recall that one of the -advantages of decision trees is the attractive and easily interpreted -diagram that results. -
- -However, when we bag a large number of trees, it is no longer -possible to represent the resulting statistical learning procedure -using a single tree, and it is no longer clear which variables are -most important to the procedure. Thus, bagging improves prediction -accuracy at the expense of interpretability. Although the collection -of bagged trees is much more difficult to interpret than a single -tree, one can obtain an overall summary of the importance of each -predictor using the MSE (for bagging regression trees) or the Gini -index (for bagging classification trees). In the case of bagging -regression trees, we can record the total amount that the MSE is -decreased due to splits over a given predictor, averaged over all \( B \) possible -trees. A large value indicates an important predictor. Similarly, in -the context of bagging classification trees, we can add up the total -amount that the Gini index is decreased by splits over a given -predictor, averaged over all \( B \) trees. -
++
@@ -425,7 +414,7 @@ predictor, averaged over all \( B \) trees.
-
Simple Voting Example, head or tail
+Bagging
- -heads_proba = 0.51
-coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
-cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)
-plt.figure(figsize=(8,3.5))
-plt.plot(cumulative_heads_ratio)
-plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
-plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
-plt.xlabel("Number of coin tosses")
-plt.ylabel("Heads ratio")
-plt.legend(loc="lower right")
-plt.axis([0, 10000, 0.42, 0.58])
-save_fig("votingsimple")
-plt.show()
-
-The plain decision trees suffer from high +variance. This means that if we split the training data into two parts +at random, and fit a decision tree to both halves, the results that we +get could be quite different. In contrast, a procedure with low +variance will yield similar results if applied repeatedly to distinct +data sets; linear regression tends to have low variance, if the ratio +of \( n \) to \( p \) is moderately large. +
+Bootstrap aggregation, or just bagging, is a +general-purpose procedure for reducing the variance of a statistical +learning method. +
@@ -436,7 +422,7 @@ plt.show()
-
Using the Voting Classifier
+More bagging
- -from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
-
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-Bagging typically results in improved accuracy +over prediction using a single tree. Unfortunately, however, it can be +difficult to interpret the resulting model. Recall that one of the +advantages of decision trees is the attractive and easily interpreted +diagram that results. +
+However, when we bag a large number of trees, it is no longer +possible to represent the resulting statistical learning procedure +using a single tree, and it is no longer clear which variables are +most important to the procedure. Thus, bagging improves prediction +accuracy at the expense of interpretability. Although the collection +of bagged trees is much more difficult to interpret than a single +tree, one can obtain an overall summary of the importance of each +predictor using the MSE (for bagging regression trees) or the Gini +index (for bagging classification trees). In the case of bagging +regression trees, we can record the total amount that the MSE is +decreased due to splits over a given predictor, averaged over all \( B \) possible +trees. A large value indicates an important predictor. Similarly, in +the context of bagging classification trees, we can add up the total +amount that the Gini index is decreased by splits over a given +predictor, averaged over all \( B \) trees. +
@@ -466,7 +432,7 @@ voting_clf.fit(X_train, y_train)
-
Please, not the moons again! Voting and Bagging
- +Simple Voting Example, head or tail
from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
-
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
+ heads_proba = 0.51
+coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
+cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)
+plt.figure(figsize=(8,3.5))
+plt.plot(cumulative_heads_ratio)
+plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
+plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
+plt.xlabel("Number of coin tosses")
+plt.ylabel("Heads ratio")
+plt.legend(loc="lower right")
+plt.axis([0, 10000, 0.42, 0.58])
+save_fig("votingsimple")
+plt.show()
-
Bagging Examples
- +Using the Voting Classifier
from sklearn.ensemble import BaggingClassifier
-from sklearn.tree import DecisionTreeClassifier
+ from sklearn.model_selection import train_test_split
+from sklearn.datasets import make_moons
-bag_clf = BaggingClassifier(
- DecisionTreeClassifier(random_state=42), n_estimators=500,
- max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
-bag_clf.fit(X_train, y_train)
-y_pred = bag_clf.predict(X_test)
-
-from sklearn.metrics import accuracy_score
-print(accuracy_score(y_test, y_pred))
-
-tree_clf = DecisionTreeClassifier(random_state=42)
-tree_clf.fit(X_train, y_train)
-y_pred_tree = tree_clf.predict(X_test)
-print(accuracy_score(y_test, y_pred_tree))
-
-from matplotlib.colors import ListedColormap
+X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
+X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
- x1s = np.linspace(axes[0], axes[1], 100)
- x2s = np.linspace(axes[2], axes[3], 100)
- x1, x2 = np.meshgrid(x1s, x2s)
- X_new = np.c_[x1.ravel(), x2.ravel()]
- y_pred = clf.predict(X_new).reshape(x1.shape)
- custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
- plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
- if contour:
- custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
- plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
- plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
- plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
- plt.axis(axes)
- plt.xlabel(r"$x_1$", fontsize=18)
- plt.ylabel(r"$x_2$", fontsize=18, rotation=0)
-plt.figure(figsize=(11,4))
-plt.subplot(121)
-plot_decision_boundary(tree_clf, X, y)
-plt.title("Decision Tree", fontsize=14)
-plt.subplot(122)
-plot_decision_boundary(bag_clf, X, y)
-plt.title("Decision Trees with Bagging", fontsize=14)
-save_fig("baggingtree")
-plt.show()
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.ensemble import VotingClassifier
+from sklearn.linear_model import LogisticRegression
+from sklearn.svm import SVC
+
+log_clf = LogisticRegression(solver="liblinear", random_state=42)
+rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
+svm_clf = SVC(gamma="auto", random_state=42)
+
+voting_clf = VotingClassifier(
+ estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
+ voting='hard')
+
+voting_clf.fit(X_train, y_train)
+
+from sklearn.metrics import accuracy_score
+
+for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
+ clf.fit(X_train, y_train)
+ y_pred = clf.predict(X_test)
+ print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
+
+log_clf = LogisticRegression(solver="liblinear", random_state=42)
+rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
+svm_clf = SVC(gamma="auto", probability=True, random_state=42)
+
+voting_clf = VotingClassifier(
+ estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
+ voting='soft')
+voting_clf.fit(X_train, y_train)
+
+from sklearn.metrics import accuracy_score
+
+for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
+ clf.fit(X_train, y_train)
+ y_pred = clf.predict(X_test)
+ print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
Making your own Bootstrap: Changing the Level of the Decision Tree
+Please, not the moons again! Voting and Bagging
-Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with -a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points \( n \)). -
import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.pipeline import make_pipeline
-from sklearn.utils import resample
-from sklearn.tree import DecisionTreeRegressor
+ from sklearn.model_selection import train_test_split
+from sklearn.datasets import make_moons
-n = 100
-n_boostraps = 100
-maxdepth = 8
+X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
+X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.ensemble import VotingClassifier
+from sklearn.linear_model import LogisticRegression
+from sklearn.svm import SVC
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-error = np.zeros(maxdepth)
-bias = np.zeros(maxdepth)
-variance = np.zeros(maxdepth)
-polydegree = np.zeros(maxdepth)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+log_clf = LogisticRegression(random_state=42)
+rnd_clf = RandomForestClassifier(random_state=42)
+svm_clf = SVC(random_state=42)
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
+voting_clf = VotingClassifier(
+ estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
+ voting='hard')
+voting_clf.fit(X_train, y_train)
+
+from sklearn.metrics import accuracy_score
-# we produce a simple tree first as benchmark
-simpletree = DecisionTreeRegressor(max_depth=3)
-simpletree.fit(X_train_scaled, y_train)
-simpleprediction = simpletree.predict(X_test_scaled)
-for degree in range(1,maxdepth):
- model = DecisionTreeRegressor(max_depth=degree)
- y_pred = np.empty((y_test.shape[0], n_boostraps))
- for i in range(n_boostraps):
- x_, y_ = resample(X_train_scaled, y_train)
- model.fit(x_, y_)
- y_pred[:, i] = model.predict(X_test_scaled)#.ravel()
+for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
+ clf.fit(X_train, y_train)
+ y_pred = clf.predict(X_test)
+ print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
+
+log_clf = LogisticRegression(random_state=42)
+rnd_clf = RandomForestClassifier(random_state=42)
+svm_clf = SVC(probability=True, random_state=42)
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
- variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
- print('Polynomial degree:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-
-mse_simpletree= np.mean( np.mean((y_test - simpleprediction)**2)
-print(mse_simpletree)
-plt.xlim(1,maxdepth)
-plt.plot(polydegree, error, label='MSE')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
-save_fig("baggingboot")
-plt.show()
+voting_clf = VotingClassifier(
+ estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
+ voting='soft')
+voting_clf.fit(X_train, y_train)
+
+from sklearn.metrics import accuracy_score
+
+for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
+ clf.fit(X_train, y_train)
+ y_pred = clf.predict(X_test)
+ print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
Why Voting?
+Bagging Examples
-The idea behind boosting, and voting as well can be phrased as follows: -Can a group of people somehow arrive at highly -reasoned decisions, despite the weak judgement of the individual -members? -
-The aim is to create a good classifier by combining several weak classifiers. -A weak classifier is a classifier which is able to produce results that are only slightly better than guessing at random. -
+ +from sklearn.ensemble import BaggingClassifier
+from sklearn.tree import DecisionTreeClassifier
-The basic approach is to apply repeatedly (in boosting this is done in an iterative way) a weak classifier to modifications of the data.
-In voting we simply apply the law of large numbers while in boosting we give more weight to misclassified data in
-each iteration.
-
+bag_clf = BaggingClassifier(
+ DecisionTreeClassifier(random_state=42), n_estimators=500,
+ max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
+bag_clf.fit(X_train, y_train)
+y_pred = bag_clf.predict(X_test)
+
+from sklearn.metrics import accuracy_score
+print(accuracy_score(y_test, y_pred))
+
+tree_clf = DecisionTreeClassifier(random_state=42)
+tree_clf.fit(X_train, y_train)
+y_pred_tree = tree_clf.predict(X_test)
+print(accuracy_score(y_test, y_pred_tree))
+
+from matplotlib.colors import ListedColormap
+
+def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
+ x1s = np.linspace(axes[0], axes[1], 100)
+ x2s = np.linspace(axes[2], axes[3], 100)
+ x1, x2 = np.meshgrid(x1s, x2s)
+ X_new = np.c_[x1.ravel(), x2.ravel()]
+ y_pred = clf.predict(X_new).reshape(x1.shape)
+ custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
+ plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
+ if contour:
+ custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
+ plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
+ plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
+ plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
+ plt.axis(axes)
+ plt.xlabel(r"$x_1$", fontsize=18)
+ plt.ylabel(r"$x_2$", fontsize=18, rotation=0)
+plt.figure(figsize=(11,4))
+plt.subplot(121)
+plot_decision_boundary(tree_clf, X, y)
+plt.title("Decision Tree", fontsize=14)
+plt.subplot(122)
+plot_decision_boundary(bag_clf, X, y)
+plt.title("Decision Trees with Bagging", fontsize=14)
+save_fig("baggingtree")
+plt.show()
+
+Decision trees play an important role as our weak classifier. They serve as the basic method.
@@ -418,7 +529,7 @@ each iteration.
-
Tossing coins
+Making your own Bootstrap: Changing the Level of the Decision Tree
-The simplest case is a so-called voting ensemble. To illustrate this, -think of yourself tossing coins with a biased outcome of 51 per cent -for heads and 49% for tails. With only few tosses, -you may not clearly see this distribution for heads and tails. However, after some -thousands of tosses, there will be a clear majority of heads. With 2000 tosses -you should see approximately 1020 heads and 980 tails. +
Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with +a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points \( n \)).
-We can then state that the outcome is a clear majority of heads. If -you do this ten thousand times, it is easy to see that there is a 97% -likelihood of a majority of heads. -
+ +import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+from sklearn.tree import DecisionTreeRegressor
-Another example would be to collect all polls before an
-election. Different polls may show different likelihoods for a
-candidate winning with say a majority of the popular vote. The majority vote
-would then consist in many polls indicating that this candidate will
-actually win.
-
+n = 100
+n_boostraps = 100
+maxdepth = 8
+
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+error = np.zeros(maxdepth)
+bias = np.zeros(maxdepth)
+variance = np.zeros(maxdepth)
+polydegree = np.zeros(maxdepth)
+X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+
+# we produce a simple tree first as benchmark
+simpletree = DecisionTreeRegressor(max_depth=3)
+simpletree.fit(X_train_scaled, y_train)
+simpleprediction = simpletree.predict(X_test_scaled)
+for degree in range(1,maxdepth):
+ model = DecisionTreeRegressor(max_depth=degree)
+ y_pred = np.empty((y_test.shape[0], n_boostraps))
+ for i in range(n_boostraps):
+ x_, y_ = resample(X_train_scaled, y_train)
+ model.fit(x_, y_)
+ y_pred[:, i] = model.predict(X_test_scaled)#.ravel()
+
+ polydegree[degree] = degree
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+ variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+ print('Polynomial degree:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+
+mse_simpletree= np.mean( np.mean((y_test - simpleprediction)**2)
+print(mse_simpletree)
+plt.xlim(1,maxdepth)
+plt.plot(polydegree, error, label='MSE')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+save_fig("baggingboot")
+plt.show()
+
+The example here shows how we can implement the coin tossing case, -clealry demostrating that after some tosses we see the law of large -numbers kicking in. -
@@ -426,7 +491,7 @@ numbers kicking in.
-
Standard imports first
+Why Voting?
+The idea behind boosting, and voting as well can be phrased as follows: +Can a group of people somehow arrive at highly +reasoned decisions, despite the weak judgement of the individual +members? +
- -# Common imports
-from IPython.display import Image
-from pydot import graph_from_dot_data
-import pandas as pd
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.tree import DecisionTreeClassifier
-from sklearn.model_selection import train_test_split
-from sklearn.tree import export_graphviz
-from sklearn.preprocessing import StandardScaler, OneHotEncoder
-from sklearn.compose import ColumnTransformer
-from IPython.display import Image
-from pydot import graph_from_dot_data
-import os
+The aim is to create a good classifier by combining several weak classifiers.
+A weak classifier is a classifier which is able to produce results that are only slightly better than guessing at random.
+
-# Where to save the figures and data files
-PROJECT_ROOT_DIR = "Results"
-FIGURE_ID = "Results/FigureFiles"
-DATA_ID = "DataFiles/"
-
-if not os.path.exists(PROJECT_ROOT_DIR):
- os.mkdir(PROJECT_ROOT_DIR)
-
-if not os.path.exists(FIGURE_ID):
- os.makedirs(FIGURE_ID)
-
-if not os.path.exists(DATA_ID):
- os.makedirs(DATA_ID)
-
-def image_path(fig_id):
- return os.path.join(FIGURE_ID, fig_id)
-
-def data_path(dat_id):
- return os.path.join(DATA_ID, dat_id)
-
-def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
-
-The basic approach is to apply repeatedly (in boosting this is done in an iterative way) a weak classifier to modifications of the data. +In voting we simply apply the law of large numbers while in boosting we give more weight to misclassified data in +each iteration. +
+Decision trees play an important role as our weak classifier. They serve as the basic method.
@@ -461,7 +425,7 @@ DATA_ID = "
The simplest case is a so-called voting ensemble. To illustrate this,
+think of yourself tossing coins with a biased outcome of 51 per cent
+for heads and 49% for tails. With only few tosses,
+you may not clearly see this distribution for heads and tails. However, after some
+thousands of tosses, there will be a clear majority of heads. With 2000 tosses
+you should see approximately 1020 heads and 980 tails.
+ We can then state that the outcome is a clear majority of heads. If
+you do this ten thousand times, it is easy to see that there is a 97%
+likelihood of a majority of heads.
+ Another example would be to collect all polls before an
+election. Different polls may show different likelihoods for a
+candidate winning with say a majority of the popular vote. The majority vote
+would then consist in many polls indicating that this candidate will
+actually win.
+ The example here shows how we can implement the coin tossing case,
+clealry demostrating that after some tosses we see the law of large
+numbers kicking in.
+
@@ -445,7 +433,7 @@ plt.show()
We can use the voting classifier on other data sets, here the exciting binary case of two distinct objects using the make moons functionality of Scikit-Learn. Random forests provide an improvement over bagged trees by way of a
-small tweak that decorrelates the trees.
- We can use the voting classifier on other data sets, here the exciting binary case of two distinct objects using the make moons functionality of Scikit-Learn. As in bagging, we build a
-number of decision trees on bootstrapped training samples. But when
-building these decision trees, each time a split in a tree is
-considered, a random sample of \( m \) predictors is chosen as split
-candidates from the full set of \( p \) predictors. The split is allowed to
-use only one of those \( m \) predictors.
- A fresh sample of \( m \) predictors is
-taken at each split, and typically we choose
- In building a random forest, at
-each split in the tree, the algorithm is not even allowed to consider
-a majority of the available predictors.
- The reason for this is rather clever. Suppose that there is one very
-strong predictor in the data set, along with a number of other
-moderately strong predictors. Then in the collection of bagged
-variable importance random forest trees, most or all of the trees will
-use this strong predictor in the top split. Consequently, all of the
-bagged trees will look quite similar to each other. Hence the
-predictions from the bagged trees will be highly correlated.
-Unfortunately, averaging many highly correlated quantities does not
-lead to as large of a reduction in variance as averaging many
-uncorrelated quantities. In particular, this means that bagging will
-not lead to a substantial reduction in variance over a single tree in
-this setting.
-
@@ -440,7 +474,7 @@ this setting.
The algorithm described here can be applied to both classification and regression problems. We will grow of forest of say \( B \) trees.
Recall that the cumulative gains curve shows the percentage of the
-overall number of cases in a given category gained by targeting a
-percentage of the total number of cases.
+ Random forests provide an improvement over bagged trees by way of a
+small tweak that decorrelates the trees.
Similarly, the receiver operating characteristic curve, or ROC curve,
-displays the diagnostic ability of a binary classifier system as its
-discrimination threshold is varied. It plots the true positive rate against the false positive rate.
+ As in bagging, we build a
+number of decision trees on bootstrapped training samples. But when
+building these decision trees, each time a split in a tree is
+considered, a random sample of \( m \) predictors is chosen as split
+candidates from the full set of \( p \) predictors. The split is allowed to
+use only one of those \( m \) predictors.
+ A fresh sample of \( m \) predictors is
+taken at each split, and typically we choose
+ In building a random forest, at
+each split in the tree, the algorithm is not even allowed to consider
+a majority of the available predictors.
+ The reason for this is rather clever. Suppose that there is one very
+strong predictor in the data set, along with a number of other
+moderately strong predictors. Then in the collection of bagged
+variable importance random forest trees, most or all of the trees will
+use this strong predictor in the top split. Consequently, all of the
+bagged trees will look quite similar to each other. Hence the
+predictions from the bagged trees will be highly correlated.
+Unfortunately, averaging many highly correlated quantities does not
+lead to as large of a reduction in variance as averaging many
+uncorrelated quantities. In particular, this means that bagging will
+not lead to a substantial reduction in variance over a single tree in
+this setting.
@@ -498,7 +447,7 @@ discrimination threshold is varied. It plots the true positive rate against the
The algorithm described here can be applied to both classification and regression problems. We will grow of forest of say \( B \) trees.
The basic idea is to combine weak classifiers in order to create a good
-classifier. With a weak classifier we often intend a classifier which
-produces results which are only slightly better than we would get by
-random guesses.
+
+ Recall that the cumulative gains curve shows the percentage of the
+overall number of cases in a given category gained by targeting a
+percentage of the total number of cases.
This is done by applying in an iterative way a weak (or a standard
-classifier like decision trees) to modify the data. In each iteration
-we emphasize those observations which are misclassified by weighting
-them with a factor.
+ Similarly, the receiver operating characteristic curve, or ROC curve,
+displays the diagnostic ability of a binary classifier system as its
+discrimination threshold is varied. It plots the true positive rate against the false positive rate.
@@ -413,7 +505,7 @@ them with a factor.
Boosting is a way of fitting an additive expansion in a set of
-elementary basis functions like for example some simple polynomials.
-Assume for example that we have a function
- where \( \beta_m \) are the expansion parameters to be determined in a
-minimization process and \( b(x;\gamma_m) \) are some simple functions of
-the multivariable parameter \( x \) which is characterized by the
-parameters \( \gamma_m \).
- As an example, consider the Sigmoid function we used in logistic
-regression. In that case, we can translate the function
-\( b(x;\gamma_m) \) into the Sigmoid function
- where \( t=\gamma_0+\gamma_1 x \) and the parameters \( \gamma_0 \) and
-\( \gamma_1 \) were determined by the Logistic Regression fitting
-algorithm.
- As another example, consider the cost function we defined for linear regression In this case the function \( f(x) \) was replaced by the design matrix
-\( \boldsymbol{X} \) and the unknown linear regression parameters \( \boldsymbol{\beta} \),
-that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we can
-simply invert a matrix and obtain the parameters \( \beta \) by
- In iterative fitting or additive modeling, we minimize the cost function with respect to the parameters \( \beta_m \) and \( \gamma_m \).
@@ -446,7 +459,7 @@ $$
The way we proceed is as follows (here we specialize to the squared-error cost function) The basic idea is to combine weak classifiers in order to create a good
+classifier. With a weak classifier we often intend a classifier which
+produces results which are only slightly better than we would get by
+random guesses.
+ We could use any of the algorithms we have discussed till now. If we
-use trees, \( \gamma \) parameterizes the split variables and split points
-at the internal nodes, and the predictions at the terminal nodes.
+ This is done by applying in an iterative way a weak (or a standard
+classifier like decision trees) to modify the data. In each iteration
+we emphasize those observations which are misclassified by weighting
+them with a factor.
@@ -418,7 +420,7 @@ at the internal nodes, and the predictions at the terminal nodes.
To better understand what happens, let us develop the steps for the iterative fitting using the above squared error function. For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \). This means that for every iteration \( m \), we need to optimize We start our iteration by simply setting \( f_0(x)=0 \).
-Taking the derivatives with respect to \( \beta \) and \( \gamma \) we obtain
+ Boosting is a way of fitting an additive expansion in a set of
+elementary basis functions like for example some simple polynomials.
+Assume for example that we have a function
and We can then rewrite these equations as (defining \( \boldsymbol{w}=\boldsymbol{e}+\gamma \boldsymbol{x}) \) with \( \boldsymbol{e} \) being the unit vector) which gives us \( \beta = \boldsymbol{w}^T\boldsymbol{y}/(\boldsymbol{w}^T\boldsymbol{w}) \). Similarly we have which leads to \( \gamma =(\boldsymbol{x}^T\boldsymbol{y}-\beta\boldsymbol{x}^T\boldsymbol{e})/(\beta\boldsymbol{x}^T\boldsymbol{x}) \). Inserting
-for \( \beta \) gives us an equation for \( \gamma \). This is a non-linear equation in the unknown \( \gamma \) and has to be solved numerically.
+ where \( \beta_m \) are the expansion parameters to be determined in a
+minimization process and \( b(x;\gamma_m) \) are some simple functions of
+the multivariable parameter \( x \) which is characterized by the
+parameters \( \gamma_m \).
The solution to these two equations gives us in turn \( \beta_1 \) and \( \gamma_1 \) leading to the new expression for \( f_1(x) \) as
-\( f_1(x) = \beta_1(1+\gamma_1x) \). Doing this \( M \) times results in our final estimate for the function \( f \).
+ As an example, consider the Sigmoid function we used in logistic
+regression. In that case, we can translate the function
+\( b(x;\gamma_m) \) into the Sigmoid function
where \( t=\gamma_0+\gamma_1 x \) and the parameters \( \gamma_0 \) and
+\( \gamma_1 \) were determined by the Logistic Regression fitting
+algorithm.
+ As another example, consider the cost function we defined for linear regression In this case the function \( f(x) \) was replaced by the design matrix
+\( \boldsymbol{X} \) and the unknown linear regression parameters \( \boldsymbol{\beta} \),
+that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we can
+simply invert a matrix and obtain the parameters \( \beta \) by
+ In iterative fitting or additive modeling, we minimize the cost function with respect to the parameters \( \beta_m \) and \( \gamma_m \).
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
-observations. We define a classification function \( G(x) \) which produces a prediction taking one or the other of the two values
-\( \{-1,1\} \).
+ The way we proceed is as follows (here we specialize to the squared-error cost function) We could use any of the algorithms we have discussed till now. If we
+use trees, \( \gamma \) parameterizes the split variables and split points
+at the internal nodes, and the predictions at the terminal nodes.
The error rate of the training sample is then The iterative procedure starts with defining a weak classifier whose
-error rate is barely better than random guessing. The iterative
-procedure in boosting is to sequentially apply a weak
-classification algorithm to repeatedly modified versions of the data
-producing a sequence of weak classifiers \( G_m(x) \).
- Here we will express our function \( f(x) \) in terms of \( G(x) \). That is will be a function of
To better understand what happens, let us develop the steps for the iterative fitting using the above squared error function. For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \). This means that for every iteration \( m \), we need to optimize In our iterative procedure we define thus The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the
-exponential cost/loss function defined as
+ We start our iteration by simply setting \( f_0(x)=0 \).
+Taking the derivatives with respect to \( \beta \) and \( \gamma \) we obtain
We optimize \( \beta \) and \( G \) for each value of \( m=1:M \) as we did in the regression case.
-This is normally done in two steps. Let us however first rewrite the cost function as
+ and We can then rewrite these equations as (defining \( \boldsymbol{w}=\boldsymbol{e}+\gamma \boldsymbol{x}) \) with \( \boldsymbol{e} \) being the unit vector) which gives us \( \beta = \boldsymbol{w}^T\boldsymbol{y}/(\boldsymbol{w}^T\boldsymbol{w}) \). Similarly we have which leads to \( \gamma =(\boldsymbol{x}^T\boldsymbol{y}-\beta\boldsymbol{x}^T\boldsymbol{e})/(\beta\boldsymbol{x}^T\boldsymbol{x}) \). Inserting
+for \( \beta \) gives us an equation for \( \gamma \). This is a non-linear equation in the unknown \( \gamma \) and has to be solved numerically.
where we have defined \( w_i^m= \exp{(-y_if_{m-1}(x_i))} \). The solution to these two equations gives us in turn \( \beta_1 \) and \( \gamma_1 \) leading to the new expression for \( f_1(x) \) as
+\( f_1(x) = \beta_1(1+\gamma_1x) \). Doing this \( M \) times results in our final estimate for the function \( f \).
+
@@ -423,7 +448,7 @@ $$
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
+observations. We define a classification function \( G(x) \) which produces a prediction taking one or the other of the two values
+\( \{-1,1\} \).
+ The error rate of the training sample is then First, for any \( \beta > 0 \), we optimize \( G \) by setting which is the classifier that minimizes the weighted error rate in predicting \( y \). The iterative procedure starts with defining a weak classifier whose
+error rate is barely better than random guessing. The iterative
+procedure in boosting is to sequentially apply a weak
+classification algorithm to repeatedly modified versions of the data
+producing a sequence of weak classifiers \( G_m(x) \).
+ We can do this by rewriting Here we will express our function \( f(x) \) in terms of \( G(x) \). That is which can be rewritten as will be a function of which leads to where we have redefined the error as which leads to an update of This leads to the new weights The algorithm here is rather straightforward. Assume that our weak
-classifier is a decision tree and we consider a binary set of outputs
-with \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
-observations. Our design matrix is given in terms of the
-feature/predictor vectors
-\( \boldsymbol{X}=[\boldsymbol{x}_0\boldsymbol{x}_1\dots\boldsymbol{x}_{p-1}] \). Finally, we define also a
-classifier determined by our data via a function \( G(x) \). This function tells us how well we are able to classify our outputs/targets \( \boldsymbol{y} \).
+ In our iterative procedure we define thus The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the
+exponential cost/loss function defined as
+ We optimize \( \beta \) and \( G \) for each value of \( m=1:M \) as we did in the regression case.
+This is normally done in two steps. Let us however first rewrite the cost function as
We have already defined the misclassification error \( \mathrm{err} \) as where the function \( I() \) is one if we misclassify and zero if we classify correctly. where we have defined \( w_i^m= \exp{(-y_if_{m-1}(x_i))} \).
@@ -417,7 +430,7 @@ $$
With the above definitions we are now ready to set up the algorithm for AdaBoost.
-The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.
- First, for any \( \beta > 0 \), we optimize \( G \) by setting which is the classifier that minimizes the weighted error rate in predicting \( y \). We can do this by rewriting which can be rewritten as which leads to where we have redefined the error as which leads to an update of This leads to the new weights For the iterations with \( m \le 2 \) the weights are modified
-individually at each steps. The observations which were misclassified
-at iteration \( m-1 \) have a weight which is larger than those which were
-classified properly. As this proceeds, the observations which were
-difficult to classifiy correctly are given a larger influence. Each
-new classification step \( m \) is then forced to concentrate on those
-observations that are missed in the previous iterations.
-
@@ -431,7 +446,7 @@ observations that are missed in the previous iterations.
Using Scikit-Learn it is easy to apply the adaptive boosting algorithm, as done here. The algorithm here is rather straightforward. Assume that our weak
+classifier is a decision tree and we consider a binary set of outputs
+with \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
+observations. Our design matrix is given in terms of the
+feature/predictor vectors
+\( \boldsymbol{X}=[\boldsymbol{x}_0\boldsymbol{x}_1\dots\boldsymbol{x}_{p-1}] \). Finally, we define also a
+classifier determined by our data via a function \( G(x) \). This function tells us how well we are able to classify our outputs/targets \( \boldsymbol{y} \).
+ We have already defined the misclassification error \( \mathrm{err} \) as where the function \( I() \) is one if we misclassify and zero if we classify correctly.
@@ -446,6 +423,8 @@ plt.show()
Gradient boosting is again a similar technique to Adaptive boosting,
-it combines so-called weak classifiers or regressors into a strong
-method via a series of iterations.
+ With the above definitions we are now ready to set up the algorithm for AdaBoost.
+The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.
In order to understand the method, let us illustrate its basics by
-bringing back the essential steps in linear regression, where our cost
-function was the least squares function.
+ For the iterations with \( m \le 2 \) the weights are modified
+individually at each steps. The observations which were misclassified
+at iteration \( m-1 \) have a weight which is larger than those which were
+classified properly. As this proceeds, the observations which were
+difficult to classifiy correctly are given a larger influence. Each
+new classification step \( m \) is then forced to concentrate on those
+observations that are missed in the previous iterations.
@@ -409,6 +436,9 @@ function was the least squares function.
We start again with our cost function \( {\cal C}(\boldsymbol{y}m\boldsymbol{f})=\sum_{i=0}^{n-1}{\cal L}(y_i, f(x_i)) \) where we want to minimize
-This means that for every iteration, we need to optimize
- Using Scikit-Learn it is easy to apply the adaptive boosting algorithm, as done here. We define a real function \( h_m(x) \) that defines our final function \( f_M(x) \) as In the steepest decent approach we approximate \( h_m(x) = -\rho_m g_m(x) \), where \( \rho_m \) is a scalar and \( g_m(x) \) the gradient defined as With the new gradient we can update \( f_m(x) = f_{m-1}(x) -\rho_m g_m(x) \). Using the above squared-error function we see that
-the gradient is \( g_m(x_i) = -2(y_i-f(x_i)) \).
- Choosing \( f_0(x)=0 \) we obtain \( g_m(x) = -2y_i \) and inserting this into the minimization problem for the cost function we have
@@ -426,6 +451,8 @@ $$
Optimizing with respect to \( \rho \) we obtain (taking the derivative) that \( \rho_1 = -1/2 \). We have then that Gradient boosting is again a similar technique to Adaptive boosting,
+it combines so-called weak classifiers or regressors into a strong
+method via a series of iterations.
+ We can then proceed and compute and find a new value for \( \rho_2=-1/2 \) and continue till we have reached \( m=M \). We can modify the steepest descent method, or steepest boosting, by introducing what is called gradient boosting. In order to understand the method, let us illustrate its basics by
+bringing back the essential steps in linear regression, where our cost
+function was the least squares function.
+
@@ -409,6 +414,8 @@ $$
Steepest descent is however not much used, since it only optimizes \( f \) at a fixed set of \( n \) points,
-so we do not learn a function that can generalize. However, we can modify the algorithm by
-fitting a weak learner to approximate the negative gradient signal.
+ We start again with our cost function \( {\cal C}(\boldsymbol{y}m\boldsymbol{f})=\sum_{i=0}^{n-1}{\cal L}(y_i, f(x_i)) \) where we want to minimize
+This means that for every iteration, we need to optimize
Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function The way we proceed in an iterative fashion is to We define a real function \( h_m(x) \) that defines our final function \( f_M(x) \) as In the steepest decent approach we approximate \( h_m(x) = -\rho_m g_m(x) \), where \( \rho_m \) is a scalar and \( g_m(x) \) the gradient defined as With the new gradient we can update \( f_m(x) = f_{m-1}(x) -\rho_m g_m(x) \). Using the above squared-error function we see that
+the gradient is \( g_m(x_i) = -2(y_i-f(x_i)) \).
+ Choosing \( f_0(x)=0 \) we obtain \( g_m(x) = -2y_i \) and inserting this into the minimization problem for the cost function we have
Optimizing with respect to \( \rho \) we obtain (taking the derivative) that \( \rho_1 = -1/2 \). We have then that We can then proceed and compute and find a new value for \( \rho_2=-1/2 \) and continue till we have reached \( m=M \). We can modify the steepest descent method, or steepest boosting, by introducing what is called gradient boosting.
@@ -463,6 +414,8 @@ plt.show()
Steepest descent is however not much used, since it only optimizes \( f \) at a fixed set of \( n \) points,
+so we do not learn a function that can generalize. However, we can modify the algorithm by
+fitting a weak learner to approximate the negative gradient signal.
+ Suppose we have a cost function \( C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i)) \) where \( y_i \) is our target and \( f(x_i) \) the function which is meant to model \( y_i \). The above cost function could be our standard squared-error function The way we proceed in an iterative fashion is to
XGBoost or Extreme Gradient
-Boosting, is an optimized distributed gradient boosting library
-designed to be highly efficient, flexible and portable. It implements
-machine learning algorithms under the Gradient Boosting
-framework. XGBoost provides a parallel tree boosting that solve many
-data science problems in a fast and accurate way. See the article by Chen and Guestrin.
- The authors design and build a highly scalable end-to-end tree
-boosting system. It has a theoretically justified weighted quantile
-sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning.
- It is now the algorithm which wins essentially all ML competitions!!!
@@ -408,6 +468,8 @@ sketch for efficient proposal calculation. It introduces a novel sparsity-aware
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now. XGBoost or Extreme Gradient
+Boosting, is an optimized distributed gradient boosting library
+designed to be highly efficient, flexible and portable. It implements
+machine learning algorithms under the Gradient Boosting
+framework. XGBoost provides a parallel tree boosting that solve many
+data science problems in a fast and accurate way. See the article by Chen and Guestrin.
+ The authors design and build a highly scalable end-to-end tree
+boosting system. It has a theoretically justified weighted quantile
+sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning.
+ It is now the algorithm which wins essentially all ML competitions!!!
@@ -466,6 +413,9 @@ plt.show()
The Gini index \( g \) gives us the degree of probability of a specific
+variable that is wrongly classified.
+ It takes values \( g \in [0,1] \),
+ It favors binary splitting. It is custom to split to a tree uising binary splits. The reason is
+that multiway splits fragment the data too quickly, leaving
+insufficient data at the next level down. Multiway splits can be
+achieved by a series of binary split and this is normally preferred.
+ The Gini index \( g \) gives us the degree of probability of a specific
+variable that is wrongly classified.
+ It takes values \( g \in [0,1] \), It favors binary splitting. It is custom to split to a tree uising binary splits. The reason is
+that multiway splits fragment the data too quickly, leaving
+insufficient data at the next level down. Multiway splits can be
+achieved by a series of binary split and this is normally preferred.
+ The Gini index \( g \) gives us the degree of probability of a specific
+variable that is wrongly classified.
+ It takes values \( g \in [0,1] \), It favors binary splitting. It is custom to split to a tree uising binary splits. The reason is
+that multiway splits fragment the data too quickly, leaving
+insufficient data at the next level down. Multiway splits can be
+achieved by a series of binary split and this is normally preferred.
+Simple Voting Example, head or tail
+Tossing coins
-
-# Common imports
-import numpy as np
-import matplotlib
-import matplotlib.pyplot as plt
-from matplotlib.colors import ListedColormap
-plt.rcParams['axes.labelsize'] = 14
-plt.rcParams['xtick.labelsize'] = 12
-plt.rcParams['ytick.labelsize'] = 12
+
-Using the Voting Classifier
+Standard imports first
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
+
# Common imports
+from IPython.display import Image
+from pydot import graph_from_dot_data
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.model_selection import train_test_split
+from sklearn.tree import export_graphviz
+from sklearn.preprocessing import StandardScaler, OneHotEncoder
+from sklearn.compose import ColumnTransformer
+from IPython.display import Image
+from pydot import graph_from_dot_data
+import os
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
+if not os.path.exists(PROJECT_ROOT_DIR):
+ os.mkdir(PROJECT_ROOT_DIR)
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", random_state=42)
+if not os.path.exists(FIGURE_ID):
+ os.makedirs(FIGURE_ID)
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
+if not os.path.exists(DATA_ID):
+ os.makedirs(DATA_ID)
-voting_clf.fit(X_train, y_train)
+def image_path(fig_id):
+ return os.path.join(FIGURE_ID, fig_id)
-from sklearn.metrics import accuracy_score
+def data_path(dat_id):
+ return os.path.join(DATA_ID, dat_id)
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", probability=True, random_state=42)
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
+def save_fig(fig_id):
+ plt.savefig(image_path(fig_id) + ".png", format='png')
Voting and Bagging
-
+Simple Voting Example, head or tail
from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
+
# Common imports
+import numpy as np
+import matplotlib
+import matplotlib.pyplot as plt
+from matplotlib.colors import ListedColormap
+plt.rcParams['axes.labelsize'] = 14
+plt.rcParams['xtick.labelsize'] = 12
+plt.rcParams['ytick.labelsize'] = 12
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
+heads_proba = 0.51
+coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
+cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)
+plt.figure(figsize=(8,3.5))
+plt.plot(cumulative_heads_ratio)
+plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
+plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
+plt.xlabel("Number of coin tosses")
+plt.ylabel("Heads ratio")
+plt.legend(loc="lower right")
+plt.axis([0, 10000, 0.42, 0.58])
+save_fig("votingsimple")
+plt.show()
Random forests
+Using the Voting Classifier
-from sklearn.model_selection import train_test_split
+from sklearn.datasets import make_moons
-
+Random Forest Algorithm
-Voting and Bagging
+
+
+
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import make_moons
+
+X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
+X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.ensemble import VotingClassifier
+from sklearn.linear_model import LogisticRegression
+from sklearn.svm import SVC
+
+log_clf = LogisticRegression(random_state=42)
+rnd_clf = RandomForestClassifier(random_state=42)
+svm_clf = SVC(random_state=42)
+
+voting_clf = VotingClassifier(
+ estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
+ voting='hard')
+voting_clf.fit(X_train, y_train)
+
+from sklearn.metrics import accuracy_score
+
+for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
+ clf.fit(X_train, y_train)
+ y_pred = clf.predict(X_test)
+ print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
+
+log_clf = LogisticRegression(random_state=42)
+rnd_clf = RandomForestClassifier(random_state=42)
+svm_clf = SVC(probability=True, random_state=42)
+
+voting_clf = VotingClassifier(
+ estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
+ voting='soft')
+voting_clf.fit(X_train, y_train)
+
+from sklearn.metrics import accuracy_score
+
+for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
+ clf.fit(X_train, y_train)
+ y_pred = clf.predict(X_test)
+ print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
+
+
-
-
-
-
-Random Forests Compared with other Methods on the Cancer Data
+Random forests
-
-import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import load_breast_cancer
-from sklearn.svm import SVC
-from sklearn.linear_model import LogisticRegression
-from sklearn.tree import DecisionTreeClassifier
-from sklearn.ensemble import BaggingClassifier
-
-# Load the data
-cancer = load_breast_cancer()
-
-X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
-print(X_train.shape)
-print(X_test.shape)
-# Logistic Regression
-logreg = LogisticRegression(solver='lbfgs')
-logreg.fit(X_train, y_train)
-print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
-# Support vector machine
-svm = SVC(gamma='auto', C=100)
-svm.fit(X_train, y_train)
-print("Test set accuracy with SVM: {:.2f}".format(svm.score(X_test,y_test)))
-# Decision Trees
-deep_tree_clf = DecisionTreeClassifier(max_depth=None)
-deep_tree_clf.fit(X_train, y_train)
-print("Test set accuracy with Decision Trees: {:.2f}".format(deep_tree_clf.score(X_test,y_test)))
-#now scale the data
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-# Logistic Regression
-logreg.fit(X_train_scaled, y_train)
-print("Test set accuracy Logistic Regression with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
-# Support Vector Machine
-svm.fit(X_train_scaled, y_train)
-print("Test set accuracy SVM with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
-# Decision Trees
-deep_tree_clf.fit(X_train_scaled, y_train)
-print("Test set accuracy with Decision Trees and scaled data: {:.2f}".format(deep_tree_clf.score(X_test_scaled,y_test)))
-
-
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.preprocessing import LabelEncoder
-from sklearn.model_selection import cross_validate
-# Data set not specificied
-#Instantiate the model with 500 trees and entropy as splitting criteria
-Random_Forest_model = RandomForestClassifier(n_estimators=500,criterion="entropy")
-Random_Forest_model.fit(X_train_scaled, y_train)
-#Cross validation
-accuracy = cross_validate(Random_Forest_model,X_test_scaled,y_test,cv=10)['test_score']
-print(accuracy)
-print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(Random_Forest_model.score(X_test_scaled,y_test)))
-
-
-import scikitplot as skplt
-y_pred = Random_Forest_model.predict(X_test_scaled)
-skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
-plt.show()
-y_probas = Random_Forest_model.predict_proba(X_test_scaled)
-skplt.metrics.plot_roc(y_test, y_probas)
-plt.show()
-skplt.metrics.plot_cumulative_gain(y_test, y_probas)
-plt.show()
-
-Compare Bagging on Trees with Random Forests
-
-
-bag_clf = BaggingClassifier(
- DecisionTreeClassifier(splitter="random", max_leaf_nodes=16, random_state=42),
- n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1, random_state=42)
-
-bag_clf.fit(X_train, y_train)
-y_pred = bag_clf.predict(X_test)
-from sklearn.ensemble import RandomForestClassifier
-rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16, n_jobs=-1, random_state=42)
-rnd_clf.fit(X_train, y_train)
-y_pred_rf = rnd_clf.predict(X_test)
-np.sum(y_pred == y_pred_rf) / len(y_pred)
-
-Random Forest Algorithm
+
+
+
+
+
+Boosting, a Bird's Eye View
+Random Forests Compared with other Methods on the Cancer Data
-import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import load_breast_cancer
+from sklearn.svm import SVC
+from sklearn.linear_model import LogisticRegression
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.ensemble import BaggingClassifier
+
+# Load the data
+cancer = load_breast_cancer()
+
+X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
+print(X_train.shape)
+print(X_test.shape)
+# Logistic Regression
+logreg = LogisticRegression(solver='lbfgs')
+logreg.fit(X_train, y_train)
+print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
+# Support vector machine
+svm = SVC(gamma='auto', C=100)
+svm.fit(X_train, y_train)
+print("Test set accuracy with SVM: {:.2f}".format(svm.score(X_test,y_test)))
+# Decision Trees
+deep_tree_clf = DecisionTreeClassifier(max_depth=None)
+deep_tree_clf.fit(X_train, y_train)
+print("Test set accuracy with Decision Trees: {:.2f}".format(deep_tree_clf.score(X_test,y_test)))
+#now scale the data
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+# Logistic Regression
+logreg.fit(X_train_scaled, y_train)
+print("Test set accuracy Logistic Regression with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
+# Support Vector Machine
+svm.fit(X_train_scaled, y_train)
+print("Test set accuracy SVM with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
+# Decision Trees
+deep_tree_clf.fit(X_train_scaled, y_train)
+print("Test set accuracy with Decision Trees and scaled data: {:.2f}".format(deep_tree_clf.score(X_test_scaled,y_test)))
+
+
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.preprocessing import LabelEncoder
+from sklearn.model_selection import cross_validate
+# Data set not specificied
+#Instantiate the model with 500 trees and entropy as splitting criteria
+Random_Forest_model = RandomForestClassifier(n_estimators=500,criterion="entropy")
+Random_Forest_model.fit(X_train_scaled, y_train)
+#Cross validation
+accuracy = cross_validate(Random_Forest_model,X_test_scaled,y_test,cv=10)['test_score']
+print(accuracy)
+print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(Random_Forest_model.score(X_test_scaled,y_test)))
+
+
+import scikitplot as skplt
+y_pred = Random_Forest_model.predict(X_test_scaled)
+skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
+plt.show()
+y_probas = Random_Forest_model.predict_proba(X_test_scaled)
+skplt.metrics.plot_roc(y_test, y_probas)
+plt.show()
+skplt.metrics.plot_cumulative_gain(y_test, y_probas)
+plt.show()
+
+What is boosting? Additive Modelling/Iterative Fitting
+Compare Bagging on Trees with Random Forests
-bag_clf = BaggingClassifier(
+ DecisionTreeClassifier(splitter="random", max_leaf_nodes=16, random_state=42),
+ n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1, random_state=42)
+
+bag_clf.fit(X_train, y_train)
+y_pred = bag_clf.predict(X_test)
+from sklearn.ensemble import RandomForestClassifier
+rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16, n_jobs=-1, random_state=42)
+rnd_clf.fit(X_train, y_train)
+y_pred_rf = rnd_clf.predict(X_test)
+np.sum(y_pred == y_pred_rf) / len(y_pred)
+
+Iterative Fitting, Regression and Squared-error Cost Function
+Boosting, a Bird's Eye View
-
-
-
-Squared-Error Example and Iterative Fitting
+What is boosting? Additive Modelling/Iterative Fitting
-
@@ -441,7 +453,7 @@ for \( \beta \) gives us an equation for \( \gamma \). This is a non-linear equa
diff --git a/doc/pub/week45/html/._week45-bs060.html b/doc/pub/week45/html/._week45-bs060.html
index 50b3550ab..04bdee513 100644
--- a/doc/pub/week45/html/._week45-bs060.html
+++ b/doc/pub/week45/html/._week45-bs060.html
@@ -85,6 +85,11 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
2,
None,
'classification-tree-how-to-split-nodes'),
+ ('Gini Index?Coefficient/Impurity',
+ 2,
+ None,
+ 'gini-index-coefficient-impurity'),
+ ('Why binary split?', 2, None, 'why-binary-split'),
('Visualizing the Tree, Classification',
2,
None,
@@ -307,62 +312,64 @@ MathJax.Hub.Config({
Iterative Fitting, Classification and AdaBoost
+Iterative Fitting, Regression and Squared-error Cost Function
-
+
+
+Adaptive Boosting, AdaBoost
+Squared-Error Example and Iterative Fitting
+
+Building up AdaBoost
+Iterative Fitting, Classification and AdaBoost
+
+Adaptive boosting: AdaBoost, Basic Algorithm
+Adaptive Boosting, AdaBoost
-Basic Steps of AdaBoost
+Building up AdaBoost
-
-
+
-
-AdaBoost Examples
+Adaptive boosting: AdaBoost, Basic Algorithm
-from sklearn.ensemble import AdaBoostClassifier
-
-ada_clf = AdaBoostClassifier(
- DecisionTreeClassifier(max_depth=1), n_estimators=200,
- algorithm="SAMME.R", learning_rate=0.5, random_state=42)
-ada_clf.fit(X_train, y_train)
-
-from sklearn.ensemble import AdaBoostClassifier
-
-ada_clf = AdaBoostClassifier(
- DecisionTreeClassifier(max_depth=1), n_estimators=200,
- algorithm="SAMME.R", learning_rate=0.5, random_state=42)
-ada_clf.fit(X_train_scaled, y_train)
-y_pred = ada_clf.predict(X_test_scaled)
-skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
-plt.show()
-y_probas = ada_clf.predict_proba(X_test_scaled)
-skplt.metrics.plot_roc(y_test, y_probas)
-plt.show()
-skplt.metrics.plot_cumulative_gain(y_test, y_probas)
-plt.show()
-
-Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent
+Basic Steps of AdaBoost
-
+
+$$
+\mathrm{\overline{err}}_m=\frac{\sum_{i=0}^{n-1}w_i^m I(y_i\ne G(x_i))}{\sum_{i=0}^{n-1}w_i},
+$$
-
+
+The Squared-Error again! Steepest Descent
+AdaBoost Examples
-from sklearn.ensemble import AdaBoostClassifier
-
+Steepest Descent Example
+Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent
-Gradient Boosting, algorithm
+The Squared-Error again! Steepest Descent
-
-
-Gradient Boosting, Examples of Regression
+Steepest Descent Example
-
-import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.ensemble import GradientBoostingRegressor
-from sklearn.preprocessing import StandardScaler
-import scikitplot as skplt
-from sklearn.metrics import mean_squared_error
+
-Gradient Boosting, Classification Example
+Gradient Boosting, algorithm
-
-import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import load_breast_cancer
-import scikitplot as skplt
-from sklearn.ensemble import GradientBoostingClassifier
-from sklearn.model_selection import cross_validate
-
-# Load the data
-cancer = load_breast_cancer()
-
-X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
-print(X_train.shape)
-print(X_test.shape)
-#now scale the data
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-gd_clf = GradientBoostingClassifier(max_depth=3, n_estimators=100, learning_rate=1.0)
-gd_clf.fit(X_train_scaled, y_train)
-#Cross validation
-accuracy = cross_validate(gd_clf,X_test_scaled,y_test,cv=10)['test_score']
-print(accuracy)
-print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(gd_clf.score(X_test_scaled,y_test)))
-
-import scikitplot as skplt
-y_pred = gd_clf.predict(X_test_scaled)
-skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
-save_fig("gdclassiffierconfusion")
-plt.show()
-y_probas = gd_clf.predict_proba(X_test_scaled)
-skplt.metrics.plot_roc(y_test, y_probas)
-save_fig("gdclassiffierroc")
-plt.show()
-skplt.metrics.plot_cumulative_gain(y_test, y_probas)
-save_fig("gdclassiffiercgain")
-plt.show()
-
-
+
+XGBoost: Extreme Gradient Boosting
+Gradient Boosting, Examples of Regression
-import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.ensemble import GradientBoostingRegressor
+from sklearn.preprocessing import StandardScaler
+import scikitplot as skplt
+from sklearn.metrics import mean_squared_error
-
+Regression Case
-
+Gradient Boosting, Classification Example
import matplotlib.pyplot as plt
import numpy as np
-from sklearn.model_selection import train_test_split
-import xgboost as xgb
-from sklearn.preprocessing import StandardScaler
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import load_breast_cancer
import scikitplot as skplt
-from sklearn.metrics import mean_squared_error
+from sklearn.ensemble import GradientBoostingClassifier
+from sklearn.model_selection import cross_validate
-n = 100
-maxdegree = 6
+# Load the data
+cancer = load_breast_cancer()
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-
-error = np.zeros(maxdegree)
-bias = np.zeros(maxdegree)
-variance = np.zeros(maxdegree)
-polydegree = np.zeros(maxdegree)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
+print(X_train.shape)
+print(X_test.shape)
+#now scale the data
+from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
-for degree in range(maxdegree):
- model = xgb.XGBRegressor(objective ='reg:squarederror', colsaobjective ='reg:squarederror', colsample_bytree = 0.3, learning_rate = 0.1,max_depth = degree, alpha = 10, n_estimators = 200)
+gd_clf = GradientBoostingClassifier(max_depth=3, n_estimators=100, learning_rate=1.0)
+gd_clf.fit(X_train_scaled, y_train)
+#Cross validation
+accuracy = cross_validate(gd_clf,X_test_scaled,y_test,cv=10)['test_score']
+print(accuracy)
+print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(gd_clf.score(X_test_scaled,y_test)))
- model.fit(X_train_scaled,y_train)
- y_pred = model.predict(X_test_scaled)
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
- variance[degree] = np.mean( np.var(y_pred) )
- print('Max depth:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-
-plt.xlim(1,maxdegree-1)
-plt.plot(polydegree, error, label='Error')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
+import scikitplot as skplt
+y_pred = gd_clf.predict(X_test_scaled)
+skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
+save_fig("gdclassiffierconfusion")
+plt.show()
+y_probas = gd_clf.predict_proba(X_test_scaled)
+skplt.metrics.plot_roc(y_test, y_probas)
+save_fig("gdclassiffierroc")
+plt.show()
+skplt.metrics.plot_cumulative_gain(y_test, y_probas)
+save_fig("gdclassiffiercgain")
plt.show()
Xgboost on the Cancer Data
+XGBoost: Extreme Gradient Boosting
-import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import load_breast_cancer
-from sklearn.preprocessing import LabelEncoder
-from sklearn.model_selection import cross_validate
-import scikitplot as skplt
-import xgboost as xgb
-# Load the data
-cancer = load_breast_cancer()
-
-X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
-print(X_train.shape)
-print(X_test.shape)
-#now scale the data
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-xg_clf = xgb.XGBClassifier()
-xg_clf.fit(X_train_scaled,y_train)
-
-y_test = xg_clf.predict(X_test_scaled)
-
-print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(xg_clf.score(X_test_scaled,y_test)))
-
-import scikitplot as skplt
-y_pred = xg_clf.predict(X_test_scaled)
-skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
-save_fig("xdclassiffierconfusion")
-plt.show()
-y_probas = xg_clf.predict_proba(X_test_scaled)
-skplt.metrics.plot_roc(y_test, y_probas)
-save_fig("xdclassiffierroc")
-plt.show()
-skplt.metrics.plot_cumulative_gain(y_test, y_probas)
-save_fig("gdclassiffiercgain")
-plt.show()
-
-
-xgb.plot_tree(xg_clf,num_trees=0)
-plt.rcParams['figure.figsize'] = [50, 10]
-save_fig("xgtree")
-plt.show()
-
-xgb.plot_importance(xg_clf)
-plt.rcParams['figure.figsize'] = [5, 5]
-save_fig("xgparams")
-plt.show()
-
-
Nov 9, 2021
+Nov 11, 2021
@@ -417,7 +424,7 @@ MathJax.Hub.Config({
Nov 9, 2021
+Nov 11, 2021
@@ -701,7 +701,7 @@ $$
$$
-g = \sum_{k=1}^K p_{mk}(1-p_{mk}).
+g = \sum_{k\ne k'} p_{mk}p_{mk'}=\sum_{k=1}^K p_{mk}(1-p_{mk}).
$$
@@ -716,6 +716,33 @@ $$
+Gini Index?Coefficient/Impurity
+
+
+
+Why binary split?
+
+Visualizing the Tree, Classification
diff --git a/doc/pub/week45/html/week45-solarized.html b/doc/pub/week45/html/week45-solarized.html
index 2f9813c35..06ee0e9ce 100644
--- a/doc/pub/week45/html/week45-solarized.html
+++ b/doc/pub/week45/html/week45-solarized.html
@@ -112,6 +112,11 @@ div.toc p,a {
2,
None,
'classification-tree-how-to-split-nodes'),
+ ('Gini Index?Coefficient/Impurity',
+ 2,
+ None,
+ 'gini-index-coefficient-impurity'),
+ ('Why binary split?', 2, None, 'why-binary-split'),
('Visualizing the Tree, Classification',
2,
None,
@@ -319,7 +324,7 @@ MathJax.Hub.Config({
Nov 9, 2021
+Nov 11, 2021
@@ -789,7 +794,7 @@ $$
@@ -800,6 +805,30 @@ s = -\sum_{k=1}^K p_{mk}\log{p_{mk}}.
$$
+
$$
-g = \sum_{k=1}^K p_{mk}(1-p_{mk}).
+g = \sum_{k\ne k'} p_{mk}p_{mk'}=\sum_{k=1}^K p_{mk}(1-p_{mk}).
$$
+Gini Index?Coefficient/Impurity
+
+
+
+
+Why binary split?
+
+
Visualizing the Tree, Classification
diff --git a/doc/pub/week45/html/week45.html b/doc/pub/week45/html/week45.html
index 7794e13ef..59127c9bb 100644
--- a/doc/pub/week45/html/week45.html
+++ b/doc/pub/week45/html/week45.html
@@ -189,6 +189,11 @@ div.toc p,a {
2,
None,
'classification-tree-how-to-split-nodes'),
+ ('Gini Index?Coefficient/Impurity',
+ 2,
+ None,
+ 'gini-index-coefficient-impurity'),
+ ('Why binary split?', 2, None, 'why-binary-split'),
('Visualizing the Tree, Classification',
2,
None,
@@ -396,7 +401,7 @@ MathJax.Hub.Config({
Nov 9, 2021
+Nov 11, 2021
@@ -866,7 +871,7 @@ $$
@@ -877,6 +882,30 @@ s = -\sum_{k=1}^K p_{mk}\log{p_{mk}}.
$$
+
+Gini Index?Coefficient/Impurity
+
+
+
+
+Why binary split?
+
+
Visualizing the Tree, Classification
diff --git a/doc/pub/week45/ipynb/ipynb-week45-src.tar.gz b/doc/pub/week45/ipynb/ipynb-week45-src.tar.gz
index b424739dd..4a2660273 100644
Binary files a/doc/pub/week45/ipynb/ipynb-week45-src.tar.gz and b/doc/pub/week45/ipynb/ipynb-week45-src.tar.gz differ
diff --git a/doc/pub/week45/ipynb/week45.ipynb b/doc/pub/week45/ipynb/week45.ipynb
index 6a20f05e4..31069eb13 100644
--- a/doc/pub/week45/ipynb/week45.ipynb
+++ b/doc/pub/week45/ipynb/week45.ipynb
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
- "id": "999587fc",
+ "id": "5029effd",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
- "id": "f3ea736a",
+ "id": "0a3a2ca8",
"metadata": {
"editable": true
},
@@ -22,14 +22,14 @@
"# Week 45: Decisions Trees, Random Forests, Bagging and Boosting\n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Nov 9, 2021**\n",
+ "Date: **Nov 11, 2021**\n",
"\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license"
]
},
{
"cell_type": "markdown",
- "id": "f4a62cf4",
+ "id": "01543c43",
"metadata": {
"editable": true
},
@@ -53,7 +53,7 @@
},
{
"cell_type": "markdown",
- "id": "7d7e0a30",
+ "id": "57a03129",
"metadata": {
"editable": true
},
@@ -84,7 +84,7 @@
},
{
"cell_type": "markdown",
- "id": "f448055c",
+ "id": "62b1b891",
"metadata": {
"editable": true
},
@@ -104,7 +104,7 @@
},
{
"cell_type": "markdown",
- "id": "0cacfbd6",
+ "id": "b58e96f8",
"metadata": {
"editable": true
},
@@ -116,7 +116,7 @@
},
{
"cell_type": "markdown",
- "id": "09cac93a",
+ "id": "e98d1b92",
"metadata": {
"editable": true
},
@@ -128,7 +128,7 @@
},
{
"cell_type": "markdown",
- "id": "9e7e3a96",
+ "id": "b36bc042",
"metadata": {
"editable": true
},
@@ -146,7 +146,7 @@
},
{
"cell_type": "markdown",
- "id": "260a1d3f",
+ "id": "6182dd01",
"metadata": {
"editable": true
},
@@ -169,7 +169,7 @@
},
{
"cell_type": "markdown",
- "id": "71e35330",
+ "id": "4474c135",
"metadata": {
"editable": true
},
@@ -192,7 +192,7 @@
},
{
"cell_type": "markdown",
- "id": "c999b0f7",
+ "id": "938cfbf4",
"metadata": {
"editable": true
},
@@ -203,7 +203,7 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "d3154072",
+ "id": "fcfdb3ca",
"metadata": {
"collapsed": false,
"editable": true
@@ -304,7 +304,7 @@
},
{
"cell_type": "markdown",
- "id": "f38fb41a",
+ "id": "d06f81d0",
"metadata": {
"editable": true
},
@@ -326,7 +326,7 @@
},
{
"cell_type": "markdown",
- "id": "3f153956",
+ "id": "30987ca1",
"metadata": {
"editable": true
},
@@ -338,7 +338,7 @@
},
{
"cell_type": "markdown",
- "id": "be1e684e",
+ "id": "973f1d0b",
"metadata": {
"editable": true
},
@@ -349,7 +349,7 @@
},
{
"cell_type": "markdown",
- "id": "69f23ebf",
+ "id": "1c6d8f1d",
"metadata": {
"editable": true
},
@@ -371,7 +371,7 @@
},
{
"cell_type": "markdown",
- "id": "0c7546ee",
+ "id": "f2d05d9e",
"metadata": {
"editable": true
},
@@ -384,7 +384,7 @@
},
{
"cell_type": "markdown",
- "id": "901c465d",
+ "id": "f43f1d0c",
"metadata": {
"editable": true
},
@@ -396,7 +396,7 @@
},
{
"cell_type": "markdown",
- "id": "f193d5dc",
+ "id": "5a522921",
"metadata": {
"editable": true
},
@@ -406,7 +406,7 @@
},
{
"cell_type": "markdown",
- "id": "5b6fee06",
+ "id": "61aac4b3",
"metadata": {
"editable": true
},
@@ -418,7 +418,7 @@
},
{
"cell_type": "markdown",
- "id": "fb7f787a",
+ "id": "0ff01949",
"metadata": {
"editable": true
},
@@ -428,7 +428,7 @@
},
{
"cell_type": "markdown",
- "id": "76179adf",
+ "id": "1fae0428",
"metadata": {
"editable": true
},
@@ -440,7 +440,7 @@
},
{
"cell_type": "markdown",
- "id": "6c8ec2b3",
+ "id": "92da3ee2",
"metadata": {
"editable": true
},
@@ -473,7 +473,7 @@
},
{
"cell_type": "markdown",
- "id": "ee74a1ea",
+ "id": "4452e6cb",
"metadata": {
"editable": true
},
@@ -497,7 +497,7 @@
},
{
"cell_type": "markdown",
- "id": "6ab80129",
+ "id": "9884a7e6",
"metadata": {
"editable": true
},
@@ -509,7 +509,7 @@
},
{
"cell_type": "markdown",
- "id": "4fb3b414",
+ "id": "c8edb67c",
"metadata": {
"editable": true
},
@@ -521,7 +521,7 @@
},
{
"cell_type": "markdown",
- "id": "4a53cd13",
+ "id": "4e7213ce",
"metadata": {
"editable": true
},
@@ -549,7 +549,7 @@
},
{
"cell_type": "markdown",
- "id": "d4972eb3",
+ "id": "52c4e553",
"metadata": {
"editable": true
},
@@ -575,7 +575,7 @@
},
{
"cell_type": "markdown",
- "id": "fa21cd89",
+ "id": "b8b0003f",
"metadata": {
"editable": true
},
@@ -598,7 +598,7 @@
},
{
"cell_type": "markdown",
- "id": "ed9e140f",
+ "id": "da726ea4",
"metadata": {
"editable": true
},
@@ -625,7 +625,7 @@
},
{
"cell_type": "markdown",
- "id": "8339299a",
+ "id": "1ca80d94",
"metadata": {
"editable": true
},
@@ -644,7 +644,7 @@
},
{
"cell_type": "markdown",
- "id": "5a01e8b0",
+ "id": "f7caed93",
"metadata": {
"editable": true
},
@@ -656,7 +656,7 @@
},
{
"cell_type": "markdown",
- "id": "3b46b1a0",
+ "id": "f810cd2b",
"metadata": {
"editable": true
},
@@ -669,7 +669,7 @@
},
{
"cell_type": "markdown",
- "id": "bf8cf60e",
+ "id": "a99accde",
"metadata": {
"editable": true
},
@@ -681,7 +681,7 @@
},
{
"cell_type": "markdown",
- "id": "91bb9314",
+ "id": "70f12422",
"metadata": {
"editable": true
},
@@ -691,19 +691,19 @@
},
{
"cell_type": "markdown",
- "id": "21041ba3",
+ "id": "2c764a1f",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "g = \\sum_{k=1}^K p_{mk}(1-p_{mk}).\n",
+ "g = \\sum_{k\\ne k'} p_{mk}p_{mk'}=\\sum_{k=1}^K p_{mk}(1-p_{mk}).\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "27f57a10",
+ "id": "36ce64fd",
"metadata": {
"editable": true
},
@@ -713,7 +713,7 @@
},
{
"cell_type": "markdown",
- "id": "28610811",
+ "id": "c7cdc315",
"metadata": {
"editable": true
},
@@ -725,7 +725,44 @@
},
{
"cell_type": "markdown",
- "id": "448db121",
+ "id": "6f9b611a",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Gini Index?Coefficient/Impurity\n",
+ "\n",
+ "The Gini index $g$ gives us the degree of probability of a specific\n",
+ "variable that is wrongly classified.\n",
+ "\n",
+ "It takes values $g \\in [0,1]$, \n",
+ "1. $g=0$ means a *pure* case where all elements belong to one class only.\n",
+ "\n",
+ "2. A value $g=1$ means that all elements are randomly distributed across various classes.\n",
+ "\n",
+ "3. A value $g=0.5$ means that the elements in a node are uniformly distributed across classes. \n",
+ "\n",
+ "It favors binary splitting."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5afd0691",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Why binary split?\n",
+ "\n",
+ "It is custom to split to a tree uising binary splits. The reason is\n",
+ "that multiway splits fragment the data too quickly, leaving\n",
+ "insufficient data at the next level down. Multiway splits can be\n",
+ "achieved by a series of binary split and this is normally preferred."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "75295fdc",
"metadata": {
"editable": true
},
@@ -736,7 +773,7 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "daa33ec4",
+ "id": "4b5e8fd0",
"metadata": {
"collapsed": false,
"editable": true
@@ -780,7 +817,7 @@
},
{
"cell_type": "markdown",
- "id": "c35660d4",
+ "id": "2a7a2bc3",
"metadata": {
"editable": true
},
@@ -791,7 +828,7 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "20d3782d",
+ "id": "404b6b46",
"metadata": {
"collapsed": false,
"editable": true
@@ -826,7 +863,7 @@
},
{
"cell_type": "markdown",
- "id": "4b50c098",
+ "id": "fa63b030",
"metadata": {
"editable": true
},
@@ -839,7 +876,7 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "df24658e",
+ "id": "e18305d7",
"metadata": {
"collapsed": false,
"editable": true
@@ -857,7 +894,7 @@
},
{
"cell_type": "markdown",
- "id": "88327e54",
+ "id": "d3c9b005",
"metadata": {
"editable": true
},
@@ -871,7 +908,7 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "cabfc0eb",
+ "id": "b975a14b",
"metadata": {
"collapsed": false,
"editable": true
@@ -890,7 +927,7 @@
},
{
"cell_type": "markdown",
- "id": "f9b97781",
+ "id": "99f98c77",
"metadata": {
"editable": true
},
@@ -910,7 +947,7 @@
},
{
"cell_type": "markdown",
- "id": "6d158b95",
+ "id": "272a7e38",
"metadata": {
"editable": true
},
@@ -927,7 +964,7 @@
},
{
"cell_type": "markdown",
- "id": "3916a0a9",
+ "id": "e413c33c",
"metadata": {
"editable": true
},
@@ -939,7 +976,7 @@
},
{
"cell_type": "markdown",
- "id": "35e64400",
+ "id": "942dab79",
"metadata": {
"editable": true
},
@@ -956,7 +993,7 @@
},
{
"cell_type": "markdown",
- "id": "6b6f2758",
+ "id": "92fb5cd3",
"metadata": {
"editable": true
},
@@ -969,7 +1006,7 @@
},
{
"cell_type": "markdown",
- "id": "56a085e6",
+ "id": "f9d1aa8e",
"metadata": {
"editable": true
},
@@ -981,7 +1018,7 @@
},
{
"cell_type": "markdown",
- "id": "fa6f435c",
+ "id": "26b91b6f",
"metadata": {
"editable": true
},
@@ -991,7 +1028,7 @@
},
{
"cell_type": "markdown",
- "id": "8afa690b",
+ "id": "7ce65e73",
"metadata": {
"editable": true
},
@@ -1003,7 +1040,7 @@
},
{
"cell_type": "markdown",
- "id": "1b9c7d84",
+ "id": "396e79d5",
"metadata": {
"editable": true
},
@@ -1013,7 +1050,7 @@
},
{
"cell_type": "markdown",
- "id": "1af13ff9",
+ "id": "396f74b5",
"metadata": {
"editable": true
},
@@ -1025,7 +1062,7 @@
},
{
"cell_type": "markdown",
- "id": "aa4dc553",
+ "id": "4cbe3954",
"metadata": {
"editable": true
},
@@ -1038,7 +1075,7 @@
},
{
"cell_type": "markdown",
- "id": "aa1afc4e",
+ "id": "5f491d9d",
"metadata": {
"editable": true
},
@@ -1082,7 +1119,7 @@
},
{
"cell_type": "markdown",
- "id": "391b026f",
+ "id": "ba9449d2",
"metadata": {
"editable": true
},
@@ -1093,7 +1130,7 @@
{
"cell_type": "code",
"execution_count": 6,
- "id": "5d786a97",
+ "id": "ddcfa964",
"metadata": {
"collapsed": false,
"editable": true
@@ -1171,7 +1208,7 @@
},
{
"cell_type": "markdown",
- "id": "2ed48016",
+ "id": "f8a112fa",
"metadata": {
"editable": true
},
@@ -1189,7 +1226,7 @@
{
"cell_type": "code",
"execution_count": 7,
- "id": "a3b26fd1",
+ "id": "bd14f8fd",
"metadata": {
"collapsed": false,
"editable": true
@@ -1260,7 +1297,7 @@
},
{
"cell_type": "markdown",
- "id": "2896b4f4",
+ "id": "b02c63b6",
"metadata": {
"editable": true
},
@@ -1299,7 +1336,7 @@
},
{
"cell_type": "markdown",
- "id": "fea0e5e6",
+ "id": "be4068bc",
"metadata": {
"editable": true
},
@@ -1310,7 +1347,7 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "1dbc948a",
+ "id": "37fe4589",
"metadata": {
"collapsed": false,
"editable": true
@@ -1362,7 +1399,7 @@
},
{
"cell_type": "markdown",
- "id": "7b0d35e1",
+ "id": "7f22a1b7",
"metadata": {
"editable": true
},
@@ -1373,7 +1410,7 @@
{
"cell_type": "code",
"execution_count": 9,
- "id": "ef709f87",
+ "id": "319313ce",
"metadata": {
"collapsed": false,
"editable": true
@@ -1448,7 +1485,7 @@
},
{
"cell_type": "markdown",
- "id": "e7a8b237",
+ "id": "db1e37fc",
"metadata": {
"editable": true
},
@@ -1459,7 +1496,7 @@
{
"cell_type": "code",
"execution_count": 10,
- "id": "c5be70d1",
+ "id": "3cb94475",
"metadata": {
"collapsed": false,
"editable": true
@@ -1490,7 +1527,7 @@
},
{
"cell_type": "markdown",
- "id": "0c4d5dab",
+ "id": "e93f2813",
"metadata": {
"editable": true
},
@@ -1501,7 +1538,7 @@
{
"cell_type": "code",
"execution_count": 11,
- "id": "0963d7fc",
+ "id": "87026915",
"metadata": {
"collapsed": false,
"editable": true
@@ -1519,7 +1556,7 @@
{
"cell_type": "code",
"execution_count": 12,
- "id": "689f738f",
+ "id": "9237d8d5",
"metadata": {
"collapsed": false,
"editable": true
@@ -1534,7 +1571,7 @@
},
{
"cell_type": "markdown",
- "id": "f476fc1b",
+ "id": "ee857f99",
"metadata": {
"editable": true
},
@@ -1545,7 +1582,7 @@
{
"cell_type": "code",
"execution_count": 13,
- "id": "ec98869a",
+ "id": "379a7a94",
"metadata": {
"collapsed": false,
"editable": true
@@ -1595,7 +1632,7 @@
{
"cell_type": "code",
"execution_count": 14,
- "id": "ae89b603",
+ "id": "4030920d",
"metadata": {
"collapsed": false,
"editable": true
@@ -1634,7 +1671,7 @@
},
{
"cell_type": "markdown",
- "id": "f4c99932",
+ "id": "18f7ece2",
"metadata": {
"editable": true
},
@@ -1658,7 +1695,7 @@
},
{
"cell_type": "markdown",
- "id": "bcc7fb3d",
+ "id": "97a1b36e",
"metadata": {
"editable": true
},
@@ -1686,7 +1723,7 @@
},
{
"cell_type": "markdown",
- "id": "029c3b45",
+ "id": "e6d882aa",
"metadata": {
"editable": true
},
@@ -1717,7 +1754,7 @@
},
{
"cell_type": "markdown",
- "id": "597ecf99",
+ "id": "5551b560",
"metadata": {
"editable": true
},
@@ -1733,7 +1770,7 @@
},
{
"cell_type": "markdown",
- "id": "34ccd8a6",
+ "id": "21dc0297",
"metadata": {
"editable": true
},
@@ -1755,7 +1792,7 @@
},
{
"cell_type": "markdown",
- "id": "60285855",
+ "id": "e65838e4",
"metadata": {
"editable": true
},
@@ -1787,7 +1824,7 @@
},
{
"cell_type": "markdown",
- "id": "da86704c",
+ "id": "cf464522",
"metadata": {
"editable": true
},
@@ -1798,7 +1835,7 @@
{
"cell_type": "code",
"execution_count": 15,
- "id": "86f01d7b",
+ "id": "9d03d9ba",
"metadata": {
"collapsed": false,
"editable": true
@@ -1822,7 +1859,7 @@
},
{
"cell_type": "markdown",
- "id": "66aa1558",
+ "id": "eaff6b2f",
"metadata": {
"editable": true
},
@@ -1833,7 +1870,7 @@
{
"cell_type": "code",
"execution_count": 16,
- "id": "1782fb88",
+ "id": "50de3ca5",
"metadata": {
"collapsed": false,
"editable": true
@@ -1887,7 +1924,7 @@
},
{
"cell_type": "markdown",
- "id": "f5aa4b7b",
+ "id": "6910515b",
"metadata": {
"editable": true
},
@@ -1898,7 +1935,7 @@
{
"cell_type": "code",
"execution_count": 17,
- "id": "3c1a494b",
+ "id": "37266fb4",
"metadata": {
"collapsed": false,
"editable": true
@@ -1928,7 +1965,7 @@
{
"cell_type": "code",
"execution_count": 18,
- "id": "15cc6ef1",
+ "id": "4e3e7e4b",
"metadata": {
"collapsed": false,
"editable": true
@@ -1946,7 +1983,7 @@
{
"cell_type": "code",
"execution_count": 19,
- "id": "d5380589",
+ "id": "84fb433e",
"metadata": {
"collapsed": false,
"editable": true
@@ -1966,7 +2003,7 @@
{
"cell_type": "code",
"execution_count": 20,
- "id": "2630974e",
+ "id": "657d10e9",
"metadata": {
"collapsed": false,
"editable": true
@@ -1983,7 +2020,7 @@
},
{
"cell_type": "markdown",
- "id": "4bb8f29b",
+ "id": "83de8cfa",
"metadata": {
"editable": true
},
@@ -1994,7 +2031,7 @@
{
"cell_type": "code",
"execution_count": 21,
- "id": "47d6424f",
+ "id": "2e82fca3",
"metadata": {
"collapsed": false,
"editable": true
@@ -2014,7 +2051,7 @@
{
"cell_type": "code",
"execution_count": 22,
- "id": "554b7c3e",
+ "id": "7a1c6024",
"metadata": {
"collapsed": false,
"editable": true
@@ -2028,7 +2065,7 @@
{
"cell_type": "code",
"execution_count": 23,
- "id": "5cdabae8",
+ "id": "305e8fb0",
"metadata": {
"collapsed": false,
"editable": true
@@ -2044,7 +2081,7 @@
{
"cell_type": "code",
"execution_count": 24,
- "id": "511564a7",
+ "id": "09e1e925",
"metadata": {
"collapsed": false,
"editable": true
@@ -2082,7 +2119,7 @@
},
{
"cell_type": "markdown",
- "id": "2179d349",
+ "id": "c93add72",
"metadata": {
"editable": true
},
@@ -2096,7 +2133,7 @@
{
"cell_type": "code",
"execution_count": 25,
- "id": "42afc36e",
+ "id": "a87db197",
"metadata": {
"collapsed": false,
"editable": true
@@ -2165,7 +2202,7 @@
},
{
"cell_type": "markdown",
- "id": "5b790359",
+ "id": "bd224b9b",
"metadata": {
"editable": true
},
@@ -2189,7 +2226,7 @@
},
{
"cell_type": "markdown",
- "id": "c3c7a9fd",
+ "id": "0905f7fd",
"metadata": {
"editable": true
},
@@ -2220,7 +2257,7 @@
},
{
"cell_type": "markdown",
- "id": "50fc15a0",
+ "id": "83e0c000",
"metadata": {
"editable": true
},
@@ -2231,7 +2268,7 @@
{
"cell_type": "code",
"execution_count": 26,
- "id": "c0b99f8c",
+ "id": "46fedf6a",
"metadata": {
"collapsed": false,
"editable": true
@@ -2279,7 +2316,7 @@
},
{
"cell_type": "markdown",
- "id": "69f8628e",
+ "id": "c5b88629",
"metadata": {
"editable": true
},
@@ -2290,7 +2327,7 @@
{
"cell_type": "code",
"execution_count": 27,
- "id": "805d4227",
+ "id": "cd753289",
"metadata": {
"collapsed": false,
"editable": true
@@ -2324,7 +2361,7 @@
},
{
"cell_type": "markdown",
- "id": "49717d4a",
+ "id": "ff2c01f5",
"metadata": {
"editable": true
},
@@ -2337,7 +2374,7 @@
{
"cell_type": "code",
"execution_count": 28,
- "id": "34a0cc41",
+ "id": "5d722d0a",
"metadata": {
"collapsed": false,
"editable": true
@@ -2390,7 +2427,7 @@
},
{
"cell_type": "markdown",
- "id": "452d0656",
+ "id": "8882a66e",
"metadata": {
"editable": true
},
@@ -2401,7 +2438,7 @@
{
"cell_type": "code",
"execution_count": 29,
- "id": "de6fd29b",
+ "id": "23939bf2",
"metadata": {
"collapsed": false,
"editable": true
@@ -2431,7 +2468,7 @@
{
"cell_type": "code",
"execution_count": 30,
- "id": "37edc304",
+ "id": "41bc3004",
"metadata": {
"collapsed": false,
"editable": true
@@ -2449,7 +2486,7 @@
{
"cell_type": "code",
"execution_count": 31,
- "id": "e1f00841",
+ "id": "9d3397a2",
"metadata": {
"collapsed": false,
"editable": true
@@ -2469,7 +2506,7 @@
{
"cell_type": "code",
"execution_count": 32,
- "id": "08dced6b",
+ "id": "6087fac8",
"metadata": {
"collapsed": false,
"editable": true
@@ -2486,7 +2523,7 @@
},
{
"cell_type": "markdown",
- "id": "0ed0228e",
+ "id": "94e1f957",
"metadata": {
"editable": true
},
@@ -2509,7 +2546,7 @@
},
{
"cell_type": "markdown",
- "id": "bc05ef74",
+ "id": "1431c18a",
"metadata": {
"editable": true
},
@@ -2521,7 +2558,7 @@
},
{
"cell_type": "markdown",
- "id": "ffb4b144",
+ "id": "2a5dfd06",
"metadata": {
"editable": true
},
@@ -2546,7 +2583,7 @@
},
{
"cell_type": "markdown",
- "id": "c75cd923",
+ "id": "00c72d63",
"metadata": {
"editable": true
},
@@ -2572,7 +2609,7 @@
},
{
"cell_type": "markdown",
- "id": "2e732ead",
+ "id": "2e412e12",
"metadata": {
"editable": true
},
@@ -2583,7 +2620,7 @@
{
"cell_type": "code",
"execution_count": 33,
- "id": "befbdbd7",
+ "id": "cf7ad4d8",
"metadata": {
"collapsed": false,
"editable": true
@@ -2660,7 +2697,7 @@
},
{
"cell_type": "markdown",
- "id": "9173c1f7",
+ "id": "322bcada",
"metadata": {
"editable": true
},
@@ -2676,7 +2713,7 @@
},
{
"cell_type": "markdown",
- "id": "9cb41a55",
+ "id": "26c6739a",
"metadata": {
"editable": true
},
@@ -2687,7 +2724,7 @@
{
"cell_type": "code",
"execution_count": 34,
- "id": "d83311e2",
+ "id": "729bb490",
"metadata": {
"collapsed": false,
"editable": true
@@ -2702,7 +2739,7 @@
{
"cell_type": "code",
"execution_count": 35,
- "id": "8cd7ddc3",
+ "id": "8e9c400f",
"metadata": {
"collapsed": false,
"editable": true
@@ -2720,7 +2757,7 @@
},
{
"cell_type": "markdown",
- "id": "27c06676",
+ "id": "dd160607",
"metadata": {
"editable": true
},
@@ -2740,7 +2777,7 @@
},
{
"cell_type": "markdown",
- "id": "42ba5198",
+ "id": "938ac422",
"metadata": {
"editable": true
},
@@ -2754,7 +2791,7 @@
},
{
"cell_type": "markdown",
- "id": "7d10fac1",
+ "id": "99742488",
"metadata": {
"editable": true
},
@@ -2766,7 +2803,7 @@
},
{
"cell_type": "markdown",
- "id": "9e3039f9",
+ "id": "29864b76",
"metadata": {
"editable": true
},
@@ -2783,7 +2820,7 @@
},
{
"cell_type": "markdown",
- "id": "afc02aa3",
+ "id": "55dfbae1",
"metadata": {
"editable": true
},
@@ -2795,7 +2832,7 @@
},
{
"cell_type": "markdown",
- "id": "a614ac0d",
+ "id": "7a5153b5",
"metadata": {
"editable": true
},
@@ -2809,7 +2846,7 @@
},
{
"cell_type": "markdown",
- "id": "a8df1b75",
+ "id": "b5fe05e5",
"metadata": {
"editable": true
},
@@ -2821,7 +2858,7 @@
},
{
"cell_type": "markdown",
- "id": "f5fc30cf",
+ "id": "843358e0",
"metadata": {
"editable": true
},
@@ -2834,7 +2871,7 @@
},
{
"cell_type": "markdown",
- "id": "c7fe95ae",
+ "id": "2b42fc98",
"metadata": {
"editable": true
},
@@ -2846,7 +2883,7 @@
},
{
"cell_type": "markdown",
- "id": "2c6f0083",
+ "id": "fa243046",
"metadata": {
"editable": true
},
@@ -2856,7 +2893,7 @@
},
{
"cell_type": "markdown",
- "id": "b73d8c2d",
+ "id": "668f3417",
"metadata": {
"editable": true
},
@@ -2884,7 +2921,7 @@
},
{
"cell_type": "markdown",
- "id": "5c6f971b",
+ "id": "738db7be",
"metadata": {
"editable": true
},
@@ -2900,7 +2937,7 @@
},
{
"cell_type": "markdown",
- "id": "e0ed27b3",
+ "id": "76855445",
"metadata": {
"editable": true
},
@@ -2912,7 +2949,7 @@
},
{
"cell_type": "markdown",
- "id": "fc9f5f4f",
+ "id": "d6fc39ce",
"metadata": {
"editable": true
},
@@ -2923,7 +2960,7 @@
},
{
"cell_type": "markdown",
- "id": "02c2ec22",
+ "id": "f4ab9329",
"metadata": {
"editable": true
},
@@ -2935,7 +2972,7 @@
},
{
"cell_type": "markdown",
- "id": "e2288bd2",
+ "id": "63fdbf59",
"metadata": {
"editable": true
},
@@ -2945,7 +2982,7 @@
},
{
"cell_type": "markdown",
- "id": "98153a85",
+ "id": "95ed39a8",
"metadata": {
"editable": true
},
@@ -2957,7 +2994,7 @@
},
{
"cell_type": "markdown",
- "id": "c39e24a0",
+ "id": "9ad3b524",
"metadata": {
"editable": true
},
@@ -2967,7 +3004,7 @@
},
{
"cell_type": "markdown",
- "id": "afdfb641",
+ "id": "a4bb913d",
"metadata": {
"editable": true
},
@@ -2979,7 +3016,7 @@
},
{
"cell_type": "markdown",
- "id": "3039d39b",
+ "id": "5731b88a",
"metadata": {
"editable": true
},
@@ -2989,7 +3026,7 @@
},
{
"cell_type": "markdown",
- "id": "2ce0aa48",
+ "id": "05cb71a7",
"metadata": {
"editable": true
},
@@ -3001,7 +3038,7 @@
},
{
"cell_type": "markdown",
- "id": "119b9640",
+ "id": "cebf470f",
"metadata": {
"editable": true
},
@@ -3015,7 +3052,7 @@
},
{
"cell_type": "markdown",
- "id": "419822da",
+ "id": "5043858f",
"metadata": {
"editable": true
},
@@ -3031,7 +3068,7 @@
},
{
"cell_type": "markdown",
- "id": "1430356a",
+ "id": "714ea4ee",
"metadata": {
"editable": true
},
@@ -3043,7 +3080,7 @@
},
{
"cell_type": "markdown",
- "id": "c3a81493",
+ "id": "3f3eafcf",
"metadata": {
"editable": true
},
@@ -3059,7 +3096,7 @@
},
{
"cell_type": "markdown",
- "id": "190028df",
+ "id": "889882d2",
"metadata": {
"editable": true
},
@@ -3071,7 +3108,7 @@
},
{
"cell_type": "markdown",
- "id": "bbbca2ac",
+ "id": "0f1b8941",
"metadata": {
"editable": true
},
@@ -3081,7 +3118,7 @@
},
{
"cell_type": "markdown",
- "id": "7aee5d30",
+ "id": "a3bb1f79",
"metadata": {
"editable": true
},
@@ -3093,7 +3130,7 @@
},
{
"cell_type": "markdown",
- "id": "16cc7ab6",
+ "id": "88df6c57",
"metadata": {
"editable": true
},
@@ -3105,7 +3142,7 @@
},
{
"cell_type": "markdown",
- "id": "0e525c09",
+ "id": "3c9d9ac3",
"metadata": {
"editable": true
},
@@ -3117,7 +3154,7 @@
},
{
"cell_type": "markdown",
- "id": "df0326b4",
+ "id": "f1f95d4a",
"metadata": {
"editable": true
},
@@ -3128,7 +3165,7 @@
},
{
"cell_type": "markdown",
- "id": "07d19584",
+ "id": "43f589be",
"metadata": {
"editable": true
},
@@ -3140,7 +3177,7 @@
},
{
"cell_type": "markdown",
- "id": "76774593",
+ "id": "d96eaaee",
"metadata": {
"editable": true
},
@@ -3151,7 +3188,7 @@
},
{
"cell_type": "markdown",
- "id": "bb66a839",
+ "id": "8b948bbb",
"metadata": {
"editable": true
},
@@ -3163,7 +3200,7 @@
},
{
"cell_type": "markdown",
- "id": "bcd0b507",
+ "id": "3f3cc06e",
"metadata": {
"editable": true
},
@@ -3173,7 +3210,7 @@
},
{
"cell_type": "markdown",
- "id": "0c203949",
+ "id": "0b1c6031",
"metadata": {
"editable": true
},
@@ -3185,7 +3222,7 @@
},
{
"cell_type": "markdown",
- "id": "415377f0",
+ "id": "1ade6746",
"metadata": {
"editable": true
},
@@ -3197,7 +3234,7 @@
},
{
"cell_type": "markdown",
- "id": "daaca279",
+ "id": "a4bbc019",
"metadata": {
"editable": true
},
@@ -3209,7 +3246,7 @@
},
{
"cell_type": "markdown",
- "id": "f7bfba93",
+ "id": "4fd3dea0",
"metadata": {
"editable": true
},
@@ -3221,7 +3258,7 @@
},
{
"cell_type": "markdown",
- "id": "4f6a2f84",
+ "id": "ca658077",
"metadata": {
"editable": true
},
@@ -3231,7 +3268,7 @@
},
{
"cell_type": "markdown",
- "id": "6858be89",
+ "id": "b9829b00",
"metadata": {
"editable": true
},
@@ -3243,7 +3280,7 @@
},
{
"cell_type": "markdown",
- "id": "eecfcfb5",
+ "id": "d744ecab",
"metadata": {
"editable": true
},
@@ -3253,7 +3290,7 @@
},
{
"cell_type": "markdown",
- "id": "cd73a579",
+ "id": "35adfc8e",
"metadata": {
"editable": true
},
@@ -3265,7 +3302,7 @@
},
{
"cell_type": "markdown",
- "id": "33302d95",
+ "id": "1705faa0",
"metadata": {
"editable": true
},
@@ -3275,7 +3312,7 @@
},
{
"cell_type": "markdown",
- "id": "ce21db0f",
+ "id": "267b593c",
"metadata": {
"editable": true
},
@@ -3287,7 +3324,7 @@
},
{
"cell_type": "markdown",
- "id": "1fd9965f",
+ "id": "e484405e",
"metadata": {
"editable": true
},
@@ -3297,7 +3334,7 @@
},
{
"cell_type": "markdown",
- "id": "2a092b92",
+ "id": "ffdf57ea",
"metadata": {
"editable": true
},
@@ -3309,7 +3346,7 @@
},
{
"cell_type": "markdown",
- "id": "1553ca38",
+ "id": "d59e62b1",
"metadata": {
"editable": true
},
@@ -3319,7 +3356,7 @@
},
{
"cell_type": "markdown",
- "id": "60bd5373",
+ "id": "87a66ea2",
"metadata": {
"editable": true
},
@@ -3331,7 +3368,7 @@
},
{
"cell_type": "markdown",
- "id": "eace9c52",
+ "id": "5f13a75a",
"metadata": {
"editable": true
},
@@ -3351,7 +3388,7 @@
},
{
"cell_type": "markdown",
- "id": "8cb0caec",
+ "id": "8a00cea3",
"metadata": {
"editable": true
},
@@ -3363,7 +3400,7 @@
},
{
"cell_type": "markdown",
- "id": "4545fb70",
+ "id": "d9a59dc0",
"metadata": {
"editable": true
},
@@ -3373,7 +3410,7 @@
},
{
"cell_type": "markdown",
- "id": "c8c0b69c",
+ "id": "6d602605",
"metadata": {
"editable": true
},
@@ -3389,7 +3426,7 @@
},
{
"cell_type": "markdown",
- "id": "9a715091",
+ "id": "69942aec",
"metadata": {
"editable": true
},
@@ -3401,7 +3438,7 @@
},
{
"cell_type": "markdown",
- "id": "ebec62af",
+ "id": "e9ec97a5",
"metadata": {
"editable": true
},
@@ -3429,7 +3466,7 @@
},
{
"cell_type": "markdown",
- "id": "6df3a47f",
+ "id": "de9119e8",
"metadata": {
"editable": true
},
@@ -3442,7 +3479,7 @@
{
"cell_type": "code",
"execution_count": 36,
- "id": "a762adfc",
+ "id": "f790730a",
"metadata": {
"collapsed": false,
"editable": true
@@ -3474,7 +3511,7 @@
},
{
"cell_type": "markdown",
- "id": "38d12d89",
+ "id": "1d164a72",
"metadata": {
"editable": true
},
@@ -3492,7 +3529,7 @@
},
{
"cell_type": "markdown",
- "id": "1b792064",
+ "id": "2aea58b1",
"metadata": {
"editable": true
},
@@ -3505,7 +3542,7 @@
},
{
"cell_type": "markdown",
- "id": "beaf036c",
+ "id": "9fe0e925",
"metadata": {
"editable": true
},
@@ -3517,7 +3554,7 @@
},
{
"cell_type": "markdown",
- "id": "702b65a0",
+ "id": "93d443b7",
"metadata": {
"editable": true
},
@@ -3527,7 +3564,7 @@
},
{
"cell_type": "markdown",
- "id": "2652a1da",
+ "id": "ff0a341f",
"metadata": {
"editable": true
},
@@ -3539,7 +3576,7 @@
},
{
"cell_type": "markdown",
- "id": "be287ca2",
+ "id": "7a54de9d",
"metadata": {
"editable": true
},
@@ -3549,7 +3586,7 @@
},
{
"cell_type": "markdown",
- "id": "ff3901b6",
+ "id": "f7b3ed21",
"metadata": {
"editable": true
},
@@ -3561,7 +3598,7 @@
},
{
"cell_type": "markdown",
- "id": "5228f5fd",
+ "id": "d76a3b92",
"metadata": {
"editable": true
},
@@ -3574,7 +3611,7 @@
},
{
"cell_type": "markdown",
- "id": "1e30448f",
+ "id": "7ebc7787",
"metadata": {
"editable": true
},
@@ -3586,7 +3623,7 @@
},
{
"cell_type": "markdown",
- "id": "5bd43cc1",
+ "id": "81f21ab2",
"metadata": {
"editable": true
},
@@ -3598,7 +3635,7 @@
},
{
"cell_type": "markdown",
- "id": "acaae452",
+ "id": "c3b3e85b",
"metadata": {
"editable": true
},
@@ -3610,7 +3647,7 @@
},
{
"cell_type": "markdown",
- "id": "022645ac",
+ "id": "75c9d363",
"metadata": {
"editable": true
},
@@ -3620,7 +3657,7 @@
},
{
"cell_type": "markdown",
- "id": "8e6bbfb4",
+ "id": "ec040a0c",
"metadata": {
"editable": true
},
@@ -3632,7 +3669,7 @@
},
{
"cell_type": "markdown",
- "id": "66ce0709",
+ "id": "1c51f8d8",
"metadata": {
"editable": true
},
@@ -3642,7 +3679,7 @@
},
{
"cell_type": "markdown",
- "id": "8ac79309",
+ "id": "40aa6a3a",
"metadata": {
"editable": true
},
@@ -3658,7 +3695,7 @@
},
{
"cell_type": "markdown",
- "id": "d562f162",
+ "id": "b90f8570",
"metadata": {
"editable": true
},
@@ -3670,7 +3707,7 @@
},
{
"cell_type": "markdown",
- "id": "9e609544",
+ "id": "8b448fbd",
"metadata": {
"editable": true
},
@@ -3691,7 +3728,7 @@
},
{
"cell_type": "markdown",
- "id": "eced6bdc",
+ "id": "05caebd1",
"metadata": {
"editable": true
},
@@ -3702,7 +3739,7 @@
{
"cell_type": "code",
"execution_count": 37,
- "id": "613e05cd",
+ "id": "fa49f235",
"metadata": {
"collapsed": false,
"editable": true
@@ -3759,7 +3796,7 @@
},
{
"cell_type": "markdown",
- "id": "77c76d85",
+ "id": "98586a10",
"metadata": {
"editable": true
},
@@ -3770,7 +3807,7 @@
{
"cell_type": "code",
"execution_count": 38,
- "id": "af5a3843",
+ "id": "d21bf444",
"metadata": {
"collapsed": false,
"editable": true
@@ -3821,7 +3858,7 @@
},
{
"cell_type": "markdown",
- "id": "0ca37c1d",
+ "id": "e0d676e3",
"metadata": {
"editable": true
},
@@ -3844,7 +3881,7 @@
},
{
"cell_type": "markdown",
- "id": "b2dd3360",
+ "id": "71519f2b",
"metadata": {
"editable": true
},
@@ -3855,7 +3892,7 @@
{
"cell_type": "code",
"execution_count": 39,
- "id": "4bf9f894",
+ "id": "b375c6ca",
"metadata": {
"collapsed": false,
"editable": true
@@ -3912,7 +3949,7 @@
},
{
"cell_type": "markdown",
- "id": "00b1f1af",
+ "id": "761ccc1e",
"metadata": {
"editable": true
},
@@ -3925,7 +3962,7 @@
{
"cell_type": "code",
"execution_count": 40,
- "id": "4454900b",
+ "id": "0673208c",
"metadata": {
"collapsed": false,
"editable": true
diff --git a/doc/src/week45/week45.do.txt b/doc/src/week45/week45.do.txt
index f7dc247d1..40c801067 100644
--- a/doc/src/week45/week45.do.txt
+++ b/doc/src/week45/week45.do.txt
@@ -428,7 +428,7 @@ p_{mk} = \frac{1}{N_m}\sum_{x_i\in R_m}I(y_i\ne k) = 1-p_{mk}.
* Gini index $g$
!bt
\[
-g = \sum_{k=1}^K p_{mk}(1-p_{mk}).
+g = \sum_{k\ne k'} p_{mk}p_{mk'}=\sum_{k=1}^K p_{mk}(1-p_{mk}).
\]
!et
* Information entropy or just entropy $s$
@@ -439,6 +439,30 @@ s = -\sum_{k=1}^K p_{mk}\log{p_{mk}}.
!et
+!split
+===== Gini Index?Coefficient/Impurity =====
+
+The Gini index $g$ gives us the degree of probability of a specific
+variable that is wrongly classified.
+
+It takes values $g \in [0,1]$,
+o $g=0$ means a *pure* case where all elements belong to one class only.
+o A value $g=1$ means that all elements are randomly distributed across various classes.
+o A value $g=0.5$ means that the elements in a node are uniformly distributed across classes.
+
+It favors binary splitting.
+
+!split
+===== Why binary split? =====
+
+It is custom to split to a tree uising binary splits. The reason is
+that multiway splits fragment the data too quickly, leaving
+insufficient data at the next level down. Multiway splits can be
+achieved by a series of binary split and this is normally preferred.
+
+
+
+
!split
===== Visualizing the Tree, Classification =====
!bc pycod