-
-
The plain decision trees suffer from high -variance. This means that if we split the training data into two parts -at random, and fit a decision tree to both halves, the results that we -get could be quite different. In contrast, a procedure with low -variance will yield similar results if applied repeatedly to distinct -data sets; linear regression tends to have low variance, if the ratio -of \( n \) to \( p \) is moderately large. +
The idea behind boosting, and voting as well can be phrased as follows: +Can a group of people somehow arrive at highly +reasoned decisions, despite the weak judgement of the individual +members?
-Bootstrap aggregation, or just bagging, is a -general-purpose procedure for reducing the variance of a statistical -learning method. +
The aim is to create a good classifier by combining several weak classifiers. +A weak classifier is a classifier which is able to produce results that are only slightly better than guessing at random.
+The basic approach is to apply repeatedly (in boosting this is done in an iterative way) a weak classifier to modifications of the data. +In voting we simply apply the law of large numbers while in boosting we give more weight to misclassified data in +each iteration. +
+ +Decision trees play an important role as our weak classifier. They serve as the basic method.
+diff --git a/doc/pub/week44/html/._week44-bs047.html b/doc/pub/week44/html/._week44-bs047.html index 1037eafcb..0bcd1ac63 100644 --- a/doc/pub/week44/html/._week44-bs047.html +++ b/doc/pub/week44/html/._week44-bs047.html @@ -149,10 +149,10 @@ doconce format html week44.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'computing-the-gini-factor'), - ('Another example, the moons again', + ('Another example, the moons', 2, None, - 'another-example-the-moons-again'), + 'another-example-the-moons'), ('Playing around with regions', 2, None, @@ -173,26 +173,6 @@ doconce format html week44.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'an-overview-of-ensemble-methods'), - ('Bagging', 2, None, 'bagging'), - ('More bagging', 2, None, 'more-bagging'), - ('Simple Voting Example, head or tail', - 2, - None, - 'simple-voting-example-head-or-tail'), - ('Using the Voting Classifier', - 2, - None, - 'using-the-voting-classifier'), - ('Please, not the moons again! Voting and Bagging', - 2, - None, - 'please-not-the-moons-again-voting-and-bagging'), - ('Bagging Examples', 2, None, 'bagging-examples'), - ('Making your own Bootstrap: Changing the Level of the Decision ' - 'Tree', - 2, - None, - 'making-your-own-bootstrap-changing-the-level-of-the-decision-tree'), ('Why Voting?', 2, None, 'why-voting'), ('Tossing coins', 2, None, 'tossing-coins'), ('Standard imports first', 2, None, 'standard-imports-first'), @@ -205,6 +185,13 @@ doconce format html week44.do.txt --html_style=bootstrap --pygments_html_style=d None, 'using-the-voting-classifier'), ('Voting and Bagging', 2, None, 'voting-and-bagging'), + ('Bagging', 2, None, 'bagging'), + ('More bagging', 2, None, 'more-bagging'), + ('Making your own Bootstrap: Changing the Level of the Decision ' + 'Tree', + 2, + None, + 'making-your-own-bootstrap-changing-the-level-of-the-decision-tree'), ('Random forests', 2, None, 'random-forests'), ('Random Forest Algorithm', 2, None, 'random-forest-algorithm'), ('Random Forests Compared with other Methods on the Cancer Data', @@ -286,7 +273,7 @@ MathJax.Hub.Config({
-
Bagging typically results in improved accuracy -over prediction using a single tree. Unfortunately, however, it can be -difficult to interpret the resulting model. Recall that one of the -advantages of decision trees is the attractive and easily interpreted -diagram that results. +
The simplest case is a so-called voting ensemble. To illustrate this, +think of yourself tossing coins with a biased outcome of 51 per cent +for heads and 49% for tails. With only few tosses, +you may not clearly see this distribution for heads and tails. However, after some +thousands of tosses, there will be a clear majority of heads. With 2000 tosses +you should see approximately 1020 heads and 980 tails.
-However, when we bag a large number of trees, it is no longer -possible to represent the resulting statistical learning procedure -using a single tree, and it is no longer clear which variables are -most important to the procedure. Thus, bagging improves prediction -accuracy at the expense of interpretability. Although the collection -of bagged trees is much more difficult to interpret than a single -tree, one can obtain an overall summary of the importance of each -predictor using the MSE (for bagging regression trees) or the Gini -index (for bagging classification trees). In the case of bagging -regression trees, we can record the total amount that the MSE is -decreased due to splits over a given predictor, averaged over all \( B \) possible -trees. A large value indicates an important predictor. Similarly, in -the context of bagging classification trees, we can add up the total -amount that the Gini index is decreased by splits over a given -predictor, averaged over all \( B \) trees. +
We can then state that the outcome is a clear majority of heads. If +you do this ten thousand times, it is easy to see that there is a 97% +likelihood of a majority of heads. +
+ +Another example would be to collect all polls before an +election. Different polls may show different likelihoods for a +candidate winning with say a majority of the popular vote. The majority vote +would then consist in many polls indicating that this candidate will +actually win. +
+ +The example here shows how we can implement the coin tossing case, +clealry demostrating that after some tosses we see the law of large +numbers kicking in.
@@ -373,7 +357,7 @@ predictor, averaged over all \( B \) trees.
-
heads_proba = 0.51
-coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
-cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)
-plt.figure(figsize=(8,3.5))
-plt.plot(cumulative_heads_ratio)
-plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
-plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
-plt.xlabel("Number of coin tosses")
-plt.ylabel("Heads ratio")
-plt.legend(loc="lower right")
-plt.axis([0, 10000, 0.42, 0.58])
-save_fig("votingsimple")
-plt.show()
+ # Common imports
+from IPython.display import Image
+from pydot import graph_from_dot_data
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.model_selection import train_test_split
+from sklearn.tree import export_graphviz
+from sklearn.preprocessing import StandardScaler, OneHotEncoder
+from sklearn.compose import ColumnTransformer
+from IPython.display import Image
+from pydot import graph_from_dot_data
+import os
+
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+ os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+ os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+ os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+ return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+ return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+ plt.savefig(image_path(fig_id) + ".png", format='png')
-
from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
+ # Common imports
+import numpy as np
+import matplotlib
+import matplotlib.pyplot as plt
+from matplotlib.colors import ListedColormap
+plt.rcParams['axes.labelsize'] = 14
+plt.rcParams['xtick.labelsize'] = 12
+plt.rcParams['ytick.labelsize'] = 12
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
+heads_proba = 0.51
+coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
+cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)
+plt.figure(figsize=(8,3.5))
+plt.plot(cumulative_heads_ratio)
+plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
+plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
+plt.xlabel("Number of coin tosses")
+plt.ylabel("Heads ratio")
+plt.legend(loc="lower right")
+plt.axis([0, 10000, 0.42, 0.58])
+save_fig("votingsimple")
+plt.show()
-
We can use the voting classifier on other data sets, here the exciting binary case of two distinct objects using the make moons functionality of Scikit-Learn.
from sklearn.metrics import accuracy_score
+
+from sklearn.metrics import accuracy_score
for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(probability=True, random_state=42)
+log_clf = LogisticRegression(solver="liblinear", random_state=42)
+rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
+svm_clf = SVC(gamma="auto", probability=True, random_state=42)
voting_clf = VotingClassifier(
estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
voting='soft')
voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
+
+from sklearn.metrics import accuracy_score
for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
clf.fit(X_train, y_train)
@@ -465,9 +396,6 @@ voting_clf.fit(X_train, y_train)
-
from sklearn.ensemble import BaggingClassifier
-from sklearn.tree import DecisionTreeClassifier
+ from sklearn.model_selection import train_test_split
+from sklearn.datasets import make_moons
-bag_clf = BaggingClassifier(
- DecisionTreeClassifier(random_state=42), n_estimators=500,
- max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
-bag_clf.fit(X_train, y_train)
-y_pred = bag_clf.predict(X_test)
+X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
+X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.ensemble import VotingClassifier
+from sklearn.linear_model import LogisticRegression
+from sklearn.svm import SVC
+
+log_clf = LogisticRegression(random_state=42)
+rnd_clf = RandomForestClassifier(random_state=42)
+svm_clf = SVC(random_state=42)
+
+voting_clf = VotingClassifier(
+ estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
+ voting='hard')
+voting_clf.fit(X_train, y_train)
from sklearn.metrics import accuracy_score
-print(accuracy_score(y_test, y_pred))
-
-tree_clf = DecisionTreeClassifier(random_state=42)
-tree_clf.fit(X_train, y_train)
-y_pred_tree = tree_clf.predict(X_test)
-print(accuracy_score(y_test, y_pred_tree))
-
-from matplotlib.colors import ListedColormap
-def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
- x1s = np.linspace(axes[0], axes[1], 100)
- x2s = np.linspace(axes[2], axes[3], 100)
- x1, x2 = np.meshgrid(x1s, x2s)
- X_new = np.c_[x1.ravel(), x2.ravel()]
- y_pred = clf.predict(X_new).reshape(x1.shape)
- custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
- plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
- if contour:
- custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
- plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
- plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
- plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
- plt.axis(axes)
- plt.xlabel(r"$x_1$", fontsize=18)
- plt.ylabel(r"$x_2$", fontsize=18, rotation=0)
-plt.figure(figsize=(11,4))
-plt.subplot(121)
-plot_decision_boundary(tree_clf, X, y)
-plt.title("Decision Tree", fontsize=14)
-plt.subplot(122)
-plot_decision_boundary(bag_clf, X, y)
-plt.title("Decision Trees with Bagging", fontsize=14)
-save_fig("baggingtree")
-plt.show()
+for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
+ clf.fit(X_train, y_train)
+ y_pred = clf.predict(X_test)
+ print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
+
+log_clf = LogisticRegression(random_state=42)
+rnd_clf = RandomForestClassifier(random_state=42)
+svm_clf = SVC(probability=True, random_state=42)
+
+voting_clf = VotingClassifier(
+ estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
+ voting='soft')
+voting_clf.fit(X_train, y_train)
+
+from sklearn.metrics import accuracy_score
+
+for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
+ clf.fit(X_train, y_train)
+ y_pred = clf.predict(X_test)
+ print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with -a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points \( n \)). +
The plain decision trees suffer from high +variance. This means that if we split the training data into two parts +at random, and fit a decision tree to both halves, the results that we +get could be quite different. In contrast, a procedure with low +variance will yield similar results if applied repeatedly to distinct +data sets; linear regression tends to have low variance, if the ratio +of \( n \) to \( p \) is moderately large.
- -import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.pipeline import make_pipeline
-from sklearn.utils import resample
-from sklearn.tree import DecisionTreeRegressor
-
-n = 100
-n_boostraps = 100
-maxdepth = 8
-
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-error = np.zeros(maxdepth)
-bias = np.zeros(maxdepth)
-variance = np.zeros(maxdepth)
-polydegree = np.zeros(maxdepth)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
-
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-# we produce a simple tree first as benchmark
-simpletree = DecisionTreeRegressor(max_depth=3)
-simpletree.fit(X_train_scaled, y_train)
-simpleprediction = simpletree.predict(X_test_scaled)
-for degree in range(1,maxdepth):
- model = DecisionTreeRegressor(max_depth=degree)
- y_pred = np.empty((y_test.shape[0], n_boostraps))
- for i in range(n_boostraps):
- x_, y_ = resample(X_train_scaled, y_train)
- model.fit(x_, y_)
- y_pred[:, i] = model.predict(X_test_scaled)#.ravel()
-
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
- variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
- print('Polynomial degree:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-
-mse_simpletree= np.mean( np.mean((y_test - simpleprediction)**2)
-print(mse_simpletree)
-plt.xlim(1,maxdepth)
-plt.plot(polydegree, error, label='MSE')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
-save_fig("baggingboot")
-plt.show()
-
-Bootstrap aggregation, or just bagging, is a +general-purpose procedure for reducing the variance of a statistical +learning method. +
@@ -428,11 +342,6 @@ plt.show()
-
The idea behind boosting, and voting as well can be phrased as follows: -Can a group of people somehow arrive at highly -reasoned decisions, despite the weak judgement of the individual -members? +
Bagging typically results in improved accuracy +over prediction using a single tree. Unfortunately, however, it can be +difficult to interpret the resulting model. Recall that one of the +advantages of decision trees is the attractive and easily interpreted +diagram that results.
-The aim is to create a good classifier by combining several weak classifiers. -A weak classifier is a classifier which is able to produce results that are only slightly better than guessing at random. +
However, when we bag a large number of trees, it is no longer +possible to represent the resulting statistical learning procedure +using a single tree, and it is no longer clear which variables are +most important to the procedure. Thus, bagging improves prediction +accuracy at the expense of interpretability. Although the collection +of bagged trees is much more difficult to interpret than a single +tree, one can obtain an overall summary of the importance of each +predictor using the MSE (for bagging regression trees) or the Gini +index (for bagging classification trees). In the case of bagging +regression trees, we can record the total amount that the MSE is +decreased due to splits over a given predictor, averaged over all \( B \) possible +trees. A large value indicates an important predictor. Similarly, in +the context of bagging classification trees, we can add up the total +amount that the Gini index is decreased by splits over a given +predictor, averaged over all \( B \) trees.
-The basic approach is to apply repeatedly (in boosting this is done in an iterative way) a weak classifier to modifications of the data. -In voting we simply apply the law of large numbers while in boosting we give more weight to misclassified data in -each iteration. -
- -Decision trees play an important role as our weak classifier. They serve as the basic method.
-diff --git a/doc/pub/week44/html/._week44-bs054.html b/doc/pub/week44/html/._week44-bs054.html index 32b0cbca1..d0f433fb8 100644 --- a/doc/pub/week44/html/._week44-bs054.html +++ b/doc/pub/week44/html/._week44-bs054.html @@ -149,10 +149,10 @@ doconce format html week44.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'computing-the-gini-factor'), - ('Another example, the moons again', + ('Another example, the moons', 2, None, - 'another-example-the-moons-again'), + 'another-example-the-moons'), ('Playing around with regions', 2, None, @@ -173,26 +173,6 @@ doconce format html week44.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'an-overview-of-ensemble-methods'), - ('Bagging', 2, None, 'bagging'), - ('More bagging', 2, None, 'more-bagging'), - ('Simple Voting Example, head or tail', - 2, - None, - 'simple-voting-example-head-or-tail'), - ('Using the Voting Classifier', - 2, - None, - 'using-the-voting-classifier'), - ('Please, not the moons again! Voting and Bagging', - 2, - None, - 'please-not-the-moons-again-voting-and-bagging'), - ('Bagging Examples', 2, None, 'bagging-examples'), - ('Making your own Bootstrap: Changing the Level of the Decision ' - 'Tree', - 2, - None, - 'making-your-own-bootstrap-changing-the-level-of-the-decision-tree'), ('Why Voting?', 2, None, 'why-voting'), ('Tossing coins', 2, None, 'tossing-coins'), ('Standard imports first', 2, None, 'standard-imports-first'), @@ -205,6 +185,13 @@ doconce format html week44.do.txt --html_style=bootstrap --pygments_html_style=d None, 'using-the-voting-classifier'), ('Voting and Bagging', 2, None, 'voting-and-bagging'), + ('Bagging', 2, None, 'bagging'), + ('More bagging', 2, None, 'more-bagging'), + ('Making your own Bootstrap: Changing the Level of the Decision ' + 'Tree', + 2, + None, + 'making-your-own-bootstrap-changing-the-level-of-the-decision-tree'), ('Random forests', 2, None, 'random-forests'), ('Random Forest Algorithm', 2, None, 'random-forest-algorithm'), ('Random Forests Compared with other Methods on the Cancer Data', @@ -286,7 +273,7 @@ MathJax.Hub.Config({
-
The simplest case is a so-called voting ensemble. To illustrate this, -think of yourself tossing coins with a biased outcome of 51 per cent -for heads and 49% for tails. With only few tosses, -you may not clearly see this distribution for heads and tails. However, after some -thousands of tosses, there will be a clear majority of heads. With 2000 tosses -you should see approximately 1020 heads and 980 tails. +
Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with +a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points \( n \)).
-We can then state that the outcome is a clear majority of heads. If -you do this ten thousand times, it is easy to see that there is a 97% -likelihood of a majority of heads. -
+ +import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+from sklearn.tree import DecisionTreeRegressor
-Another example would be to collect all polls before an
-election. Different polls may show different likelihoods for a
-candidate winning with say a majority of the popular vote. The majority vote
-would then consist in many polls indicating that this candidate will
-actually win.
-
+n = 100
+n_boostraps = 100
+maxdepth = 8
+
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+error = np.zeros(maxdepth)
+bias = np.zeros(maxdepth)
+variance = np.zeros(maxdepth)
+polydegree = np.zeros(maxdepth)
+X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+
+# we produce a simple tree first as benchmark
+simpletree = DecisionTreeRegressor(max_depth=3)
+simpletree.fit(X_train_scaled, y_train)
+simpleprediction = simpletree.predict(X_test_scaled)
+for degree in range(1,maxdepth):
+ model = DecisionTreeRegressor(max_depth=degree)
+ y_pred = np.empty((y_test.shape[0], n_boostraps))
+ for i in range(n_boostraps):
+ x_, y_ = resample(X_train_scaled, y_train)
+ model.fit(x_, y_)
+ y_pred[:, i] = model.predict(X_test_scaled)#.ravel()
+
+ polydegree[degree] = degree
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+ variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+ print('Polynomial degree:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+
+mse_simpletree= np.mean( np.mean((y_test - simpleprediction)**2)
+print(mse_simpletree)
+plt.xlim(1,maxdepth)
+plt.plot(polydegree, error, label='MSE')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+save_fig("baggingboot")
+plt.show()
+
+The example here shows how we can implement the coin tossing case, -clealry demostrating that after some tosses we see the law of large -numbers kicking in. -
@@ -368,10 +409,6 @@ numbers kicking in.
-
Random forests provide an improvement over bagged trees by way of a +small tweak that decorrelates the trees. +
- -# Common imports
-from IPython.display import Image
-from pydot import graph_from_dot_data
-import pandas as pd
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.tree import DecisionTreeClassifier
-from sklearn.model_selection import train_test_split
-from sklearn.tree import export_graphviz
-from sklearn.preprocessing import StandardScaler, OneHotEncoder
-from sklearn.compose import ColumnTransformer
-from IPython.display import Image
-from pydot import graph_from_dot_data
-import os
+As in bagging, we build a
+number of decision trees on bootstrapped training samples. But when
+building these decision trees, each time a split in a tree is
+considered, a random sample of \( m \) predictors is chosen as split
+candidates from the full set of \( p \) predictors. The split is allowed to
+use only one of those \( m \) predictors.
+
-# Where to save the figures and data files
-PROJECT_ROOT_DIR = "Results"
-FIGURE_ID = "Results/FigureFiles"
-DATA_ID = "DataFiles/"
+A fresh sample of \( m \) predictors is
+taken at each split, and typically we choose
+
-if not os.path.exists(PROJECT_ROOT_DIR):
- os.mkdir(PROJECT_ROOT_DIR)
+$$
+m\approx \sqrt{p}.
+$$
-if not os.path.exists(FIGURE_ID):
- os.makedirs(FIGURE_ID)
-
-if not os.path.exists(DATA_ID):
- os.makedirs(DATA_ID)
-
-def image_path(fig_id):
- return os.path.join(FIGURE_ID, fig_id)
-
-def data_path(dat_id):
- return os.path.join(DATA_ID, dat_id)
-
-def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
-
-In building a random forest, at +each split in the tree, the algorithm is not even allowed to consider +a majority of the available predictors. +
+The reason for this is rather clever. Suppose that there is one very +strong predictor in the data set, along with a number of other +moderately strong predictors. Then in the collection of bagged +variable importance random forest trees, most or all of the trees will +use this strong predictor in the top split. Consequently, all of the +bagged trees will look quite similar to each other. Hence the +predictions from the bagged trees will be highly correlated. +Unfortunately, averaging many highly correlated quantities does not +lead to as large of a reduction in variance as averaging many +uncorrelated quantities. In particular, this means that bagging will +not lead to a substantial reduction in variance over a single tree in +this setting. +
@@ -402,10 +364,6 @@ DATA_ID = "
The algorithm described here can be applied to both classification and regression problems. We will grow of forest of say \( B \) trees.
We can use the voting classifier on other data sets, here the exciting binary case of two distinct objects using the make moons functionality of Scikit-Learn. Recall that the cumulative gains curve shows the percentage of the
+overall number of cases in a given category gained by targeting a
+percentage of the total number of cases.
+ Similarly, the receiver operating characteristic curve, or ROC curve,
+displays the diagnostic ability of a binary classifier system as its
+discrimination threshold is varied. It plots the true positive rate against the false positive rate.
+
@@ -406,10 +415,6 @@ voting_clf.fit(X_train, y_train)
The plain decision trees suffer from high
-variance. This means that if we split the training data into two parts
-at random, and fit a decision tree to both halves, the results that we
-get could be quite different. In contrast, a procedure with low
-variance will yield similar results if applied repeatedly to distinct
-data sets; linear regression tends to have low variance, if the ratio
-of \( n \) to \( p \) is moderately large.
- Bootstrap aggregation, or just bagging, is a
-general-purpose procedure for reducing the variance of a statistical
-learning method.
- Bagging typically results in improved accuracy
-over prediction using a single tree. Unfortunately, however, it can be
-difficult to interpret the resulting model. Recall that one of the
-advantages of decision trees is the attractive and easily interpreted
-diagram that results.
- However, when we bag a large number of trees, it is no longer
-possible to represent the resulting statistical learning procedure
-using a single tree, and it is no longer clear which variables are
-most important to the procedure. Thus, bagging improves prediction
-accuracy at the expense of interpretability. Although the collection
-of bagged trees is much more difficult to interpret than a single
-tree, one can obtain an overall summary of the importance of each
-predictor using the MSE (for bagging regression trees) or the Gini
-index (for bagging classification trees). In the case of bagging
-regression trees, we can record the total amount that the MSE is
-decreased due to splits over a given predictor, averaged over all \( B \) possible
-trees. A large value indicates an important predictor. Similarly, in
-the context of bagging classification trees, we can add up the total
-amount that the Gini index is decreased by splits over a given
-predictor, averaged over all \( B \) trees.
- Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with
-a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points \( n \)).
- The plain decision trees suffer from high
+variance. This means that if we split the training data into two parts
+at random, and fit a decision tree to both halves, the results that we
+get could be quite different. In contrast, a procedure with low
+variance will yield similar results if applied repeatedly to distinct
+data sets; linear regression tends to have low variance, if the ratio
+of \( n \) to \( p \) is moderately large.
+ Bootstrap aggregation, or just bagging, is a
+general-purpose procedure for reducing the variance of a statistical
+learning method.
+ Bagging typically results in improved accuracy
+over prediction using a single tree. Unfortunately, however, it can be
+difficult to interpret the resulting model. Recall that one of the
+advantages of decision trees is the attractive and easily interpreted
+diagram that results.
+ However, when we bag a large number of trees, it is no longer
+possible to represent the resulting statistical learning procedure
+using a single tree, and it is no longer clear which variables are
+most important to the procedure. Thus, bagging improves prediction
+accuracy at the expense of interpretability. Although the collection
+of bagged trees is much more difficult to interpret than a single
+tree, one can obtain an overall summary of the importance of each
+predictor using the MSE (for bagging regression trees) or the Gini
+index (for bagging classification trees). In the case of bagging
+regression trees, we can record the total amount that the MSE is
+decreased due to splits over a given predictor, averaged over all \( B \) possible
+trees. A large value indicates an important predictor. Similarly, in
+the context of bagging classification trees, we can add up the total
+amount that the Gini index is decreased by splits over a given
+predictor, averaged over all \( B \) trees.
+ Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with
+a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points \( n \)).
+ The plain decision trees suffer from high
-variance. This means that if we split the training data into two parts
-at random, and fit a decision tree to both halves, the results that we
-get could be quite different. In contrast, a procedure with low
-variance will yield similar results if applied repeatedly to distinct
-data sets; linear regression tends to have low variance, if the ratio
-of \( n \) to \( p \) is moderately large.
- Bootstrap aggregation, or just bagging, is a
-general-purpose procedure for reducing the variance of a statistical
-learning method.
- Bagging typically results in improved accuracy
-over prediction using a single tree. Unfortunately, however, it can be
-difficult to interpret the resulting model. Recall that one of the
-advantages of decision trees is the attractive and easily interpreted
-diagram that results.
- However, when we bag a large number of trees, it is no longer
-possible to represent the resulting statistical learning procedure
-using a single tree, and it is no longer clear which variables are
-most important to the procedure. Thus, bagging improves prediction
-accuracy at the expense of interpretability. Although the collection
-of bagged trees is much more difficult to interpret than a single
-tree, one can obtain an overall summary of the importance of each
-predictor using the MSE (for bagging regression trees) or the Gini
-index (for bagging classification trees). In the case of bagging
-regression trees, we can record the total amount that the MSE is
-decreased due to splits over a given predictor, averaged over all \( B \) possible
-trees. A large value indicates an important predictor. Similarly, in
-the context of bagging classification trees, we can add up the total
-amount that the Gini index is decreased by splits over a given
-predictor, averaged over all \( B \) trees.
- Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with
-a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points \( n \)).
- The plain decision trees suffer from high
+variance. This means that if we split the training data into two parts
+at random, and fit a decision tree to both halves, the results that we
+get could be quite different. In contrast, a procedure with low
+variance will yield similar results if applied repeatedly to distinct
+data sets; linear regression tends to have low variance, if the ratio
+of \( n \) to \( p \) is moderately large.
+ Bootstrap aggregation, or just bagging, is a
+general-purpose procedure for reducing the variance of a statistical
+learning method.
+ Bagging typically results in improved accuracy
+over prediction using a single tree. Unfortunately, however, it can be
+difficult to interpret the resulting model. Recall that one of the
+advantages of decision trees is the attractive and easily interpreted
+diagram that results.
+ However, when we bag a large number of trees, it is no longer
+possible to represent the resulting statistical learning procedure
+using a single tree, and it is no longer clear which variables are
+most important to the procedure. Thus, bagging improves prediction
+accuracy at the expense of interpretability. Although the collection
+of bagged trees is much more difficult to interpret than a single
+tree, one can obtain an overall summary of the importance of each
+predictor using the MSE (for bagging regression trees) or the Gini
+index (for bagging classification trees). In the case of bagging
+regression trees, we can record the total amount that the MSE is
+decreased due to splits over a given predictor, averaged over all \( B \) possible
+trees. A large value indicates an important predictor. Similarly, in
+the context of bagging classification trees, we can add up the total
+amount that the Gini index is decreased by splits over a given
+predictor, averaged over all \( B \) trees.
+ Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with
+a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points \( n \)).
+ The plain decision trees suffer from high
-variance. This means that if we split the training data into two parts
-at random, and fit a decision tree to both halves, the results that we
-get could be quite different. In contrast, a procedure with low
-variance will yield similar results if applied repeatedly to distinct
-data sets; linear regression tends to have low variance, if the ratio
-of \( n \) to \( p \) is moderately large.
- Bootstrap aggregation, or just bagging, is a
-general-purpose procedure for reducing the variance of a statistical
-learning method.
- Bagging typically results in improved accuracy
-over prediction using a single tree. Unfortunately, however, it can be
-difficult to interpret the resulting model. Recall that one of the
-advantages of decision trees is the attractive and easily interpreted
-diagram that results.
- However, when we bag a large number of trees, it is no longer
-possible to represent the resulting statistical learning procedure
-using a single tree, and it is no longer clear which variables are
-most important to the procedure. Thus, bagging improves prediction
-accuracy at the expense of interpretability. Although the collection
-of bagged trees is much more difficult to interpret than a single
-tree, one can obtain an overall summary of the importance of each
-predictor using the MSE (for bagging regression trees) or the Gini
-index (for bagging classification trees). In the case of bagging
-regression trees, we can record the total amount that the MSE is
-decreased due to splits over a given predictor, averaged over all \( B \) possible
-trees. A large value indicates an important predictor. Similarly, in
-the context of bagging classification trees, we can add up the total
-amount that the Gini index is decreased by splits over a given
-predictor, averaged over all \( B \) trees.
- Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with
-a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points \( n \)).
- The plain decision trees suffer from high
+variance. This means that if we split the training data into two parts
+at random, and fit a decision tree to both halves, the results that we
+get could be quite different. In contrast, a procedure with low
+variance will yield similar results if applied repeatedly to distinct
+data sets; linear regression tends to have low variance, if the ratio
+of \( n \) to \( p \) is moderately large.
+ Bootstrap aggregation, or just bagging, is a
+general-purpose procedure for reducing the variance of a statistical
+learning method.
+ Bagging typically results in improved accuracy
+over prediction using a single tree. Unfortunately, however, it can be
+difficult to interpret the resulting model. Recall that one of the
+advantages of decision trees is the attractive and easily interpreted
+diagram that results.
+ However, when we bag a large number of trees, it is no longer
+possible to represent the resulting statistical learning procedure
+using a single tree, and it is no longer clear which variables are
+most important to the procedure. Thus, bagging improves prediction
+accuracy at the expense of interpretability. Although the collection
+of bagged trees is much more difficult to interpret than a single
+tree, one can obtain an overall summary of the importance of each
+predictor using the MSE (for bagging regression trees) or the Gini
+index (for bagging classification trees). In the case of bagging
+regression trees, we can record the total amount that the MSE is
+decreased due to splits over a given predictor, averaged over all \( B \) possible
+trees. A large value indicates an important predictor. Similarly, in
+the context of bagging classification trees, we can add up the total
+amount that the Gini index is decreased by splits over a given
+predictor, averaged over all \( B \) trees.
+ Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with
+a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points \( n \)).
+Simple Voting Example, head or tail
-
-
-# Common imports
-import numpy as np
-import matplotlib
-import matplotlib.pyplot as plt
-from matplotlib.colors import ListedColormap
-plt.rcParams['axes.labelsize'] = 14
-plt.rcParams['xtick.labelsize'] = 12
-plt.rcParams['ytick.labelsize'] = 12
-
-heads_proba = 0.51
-coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
-cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)
-plt.figure(figsize=(8,3.5))
-plt.plot(cumulative_heads_ratio)
-plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
-plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
-plt.xlabel("Number of coin tosses")
-plt.ylabel("Heads ratio")
-plt.legend(loc="lower right")
-plt.axis([0, 10000, 0.42, 0.58])
-save_fig("votingsimple")
-plt.show()
-
-Random Forest Algorithm
+
+
+
+
+
+Using the Voting Classifier
-
-Random Forests Compared with other Methods on the Cancer Data
from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
+
import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import load_breast_cancer
+from sklearn.svm import SVC
+from sklearn.linear_model import LogisticRegression
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.ensemble import BaggingClassifier
+
+# Load the data
+cancer = load_breast_cancer()
+
+X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
+print(X_train.shape)
+print(X_test.shape)
+#define methods
+# Logistic Regression
+logreg = LogisticRegression(solver='lbfgs')
+# Support vector machine
+svm = SVC(gamma='auto', C=100)
+# Decision Trees
+deep_tree_clf = DecisionTreeClassifier(max_depth=None)
+#Scale the data
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+# Logistic Regression
+logreg.fit(X_train_scaled, y_train)
+print("Test set accuracy Logistic Regression with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
+# Support Vector Machine
+svm.fit(X_train_scaled, y_train)
+print("Test set accuracy SVM with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
+# Decision Trees
+deep_tree_clf.fit(X_train_scaled, y_train)
+print("Test set accuracy with Decision Trees and scaled data: {:.2f}".format(deep_tree_clf.score(X_test_scaled,y_test)))
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
+from sklearn.preprocessing import LabelEncoder
+from sklearn.model_selection import cross_validate
+# Data set not specificied
+#Instantiate the model with 500 trees and entropy as splitting criteria
+Random_Forest_model = RandomForestClassifier(n_estimators=500,criterion="entropy")
+Random_Forest_model.fit(X_train_scaled, y_train)
+#Cross validation
+accuracy = cross_validate(Random_Forest_model,X_test_scaled,y_test,cv=10)['test_score']
+print(accuracy)
+print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(Random_Forest_model.score(X_test_scaled,y_test)))
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", random_state=42)
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", probability=True, random_state=42)
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
+import scikitplot as skplt
+y_pred = Random_Forest_model.predict(X_test_scaled)
+skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
+plt.show()
+y_probas = Random_Forest_model.predict_proba(X_test_scaled)
+skplt.metrics.plot_roc(y_test, y_probas)
+plt.show()
+skplt.metrics.plot_cumulative_gain(y_test, y_probas)
+plt.show()
Voting and Bagging
-
+Compare Bagging on Trees with Random Forests
from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
-
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
+
bag_clf = BaggingClassifier(
+ DecisionTreeClassifier(splitter="random", max_leaf_nodes=16, random_state=42),
+ n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1, random_state=42)
+
+bag_clf.fit(X_train, y_train)
+y_pred = bag_clf.predict(X_test)
from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
+rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16, n_jobs=-1, random_state=42)
+rnd_clf.fit(X_train, y_train)
+y_pred_rf = rnd_clf.predict(X_test)
+np.sum(y_pred == y_pred_rf) / len(y_pred)
Another example, the moons again
+Another example, the moons
-Bagging
-
-More bagging
-
-Simple Voting Example, head or tail
-
-
-heads_proba = 0.51
-coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
-cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)
-plt.figure(figsize=(8,3.5))
-plt.plot(cumulative_heads_ratio)
-plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
-plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
-plt.xlabel("Number of coin tosses")
-plt.ylabel("Heads ratio")
-plt.legend(loc="lower right")
-plt.axis([0, 10000, 0.42, 0.58])
-save_fig("votingsimple")
-plt.show()
-
-Using the Voting Classifier
-
-
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
-
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-Please, not the moons again! Voting and Bagging
-
-
-
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
-
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-Bagging Examples
-
-
-
-from sklearn.ensemble import BaggingClassifier
-from sklearn.tree import DecisionTreeClassifier
-
-bag_clf = BaggingClassifier(
- DecisionTreeClassifier(random_state=42), n_estimators=500,
- max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
-bag_clf.fit(X_train, y_train)
-y_pred = bag_clf.predict(X_test)
-
-from sklearn.metrics import accuracy_score
-print(accuracy_score(y_test, y_pred))
-
-tree_clf = DecisionTreeClassifier(random_state=42)
-tree_clf.fit(X_train, y_train)
-y_pred_tree = tree_clf.predict(X_test)
-print(accuracy_score(y_test, y_pred_tree))
-
-from matplotlib.colors import ListedColormap
-
-def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
- x1s = np.linspace(axes[0], axes[1], 100)
- x2s = np.linspace(axes[2], axes[3], 100)
- x1, x2 = np.meshgrid(x1s, x2s)
- X_new = np.c_[x1.ravel(), x2.ravel()]
- y_pred = clf.predict(X_new).reshape(x1.shape)
- custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
- plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
- if contour:
- custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
- plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
- plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
- plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
- plt.axis(axes)
- plt.xlabel(r"$x_1$", fontsize=18)
- plt.ylabel(r"$x_2$", fontsize=18, rotation=0)
-plt.figure(figsize=(11,4))
-plt.subplot(121)
-plot_decision_boundary(tree_clf, X, y)
-plt.title("Decision Tree", fontsize=14)
-plt.subplot(122)
-plot_decision_boundary(bag_clf, X, y)
-plt.title("Decision Trees with Bagging", fontsize=14)
-save_fig("baggingtree")
-plt.show()
-
-Making your own Bootstrap: Changing the Level of the Decision Tree
-
-import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.pipeline import make_pipeline
-from sklearn.utils import resample
-from sklearn.tree import DecisionTreeRegressor
-
-n = 100
-n_boostraps = 100
-maxdepth = 8
-
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-error = np.zeros(maxdepth)
-bias = np.zeros(maxdepth)
-variance = np.zeros(maxdepth)
-polydegree = np.zeros(maxdepth)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
-
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-# we produce a simple tree first as benchmark
-simpletree = DecisionTreeRegressor(max_depth=3)
-simpletree.fit(X_train_scaled, y_train)
-simpleprediction = simpletree.predict(X_test_scaled)
-for degree in range(1,maxdepth):
- model = DecisionTreeRegressor(max_depth=degree)
- y_pred = np.empty((y_test.shape[0], n_boostraps))
- for i in range(n_boostraps):
- x_, y_ = resample(X_train_scaled, y_train)
- model.fit(x_, y_)
- y_pred[:, i] = model.predict(X_test_scaled)#.ravel()
-
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
- variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
- print('Polynomial degree:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-
-mse_simpletree= np.mean( np.mean((y_test - simpleprediction)**2)
-print(mse_simpletree)
-plt.xlim(1,maxdepth)
-plt.plot(polydegree, error, label='MSE')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
-save_fig("baggingboot")
-plt.show()
-
-Why Voting?
@@ -2729,6 +2246,138 @@ voting_clf.fit(X_train, y_train)
Bagging
+
+More bagging
+
+Making your own Bootstrap: Changing the Level of the Decision Tree
+
+import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+from sklearn.tree import DecisionTreeRegressor
+
+n = 100
+n_boostraps = 100
+maxdepth = 8
+
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+error = np.zeros(maxdepth)
+bias = np.zeros(maxdepth)
+variance = np.zeros(maxdepth)
+polydegree = np.zeros(maxdepth)
+X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+
+# we produce a simple tree first as benchmark
+simpletree = DecisionTreeRegressor(max_depth=3)
+simpletree.fit(X_train_scaled, y_train)
+simpleprediction = simpletree.predict(X_test_scaled)
+for degree in range(1,maxdepth):
+ model = DecisionTreeRegressor(max_depth=degree)
+ y_pred = np.empty((y_test.shape[0], n_boostraps))
+ for i in range(n_boostraps):
+ x_, y_ = resample(X_train_scaled, y_train)
+ model.fit(x_, y_)
+ y_pred[:, i] = model.predict(X_test_scaled)#.ravel()
+
+ polydegree[degree] = degree
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+ variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+ print('Polynomial degree:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+
+mse_simpletree= np.mean( np.mean((y_test - simpleprediction)**2)
+print(mse_simpletree)
+plt.xlim(1,maxdepth)
+plt.plot(polydegree, error, label='MSE')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+save_fig("baggingboot")
+plt.show()
+
+Random forests
@@ -2825,19 +2474,14 @@ cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
print(X_train.shape)
print(X_test.shape)
+#define methods
# Logistic Regression
logreg = LogisticRegression(solver='lbfgs')
-logreg.fit(X_train, y_train)
-print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
# Support vector machine
svm = SVC(gamma='auto', C=100)
-svm.fit(X_train, y_train)
-print("Test set accuracy with SVM: {:.2f}".format(svm.score(X_test,y_test)))
# Decision Trees
deep_tree_clf = DecisionTreeClassifier(max_depth=None)
-deep_tree_clf.fit(X_train, y_train)
-print("Test set accuracy with Decision Trees: {:.2f}".format(deep_tree_clf.score(X_test,y_test)))
-#now scale the data
+#Scale the data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
diff --git a/doc/pub/week44/html/week44-solarized.html b/doc/pub/week44/html/week44-solarized.html
index bcfd96b38..cfb9094ee 100644
--- a/doc/pub/week44/html/week44-solarized.html
+++ b/doc/pub/week44/html/week44-solarized.html
@@ -176,10 +176,10 @@ div.toc p,a {
2,
None,
'computing-the-gini-factor'),
- ('Another example, the moons again',
+ ('Another example, the moons',
2,
None,
- 'another-example-the-moons-again'),
+ 'another-example-the-moons'),
('Playing around with regions',
2,
None,
@@ -200,26 +200,6 @@ div.toc p,a {
2,
None,
'an-overview-of-ensemble-methods'),
- ('Bagging', 2, None, 'bagging'),
- ('More bagging', 2, None, 'more-bagging'),
- ('Simple Voting Example, head or tail',
- 2,
- None,
- 'simple-voting-example-head-or-tail'),
- ('Using the Voting Classifier',
- 2,
- None,
- 'using-the-voting-classifier'),
- ('Please, not the moons again! Voting and Bagging',
- 2,
- None,
- 'please-not-the-moons-again-voting-and-bagging'),
- ('Bagging Examples', 2, None, 'bagging-examples'),
- ('Making your own Bootstrap: Changing the Level of the Decision '
- 'Tree',
- 2,
- None,
- 'making-your-own-bootstrap-changing-the-level-of-the-decision-tree'),
('Why Voting?', 2, None, 'why-voting'),
('Tossing coins', 2, None, 'tossing-coins'),
('Standard imports first', 2, None, 'standard-imports-first'),
@@ -232,6 +212,13 @@ div.toc p,a {
None,
'using-the-voting-classifier'),
('Voting and Bagging', 2, None, 'voting-and-bagging'),
+ ('Bagging', 2, None, 'bagging'),
+ ('More bagging', 2, None, 'more-bagging'),
+ ('Making your own Bootstrap: Changing the Level of the Decision '
+ 'Tree',
+ 2,
+ None,
+ 'making-your-own-bootstrap-changing-the-level-of-the-decision-tree'),
('Random forests', 2, None, 'random-forests'),
('Random Forest Algorithm', 2, None, 'random-forest-algorithm'),
('Random Forests Compared with other Methods on the Cancer Data',
@@ -1556,7 +1543,7 @@ split = get_split(dataset)
-Another example, the moons again
+Another example, the moons
-
-Bagging
-
-
-More bagging
-
-
-Simple Voting Example, head or tail
-
-
-heads_proba = 0.51
-coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
-cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)
-plt.figure(figsize=(8,3.5))
-plt.plot(cumulative_heads_ratio)
-plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
-plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
-plt.xlabel("Number of coin tosses")
-plt.ylabel("Heads ratio")
-plt.legend(loc="lower right")
-plt.axis([0, 10000, 0.42, 0.58])
-save_fig("votingsimple")
-plt.show()
-
-
-Using the Voting Classifier
-
-
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
-
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-
-Please, not the moons again! Voting and Bagging
-
-
-
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
-
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-
-Bagging Examples
-
-
-
-from sklearn.ensemble import BaggingClassifier
-from sklearn.tree import DecisionTreeClassifier
-
-bag_clf = BaggingClassifier(
- DecisionTreeClassifier(random_state=42), n_estimators=500,
- max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
-bag_clf.fit(X_train, y_train)
-y_pred = bag_clf.predict(X_test)
-
-from sklearn.metrics import accuracy_score
-print(accuracy_score(y_test, y_pred))
-
-tree_clf = DecisionTreeClassifier(random_state=42)
-tree_clf.fit(X_train, y_train)
-y_pred_tree = tree_clf.predict(X_test)
-print(accuracy_score(y_test, y_pred_tree))
-
-from matplotlib.colors import ListedColormap
-
-def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
- x1s = np.linspace(axes[0], axes[1], 100)
- x2s = np.linspace(axes[2], axes[3], 100)
- x1, x2 = np.meshgrid(x1s, x2s)
- X_new = np.c_[x1.ravel(), x2.ravel()]
- y_pred = clf.predict(X_new).reshape(x1.shape)
- custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
- plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
- if contour:
- custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
- plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
- plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
- plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
- plt.axis(axes)
- plt.xlabel(r"$x_1$", fontsize=18)
- plt.ylabel(r"$x_2$", fontsize=18, rotation=0)
-plt.figure(figsize=(11,4))
-plt.subplot(121)
-plot_decision_boundary(tree_clf, X, y)
-plt.title("Decision Tree", fontsize=14)
-plt.subplot(122)
-plot_decision_boundary(bag_clf, X, y)
-plt.title("Decision Trees with Bagging", fontsize=14)
-save_fig("baggingtree")
-plt.show()
-
-
-Making your own Bootstrap: Changing the Level of the Decision Tree
-
-import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.pipeline import make_pipeline
-from sklearn.utils import resample
-from sklearn.tree import DecisionTreeRegressor
-
-n = 100
-n_boostraps = 100
-maxdepth = 8
-
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-error = np.zeros(maxdepth)
-bias = np.zeros(maxdepth)
-variance = np.zeros(maxdepth)
-polydegree = np.zeros(maxdepth)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
-
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-# we produce a simple tree first as benchmark
-simpletree = DecisionTreeRegressor(max_depth=3)
-simpletree.fit(X_train_scaled, y_train)
-simpleprediction = simpletree.predict(X_test_scaled)
-for degree in range(1,maxdepth):
- model = DecisionTreeRegressor(max_depth=degree)
- y_pred = np.empty((y_test.shape[0], n_boostraps))
- for i in range(n_boostraps):
- x_, y_ = resample(X_train_scaled, y_train)
- model.fit(x_, y_)
- y_pred[:, i] = model.predict(X_test_scaled)#.ravel()
-
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
- variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
- print('Polynomial degree:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-
-mse_simpletree= np.mean( np.mean((y_test - simpleprediction)**2)
-print(mse_simpletree)
-plt.xlim(1,maxdepth)
-plt.plot(polydegree, error, label='MSE')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
-save_fig("baggingboot")
-plt.show()
-
-
Why Voting?
@@ -2744,6 +2250,136 @@ voting_clf.fit(X_train, y_train)
+Bagging
+
+
+More bagging
+
+
+Making your own Bootstrap: Changing the Level of the Decision Tree
+
+import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+from sklearn.tree import DecisionTreeRegressor
+
+n = 100
+n_boostraps = 100
+maxdepth = 8
+
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+error = np.zeros(maxdepth)
+bias = np.zeros(maxdepth)
+variance = np.zeros(maxdepth)
+polydegree = np.zeros(maxdepth)
+X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+
+# we produce a simple tree first as benchmark
+simpletree = DecisionTreeRegressor(max_depth=3)
+simpletree.fit(X_train_scaled, y_train)
+simpleprediction = simpletree.predict(X_test_scaled)
+for degree in range(1,maxdepth):
+ model = DecisionTreeRegressor(max_depth=degree)
+ y_pred = np.empty((y_test.shape[0], n_boostraps))
+ for i in range(n_boostraps):
+ x_, y_ = resample(X_train_scaled, y_train)
+ model.fit(x_, y_)
+ y_pred[:, i] = model.predict(X_test_scaled)#.ravel()
+
+ polydegree[degree] = degree
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+ variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+ print('Polynomial degree:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+
+mse_simpletree= np.mean( np.mean((y_test - simpleprediction)**2)
+print(mse_simpletree)
+plt.xlim(1,maxdepth)
+plt.plot(polydegree, error, label='MSE')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+save_fig("baggingboot")
+plt.show()
+
+
Random forests
@@ -2828,19 +2464,14 @@ cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
print(X_train.shape)
print(X_test.shape)
+#define methods
# Logistic Regression
logreg = LogisticRegression(solver='lbfgs')
-logreg.fit(X_train, y_train)
-print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
# Support vector machine
svm = SVC(gamma='auto', C=100)
-svm.fit(X_train, y_train)
-print("Test set accuracy with SVM: {:.2f}".format(svm.score(X_test,y_test)))
# Decision Trees
deep_tree_clf = DecisionTreeClassifier(max_depth=None)
-deep_tree_clf.fit(X_train, y_train)
-print("Test set accuracy with Decision Trees: {:.2f}".format(deep_tree_clf.score(X_test,y_test)))
-#now scale the data
+#Scale the data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
diff --git a/doc/pub/week44/html/week44.html b/doc/pub/week44/html/week44.html
index 3d7b78d9f..9b2b756de 100644
--- a/doc/pub/week44/html/week44.html
+++ b/doc/pub/week44/html/week44.html
@@ -253,10 +253,10 @@ div.toc p,a {
2,
None,
'computing-the-gini-factor'),
- ('Another example, the moons again',
+ ('Another example, the moons',
2,
None,
- 'another-example-the-moons-again'),
+ 'another-example-the-moons'),
('Playing around with regions',
2,
None,
@@ -277,26 +277,6 @@ div.toc p,a {
2,
None,
'an-overview-of-ensemble-methods'),
- ('Bagging', 2, None, 'bagging'),
- ('More bagging', 2, None, 'more-bagging'),
- ('Simple Voting Example, head or tail',
- 2,
- None,
- 'simple-voting-example-head-or-tail'),
- ('Using the Voting Classifier',
- 2,
- None,
- 'using-the-voting-classifier'),
- ('Please, not the moons again! Voting and Bagging',
- 2,
- None,
- 'please-not-the-moons-again-voting-and-bagging'),
- ('Bagging Examples', 2, None, 'bagging-examples'),
- ('Making your own Bootstrap: Changing the Level of the Decision '
- 'Tree',
- 2,
- None,
- 'making-your-own-bootstrap-changing-the-level-of-the-decision-tree'),
('Why Voting?', 2, None, 'why-voting'),
('Tossing coins', 2, None, 'tossing-coins'),
('Standard imports first', 2, None, 'standard-imports-first'),
@@ -309,6 +289,13 @@ div.toc p,a {
None,
'using-the-voting-classifier'),
('Voting and Bagging', 2, None, 'voting-and-bagging'),
+ ('Bagging', 2, None, 'bagging'),
+ ('More bagging', 2, None, 'more-bagging'),
+ ('Making your own Bootstrap: Changing the Level of the Decision '
+ 'Tree',
+ 2,
+ None,
+ 'making-your-own-bootstrap-changing-the-level-of-the-decision-tree'),
('Random forests', 2, None, 'random-forests'),
('Random Forest Algorithm', 2, None, 'random-forest-algorithm'),
('Random Forests Compared with other Methods on the Cancer Data',
@@ -1633,7 +1620,7 @@ split = get_split(dataset)
-Another example, the moons again
+Another example, the moons
-
-Bagging
-
-
-More bagging
-
-
-Simple Voting Example, head or tail
-
-
-heads_proba = 0.51
-coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
-cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)
-plt.figure(figsize=(8,3.5))
-plt.plot(cumulative_heads_ratio)
-plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
-plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
-plt.xlabel("Number of coin tosses")
-plt.ylabel("Heads ratio")
-plt.legend(loc="lower right")
-plt.axis([0, 10000, 0.42, 0.58])
-save_fig("votingsimple")
-plt.show()
-
-
-Using the Voting Classifier
-
-
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
-
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-
-Please, not the moons again! Voting and Bagging
-
-
-
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
-
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-
-Bagging Examples
-
-
-
-from sklearn.ensemble import BaggingClassifier
-from sklearn.tree import DecisionTreeClassifier
-
-bag_clf = BaggingClassifier(
- DecisionTreeClassifier(random_state=42), n_estimators=500,
- max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
-bag_clf.fit(X_train, y_train)
-y_pred = bag_clf.predict(X_test)
-
-from sklearn.metrics import accuracy_score
-print(accuracy_score(y_test, y_pred))
-
-tree_clf = DecisionTreeClassifier(random_state=42)
-tree_clf.fit(X_train, y_train)
-y_pred_tree = tree_clf.predict(X_test)
-print(accuracy_score(y_test, y_pred_tree))
-
-from matplotlib.colors import ListedColormap
-
-def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
- x1s = np.linspace(axes[0], axes[1], 100)
- x2s = np.linspace(axes[2], axes[3], 100)
- x1, x2 = np.meshgrid(x1s, x2s)
- X_new = np.c_[x1.ravel(), x2.ravel()]
- y_pred = clf.predict(X_new).reshape(x1.shape)
- custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
- plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
- if contour:
- custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
- plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
- plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
- plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
- plt.axis(axes)
- plt.xlabel(r"$x_1$", fontsize=18)
- plt.ylabel(r"$x_2$", fontsize=18, rotation=0)
-plt.figure(figsize=(11,4))
-plt.subplot(121)
-plot_decision_boundary(tree_clf, X, y)
-plt.title("Decision Tree", fontsize=14)
-plt.subplot(122)
-plot_decision_boundary(bag_clf, X, y)
-plt.title("Decision Trees with Bagging", fontsize=14)
-save_fig("baggingtree")
-plt.show()
-
-
-Making your own Bootstrap: Changing the Level of the Decision Tree
-
-import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.pipeline import make_pipeline
-from sklearn.utils import resample
-from sklearn.tree import DecisionTreeRegressor
-
-n = 100
-n_boostraps = 100
-maxdepth = 8
-
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-error = np.zeros(maxdepth)
-bias = np.zeros(maxdepth)
-variance = np.zeros(maxdepth)
-polydegree = np.zeros(maxdepth)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
-
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-# we produce a simple tree first as benchmark
-simpletree = DecisionTreeRegressor(max_depth=3)
-simpletree.fit(X_train_scaled, y_train)
-simpleprediction = simpletree.predict(X_test_scaled)
-for degree in range(1,maxdepth):
- model = DecisionTreeRegressor(max_depth=degree)
- y_pred = np.empty((y_test.shape[0], n_boostraps))
- for i in range(n_boostraps):
- x_, y_ = resample(X_train_scaled, y_train)
- model.fit(x_, y_)
- y_pred[:, i] = model.predict(X_test_scaled)#.ravel()
-
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
- variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
- print('Polynomial degree:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-
-mse_simpletree= np.mean( np.mean((y_test - simpleprediction)**2)
-print(mse_simpletree)
-plt.xlim(1,maxdepth)
-plt.plot(polydegree, error, label='MSE')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
-save_fig("baggingboot")
-plt.show()
-
-
Why Voting?
@@ -2821,6 +2327,136 @@ voting_clf.fit(X_train, y_train)
+Bagging
+
+
+More bagging
+
+
+Making your own Bootstrap: Changing the Level of the Decision Tree
+
+import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+from sklearn.tree import DecisionTreeRegressor
+
+n = 100
+n_boostraps = 100
+maxdepth = 8
+
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+error = np.zeros(maxdepth)
+bias = np.zeros(maxdepth)
+variance = np.zeros(maxdepth)
+polydegree = np.zeros(maxdepth)
+X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+
+# we produce a simple tree first as benchmark
+simpletree = DecisionTreeRegressor(max_depth=3)
+simpletree.fit(X_train_scaled, y_train)
+simpleprediction = simpletree.predict(X_test_scaled)
+for degree in range(1,maxdepth):
+ model = DecisionTreeRegressor(max_depth=degree)
+ y_pred = np.empty((y_test.shape[0], n_boostraps))
+ for i in range(n_boostraps):
+ x_, y_ = resample(X_train_scaled, y_train)
+ model.fit(x_, y_)
+ y_pred[:, i] = model.predict(X_test_scaled)#.ravel()
+
+ polydegree[degree] = degree
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+ variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+ print('Polynomial degree:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+
+mse_simpletree= np.mean( np.mean((y_test - simpleprediction)**2)
+print(mse_simpletree)
+plt.xlim(1,maxdepth)
+plt.plot(polydegree, error, label='MSE')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+save_fig("baggingboot")
+plt.show()
+
+
Random forests
@@ -2905,19 +2541,14 @@ cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
print(X_train.shape)
print(X_test.shape)
+#define methods
# Logistic Regression
logreg = LogisticRegression(solver='lbfgs')
-logreg.fit(X_train, y_train)
-print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
# Support vector machine
svm = SVC(gamma='auto', C=100)
-svm.fit(X_train, y_train)
-print("Test set accuracy with SVM: {:.2f}".format(svm.score(X_test,y_test)))
# Decision Trees
deep_tree_clf = DecisionTreeClassifier(max_depth=None)
-deep_tree_clf.fit(X_train, y_train)
-print("Test set accuracy with Decision Trees: {:.2f}".format(deep_tree_clf.score(X_test,y_test)))
-#now scale the data
+#Scale the data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
diff --git a/doc/pub/week44/ipynb/Datafiles/cancer.dot b/doc/pub/week44/ipynb/Datafiles/cancer.dot
index c000d8c4c..75af04abe 100644
--- a/doc/pub/week44/ipynb/Datafiles/cancer.dot
+++ b/doc/pub/week44/ipynb/Datafiles/cancer.dot
@@ -1,22 +1,22 @@
digraph Tree {
-node [shape=box, style="filled, rounded", color="black", fontname=helvetica] ;
-edge [fontname=helvetica] ;
+node [shape=box, style="filled, rounded", color="black", fontname="helvetica"] ;
+edge [fontname="helvetica"] ;
0 [label="worst perimeter <= 106.05\ngini = 0.465\nsamples = 426\nvalue = [[269, 157]\n[157, 269]]", fillcolor="#fefbf9"] ;
1 [label="worst concave points <= 0.159\ngini = 0.067\nsamples = 259\nvalue = [[250, 9]\n[9, 250]]", fillcolor="#e99355"] ;
0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ;
2 [label="worst concave points <= 0.135\ngini = 0.031\nsamples = 253\nvalue = [[249, 4]\n[4, 249]]", fillcolor="#e78946"] ;
1 -> 2 ;
-3 [label="radius error <= 0.643\ngini = 0.008\nsamples = 242\nvalue = [[241, 1]\n[1, 241]]", fillcolor="#e5833c"] ;
+3 [label="area error <= 48.975\ngini = 0.008\nsamples = 242\nvalue = [[241, 1]\n[1, 241]]", fillcolor="#e5833c"] ;
2 -> 3 ;
4 [label="gini = 0.0\nsamples = 239\nvalue = [[239, 0]\n[0, 239]]", fillcolor="#e58139"] ;
3 -> 4 ;
-5 [label="concavity error <= 0.017\ngini = 0.444\nsamples = 3\nvalue = [[2, 1]\n[1, 2]]", fillcolor="#fdf6f0"] ;
+5 [label="perimeter error <= 4.249\ngini = 0.444\nsamples = 3\nvalue = [[2, 1]\n[1, 2]]", fillcolor="#fdf6f0"] ;
3 -> 5 ;
6 [label="gini = 0.0\nsamples = 1\nvalue = [[0, 1]\n[1, 0]]", fillcolor="#e58139"] ;
5 -> 6 ;
7 [label="gini = 0.0\nsamples = 2\nvalue = [[2, 0]\n[0, 2]]", fillcolor="#e58139"] ;
5 -> 7 ;
-8 [label="worst texture <= 29.455\ngini = 0.397\nsamples = 11\nvalue = [[8, 3]\n[3, 8]]", fillcolor="#fae9dd"] ;
+8 [label="mean texture <= 20.84\ngini = 0.397\nsamples = 11\nvalue = [[8, 3]\n[3, 8]]", fillcolor="#fae9dd"] ;
2 -> 8 ;
9 [label="gini = 0.0\nsamples = 8\nvalue = [[8, 0]\n[0, 8]]", fillcolor="#e58139"] ;
8 -> 9 ;
@@ -48,10 +48,10 @@ edge [fontname=helvetica] ;
21 -> 22 ;
23 [label="gini = 0.0\nsamples = 6\nvalue = [[6, 0]\n[0, 6]]", fillcolor="#e58139"] ;
21 -> 23 ;
-24 [label="fractal dimension error <= 0.013\ngini = 0.015\nsamples = 136\nvalue = [[1, 135]\n[135, 1]]", fillcolor="#e6853f"] ;
+24 [label="mean smoothness <= 0.079\ngini = 0.015\nsamples = 136\nvalue = [[1, 135]\n[135, 1]]", fillcolor="#e6853f"] ;
20 -> 24 ;
-25 [label="gini = 0.0\nsamples = 135\nvalue = [[0, 135]\n[135, 0]]", fillcolor="#e58139"] ;
+25 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139"] ;
24 -> 25 ;
-26 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139"] ;
+26 [label="gini = 0.0\nsamples = 135\nvalue = [[0, 135]\n[135, 0]]", fillcolor="#e58139"] ;
24 -> 26 ;
}
\ No newline at end of file
diff --git a/doc/pub/week44/ipynb/Datafiles/cancer.png b/doc/pub/week44/ipynb/Datafiles/cancer.png
index 5f003c0bb..c0f1f7548 100644
Binary files a/doc/pub/week44/ipynb/Datafiles/cancer.png and b/doc/pub/week44/ipynb/Datafiles/cancer.png differ
diff --git a/doc/pub/week44/ipynb/Datafiles/moons.dot b/doc/pub/week44/ipynb/Datafiles/moons.dot
index 05d91c44c..b06d0a543 100644
--- a/doc/pub/week44/ipynb/Datafiles/moons.dot
+++ b/doc/pub/week44/ipynb/Datafiles/moons.dot
@@ -1,6 +1,6 @@
digraph Tree {
-node [shape=box, style="filled, rounded", color="black", fontname=helvetica] ;
-edge [fontname=helvetica] ;
+node [shape=box, style="filled, rounded", color="black", fontname="helvetica"] ;
+edge [fontname="helvetica"] ;
0 [label="X[1] <= 0.115\ngini = 0.5\nsamples = 75\nvalue = [37, 38]", fillcolor="#fafcfe"] ;
1 [label="X[0] <= -0.686\ngini = 0.225\nsamples = 31\nvalue = [4, 27]", fillcolor="#56ace9"] ;
0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ;
diff --git a/doc/pub/week44/ipynb/Datafiles/ride.dot b/doc/pub/week44/ipynb/Datafiles/ride.dot
index b9d720cda..4e3b65b18 100644
--- a/doc/pub/week44/ipynb/Datafiles/ride.dot
+++ b/doc/pub/week44/ipynb/Datafiles/ride.dot
@@ -1,6 +1,6 @@
digraph Tree {
-node [shape=box, style="filled, rounded", color="black", fontname=helvetica] ;
-edge [fontname=helvetica] ;
+node [shape=box, style="filled, rounded", color="black", fontname="helvetica"] ;
+edge [fontname="helvetica"] ;
0 [label="X[9] <= 0.5\ngini = 0.48\nsamples = 15\nvalue = [4, 10, 1]", fillcolor="#93f1ba"] ;
1 [label="X[1] <= 0.5\ngini = 0.408\nsamples = 14\nvalue = [4, 10, 0]", fillcolor="#88efb3"] ;
0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ;
diff --git a/doc/pub/week44/ipynb/Results/FigureFiles/baggingboot.png b/doc/pub/week44/ipynb/Results/FigureFiles/baggingboot.png
index d1e3e934f..512a5a20f 100644
Binary files a/doc/pub/week44/ipynb/Results/FigureFiles/baggingboot.png and b/doc/pub/week44/ipynb/Results/FigureFiles/baggingboot.png differ
diff --git a/doc/pub/week44/ipynb/Results/FigureFiles/baggingtree.png b/doc/pub/week44/ipynb/Results/FigureFiles/baggingtree.png
index 5c83b6226..29aa92b46 100644
Binary files a/doc/pub/week44/ipynb/Results/FigureFiles/baggingtree.png and b/doc/pub/week44/ipynb/Results/FigureFiles/baggingtree.png differ
diff --git a/doc/pub/week44/ipynb/Results/FigureFiles/votingsimple.png b/doc/pub/week44/ipynb/Results/FigureFiles/votingsimple.png
index 7fa095803..7ba47edbc 100644
Binary files a/doc/pub/week44/ipynb/Results/FigureFiles/votingsimple.png and b/doc/pub/week44/ipynb/Results/FigureFiles/votingsimple.png differ
diff --git a/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz b/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz
index ede2ebeca..d1a5e31f5 100644
Binary files a/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz and b/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz differ
diff --git a/doc/pub/week44/ipynb/week44.ipynb b/doc/pub/week44/ipynb/week44.ipynb
index 51a466001..23dd53ac1 100644
--- a/doc/pub/week44/ipynb/week44.ipynb
+++ b/doc/pub/week44/ipynb/week44.ipynb
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
- "id": "5c948236",
+ "id": "4c7a5155",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
- "id": "2e712ddb",
+ "id": "4ac9d0cb",
"metadata": {
"editable": true
},
@@ -29,7 +29,7 @@
},
{
"cell_type": "markdown",
- "id": "ce77630c",
+ "id": "e0a4b5a7",
"metadata": {
"editable": true
},
@@ -51,7 +51,7 @@
},
{
"cell_type": "markdown",
- "id": "fbc49098",
+ "id": "3d46e4eb",
"metadata": {
"editable": true
},
@@ -69,7 +69,7 @@
},
{
"cell_type": "markdown",
- "id": "99be5592",
+ "id": "b7dc5b6f",
"metadata": {
"editable": true
},
@@ -83,7 +83,7 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "321af0c0",
+ "id": "b2d6c39b",
"metadata": {
"collapsed": false,
"editable": true
@@ -199,7 +199,7 @@
},
{
"cell_type": "markdown",
- "id": "b8f4d755",
+ "id": "741b7207",
"metadata": {
"editable": true
},
@@ -230,7 +230,7 @@
},
{
"cell_type": "markdown",
- "id": "0477ca95",
+ "id": "7c2e0aec",
"metadata": {
"editable": true
},
@@ -250,7 +250,7 @@
},
{
"cell_type": "markdown",
- "id": "96526d34",
+ "id": "d6ebe102",
"metadata": {
"editable": true
},
@@ -264,7 +264,7 @@
},
{
"cell_type": "markdown",
- "id": "f1d105d9",
+ "id": "d9789025",
"metadata": {
"editable": true
},
@@ -277,7 +277,7 @@
},
{
"cell_type": "markdown",
- "id": "6def974b",
+ "id": "b21ba8d6",
"metadata": {
"editable": true
},
@@ -295,7 +295,7 @@
},
{
"cell_type": "markdown",
- "id": "4cee402a",
+ "id": "84826586",
"metadata": {
"editable": true
},
@@ -318,7 +318,7 @@
},
{
"cell_type": "markdown",
- "id": "b0ae8c62",
+ "id": "b7b60eb0",
"metadata": {
"editable": true
},
@@ -341,7 +341,7 @@
},
{
"cell_type": "markdown",
- "id": "5cc2e727",
+ "id": "c2729e10",
"metadata": {
"editable": true
},
@@ -352,7 +352,7 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "2fa9aac3",
+ "id": "7879427c",
"metadata": {
"collapsed": false,
"editable": true
@@ -451,7 +451,7 @@
},
{
"cell_type": "markdown",
- "id": "2bb0ea43",
+ "id": "64a60119",
"metadata": {
"editable": true
},
@@ -473,7 +473,7 @@
},
{
"cell_type": "markdown",
- "id": "f7fd9801",
+ "id": "aef71098",
"metadata": {
"editable": true
},
@@ -485,7 +485,7 @@
},
{
"cell_type": "markdown",
- "id": "77d8e179",
+ "id": "c1819a6f",
"metadata": {
"editable": true
},
@@ -496,7 +496,7 @@
},
{
"cell_type": "markdown",
- "id": "90b48cd5",
+ "id": "b2aa9527",
"metadata": {
"editable": true
},
@@ -518,7 +518,7 @@
},
{
"cell_type": "markdown",
- "id": "1c1ee9be",
+ "id": "3bd07e22",
"metadata": {
"editable": true
},
@@ -531,7 +531,7 @@
},
{
"cell_type": "markdown",
- "id": "b3540862",
+ "id": "d087d425",
"metadata": {
"editable": true
},
@@ -543,7 +543,7 @@
},
{
"cell_type": "markdown",
- "id": "412be88b",
+ "id": "6dddbbe4",
"metadata": {
"editable": true
},
@@ -553,7 +553,7 @@
},
{
"cell_type": "markdown",
- "id": "26943d25",
+ "id": "a53486a7",
"metadata": {
"editable": true
},
@@ -565,7 +565,7 @@
},
{
"cell_type": "markdown",
- "id": "238ef636",
+ "id": "ed5534f3",
"metadata": {
"editable": true
},
@@ -575,7 +575,7 @@
},
{
"cell_type": "markdown",
- "id": "7d1f6683",
+ "id": "bfbb12ea",
"metadata": {
"editable": true
},
@@ -587,7 +587,7 @@
},
{
"cell_type": "markdown",
- "id": "ed814361",
+ "id": "254338e6",
"metadata": {
"editable": true
},
@@ -620,7 +620,7 @@
},
{
"cell_type": "markdown",
- "id": "b0389394",
+ "id": "57be982b",
"metadata": {
"editable": true
},
@@ -644,7 +644,7 @@
},
{
"cell_type": "markdown",
- "id": "d7428112",
+ "id": "9fbb4348",
"metadata": {
"editable": true
},
@@ -656,7 +656,7 @@
},
{
"cell_type": "markdown",
- "id": "365f30ff",
+ "id": "bd62b866",
"metadata": {
"editable": true
},
@@ -668,7 +668,7 @@
},
{
"cell_type": "markdown",
- "id": "ba0684e8",
+ "id": "7d8fbd88",
"metadata": {
"editable": true
},
@@ -696,7 +696,7 @@
},
{
"cell_type": "markdown",
- "id": "61e97043",
+ "id": "4ff5152f",
"metadata": {
"editable": true
},
@@ -722,7 +722,7 @@
},
{
"cell_type": "markdown",
- "id": "7970c5bc",
+ "id": "1a23ea40",
"metadata": {
"editable": true
},
@@ -745,7 +745,7 @@
},
{
"cell_type": "markdown",
- "id": "9d478b04",
+ "id": "a80b8a59",
"metadata": {
"editable": true
},
@@ -772,7 +772,7 @@
},
{
"cell_type": "markdown",
- "id": "d11cb0d7",
+ "id": "bb0ced76",
"metadata": {
"editable": true
},
@@ -791,7 +791,7 @@
},
{
"cell_type": "markdown",
- "id": "46022c8c",
+ "id": "087aecb8",
"metadata": {
"editable": true
},
@@ -803,7 +803,7 @@
},
{
"cell_type": "markdown",
- "id": "0166e45a",
+ "id": "b1dfaa5d",
"metadata": {
"editable": true
},
@@ -816,7 +816,7 @@
},
{
"cell_type": "markdown",
- "id": "f65fada9",
+ "id": "5d9e1c22",
"metadata": {
"editable": true
},
@@ -828,7 +828,7 @@
},
{
"cell_type": "markdown",
- "id": "078674e5",
+ "id": "834faf0f",
"metadata": {
"editable": true
},
@@ -838,7 +838,7 @@
},
{
"cell_type": "markdown",
- "id": "a1ac2fe9",
+ "id": "9bd81347",
"metadata": {
"editable": true
},
@@ -850,7 +850,7 @@
},
{
"cell_type": "markdown",
- "id": "1a4f2f62",
+ "id": "017dd22e",
"metadata": {
"editable": true
},
@@ -860,7 +860,7 @@
},
{
"cell_type": "markdown",
- "id": "00f8b061",
+ "id": "cd04c09c",
"metadata": {
"editable": true
},
@@ -872,7 +872,7 @@
},
{
"cell_type": "markdown",
- "id": "360ff461",
+ "id": "5cf9df71",
"metadata": {
"editable": true
},
@@ -883,7 +883,7 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "35297e26",
+ "id": "53b53796",
"metadata": {
"collapsed": false,
"editable": true
@@ -927,7 +927,7 @@
},
{
"cell_type": "markdown",
- "id": "cb145416",
+ "id": "5ddb1789",
"metadata": {
"editable": true
},
@@ -938,7 +938,7 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "6dc54743",
+ "id": "65092d3a",
"metadata": {
"collapsed": false,
"editable": true
@@ -973,7 +973,7 @@
},
{
"cell_type": "markdown",
- "id": "af20ea2d",
+ "id": "f014d974",
"metadata": {
"editable": true
},
@@ -986,7 +986,7 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "0c09dfdf",
+ "id": "43829fb1",
"metadata": {
"collapsed": false,
"editable": true
@@ -1004,7 +1004,7 @@
},
{
"cell_type": "markdown",
- "id": "7b3c3bcb",
+ "id": "3602edc7",
"metadata": {
"editable": true
},
@@ -1018,7 +1018,7 @@
{
"cell_type": "code",
"execution_count": 6,
- "id": "fb0cc872",
+ "id": "db8f56d7",
"metadata": {
"collapsed": false,
"editable": true
@@ -1037,7 +1037,7 @@
},
{
"cell_type": "markdown",
- "id": "8d8d8d82",
+ "id": "79548471",
"metadata": {
"editable": true
},
@@ -1057,7 +1057,7 @@
},
{
"cell_type": "markdown",
- "id": "306a610b",
+ "id": "d0b42d59",
"metadata": {
"editable": true
},
@@ -1074,7 +1074,7 @@
},
{
"cell_type": "markdown",
- "id": "39ece8a3",
+ "id": "b5a14a97",
"metadata": {
"editable": true
},
@@ -1086,7 +1086,7 @@
},
{
"cell_type": "markdown",
- "id": "4bdf6600",
+ "id": "9a774778",
"metadata": {
"editable": true
},
@@ -1103,7 +1103,7 @@
},
{
"cell_type": "markdown",
- "id": "aef73f0c",
+ "id": "be8b5c64",
"metadata": {
"editable": true
},
@@ -1116,7 +1116,7 @@
},
{
"cell_type": "markdown",
- "id": "31810fc0",
+ "id": "84b496a4",
"metadata": {
"editable": true
},
@@ -1128,7 +1128,7 @@
},
{
"cell_type": "markdown",
- "id": "d08aca8f",
+ "id": "10223626",
"metadata": {
"editable": true
},
@@ -1138,7 +1138,7 @@
},
{
"cell_type": "markdown",
- "id": "ddd28bbd",
+ "id": "bdc0ba64",
"metadata": {
"editable": true
},
@@ -1150,7 +1150,7 @@
},
{
"cell_type": "markdown",
- "id": "a12364d0",
+ "id": "8fbc3414",
"metadata": {
"editable": true
},
@@ -1160,7 +1160,7 @@
},
{
"cell_type": "markdown",
- "id": "4946666e",
+ "id": "d4d2fcd8",
"metadata": {
"editable": true
},
@@ -1172,7 +1172,7 @@
},
{
"cell_type": "markdown",
- "id": "b4d030a4",
+ "id": "ce399959",
"metadata": {
"editable": true
},
@@ -1185,7 +1185,7 @@
},
{
"cell_type": "markdown",
- "id": "c7b6da02",
+ "id": "6faf3fd5",
"metadata": {
"editable": true
},
@@ -1200,7 +1200,7 @@
},
{
"cell_type": "markdown",
- "id": "f6aa3363",
+ "id": "f309b16f",
"metadata": {
"editable": true
},
@@ -1226,7 +1226,7 @@
},
{
"cell_type": "markdown",
- "id": "e7a581e8",
+ "id": "186e0731",
"metadata": {
"editable": true
},
@@ -1254,7 +1254,7 @@
},
{
"cell_type": "markdown",
- "id": "13b84c42",
+ "id": "6c94c555",
"metadata": {
"editable": true
},
@@ -1271,7 +1271,7 @@
},
{
"cell_type": "markdown",
- "id": "7ae7c7ea",
+ "id": "e606b67d",
"metadata": {
"editable": true
},
@@ -1285,7 +1285,7 @@
},
{
"cell_type": "markdown",
- "id": "fc8a3e67",
+ "id": "5c656c54",
"metadata": {
"editable": true
},
@@ -1301,7 +1301,7 @@
},
{
"cell_type": "markdown",
- "id": "53850a13",
+ "id": "d6a5f46c",
"metadata": {
"editable": true
},
@@ -1312,7 +1312,7 @@
{
"cell_type": "code",
"execution_count": 7,
- "id": "4be4a43d",
+ "id": "68650507",
"metadata": {
"collapsed": false,
"editable": true
@@ -1390,7 +1390,7 @@
},
{
"cell_type": "markdown",
- "id": "90e95d49",
+ "id": "994250a4",
"metadata": {
"editable": true
},
@@ -1434,7 +1434,7 @@
},
{
"cell_type": "markdown",
- "id": "4e8a69a8",
+ "id": "dc7f6fc5",
"metadata": {
"editable": true
},
@@ -1445,7 +1445,7 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "3f6340fc",
+ "id": "121d45cf",
"metadata": {
"collapsed": false,
"editable": true
@@ -1523,7 +1523,7 @@
},
{
"cell_type": "markdown",
- "id": "9a18ffbd",
+ "id": "9c174840",
"metadata": {
"editable": true
},
@@ -1541,7 +1541,7 @@
{
"cell_type": "code",
"execution_count": 9,
- "id": "dc01ad75",
+ "id": "b077c4d1",
"metadata": {
"collapsed": false,
"editable": true
@@ -1612,18 +1612,18 @@
},
{
"cell_type": "markdown",
- "id": "a3d846cd",
+ "id": "fb1a615b",
"metadata": {
"editable": true
},
"source": [
- "## Another example, the moons again"
+ "## Another example, the moons"
]
},
{
"cell_type": "code",
"execution_count": 10,
- "id": "f41353c4",
+ "id": "7cdd567c",
"metadata": {
"collapsed": false,
"editable": true
@@ -1698,7 +1698,7 @@
},
{
"cell_type": "markdown",
- "id": "0d75cb36",
+ "id": "c74c9eaf",
"metadata": {
"editable": true
},
@@ -1709,7 +1709,7 @@
{
"cell_type": "code",
"execution_count": 11,
- "id": "270db370",
+ "id": "4f3c7805",
"metadata": {
"collapsed": false,
"editable": true
@@ -1740,7 +1740,7 @@
},
{
"cell_type": "markdown",
- "id": "944dd7a5",
+ "id": "5eeaf397",
"metadata": {
"editable": true
},
@@ -1751,7 +1751,7 @@
{
"cell_type": "code",
"execution_count": 12,
- "id": "6f9de8a4",
+ "id": "cd9e1562",
"metadata": {
"collapsed": false,
"editable": true
@@ -1769,7 +1769,7 @@
{
"cell_type": "code",
"execution_count": 13,
- "id": "e8c9f095",
+ "id": "1f11a97f",
"metadata": {
"collapsed": false,
"editable": true
@@ -1784,7 +1784,7 @@
},
{
"cell_type": "markdown",
- "id": "67f0f549",
+ "id": "c39125c9",
"metadata": {
"editable": true
},
@@ -1795,7 +1795,7 @@
{
"cell_type": "code",
"execution_count": 14,
- "id": "4587efe8",
+ "id": "f999098e",
"metadata": {
"collapsed": false,
"editable": true
@@ -1845,7 +1845,7 @@
{
"cell_type": "code",
"execution_count": 15,
- "id": "5e036165",
+ "id": "5c31d316",
"metadata": {
"collapsed": false,
"editable": true
@@ -1884,7 +1884,7 @@
},
{
"cell_type": "markdown",
- "id": "08cb1573",
+ "id": "a72112df",
"metadata": {
"editable": true
},
@@ -1908,7 +1908,7 @@
},
{
"cell_type": "markdown",
- "id": "44b5f52e",
+ "id": "3a11c269",
"metadata": {
"editable": true
},
@@ -1936,7 +1936,7 @@
},
{
"cell_type": "markdown",
- "id": "daf142b0",
+ "id": "1fdc0348",
"metadata": {
"editable": true
},
@@ -1967,7 +1967,7 @@
},
{
"cell_type": "markdown",
- "id": "038cda56",
+ "id": "9e124da2",
"metadata": {
"editable": true
},
@@ -1983,61 +1983,121 @@
},
{
"cell_type": "markdown",
- "id": "d6881d19",
+ "id": "1f9afd64",
"metadata": {
"editable": true
},
"source": [
- "## Bagging\n",
+ "## Why Voting?\n",
"\n",
- "The **plain** decision trees suffer from high\n",
- "variance. This means that if we split the training data into two parts\n",
- "at random, and fit a decision tree to both halves, the results that we\n",
- "get could be quite different. In contrast, a procedure with low\n",
- "variance will yield similar results if applied repeatedly to distinct\n",
- "data sets; linear regression tends to have low variance, if the ratio\n",
- "of $n$ to $p$ is moderately large. \n",
+ "The idea behind boosting, and voting as well can be phrased as follows:\n",
+ "**Can a group of people somehow arrive at highly\n",
+ "reasoned decisions, despite the weak judgement of the individual\n",
+ "members?**\n",
"\n",
- "**Bootstrap aggregation**, or just **bagging**, is a\n",
- "general-purpose procedure for reducing the variance of a statistical\n",
- "learning method."
+ "The aim is to create a good classifier by combining several weak classifiers.\n",
+ "**A weak classifier is a classifier which is able to produce results that are only slightly better than guessing at random.**\n",
+ "\n",
+ "The basic approach is to apply repeatedly (in boosting this is done in an iterative way) a weak classifier to modifications of the data.\n",
+ "In voting we simply apply the law of large numbers while in boosting we give more weight to misclassified data in\n",
+ "each iteration. \n",
+ "\n",
+ "Decision trees play an important role as our weak classifier. They serve as the basic method."
]
},
{
"cell_type": "markdown",
- "id": "6ca7617f",
+ "id": "72766be2",
"metadata": {
"editable": true
},
"source": [
- "## More bagging\n",
+ "## Tossing coins\n",
"\n",
- "Bagging typically results in improved accuracy\n",
- "over prediction using a single tree. Unfortunately, however, it can be\n",
- "difficult to interpret the resulting model. Recall that one of the\n",
- "advantages of decision trees is the attractive and easily interpreted\n",
- "diagram that results.\n",
+ "The simplest case is a so-called voting ensemble. To illustrate this,\n",
+ "think of yourself tossing coins with a biased outcome of 51 per cent\n",
+ "for heads and 49% for tails. With only few tosses,\n",
+ "you may not clearly see this distribution for heads and tails. However, after some\n",
+ "thousands of tosses, there will be a clear majority of heads. With 2000 tosses\n",
+ "you should see approximately 1020 heads and 980 tails.\n",
"\n",
- "However, when we bag a large number of trees, it is no longer\n",
- "possible to represent the resulting statistical learning procedure\n",
- "using a single tree, and it is no longer clear which variables are\n",
- "most important to the procedure. Thus, bagging improves prediction\n",
- "accuracy at the expense of interpretability. Although the collection\n",
- "of bagged trees is much more difficult to interpret than a single\n",
- "tree, one can obtain an overall summary of the importance of each\n",
- "predictor using the MSE (for bagging regression trees) or the Gini\n",
- "index (for bagging classification trees). In the case of bagging\n",
- "regression trees, we can record the total amount that the MSE is\n",
- "decreased due to splits over a given predictor, averaged over all $B$ possible\n",
- "trees. A large value indicates an important predictor. Similarly, in\n",
- "the context of bagging classification trees, we can add up the total\n",
- "amount that the Gini index is decreased by splits over a given\n",
- "predictor, averaged over all $B$ trees."
+ "We can then state that the outcome is a clear majority of heads. If\n",
+ "you do this ten thousand times, it is easy to see that there is a 97%\n",
+ "likelihood of a majority of heads.\n",
+ "\n",
+ "Another example would be to collect all polls before an\n",
+ "election. Different polls may show different likelihoods for a\n",
+ "candidate winning with say a majority of the popular vote. The majority vote\n",
+ "would then consist in many polls indicating that this candidate will\n",
+ "actually win.\n",
+ "\n",
+ "The example here shows how we can implement the coin tossing case,\n",
+ "clealry demostrating that after some tosses we see the [law of large](https://en.wikipedia.org/wiki/Law_of_large_numbers)\n",
+ "numbers kicking in."
]
},
{
"cell_type": "markdown",
- "id": "ccab6c19",
+ "id": "31fd7020",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Standard imports first"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "id": "8f3556b4",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
+ "source": [
+ "# Common imports\n",
+ "from IPython.display import Image \n",
+ "from pydot import graph_from_dot_data\n",
+ "import pandas as pd\n",
+ "import numpy as np\n",
+ "import matplotlib.pyplot as plt\n",
+ "from sklearn.tree import DecisionTreeClassifier\n",
+ "from sklearn.model_selection import train_test_split\n",
+ "from sklearn.tree import export_graphviz\n",
+ "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
+ "from sklearn.compose import ColumnTransformer\n",
+ "from IPython.display import Image \n",
+ "from pydot import graph_from_dot_data\n",
+ "import os\n",
+ "\n",
+ "# Where to save the figures and data files\n",
+ "PROJECT_ROOT_DIR = \"Results\"\n",
+ "FIGURE_ID = \"Results/FigureFiles\"\n",
+ "DATA_ID = \"DataFiles/\"\n",
+ "\n",
+ "if not os.path.exists(PROJECT_ROOT_DIR):\n",
+ " os.mkdir(PROJECT_ROOT_DIR)\n",
+ "\n",
+ "if not os.path.exists(FIGURE_ID):\n",
+ " os.makedirs(FIGURE_ID)\n",
+ "\n",
+ "if not os.path.exists(DATA_ID):\n",
+ " os.makedirs(DATA_ID)\n",
+ "\n",
+ "def image_path(fig_id):\n",
+ " return os.path.join(FIGURE_ID, fig_id)\n",
+ "\n",
+ "def data_path(dat_id):\n",
+ " return os.path.join(DATA_ID, dat_id)\n",
+ "\n",
+ "def save_fig(fig_id):\n",
+ " plt.savefig(image_path(fig_id) + \".png\", format='png')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7e4f5d08",
"metadata": {
"editable": true
},
@@ -2047,14 +2107,24 @@
},
{
"cell_type": "code",
- "execution_count": 16,
- "id": "ee9af3b9",
+ "execution_count": 17,
+ "id": "4b2a27f1",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
+ "\n",
+ "# Common imports\n",
+ "import numpy as np\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "from matplotlib.colors import ListedColormap\n",
+ "plt.rcParams['axes.labelsize'] = 14\n",
+ "plt.rcParams['xtick.labelsize'] = 12\n",
+ "plt.rcParams['ytick.labelsize'] = 12\n",
+ "\n",
"heads_proba = 0.51\n",
"coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)\n",
"cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)\n",
@@ -2072,18 +2142,20 @@
},
{
"cell_type": "markdown",
- "id": "cd895813",
+ "id": "facdb6a9",
"metadata": {
"editable": true
},
"source": [
- "## Using the Voting Classifier"
+ "## Using the Voting Classifier\n",
+ "\n",
+ "We can use the voting classifier on other data sets, here the exciting binary case of two distinct objects using the make moons functionality of **Scikit-Learn**."
]
},
{
"cell_type": "code",
- "execution_count": 17,
- "id": "d5e48af3",
+ "execution_count": 18,
+ "id": "c7986569",
"metadata": {
"collapsed": false,
"editable": true
@@ -2121,7 +2193,6 @@
"log_clf = LogisticRegression(solver=\"liblinear\", random_state=42)\n",
"rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)\n",
"svm_clf = SVC(gamma=\"auto\", probability=True, random_state=42)\n",
- "\n",
"voting_clf = VotingClassifier(\n",
" estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],\n",
" voting='soft')\n",
@@ -2137,18 +2208,18 @@
},
{
"cell_type": "markdown",
- "id": "c64b5374",
+ "id": "090cc2cf",
"metadata": {
"editable": true
},
"source": [
- "## Please, not the moons again! Voting and Bagging"
+ "## Voting and Bagging"
]
},
{
"cell_type": "code",
- "execution_count": 18,
- "id": "76d0f2ea",
+ "execution_count": 19,
+ "id": "44f30cb2",
"metadata": {
"collapsed": false,
"editable": true
@@ -2177,8 +2248,8 @@
},
{
"cell_type": "code",
- "execution_count": 19,
- "id": "7fcccf7d",
+ "execution_count": 20,
+ "id": "d28826ab",
"metadata": {
"collapsed": false,
"editable": true
@@ -2195,8 +2266,8 @@
},
{
"cell_type": "code",
- "execution_count": 20,
- "id": "e380253c",
+ "execution_count": 21,
+ "id": "b02c5322",
"metadata": {
"collapsed": false,
"editable": true
@@ -2215,8 +2286,8 @@
},
{
"cell_type": "code",
- "execution_count": 21,
- "id": "f90ec503",
+ "execution_count": 22,
+ "id": "d542a2be",
"metadata": {
"collapsed": false,
"editable": true
@@ -2233,106 +2304,61 @@
},
{
"cell_type": "markdown",
- "id": "23aac3ac",
+ "id": "a7cdad96",
"metadata": {
"editable": true
},
"source": [
- "## Bagging Examples"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 22,
- "id": "de7988eb",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "from sklearn.ensemble import BaggingClassifier\n",
- "from sklearn.tree import DecisionTreeClassifier\n",
+ "## Bagging\n",
"\n",
- "bag_clf = BaggingClassifier(\n",
- " DecisionTreeClassifier(random_state=42), n_estimators=500,\n",
- " max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)\n",
- "bag_clf.fit(X_train, y_train)\n",
- "y_pred = bag_clf.predict(X_test)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 23,
- "id": "6c8ee680",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "from sklearn.metrics import accuracy_score\n",
- "print(accuracy_score(y_test, y_pred))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 24,
- "id": "6c08a9b3",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "tree_clf = DecisionTreeClassifier(random_state=42)\n",
- "tree_clf.fit(X_train, y_train)\n",
- "y_pred_tree = tree_clf.predict(X_test)\n",
- "print(accuracy_score(y_test, y_pred_tree))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 25,
- "id": "a2365657",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "from matplotlib.colors import ListedColormap\n",
+ "The **plain** decision trees suffer from high\n",
+ "variance. This means that if we split the training data into two parts\n",
+ "at random, and fit a decision tree to both halves, the results that we\n",
+ "get could be quite different. In contrast, a procedure with low\n",
+ "variance will yield similar results if applied repeatedly to distinct\n",
+ "data sets; linear regression tends to have low variance, if the ratio\n",
+ "of $n$ to $p$ is moderately large. \n",
"\n",
- "def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):\n",
- " x1s = np.linspace(axes[0], axes[1], 100)\n",
- " x2s = np.linspace(axes[2], axes[3], 100)\n",
- " x1, x2 = np.meshgrid(x1s, x2s)\n",
- " X_new = np.c_[x1.ravel(), x2.ravel()]\n",
- " y_pred = clf.predict(X_new).reshape(x1.shape)\n",
- " custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])\n",
- " plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)\n",
- " if contour:\n",
- " custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])\n",
- " plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)\n",
- " plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"yo\", alpha=alpha)\n",
- " plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"bs\", alpha=alpha)\n",
- " plt.axis(axes)\n",
- " plt.xlabel(r\"$x_1$\", fontsize=18)\n",
- " plt.ylabel(r\"$x_2$\", fontsize=18, rotation=0)\n",
- "plt.figure(figsize=(11,4))\n",
- "plt.subplot(121)\n",
- "plot_decision_boundary(tree_clf, X, y)\n",
- "plt.title(\"Decision Tree\", fontsize=14)\n",
- "plt.subplot(122)\n",
- "plot_decision_boundary(bag_clf, X, y)\n",
- "plt.title(\"Decision Trees with Bagging\", fontsize=14)\n",
- "save_fig(\"baggingtree\")\n",
- "plt.show()"
+ "**Bootstrap aggregation**, or just **bagging**, is a\n",
+ "general-purpose procedure for reducing the variance of a statistical\n",
+ "learning method."
]
},
{
"cell_type": "markdown",
- "id": "50147ee0",
+ "id": "96ae19e9",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## More bagging\n",
+ "\n",
+ "Bagging typically results in improved accuracy\n",
+ "over prediction using a single tree. Unfortunately, however, it can be\n",
+ "difficult to interpret the resulting model. Recall that one of the\n",
+ "advantages of decision trees is the attractive and easily interpreted\n",
+ "diagram that results.\n",
+ "\n",
+ "However, when we bag a large number of trees, it is no longer\n",
+ "possible to represent the resulting statistical learning procedure\n",
+ "using a single tree, and it is no longer clear which variables are\n",
+ "most important to the procedure. Thus, bagging improves prediction\n",
+ "accuracy at the expense of interpretability. Although the collection\n",
+ "of bagged trees is much more difficult to interpret than a single\n",
+ "tree, one can obtain an overall summary of the importance of each\n",
+ "predictor using the MSE (for bagging regression trees) or the Gini\n",
+ "index (for bagging classification trees). In the case of bagging\n",
+ "regression trees, we can record the total amount that the MSE is\n",
+ "decreased due to splits over a given predictor, averaged over all $B$ possible\n",
+ "trees. A large value indicates an important predictor. Similarly, in\n",
+ "the context of bagging classification trees, we can add up the total\n",
+ "amount that the Gini index is decreased by splits over a given\n",
+ "predictor, averaged over all $B$ trees."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0b821772",
"metadata": {
"editable": true
},
@@ -2345,8 +2371,8 @@
},
{
"cell_type": "code",
- "execution_count": 26,
- "id": "5c2c80a2",
+ "execution_count": 23,
+ "id": "c69ac3f2",
"metadata": {
"collapsed": false,
"editable": true
@@ -2415,328 +2441,7 @@
},
{
"cell_type": "markdown",
- "id": "1178fafb",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Why Voting?\n",
- "\n",
- "The idea behind boosting, and voting as well can be phrased as follows:\n",
- "**Can a group of people somehow arrive at highly\n",
- "reasoned decisions, despite the weak judgement of the individual\n",
- "members?**\n",
- "\n",
- "The aim is to create a good classifier by combining several weak classifiers.\n",
- "**A weak classifier is a classifier which is able to produce results that are only slightly better than guessing at random.**\n",
- "\n",
- "The basic approach is to apply repeatedly (in boosting this is done in an iterative way) a weak classifier to modifications of the data.\n",
- "In voting we simply apply the law of large numbers while in boosting we give more weight to misclassified data in\n",
- "each iteration. \n",
- "\n",
- "Decision trees play an important role as our weak classifier. They serve as the basic method."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a3a30aa4",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Tossing coins\n",
- "\n",
- "The simplest case is a so-called voting ensemble. To illustrate this,\n",
- "think of yourself tossing coins with a biased outcome of 51 per cent\n",
- "for heads and 49% for tails. With only few tosses,\n",
- "you may not clearly see this distribution for heads and tails. However, after some\n",
- "thousands of tosses, there will be a clear majority of heads. With 2000 tosses\n",
- "you should see approximately 1020 heads and 980 tails.\n",
- "\n",
- "We can then state that the outcome is a clear majority of heads. If\n",
- "you do this ten thousand times, it is easy to see that there is a 97%\n",
- "likelihood of a majority of heads.\n",
- "\n",
- "Another example would be to collect all polls before an\n",
- "election. Different polls may show different likelihoods for a\n",
- "candidate winning with say a majority of the popular vote. The majority vote\n",
- "would then consist in many polls indicating that this candidate will\n",
- "actually win.\n",
- "\n",
- "The example here shows how we can implement the coin tossing case,\n",
- "clealry demostrating that after some tosses we see the [law of large](https://en.wikipedia.org/wiki/Law_of_large_numbers)\n",
- "numbers kicking in."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2fe724f5",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Standard imports first"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 27,
- "id": "a5ee936a",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "# Common imports\n",
- "from IPython.display import Image \n",
- "from pydot import graph_from_dot_data\n",
- "import pandas as pd\n",
- "import numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "from sklearn.tree import DecisionTreeClassifier\n",
- "from sklearn.model_selection import train_test_split\n",
- "from sklearn.tree import export_graphviz\n",
- "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
- "from sklearn.compose import ColumnTransformer\n",
- "from IPython.display import Image \n",
- "from pydot import graph_from_dot_data\n",
- "import os\n",
- "\n",
- "# Where to save the figures and data files\n",
- "PROJECT_ROOT_DIR = \"Results\"\n",
- "FIGURE_ID = \"Results/FigureFiles\"\n",
- "DATA_ID = \"DataFiles/\"\n",
- "\n",
- "if not os.path.exists(PROJECT_ROOT_DIR):\n",
- " os.mkdir(PROJECT_ROOT_DIR)\n",
- "\n",
- "if not os.path.exists(FIGURE_ID):\n",
- " os.makedirs(FIGURE_ID)\n",
- "\n",
- "if not os.path.exists(DATA_ID):\n",
- " os.makedirs(DATA_ID)\n",
- "\n",
- "def image_path(fig_id):\n",
- " return os.path.join(FIGURE_ID, fig_id)\n",
- "\n",
- "def data_path(dat_id):\n",
- " return os.path.join(DATA_ID, dat_id)\n",
- "\n",
- "def save_fig(fig_id):\n",
- " plt.savefig(image_path(fig_id) + \".png\", format='png')"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "780b0b45",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Simple Voting Example, head or tail"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 28,
- "id": "2aa03ca6",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "\n",
- "# Common imports\n",
- "import numpy as np\n",
- "import matplotlib\n",
- "import matplotlib.pyplot as plt\n",
- "from matplotlib.colors import ListedColormap\n",
- "plt.rcParams['axes.labelsize'] = 14\n",
- "plt.rcParams['xtick.labelsize'] = 12\n",
- "plt.rcParams['ytick.labelsize'] = 12\n",
- "\n",
- "heads_proba = 0.51\n",
- "coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)\n",
- "cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)\n",
- "plt.figure(figsize=(8,3.5))\n",
- "plt.plot(cumulative_heads_ratio)\n",
- "plt.plot([0, 10000], [0.51, 0.51], \"k--\", linewidth=2, label=\"51%\")\n",
- "plt.plot([0, 10000], [0.5, 0.5], \"k-\", label=\"50%\")\n",
- "plt.xlabel(\"Number of coin tosses\")\n",
- "plt.ylabel(\"Heads ratio\")\n",
- "plt.legend(loc=\"lower right\")\n",
- "plt.axis([0, 10000, 0.42, 0.58])\n",
- "save_fig(\"votingsimple\")\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "edaded51",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Using the Voting Classifier\n",
- "\n",
- "We can use the voting classifier on other data sets, here the exciting binary case of two distinct objects using the make moons functionality of **Scikit-Learn**."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 29,
- "id": "ea39ac21",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "from sklearn.model_selection import train_test_split\n",
- "from sklearn.datasets import make_moons\n",
- "\n",
- "X, y = make_moons(n_samples=500, noise=0.30, random_state=42)\n",
- "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)\n",
- "\n",
- "from sklearn.ensemble import RandomForestClassifier\n",
- "from sklearn.ensemble import VotingClassifier\n",
- "from sklearn.linear_model import LogisticRegression\n",
- "from sklearn.svm import SVC\n",
- "\n",
- "log_clf = LogisticRegression(solver=\"liblinear\", random_state=42)\n",
- "rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)\n",
- "svm_clf = SVC(gamma=\"auto\", random_state=42)\n",
- "\n",
- "voting_clf = VotingClassifier(\n",
- " estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],\n",
- " voting='hard')\n",
- "\n",
- "voting_clf.fit(X_train, y_train)\n",
- "\n",
- "from sklearn.metrics import accuracy_score\n",
- "\n",
- "for clf in (log_clf, rnd_clf, svm_clf, voting_clf):\n",
- " clf.fit(X_train, y_train)\n",
- " y_pred = clf.predict(X_test)\n",
- " print(clf.__class__.__name__, accuracy_score(y_test, y_pred))\n",
- "\n",
- "log_clf = LogisticRegression(solver=\"liblinear\", random_state=42)\n",
- "rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)\n",
- "svm_clf = SVC(gamma=\"auto\", probability=True, random_state=42)\n",
- "voting_clf = VotingClassifier(\n",
- " estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],\n",
- " voting='soft')\n",
- "voting_clf.fit(X_train, y_train)\n",
- "\n",
- "from sklearn.metrics import accuracy_score\n",
- "\n",
- "for clf in (log_clf, rnd_clf, svm_clf, voting_clf):\n",
- " clf.fit(X_train, y_train)\n",
- " y_pred = clf.predict(X_test)\n",
- " print(clf.__class__.__name__, accuracy_score(y_test, y_pred))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2c612ac2",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Voting and Bagging"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 30,
- "id": "d700c338",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "from sklearn.model_selection import train_test_split\n",
- "from sklearn.datasets import make_moons\n",
- "\n",
- "X, y = make_moons(n_samples=500, noise=0.30, random_state=42)\n",
- "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)\n",
- "from sklearn.ensemble import RandomForestClassifier\n",
- "from sklearn.ensemble import VotingClassifier\n",
- "from sklearn.linear_model import LogisticRegression\n",
- "from sklearn.svm import SVC\n",
- "\n",
- "log_clf = LogisticRegression(random_state=42)\n",
- "rnd_clf = RandomForestClassifier(random_state=42)\n",
- "svm_clf = SVC(random_state=42)\n",
- "\n",
- "voting_clf = VotingClassifier(\n",
- " estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],\n",
- " voting='hard')\n",
- "voting_clf.fit(X_train, y_train)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 31,
- "id": "9a5ddf88",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "from sklearn.metrics import accuracy_score\n",
- "\n",
- "for clf in (log_clf, rnd_clf, svm_clf, voting_clf):\n",
- " clf.fit(X_train, y_train)\n",
- " y_pred = clf.predict(X_test)\n",
- " print(clf.__class__.__name__, accuracy_score(y_test, y_pred))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 32,
- "id": "1760ed86",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "log_clf = LogisticRegression(random_state=42)\n",
- "rnd_clf = RandomForestClassifier(random_state=42)\n",
- "svm_clf = SVC(probability=True, random_state=42)\n",
- "\n",
- "voting_clf = VotingClassifier(\n",
- " estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],\n",
- " voting='soft')\n",
- "voting_clf.fit(X_train, y_train)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 33,
- "id": "4ab23f0f",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "from sklearn.metrics import accuracy_score\n",
- "\n",
- "for clf in (log_clf, rnd_clf, svm_clf, voting_clf):\n",
- " clf.fit(X_train, y_train)\n",
- " y_pred = clf.predict(X_test)\n",
- " print(clf.__class__.__name__, accuracy_score(y_test, y_pred))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0355abcb",
+ "id": "9a85f577",
"metadata": {
"editable": true
},
@@ -2759,7 +2464,7 @@
},
{
"cell_type": "markdown",
- "id": "e543eba5",
+ "id": "b020b475",
"metadata": {
"editable": true
},
@@ -2771,7 +2476,7 @@
},
{
"cell_type": "markdown",
- "id": "d3f906c7",
+ "id": "5121590b",
"metadata": {
"editable": true
},
@@ -2796,7 +2501,7 @@
},
{
"cell_type": "markdown",
- "id": "3582fecd",
+ "id": "ef8879c2",
"metadata": {
"editable": true
},
@@ -2822,7 +2527,7 @@
},
{
"cell_type": "markdown",
- "id": "9dba7602",
+ "id": "f53f3987",
"metadata": {
"editable": true
},
@@ -2832,8 +2537,8 @@
},
{
"cell_type": "code",
- "execution_count": 34,
- "id": "b9bcda80",
+ "execution_count": 24,
+ "id": "caef766a",
"metadata": {
"collapsed": false,
"editable": true
@@ -2855,19 +2560,14 @@
"X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)\n",
"print(X_train.shape)\n",
"print(X_test.shape)\n",
+ "#define methods\n",
"# Logistic Regression\n",
"logreg = LogisticRegression(solver='lbfgs')\n",
- "logreg.fit(X_train, y_train)\n",
- "print(\"Test set accuracy with Logistic Regression: {:.2f}\".format(logreg.score(X_test,y_test)))\n",
"# Support vector machine\n",
"svm = SVC(gamma='auto', C=100)\n",
- "svm.fit(X_train, y_train)\n",
- "print(\"Test set accuracy with SVM: {:.2f}\".format(svm.score(X_test,y_test)))\n",
"# Decision Trees\n",
"deep_tree_clf = DecisionTreeClassifier(max_depth=None)\n",
- "deep_tree_clf.fit(X_train, y_train)\n",
- "print(\"Test set accuracy with Decision Trees: {:.2f}\".format(deep_tree_clf.score(X_test,y_test)))\n",
- "#now scale the data\n",
+ "#Scale the data\n",
"from sklearn.preprocessing import StandardScaler\n",
"scaler = StandardScaler()\n",
"scaler.fit(X_train)\n",
@@ -2910,7 +2610,7 @@
},
{
"cell_type": "markdown",
- "id": "699bd8fd",
+ "id": "51dc82ac",
"metadata": {
"editable": true
},
@@ -2926,7 +2626,7 @@
},
{
"cell_type": "markdown",
- "id": "b66925ee",
+ "id": "19c89a3b",
"metadata": {
"editable": true
},
@@ -2936,8 +2636,8 @@
},
{
"cell_type": "code",
- "execution_count": 35,
- "id": "6deb494a",
+ "execution_count": 25,
+ "id": "61df8785",
"metadata": {
"collapsed": false,
"editable": true
@@ -2951,8 +2651,8 @@
},
{
"cell_type": "code",
- "execution_count": 36,
- "id": "118dcd30",
+ "execution_count": 26,
+ "id": "4431c086",
"metadata": {
"collapsed": false,
"editable": true
diff --git a/doc/src/week44/week44.do.txt b/doc/src/week44/week44.do.txt
index 7ec45cf41..b42ca4f2b 100644
--- a/doc/src/week44/week44.do.txt
+++ b/doc/src/week44/week44.do.txt
@@ -1062,7 +1062,7 @@ print('Split: [X%d < %.3f]' % ((split['index']+1), split['value']))
!split
-===== Another example, the moons again =====
+===== Another example, the moons =====
!bc pycod
from __future__ import division, print_function, unicode_literals
@@ -1306,300 +1306,6 @@ We discuss these methods here.
FIGURE: [DataFiles/ensembleoverview.png, width=600 frac=0.8]
-
-
-!split
-===== Bagging =====
-
-The _plain_ decision trees suffer from high
-variance. This means that if we split the training data into two parts
-at random, and fit a decision tree to both halves, the results that we
-get could be quite different. In contrast, a procedure with low
-variance will yield similar results if applied repeatedly to distinct
-data sets; linear regression tends to have low variance, if the ratio
-of $n$ to $p$ is moderately large.
-
-_Bootstrap aggregation_, or just _bagging_, is a
-general-purpose procedure for reducing the variance of a statistical
-learning method.
-
-
-!split
-===== More bagging =====
-
-Bagging typically results in improved accuracy
-over prediction using a single tree. Unfortunately, however, it can be
-difficult to interpret the resulting model. Recall that one of the
-advantages of decision trees is the attractive and easily interpreted
-diagram that results.
-
-However, when we bag a large number of trees, it is no longer
-possible to represent the resulting statistical learning procedure
-using a single tree, and it is no longer clear which variables are
-most important to the procedure. Thus, bagging improves prediction
-accuracy at the expense of interpretability. Although the collection
-of bagged trees is much more difficult to interpret than a single
-tree, one can obtain an overall summary of the importance of each
-predictor using the MSE (for bagging regression trees) or the Gini
-index (for bagging classification trees). In the case of bagging
-regression trees, we can record the total amount that the MSE is
-decreased due to splits over a given predictor, averaged over all $B$ possible
-trees. A large value indicates an important predictor. Similarly, in
-the context of bagging classification trees, we can add up the total
-amount that the Gini index is decreased by splits over a given
-predictor, averaged over all $B$ trees.
-
-!split
-===== Simple Voting Example, head or tail =====
-!bc pycod
-heads_proba = 0.51
-coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
-cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)
-plt.figure(figsize=(8,3.5))
-plt.plot(cumulative_heads_ratio)
-plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
-plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
-plt.xlabel("Number of coin tosses")
-plt.ylabel("Heads ratio")
-plt.legend(loc="lower right")
-plt.axis([0, 10000, 0.42, 0.58])
-save_fig("votingsimple")
-plt.show()
-
-!ec
-
-!split
-===== Using the Voting Classifier =====
-!bc pycod
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
-
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-log_clf = LogisticRegression(solver="liblinear", random_state=42)
-rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
-svm_clf = SVC(gamma="auto", probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-
-!ec
-
-!split
-===== Please, not the moons again! Voting and Bagging =====
-
-!bc pycod
-from sklearn.model_selection import train_test_split
-from sklearn.datasets import make_moons
-
-X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
-X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.ensemble import VotingClassifier
-from sklearn.linear_model import LogisticRegression
-from sklearn.svm import SVC
-
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='hard')
-voting_clf.fit(X_train, y_train)
-!ec
-
-!bc pycod
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-!ec
-
-!bc pycod
-log_clf = LogisticRegression(random_state=42)
-rnd_clf = RandomForestClassifier(random_state=42)
-svm_clf = SVC(probability=True, random_state=42)
-
-voting_clf = VotingClassifier(
- estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
- voting='soft')
-voting_clf.fit(X_train, y_train)
-!ec
-
-!bc pycod
-from sklearn.metrics import accuracy_score
-
-for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
- clf.fit(X_train, y_train)
- y_pred = clf.predict(X_test)
- print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
-!ec
-
-!split
-===== Bagging Examples =====
-
-!bc pycod
-from sklearn.ensemble import BaggingClassifier
-from sklearn.tree import DecisionTreeClassifier
-
-bag_clf = BaggingClassifier(
- DecisionTreeClassifier(random_state=42), n_estimators=500,
- max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
-bag_clf.fit(X_train, y_train)
-y_pred = bag_clf.predict(X_test)
-!ec
-
-
-!bc pycod
-from sklearn.metrics import accuracy_score
-print(accuracy_score(y_test, y_pred))
-!ec
-
-!bc pycod
-tree_clf = DecisionTreeClassifier(random_state=42)
-tree_clf.fit(X_train, y_train)
-y_pred_tree = tree_clf.predict(X_test)
-print(accuracy_score(y_test, y_pred_tree))
-!ec
-
-!bc pycod
-from matplotlib.colors import ListedColormap
-
-def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
- x1s = np.linspace(axes[0], axes[1], 100)
- x2s = np.linspace(axes[2], axes[3], 100)
- x1, x2 = np.meshgrid(x1s, x2s)
- X_new = np.c_[x1.ravel(), x2.ravel()]
- y_pred = clf.predict(X_new).reshape(x1.shape)
- custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
- plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
- if contour:
- custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
- plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
- plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
- plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
- plt.axis(axes)
- plt.xlabel(r"$x_1$", fontsize=18)
- plt.ylabel(r"$x_2$", fontsize=18, rotation=0)
-plt.figure(figsize=(11,4))
-plt.subplot(121)
-plot_decision_boundary(tree_clf, X, y)
-plt.title("Decision Tree", fontsize=14)
-plt.subplot(122)
-plot_decision_boundary(bag_clf, X, y)
-plt.title("Decision Trees with Bagging", fontsize=14)
-save_fig("baggingtree")
-plt.show()
-!ec
-
-
-
-!split
-===== Making your own Bootstrap: Changing the Level of the Decision Tree =====
-
-Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with
-a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points $n$).
-!bc pycod
-
-import matplotlib.pyplot as plt
-import numpy as np
-from sklearn.model_selection import train_test_split
-from sklearn.pipeline import make_pipeline
-from sklearn.utils import resample
-from sklearn.tree import DecisionTreeRegressor
-
-n = 100
-n_boostraps = 100
-maxdepth = 8
-
-# Make data set.
-x = np.linspace(-3, 3, n).reshape(-1, 1)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-error = np.zeros(maxdepth)
-bias = np.zeros(maxdepth)
-variance = np.zeros(maxdepth)
-polydegree = np.zeros(maxdepth)
-X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
-
-from sklearn.preprocessing import StandardScaler
-scaler = StandardScaler()
-scaler.fit(X_train)
-X_train_scaled = scaler.transform(X_train)
-X_test_scaled = scaler.transform(X_test)
-
-# we produce a simple tree first as benchmark
-simpletree = DecisionTreeRegressor(max_depth=3)
-simpletree.fit(X_train_scaled, y_train)
-simpleprediction = simpletree.predict(X_test_scaled)
-for degree in range(1,maxdepth):
- model = DecisionTreeRegressor(max_depth=degree)
- y_pred = np.empty((y_test.shape[0], n_boostraps))
- for i in range(n_boostraps):
- x_, y_ = resample(X_train_scaled, y_train)
- model.fit(x_, y_)
- y_pred[:, i] = model.predict(X_test_scaled)#.ravel()
-
- polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
- variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
- print('Polynomial degree:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
-
-mse_simpletree= np.mean( np.mean((y_test - simpleprediction)**2)
-print(mse_simpletree)
-plt.xlim(1,maxdepth)
-plt.plot(polydegree, error, label='MSE')
-plt.plot(polydegree, bias, label='bias')
-plt.plot(polydegree, variance, label='Variance')
-plt.legend()
-save_fig("baggingboot")
-plt.show()
-
-!ec
-
-
-
-
-
!split
===== Why Voting? =====
@@ -1828,6 +1534,122 @@ for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
+!split
+===== Bagging =====
+
+The _plain_ decision trees suffer from high
+variance. This means that if we split the training data into two parts
+at random, and fit a decision tree to both halves, the results that we
+get could be quite different. In contrast, a procedure with low
+variance will yield similar results if applied repeatedly to distinct
+data sets; linear regression tends to have low variance, if the ratio
+of $n$ to $p$ is moderately large.
+
+_Bootstrap aggregation_, or just _bagging_, is a
+general-purpose procedure for reducing the variance of a statistical
+learning method.
+
+
+!split
+===== More bagging =====
+
+Bagging typically results in improved accuracy
+over prediction using a single tree. Unfortunately, however, it can be
+difficult to interpret the resulting model. Recall that one of the
+advantages of decision trees is the attractive and easily interpreted
+diagram that results.
+
+However, when we bag a large number of trees, it is no longer
+possible to represent the resulting statistical learning procedure
+using a single tree, and it is no longer clear which variables are
+most important to the procedure. Thus, bagging improves prediction
+accuracy at the expense of interpretability. Although the collection
+of bagged trees is much more difficult to interpret than a single
+tree, one can obtain an overall summary of the importance of each
+predictor using the MSE (for bagging regression trees) or the Gini
+index (for bagging classification trees). In the case of bagging
+regression trees, we can record the total amount that the MSE is
+decreased due to splits over a given predictor, averaged over all $B$ possible
+trees. A large value indicates an important predictor. Similarly, in
+the context of bagging classification trees, we can add up the total
+amount that the Gini index is decreased by splits over a given
+predictor, averaged over all $B$ trees.
+
+
+
+!split
+===== Making your own Bootstrap: Changing the Level of the Decision Tree =====
+
+Let us bring up our good old boostrap example from the linear regression lectures. We change the linerar regression algorithm with
+a decision tree wth different depths and perform a bootstrap aggregate (in this case we perform as many bootstraps as data points $n$).
+!bc pycod
+
+import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+from sklearn.tree import DecisionTreeRegressor
+
+n = 100
+n_boostraps = 100
+maxdepth = 8
+
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+error = np.zeros(maxdepth)
+bias = np.zeros(maxdepth)
+variance = np.zeros(maxdepth)
+polydegree = np.zeros(maxdepth)
+X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+
+# we produce a simple tree first as benchmark
+simpletree = DecisionTreeRegressor(max_depth=3)
+simpletree.fit(X_train_scaled, y_train)
+simpleprediction = simpletree.predict(X_test_scaled)
+for degree in range(1,maxdepth):
+ model = DecisionTreeRegressor(max_depth=degree)
+ y_pred = np.empty((y_test.shape[0], n_boostraps))
+ for i in range(n_boostraps):
+ x_, y_ = resample(X_train_scaled, y_train)
+ model.fit(x_, y_)
+ y_pred[:, i] = model.predict(X_test_scaled)#.ravel()
+
+ polydegree[degree] = degree
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+ variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+ print('Polynomial degree:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+
+mse_simpletree= np.mean( np.mean((y_test - simpleprediction)**2)
+print(mse_simpletree)
+plt.xlim(1,maxdepth)
+plt.plot(polydegree, error, label='MSE')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+save_fig("baggingboot")
+plt.show()
+
+!ec
+
+
+
+
+
+
+
!split
===== Random forests =====
@@ -1902,19 +1724,14 @@ cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
print(X_train.shape)
print(X_test.shape)
+#define methods
# Logistic Regression
logreg = LogisticRegression(solver='lbfgs')
-logreg.fit(X_train, y_train)
-print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
# Support vector machine
svm = SVC(gamma='auto', C=100)
-svm.fit(X_train, y_train)
-print("Test set accuracy with SVM: {:.2f}".format(svm.score(X_test,y_test)))
# Decision Trees
deep_tree_clf = DecisionTreeClassifier(max_depth=None)
-deep_tree_clf.fit(X_train, y_train)
-print("Test set accuracy with Decision Trees: {:.2f}".format(deep_tree_clf.score(X_test,y_test)))
-#now scale the data
+#Scale the data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)