lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added zoo data

Browse Source
This commit is contained in:
mhjensen
2018-11-09 05:21:24 +01:00
parent 82f7105add
commit 47bf3582ee
9 changed files with 254 additions and 72 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/DecisionTrees/html/DecisionTrees-bs.html
+60 -22
View File
@@ -56,11 +56,13 @@ Automatically generated HTML file from DocOnce source
('A schematic procedure', 2, None, '___sec10'),
('A classification tree', 2, None, '___sec11'),
('Growing a classification tree', 2, None, '___sec12'),
('Pros and cons of trees, pros', 2, None, '___sec13'),
('Disadvantages', 2, None, '___sec14'),
('Bagging', 2, None, '___sec15'),
('Random forests', 2, None, '___sec16'),
('A simple scikit-learn example', 2, None, '___sec17')]}
('The zoo data', 2, None, '___sec13'),
('Pros and cons of trees, pros', 2, None, '___sec14'),
('Disadvantages', 2, None, '___sec15'),
('Bagging', 2, None, '___sec16'),
('Random forests', 2, None, '___sec17'),
('A simple scikit-learn example', 2, None, '___sec18'),
('Boosting', 2, None, '___sec19')]}
end of tocinfo -->
<body>
@@ -111,11 +113,13 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="#___sec10" style="font-size: 80%;">A schematic procedure</a></li>
<!-- navigation toc: --> <li><a href="#___sec11" style="font-size: 80%;">A classification tree</a></li>
<!-- navigation toc: --> <li><a href="#___sec12" style="font-size: 80%;">Growing a classification tree</a></li>
<!-- navigation toc: --> <li><a href="#___sec13" style="font-size: 80%;">Pros and cons of trees, pros</a></li>
<!-- navigation toc: --> <li><a href="#___sec14" style="font-size: 80%;">Disadvantages</a></li>
<!-- navigation toc: --> <li><a href="#___sec15" style="font-size: 80%;">Bagging</a></li>
<!-- navigation toc: --> <li><a href="#___sec16" style="font-size: 80%;">Random forests</a></li>
<!-- navigation toc: --> <li><a href="#___sec17" style="font-size: 80%;">A simple scikit-learn example</a></li>
<!-- navigation toc: --> <li><a href="#___sec13" style="font-size: 80%;">The zoo data</a></li>
<!-- navigation toc: --> <li><a href="#___sec14" style="font-size: 80%;">Pros and cons of trees, pros</a></li>
<!-- navigation toc: --> <li><a href="#___sec15" style="font-size: 80%;">Disadvantages</a></li>
<!-- navigation toc: --> <li><a href="#___sec16" style="font-size: 80%;">Bagging</a></li>
<!-- navigation toc: --> <li><a href="#___sec17" style="font-size: 80%;">Random forests</a></li>
<!-- navigation toc: --> <li><a href="#___sec18" style="font-size: 80%;">A simple scikit-learn example</a></li>
<!-- navigation toc: --> <li><a href="#___sec19" style="font-size: 80%;">Boosting</a></li>
</ul>
</li>
@@ -552,7 +556,7 @@ $$
is as small as possible. Here \( \overline{T} \) is
the number of terminal nodes of the tree \( T \) , \( R_m \) is the
rectangle (i.e. the subset of predictor space) corresponding to the $m$th terminal node.
rectangle (i.e. the subset of predictor space) corresponding to the \( m \)-th terminal node.
<p>
The tuning parameter \( \alpha \) controls a trade-off between the subtree&#8217;s
@@ -583,14 +587,16 @@ subtree corresponding to \( \alpha \).
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
<ol>
<li> Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
<ol type="a"></li>
<li> Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.</li>
<li> Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of \( \alpha \).</li>
</ol>
<li> Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we:</li>
<ul>
<li> repeat steps 1 and 2 on all but the \( k \)-th fold of the training data.</li>
<li> Then we valuate the mean squared prediction error on the data in the left-out \( k \)-th fold, as a function of \( \alpha \).</li>
<li> Finally we average the results for each value of \( alpha \), and pick \( \alpha \) to minimize the average error.</li>
</ul>
<li> Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we repeat Steps 1 and 2 on all but the $k$th fold of the training data. Then we valuate the mean squared prediction error on the data in the left-out $k$th fold, as a function of \( \alpha \).</li>
<li> Then we average the results for each value of \( alpha \), and pick \( \alpha \) to minimize the average error.</li>
<li> Return the subtree from Step 2 that corresponds to the chosen value of \( \alpha \).</li>
</ol>
</div>
@@ -643,7 +649,35 @@ than is the classification error rate.
<p>
<!-- !split -->
<h2 id="___sec13" class="anchor">Pros and cons of trees, pros </h2>
<h2 id="___sec13" class="anchor">The zoo data </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">pprint</span> <span style="color: #008000; font-weight: bold">import</span> pprint
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.tree</span> <span style="color: #008000; font-weight: bold">import</span> DecisionTreeClassifier
<span style="color: #408080; font-style: italic">#Import the dataset </span>
dataset <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>read_csv(<span style="color: #BA2121">&#39;data/zoo.csv&#39;</span>)
<span style="color: #408080; font-style: italic">#We drop the animal names since this is not a good feature to split the data on</span>
<span style="color: #408080; font-style: italic">#dataset=dataset.drop(&#39;animal_name&#39;,axis=1)</span>
<span style="color: #408080; font-style: italic">#Split the data into a training and a testing set</span>
train_features <span style="color: #666666">=</span> dataset<span style="color: #666666">.</span>iloc[:<span style="color: #666666">80</span>,:<span style="color: #666666">-1</span>]
test_features <span style="color: #666666">=</span> dataset<span style="color: #666666">.</span>iloc[<span style="color: #666666">80</span>:,:<span style="color: #666666">-1</span>]
train_targets <span style="color: #666666">=</span> dataset<span style="color: #666666">.</span>iloc[:<span style="color: #666666">80</span>,<span style="color: #666666">-1</span>]
test_targets <span style="color: #666666">=</span> dataset<span style="color: #666666">.</span>iloc[<span style="color: #666666">80</span>:,<span style="color: #666666">-1</span>]
<span style="color: #408080; font-style: italic">#Train the model</span>
tree <span style="color: #666666">=</span> DecisionTreeClassifier(criterion <span style="color: #666666">=</span> <span style="color: #BA2121">&#39;entropy&#39;</span>)<span style="color: #666666">.</span>fit(train_features,train_targets)
<span style="color: #408080; font-style: italic">#Predict the classes of new, unseen data</span>
prediction <span style="color: #666666">=</span> tree<span style="color: #666666">.</span>predict(test_features)
<span style="color: #408080; font-style: italic">#Check the accuracy</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The prediction accuracy is: &quot;</span>,tree<span style="color: #666666">.</span>score(test_features,test_targets)<span style="color: #666666">*100</span>,<span style="color: #BA2121">&quot;%&quot;</span>)
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec14" class="anchor">Pros and cons of trees, pros </h2>
<ul>
<li> White box, easy to interpret model. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches discussed earlier (think of support vector machines)</li>
@@ -657,7 +691,7 @@ than is the classification error rate.
<!-- !split -->
<h2 id="___sec14" class="anchor">Disadvantages </h2>
<h2 id="___sec15" class="anchor">Disadvantages </h2>
<ul>
<li> Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches</li>
@@ -674,7 +708,7 @@ However, by aggregating many decision trees, using methods like bagging, random
<p>
<!-- !split -->
<h2 id="___sec15" class="anchor">Bagging </h2>
<h2 id="___sec16" class="anchor">Bagging </h2>
<p>
The <b>plain</b> decision trees suffer from high
@@ -717,7 +751,7 @@ predictor, averaged over all \( B \) trees.
<p>
<!-- !split -->
<h2 id="___sec16" class="anchor">Random forests </h2>
<h2 id="___sec17" class="anchor">Random forests </h2>
<p>
Random forests provide an improvement over bagged trees by way of a
@@ -759,7 +793,7 @@ setting.
<p>
<!-- !split -->
<h2 id="___sec17" class="anchor">A simple scikit-learn example </h2>
<h2 id="___sec18" class="anchor">A simple scikit-learn example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -775,6 +809,10 @@ Random_Forest_model <span style="color: #666666">=</span> RandomForestClassifier
accuracy <span style="color: #666666">=</span> cross_validate(Random_Forest_model,X,Y,cv<span style="color: #666666">=10</span>)[<span style="color: #BA2121">&#39;test_score&#39;</span>]
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec19" class="anchor">Boosting </h2>
More material to come here.
<!-- ------------------- end of main content --------------- -->
doc/pub/DecisionTrees/html/DecisionTrees-reveal.html
+52 -12
View File
@@ -565,7 +565,7 @@ $$
is as small as possible. Here \( \overline{T} \) is
the number of terminal nodes of the tree \( T \) , \( R_m \) is the
rectangle (i.e. the subset of predictor space) corresponding to the $m$th terminal node.
rectangle (i.e. the subset of predictor space) corresponding to the \( m \)-th terminal node.
<p>
The tuning parameter \( \alpha \) controls a trade-off between the subtree&#8217;s
@@ -594,13 +594,18 @@ subtree corresponding to \( \alpha \).
<div class="alert alert-block alert-block alert-text-normal">
<b>Building a Regression Tree.</b>
<ol>
<p><li> Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
<ol type="a"></li>
<p><li> Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.</li>
<p><li> Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of \( \alpha \).</li>
</ol>
<p><li> Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we repeat Steps 1 and 2 on all but the $k$th fold of the training data. Then we valuate the mean squared prediction error on the data in the left-out $k$th fold, as a function of \( \alpha \).</li>
<p><li> Then we average the results for each value of \( alpha \), and pick \( \alpha \) to minimize the average error.</li>
<p><li> Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we:</li>
<ul>
<p><li> repeat steps 1 and 2 on all but the \( k \)-th fold of the training data.</li>
<p><li> Then we valuate the mean squared prediction error on the data in the left-out \( k \)-th fold, as a function of \( \alpha \).</li>
<p><li> Finally we average the results for each value of \( alpha \), and pick \( \alpha \) to minimize the average error.</li>
</ul>
<p><li> Return the subtree from Step 2 that corresponds to the chosen value of \( \alpha \).</li>
</ol>
</div>
@@ -651,7 +656,36 @@ than is the classification error rate.
<section>
<h2 id="___sec13">Pros and cons of trees, pros </h2>
<h2 id="___sec13">The zoo data </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">pandas</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pd</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">pprint</span> <span style="color: #8B008B; font-weight: bold">import</span> pprint
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.tree</span> <span style="color: #8B008B; font-weight: bold">import</span> DecisionTreeClassifier
<span style="color: #228B22">#Import the dataset </span>
dataset = pd.read_csv(<span style="color: #CD5555">&#39;data/zoo.csv&#39;</span>)
<span style="color: #228B22">#We drop the animal names since this is not a good feature to split the data on</span>
<span style="color: #228B22">#dataset=dataset.drop(&#39;animal_name&#39;,axis=1)</span>
<span style="color: #228B22">#Split the data into a training and a testing set</span>
train_features = dataset.iloc[:<span style="color: #B452CD">80</span>,:-<span style="color: #B452CD">1</span>]
test_features = dataset.iloc[<span style="color: #B452CD">80</span>:,:-<span style="color: #B452CD">1</span>]
train_targets = dataset.iloc[:<span style="color: #B452CD">80</span>,-<span style="color: #B452CD">1</span>]
test_targets = dataset.iloc[<span style="color: #B452CD">80</span>:,-<span style="color: #B452CD">1</span>]
<span style="color: #228B22">#Train the model</span>
tree = DecisionTreeClassifier(criterion = <span style="color: #CD5555">&#39;entropy&#39;</span>).fit(train_features,train_targets)
<span style="color: #228B22">#Predict the classes of new, unseen data</span>
prediction = tree.predict(test_features)
<span style="color: #228B22">#Check the accuracy</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The prediction accuracy is: &quot;</span>,tree.score(test_features,test_targets)*<span style="color: #B452CD">100</span>,<span style="color: #CD5555">&quot;%&quot;</span>)
</pre></div>
</section>
<section>
<h2 id="___sec14">Pros and cons of trees, pros </h2>
<ul>
<p><li> White box, easy to interpret model. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches discussed earlier (think of support vector machines)</li>
@@ -666,7 +700,7 @@ than is the classification error rate.
<section>
<h2 id="___sec14">Disadvantages </h2>
<h2 id="___sec15">Disadvantages </h2>
<ul>
<p><li> Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches</li>
@@ -684,7 +718,7 @@ However, by aggregating many decision trees, using methods like bagging, random
<section>
<h2 id="___sec15">Bagging </h2>
<h2 id="___sec16">Bagging </h2>
<p>
The <b>plain</b> decision trees suffer from high
@@ -727,7 +761,7 @@ predictor, averaged over all \( B \) trees.
<section>
<h2 id="___sec16">Random forests </h2>
<h2 id="___sec17">Random forests </h2>
<p>
Random forests provide an improvement over bagged trees by way of a
@@ -771,7 +805,7 @@ setting.
<section>
<h2 id="___sec17">A simple scikit-learn example </h2>
<h2 id="___sec18">A simple scikit-learn example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -789,6 +823,12 @@ accuracy = cross_validate(Random_Forest_model,X,Y,cv=<span style="color: #B452CD
</section>
<section>
<h2 id="___sec19">Boosting </h2>
More material to come here.
</section>
</div> <!-- class="slides" -->
</div> <!-- class="reveal" -->
doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
+53 -17
View File
@@ -76,11 +76,13 @@ div { text-align: justify; text-justify: inter-word; }
('A schematic procedure', 2, None, '___sec10'),
('A classification tree', 2, None, '___sec11'),
('Growing a classification tree', 2, None, '___sec12'),
('Pros and cons of trees, pros', 2, None, '___sec13'),
('Disadvantages', 2, None, '___sec14'),
('Bagging', 2, None, '___sec15'),
('Random forests', 2, None, '___sec16'),
('A simple scikit-learn example', 2, None, '___sec17')]}
('The zoo data', 2, None, '___sec13'),
('Pros and cons of trees, pros', 2, None, '___sec14'),
('Disadvantages', 2, None, '___sec15'),
('Bagging', 2, None, '___sec16'),
('Random forests', 2, None, '___sec17'),
('A simple scikit-learn example', 2, None, '___sec18'),
('Boosting', 2, None, '___sec19')]}
end of tocinfo -->
<body>
@@ -521,7 +523,7 @@ $$
is as small as possible. Here \( \overline{T} \) is
the number of terminal nodes of the tree \( T \) , \( R_m \) is the
rectangle (i.e. the subset of predictor space) corresponding to the $m$th terminal node.
rectangle (i.e. the subset of predictor space) corresponding to the \( m \)-th terminal node.
<p>
The tuning parameter \( \alpha \) controls a trade-off between the subtree&#8217;s
@@ -552,14 +554,16 @@ subtree corresponding to \( \alpha \).
<p>
<ol>
<li> Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
<ol type="a"></li>
<li> Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.</li>
<li> Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of \( \alpha \).</li>
</ol>
<li> Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we:</li>
<ul>
<li> repeat steps 1 and 2 on all but the \( k \)-th fold of the training data.</li>
<li> Then we valuate the mean squared prediction error on the data in the left-out \( k \)-th fold, as a function of \( \alpha \).</li>
<li> Finally we average the results for each value of \( alpha \), and pick \( \alpha \) to minimize the average error.</li>
</ul>
<li> Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we repeat Steps 1 and 2 on all but the $k$th fold of the training data. Then we valuate the mean squared prediction error on the data in the left-out $k$th fold, as a function of \( \alpha \).</li>
<li> Then we average the results for each value of \( alpha \), and pick \( \alpha \) to minimize the average error.</li>
<li> Return the subtree from Step 2 that corresponds to the chosen value of \( \alpha \).</li>
</ol>
</div>
@@ -611,7 +615,35 @@ than is the classification error rate.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Pros and cons of trees, pros </h2>
<h2 id="___sec13">The zoo data </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">pandas</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pd</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">pprint</span> <span style="color: #8B008B; font-weight: bold">import</span> pprint
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.tree</span> <span style="color: #8B008B; font-weight: bold">import</span> DecisionTreeClassifier
<span style="color: #228B22">#Import the dataset </span>
dataset = pd.read_csv(<span style="color: #CD5555">&#39;data/zoo.csv&#39;</span>)
<span style="color: #228B22">#We drop the animal names since this is not a good feature to split the data on</span>
<span style="color: #228B22">#dataset=dataset.drop(&#39;animal_name&#39;,axis=1)</span>
<span style="color: #228B22">#Split the data into a training and a testing set</span>
train_features = dataset.iloc[:<span style="color: #B452CD">80</span>,:-<span style="color: #B452CD">1</span>]
test_features = dataset.iloc[<span style="color: #B452CD">80</span>:,:-<span style="color: #B452CD">1</span>]
train_targets = dataset.iloc[:<span style="color: #B452CD">80</span>,-<span style="color: #B452CD">1</span>]
test_targets = dataset.iloc[<span style="color: #B452CD">80</span>:,-<span style="color: #B452CD">1</span>]
<span style="color: #228B22">#Train the model</span>
tree = DecisionTreeClassifier(criterion = <span style="color: #CD5555">&#39;entropy&#39;</span>).fit(train_features,train_targets)
<span style="color: #228B22">#Predict the classes of new, unseen data</span>
prediction = tree.predict(test_features)
<span style="color: #228B22">#Check the accuracy</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The prediction accuracy is: &quot;</span>,tree.score(test_features,test_targets)*<span style="color: #B452CD">100</span>,<span style="color: #CD5555">&quot;%&quot;</span>)
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">Pros and cons of trees, pros </h2>
<ul>
<li> White box, easy to interpret model. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches discussed earlier (think of support vector machines)</li>
@@ -625,7 +657,7 @@ than is the classification error rate.
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">Disadvantages </h2>
<h2 id="___sec15">Disadvantages </h2>
<ul>
<li> Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches</li>
@@ -642,7 +674,7 @@ However, by aggregating many decision trees, using methods like bagging, random
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Bagging </h2>
<h2 id="___sec16">Bagging </h2>
<p>
The <b>plain</b> decision trees suffer from high
@@ -685,7 +717,7 @@ predictor, averaged over all \( B \) trees.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">Random forests </h2>
<h2 id="___sec17">Random forests </h2>
<p>
Random forests provide an improvement over bagged trees by way of a
@@ -727,7 +759,7 @@ setting.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">A simple scikit-learn example </h2>
<h2 id="___sec18">A simple scikit-learn example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -743,6 +775,10 @@ Random_Forest_model = RandomForestClassifier(n_estimators=<span style="color: #B
accuracy = cross_validate(Random_Forest_model,X,Y,cv=<span style="color: #B452CD">10</span>)[<span style="color: #CD5555">&#39;test_score&#39;</span>]
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec19">Boosting </h2>
More material to come here.
<!-- ------------------- end of main content --------------- -->
doc/pub/DecisionTrees/html/DecisionTrees.html
+53 -17
View File
@@ -81,11 +81,13 @@ div { text-align: justify; text-justify: inter-word; }
('A schematic procedure', 2, None, '___sec10'),
('A classification tree', 2, None, '___sec11'),
('Growing a classification tree', 2, None, '___sec12'),
('Pros and cons of trees, pros', 2, None, '___sec13'),
('Disadvantages', 2, None, '___sec14'),
('Bagging', 2, None, '___sec15'),
('Random forests', 2, None, '___sec16'),
('A simple scikit-learn example', 2, None, '___sec17')]}
('The zoo data', 2, None, '___sec13'),
('Pros and cons of trees, pros', 2, None, '___sec14'),
('Disadvantages', 2, None, '___sec15'),
('Bagging', 2, None, '___sec16'),
('Random forests', 2, None, '___sec17'),
('A simple scikit-learn example', 2, None, '___sec18'),
('Boosting', 2, None, '___sec19')]}
end of tocinfo -->
<body>
@@ -526,7 +528,7 @@ $$
is as small as possible. Here \( \overline{T} \) is
the number of terminal nodes of the tree \( T \) , \( R_m \) is the
rectangle (i.e. the subset of predictor space) corresponding to the $m$th terminal node.
rectangle (i.e. the subset of predictor space) corresponding to the \( m \)-th terminal node.
<p>
The tuning parameter \( \alpha \) controls a trade-off between the subtree&#8217;s
@@ -557,14 +559,16 @@ subtree corresponding to \( \alpha \).
<p>
<ol>
<li> Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
<ol type="a"></li>
<li> Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.</li>
<li> Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of \( \alpha \).</li>
</ol>
<li> Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we:</li>
<ul>
<li> repeat steps 1 and 2 on all but the \( k \)-th fold of the training data.</li>
<li> Then we valuate the mean squared prediction error on the data in the left-out \( k \)-th fold, as a function of \( \alpha \).</li>
<li> Finally we average the results for each value of \( alpha \), and pick \( \alpha \) to minimize the average error.</li>
</ul>
<li> Use for example \( K \)-fold cross-validation to choose \( \alpha \). Divide the training observations into \( K \) folds. For each \( k=1,2,\dots,K \) we repeat Steps 1 and 2 on all but the $k$th fold of the training data. Then we valuate the mean squared prediction error on the data in the left-out $k$th fold, as a function of \( \alpha \).</li>
<li> Then we average the results for each value of \( alpha \), and pick \( \alpha \) to minimize the average error.</li>
<li> Return the subtree from Step 2 that corresponds to the chosen value of \( \alpha \).</li>
</ol>
</div>
@@ -616,7 +620,35 @@ than is the classification error rate.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Pros and cons of trees, pros </h2>
<h2 id="___sec13">The zoo data </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">pprint</span> <span style="color: #008000; font-weight: bold">import</span> pprint
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.tree</span> <span style="color: #008000; font-weight: bold">import</span> DecisionTreeClassifier
<span style="color: #408080; font-style: italic">#Import the dataset </span>
dataset <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>read_csv(<span style="color: #BA2121">&#39;data/zoo.csv&#39;</span>)
<span style="color: #408080; font-style: italic">#We drop the animal names since this is not a good feature to split the data on</span>
<span style="color: #408080; font-style: italic">#dataset=dataset.drop(&#39;animal_name&#39;,axis=1)</span>
<span style="color: #408080; font-style: italic">#Split the data into a training and a testing set</span>
train_features <span style="color: #666666">=</span> dataset<span style="color: #666666">.</span>iloc[:<span style="color: #666666">80</span>,:<span style="color: #666666">-1</span>]
test_features <span style="color: #666666">=</span> dataset<span style="color: #666666">.</span>iloc[<span style="color: #666666">80</span>:,:<span style="color: #666666">-1</span>]
train_targets <span style="color: #666666">=</span> dataset<span style="color: #666666">.</span>iloc[:<span style="color: #666666">80</span>,<span style="color: #666666">-1</span>]
test_targets <span style="color: #666666">=</span> dataset<span style="color: #666666">.</span>iloc[<span style="color: #666666">80</span>:,<span style="color: #666666">-1</span>]
<span style="color: #408080; font-style: italic">#Train the model</span>
tree <span style="color: #666666">=</span> DecisionTreeClassifier(criterion <span style="color: #666666">=</span> <span style="color: #BA2121">&#39;entropy&#39;</span>)<span style="color: #666666">.</span>fit(train_features,train_targets)
<span style="color: #408080; font-style: italic">#Predict the classes of new, unseen data</span>
prediction <span style="color: #666666">=</span> tree<span style="color: #666666">.</span>predict(test_features)
<span style="color: #408080; font-style: italic">#Check the accuracy</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The prediction accuracy is: &quot;</span>,tree<span style="color: #666666">.</span>score(test_features,test_targets)<span style="color: #666666">*100</span>,<span style="color: #BA2121">&quot;%&quot;</span>)
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">Pros and cons of trees, pros </h2>
<ul>
<li> White box, easy to interpret model. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches discussed earlier (think of support vector machines)</li>
@@ -630,7 +662,7 @@ than is the classification error rate.
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">Disadvantages </h2>
<h2 id="___sec15">Disadvantages </h2>
<ul>
<li> Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches</li>
@@ -647,7 +679,7 @@ However, by aggregating many decision trees, using methods like bagging, random
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Bagging </h2>
<h2 id="___sec16">Bagging </h2>
<p>
The <b>plain</b> decision trees suffer from high
@@ -690,7 +722,7 @@ predictor, averaged over all \( B \) trees.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">Random forests </h2>
<h2 id="___sec17">Random forests </h2>
<p>
Random forests provide an improvement over bagged trees by way of a
@@ -732,7 +764,7 @@ setting.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">A simple scikit-learn example </h2>
<h2 id="___sec18">A simple scikit-learn example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -748,6 +780,10 @@ Random_Forest_model <span style="color: #666666">=</span> RandomForestClassifier
accuracy <span style="color: #666666">=</span> cross_validate(Random_Forest_model,X,Y,cv<span style="color: #666666">=10</span>)[<span style="color: #BA2121">&#39;test_score&#39;</span>]
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec19">Boosting </h2>
More material to come here.
<!-- ------------------- end of main content --------------- -->
doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/DecisionTrees/pdf/DecisionTrees-beamer-handouts2x3.pdf
BIN
View File
Binary file not shown.
doc/pub/DecisionTrees/pdf/DecisionTrees-beamer.pdf
BIN
View File
Binary file not shown.
doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf
BIN
View File
Binary file not shown.
doc/src/DecisionTrees/DecisionTrees.do.txt
+36 -4
View File
@@ -370,7 +370,7 @@ For each value of $\alpha$ there corresponds a subtree $T \in T_0$ such that
!et
is as small as possible. Here $\overline{T}$ is
the number of terminal nodes of the tree $T$ , $R_m$ is the
rectangle (i.e. the subset of predictor space) corresponding to the $m$th terminal node.
rectangle (i.e. the subset of predictor space) corresponding to the $m$-th terminal node.
The tuning parameter $\alpha$ controls a trade-off between the subtree’s
com- plexity and its fit to the training data. When $\alpha = 0$, then the
@@ -395,9 +395,11 @@ subtree corresponding to $\alpha$.
!bblock Building a Regression Tree
o Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
o Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of $\alpha$.
o Use for example $K$-fold cross-validation to choose $\alpha$. Divide the training observations into $K$ folds. For each $k=1,2,\dots,K$ we repeat Steps 1 and 2 on all but the $k$th fold of the training data. Then we valuate the mean squared prediction error on the data in the left-out $k$th fold, as a function of $\alpha$.
o Then we average the results for each value of $alpha$, and pick $\alpha$ to minimize the average error.
o Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of $\alpha$.
o Use for example $K$-fold cross-validation to choose $\alpha$. Divide the training observations into $K$ folds. For each $k=1,2,\dots,K$ we:
* repeat steps 1 and 2 on all but the $k$-th fold of the training data.
* Then we valuate the mean squared prediction error on the data in the left-out $k$-th fold, as a function of $\alpha$.
* Finally we average the results for each value of $alpha$, and pick $\alpha$ to minimize the average error.
o Return the subtree from Step 2 that corresponds to the chosen value of $\alpha$.
!eblock
@@ -439,6 +441,31 @@ split, since these two approaches are more sensitive to node purity
than is the classification error rate.
!split
===== The zoo data =====
!bc pycod
import pandas as pd
import numpy as np
from pprint import pprint
from sklearn.tree import DecisionTreeClassifier
#Import the dataset
dataset = pd.read_csv('data/zoo.csv')
#We drop the animal names since this is not a good feature to split the data on
#dataset=dataset.drop('animal_name',axis=1)
#Split the data into a training and a testing set
train_features = dataset.iloc[:80,:-1]
test_features = dataset.iloc[80:,:-1]
train_targets = dataset.iloc[:80,-1]
test_targets = dataset.iloc[80:,-1]
#Train the model
tree = DecisionTreeClassifier(criterion = 'entropy').fit(train_features,train_targets)
#Predict the classes of new, unseen data
prediction = tree.predict(test_features)
#Check the accuracy
print("The prediction accuracy is: ",tree.score(test_features,test_targets)*100,"%")
!ec
!split
===== Pros and cons of trees, pros =====
@@ -553,3 +580,8 @@ Random_Forest_model = RandomForestClassifier(n_estimators=100,criterion="entropy
#Cross validation
accuracy = cross_validate(Random_Forest_model,X,Y,cv=10)['test_score']
!ec
!split
===== Boosting =====
More material to come here.