lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
efa30bcf9e512b04a00a1a196151e6aa948bd6de
FYS-STK4155/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
T
mhjensen 50471e835b updated material about trees, more to come 
nothing about random forests
2018-11-02 06:05:54 +01:00

378 lines
20 KiB
HTML
Raw Blame History

<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Trees, forests and all that">
<title>Data Analysis and Machine Learning: Trees, forests and all that</title>
<link href="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
<script src="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/js/highlight.pack.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
<link href="https://thomasf.github.io/solarized-css/solarized-light.min.css" rel="stylesheet">
<style type="text/css">
h1 {color: #b58900;} /* yellow */
/* h1 {color: #cb4b16;} orange */
/* h1 {color: #d33682;} magenta, the original choice of thomasf */
code { padding: 0px; background-color: inherit; }
pre {
border: 0pt solid #93a1a1;
box-shadow: none;
}
.alert-text-small { font-size: 80%; }
.alert-text-large { font-size: 130%; }
.alert-text-normal { font-size: 90%; }
.alert {
padding:8px 35px 8px 14px; margin-bottom:18px;
text-shadow:0 1px 0 rgba(255,255,255,0.5);
border:1px solid #93a1a1;
border-radius: 4px;
-webkit-border-radius: 4px;
-moz-border-radius: 4px;
color: #555;
background-color: #eee8d5;
background-position: 10px 5px;
background-repeat: no-repeat;
background-size: 38px;
padding-left: 55px;
width: 75%;
}
.alert-block {padding-top:14px; padding-bottom:14px}
.alert-block > p, .alert-block > ul {margin-bottom:1em}
.alert li {margin-top: 1em}
.alert-block p+p {margin-top:5px}
.alert-notice { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_question.png); }
div { text-align: justify; text-justify: inter-word; }
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Decision trees, overarching aims', 2, None, '___sec0'),
('Nodes, leafs, roots and branches', 2, None, '___sec1'),
('How do we set it up?', 2, None, '___sec2'),
('Decision trees and Regression', 2, None, '___sec3'),
('Maxwell-Boltzmann velocity distribution', 2, None, '___sec4')]}
end of tocinfo -->
<body>
<!-- ------------------- main content ---------------------- -->
<center><h1>Data Analysis and Machine Learning: Trees, forests and all that</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
<center>
<b>Morten Hjorth-Jensen</b> [1, 2]
</center>
<p>
<!-- institution(s) -->
<center>[1] <b>Department of Physics, University of Oslo</b></center>
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Decision trees, overarching aims </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>
Decision trees are supervised learning algorithms used for both,
classification and regression tasks where we will concentrate on
classification in this first part of our decision tree tutorial.
Decision trees are assigned to the information based learning
algorithms which use different measures of information gain for
learning. We can use decision trees for issues where we have
continuous but also categorical input and target features.
</div>
<p>
<!-- !split -->
<h2 id="___sec1">Nodes, leafs, roots and branches </h2>
<p>
The main idea of decision trees
is to find those descriptive features which contain the most
<b>information</b> regarding the target feature and then split the dataset
along the values of these features such that the target feature values
for the resulting sub\_datasets are as pure as possible.
<p>
The descriptive feature which leaves the target feature most purely is said
to be the most informative one. This process of finding the <b>most
informative</b> feature is done until we accomplish a stopping criteria
where we then finally end up in so called <b>leaf nodes</b>.
<p>
The leaf nodes
contain the predictions we will make for new query instances presented
to our trained model. This is possible since the model has kind of
learned the underlying structure of the training data and hence can,
given some assumptions, make predictions about the target feature value
(class) of unseen query instances.
<p>
A decision tree mainly contains of a <b>root node</b>, <b>interior nodes</b>,
and <b>leaf nodes</b> which are then connected by <b>branches</b>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">How do we set it up? </h2>
<p>
In simplified terms, the process of training a decision tree and
predicting the target features of query instances is as follows:
<ol>
<li> Present a dataset containing of a number of training instances characterized by a number of descriptive features and a target feature</li>
<li> Train the decision tree model by continuously splitting the target feature along the values of the descriptive features using a measure of information gain during the training process</li>
<li> Grow the tree until we accomplish a stopping criteria create leaf nodes which represent the <em>predictions</em> we want to make for new query instances</li>
<li> Show query instances to the tree and run down the tree until we arrive at leaf nodes</li>
</ol>
Then we are essentially done!
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Decision trees and Regression </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> PolynomialFeatures
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression
steps=<span style="color: #B452CD">250</span>
distance=<span style="color: #B452CD">0</span>
x=<span style="color: #B452CD">0</span>
distance_list=[]
steps_list=[]
<span style="color: #8B008B; font-weight: bold">while</span> x&lt;steps:
distance+=np.random.randint(-<span style="color: #B452CD">1</span>,<span style="color: #B452CD">2</span>)
distance_list.append(distance)
x+=<span style="color: #B452CD">1</span>
steps_list.append(x)
plt.plot(steps_list,distance_list, color=<span style="color: #CD5555">&#39;green&#39;</span>, label=<span style="color: #CD5555">&quot;Random Walk Data&quot;</span>)
steps_list=np.asarray(steps_list)
distance_list=np.asarray(distance_list)
X=steps_list[:,np.newaxis]
<span style="color: #228B22">#Polynomial fits</span>
<span style="color: #228B22">#Degree 2</span>
poly_features=PolynomialFeatures(degree=<span style="color: #B452CD">2</span>, include_bias=<span style="color: #658b00">False</span>)
X_poly=poly_features.fit_transform(X)
lin_reg=LinearRegression()
poly_fit=lin_reg.fit(X_poly,distance_list)
b=lin_reg.coef_
c=lin_reg.intercept_
<span style="color: #8B008B; font-weight: bold">print</span> (<span style="color: #CD5555">&quot;2nd degree coefficients:&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span> (<span style="color: #CD5555">&quot;zero power: &quot;</span>,c)
<span style="color: #8B008B; font-weight: bold">print</span> (<span style="color: #CD5555">&quot;first power: &quot;</span>, b[<span style="color: #B452CD">0</span>])
<span style="color: #8B008B; font-weight: bold">print</span> (<span style="color: #CD5555">&quot;second power: &quot;</span>,b[<span style="color: #B452CD">1</span>])
z = np.arange(<span style="color: #B452CD">0</span>, steps, .<span style="color: #B452CD">01</span>)
z_mod=b[<span style="color: #B452CD">1</span>]*z**<span style="color: #B452CD">2</span>+b[<span style="color: #B452CD">0</span>]*z+c
fit_mod=b[<span style="color: #B452CD">1</span>]*X**<span style="color: #B452CD">2</span>+b[<span style="color: #B452CD">0</span>]*X+c
plt.plot(z, z_mod, color=<span style="color: #CD5555">&#39;r&#39;</span>, label=<span style="color: #CD5555">&quot;2nd Degree Fit&quot;</span>)
plt.title(<span style="color: #CD5555">&quot;Polynomial Regression&quot;</span>)
plt.xlabel(<span style="color: #CD5555">&quot;Steps&quot;</span>)
plt.ylabel(<span style="color: #CD5555">&quot;Distance&quot;</span>)
<span style="color: #228B22">#Degree 10</span>
poly_features10=PolynomialFeatures(degree=<span style="color: #B452CD">10</span>, include_bias=<span style="color: #658b00">False</span>)
X_poly10=poly_features10.fit_transform(X)
poly_fit10=lin_reg.fit(X_poly10,distance_list)
y_plot=poly_fit10.predict(X_poly10)
plt.plot(X, y_plot, color=<span style="color: #CD5555">&#39;black&#39;</span>, label=<span style="color: #CD5555">&quot;10th Degree Fit&quot;</span>)
plt.legend()
plt.show()
<span style="color: #228B22">#Decision Tree Regression</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.tree</span> <span style="color: #8B008B; font-weight: bold">import</span> DecisionTreeRegressor
regr_1=DecisionTreeRegressor(max_depth=<span style="color: #B452CD">2</span>)
regr_2=DecisionTreeRegressor(max_depth=<span style="color: #B452CD">5</span>)
regr_3=DecisionTreeRegressor(max_depth=<span style="color: #B452CD">7</span>)
regr_1.fit(X, distance_list)
regr_2.fit(X, distance_list)
regr_3.fit(X, distance_list)
X_test = np.arange(<span style="color: #B452CD">0.0</span>, steps, <span style="color: #B452CD">0.01</span>)[:, np.newaxis]
y_1 = regr_1.predict(X_test)
y_2 = regr_2.predict(X_test)
y_3=regr_3.predict(X_test)
<span style="color: #228B22"># Plot the results</span>
plt.figure()
plt.scatter(X, distance_list, s=<span style="color: #B452CD">2.5</span>, c=<span style="color: #CD5555">&quot;black&quot;</span>, label=<span style="color: #CD5555">&quot;data&quot;</span>)
plt.plot(X_test, y_1, color=<span style="color: #CD5555">&quot;red&quot;</span>,
label=<span style="color: #CD5555">&quot;max_depth=2&quot;</span>, linewidth=<span style="color: #B452CD">2</span>)
plt.plot(X_test, y_2, color=<span style="color: #CD5555">&quot;green&quot;</span>, label=<span style="color: #CD5555">&quot;max_depth=5&quot;</span>, linewidth=<span style="color: #B452CD">2</span>)
plt.plot(X_test, y_3, color=<span style="color: #CD5555">&quot;m&quot;</span>, label=<span style="color: #CD5555">&quot;max_depth=7&quot;</span>, linewidth=<span style="color: #B452CD">2</span>)
plt.xlabel(<span style="color: #CD5555">&quot;Data&quot;</span>)
plt.ylabel(<span style="color: #CD5555">&quot;Darget&quot;</span>)
plt.title(<span style="color: #CD5555">&quot;Decision Tree Regression&quot;</span>)
plt.legend()
plt.show()
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Maxwell-Boltzmann velocity distribution </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># Program to test the Metropolis algorithm with one particle at given temp in</span>
<span style="color: #228B22"># one dimension</span>
<span style="color: #228B22">#!/usr/bin/env python</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.mlab</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">mlab</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">random</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">math</span> <span style="color: #8B008B; font-weight: bold">import</span> sqrt, exp, log
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> PolynomialFeatures
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression
<span style="color: #228B22"># initialize the rng with a seed</span>
random.seed()
<span style="color: #228B22"># Hard coding of input parameters</span>
MCcycles = <span style="color: #B452CD">100000</span>
Temperature = <span style="color: #B452CD">2.0</span>
beta = <span style="color: #B452CD">1.</span>/Temperature
InitialVelocity = -<span style="color: #B452CD">2.0</span>
CurrentVelocity = InitialVelocity
Energy = <span style="color: #B452CD">0.5</span>*InitialVelocity*InitialVelocity
VelocityRange = <span style="color: #B452CD">10</span>*sqrt(Temperature)
VelocityStep = <span style="color: #B452CD">2</span>*VelocityRange/<span style="color: #B452CD">10.</span>
AverageEnergy = Energy
AverageEnergy2 = Energy*Energy
VelocityValues = np.zeros(MCcycles)
<span style="color: #228B22"># The Monte Carlo sampling with Metropolis starts here</span>
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span> (<span style="color: #B452CD">1</span>, MCcycles, <span style="color: #B452CD">1</span>):
TrialVelocity = CurrentVelocity + (<span style="color: #B452CD">2.0</span>*random.random() - <span style="color: #B452CD">1.0</span>)*VelocityStep
EnergyChange = <span style="color: #B452CD">0.5</span>*(TrialVelocity*TrialVelocity -CurrentVelocity*CurrentVelocity);
<span style="color: #8B008B; font-weight: bold">if</span> random.random() &lt;= exp(-beta*EnergyChange):
CurrentVelocity = TrialVelocity
Energy += EnergyChange
VelocityValues[i] = CurrentVelocity
AverageEnergy += Energy
AverageEnergy2 += Energy*Energy
<span style="color: #228B22">#Final averages</span>
AverageEnergy = AverageEnergy/MCcycles
AverageEnergy2 = AverageEnergy2/MCcycles
Variance = AverageEnergy2 - AverageEnergy*AverageEnergy
<span style="color: #8B008B; font-weight: bold">print</span>(AverageEnergy, Variance)
n, bins, patches = plt.hist(VelocityValues, <span style="color: #B452CD">400</span>, facecolor=<span style="color: #CD5555">&#39;green&#39;</span>)
plt.xlabel(<span style="color: #CD5555">&#39;$v$&#39;</span>)
plt.ylabel(<span style="color: #CD5555">&#39;Velocity distribution P(v)&#39;</span>)
plt.title(<span style="color: #CD5555">r&#39;Velocity histogram at $k_BT=2$&#39;</span>)
plt.axis([-<span style="color: #B452CD">5</span>, <span style="color: #B452CD">5</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">600</span>])
plt.grid(<span style="color: #658b00">True</span>)
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">collections</span> <span style="color: #8B008B; font-weight: bold">import</span> Counter
<span style="color: #228B22">#print (Counter(VelocityValues))</span>
<span style="color: #8B008B; font-weight: bold">print</span> (VelocityValues[:<span style="color: #B452CD">20</span>])
VelocityValues=<span style="color: #658b00">list</span>(Counter(VelocityValues).keys())
d=<span style="color: #658b00">list</span>(Counter(VelocityValues).values())
VelocityValues=np.asarray(VelocityValues)[:, np.newaxis]
d=np.asarray(d)
<span style="color: #8B008B; font-weight: bold">print</span> (VelocityValues.shape, d.shape)
plt.scatter(VelocityValues, d)
plt.show()
<span style="color: #228B22">#2nd Degree Polynomial</span>
poly_feat=PolynomialFeatures(degree=<span style="color: #B452CD">20</span>, include_bias=<span style="color: #658b00">False</span>)
X_poly=poly_feat.fit_transform(VelocityValues)
lin_reg=LinearRegression()
poly_fit=lin_reg.fit(X_poly,d)
y_plot=poly_fit.predict(X_poly)
plt.title(<span style="color: #CD5555">&quot;Polynomial Fit&quot;</span>)
plt.plot(VelocityValues, y_plot, color=<span style="color: #CD5555">&#39;black&#39;</span>, label=<span style="color: #CD5555">&quot;Fit&quot;</span>)
plt.show()
<span style="color: #228B22">#Decision Trees</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.tree</span> <span style="color: #8B008B; font-weight: bold">import</span> DecisionTreeRegressor
regr_1=DecisionTreeRegressor(max_depth=<span style="color: #B452CD">2</span>)
regr_2=DecisionTreeRegressor(max_depth=<span style="color: #B452CD">5</span>)
regr_3=DecisionTreeRegressor(max_depth=<span style="color: #B452CD">7</span>)
regr_1.fit(VelocityValues, d)
regr_2.fit(VelocityValues, d)
regr_3.fit(VelocityValues, d)
X_test = np.arange(<span style="color: #B452CD">0.0</span>, MCcycles, <span style="color: #B452CD">0.01</span>)[:, np.newaxis]
y_1=regr_1.predict(X_test)
y_2=regr_2.predict(X_test)
y_3=regr_3.predict(X_test)
plt.title(<span style="color: #CD5555">&quot;Decision Tree&quot;</span>)
plt.plot(X_test, y_1, color=<span style="color: #CD5555">&quot;red&quot;</span>, label=<span style="color: #CD5555">&quot;max_depth=2&quot;</span>, linewidth=<span style="color: #B452CD">2</span>)
plt.plot(X_test, y_2, color=<span style="color: #CD5555">&quot;green&quot;</span>, label=<span style="color: #CD5555">&quot;max_depth=5&quot;</span>, linewidth=<span style="color: #B452CD">2</span>)
plt.plot(X_test, y_3, color=<span style="color: #CD5555">&quot;m&quot;</span>, label=<span style="color: #CD5555">&quot;max_depth=7&quot;</span>, linewidth=<span style="color: #B452CD">2</span>)
plt.show()
<span style="color: #228B22">#Separate each frequency not in one specific velocity, but in a range of values,</span>
<span style="color: #228B22">#i.e. frequency of all velocities in range -5 to -4.9, -4.9 to -4.8, etc...</span>
</pre></div>
<p>
<!-- ------------------- end of main content --------------- -->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink