lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updated material about trees, more to come

Browse Source 
nothing about random forests
This commit is contained in:
mhjensen
2018-11-02 06:05:54 +01:00
parent 03bc12598b
commit 50471e835b
9 changed files with 270 additions and 305 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/DecisionTrees/html/DecisionTrees-bs.html
+60 -66
View File
@@ -6,9 +6,9 @@ Automatically generated HTML file from DocOnce source
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees">
<meta name="description" content="Data Analysis and Machine Learning: Trees, forests and all that">
<title>Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees</title>
<title>Data Analysis and Machine Learning: Trees, forests and all that</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
@@ -41,8 +41,10 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('Decision trees, overarching aims', 2, None, '___sec0'),
('Nearest Neighbors', 2, None, '___sec1'),
('Decision trees and Regression', 2, None, '___sec2')]}
('Nodes, leafs, roots and branches', 2, None, '___sec1'),
('How do we set it up?', 2, None, '___sec2'),
('Decision trees and Regression', 2, None, '___sec3'),
('Maxwell-Boltzmann velocity distribution', 2, None, '___sec4')]}
end of tocinfo -->
<body>
@@ -56,7 +58,7 @@ end of tocinfo -->
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="DecisionTrees-bs.html">Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees</a>
<a class="navbar-brand" href="DecisionTrees-bs.html">Data Analysis and Machine Learning: Trees, forests and all that</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
@@ -65,8 +67,10 @@ end of tocinfo -->
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">Decision trees, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Nearest Neighbors</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Decision trees and Regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Nodes, leafs, roots and branches</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">How do we set it up?</a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">Decision trees and Regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">Maxwell-Boltzmann velocity distribution</a></li>
</ul>
</li>
@@ -84,7 +88,7 @@ end of tocinfo -->
<div class="jumbotron">
<center><h1>Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees</h1></center> <!-- document title -->
<center><h1>Data Analysis and Machine Learning: Trees, forests and all that</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -100,7 +104,7 @@ end of tocinfo -->
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 1, 2018</h4></center> <!-- date -->
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- potential-jumbotron-button -->
@@ -128,72 +132,57 @@ continuous but also categorical input and target features.
<p>
<!-- !split -->
<!-- !split -->
<h2 id="___sec1" class="anchor">Nodes, leafs, roots and branches </h2>
<h2 id="___sec1" class="anchor">Nearest Neighbors </h2>
<p>
The main idea of decision trees
is to find those descriptive features which contain the most
<b>information</b> regarding the target feature and then split the dataset
along the values of these features such that the target feature values
for the resulting sub\_datasets are as pure as possible.
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">mglearn</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> linear_model
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LinearRegression
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> PolynomialFeatures
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.neighbors</span> <span style="color: #008000; font-weight: bold">import</span> KNeighborsClassifier
<p>
The descriptive feature which leaves the target feature most purely is said
to be the most informative one. This process of finding the <b>most
informative</b> feature is done until we accomplish a stopping criteria
where we then finally end up in so called <b>leaf nodes</b>.
<span style="color: #408080; font-style: italic"># Generate sample data</span>
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sort(<span style="color: #666666">5*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">40</span>,<span style="color: #666666">1</span>), axis<span style="color: #666666">=0</span>)
y <span style="color: #666666">=</span> X<span style="color: #666666">**3</span>
y<span style="color: #666666">=</span>y<span style="color: #666666">.</span>ravel()
<p>
The leaf nodes
contain the predictions we will make for new query instances presented
to our trained model. This is possible since the model has kind of
learned the underlying structure of the training data and hence can,
given some assumptions, make predictions about the target feature value
(class) of unseen query instances.
<span style="color: #408080; font-style: italic"># Add noise to targets</span>
X[::<span style="color: #666666">4</span>] <span style="color: #666666">+=</span>(<span style="color: #666666">0.5</span> <span style="color: #666666">-</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">1</span>))
y[::<span style="color: #666666">5</span>] <span style="color: #666666">+=</span>(<span style="color: #666666">0.5</span> <span style="color: #666666">-</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">8</span>))
<p>
A decision tree mainly contains of a <b>root node</b>, <b>interior nodes</b>,
and <b>leaf nodes</b> which are then connected by <b>branches</b>.
a<span style="color: #666666">=</span>np<span style="color: #666666">.</span>array(X)
b<span style="color: #666666">=</span>np<span style="color: #666666">.</span>array(y)
X_train<span style="color: #666666">=</span>a[:<span style="color: #666666">19</span>]
X_test<span style="color: #666666">=</span>a[<span style="color: #666666">19</span>:]
y_train<span style="color: #666666">=</span>b[:<span style="color: #666666">19</span>]
y_test<span style="color: #666666">=</span>b[<span style="color: #666666">19</span>:]
model<span style="color: #666666">=</span>Pipeline([(<span style="color: #BA2121">&#39;poly&#39;</span>, PolynomialFeatures(degree<span style="color: #666666">=3</span>)),(<span style="color: #BA2121">&#39;linear&#39;</span>, LinearRegression(fit_intercept<span style="color: #666666">=</span><span style="color: #008000">False</span>))])
model<span style="color: #666666">=</span>model<span style="color: #666666">.</span>fit(X_train, y_train)
pred<span style="color: #666666">=</span>model<span style="color: #666666">.</span>predict(X_test)
poly<span style="color: #666666">=</span>PolynomialFeatures(degree<span style="color: #666666">=3</span>)
poly<span style="color: #666666">.</span>fit_transform(X_train, y_train)
plt<span style="color: #666666">.</span>scatter(X_test, y_test)
plt<span style="color: #666666">.</span>plot(X_test, pred, color<span style="color: #666666">=</span><span style="color: #BA2121">&#39;green&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #008000; font-weight: bold">print</span> (model<span style="color: #666666">.</span>score(X_test,y_test))
<span style="color: #008000; font-weight: bold">print</span> (<span style="color: #BA2121">&quot;---------K-Nearest Neighbors-------&quot;</span>)
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;neighbors_settings=range(1,11)</span>
<span style="color: #BA2121; font-style: italic">for n_neighbors in neighbors_settings:</span>
<span style="color: #BA2121; font-style: italic"> clf=KNeighborsClassifier(n_neighbors=n_neighbors)</span>
<span style="color: #BA2121; font-style: italic"> clf.fit(X_train, y_train)</span>
<span style="color: #BA2121; font-style: italic"> training_accuracy.append(clf.score(X_train, y_train))</span>
<span style="color: #BA2121; font-style: italic"> test_accuracy.append(clf.score(X_test, y_test))</span>
<span style="color: #BA2121; font-style: italic">print (mglearn.plots.plot_knn_regression(n_neighbors=3))&quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.neighbors</span> <span style="color: #008000; font-weight: bold">import</span> KNeighborsRegressor
X, y<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>datasets<span style="color: #666666">.</span>make_wave(n_samples<span style="color: #666666">=40</span>)
reg <span style="color: #666666">=</span> KNeighborsRegressor(n_neighbors<span style="color: #666666">=3</span>)
reg<span style="color: #666666">.</span>fit(X_train, y_train)
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec2" class="anchor">Decision trees and Regression </h2>
<h2 id="___sec2" class="anchor">How do we set it up? </h2>
<p>
In simplified terms, the process of training a decision tree and
predicting the target features of query instances is as follows:
<ol>
<li> Present a dataset containing of a number of training instances characterized by a number of descriptive features and a target feature</li>
<li> Train the decision tree model by continuously splitting the target feature along the values of the descriptive features using a measure of information gain during the training process</li>
<li> Grow the tree until we accomplish a stopping criteria create leaf nodes which represent the <em>predictions</em> we want to make for new query instances</li>
<li> Show query instances to the tree and run down the tree until we arrive at leaf nodes</li>
</ol>
Then we are essentially done!
<p>
<!-- !split -->
<h2 id="___sec3" class="anchor">Decision trees and Regression </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -286,6 +275,11 @@ plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot
plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec4" class="anchor">Maxwell-Boltzmann velocity distribution </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
doc/pub/DecisionTrees/html/DecisionTrees-reveal.html
+52 -61
View File
@@ -3,9 +3,9 @@
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees">
<meta name="description" content="Data Analysis and Machine Learning: Trees, forests and all that">
<title>Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees</title>
<title>Data Analysis and Machine Learning: Trees, forests and all that</title>
@@ -116,7 +116,7 @@ td.padding {
<center><h1 style="text-align: center;">Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees</h1></center> <!-- document title -->
<center><h1 style="text-align: center;">Data Analysis and Machine Learning: Trees, forests and all that</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -132,7 +132,7 @@ td.padding {
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Nov 1, 2018</h4></center> <!-- date -->
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<br>
<p>
@@ -161,71 +161,56 @@ continuous but also categorical input and target features.
<section>
<h2 id="___sec1">Nearest Neighbors </h2>
<h2 id="___sec1">Nodes, leafs, roots and branches </h2>
<p>
The main idea of decision trees
is to find those descriptive features which contain the most
<b>information</b> regarding the target feature and then split the dataset
along the values of these features such that the target feature values
for the resulting sub\_datasets are as pure as possible.
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">mglearn</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn</span> <span style="color: #8B008B; font-weight: bold">import</span> linear_model
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> PolynomialFeatures
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.pipeline</span> <span style="color: #8B008B; font-weight: bold">import</span> Pipeline
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.neighbors</span> <span style="color: #8B008B; font-weight: bold">import</span> KNeighborsClassifier
<p>
The descriptive feature which leaves the target feature most purely is said
to be the most informative one. This process of finding the <b>most
informative</b> feature is done until we accomplish a stopping criteria
where we then finally end up in so called <b>leaf nodes</b>.
<span style="color: #228B22"># Generate sample data</span>
X = np.sort(<span style="color: #B452CD">5</span>*np.random.rand(<span style="color: #B452CD">40</span>,<span style="color: #B452CD">1</span>), axis=<span style="color: #B452CD">0</span>)
y = X**<span style="color: #B452CD">3</span>
y=y.ravel()
<p>
The leaf nodes
contain the predictions we will make for new query instances presented
to our trained model. This is possible since the model has kind of
learned the underlying structure of the training data and hence can,
given some assumptions, make predictions about the target feature value
(class) of unseen query instances.
<span style="color: #228B22"># Add noise to targets</span>
X[::<span style="color: #B452CD">4</span>] +=(<span style="color: #B452CD">0.5</span> - np.random.rand(<span style="color: #B452CD">1</span>))
y[::<span style="color: #B452CD">5</span>] +=(<span style="color: #B452CD">0.5</span> - np.random.rand(<span style="color: #B452CD">8</span>))
a=np.array(X)
b=np.array(y)
X_train=a[:<span style="color: #B452CD">19</span>]
X_test=a[<span style="color: #B452CD">19</span>:]
y_train=b[:<span style="color: #B452CD">19</span>]
y_test=b[<span style="color: #B452CD">19</span>:]
model=Pipeline([(<span style="color: #CD5555">&#39;poly&#39;</span>, PolynomialFeatures(degree=<span style="color: #B452CD">3</span>)),(<span style="color: #CD5555">&#39;linear&#39;</span>, LinearRegression(fit_intercept=<span style="color: #658b00">False</span>))])
model=model.fit(X_train, y_train)
pred=model.predict(X_test)
poly=PolynomialFeatures(degree=<span style="color: #B452CD">3</span>)
poly.fit_transform(X_train, y_train)
plt.scatter(X_test, y_test)
plt.plot(X_test, pred, color=<span style="color: #CD5555">&#39;green&#39;</span>)
plt.show()
<span style="color: #8B008B; font-weight: bold">print</span> (model.score(X_test,y_test))
<span style="color: #8B008B; font-weight: bold">print</span> (<span style="color: #CD5555">&quot;---------K-Nearest Neighbors-------&quot;</span>)
<span style="color: #CD5555">&quot;&quot;&quot;neighbors_settings=range(1,11)</span>
<span style="color: #CD5555">for n_neighbors in neighbors_settings:</span>
<span style="color: #CD5555"> clf=KNeighborsClassifier(n_neighbors=n_neighbors)</span>
<span style="color: #CD5555"> clf.fit(X_train, y_train)</span>
<span style="color: #CD5555"> training_accuracy.append(clf.score(X_train, y_train))</span>
<span style="color: #CD5555"> test_accuracy.append(clf.score(X_test, y_test))</span>
<span style="color: #CD5555">print (mglearn.plots.plot_knn_regression(n_neighbors=3))&quot;&quot;&quot;</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.neighbors</span> <span style="color: #8B008B; font-weight: bold">import</span> KNeighborsRegressor
X, y=mglearn.datasets.make_wave(n_samples=<span style="color: #B452CD">40</span>)
reg = KNeighborsRegressor(n_neighbors=<span style="color: #B452CD">3</span>)
reg.fit(X_train, y_train)
</pre></div>
<p>
A decision tree mainly contains of a <b>root node</b>, <b>interior nodes</b>,
and <b>leaf nodes</b> which are then connected by <b>branches</b>.
</section>
<section>
<h2 id="___sec2">Decision trees and Regression </h2>
<h2 id="___sec2">How do we set it up? </h2>
<p>
In simplified terms, the process of training a decision tree and
predicting the target features of query instances is as follows:
<ol>
<p><li> Present a dataset containing of a number of training instances characterized by a number of descriptive features and a target feature</li>
<p><li> Train the decision tree model by continuously splitting the target feature along the values of the descriptive features using a measure of information gain during the training process</li>
<p><li> Grow the tree until we accomplish a stopping criteria create leaf nodes which represent the <em>predictions</em> we want to make for new query instances</li>
<p><li> Show query instances to the tree and run down the tree until we arrive at leaf nodes</li>
</ol>
<p>
Then we are essentially done!
</section>
<section>
<h2 id="___sec3">Decision trees and Regression </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -318,6 +303,12 @@ plt.title(<span style="color: #CD5555">&quot;Decision Tree Regression&quot;</spa
plt.legend()
plt.show()
</pre></div>
</section>
<section>
<h2 id="___sec4">Maxwell-Boltzmann velocity distribution </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
+55 -63
View File
@@ -6,9 +6,9 @@ Automatically generated HTML file from DocOnce source
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees">
<meta name="description" content="Data Analysis and Machine Learning: Trees, forests and all that">
<title>Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees</title>
<title>Data Analysis and Machine Learning: Trees, forests and all that</title>
<link href="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
@@ -61,8 +61,10 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('Decision trees, overarching aims', 2, None, '___sec0'),
('Nearest Neighbors', 2, None, '___sec1'),
('Decision trees and Regression', 2, None, '___sec2')]}
('Nodes, leafs, roots and branches', 2, None, '___sec1'),
('How do we set it up?', 2, None, '___sec2'),
('Decision trees and Regression', 2, None, '___sec3'),
('Maxwell-Boltzmann velocity distribution', 2, None, '___sec4')]}
end of tocinfo -->
<body>
@@ -72,7 +74,7 @@ end of tocinfo -->
<center><h1>Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees</h1></center> <!-- document title -->
<center><h1>Data Analysis and Machine Learning: Trees, forests and all that</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -88,7 +90,7 @@ end of tocinfo -->
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 1, 2018</h4></center> <!-- date -->
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -112,72 +114,57 @@ continuous but also categorical input and target features.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<!-- !split -->
<h2 id="___sec1">Nodes, leafs, roots and branches </h2>
<h2 id="___sec1">Nearest Neighbors </h2>
<p>
The main idea of decision trees
is to find those descriptive features which contain the most
<b>information</b> regarding the target feature and then split the dataset
along the values of these features such that the target feature values
for the resulting sub\_datasets are as pure as possible.
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">mglearn</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn</span> <span style="color: #8B008B; font-weight: bold">import</span> linear_model
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> PolynomialFeatures
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.pipeline</span> <span style="color: #8B008B; font-weight: bold">import</span> Pipeline
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.neighbors</span> <span style="color: #8B008B; font-weight: bold">import</span> KNeighborsClassifier
<p>
The descriptive feature which leaves the target feature most purely is said
to be the most informative one. This process of finding the <b>most
informative</b> feature is done until we accomplish a stopping criteria
where we then finally end up in so called <b>leaf nodes</b>.
<span style="color: #228B22"># Generate sample data</span>
X = np.sort(<span style="color: #B452CD">5</span>*np.random.rand(<span style="color: #B452CD">40</span>,<span style="color: #B452CD">1</span>), axis=<span style="color: #B452CD">0</span>)
y = X**<span style="color: #B452CD">3</span>
y=y.ravel()
<p>
The leaf nodes
contain the predictions we will make for new query instances presented
to our trained model. This is possible since the model has kind of
learned the underlying structure of the training data and hence can,
given some assumptions, make predictions about the target feature value
(class) of unseen query instances.
<span style="color: #228B22"># Add noise to targets</span>
X[::<span style="color: #B452CD">4</span>] +=(<span style="color: #B452CD">0.5</span> - np.random.rand(<span style="color: #B452CD">1</span>))
y[::<span style="color: #B452CD">5</span>] +=(<span style="color: #B452CD">0.5</span> - np.random.rand(<span style="color: #B452CD">8</span>))
<p>
A decision tree mainly contains of a <b>root node</b>, <b>interior nodes</b>,
and <b>leaf nodes</b> which are then connected by <b>branches</b>.
a=np.array(X)
b=np.array(y)
X_train=a[:<span style="color: #B452CD">19</span>]
X_test=a[<span style="color: #B452CD">19</span>:]
y_train=b[:<span style="color: #B452CD">19</span>]
y_test=b[<span style="color: #B452CD">19</span>:]
model=Pipeline([(<span style="color: #CD5555">&#39;poly&#39;</span>, PolynomialFeatures(degree=<span style="color: #B452CD">3</span>)),(<span style="color: #CD5555">&#39;linear&#39;</span>, LinearRegression(fit_intercept=<span style="color: #658b00">False</span>))])
model=model.fit(X_train, y_train)
pred=model.predict(X_test)
poly=PolynomialFeatures(degree=<span style="color: #B452CD">3</span>)
poly.fit_transform(X_train, y_train)
plt.scatter(X_test, y_test)
plt.plot(X_test, pred, color=<span style="color: #CD5555">&#39;green&#39;</span>)
plt.show()
<span style="color: #8B008B; font-weight: bold">print</span> (model.score(X_test,y_test))
<span style="color: #8B008B; font-weight: bold">print</span> (<span style="color: #CD5555">&quot;---------K-Nearest Neighbors-------&quot;</span>)
<span style="color: #CD5555">&quot;&quot;&quot;neighbors_settings=range(1,11)</span>
<span style="color: #CD5555">for n_neighbors in neighbors_settings:</span>
<span style="color: #CD5555"> clf=KNeighborsClassifier(n_neighbors=n_neighbors)</span>
<span style="color: #CD5555"> clf.fit(X_train, y_train)</span>
<span style="color: #CD5555"> training_accuracy.append(clf.score(X_train, y_train))</span>
<span style="color: #CD5555"> test_accuracy.append(clf.score(X_test, y_test))</span>
<span style="color: #CD5555">print (mglearn.plots.plot_knn_regression(n_neighbors=3))&quot;&quot;&quot;</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.neighbors</span> <span style="color: #8B008B; font-weight: bold">import</span> KNeighborsRegressor
X, y=mglearn.datasets.make_wave(n_samples=<span style="color: #B452CD">40</span>)
reg = KNeighborsRegressor(n_neighbors=<span style="color: #B452CD">3</span>)
reg.fit(X_train, y_train)
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Decision trees and Regression </h2>
<h2 id="___sec2">How do we set it up? </h2>
<p>
In simplified terms, the process of training a decision tree and
predicting the target features of query instances is as follows:
<ol>
<li> Present a dataset containing of a number of training instances characterized by a number of descriptive features and a target feature</li>
<li> Train the decision tree model by continuously splitting the target feature along the values of the descriptive features using a measure of information gain during the training process</li>
<li> Grow the tree until we accomplish a stopping criteria create leaf nodes which represent the <em>predictions</em> we want to make for new query instances</li>
<li> Show query instances to the tree and run down the tree until we arrive at leaf nodes</li>
</ol>
Then we are essentially done!
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Decision trees and Regression </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -270,6 +257,11 @@ plt.title(<span style="color: #CD5555">&quot;Decision Tree Regression&quot;</spa
plt.legend()
plt.show()
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Maxwell-Boltzmann velocity distribution </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
doc/pub/DecisionTrees/html/DecisionTrees.html
+55 -63
View File
@@ -6,9 +6,9 @@ Automatically generated HTML file from DocOnce source
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees">
<meta name="description" content="Data Analysis and Machine Learning: Trees, forests and all that">
<title>Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees</title>
<title>Data Analysis and Machine Learning: Trees, forests and all that</title>
<style type="text/css">
@@ -66,8 +66,10 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('Decision trees, overarching aims', 2, None, '___sec0'),
('Nearest Neighbors', 2, None, '___sec1'),
('Decision trees and Regression', 2, None, '___sec2')]}
('Nodes, leafs, roots and branches', 2, None, '___sec1'),
('How do we set it up?', 2, None, '___sec2'),
('Decision trees and Regression', 2, None, '___sec3'),
('Maxwell-Boltzmann velocity distribution', 2, None, '___sec4')]}
end of tocinfo -->
<body>
@@ -77,7 +79,7 @@ end of tocinfo -->
<center><h1>Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees</h1></center> <!-- document title -->
<center><h1>Data Analysis and Machine Learning: Trees, forests and all that</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -93,7 +95,7 @@ end of tocinfo -->
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 1, 2018</h4></center> <!-- date -->
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -117,72 +119,57 @@ continuous but also categorical input and target features.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<!-- !split -->
<h2 id="___sec1">Nodes, leafs, roots and branches </h2>
<h2 id="___sec1">Nearest Neighbors </h2>
<p>
The main idea of decision trees
is to find those descriptive features which contain the most
<b>information</b> regarding the target feature and then split the dataset
along the values of these features such that the target feature values
for the resulting sub\_datasets are as pure as possible.
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">mglearn</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> linear_model
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LinearRegression
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> PolynomialFeatures
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.neighbors</span> <span style="color: #008000; font-weight: bold">import</span> KNeighborsClassifier
<p>
The descriptive feature which leaves the target feature most purely is said
to be the most informative one. This process of finding the <b>most
informative</b> feature is done until we accomplish a stopping criteria
where we then finally end up in so called <b>leaf nodes</b>.
<span style="color: #408080; font-style: italic"># Generate sample data</span>
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sort(<span style="color: #666666">5*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">40</span>,<span style="color: #666666">1</span>), axis<span style="color: #666666">=0</span>)
y <span style="color: #666666">=</span> X<span style="color: #666666">**3</span>
y<span style="color: #666666">=</span>y<span style="color: #666666">.</span>ravel()
<p>
The leaf nodes
contain the predictions we will make for new query instances presented
to our trained model. This is possible since the model has kind of
learned the underlying structure of the training data and hence can,
given some assumptions, make predictions about the target feature value
(class) of unseen query instances.
<span style="color: #408080; font-style: italic"># Add noise to targets</span>
X[::<span style="color: #666666">4</span>] <span style="color: #666666">+=</span>(<span style="color: #666666">0.5</span> <span style="color: #666666">-</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">1</span>))
y[::<span style="color: #666666">5</span>] <span style="color: #666666">+=</span>(<span style="color: #666666">0.5</span> <span style="color: #666666">-</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">8</span>))
<p>
A decision tree mainly contains of a <b>root node</b>, <b>interior nodes</b>,
and <b>leaf nodes</b> which are then connected by <b>branches</b>.
a<span style="color: #666666">=</span>np<span style="color: #666666">.</span>array(X)
b<span style="color: #666666">=</span>np<span style="color: #666666">.</span>array(y)
X_train<span style="color: #666666">=</span>a[:<span style="color: #666666">19</span>]
X_test<span style="color: #666666">=</span>a[<span style="color: #666666">19</span>:]
y_train<span style="color: #666666">=</span>b[:<span style="color: #666666">19</span>]
y_test<span style="color: #666666">=</span>b[<span style="color: #666666">19</span>:]
model<span style="color: #666666">=</span>Pipeline([(<span style="color: #BA2121">&#39;poly&#39;</span>, PolynomialFeatures(degree<span style="color: #666666">=3</span>)),(<span style="color: #BA2121">&#39;linear&#39;</span>, LinearRegression(fit_intercept<span style="color: #666666">=</span><span style="color: #008000">False</span>))])
model<span style="color: #666666">=</span>model<span style="color: #666666">.</span>fit(X_train, y_train)
pred<span style="color: #666666">=</span>model<span style="color: #666666">.</span>predict(X_test)
poly<span style="color: #666666">=</span>PolynomialFeatures(degree<span style="color: #666666">=3</span>)
poly<span style="color: #666666">.</span>fit_transform(X_train, y_train)
plt<span style="color: #666666">.</span>scatter(X_test, y_test)
plt<span style="color: #666666">.</span>plot(X_test, pred, color<span style="color: #666666">=</span><span style="color: #BA2121">&#39;green&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #008000; font-weight: bold">print</span> (model<span style="color: #666666">.</span>score(X_test,y_test))
<span style="color: #008000; font-weight: bold">print</span> (<span style="color: #BA2121">&quot;---------K-Nearest Neighbors-------&quot;</span>)
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;neighbors_settings=range(1,11)</span>
<span style="color: #BA2121; font-style: italic">for n_neighbors in neighbors_settings:</span>
<span style="color: #BA2121; font-style: italic"> clf=KNeighborsClassifier(n_neighbors=n_neighbors)</span>
<span style="color: #BA2121; font-style: italic"> clf.fit(X_train, y_train)</span>
<span style="color: #BA2121; font-style: italic"> training_accuracy.append(clf.score(X_train, y_train))</span>
<span style="color: #BA2121; font-style: italic"> test_accuracy.append(clf.score(X_test, y_test))</span>
<span style="color: #BA2121; font-style: italic">print (mglearn.plots.plot_knn_regression(n_neighbors=3))&quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.neighbors</span> <span style="color: #008000; font-weight: bold">import</span> KNeighborsRegressor
X, y<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>datasets<span style="color: #666666">.</span>make_wave(n_samples<span style="color: #666666">=40</span>)
reg <span style="color: #666666">=</span> KNeighborsRegressor(n_neighbors<span style="color: #666666">=3</span>)
reg<span style="color: #666666">.</span>fit(X_train, y_train)
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Decision trees and Regression </h2>
<h2 id="___sec2">How do we set it up? </h2>
<p>
In simplified terms, the process of training a decision tree and
predicting the target features of query instances is as follows:
<ol>
<li> Present a dataset containing of a number of training instances characterized by a number of descriptive features and a target feature</li>
<li> Train the decision tree model by continuously splitting the target feature along the values of the descriptive features using a measure of information gain during the training process</li>
<li> Grow the tree until we accomplish a stopping criteria create leaf nodes which represent the <em>predictions</em> we want to make for new query instances</li>
<li> Show query instances to the tree and run down the tree until we arrive at leaf nodes</li>
</ol>
Then we are essentially done!
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Decision trees and Regression </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -275,6 +262,11 @@ plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot
plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Maxwell-Boltzmann velocity distribution </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/DecisionTrees/pdf/DecisionTrees-beamer-handouts2x3.pdf
BIN
View File
Binary file not shown.
doc/pub/DecisionTrees/pdf/DecisionTrees-beamer.pdf
BIN
View File
Binary file not shown.
doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf
BIN
View File
Binary file not shown.
doc/src/DecisionTrees/DecisionTrees.do.txt
+48 -52
View File
@@ -1,4 +1,4 @@
TITLE: Data Analysis and Machine Learning: Nearest Neighbors and Decision Trees
TITLE: Data Analysis and Machine Learning: Trees, forests and all that
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
DATE: today
@@ -17,65 +17,53 @@ continuous but also categorical input and target features.
!eblock
!split
===== Nodes, leafs, roots and branches =====
The main idea of decision trees
is to find those descriptive features which contain the most
_information_ regarding the target feature and then split the dataset
along the values of these features such that the target feature values
for the resulting sub\_datasets are as pure as possible.
The descriptive feature which leaves the target feature most purely is said
to be the most informative one. This process of finding the _most
informative_ feature is done until we accomplish a stopping criteria
where we then finally end up in so called _leaf nodes_.
The leaf nodes
contain the predictions we will make for new query instances presented
to our trained model. This is possible since the model has kind of
learned the underlying structure of the training data and hence can,
given some assumptions, make predictions about the target feature value
(class) of unseen query instances.
A decision tree mainly contains of a _root node_, _interior nodes_,
and _leaf nodes_ which are then connected by _branches_.
!split
===== Nearest Neighbors =====
!bc pycod
import mglearn
import numpy as np
from sklearn import linear_model
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import Pipeline
from sklearn.neighbors import KNeighborsClassifier
# Generate sample data
X = np.sort(5*np.random.rand(40,1), axis=0)
y = X**3
y=y.ravel()
# Add noise to targets
X[::4] +=(0.5 - np.random.rand(1))
y[::5] +=(0.5 - np.random.rand(8))
a=np.array(X)
b=np.array(y)
X_train=a[:19]
X_test=a[19:]
y_train=b[:19]
y_test=b[19:]
model=Pipeline([('poly', PolynomialFeatures(degree=3)),('linear', LinearRegression(fit_intercept=False))])
model=model.fit(X_train, y_train)
pred=model.predict(X_test)
===== How do we set it up? =====
poly=PolynomialFeatures(degree=3)
poly.fit_transform(X_train, y_train)
plt.scatter(X_test, y_test)
plt.plot(X_test, pred, color='green')
plt.show()
In simplified terms, the process of training a decision tree and
predicting the target features of query instances is as follows:
print (model.score(X_test,y_test))
o Present a dataset containing of a number of training instances characterized by a number of descriptive features and a target feature
print ("---------K-Nearest Neighbors-------")
"""neighbors_settings=range(1,11)
for n_neighbors in neighbors_settings:
clf=KNeighborsClassifier(n_neighbors=n_neighbors)
clf.fit(X_train, y_train)
training_accuracy.append(clf.score(X_train, y_train))
test_accuracy.append(clf.score(X_test, y_test))
o Train the decision tree model by continuously splitting the target feature along the values of the descriptive features using a measure of information gain during the training process
o Grow the tree until we accomplish a stopping criteria create leaf nodes which represent the *predictions* we want to make for new query instances
o Show query instances to the tree and run down the tree until we arrive at leaf nodes
Then we are essentially done!
print (mglearn.plots.plot_knn_regression(n_neighbors=3))"""
from sklearn.neighbors import KNeighborsRegressor
X, y=mglearn.datasets.make_wave(n_samples=40)
reg = KNeighborsRegressor(n_neighbors=3)
reg.fit(X_train, y_train)
!ec
!split
===== Decision trees and Regression =====
@@ -171,6 +159,9 @@ plt.show()
!ec
!split
===== Maxwell-Boltzmann velocity distribution =====
!bc pycod
# Program to test the Metropolis algorithm with one particle at given temp in
# one dimension
@@ -269,3 +260,8 @@ plt.show()
#i.e. frequency of all velocities in range -5 to -4.9, -4.9 to -4.8, etc...
!ec