mhjensen
247 lines
20 KiB
HTML
247 lines
20 KiB
HTML
<!--
|
|
Automatically generated HTML file from DocOnce source
|
|
(https://github.com/hplgit/doconce/)
|
|
-->
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
|
<meta name="description" content="Data Analysis and Machine Learning: Support Vector Machines">
|
|
|
|
<title>Data Analysis and Machine Learning: Support Vector Machines</title>
|
|
|
|
|
|
<style type="text/css">
|
|
/* bloodish style */
|
|
|
|
body {
|
|
font-family: Helvetica, Verdana, Arial, Sans-serif;
|
|
color: #404040;
|
|
background: #ffffff;
|
|
}
|
|
h1 { font-size: 1.8em; color: #8A0808; }
|
|
h2 { font-size: 1.6em; color: #8A0808; }
|
|
h3 { font-size: 1.4em; color: #8A0808; }
|
|
h4 { color: #8A0808; }
|
|
a { color: #8A0808; text-decoration:none; }
|
|
tt { font-family: "Courier New", Courier; }
|
|
/* pre style removed because it will interfer with pygments */
|
|
p { text-indent: 0px; }
|
|
hr { border: 0; width: 80%; border-bottom: 1px solid #aaa}
|
|
p.caption { width: 80%; font-style: normal; text-align: left; }
|
|
hr.figure { border: 0; width: 80%; border-bottom: 1px solid #aaa}
|
|
|
|
div { text-align: justify; text-justify: inter-word; }
|
|
</style>
|
|
|
|
|
|
</head>
|
|
|
|
<!-- tocinfo
|
|
{'highest level': 2,
|
|
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
|
|
('Strength and weakness', 2, None, '___sec1'),
|
|
('Examples with kernels', 2, None, '___sec2')]}
|
|
end of tocinfo -->
|
|
|
|
<body>
|
|
|
|
|
|
|
|
<script type="text/x-mathjax-config">
|
|
MathJax.Hub.Config({
|
|
TeX: {
|
|
equationNumbers: { autoNumber: "AMS" },
|
|
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
|
}
|
|
});
|
|
</script>
|
|
<script type="text/javascript" async
|
|
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
|
</script>
|
|
|
|
|
|
|
|
|
|
<!-- ------------------- main content ---------------------- -->
|
|
|
|
|
|
|
|
<center><h1>Data Analysis and Machine Learning: Support Vector Machines</h1></center> <!-- document title -->
|
|
|
|
<p>
|
|
<!-- author(s): Morten Hjorth-Jensen -->
|
|
|
|
<center>
|
|
<b>Morten Hjorth-Jensen</b> [1, 2]
|
|
</center>
|
|
|
|
<p>
|
|
<!-- institution(s) -->
|
|
|
|
<center>[1] <b>Department of Physics, University of Oslo</b></center>
|
|
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
|
<br>
|
|
<p>
|
|
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
|
|
<br>
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec0">Support Vector Machines, overarching aims </h2>
|
|
|
|
<p>
|
|
A Support Vector Machine (SVM) is a very powerful and versatile
|
|
Machine Learning model, capable of performing linear or nonlinear
|
|
classification, regression, and even outlier detection. It is one of
|
|
the most popular models in Machine Learning, and anyone interested in
|
|
Machine Learning should have it in their toolbox. SVMs are
|
|
particularly well suited for classification of complex but small-sized or
|
|
medium-sized datasets.
|
|
|
|
<p>
|
|
The basic mathematics relies on the definition of hyperplanes and the definition of a <b>margin</b> which separates
|
|
classes (in case of classification problems) of variables. It is also used for regression problems.
|
|
|
|
<p>
|
|
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
|
|
We distringuish also between linearn and non-linear approaches.
|
|
|
|
<p>
|
|
<b>These notes will be updated shortly with more material</b>
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec1">Strength and weakness </h2>
|
|
When we implement a linear support vector machine, the main parameter is the constant \( C \). Small values of \( C \) mean simple models.
|
|
These models are fast to train and also fast to predict and scale to very large data sets and work well with sparse data. Linear support vector machines make it easy to understand how a prediction is made, however it is often not easy to understand why coefficients are the way they are. These models work also well in higer dimensions.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec2">Examples with kernels </h2>
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">IPython.display</span> <span style="color: #008000; font-weight: bold">import</span> set_matplotlib_formats, display
|
|
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
|
|
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
|
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
|
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">mglearn</span>
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">cycler</span> <span style="color: #008000; font-weight: bold">import</span> cycler
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LogisticRegression
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVC
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> make_blobs
|
|
|
|
|
|
X, y <span style="color: #666666">=</span> make_blobs(centers<span style="color: #666666">=4</span>, random_state<span style="color: #666666">=8</span>)
|
|
y <span style="color: #666666">=</span> y <span style="color: #666666">%</span> <span style="color: #666666">2</span>
|
|
|
|
mglearn<span style="color: #666666">.</span>discrete_scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], y)
|
|
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">"Feature 0"</span>)
|
|
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">"Feature 1"</span>)
|
|
plt<span style="color: #666666">.</span>show()
|
|
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVC
|
|
linear_svm <span style="color: #666666">=</span> LinearSVC()<span style="color: #666666">.</span>fit(X, y)
|
|
|
|
mglearn<span style="color: #666666">.</span>plots<span style="color: #666666">.</span>plot_2d_separator(linear_svm, X)
|
|
mglearn<span style="color: #666666">.</span>discrete_scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], y)
|
|
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">"Feature 0"</span>)
|
|
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">"Feature 1"</span>)
|
|
</pre></div>
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># add the squared first feature</span>
|
|
X_new <span style="color: #666666">=</span> np<span style="color: #666666">.</span>hstack([X, X[:, <span style="color: #666666">1</span>:] <span style="color: #666666">**</span> <span style="color: #666666">2</span>])
|
|
|
|
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">mpl_toolkits.mplot3d</span> <span style="color: #008000; font-weight: bold">import</span> Axes3D, axes3d
|
|
figure <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
|
|
<span style="color: #408080; font-style: italic"># visualize in 3D</span>
|
|
ax <span style="color: #666666">=</span> Axes3D(figure, elev<span style="color: #666666">=-152</span>, azim<span style="color: #666666">=-26</span>)
|
|
<span style="color: #408080; font-style: italic"># plot first all the points with y==0, then all with y == 1</span>
|
|
mask <span style="color: #666666">=</span> y <span style="color: #666666">==</span> <span style="color: #666666">0</span>
|
|
ax<span style="color: #666666">.</span>scatter(X_new[mask, <span style="color: #666666">0</span>], X_new[mask, <span style="color: #666666">1</span>], X_new[mask, <span style="color: #666666">2</span>], c<span style="color: #666666">=</span><span style="color: #BA2121">'b'</span>,
|
|
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, s<span style="color: #666666">=60</span>, edgecolor<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
|
ax<span style="color: #666666">.</span>scatter(X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">0</span>], X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">1</span>], X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">2</span>], c<span style="color: #666666">=</span><span style="color: #BA2121">'r'</span>, marker<span style="color: #666666">=</span><span style="color: #BA2121">'^'</span>,
|
|
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, s<span style="color: #666666">=60</span>, edgecolor<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
|
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">"feature0"</span>)
|
|
ax<span style="color: #666666">.</span>set_ylabel(<span style="color: #BA2121">"feature1"</span>)
|
|
ax<span style="color: #666666">.</span>set_zlabel(<span style="color: #BA2121">"feature1 ** 2"</span>)
|
|
</pre></div>
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>linear_svm_3d <span style="color: #666666">=</span> LinearSVC()<span style="color: #666666">.</span>fit(X_new, y)
|
|
coef, intercept <span style="color: #666666">=</span> linear_svm_3d<span style="color: #666666">.</span>coef_<span style="color: #666666">.</span>ravel(), linear_svm_3d<span style="color: #666666">.</span>intercept_
|
|
|
|
<span style="color: #408080; font-style: italic"># show linear decision boundary</span>
|
|
figure <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
|
|
ax <span style="color: #666666">=</span> Axes3D(figure, elev<span style="color: #666666">=-152</span>, azim<span style="color: #666666">=-26</span>)
|
|
xx <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(X_new[:, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>min() <span style="color: #666666">-</span> <span style="color: #666666">2</span>, X_new[:, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>max() <span style="color: #666666">+</span> <span style="color: #666666">2</span>, <span style="color: #666666">50</span>)
|
|
yy <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(X_new[:, <span style="color: #666666">1</span>]<span style="color: #666666">.</span>min() <span style="color: #666666">-</span> <span style="color: #666666">2</span>, X_new[:, <span style="color: #666666">1</span>]<span style="color: #666666">.</span>max() <span style="color: #666666">+</span> <span style="color: #666666">2</span>, <span style="color: #666666">50</span>)
|
|
|
|
XX, YY <span style="color: #666666">=</span> np<span style="color: #666666">.</span>meshgrid(xx, yy)
|
|
ZZ <span style="color: #666666">=</span> (coef[<span style="color: #666666">0</span>] <span style="color: #666666">*</span> XX <span style="color: #666666">+</span> coef[<span style="color: #666666">1</span>] <span style="color: #666666">*</span> YY <span style="color: #666666">+</span> intercept) <span style="color: #666666">/</span> <span style="color: #666666">-</span>coef[<span style="color: #666666">2</span>]
|
|
ax<span style="color: #666666">.</span>plot_surface(XX, YY, ZZ, rstride<span style="color: #666666">=8</span>, cstride<span style="color: #666666">=8</span>, alpha<span style="color: #666666">=0.3</span>)
|
|
ax<span style="color: #666666">.</span>scatter(X_new[mask, <span style="color: #666666">0</span>], X_new[mask, <span style="color: #666666">1</span>], X_new[mask, <span style="color: #666666">2</span>], c<span style="color: #666666">=</span><span style="color: #BA2121">'b'</span>,
|
|
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, s<span style="color: #666666">=60</span>, edgecolor<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
|
ax<span style="color: #666666">.</span>scatter(X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">0</span>], X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">1</span>], X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">2</span>], c<span style="color: #666666">=</span><span style="color: #BA2121">'r'</span>, marker<span style="color: #666666">=</span><span style="color: #BA2121">'^'</span>,
|
|
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, s<span style="color: #666666">=60</span>, edgecolor<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
|
|
|
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">"feature0"</span>)
|
|
ax<span style="color: #666666">.</span>set_ylabel(<span style="color: #BA2121">"feature1"</span>)
|
|
ax<span style="color: #666666">.</span>set_zlabel(<span style="color: #BA2121">"feature1 ** 2"</span>)
|
|
|
|
ZZ <span style="color: #666666">=</span> YY <span style="color: #666666">**</span> <span style="color: #666666">2</span>
|
|
dec <span style="color: #666666">=</span> linear_svm_3d<span style="color: #666666">.</span>decision_function(np<span style="color: #666666">.</span>c_[XX<span style="color: #666666">.</span>ravel(), YY<span style="color: #666666">.</span>ravel(), ZZ<span style="color: #666666">.</span>ravel()])
|
|
plt<span style="color: #666666">.</span>contourf(XX, YY, dec<span style="color: #666666">.</span>reshape(XX<span style="color: #666666">.</span>shape), levels<span style="color: #666666">=</span>[dec<span style="color: #666666">.</span>min(), <span style="color: #666666">0</span>, dec<span style="color: #666666">.</span>max()],
|
|
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, alpha<span style="color: #666666">=0.5</span>)
|
|
mglearn<span style="color: #666666">.</span>discrete_scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], y)
|
|
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">"Feature 0"</span>)
|
|
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">"Feature 1"</span>)
|
|
</pre></div>
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
|
|
X, y <span style="color: #666666">=</span> mglearn<span style="color: #666666">.</span>tools<span style="color: #666666">.</span>make_handcrafted_dataset()
|
|
svm <span style="color: #666666">=</span> SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">'rbf'</span>, C<span style="color: #666666">=10</span>, gamma<span style="color: #666666">=0.1</span>)<span style="color: #666666">.</span>fit(X, y)
|
|
mglearn<span style="color: #666666">.</span>plots<span style="color: #666666">.</span>plot_2d_separator(svm, X, eps<span style="color: #666666">=.5</span>)
|
|
mglearn<span style="color: #666666">.</span>discrete_scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], y)
|
|
<span style="color: #408080; font-style: italic"># plot support vectors</span>
|
|
sv <span style="color: #666666">=</span> svm<span style="color: #666666">.</span>support_vectors_
|
|
<span style="color: #408080; font-style: italic"># class labels of support vectors are given by the sign of the dual coefficients</span>
|
|
sv_labels <span style="color: #666666">=</span> svm<span style="color: #666666">.</span>dual_coef_<span style="color: #666666">.</span>ravel() <span style="color: #666666">></span> <span style="color: #666666">0</span>
|
|
mglearn<span style="color: #666666">.</span>discrete_scatter(sv[:, <span style="color: #666666">0</span>], sv[:, <span style="color: #666666">1</span>], sv_labels, s<span style="color: #666666">=15</span>, markeredgewidth<span style="color: #666666">=3</span>)
|
|
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">"Feature 0"</span>)
|
|
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">"Feature 1"</span>)
|
|
|
|
fig, axes <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>subplots(<span style="color: #666666">3</span>, <span style="color: #666666">3</span>, figsize<span style="color: #666666">=</span>(<span style="color: #666666">15</span>, <span style="color: #666666">10</span>))
|
|
|
|
<span style="color: #008000; font-weight: bold">for</span> ax, C <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(axes, [<span style="color: #666666">-1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">3</span>]):
|
|
<span style="color: #008000; font-weight: bold">for</span> a, gamma <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(ax, <span style="color: #008000">range</span>(<span style="color: #666666">-1</span>, <span style="color: #666666">2</span>)):
|
|
mglearn<span style="color: #666666">.</span>plots<span style="color: #666666">.</span>plot_svm(log_C<span style="color: #666666">=</span>C, log_gamma<span style="color: #666666">=</span>gamma, ax<span style="color: #666666">=</span>a)
|
|
|
|
axes[<span style="color: #666666">0</span>, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>legend([<span style="color: #BA2121">"class 0"</span>, <span style="color: #BA2121">"class 1"</span>, <span style="color: #BA2121">"sv class 0"</span>, <span style="color: #BA2121">"sv class 1"</span>],
|
|
ncol<span style="color: #666666">=4</span>, loc<span style="color: #666666">=</span>(<span style="color: #666666">.9</span>, <span style="color: #666666">1.2</span>))
|
|
</pre></div>
|
|
<p>
|
|
|
|
<!-- ------------------- end of main content --------------- -->
|
|
|
|
|
|
<center style="font-size:80%">
|
|
<!-- copyright --> © 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
|
|
</center>
|
|
|
|
|
|
</body>
|
|
</html>
|
|
|
|
|