lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7940c2bcc4ab89d9123cbf2cafbd0b3fd37fdebf
FYS-STK4155/doc/pub/svm/html/svm-bs.html
T
mhjensen 03bc12598b sum update
2018-11-02 04:49:53 +01:00

294 lines
21 KiB
HTML
Raw Blame History

<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Support Vector Machines">
<title>Data Analysis and Machine Learning: Support Vector Machines</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
('Strength and weakness', 2, None, '___sec1'),
('Examples with kernels', 2, None, '___sec2')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "AMS" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="svm-bs.html">Data Analysis and Machine Learning: Support Vector Machines</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">Support Vector Machines, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Strength and weakness</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Examples with kernels</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<!-- ------------------- main content ---------------------- -->
<div class="jumbotron">
<center><h1>Data Analysis and Machine Learning: Support Vector Machines</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
<center>
<b>Morten Hjorth-Jensen</b> [1, 2]
</center>
<p>
<!-- institution(s) -->
<center>[1] <b>Department of Physics, University of Oslo</b></center>
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- potential-jumbotron-button -->
</div> <!-- end jumbotron -->
<!-- !split -->
<h2 id="___sec0" class="anchor">Support Vector Machines, overarching aims </h2>
<p>
A Support Vector Machine (SVM) is a very powerful and versatile
Machine Learning model, capable of performing linear or nonlinear
classification, regression, and even outlier detection. It is one of
the most popular models in Machine Learning, and anyone interested in
Machine Learning should have it in their toolbox. SVMs are
particularly well suited for classification of complex but small-sized or
medium-sized datasets.
<p>
The basic mathematics relies on the definition of hyperplanes and the definition of a <b>margin</b> which separates
classes (in case of classification problems) of variables. It is also used for regression problems.
<p>
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
We distringuish also between linearn and non-linear approaches.
<p>
<b>These notes will be updated shortly with more material</b>
<p>
<!-- !split -->
<h2 id="___sec1" class="anchor">Strength and weakness </h2>
When we implement a linear support vector machine, the main parameter is the constant \( C \). Small values of \( C \) mean simple models.
These models are fast to train and also fast to predict and scale to very large data sets and work well with sparse data. Linear support vector machines make it easy to understand how a prediction is made, however it is often not easy to understand why coefficients are the way they are. These models work also well in higer dimensions.
<p>
<!-- !split -->
<h2 id="___sec2" class="anchor">Examples with kernels </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">IPython.display</span> <span style="color: #008000; font-weight: bold">import</span> set_matplotlib_formats, display
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">mglearn</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">cycler</span> <span style="color: #008000; font-weight: bold">import</span> cycler
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LogisticRegression
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVC
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> make_blobs
X, y <span style="color: #666666">=</span> make_blobs(centers<span style="color: #666666">=4</span>, random_state<span style="color: #666666">=8</span>)
y <span style="color: #666666">=</span> y <span style="color: #666666">%</span> <span style="color: #666666">2</span>
mglearn<span style="color: #666666">.</span>discrete_scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], y)
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">&quot;Feature 0&quot;</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&quot;Feature 1&quot;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVC
linear_svm <span style="color: #666666">=</span> LinearSVC()<span style="color: #666666">.</span>fit(X, y)
mglearn<span style="color: #666666">.</span>plots<span style="color: #666666">.</span>plot_2d_separator(linear_svm, X)
mglearn<span style="color: #666666">.</span>discrete_scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], y)
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">&quot;Feature 0&quot;</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&quot;Feature 1&quot;</span>)
</pre></div>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># add the squared first feature</span>
X_new <span style="color: #666666">=</span> np<span style="color: #666666">.</span>hstack([X, X[:, <span style="color: #666666">1</span>:] <span style="color: #666666">**</span> <span style="color: #666666">2</span>])
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">mpl_toolkits.mplot3d</span> <span style="color: #008000; font-weight: bold">import</span> Axes3D, axes3d
figure <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
<span style="color: #408080; font-style: italic"># visualize in 3D</span>
ax <span style="color: #666666">=</span> Axes3D(figure, elev<span style="color: #666666">=-152</span>, azim<span style="color: #666666">=-26</span>)
<span style="color: #408080; font-style: italic"># plot first all the points with y==0, then all with y == 1</span>
mask <span style="color: #666666">=</span> y <span style="color: #666666">==</span> <span style="color: #666666">0</span>
ax<span style="color: #666666">.</span>scatter(X_new[mask, <span style="color: #666666">0</span>], X_new[mask, <span style="color: #666666">1</span>], X_new[mask, <span style="color: #666666">2</span>], c<span style="color: #666666">=</span><span style="color: #BA2121">&#39;b&#39;</span>,
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, s<span style="color: #666666">=60</span>, edgecolor<span style="color: #666666">=</span><span style="color: #BA2121">&#39;k&#39;</span>)
ax<span style="color: #666666">.</span>scatter(X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">0</span>], X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">1</span>], X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">2</span>], c<span style="color: #666666">=</span><span style="color: #BA2121">&#39;r&#39;</span>, marker<span style="color: #666666">=</span><span style="color: #BA2121">&#39;^&#39;</span>,
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, s<span style="color: #666666">=60</span>, edgecolor<span style="color: #666666">=</span><span style="color: #BA2121">&#39;k&#39;</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&quot;feature0&quot;</span>)
ax<span style="color: #666666">.</span>set_ylabel(<span style="color: #BA2121">&quot;feature1&quot;</span>)
ax<span style="color: #666666">.</span>set_zlabel(<span style="color: #BA2121">&quot;feature1 ** 2&quot;</span>)
</pre></div>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>linear_svm_3d <span style="color: #666666">=</span> LinearSVC()<span style="color: #666666">.</span>fit(X_new, y)
coef, intercept <span style="color: #666666">=</span> linear_svm_3d<span style="color: #666666">.</span>coef_<span style="color: #666666">.</span>ravel(), linear_svm_3d<span style="color: #666666">.</span>intercept_
<span style="color: #408080; font-style: italic"># show linear decision boundary</span>
figure <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> Axes3D(figure, elev<span style="color: #666666">=-152</span>, azim<span style="color: #666666">=-26</span>)
xx <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(X_new[:, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>min() <span style="color: #666666">-</span> <span style="color: #666666">2</span>, X_new[:, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>max() <span style="color: #666666">+</span> <span style="color: #666666">2</span>, <span style="color: #666666">50</span>)
yy <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(X_new[:, <span style="color: #666666">1</span>]<span style="color: #666666">.</span>min() <span style="color: #666666">-</span> <span style="color: #666666">2</span>, X_new[:, <span style="color: #666666">1</span>]<span style="color: #666666">.</span>max() <span style="color: #666666">+</span> <span style="color: #666666">2</span>, <span style="color: #666666">50</span>)
XX, YY <span style="color: #666666">=</span> np<span style="color: #666666">.</span>meshgrid(xx, yy)
ZZ <span style="color: #666666">=</span> (coef[<span style="color: #666666">0</span>] <span style="color: #666666">*</span> XX <span style="color: #666666">+</span> coef[<span style="color: #666666">1</span>] <span style="color: #666666">*</span> YY <span style="color: #666666">+</span> intercept) <span style="color: #666666">/</span> <span style="color: #666666">-</span>coef[<span style="color: #666666">2</span>]
ax<span style="color: #666666">.</span>plot_surface(XX, YY, ZZ, rstride<span style="color: #666666">=8</span>, cstride<span style="color: #666666">=8</span>, alpha<span style="color: #666666">=0.3</span>)
ax<span style="color: #666666">.</span>scatter(X_new[mask, <span style="color: #666666">0</span>], X_new[mask, <span style="color: #666666">1</span>], X_new[mask, <span style="color: #666666">2</span>], c<span style="color: #666666">=</span><span style="color: #BA2121">&#39;b&#39;</span>,
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, s<span style="color: #666666">=60</span>, edgecolor<span style="color: #666666">=</span><span style="color: #BA2121">&#39;k&#39;</span>)
ax<span style="color: #666666">.</span>scatter(X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">0</span>], X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">1</span>], X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">2</span>], c<span style="color: #666666">=</span><span style="color: #BA2121">&#39;r&#39;</span>, marker<span style="color: #666666">=</span><span style="color: #BA2121">&#39;^&#39;</span>,
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, s<span style="color: #666666">=60</span>, edgecolor<span style="color: #666666">=</span><span style="color: #BA2121">&#39;k&#39;</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&quot;feature0&quot;</span>)
ax<span style="color: #666666">.</span>set_ylabel(<span style="color: #BA2121">&quot;feature1&quot;</span>)
ax<span style="color: #666666">.</span>set_zlabel(<span style="color: #BA2121">&quot;feature1 ** 2&quot;</span>)
ZZ <span style="color: #666666">=</span> YY <span style="color: #666666">**</span> <span style="color: #666666">2</span>
dec <span style="color: #666666">=</span> linear_svm_3d<span style="color: #666666">.</span>decision_function(np<span style="color: #666666">.</span>c_[XX<span style="color: #666666">.</span>ravel(), YY<span style="color: #666666">.</span>ravel(), ZZ<span style="color: #666666">.</span>ravel()])
plt<span style="color: #666666">.</span>contourf(XX, YY, dec<span style="color: #666666">.</span>reshape(XX<span style="color: #666666">.</span>shape), levels<span style="color: #666666">=</span>[dec<span style="color: #666666">.</span>min(), <span style="color: #666666">0</span>, dec<span style="color: #666666">.</span>max()],
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, alpha<span style="color: #666666">=0.5</span>)
mglearn<span style="color: #666666">.</span>discrete_scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], y)
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">&quot;Feature 0&quot;</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&quot;Feature 1&quot;</span>)
</pre></div>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
X, y <span style="color: #666666">=</span> mglearn<span style="color: #666666">.</span>tools<span style="color: #666666">.</span>make_handcrafted_dataset()
svm <span style="color: #666666">=</span> SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">&#39;rbf&#39;</span>, C<span style="color: #666666">=10</span>, gamma<span style="color: #666666">=0.1</span>)<span style="color: #666666">.</span>fit(X, y)
mglearn<span style="color: #666666">.</span>plots<span style="color: #666666">.</span>plot_2d_separator(svm, X, eps<span style="color: #666666">=.5</span>)
mglearn<span style="color: #666666">.</span>discrete_scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], y)
<span style="color: #408080; font-style: italic"># plot support vectors</span>
sv <span style="color: #666666">=</span> svm<span style="color: #666666">.</span>support_vectors_
<span style="color: #408080; font-style: italic"># class labels of support vectors are given by the sign of the dual coefficients</span>
sv_labels <span style="color: #666666">=</span> svm<span style="color: #666666">.</span>dual_coef_<span style="color: #666666">.</span>ravel() <span style="color: #666666">&gt;</span> <span style="color: #666666">0</span>
mglearn<span style="color: #666666">.</span>discrete_scatter(sv[:, <span style="color: #666666">0</span>], sv[:, <span style="color: #666666">1</span>], sv_labels, s<span style="color: #666666">=15</span>, markeredgewidth<span style="color: #666666">=3</span>)
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">&quot;Feature 0&quot;</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&quot;Feature 1&quot;</span>)
fig, axes <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>subplots(<span style="color: #666666">3</span>, <span style="color: #666666">3</span>, figsize<span style="color: #666666">=</span>(<span style="color: #666666">15</span>, <span style="color: #666666">10</span>))
<span style="color: #008000; font-weight: bold">for</span> ax, C <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(axes, [<span style="color: #666666">-1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">3</span>]):
<span style="color: #008000; font-weight: bold">for</span> a, gamma <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(ax, <span style="color: #008000">range</span>(<span style="color: #666666">-1</span>, <span style="color: #666666">2</span>)):
mglearn<span style="color: #666666">.</span>plots<span style="color: #666666">.</span>plot_svm(log_C<span style="color: #666666">=</span>C, log_gamma<span style="color: #666666">=</span>gamma, ax<span style="color: #666666">=</span>a)
axes[<span style="color: #666666">0</span>, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>legend([<span style="color: #BA2121">&quot;class 0&quot;</span>, <span style="color: #BA2121">&quot;class 1&quot;</span>, <span style="color: #BA2121">&quot;sv class 0&quot;</span>, <span style="color: #BA2121">&quot;sv class 1&quot;</span>],
ncol<span style="color: #666666">=4</span>, loc<span style="color: #666666">=</span>(<span style="color: #666666">.9</span>, <span style="color: #666666">1.2</span>))
</pre></div>
<p>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink