mhjensen
264 lines
13 KiB
HTML
264 lines
13 KiB
HTML
<!--
|
|
Automatically generated HTML file from DocOnce source
|
|
(https://github.com/hplgit/doconce/)
|
|
-->
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
|
<meta name="description" content="Data Analysis and Machine Learning: Support Vector Machines">
|
|
|
|
<title>Data Analysis and Machine Learning: Support Vector Machines</title>
|
|
|
|
<!-- Bootstrap style: bootstrap -->
|
|
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
|
<!-- not necessary
|
|
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
|
-->
|
|
|
|
<style type="text/css">
|
|
|
|
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
|
.dropdown-menu {
|
|
height: auto;
|
|
max-height: 400px;
|
|
overflow-x: hidden;
|
|
}
|
|
|
|
/* Adds an invisible element before each target to offset for the navigation
|
|
bar */
|
|
.anchor::before {
|
|
content:"";
|
|
display:block;
|
|
height:50px; /* fixed header height for style bootstrap */
|
|
margin:-50px 0 0; /* negative fixed header height */
|
|
}
|
|
</style>
|
|
|
|
|
|
</head>
|
|
|
|
<!-- tocinfo
|
|
{'highest level': 2,
|
|
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
|
|
('SVMs, basics with scikit-learn', 2, None, '___sec1'),
|
|
('SVMs, adding polynomial features', 2, None, '___sec2'),
|
|
('SVMs, polynomials and kernels', 2, None, '___sec3'),
|
|
('SVMs, regression', 2, None, '___sec4')]}
|
|
end of tocinfo -->
|
|
|
|
<body>
|
|
|
|
|
|
<!-- Bootstrap navigation bar -->
|
|
<div class="navbar navbar-default navbar-fixed-top">
|
|
<div class="navbar-header">
|
|
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
</button>
|
|
<a class="navbar-brand" href="svm-bs.html">Data Analysis and Machine Learning: Support Vector Machines</a>
|
|
</div>
|
|
|
|
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
|
<ul class="nav navbar-nav navbar-right">
|
|
<li class="dropdown">
|
|
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
|
<ul class="dropdown-menu">
|
|
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">Support Vector Machines, overarching aims</a></li>
|
|
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">SVMs, basics with scikit-learn</a></li>
|
|
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">SVMs, adding polynomial features</a></li>
|
|
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">SVMs, polynomials and kernels</a></li>
|
|
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">SVMs, regression</a></li>
|
|
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
</div> <!-- end of navigation bar -->
|
|
|
|
<div class="container">
|
|
|
|
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
|
|
|
<!-- ------------------- main content ---------------------- -->
|
|
|
|
|
|
|
|
<div class="jumbotron">
|
|
<center><h1>Data Analysis and Machine Learning: Support Vector Machines</h1></center> <!-- document title -->
|
|
|
|
<p>
|
|
<!-- author(s): Morten Hjorth-Jensen -->
|
|
|
|
<center>
|
|
<b>Morten Hjorth-Jensen</b> [1, 2]
|
|
</center>
|
|
|
|
<p>
|
|
<!-- institution(s) -->
|
|
|
|
<center>[1] <b>Department of Physics, University of Oslo</b></center>
|
|
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
|
<br>
|
|
<p>
|
|
<center><h4>Nov 1, 2018</h4></center> <!-- date -->
|
|
<br>
|
|
<p>
|
|
<!-- potential-jumbotron-button -->
|
|
</div> <!-- end jumbotron -->
|
|
|
|
<!-- !split -->
|
|
|
|
<h2 id="___sec0" class="anchor">Support Vector Machines, overarching aims </h2>
|
|
|
|
<p>
|
|
A Support Vector Machine (SVM) is a very powerful and versatile
|
|
Machine Learning model, capable of performing linear or nonlinear
|
|
classification, regression, and even outlier detection. It is one of
|
|
the most popular models in Machine Learning, and anyone interested in
|
|
Machine Learning should have it in their toolbox. SVMs are
|
|
particularly well suited for classification of complex but small-sized or
|
|
medium-sized datasets.
|
|
|
|
<p>
|
|
The basic mathematics relies on the definition of hyperplanes and the definition of a <b>margin</b> which separates
|
|
classes (in case of classification problems) of variables. It is also used for regression problems.
|
|
|
|
<p>
|
|
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
|
|
We distringuish also between linearn and non-linear approaches.
|
|
|
|
<p>
|
|
<b>These notes will be updated shortly with more material</b>
|
|
|
|
<p>
|
|
<!-- !split -->
|
|
|
|
<h2 id="___sec1" class="anchor">SVMs, basics with scikit-learn </h2>
|
|
|
|
<p>
|
|
The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM
|
|
model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect
|
|
Iris-Virginica flowers.
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVC
|
|
iris <span style="color: #666666">=</span> datasets<span style="color: #666666">.</span>load_iris()
|
|
X <span style="color: #666666">=</span> iris[<span style="color: #BA2121">"data"</span>][:, (<span style="color: #666666">2</span>, <span style="color: #666666">3</span>)] <span style="color: #408080; font-style: italic"># petal length, petal width</span>
|
|
y <span style="color: #666666">=</span> (iris[<span style="color: #BA2121">"target"</span>] <span style="color: #666666">==</span> <span style="color: #666666">2</span>)<span style="color: #666666">.</span>astype(np<span style="color: #666666">.</span>float64) <span style="color: #408080; font-style: italic"># Iris-Virginica</span>
|
|
svm_clf <span style="color: #666666">=</span> Pipeline((
|
|
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
|
(<span style="color: #BA2121">"linear_svc"</span>, LinearSVC(C<span style="color: #666666">=1</span>, loss<span style="color: #666666">=</span><span style="color: #BA2121">"hinge"</span>)),
|
|
))
|
|
svm_clf<span style="color: #666666">.</span>fit(X_scaled, y)
|
|
</pre></div>
|
|
<p>
|
|
Alternatively, you could use the SVC class, using <b>SVC(kernel="linear", C=1)</b>, but it is much slower,
|
|
especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier
|
|
class, with <b>SGDClassifier(loss="hinge", alpha=1/(m*C))</b>. This applies regular Stochastic
|
|
Gradient Descentto train a linear SVM classifier. It does not converge as fast as the
|
|
LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core
|
|
training), or to handle online classification tasks.
|
|
|
|
<p>
|
|
<!-- !split -->
|
|
|
|
<h2 id="___sec2" class="anchor">SVMs, adding polynomial features </h2>
|
|
|
|
<p>
|
|
Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets
|
|
are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more
|
|
features, such as polynomial features. In some cases this can result in a linearly
|
|
separable dataset.
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> make_moons
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> PolynomialFeatures
|
|
polynomial_svm_clf <span style="color: #666666">=</span> Pipeline((
|
|
(<span style="color: #BA2121">"poly_features"</span>, PolynomialFeatures(degree<span style="color: #666666">=3</span>)),
|
|
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
|
(<span style="color: #BA2121">"svm_clf"</span>, LinearSVC(C<span style="color: #666666">=10</span>, loss<span style="color: #666666">=</span><span style="color: #BA2121">"hinge"</span>))
|
|
))
|
|
polynomial_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
|
</pre></div>
|
|
<p>
|
|
<!-- !split -->
|
|
|
|
<h2 id="___sec3" class="anchor">SVMs, polynomials and kernels </h2>
|
|
|
|
<p>
|
|
Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning
|
|
algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets,
|
|
and with a high polynomial degree it creates a huge number of features, making the model too slow.
|
|
Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the
|
|
kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many
|
|
polynomial features, even with very high-degree polynomials, without actually having to add them. So
|
|
there is no combinatorial explosion of the number of features since you don’t actually add any features.
|
|
This trick is implemented by the SVC class.
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
|
|
poly_kernel_svm_clf <span style="color: #666666">=</span> Pipeline((
|
|
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
|
(<span style="color: #BA2121">"svm_clf"</span>, SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">"poly"</span>, degree<span style="color: #666666">=3</span>, coef0<span style="color: #666666">=1</span>, C<span style="color: #666666">=5</span>))
|
|
))
|
|
poly_kernel_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
|
</pre></div>
|
|
<p>
|
|
This code trains an SVM classifier using a 3rd-degree polynomial kernel.
|
|
|
|
<p>
|
|
<!-- !split -->
|
|
|
|
<h2 id="___sec4" class="anchor">SVMs, regression </h2>
|
|
|
|
<p>
|
|
You can use Scikit-Learn’s LinearSVR class to perform linear SVM Regression. The following code
|
|
shows how to use the regression option (more material to come)
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVR
|
|
svm_reg <span style="color: #666666">=</span> LinearSVR(epsilon<span style="color: #666666">=1.5</span>)
|
|
svm_reg<span style="color: #666666">.</span>fit(X, y)
|
|
</pre></div>
|
|
<p>
|
|
To tackle nonlinear regression tasks, you can use a kernelized SVM model
|
|
|
|
<!-- ------------------- end of main content --------------- -->
|
|
|
|
</div> <!-- end container -->
|
|
<!-- include javascript, jQuery *first* -->
|
|
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
|
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
|
|
|
<!-- Bootstrap footer
|
|
<footer>
|
|
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
|
</footer>
|
|
-->
|
|
|
|
|
|
<center style="font-size:80%">
|
|
<!-- copyright --> © 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
|
|
</center>
|
|
|
|
|
|
</body>
|
|
</html>
|
|
|
|
|