mhjensen
215 lines
11 KiB
HTML
215 lines
11 KiB
HTML
<!--
|
|
Automatically generated HTML file from DocOnce source
|
|
(https://github.com/hplgit/doconce/)
|
|
-->
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
|
<meta name="description" content="Data Analysis and Machine Learning: Support Vector Machines">
|
|
|
|
<title>Data Analysis and Machine Learning: Support Vector Machines</title>
|
|
|
|
|
|
<style type="text/css">
|
|
/* bloodish style */
|
|
|
|
body {
|
|
font-family: Helvetica, Verdana, Arial, Sans-serif;
|
|
color: #404040;
|
|
background: #ffffff;
|
|
}
|
|
h1 { font-size: 1.8em; color: #8A0808; }
|
|
h2 { font-size: 1.6em; color: #8A0808; }
|
|
h3 { font-size: 1.4em; color: #8A0808; }
|
|
h4 { color: #8A0808; }
|
|
a { color: #8A0808; text-decoration:none; }
|
|
tt { font-family: "Courier New", Courier; }
|
|
/* pre style removed because it will interfer with pygments */
|
|
p { text-indent: 0px; }
|
|
hr { border: 0; width: 80%; border-bottom: 1px solid #aaa}
|
|
p.caption { width: 80%; font-style: normal; text-align: left; }
|
|
hr.figure { border: 0; width: 80%; border-bottom: 1px solid #aaa}
|
|
|
|
div { text-align: justify; text-justify: inter-word; }
|
|
</style>
|
|
|
|
|
|
</head>
|
|
|
|
<!-- tocinfo
|
|
{'highest level': 2,
|
|
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
|
|
('SVMs, basics with scikit-learn', 2, None, '___sec1'),
|
|
('SVMs, adding polynomial features', 2, None, '___sec2'),
|
|
('SVMs, polynomials and kernels', 2, None, '___sec3'),
|
|
('SVMs, regression', 2, None, '___sec4')]}
|
|
end of tocinfo -->
|
|
|
|
<body>
|
|
|
|
|
|
<!-- ------------------- main content ---------------------- -->
|
|
|
|
|
|
|
|
<center><h1>Data Analysis and Machine Learning: Support Vector Machines</h1></center> <!-- document title -->
|
|
|
|
<p>
|
|
<!-- author(s): Morten Hjorth-Jensen -->
|
|
|
|
<center>
|
|
<b>Morten Hjorth-Jensen</b> [1, 2]
|
|
</center>
|
|
|
|
<p>
|
|
<!-- institution(s) -->
|
|
|
|
<center>[1] <b>Department of Physics, University of Oslo</b></center>
|
|
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
|
<br>
|
|
<p>
|
|
<center><h4>Nov 1, 2018</h4></center> <!-- date -->
|
|
<br>
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec0">Support Vector Machines, overarching aims </h2>
|
|
|
|
<p>
|
|
A Support Vector Machine (SVM) is a very powerful and versatile
|
|
Machine Learning model, capable of performing linear or nonlinear
|
|
classification, regression, and even outlier detection. It is one of
|
|
the most popular models in Machine Learning, and anyone interested in
|
|
Machine Learning should have it in their toolbox. SVMs are
|
|
particularly well suited for classification of complex but small-sized or
|
|
medium-sized datasets.
|
|
|
|
<p>
|
|
The basic mathematics relies on the definition of hyperplanes and the definition of a <b>margin</b> which separates
|
|
classes (in case of classification problems) of variables. It is also used for regression problems.
|
|
|
|
<p>
|
|
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
|
|
We distringuish also between linearn and non-linear approaches.
|
|
|
|
<p>
|
|
<b>These notes will be updated shortly with more material</b>
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec1">SVMs, basics with scikit-learn </h2>
|
|
|
|
<p>
|
|
The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM
|
|
model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect
|
|
Iris-Virginica flowers.
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVC
|
|
iris <span style="color: #666666">=</span> datasets<span style="color: #666666">.</span>load_iris()
|
|
X <span style="color: #666666">=</span> iris[<span style="color: #BA2121">"data"</span>][:, (<span style="color: #666666">2</span>, <span style="color: #666666">3</span>)] <span style="color: #408080; font-style: italic"># petal length, petal width</span>
|
|
y <span style="color: #666666">=</span> (iris[<span style="color: #BA2121">"target"</span>] <span style="color: #666666">==</span> <span style="color: #666666">2</span>)<span style="color: #666666">.</span>astype(np<span style="color: #666666">.</span>float64) <span style="color: #408080; font-style: italic"># Iris-Virginica</span>
|
|
svm_clf <span style="color: #666666">=</span> Pipeline((
|
|
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
|
(<span style="color: #BA2121">"linear_svc"</span>, LinearSVC(C<span style="color: #666666">=1</span>, loss<span style="color: #666666">=</span><span style="color: #BA2121">"hinge"</span>)),
|
|
))
|
|
svm_clf<span style="color: #666666">.</span>fit(X_scaled, y)
|
|
</pre></div>
|
|
<p>
|
|
Alternatively, you could use the SVC class, using <b>SVC(kernel="linear", C=1)</b>, but it is much slower,
|
|
especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier
|
|
class, with <b>SGDClassifier(loss="hinge", alpha=1/(m*C))</b>. This applies regular Stochastic
|
|
Gradient Descentto train a linear SVM classifier. It does not converge as fast as the
|
|
LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core
|
|
training), or to handle online classification tasks.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec2">SVMs, adding polynomial features </h2>
|
|
|
|
<p>
|
|
Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets
|
|
are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more
|
|
features, such as polynomial features. In some cases this can result in a linearly
|
|
separable dataset.
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> make_moons
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> PolynomialFeatures
|
|
polynomial_svm_clf <span style="color: #666666">=</span> Pipeline((
|
|
(<span style="color: #BA2121">"poly_features"</span>, PolynomialFeatures(degree<span style="color: #666666">=3</span>)),
|
|
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
|
(<span style="color: #BA2121">"svm_clf"</span>, LinearSVC(C<span style="color: #666666">=10</span>, loss<span style="color: #666666">=</span><span style="color: #BA2121">"hinge"</span>))
|
|
))
|
|
polynomial_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
|
</pre></div>
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec3">SVMs, polynomials and kernels </h2>
|
|
|
|
<p>
|
|
Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning
|
|
algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets,
|
|
and with a high polynomial degree it creates a huge number of features, making the model too slow.
|
|
Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the
|
|
kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many
|
|
polynomial features, even with very high-degree polynomials, without actually having to add them. So
|
|
there is no combinatorial explosion of the number of features since you don’t actually add any features.
|
|
This trick is implemented by the SVC class.
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
|
|
poly_kernel_svm_clf <span style="color: #666666">=</span> Pipeline((
|
|
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
|
(<span style="color: #BA2121">"svm_clf"</span>, SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">"poly"</span>, degree<span style="color: #666666">=3</span>, coef0<span style="color: #666666">=1</span>, C<span style="color: #666666">=5</span>))
|
|
))
|
|
poly_kernel_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
|
</pre></div>
|
|
<p>
|
|
This code trains an SVM classifier using a 3rd-degree polynomial kernel.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec4">SVMs, regression </h2>
|
|
|
|
<p>
|
|
You can use Scikit-Learn’s LinearSVR class to perform linear SVM Regression. The following code
|
|
shows how to use the regression option (more material to come)
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVR
|
|
svm_reg <span style="color: #666666">=</span> LinearSVR(epsilon<span style="color: #666666">=1.5</span>)
|
|
svm_reg<span style="color: #666666">.</span>fit(X, y)
|
|
</pre></div>
|
|
<p>
|
|
To tackle nonlinear regression tasks, you can use a kernelized SVM model
|
|
|
|
<!-- ------------------- end of main content --------------- -->
|
|
|
|
|
|
<center style="font-size:80%">
|
|
<!-- copyright --> © 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
|
|
</center>
|
|
|
|
|
|
</body>
|
|
</html>
|
|
|
|
|