mhjensen
642 lines
33 KiB
HTML
642 lines
33 KiB
HTML
<!--
|
|
Automatically generated HTML file from DocOnce source
|
|
(https://github.com/hplgit/doconce/)
|
|
-->
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
|
<meta name="description" content="Data Analysis and Machine Learning: Support Vector Machines">
|
|
|
|
<title>Data Analysis and Machine Learning: Support Vector Machines</title>
|
|
|
|
|
|
<style type="text/css">
|
|
/* bloodish style */
|
|
|
|
body {
|
|
font-family: Helvetica, Verdana, Arial, Sans-serif;
|
|
color: #404040;
|
|
background: #ffffff;
|
|
}
|
|
h1 { font-size: 1.8em; color: #8A0808; }
|
|
h2 { font-size: 1.6em; color: #8A0808; }
|
|
h3 { font-size: 1.4em; color: #8A0808; }
|
|
h4 { color: #8A0808; }
|
|
a { color: #8A0808; text-decoration:none; }
|
|
tt { font-family: "Courier New", Courier; }
|
|
/* pre style removed because it will interfer with pygments */
|
|
p { text-indent: 0px; }
|
|
hr { border: 0; width: 80%; border-bottom: 1px solid #aaa}
|
|
p.caption { width: 80%; font-style: normal; text-align: left; }
|
|
hr.figure { border: 0; width: 80%; border-bottom: 1px solid #aaa}
|
|
|
|
div { text-align: justify; text-justify: inter-word; }
|
|
</style>
|
|
|
|
|
|
</head>
|
|
|
|
<!-- tocinfo
|
|
{'highest level': 2,
|
|
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
|
|
('Hyperplanes and all that', 2, None, '___sec1'),
|
|
('What is a hyperplane?', 2, None, '___sec2'),
|
|
('A $p$-dimensional space of features', 2, None, '___sec3'),
|
|
('The two-dimensional case', 2, None, '___sec4'),
|
|
('Getting into the details', 2, None, '___sec5'),
|
|
('First attempt at a minimization approach', 2, None, '___sec6'),
|
|
('Solving the equations', 2, None, '___sec7'),
|
|
('A better approach', 2, None, '___sec8'),
|
|
('A quick reminder on Lagrangian multipliers',
|
|
2,
|
|
None,
|
|
'___sec9'),
|
|
('Adding the muliplier', 2, None, '___sec10'),
|
|
('Setting up the problem', 2, None, '___sec11'),
|
|
('The problem to solve', 2, None, '___sec12'),
|
|
('Examples with kernels', 2, None, '___sec13')]}
|
|
end of tocinfo -->
|
|
|
|
<body>
|
|
|
|
|
|
|
|
<script type="text/x-mathjax-config">
|
|
MathJax.Hub.Config({
|
|
TeX: {
|
|
equationNumbers: { autoNumber: "AMS" },
|
|
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
|
}
|
|
});
|
|
</script>
|
|
<script type="text/javascript" async
|
|
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
|
</script>
|
|
|
|
|
|
|
|
|
|
<!-- ------------------- main content ---------------------- -->
|
|
|
|
|
|
|
|
<center><h1>Data Analysis and Machine Learning: Support Vector Machines</h1></center> <!-- document title -->
|
|
|
|
<p>
|
|
<!-- author(s): Morten Hjorth-Jensen -->
|
|
|
|
<center>
|
|
<b>Morten Hjorth-Jensen</b> [1, 2]
|
|
</center>
|
|
|
|
<p>
|
|
<!-- institution(s) -->
|
|
|
|
<center>[1] <b>Department of Physics, University of Oslo</b></center>
|
|
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
|
<br>
|
|
<p>
|
|
<center><h4>Nov 5, 2018</h4></center> <!-- date -->
|
|
<br>
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec0">Support Vector Machines, overarching aims </h2>
|
|
|
|
<p>
|
|
A Support Vector Machine (SVM) is a very powerful and versatile
|
|
Machine Learning model, capable of performing linear or nonlinear
|
|
classification, regression, and even outlier detection. It is one of
|
|
the most popular models in Machine Learning, and anyone interested in
|
|
Machine Learning should have it in their toolbox. SVMs are
|
|
particularly well suited for classification of complex but small-sized or
|
|
medium-sized datasets.
|
|
|
|
<p>
|
|
The case with two well-separated classes only can be understood in an intuitive way in terms of lines in a two-dimensional space separating the two classes (see figure below).
|
|
|
|
<p>
|
|
The basic mathematics behind the SVM is however less familiar to most of us.
|
|
It relies on the definition of hyperplanes and the
|
|
definition of a <b>margin</b> which separates classes (in case of
|
|
classification problems) of variables. It is also used for regression
|
|
problems.
|
|
|
|
<p>
|
|
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
|
|
We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec1">Hyperplanes and all that </h2>
|
|
|
|
<p>
|
|
The theory behind support vector machines (SVM hereafter) is based on
|
|
the mathematical description of so-called hyperplanes. Let us start
|
|
with a two-dimensional case. This will also allow us to introduce our
|
|
first SVM examples. These will be tailored to the case of two specific
|
|
classes, as displayed in the figure here.
|
|
|
|
<p>
|
|
We assume here that our data set can be well separated into two
|
|
domains, where a straight line does the job in the separating the two
|
|
classes. Here the two classes are represented by either crosses or
|
|
circles.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec2">What is a hyperplane? </h2>
|
|
|
|
<p>
|
|
The aim of the SVM algorithm is to find a hyperplane in an \( p \)-dimensional space, where \( p \) is the number of features that distinctly classifies the data points.
|
|
|
|
<p>
|
|
In a \( p \)-dimensional space, a hyperplane is what we call an affine subspace of dimension of \( p-1 \).
|
|
As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is
|
|
a two-dimensional subspace, or stated simply, a plane.
|
|
|
|
<p>
|
|
In two dimensions, with the variables \( x_1 \) and \( x_2 \), the hyperplane is defined as
|
|
$$
|
|
b+w_1x_1+w_2x_2=0,
|
|
$$
|
|
|
|
where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line
|
|
\( b+w_1x_1+w_2x_2=0 \).
|
|
In two dimensions we define the vectors \( \hat{x} =[x1,x2] \) and \( \hat{w}=[w1,w2] \).
|
|
We can then rewrite the above equation as
|
|
$$
|
|
\hat{w}^T\hat{x}+b=0.
|
|
$$
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec3">A \( p \)-dimensional space of features </h2>
|
|
|
|
<p>
|
|
We limit ourselves to two classes of outputs \( y_i \) and assign these classes the values \( y_i = \pm 1 \).
|
|
In a \( p \)-dimensional space of say \( p \) features we have a hyperplane defines as
|
|
$$
|
|
b+wx_1+w_2x_2+\dots +w_px_p=0.
|
|
$$
|
|
|
|
If we define a
|
|
matrix \( \hat{X}=\left[\hat{x}_1,\hat{x}_2,\dots, \hat{x}_p\right] \)
|
|
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \hat{X} \),
|
|
$$
|
|
\hat{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
|
|
$$
|
|
|
|
If the above condition is not met for a given vector \( \hat{x}_i \) we have
|
|
$$
|
|
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
|
|
$$
|
|
|
|
if our output \( y_i=1 \).
|
|
In this case we say that \( \hat{x}_i \) lies on one of the sides of the hyperplane and if
|
|
$$
|
|
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
|
|
$$
|
|
|
|
for the class of observations \( y_i=-1 \),
|
|
then \( \hat{x}_i \) lies on the other side.
|
|
|
|
<p>
|
|
Equivalently, for the two classes of observations we have
|
|
$$
|
|
y_i\left(b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip}\right) > 0.
|
|
$$
|
|
|
|
<p>
|
|
When we try to separate hyperplanes, if it exists, we can use it to construct a natural classifier: a test observation is assigned a given class depending on which side of the hyperplane it is located.
|
|
|
|
<p>
|
|
<!-- !split -->
|
|
|
|
<h2 id="___sec4">The two-dimensional case </h2>
|
|
|
|
<p>
|
|
Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional
|
|
plane. To separate the two classes of data points, there are many
|
|
possible lines (hyperplanes if you prefer a more strict naming)
|
|
that could be chosen. Our objective is to find a
|
|
plane that has the maximum margin, i.e the maximum distance between
|
|
data points of both classes. Maximizing the margin distance provides
|
|
some reinforcement so that future data points can be classified with
|
|
more confidence.
|
|
|
|
<p>
|
|
What a linear classifier attempts to accomplish is to split the
|
|
feature space into two half spaces by placing a hyperplane between the
|
|
data points. This hyperplane will be our decision boundary. All
|
|
points on one side of the plane will belong to class one and all points
|
|
on the other side of the plane will belong to the second class two.
|
|
|
|
<p>
|
|
Unfortunately there are many ways in which we can place a hyperplane
|
|
to divide the data. Below is an example of two candidate hyperplanes
|
|
for our data sample.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec5">Getting into the details </h2>
|
|
|
|
<p>
|
|
Let us define the function
|
|
$$
|
|
f(x) = \hat{w}^T\hat{x}+b = 0,
|
|
$$
|
|
|
|
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
|
|
|
|
<p>
|
|
Any point defined by \( \hat{x}_i \) and \( \hat{x}_2 \) on the line \( L \) will satisfy \( \hat{w}^T(\hat{x}_1-\hat{x}_2)=0 \).
|
|
|
|
<p>
|
|
The signed distance \( \delta \) from any point defined by a vector \( \hat{x} \) and a point \( \hat{x}_0 \) on the line \( L \) is then
|
|
$$
|
|
\delta = \frac{1}{\vert\vert \hat{w}\vert\vert}(\hat{w}^T\hat{x}+b).
|
|
$$
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec6">First attempt at a minimization approach </h2>
|
|
|
|
<p>
|
|
How do we find the parameter \( b \) and the vector \( \hat{w} \)? What we could
|
|
do is to define a cost function which now contains the set of all
|
|
misclassified points \( M \) and attempt to minimize this function
|
|
|
|
$$
|
|
C(\hat{w},b) = -\sum_{i\in M} y_i(\hat{w}^T\hat{x}_i+b).
|
|
$$
|
|
|
|
<p>
|
|
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \hat{w}^T\hat{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
|
|
$$
|
|
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
|
|
$$
|
|
|
|
and
|
|
$$
|
|
\frac{\partial C}{\partial \hat{w}} = -\sum_{i\in M} y_ix_i.
|
|
$$
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec7">Solving the equations </h2>
|
|
|
|
<p>
|
|
We can now use the Newton-Raphson method or gradient descent to solve the equations
|
|
$$
|
|
b \leftarrow b +\eta \frac{\partial C}{\partial b},
|
|
$$
|
|
|
|
and
|
|
$$
|
|
\hat{w} \leftarrow \hat{w} +\eta \frac{\partial C}{\partial \hat{w}},
|
|
$$
|
|
|
|
where \( \eta \) is our by now well-known learning rate.
|
|
|
|
<p>
|
|
There are however problems with this approach, although it looks
|
|
pretty straightforward to implement. In case we separate our data into
|
|
two distinct classes, we may up with many possible lines, as indicated
|
|
in the figure and shown by running the following program. For small
|
|
gaps between the entries, we may also end up needing many iterations
|
|
before the solutions converge and if the data cannot be separated
|
|
properly into two distinct classes, we may not experience a converge
|
|
at all.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec8">A better approach </h2>
|
|
|
|
<p>
|
|
A better approach is rather to try to define a large margin between
|
|
the two classes (if they are well separated from the beginning).
|
|
|
|
<p>
|
|
Thus, we wish to find a margin \( M \) with \( \hat{w} \) normalized to
|
|
\( \vert\vert \hat{w}\vert\vert =1 \) subject to the condition
|
|
|
|
$$
|
|
y_i(\hat{w}^T\hat{x}_i+b) \geq M \forall i=1,2,\dots, p.
|
|
$$
|
|
|
|
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.
|
|
|
|
<p>
|
|
We seek thus the largest value \( M \) defined by
|
|
$$
|
|
\frac{1}{\vert \vert \hat{w}\vert\vert}y_i(\hat{w}^T\hat{x}_i+b) \geq M \forall i=1,2,\dots, n,
|
|
$$
|
|
|
|
or just
|
|
$$
|
|
y_i(\hat{w}^T\hat{x}_i+b) \geq M\vert \vert \hat{w}\vert\vert \forall i.
|
|
$$
|
|
|
|
If we scale the equation so that \( \vert \vert \hat{w}\vert\vert = 1/M \), we have to find the minimum of
|
|
\( \hat{w}^T\hat{w}=\vert \vert \hat{w}\vert\vert \) (the norm) subject to the condition
|
|
$$
|
|
y_i(\hat{w}^T\hat{x}_i+b) \geq 1 \forall i.
|
|
$$
|
|
|
|
<p>
|
|
We have thus defined our margin as the invers of the norm of \( \hat{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec9">A quick reminder on Lagrangian multipliers </h2>
|
|
|
|
<p>
|
|
Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an
|
|
extreme we have
|
|
$$
|
|
df=0.
|
|
$$
|
|
|
|
A necessary and sufficient condition is
|
|
$$
|
|
\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0,
|
|
$$
|
|
|
|
due to
|
|
$$
|
|
df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz.
|
|
$$
|
|
|
|
In many problems the variables \( x,y,z \) are often subject to constraints (such as those above for the margin)
|
|
so that they are no longer all independent. It is possible at least in principle to use each
|
|
constraint to eliminate one variable
|
|
and to proceed with a new and smaller set of independent varables.
|
|
|
|
<p>
|
|
The use of so-called Lagrangian multipliers is an alternative technique when the elimination
|
|
of variables is incovenient or undesirable. Assume that we have an equation of constraint on
|
|
the variables \( x,y,z \)
|
|
$$
|
|
\phi(x,y,z) = 0,
|
|
$$
|
|
|
|
resulting in
|
|
$$
|
|
d\phi = \frac{\partial \phi}{\partial x}dx+\frac{\partial \phi}{\partial y}dy+\frac{\partial \phi}{\partial z}dz =0.
|
|
$$
|
|
|
|
Now we cannot set anymore
|
|
$$
|
|
\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0,
|
|
$$
|
|
|
|
if \( df=0 \) is wanted
|
|
because there are now only two independent variables! Assume \( x \) and \( y \) are the independent
|
|
variables.
|
|
Then \( dz \) is no longer arbitrary.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec10">Adding the muliplier </h2>
|
|
|
|
<p>
|
|
However, we can add to
|
|
$$
|
|
df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz,
|
|
$$
|
|
|
|
a multiplum of \( d\phi \), viz. \( \lambda d\phi \), resulting in
|
|
$$
|
|
df+\lambda d\phi = (\frac{\partial f}{\partial z}+\lambda
|
|
\frac{\partial \phi}{\partial x})dx+(\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y})dy+
|
|
(\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z})dz =0.
|
|
$$
|
|
|
|
Our multiplier is chosen so that
|
|
$$
|
|
\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z} =0.
|
|
$$
|
|
|
|
<p>
|
|
We need to remember that we took \( dx \) and \( dy \) to be arbitrary and thus we must have
|
|
$$
|
|
\frac{\partial f}{\partial x}+\lambda\frac{\partial \phi}{\partial x} =0,
|
|
$$
|
|
|
|
and
|
|
$$
|
|
\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y} =0.
|
|
$$
|
|
|
|
When all these equations are satisfied, \( df=0 \). We have four unknowns, \( x,y,z \) and
|
|
\( \lambda \). Actually we want only \( x,y,z \), \( \lambda \) needs not to be determined,
|
|
it is therefore often called
|
|
Lagrange's undetermined multiplier.
|
|
If we have a set of constraints \( \phi_k \) we have the equations
|
|
$$
|
|
\frac{\partial f}{\partial x_i}+\sum_k\lambda_k\frac{\partial \phi_k}{\partial x_i} =0.
|
|
$$
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec11">Setting up the problem </h2>
|
|
In order to solve the above problem, we define the following Lagrangian function to be minimized
|
|
$$
|
|
{\cal L}(\lambda,b,\hat{w})=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-1\right],
|
|
$$
|
|
|
|
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
|
|
|
|
<p>
|
|
Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain
|
|
$$
|
|
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
|
|
$$
|
|
|
|
and
|
|
$$
|
|
\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i.
|
|
$$
|
|
|
|
Inserting these constraints into the equation for \( {\cal L} \) we obtain
|
|
$$
|
|
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
|
|
$$
|
|
|
|
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
|
|
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_blank">Karush-Kuhn-Tucker</a> (KKT) condition
|
|
$$
|
|
\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -1\right] \forall i.
|
|
$$
|
|
|
|
|
|
<ol>
|
|
<li> If \( \lambda_i > 0 \), then \( y_i(\hat{w}^T\hat{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
|
|
<li> If \( y_i(\hat{w}^T\hat{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
|
|
</ol>
|
|
|
|
When \( \lambda_i > 0 \), the vectors \( \hat{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec12">The problem to solve </h2>
|
|
|
|
<p>
|
|
We can rewrite
|
|
$$
|
|
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
|
|
$$
|
|
|
|
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
|
|
$$
|
|
\frac{1}{2} \hat{\lambda}^T\begin{bmatrix} y_1y_1\hat{x}_1^T\hat{x}_1 & y_1y_2\hat{x}_1^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_1^T\hat{x}_n \\
|
|
y_2y_1\hat{x}_2^T\hat{x}_1 & y_2y_2\hat{x}_2^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_2^T\hat{x}_n \\
|
|
\dots & \dots & \dots & \dots & \dots \\
|
|
\dots & \dots & \dots & \dots & \dots \\
|
|
y_ny_1\hat{x}_n^T\hat{x}_1 & y_ny_2\hat{x}_n^T\hat{x}_2 & \dots & \dots & y_ny_n\hat{x}_n^T\hat{x}_n \\
|
|
\end{bmatrix}\hat{\lambda}-\mathbb{1}\hat{\lambda},
|
|
$$
|
|
|
|
subject to \( \hat{y}^T\hat{\lambda}=0 \).
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec13">Examples with kernels </h2>
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">IPython.display</span> <span style="color: #008000; font-weight: bold">import</span> set_matplotlib_formats, display
|
|
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
|
|
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
|
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
|
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">mglearn</span>
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">cycler</span> <span style="color: #008000; font-weight: bold">import</span> cycler
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LogisticRegression
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVC
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> make_blobs
|
|
|
|
|
|
X, y <span style="color: #666666">=</span> make_blobs(centers<span style="color: #666666">=4</span>, random_state<span style="color: #666666">=8</span>)
|
|
y <span style="color: #666666">=</span> y <span style="color: #666666">%</span> <span style="color: #666666">2</span>
|
|
|
|
mglearn<span style="color: #666666">.</span>discrete_scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], y)
|
|
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">"Feature 0"</span>)
|
|
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">"Feature 1"</span>)
|
|
plt<span style="color: #666666">.</span>show()
|
|
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVC
|
|
linear_svm <span style="color: #666666">=</span> LinearSVC()<span style="color: #666666">.</span>fit(X, y)
|
|
|
|
mglearn<span style="color: #666666">.</span>plots<span style="color: #666666">.</span>plot_2d_separator(linear_svm, X)
|
|
mglearn<span style="color: #666666">.</span>discrete_scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], y)
|
|
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">"Feature 0"</span>)
|
|
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">"Feature 1"</span>)
|
|
</pre></div>
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># add the squared first feature</span>
|
|
X_new <span style="color: #666666">=</span> np<span style="color: #666666">.</span>hstack([X, X[:, <span style="color: #666666">1</span>:] <span style="color: #666666">**</span> <span style="color: #666666">2</span>])
|
|
|
|
|
|
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">mpl_toolkits.mplot3d</span> <span style="color: #008000; font-weight: bold">import</span> Axes3D, axes3d
|
|
figure <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
|
|
<span style="color: #408080; font-style: italic"># visualize in 3D</span>
|
|
ax <span style="color: #666666">=</span> Axes3D(figure, elev<span style="color: #666666">=-152</span>, azim<span style="color: #666666">=-26</span>)
|
|
<span style="color: #408080; font-style: italic"># plot first all the points with y==0, then all with y == 1</span>
|
|
mask <span style="color: #666666">=</span> y <span style="color: #666666">==</span> <span style="color: #666666">0</span>
|
|
ax<span style="color: #666666">.</span>scatter(X_new[mask, <span style="color: #666666">0</span>], X_new[mask, <span style="color: #666666">1</span>], X_new[mask, <span style="color: #666666">2</span>], c<span style="color: #666666">=</span><span style="color: #BA2121">'b'</span>,
|
|
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, s<span style="color: #666666">=60</span>, edgecolor<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
|
ax<span style="color: #666666">.</span>scatter(X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">0</span>], X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">1</span>], X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">2</span>], c<span style="color: #666666">=</span><span style="color: #BA2121">'r'</span>, marker<span style="color: #666666">=</span><span style="color: #BA2121">'^'</span>,
|
|
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, s<span style="color: #666666">=60</span>, edgecolor<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
|
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">"feature0"</span>)
|
|
ax<span style="color: #666666">.</span>set_ylabel(<span style="color: #BA2121">"feature1"</span>)
|
|
ax<span style="color: #666666">.</span>set_zlabel(<span style="color: #BA2121">"feature1 ** 2"</span>)
|
|
</pre></div>
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>linear_svm_3d <span style="color: #666666">=</span> LinearSVC()<span style="color: #666666">.</span>fit(X_new, y)
|
|
coef, intercept <span style="color: #666666">=</span> linear_svm_3d<span style="color: #666666">.</span>coef_<span style="color: #666666">.</span>ravel(), linear_svm_3d<span style="color: #666666">.</span>intercept_
|
|
|
|
<span style="color: #408080; font-style: italic"># show linear decision boundary</span>
|
|
figure <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
|
|
ax <span style="color: #666666">=</span> Axes3D(figure, elev<span style="color: #666666">=-152</span>, azim<span style="color: #666666">=-26</span>)
|
|
xx <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(X_new[:, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>min() <span style="color: #666666">-</span> <span style="color: #666666">2</span>, X_new[:, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>max() <span style="color: #666666">+</span> <span style="color: #666666">2</span>, <span style="color: #666666">50</span>)
|
|
yy <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(X_new[:, <span style="color: #666666">1</span>]<span style="color: #666666">.</span>min() <span style="color: #666666">-</span> <span style="color: #666666">2</span>, X_new[:, <span style="color: #666666">1</span>]<span style="color: #666666">.</span>max() <span style="color: #666666">+</span> <span style="color: #666666">2</span>, <span style="color: #666666">50</span>)
|
|
|
|
XX, YY <span style="color: #666666">=</span> np<span style="color: #666666">.</span>meshgrid(xx, yy)
|
|
ZZ <span style="color: #666666">=</span> (coef[<span style="color: #666666">0</span>] <span style="color: #666666">*</span> XX <span style="color: #666666">+</span> coef[<span style="color: #666666">1</span>] <span style="color: #666666">*</span> YY <span style="color: #666666">+</span> intercept) <span style="color: #666666">/</span> <span style="color: #666666">-</span>coef[<span style="color: #666666">2</span>]
|
|
ax<span style="color: #666666">.</span>plot_surface(XX, YY, ZZ, rstride<span style="color: #666666">=8</span>, cstride<span style="color: #666666">=8</span>, alpha<span style="color: #666666">=0.3</span>)
|
|
ax<span style="color: #666666">.</span>scatter(X_new[mask, <span style="color: #666666">0</span>], X_new[mask, <span style="color: #666666">1</span>], X_new[mask, <span style="color: #666666">2</span>], c<span style="color: #666666">=</span><span style="color: #BA2121">'b'</span>,
|
|
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, s<span style="color: #666666">=60</span>, edgecolor<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
|
ax<span style="color: #666666">.</span>scatter(X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">0</span>], X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">1</span>], X_new[<span style="color: #666666">~</span>mask, <span style="color: #666666">2</span>], c<span style="color: #666666">=</span><span style="color: #BA2121">'r'</span>, marker<span style="color: #666666">=</span><span style="color: #BA2121">'^'</span>,
|
|
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, s<span style="color: #666666">=60</span>, edgecolor<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
|
|
|
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">"feature0"</span>)
|
|
ax<span style="color: #666666">.</span>set_ylabel(<span style="color: #BA2121">"feature1"</span>)
|
|
ax<span style="color: #666666">.</span>set_zlabel(<span style="color: #BA2121">"feature1 ** 2"</span>)
|
|
|
|
ZZ <span style="color: #666666">=</span> YY <span style="color: #666666">**</span> <span style="color: #666666">2</span>
|
|
dec <span style="color: #666666">=</span> linear_svm_3d<span style="color: #666666">.</span>decision_function(np<span style="color: #666666">.</span>c_[XX<span style="color: #666666">.</span>ravel(), YY<span style="color: #666666">.</span>ravel(), ZZ<span style="color: #666666">.</span>ravel()])
|
|
plt<span style="color: #666666">.</span>contourf(XX, YY, dec<span style="color: #666666">.</span>reshape(XX<span style="color: #666666">.</span>shape), levels<span style="color: #666666">=</span>[dec<span style="color: #666666">.</span>min(), <span style="color: #666666">0</span>, dec<span style="color: #666666">.</span>max()],
|
|
cmap<span style="color: #666666">=</span>mglearn<span style="color: #666666">.</span>cm2, alpha<span style="color: #666666">=0.5</span>)
|
|
mglearn<span style="color: #666666">.</span>discrete_scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], y)
|
|
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">"Feature 0"</span>)
|
|
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">"Feature 1"</span>)
|
|
</pre></div>
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
|
|
X, y <span style="color: #666666">=</span> mglearn<span style="color: #666666">.</span>tools<span style="color: #666666">.</span>make_handcrafted_dataset()
|
|
svm <span style="color: #666666">=</span> SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">'rbf'</span>, C<span style="color: #666666">=10</span>, gamma<span style="color: #666666">=0.1</span>)<span style="color: #666666">.</span>fit(X, y)
|
|
mglearn<span style="color: #666666">.</span>plots<span style="color: #666666">.</span>plot_2d_separator(svm, X, eps<span style="color: #666666">=.5</span>)
|
|
mglearn<span style="color: #666666">.</span>discrete_scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], y)
|
|
<span style="color: #408080; font-style: italic"># plot support vectors</span>
|
|
sv <span style="color: #666666">=</span> svm<span style="color: #666666">.</span>support_vectors_
|
|
<span style="color: #408080; font-style: italic"># class labels of support vectors are given by the sign of the dual coefficients</span>
|
|
sv_labels <span style="color: #666666">=</span> svm<span style="color: #666666">.</span>dual_coef_<span style="color: #666666">.</span>ravel() <span style="color: #666666">></span> <span style="color: #666666">0</span>
|
|
mglearn<span style="color: #666666">.</span>discrete_scatter(sv[:, <span style="color: #666666">0</span>], sv[:, <span style="color: #666666">1</span>], sv_labels, s<span style="color: #666666">=15</span>, markeredgewidth<span style="color: #666666">=3</span>)
|
|
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">"Feature 0"</span>)
|
|
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">"Feature 1"</span>)
|
|
|
|
fig, axes <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>subplots(<span style="color: #666666">3</span>, <span style="color: #666666">3</span>, figsize<span style="color: #666666">=</span>(<span style="color: #666666">15</span>, <span style="color: #666666">10</span>))
|
|
|
|
<span style="color: #008000; font-weight: bold">for</span> ax, C <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(axes, [<span style="color: #666666">-1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">3</span>]):
|
|
<span style="color: #008000; font-weight: bold">for</span> a, gamma <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(ax, <span style="color: #008000">range</span>(<span style="color: #666666">-1</span>, <span style="color: #666666">2</span>)):
|
|
mglearn<span style="color: #666666">.</span>plots<span style="color: #666666">.</span>plot_svm(log_C<span style="color: #666666">=</span>C, log_gamma<span style="color: #666666">=</span>gamma, ax<span style="color: #666666">=</span>a)
|
|
|
|
axes[<span style="color: #666666">0</span>, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>legend([<span style="color: #BA2121">"class 0"</span>, <span style="color: #BA2121">"class 1"</span>, <span style="color: #BA2121">"sv class 0"</span>, <span style="color: #BA2121">"sv class 1"</span>],
|
|
ncol<span style="color: #666666">=4</span>, loc<span style="color: #666666">=</span>(<span style="color: #666666">.9</span>, <span style="color: #666666">1.2</span>))
|
|
</pre></div>
|
|
<p>
|
|
|
|
<!-- ------------------- end of main content --------------- -->
|
|
|
|
|
|
<center style="font-size:80%">
|
|
<!-- copyright --> © 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
|
|
</center>
|
|
|
|
|
|
</body>
|
|
</html>
|
|
|
|
|