+ + + + + + +
+ + +
+ + +
+
+ + +
+ + ++ +
+ + +
+ + + + +
+A Support Vector Machine (SVM) is a very powerful and versatile +Machine Learning model, capable of performing linear or nonlinear +classification, regression, and even outlier detection. It is one of +the most popular models in Machine Learning, and anyone interested in +Machine Learning should have it in their toolbox. SVMs are +particularly well suited for classification of complex but small-sized or +medium-sized datasets. + +
+The case with two well-separated classes only can be understood in an intuitive way in terms of lines in a two-dimensional space separating the two classes (see figure below). + +
+The basic mathematics behind the SVM is however less familiar to most of us. +It relies on the definition of hyperplanes and the +definition of a margin which separates classes (in case of +classification problems) of variables. It is also used for regression +problems. + +
+With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below. +We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines. + +
+
+ +
+ + +
+ + + + +
+The theory behind support vector machines (SVM hereafter) is based on +the mathematical description of so-called hyperplanes. Let us start +with a two-dimensional case. This will also allow us to introduce our +first SVM examples. These will be tailored to the case of two specific +classes, as displayed in the figure here. + +
+We assume here that our data set can be well separated into two +domains, where a straight line does the job in the separating the two +classes. Here the two classes are represented by either crosses or +circles. + +
+
+ +
+ + +
+ + + + +
+The aim of the SVM algorithm is to find a hyperplane in an \( p \)-dimensional space, where \( p \) is the number of features that distinctly classifies the data points. + +
+In a \( p \)-dimensional space, a hyperplane is what we call an affine subspace of dimension of \( p-1 \). +As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is +a two-dimensional subspace, or stated simply, a plane. + +
+In two dimensions, with the variables \( x_1 \) and \( x_2 \), the hyperplane is defined as +$$ +b+w_1x_1+w_2x_2=0, +$$ + +where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line +\( b+w_1x_1+w_2x_2=0 \). +In two dimensions we define the vectors \( \hat{x} =[x1,x2] \) and \( \hat{w}=[w1,w2] \). +We can then rewrite the above equation as +$$ +\hat{w}^T\hat{x}+b=0. +$$ + +
+
+ +
+ + +
+ + + + +
+We limit ourselves to two classes of outputs \( y_i \) and assign these classes the values \( y_i = \pm 1 \). +In a \( p \)-dimensional space of say \( p \) features we have a hyperplane defines as +$$ +b+wx_1+w_2x_2+\dots +w_px_p=0. +$$ + +If we define a +matrix \( \hat{X}=\left[\hat{x}_1,\hat{x}_2,\dots, \hat{x}_p\right] \) +of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \hat{X} \), +$$ +\hat{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. +$$ + +If the above condition is not met for a given vector \( \hat{x}_i \) we have +$$ +b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0, +$$ + +if our output \( y_i=1 \). +In this case we say that \( \hat{x}_i \) lies on one of the sides of the hyperplane and if +$$ +b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0, +$$ + +for the class of observations \( y_i=-1 \), +then \( \hat{x}_i \) lies on the other side. + +
+Equivalently, for the two classes of observations we have +$$ +y_i\left(b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip}\right) > 0. +$$ + +
+When we try to separate hyperplanes, if it exists, we can use it to construct a natural classifier: a test observation is assigned a given class depending on which side of the hyperplane it is located. + +
+
+ +
+ + +
+ + + + +
+Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional +plane. To separate the two classes of data points, there are many +possible lines (hyperplanes if you prefer a more strict naming) +that could be chosen. Our objective is to find a +plane that has the maximum margin, i.e the maximum distance between +data points of both classes. Maximizing the margin distance provides +some reinforcement so that future data points can be classified with +more confidence. + +
+What a linear classifier attempts to accomplish is to split the +feature space into two half spaces by placing a hyperplane between the +data points. This hyperplane will be our decision boundary. All +points on one side of the plane will belong to class one and all points +on the other side of the plane will belong to the second class two. + +
+Unfortunately there are many ways in which we can place a hyperplane +to divide the data. Below is an example of two candidate hyperplanes +for our data sample. + +
+
+ +
+ + +
+ + + + +
+Let us define the function +$$ +f(x) = \hat{w}^T\hat{x}+b = 0, +$$ + +as the function that determines the line \( L \) that separates two classes (our two features), see the figure here. + +
+Any point defined by \( \hat{x}_i \) and \( \hat{x}_2 \) on the line \( L \) will satisfy \( \hat{w}^T(\hat{x}_1-\hat{x}_2)=0 \). + +
+The signed distance \( \delta \) from any point defined by a vector \( \hat{x} \) and a point \( \hat{x}_0 \) on the line \( L \) is then +$$ +\delta = \frac{1}{\vert\vert \hat{w}\vert\vert}(\hat{w}^T\hat{x}+b). +$$ + +
+
+ +
+ + +
+ + + + +
+How do we find the parameter \( b \) and the vector \( \hat{w} \)? What we could +do is to define a cost function which now contains the set of all +misclassified points \( M \) and attempt to minimize this function + +$$ +C(\hat{w},b) = -\sum_{i\in M} y_i(\hat{w}^T\hat{x}_i+b). +$$ + +
+We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \hat{w}^T\hat{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us +$$ +\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i, +$$ + +and +$$ +\frac{\partial C}{\partial \hat{w}} = -\sum_{i\in M} y_ix_i. +$$ + +
+
+ +
+ + +
+ + + + +
+We can now use the Newton-Raphson method or gradient descent to solve the equations +$$ +b \leftarrow b +\eta \frac{\partial C}{\partial b}, +$$ + +and +$$ +\hat{w} \leftarrow \hat{w} +\eta \frac{\partial C}{\partial \hat{w}}, +$$ + +where \( \eta \) is our by now well-known learning rate. + +
+There are however problems with this approach, although it looks +pretty straightforward to implement. In case we separate our data into +two distinct classes, we may up with many possible lines, as indicated +in the figure and shown by running the following program. For small +gaps between the entries, we may also end up needing many iterations +before the solutions converge and if the data cannot be separated +properly into two distinct classes, we may not experience a converge +at all. + +
+
+ +
+ + +
+ + + + +
+A better approach is rather to try to define a large margin between +the two classes (if they are well separated from the beginning). + +
+Thus, we wish to find a margin \( M \) with \( \hat{w} \) normalized to +\( \vert\vert \hat{w}\vert\vert =1 \) subject to the condition + +$$ +y_i(\hat{w}^T\hat{x}_i+b) \geq M \forall i=1,2,\dots, p. +$$ + +All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line. + +
+We seek thus the largest value \( M \) defined by +$$ +\frac{1}{\vert \vert \hat{w}\vert\vert}y_i(\hat{w}^T\hat{x}_i+b) \geq M \forall i=1,2,\dots, n, +$$ + +or just +$$ +y_i(\hat{w}^T\hat{x}_i+b) \geq M\vert \vert \hat{w}\vert\vert \forall i. +$$ + +If we scale the equation so that \( \vert \vert \hat{w}\vert\vert = 1/M \), we have to find the minimum of +\( \hat{w}^T\hat{w}=\vert \vert \hat{w}\vert\vert \) (the norm) subject to the condition +$$ +y_i(\hat{w}^T\hat{x}_i+b) \geq 1 \forall i. +$$ + +
+We have thus defined our margin as the invers of the norm of \( \hat{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers. + +
+
+ +
+ + +
+ + + + +
+Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an +extreme we have +$$ +df=0. +$$ + +A necessary and sufficient condition is +$$ +\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0, +$$ + +due to +$$ +df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz. +$$ + +In many problems the variables \( x,y,z \) are often subject to constraints (such as those above for the margin) +so that they are no longer all independent. It is possible at least in principle to use each +constraint to eliminate one variable +and to proceed with a new and smaller set of independent varables. + +
+The use of so-called Lagrangian multipliers is an alternative technique when the elimination +of variables is incovenient or undesirable. Assume that we have an equation of constraint on +the variables \( x,y,z \) +$$ +\phi(x,y,z) = 0, +$$ + + resulting in +$$ +d\phi = \frac{\partial \phi}{\partial x}dx+\frac{\partial \phi}{\partial y}dy+\frac{\partial \phi}{\partial z}dz =0. +$$ + +Now we cannot set anymore +$$ +\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0, +$$ + +if \( df=0 \) is wanted +because there are now only two independent variables! Assume \( x \) and \( y \) are the independent +variables. +Then \( dz \) is no longer arbitrary. + +
+
+ +
+ + +
+ + + + +
+However, we can add to +$$ +df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz, +$$ + +a multiplum of \( d\phi \), viz. \( \lambda d\phi \), resulting in +$$ +df+\lambda d\phi = (\frac{\partial f}{\partial z}+\lambda +\frac{\partial \phi}{\partial x})dx+(\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y})dy+ +(\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z})dz =0. +$$ + +Our multiplier is chosen so that +$$ +\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z} =0. +$$ + +
+We need to remember that we took \( dx \) and \( dy \) to be arbitrary and thus we must have +$$ +\frac{\partial f}{\partial x}+\lambda\frac{\partial \phi}{\partial x} =0, +$$ + +and +$$ +\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y} =0. +$$ + +When all these equations are satisfied, \( df=0 \). We have four unknowns, \( x,y,z \) and +\( \lambda \). Actually we want only \( x,y,z \), \( \lambda \) needs not to be determined, +it is therefore often called +Lagrange's undetermined multiplier. +If we have a set of constraints \( \phi_k \) we have the equations +$$ +\frac{\partial f}{\partial x_i}+\sum_k\lambda_k\frac{\partial \phi_k}{\partial x_i} =0. +$$ + +
+
+ +
+ + +
+ + + + +
+Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain +$$ +\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, +$$ + +and +$$ +\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i. +$$ + +Inserting these constraints into the equation for \( {\cal L} \) we obtain +$$ +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +$$ + +subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). +We must in addition satisfy the Karush-Kuhn-Tucker (KKT) condition +$$ +\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -1\right] \forall i. +$$ + + +
+
+ +
+ + +
+ + + + +
+We can rewrite +$$ +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +$$ + +and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem +$$ +\frac{1}{2} \hat{\lambda}^T\begin{bmatrix} y_1y_1\hat{x}_1^T\hat{x}_1 & y_1y_2\hat{x}_1^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_1^T\hat{x}_n \\ +y_2y_1\hat{x}_2^T\hat{x}_1 & y_2y_2\hat{x}_2^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_2^T\hat{x}_n \\ +\dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots \\ +y_ny_1\hat{x}_n^T\hat{x}_1 & y_ny_2\hat{x}_n^T\hat{x}_2 & \dots & \dots & y_ny_n\hat{x}_n^T\hat{x}_n \\ +\end{bmatrix}\hat{\lambda}-\mathbb{1}\hat{\lambda}, +$$ + +subject to \( \hat{y}^T\hat{\lambda}=0 \). + +
+
+ +
+ + +
+ + + + +
+ + +
from IPython.display import set_matplotlib_formats, display
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+import mglearn
+from cycler import cycler
+from sklearn.linear_model import LogisticRegression
+from sklearn.svm import LinearSVC
+from sklearn.datasets import make_blobs
+
+
+X, y = make_blobs(centers=4, random_state=8)
+y = y % 2
+
+mglearn.discrete_scatter(X[:, 0], X[:, 1], y)
+plt.xlabel("Feature 0")
+plt.ylabel("Feature 1")
+plt.show()
+
+from sklearn.svm import LinearSVC
+linear_svm = LinearSVC().fit(X, y)
+
+mglearn.plots.plot_2d_separator(linear_svm, X)
+mglearn.discrete_scatter(X[:, 0], X[:, 1], y)
+plt.xlabel("Feature 0")
+plt.ylabel("Feature 1")
++ + +
# add the squared first feature
+X_new = np.hstack([X, X[:, 1:] ** 2])
+
+
+from mpl_toolkits.mplot3d import Axes3D, axes3d
+figure = plt.figure()
+# visualize in 3D
+ax = Axes3D(figure, elev=-152, azim=-26)
+# plot first all the points with y==0, then all with y == 1
+mask = y == 0
+ax.scatter(X_new[mask, 0], X_new[mask, 1], X_new[mask, 2], c='b',
+ cmap=mglearn.cm2, s=60, edgecolor='k')
+ax.scatter(X_new[~mask, 0], X_new[~mask, 1], X_new[~mask, 2], c='r', marker='^',
+ cmap=mglearn.cm2, s=60, edgecolor='k')
+ax.set_xlabel("feature0")
+ax.set_ylabel("feature1")
+ax.set_zlabel("feature1 ** 2")
++ + +
linear_svm_3d = LinearSVC().fit(X_new, y)
+coef, intercept = linear_svm_3d.coef_.ravel(), linear_svm_3d.intercept_
+
+# show linear decision boundary
+figure = plt.figure()
+ax = Axes3D(figure, elev=-152, azim=-26)
+xx = np.linspace(X_new[:, 0].min() - 2, X_new[:, 0].max() + 2, 50)
+yy = np.linspace(X_new[:, 1].min() - 2, X_new[:, 1].max() + 2, 50)
+
+XX, YY = np.meshgrid(xx, yy)
+ZZ = (coef[0] * XX + coef[1] * YY + intercept) / -coef[2]
+ax.plot_surface(XX, YY, ZZ, rstride=8, cstride=8, alpha=0.3)
+ax.scatter(X_new[mask, 0], X_new[mask, 1], X_new[mask, 2], c='b',
+ cmap=mglearn.cm2, s=60, edgecolor='k')
+ax.scatter(X_new[~mask, 0], X_new[~mask, 1], X_new[~mask, 2], c='r', marker='^',
+ cmap=mglearn.cm2, s=60, edgecolor='k')
+
+ax.set_xlabel("feature0")
+ax.set_ylabel("feature1")
+ax.set_zlabel("feature1 ** 2")
+
+ZZ = YY ** 2
+dec = linear_svm_3d.decision_function(np.c_[XX.ravel(), YY.ravel(), ZZ.ravel()])
+plt.contourf(XX, YY, dec.reshape(XX.shape), levels=[dec.min(), 0, dec.max()],
+ cmap=mglearn.cm2, alpha=0.5)
+mglearn.discrete_scatter(X[:, 0], X[:, 1], y)
+plt.xlabel("Feature 0")
+plt.ylabel("Feature 1")
++ + +
from sklearn.svm import SVC
+X, y = mglearn.tools.make_handcrafted_dataset()
+svm = SVC(kernel='rbf', C=10, gamma=0.1).fit(X, y)
+mglearn.plots.plot_2d_separator(svm, X, eps=.5)
+mglearn.discrete_scatter(X[:, 0], X[:, 1], y)
+# plot support vectors
+sv = svm.support_vectors_
+# class labels of support vectors are given by the sign of the dual coefficients
+sv_labels = svm.dual_coef_.ravel() > 0
+mglearn.discrete_scatter(sv[:, 0], sv[:, 1], sv_labels, s=15, markeredgewidth=3)
+plt.xlabel("Feature 0")
+plt.ylabel("Feature 1")
+
+fig, axes = plt.subplots(3, 3, figsize=(15, 10))
+
+for ax, C in zip(axes, [-1, 0, 3]):
+ for a, gamma in zip(ax, range(-1, 2)):
+ mglearn.plots.plot_svm(log_C=C, log_gamma=gamma, ax=a)
+
+axes[0, 0].legend(["class 0", "class 1", "sv class 0", "sv class 1"],
+ ncol=4, loc=(.9, 1.2))
++ +
+ +
+ + +