mhjensen
185 lines
7.2 KiB
Plaintext
185 lines
7.2 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- dom:TITLE: Data Analysis and Machine Learning: Support Vector Machines -->\n",
|
|
"# Data Analysis and Machine Learning: Support Vector Machines\n",
|
|
"<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
|
|
"<!-- Author: --> \n",
|
|
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
|
|
"\n",
|
|
"Date: **Nov 2, 2018**\n",
|
|
"\n",
|
|
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"## Support Vector Machines, overarching aims\n",
|
|
"\n",
|
|
"A Support Vector Machine (SVM) is a very powerful and versatile\n",
|
|
"Machine Learning model, capable of performing linear or nonlinear\n",
|
|
"classification, regression, and even outlier detection. It is one of\n",
|
|
"the most popular models in Machine Learning, and anyone interested in\n",
|
|
"Machine Learning should have it in their toolbox. SVMs are\n",
|
|
"particularly well suited for classification of complex but small-sized or\n",
|
|
"medium-sized datasets. \n",
|
|
"\n",
|
|
"The basic mathematics relies on the definition of hyperplanes and the definition of a **margin** which separates\n",
|
|
"classes (in case of classification problems) of variables. It is also used for regression problems.\n",
|
|
"\n",
|
|
"With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.\n",
|
|
"We distringuish also between linearn and non-linear approaches.\n",
|
|
"\n",
|
|
"**These notes will be updated shortly with more material**\n",
|
|
"\n",
|
|
"\n",
|
|
"## Strength and weakness\n",
|
|
"When we implement a linear support vector machine, the main parameter is the constant $C$. Small values of $C$ mean simple models.\n",
|
|
"These models are fast to train and also fast to predict and scale to very large data sets and work well with sparse data. Linear support vector machines make it easy to understand how a prediction is made, however it is often not easy to understand why coefficients are the way they are. These models work also well in higer dimensions. \n",
|
|
"\n",
|
|
"\n",
|
|
"## Examples with kernels"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"from IPython.display import set_matplotlib_formats, display\n",
|
|
"import pandas as pd\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import mglearn\n",
|
|
"from cycler import cycler\n",
|
|
"from sklearn.linear_model import LogisticRegression\n",
|
|
"from sklearn.svm import LinearSVC\n",
|
|
"from sklearn.datasets import make_blobs\n",
|
|
"\n",
|
|
"\n",
|
|
"X, y = make_blobs(centers=4, random_state=8)\n",
|
|
"y = y % 2\n",
|
|
"\n",
|
|
"mglearn.discrete_scatter(X[:, 0], X[:, 1], y)\n",
|
|
"plt.xlabel(\"Feature 0\")\n",
|
|
"plt.ylabel(\"Feature 1\")\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"from sklearn.svm import LinearSVC\n",
|
|
"linear_svm = LinearSVC().fit(X, y)\n",
|
|
"\n",
|
|
"mglearn.plots.plot_2d_separator(linear_svm, X)\n",
|
|
"mglearn.discrete_scatter(X[:, 0], X[:, 1], y)\n",
|
|
"plt.xlabel(\"Feature 0\")\n",
|
|
"plt.ylabel(\"Feature 1\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# add the squared first feature\n",
|
|
"X_new = np.hstack([X, X[:, 1:] ** 2])\n",
|
|
"\n",
|
|
"\n",
|
|
"from mpl_toolkits.mplot3d import Axes3D, axes3d\n",
|
|
"figure = plt.figure()\n",
|
|
"# visualize in 3D\n",
|
|
"ax = Axes3D(figure, elev=-152, azim=-26)\n",
|
|
"# plot first all the points with y==0, then all with y == 1\n",
|
|
"mask = y == 0\n",
|
|
"ax.scatter(X_new[mask, 0], X_new[mask, 1], X_new[mask, 2], c='b',\n",
|
|
" cmap=mglearn.cm2, s=60, edgecolor='k')\n",
|
|
"ax.scatter(X_new[~mask, 0], X_new[~mask, 1], X_new[~mask, 2], c='r', marker='^',\n",
|
|
" cmap=mglearn.cm2, s=60, edgecolor='k')\n",
|
|
"ax.set_xlabel(\"feature0\")\n",
|
|
"ax.set_ylabel(\"feature1\")\n",
|
|
"ax.set_zlabel(\"feature1 ** 2\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"linear_svm_3d = LinearSVC().fit(X_new, y)\n",
|
|
"coef, intercept = linear_svm_3d.coef_.ravel(), linear_svm_3d.intercept_\n",
|
|
"\n",
|
|
"# show linear decision boundary\n",
|
|
"figure = plt.figure()\n",
|
|
"ax = Axes3D(figure, elev=-152, azim=-26)\n",
|
|
"xx = np.linspace(X_new[:, 0].min() - 2, X_new[:, 0].max() + 2, 50)\n",
|
|
"yy = np.linspace(X_new[:, 1].min() - 2, X_new[:, 1].max() + 2, 50)\n",
|
|
"\n",
|
|
"XX, YY = np.meshgrid(xx, yy)\n",
|
|
"ZZ = (coef[0] * XX + coef[1] * YY + intercept) / -coef[2]\n",
|
|
"ax.plot_surface(XX, YY, ZZ, rstride=8, cstride=8, alpha=0.3)\n",
|
|
"ax.scatter(X_new[mask, 0], X_new[mask, 1], X_new[mask, 2], c='b',\n",
|
|
" cmap=mglearn.cm2, s=60, edgecolor='k')\n",
|
|
"ax.scatter(X_new[~mask, 0], X_new[~mask, 1], X_new[~mask, 2], c='r', marker='^',\n",
|
|
" cmap=mglearn.cm2, s=60, edgecolor='k')\n",
|
|
"\n",
|
|
"ax.set_xlabel(\"feature0\")\n",
|
|
"ax.set_ylabel(\"feature1\")\n",
|
|
"ax.set_zlabel(\"feature1 ** 2\")\n",
|
|
"\n",
|
|
"ZZ = YY ** 2\n",
|
|
"dec = linear_svm_3d.decision_function(np.c_[XX.ravel(), YY.ravel(), ZZ.ravel()])\n",
|
|
"plt.contourf(XX, YY, dec.reshape(XX.shape), levels=[dec.min(), 0, dec.max()],\n",
|
|
" cmap=mglearn.cm2, alpha=0.5)\n",
|
|
"mglearn.discrete_scatter(X[:, 0], X[:, 1], y)\n",
|
|
"plt.xlabel(\"Feature 0\")\n",
|
|
"plt.ylabel(\"Feature 1\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn.svm import SVC\n",
|
|
"X, y = mglearn.tools.make_handcrafted_dataset() \n",
|
|
"svm = SVC(kernel='rbf', C=10, gamma=0.1).fit(X, y)\n",
|
|
"mglearn.plots.plot_2d_separator(svm, X, eps=.5)\n",
|
|
"mglearn.discrete_scatter(X[:, 0], X[:, 1], y)\n",
|
|
"# plot support vectors\n",
|
|
"sv = svm.support_vectors_\n",
|
|
"# class labels of support vectors are given by the sign of the dual coefficients\n",
|
|
"sv_labels = svm.dual_coef_.ravel() > 0\n",
|
|
"mglearn.discrete_scatter(sv[:, 0], sv[:, 1], sv_labels, s=15, markeredgewidth=3)\n",
|
|
"plt.xlabel(\"Feature 0\")\n",
|
|
"plt.ylabel(\"Feature 1\")\n",
|
|
"\n",
|
|
"fig, axes = plt.subplots(3, 3, figsize=(15, 10))\n",
|
|
"\n",
|
|
"for ax, C in zip(axes, [-1, 0, 3]):\n",
|
|
" for a, gamma in zip(ax, range(-1, 2)):\n",
|
|
" mglearn.plots.plot_svm(log_C=C, log_gamma=gamma, ax=a)\n",
|
|
" \n",
|
|
"axes[0, 0].legend([\"class 0\", \"class 1\", \"sv class 0\", \"sv class 1\"],\n",
|
|
" ncol=4, loc=(.9, 1.2))"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|