FYS-STK4155/doc/pub/svm/ipynb/svm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- dom:TITLE: Data Analysis and Machine Learning: Support Vector Machines -->\n",
    "# Data Analysis and Machine Learning: Support Vector Machines\n",
    "<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
    "<!-- Author: -->  \n",
    "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
    "\n",
    "Date: **Nov 1, 2018**\n",
    "\n",
    "Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
    "\n",
    "\n",
    "\n",
    "## Support Vector Machines, overarching aims\n",
    "\n",
    "A Support Vector Machine (SVM) is a very powerful and versatile\n",
    "Machine Learning model, capable of performing linear or nonlinear\n",
    "classification, regression, and even outlier detection. It is one of\n",
    "the most popular models in Machine Learning, and anyone interested in\n",
    "Machine Learning should have it in their toolbox. SVMs are\n",
    "particularly well suited for classification of complex but small-sized or\n",
    "medium-sized datasets.  \n",
    "\n",
    "The basic mathematics relies on the definition of hyperplanes and the definition of a **margin** which separates\n",
    "classes (in case of classification problems) of variables. It is also used for regression problems.\n",
    "\n",
    "With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.\n",
    "We distringuish also between linearn and non-linear approaches.\n",
    "\n",
    "**These notes will be updated shortly with more material**\n",
    "\n",
    "## SVMs, basics with scikit-learn\n",
    "\n",
    "The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM\n",
    "model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect\n",
    "Iris-Virginica flowers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn import datasets\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.svm import LinearSVC\n",
    "iris = datasets.load_iris()\n",
    "X = iris[\"data\"][:, (2, 3)] # petal length, petal width\n",
    "y = (iris[\"target\"] == 2).astype(np.float64) # Iris-Virginica\n",
    "svm_clf = Pipeline((\n",
    "(\"scaler\", StandardScaler()),\n",
    "(\"linear_svc\", LinearSVC(C=1, loss=\"hinge\")),\n",
    "))\n",
    "svm_clf.fit(X_scaled, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you could use the SVC class, using **SVC(kernel=\"linear\", C=1)**, but it is much slower,\n",
    "especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier\n",
    "class, with **SGDClassifier(loss=\"hinge\", alpha=1/(m*C))**. This applies regular Stochastic\n",
    "Gradient Descentto train a linear SVM classifier. It does not converge as fast as the\n",
    "LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core\n",
    "training), or to handle online classification tasks.\n",
    "\n",
    "## SVMs, adding polynomial features\n",
    "\n",
    "Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets\n",
    "are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more\n",
    "features, such as polynomial features. In some cases this can result in a linearly\n",
    "separable dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "polynomial_svm_clf = Pipeline((\n",
    "(\"poly_features\", PolynomialFeatures(degree=3)),\n",
    "(\"scaler\", StandardScaler()),\n",
    "(\"svm_clf\", LinearSVC(C=10, loss=\"hinge\"))\n",
    "))\n",
    "polynomial_svm_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVMs, polynomials and kernels\n",
    "\n",
    "Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning\n",
    "algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets,\n",
    "and with a high polynomial degree it creates a huge number of features, making the model too slow.\n",
    "Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the\n",
    "kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many\n",
    "polynomial features, even with very high-degree polynomials, without actually having to add them. So\n",
    "there is no combinatorial explosion of the number of features since you don’t actually add any features.\n",
    "This trick is implemented by the SVC class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "poly_kernel_svm_clf = Pipeline((\n",
    "(\"scaler\", StandardScaler()),\n",
    "(\"svm_clf\", SVC(kernel=\"poly\", degree=3, coef0=1, C=5))\n",
    "))\n",
    "poly_kernel_svm_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code trains an SVM classifier using a 3rd-degree polynomial kernel.\n",
    "\n",
    "\n",
    "## SVMs, regression\n",
    "\n",
    "You can use Scikit-Learn’s LinearSVR class to perform linear SVM Regression. The following code\n",
    "shows how to use the regression option (more material to come)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import LinearSVR\n",
    "svm_reg = LinearSVR(epsilon=1.5)\n",
    "svm_reg.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To tackle nonlinear regression tasks, you can use a kernelized SVM model"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 2
}