lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
eb31c10d1ca198eaaca4c2c83e8f5cdb0aeb69e4
FYS-STK4155/doc/LectureNotes/_build/html/chapter5.html
T
Morten Hjorth-Jensen eb31c10d1c addex exercise week 37
2025-09-01 13:43:58 +02:00

1497 lines
76 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en" data-content_root="./" >
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>8. Support Vector Machines, overarching aims &#8212; Applied Data Analysis and Machine Learning</title>
<script data-cfasync="false">
document.documentElement.dataset.mode = localStorage.getItem("mode") || "";
document.documentElement.dataset.theme = localStorage.getItem("theme") || "";
</script>
<!-- Loaded before other Sphinx assets -->
<link href="_static/styles/theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="_static/styles/bootstrap.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="_static/styles/pydata-sphinx-theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="_static/vendor/fontawesome/6.5.2/css/all.min.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-solid-900.woff2" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-brands-400.woff2" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-regular-400.woff2" />
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="_static/styles/sphinx-book-theme.css?v=eba8b062" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css?v=13237357" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css?v=76b2166b" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css?v=6644e6bb" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css?v=4fa983c6" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-design.min.css?v=95c83b7e" />
<!-- Pre-loaded scripts that we'll load fully later -->
<link rel="preload" as="script" href="_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b" />
<link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b" />
<script src="_static/vendor/fontawesome/6.5.2/js/all.min.js?digest=dfe6caa3a7d634c4db9b"></script>
<script src="_static/documentation_options.js?v=9eb32ce0"></script>
<script src="_static/doctools.js?v=9a2dae69"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/clipboard.min.js?v=a7894cd8"></script>
<script src="_static/copybutton.js?v=f281be69"></script>
<script src="_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
<script>let toggleHintShow = 'Click to show';</script>
<script>let toggleHintHide = 'Click to hide';</script>
<script>let toggleOpenOnPrint = 'true';</script>
<script src="_static/togglebutton.js?v=4a39c7ea"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
<script src="_static/design-tabs.js?v=f930bc37"></script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
<script async="async" src="_static/sphinx-thebe.js?v=c100c467"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
<script>window.MathJax = {"options": {"processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script>
<script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<script>DOCUMENTATION_OPTIONS.pagename = 'chapter5';</script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="9. Decision trees, overarching aims" href="chapter6.html" />
<link rel="prev" title="7. Optimization, the central part of any Machine Learning algortithm" href="chapteroptimization.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
</head>
<body data-bs-spy="scroll" data-bs-target=".bd-toc-nav" data-offset="180" data-bs-root-margin="0px 0px -60%" data-default-mode="">
<div id="pst-skip-link" class="skip-link d-print-none"><a href="#main-content">Skip to main content</a></div>
<div id="pst-scroll-pixel-helper"></div>
<button type="button" class="btn rounded-pill" id="pst-back-to-top">
<i class="fa-solid fa-arrow-up"></i>Back to top</button>
<input type="checkbox"
class="sidebar-toggle"
id="pst-primary-sidebar-checkbox"/>
<label class="overlay overlay-primary" for="pst-primary-sidebar-checkbox"></label>
<input type="checkbox"
class="sidebar-toggle"
id="pst-secondary-sidebar-checkbox"/>
<label class="overlay overlay-secondary" for="pst-secondary-sidebar-checkbox"></label>
<div class="search-button__wrapper">
<div class="search-button__overlay"></div>
<div class="search-button__search-container">
<form class="bd-search d-flex align-items-center"
action="search.html"
method="get">
<i class="fa-solid fa-magnifying-glass"></i>
<input type="search"
class="form-control"
name="q"
id="search-input"
placeholder="Search this book..."
aria-label="Search this book..."
autocomplete="off"
autocorrect="off"
autocapitalize="off"
spellcheck="false"/>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd>K</kbd></span>
</form></div>
</div>
<div class="pst-async-banner-revealer d-none">
<aside id="bd-header-version-warning" class="d-none d-print-none" aria-label="Version warning"></aside>
</div>
<header class="bd-header navbar navbar-expand-lg bd-navbar d-print-none">
</header>
<div class="bd-container">
<div class="bd-container__inner bd-page-width">
<div class="bd-sidebar-primary bd-sidebar">
<div class="sidebar-header-items sidebar-primary__section">
</div>
<div class="sidebar-primary-items__start sidebar-primary__section">
<div class="sidebar-primary-item">
<a class="navbar-brand logo" href="intro.html">
<img src="_static/logo.png" class="logo__image only-light" alt="Applied Data Analysis and Machine Learning - Home"/>
<script>document.write(`<img src="_static/logo.png" class="logo__image only-dark" alt="Applied Data Analysis and Machine Learning - Home"/>`);</script>
</a></div>
<div class="sidebar-primary-item">
<script>
document.write(`
<button class="btn search-button-field search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass"></i>
<span class="search-button__default-text">Search</span>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
</button>
`);
</script></div>
<div class="sidebar-primary-item"><nav class="bd-links bd-docs-nav" aria-label="Main">
<div class="bd-toc-item navbar-nav active">
<ul class="nav bd-sidenav bd-sidenav__home-link">
<li class="toctree-l1">
<a class="reference internal" href="intro.html">
Applied Data Analysis and Machine Learning
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">About the course</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="schedule.html">Course setting</a></li>
<li class="toctree-l1"><a class="reference internal" href="teachers.html">Teachers and Grading</a></li>
<li class="toctree-l1"><a class="reference internal" href="textbooks.html">Textbooks</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Review of Statistics with Resampling Techniques and Linear Algebra</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="statistics.html">1. Elements of Probability Theory and Statistical Data Analysis</a></li>
<li class="toctree-l1"><a class="reference internal" href="linalg.html">2. Linear Algebra, Handling of Arrays and more Python Features</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">From Regression to Support Vector Machines</span></p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapter1.html">3. Linear Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter2.html">4. Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter3.html">5. Resampling Methods</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter4.html">6. Logistic Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapteroptimization.html">7. Optimization, the central part of any Machine Learning algortithm</a></li>
<li class="toctree-l1 current active"><a class="current reference internal" href="#">8. Support Vector Machines, overarching aims</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Decision Trees, Ensemble Methods and Boosting</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapter6.html">9. Decision trees, overarching aims</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter7.html">10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Dimensionality Reduction</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapter8.html">11. Basic ideas of the Principal Component Analysis (PCA)</a></li>
<li class="toctree-l1"><a class="reference internal" href="clustering.html">12. Clustering and Unsupervised Learning</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Deep Learning Methods</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapter9.html">13. Neural networks</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter10.html">14. Building a Feed Forward Neural Network</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter11.html">15. Solving Differential Equations with Deep Learning</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter12.html">16. Convolutional Neural Networks</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter13.html">17. Recurrent neural networks: Overarching view</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek35.html">Exercises week 35</a></li>
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
</nav></div>
</div>
<div class="sidebar-primary-items__end sidebar-primary__section">
</div>
<div id="rtd-footer-container"></div>
</div>
<main id="main-content" class="bd-main" role="main">
<div class="sbt-scroll-pixel-helper"></div>
<div class="bd-content">
<div class="bd-article-container">
<div class="bd-header-article d-print-none">
<div class="header-article-items header-article__inner">
<div class="header-article-items__start">
<div class="header-article-item"><button class="sidebar-toggle primary-toggle btn btn-sm" title="Toggle primary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
<span class="fa-solid fa-bars"></span>
</button></div>
</div>
<div class="header-article-items__end">
<div class="header-article-item">
<div class="article-header-buttons">
<div class="dropdown dropdown-download-buttons">
<button class="btn dropdown-toggle" type="button" data-bs-toggle="dropdown" aria-expanded="false" aria-label="Download this page">
<i class="fas fa-download"></i>
</button>
<ul class="dropdown-menu">
<li><a href="_sources/chapter5.ipynb" target="_blank"
class="btn btn-sm btn-download-source-button dropdown-item"
title="Download source file"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-file"></i>
</span>
<span class="btn__text-container">.ipynb</span>
</a>
</li>
<li>
<button onclick="window.print()"
class="btn btn-sm btn-download-pdf-button dropdown-item"
title="Print to PDF"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-file-pdf"></i>
</span>
<span class="btn__text-container">.pdf</span>
</button>
</li>
</ul>
</div>
<button onclick="toggleFullScreen()"
class="btn btn-sm btn-fullscreen-button"
title="Fullscreen mode"
data-bs-placement="bottom" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-expand"></i>
</span>
</button>
<script>
document.write(`
<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button" title="light/dark" aria-label="light/dark" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light"></i>
<i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark"></i>
<i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto"></i>
</button>
`);
</script>
<script>
document.write(`
<button class="btn btn-sm pst-navbar-icon search-button search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass fa-lg"></i>
</button>
`);
</script>
<button class="sidebar-toggle secondary-toggle btn btn-sm" title="Toggle secondary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
<span class="fa-solid fa-list"></span>
</button>
</div></div>
</div>
</div>
</div>
<div id="jb-print-docs-body" class="onlyprint">
<h1>Support Vector Machines, overarching aims</h1>
<!-- Table of contents -->
<div id="print-main-content">
<div id="jb-print-toc">
<div>
<h2> Contents </h2>
</div>
<nav aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#hyperplanes-and-all-that">8.1. Hyperplanes and all that</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#the-two-dimensional-case">8.1.1. The two-dimensional case</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#a-better-approach">8.1.2. A better approach</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#a-quick-reminder-on-lagrangian-multipliers">8.2. A quick Reminder on Lagrangian Multipliers</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#a-soft-classifier">8.3. A soft classifier</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#kernels-and-non-linearity">8.4. Kernels and non-linearity</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#different-kernels-and-mercer-s-theorem">8.5. Different kernels and Mercer’s theorem</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-moons-example">8.6. The moons example</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#mathematical-optimization-of-convex-functions">8.7. Mathematical optimization of convex functions</a></li>
</ul>
</nav>
</div>
</div>
</div>
<div id="searchbox"></div>
<article class="bd-article">
<section class="tex2jax_ignore mathjax_ignore" id="support-vector-machines-overarching-aims">
<h1><span class="section-number">8. </span>Support Vector Machines, overarching aims<a class="headerlink" href="#support-vector-machines-overarching-aims" title="Link to this heading">#</a></h1>
<p>A Support Vector Machine (SVM) is a very powerful and versatile
Machine Learning method, capable of performing linear or nonlinear
classification, regression, and even outlier detection. It is one of
the most popular models in Machine Learning, and anyone interested in
Machine Learning should have it in their toolbox. SVMs are
particularly well suited for classification of complex but small-sized or
medium-sized datasets.</p>
<p>The case with two well-separated classes only can be understood in an
intuitive way in terms of lines in a two-dimensional space separating
the two classes (see figure below).</p>
<p>The basic mathematics behind the SVM is however less familiar to most of us.
It relies on the definition of hyperplanes and the
definition of a <strong>margin</strong> which separates classes (in case of
classification problems) of variables. It is also used for regression
problems.</p>
<p>With SVMs we distinguish between hard margin and soft margins. The
latter introduces a so-called softening parameter to be discussed
below. We distinguish also between linear and non-linear
approaches. The latter are the most frequent ones since it is rather
unlikely that we can separate classes easily by say straight lines.</p>
<section id="hyperplanes-and-all-that">
<h2><span class="section-number">8.1. </span>Hyperplanes and all that<a class="headerlink" href="#hyperplanes-and-all-that" title="Link to this heading">#</a></h2>
<p>The theory behind support vector machines (SVM hereafter) is based on
the mathematical description of so-called hyperplanes. Let us start
with a two-dimensional case. This will also allow us to introduce our
first SVM examples. These will be tailored to the case of two specific
classes, as displayed in the figure here based on the usage of the petal data.</p>
<p>We assume here that our data set can be well separated into two
domains, where a straight line does the job in the separating the two
classes. Here the two classes are represented by either squares or
circles.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>%matplotlib inline
from sklearn import datasets
from sklearn.svm import SVC, LinearSVC
from sklearn.linear_model import SGDClassifier
from sklearn.preprocessing import StandardScaler
import matplotlib
import matplotlib.pyplot as plt
plt.rcParams[&#39;axes.labelsize&#39;] = 14
plt.rcParams[&#39;xtick.labelsize&#39;] = 12
plt.rcParams[&#39;ytick.labelsize&#39;] = 12
iris = datasets.load_iris()
X = iris[&quot;data&quot;][:, (2, 3)] # petal length, petal width
y = iris[&quot;target&quot;]
setosa_or_versicolor = (y == 0) | (y == 1)
X = X[setosa_or_versicolor]
y = y[setosa_or_versicolor]
C = 5
alpha = 1 / (C * len(X))
lin_clf = LinearSVC(loss=&quot;hinge&quot;, C=C, random_state=42)
svm_clf = SVC(kernel=&quot;linear&quot;, C=C)
sgd_clf = SGDClassifier(loss=&quot;hinge&quot;, learning_rate=&quot;constant&quot;, eta0=0.001, alpha=alpha,
max_iter=100000, random_state=42)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
lin_clf.fit(X_scaled, y)
svm_clf.fit(X_scaled, y)
sgd_clf.fit(X_scaled, y)
print(&quot;LinearSVC: &quot;, lin_clf.intercept_, lin_clf.coef_)
print(&quot;SVC: &quot;, svm_clf.intercept_, svm_clf.coef_)
print(&quot;SGDClassifier(alpha={:.5f}):&quot;.format(sgd_clf.alpha), sgd_clf.intercept_, sgd_clf.coef_)
# Compute the slope and bias of each decision boundary
w1 = -lin_clf.coef_[0, 0]/lin_clf.coef_[0, 1]
b1 = -lin_clf.intercept_[0]/lin_clf.coef_[0, 1]
w2 = -svm_clf.coef_[0, 0]/svm_clf.coef_[0, 1]
b2 = -svm_clf.intercept_[0]/svm_clf.coef_[0, 1]
w3 = -sgd_clf.coef_[0, 0]/sgd_clf.coef_[0, 1]
b3 = -sgd_clf.intercept_[0]/sgd_clf.coef_[0, 1]
# Transform the decision boundary lines back to the original scale
line1 = scaler.inverse_transform([[-10, -10 * w1 + b1], [10, 10 * w1 + b1]])
line2 = scaler.inverse_transform([[-10, -10 * w2 + b2], [10, 10 * w2 + b2]])
line3 = scaler.inverse_transform([[-10, -10 * w3 + b3], [10, 10 * w3 + b3]])
# Plot all three decision boundaries
plt.figure(figsize=(11, 4))
plt.plot(line1[:, 0], line1[:, 1], &quot;k:&quot;, label=&quot;LinearSVC&quot;)
plt.plot(line2[:, 0], line2[:, 1], &quot;b--&quot;, linewidth=2, label=&quot;SVC&quot;)
plt.plot(line3[:, 0], line3[:, 1], &quot;r-&quot;, label=&quot;SGDClassifier&quot;)
plt.plot(X[:, 0][y==1], X[:, 1][y==1], &quot;bs&quot;) # label=&quot;Iris-Versicolor&quot;
plt.plot(X[:, 0][y==0], X[:, 1][y==0], &quot;yo&quot;) # label=&quot;Iris-Setosa&quot;
plt.xlabel(&quot;Petal length&quot;, fontsize=14)
plt.ylabel(&quot;Petal width&quot;, fontsize=14)
plt.legend(loc=&quot;upper center&quot;, fontsize=14)
plt.axis([0, 5.5, 0, 2])
plt.show()
</pre></div>
</div>
</div>
</div>
<p>The aim of the SVM algorithm is to find a hyperplane in a
<span class="math notranslate nohighlight">\(p\)</span>-dimensional space, where <span class="math notranslate nohighlight">\(p\)</span> is the number of features that
distinctly classifies the data points.</p>
<p>In a <span class="math notranslate nohighlight">\(p\)</span>-dimensional space, a hyperplane is what we call an affine subspace of dimension of <span class="math notranslate nohighlight">\(p-1\)</span>.
As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is
a two-dimensional subspace, or stated simply, a plane.</p>
<p>In two dimensions, with the variables <span class="math notranslate nohighlight">\(x_1\)</span> and <span class="math notranslate nohighlight">\(x_2\)</span>, the hyperplane is defined as</p>
<div class="math notranslate nohighlight">
\[
b+w_1x_1+w_2x_2=0,
\]</div>
<p>where <span class="math notranslate nohighlight">\(b\)</span> is the intercept and <span class="math notranslate nohighlight">\(w_1\)</span> and <span class="math notranslate nohighlight">\(w_2\)</span> define the elements of a vector orthogonal to the line
<span class="math notranslate nohighlight">\(b+w_1x_1+w_2x_2=0\)</span>.
In two dimensions we define the vectors <span class="math notranslate nohighlight">\(\boldsymbol{x} =[x1,x2]\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{w}=[w1,w2]\)</span>.
We can then rewrite the above equation as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{x}^T\boldsymbol{w}+b=0.
\]</div>
<p>We limit ourselves to two classes of outputs <span class="math notranslate nohighlight">\(y_i\)</span> and assign these classes the values <span class="math notranslate nohighlight">\(y_i = \pm 1\)</span>.
In a <span class="math notranslate nohighlight">\(p\)</span>-dimensional space of say <span class="math notranslate nohighlight">\(p\)</span> features we have a hyperplane defines as</p>
<div class="math notranslate nohighlight">
\[
b+wx_1+w_2x_2+\dots +w_px_p=0.
\]</div>
<p>If we define a
matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right]\)</span>
of dimension <span class="math notranslate nohighlight">\(n\times p\)</span>, where <span class="math notranslate nohighlight">\(n\)</span> represents the observations for each feature and each vector <span class="math notranslate nohighlight">\(x_i\)</span> is a column vector of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>,</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\end{split}\]</div>
<p>If the above condition is not met for a given vector <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> we have</p>
<div class="math notranslate nohighlight">
\[
b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} &gt;0,
\]</div>
<p>if our output <span class="math notranslate nohighlight">\(y_i=1\)</span>.
In this case we say that <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> lies on one of the sides of the hyperplane and if</p>
<div class="math notranslate nohighlight">
\[
b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} &lt; 0,
\]</div>
<p>for the class of observations <span class="math notranslate nohighlight">\(y_i=-1\)</span>,
then <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> lies on the other side.</p>
<p>Equivalently, for the two classes of observations we have</p>
<div class="math notranslate nohighlight">
\[
y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) &gt; 0.
\]</div>
<p>When we try to separate hyperplanes, if it exists, we can use it to construct a natural classifier: a test observation is assigned a given class depending on which side of the hyperplane it is located.</p>
<section id="the-two-dimensional-case">
<h3><span class="section-number">8.1.1. </span>The two-dimensional case<a class="headerlink" href="#the-two-dimensional-case" title="Link to this heading">#</a></h3>
<p>Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional
plane. To separate the two classes of data points, there are many
possible lines (hyperplanes if you prefer a more strict naming)<br />
that could be chosen. Our objective is to find a
plane that has the maximum margin, i.e the maximum distance between
data points of both classes. Maximizing the margin distance provides
some reinforcement so that future data points can be classified with
more confidence.</p>
<p>What a linear classifier attempts to accomplish is to split the
feature space into two half spaces by placing a hyperplane between the
data points. This hyperplane will be our decision boundary. All
points on one side of the plane will belong to class one and all points
on the other side of the plane will belong to the second class two.</p>
<p>Unfortunately there are many ways in which we can place a hyperplane
to divide the data. Below is an example of two candidate hyperplanes
for our data sample.</p>
<p>Let us define the function</p>
<div class="math notranslate nohighlight">
\[
f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0,
\]</div>
<p>as the function that determines the line <span class="math notranslate nohighlight">\(L\)</span> that separates two classes (our two features), see the figure here.</p>
<p>Any point defined by <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{x}_2\)</span> on the line <span class="math notranslate nohighlight">\(L\)</span> will satisfy <span class="math notranslate nohighlight">\(\boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0\)</span>.</p>
<p>The signed distance <span class="math notranslate nohighlight">\(\delta\)</span> from any point defined by a vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> and a point <span class="math notranslate nohighlight">\(\boldsymbol{x}_0\)</span> on the line <span class="math notranslate nohighlight">\(L\)</span> is then</p>
<div class="math notranslate nohighlight">
\[
\delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b).
\]</div>
<p>How do we find the parameter <span class="math notranslate nohighlight">\(b\)</span> and the vector <span class="math notranslate nohighlight">\(\boldsymbol{w}\)</span>? What we could
do is to define a cost function which now contains the set of all
misclassified points <span class="math notranslate nohighlight">\(M\)</span> and attempt to minimize this function</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b).
\]</div>
<p>We could now for example define all values <span class="math notranslate nohighlight">\(y_i =1\)</span> as misclassified in case we have <span class="math notranslate nohighlight">\(\boldsymbol{w}^T\boldsymbol{x}_i+b &lt; 0\)</span> and the opposite if we have <span class="math notranslate nohighlight">\(y_i=-1\)</span>. Taking the derivatives gives us</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i.
\]</div>
<p>We can now use the Newton-Raphson method or different variants of the gradient descent family (from plain gradient descent to various stochastic gradient descent approaches) to solve the equations</p>
<div class="math notranslate nohighlight">
\[
b \leftarrow b +\eta \frac{\partial C}{\partial b},
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}},
\]</div>
<p>where <span class="math notranslate nohighlight">\(\eta\)</span> is our by now well-known learning rate.</p>
<p>The equations we discussed above can be coded rather easily (the
framework is similar to what we developed for logistic
regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way.</p>
<p>There are however problems with this approach, although it looks
pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes.</p>
<p>For small
gaps between the entries, we may also end up needing many iterations
before the solutions converge and if the data cannot be separated
properly into two distinct classes, we may not experience a converge
at all.</p>
</section>
<section id="a-better-approach">
<h3><span class="section-number">8.1.2. </span>A better approach<a class="headerlink" href="#a-better-approach" title="Link to this heading">#</a></h3>
<p>A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).</p>
<p>Thus, we wish to find a margin <span class="math notranslate nohighlight">\(M\)</span> with <span class="math notranslate nohighlight">\(\boldsymbol{w}\)</span> normalized to
<span class="math notranslate nohighlight">\(\vert\vert \boldsymbol{w}\vert\vert =1\)</span> subject to the condition</p>
<div class="math notranslate nohighlight">
\[
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
\]</div>
<p>All points are thus at a signed distance from the decision boundary defined by the line <span class="math notranslate nohighlight">\(L\)</span>. The parameters <span class="math notranslate nohighlight">\(b\)</span> and <span class="math notranslate nohighlight">\(w_1\)</span> and <span class="math notranslate nohighlight">\(w_2\)</span> define this line.</p>
<p>We seek thus the largest value <span class="math notranslate nohighlight">\(M\)</span> defined by</p>
<div class="math notranslate nohighlight">
\[
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\]</div>
<p>or just</p>
<div class="math notranslate nohighlight">
\[
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
\]</div>
<p>If we scale the equation so that <span class="math notranslate nohighlight">\(\vert \vert \boldsymbol{w}\vert\vert = 1/M\)</span>, we have to find the minimum of
<span class="math notranslate nohighlight">\(\boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert\)</span> (the norm) subject to the condition</p>
<div class="math notranslate nohighlight">
\[
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
\]</div>
<p>We have thus defined our margin as the invers of the norm of
<span class="math notranslate nohighlight">\(\boldsymbol{w}\)</span>. We want to minimize the norm in order to have a as large as
possible margin <span class="math notranslate nohighlight">\(M\)</span>. Before we proceed, we need to remind ourselves
about Lagrangian multipliers.</p>
</section>
</section>
<section id="a-quick-reminder-on-lagrangian-multipliers">
<h2><span class="section-number">8.2. </span>A quick Reminder on Lagrangian Multipliers<a class="headerlink" href="#a-quick-reminder-on-lagrangian-multipliers" title="Link to this heading">#</a></h2>
<p>Consider a function of three independent variables <span class="math notranslate nohighlight">\(f(x,y,z)\)</span> . For the function <span class="math notranslate nohighlight">\(f\)</span> to be an
extreme we have</p>
<div class="math notranslate nohighlight">
\[
df=0.
\]</div>
<p>A necessary and sufficient condition is</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0,
\]</div>
<p>due to</p>
<div class="math notranslate nohighlight">
\[
df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz.
\]</div>
<p>In many problems the variables <span class="math notranslate nohighlight">\(x,y,z\)</span> are often subject to constraints (such as those above for the margin)
so that they are no longer all independent. It is possible at least in principle to use each
constraint to eliminate one variable
and to proceed with a new and smaller set of independent varables.</p>
<p>The use of so-called Lagrangian multipliers is an alternative technique when the elimination
of variables is incovenient or undesirable. Assume that we have an equation of constraint on
the variables <span class="math notranslate nohighlight">\(x,y,z\)</span></p>
<div class="math notranslate nohighlight">
\[
\phi(x,y,z) = 0,
\]</div>
<p>resulting in</p>
<div class="math notranslate nohighlight">
\[
d\phi = \frac{\partial \phi}{\partial x}dx+\frac{\partial \phi}{\partial y}dy+\frac{\partial \phi}{\partial z}dz =0.
\]</div>
<p>Now we cannot set anymore</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0,
\]</div>
<p>if <span class="math notranslate nohighlight">\(df=0\)</span> is wanted
because there are now only two independent variables! Assume <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> are the independent
variables.
Then <span class="math notranslate nohighlight">\(dz\)</span> is no longer arbitrary.</p>
<p>However, we can add to</p>
<div class="math notranslate nohighlight">
\[
df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz,
\]</div>
<p>a multiplum of <span class="math notranslate nohighlight">\(d\phi\)</span>, viz. <span class="math notranslate nohighlight">\(\lambda d\phi\)</span>, resulting in</p>
<div class="math notranslate nohighlight">
\[
df+\lambda d\phi = (\frac{\partial f}{\partial z}+\lambda
\frac{\partial \phi}{\partial x})dx+(\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y})dy+
(\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z})dz =0.
\]</div>
<p>Our multiplier is chosen so that</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z} =0.
\]</div>
<p>We need to remember that we took <span class="math notranslate nohighlight">\(dx\)</span> and <span class="math notranslate nohighlight">\(dy\)</span> to be arbitrary and thus we must have</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial f}{\partial x}+\lambda\frac{\partial \phi}{\partial x} =0,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y} =0.
\]</div>
<p>When all these equations are satisfied, <span class="math notranslate nohighlight">\(df=0\)</span>. We have four unknowns, <span class="math notranslate nohighlight">\(x,y,z\)</span> and
<span class="math notranslate nohighlight">\(\lambda\)</span>. Actually we want only <span class="math notranslate nohighlight">\(x,y,z\)</span>, <span class="math notranslate nohighlight">\(\lambda\)</span> needs not to be determined,
it is therefore often called
Lagrange’s undetermined multiplier.
If we have a set of constraints <span class="math notranslate nohighlight">\(\phi_k\)</span> we have the equations</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial f}{\partial x_i}+\sum_k\lambda_k\frac{\partial \phi_k}{\partial x_i} =0.
\]</div>
<p>In order to solve the above problem, we define the following Lagrangian function to be minimized</p>
<div class="math notranslate nohighlight">
\[
\cal{L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right],
\]</div>
<p>where <span class="math notranslate nohighlight">\(\lambda_i\)</span> is a so-called Lagrange multiplier subject to the condition <span class="math notranslate nohighlight">\(\lambda_i \geq 0\)</span>.</p>
<p>Taking the derivatives with respect to <span class="math notranslate nohighlight">\(b\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{w}\)</span> we obtain</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \cal{L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \cal{L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
\]</div>
<p>Inserting these constraints into the equation for <span class="math notranslate nohighlight">\(\cal{L}\)</span> we obtain</p>
<div class="math notranslate nohighlight">
\[
\cal{L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
\]</div>
<p>subject to the constraints <span class="math notranslate nohighlight">\(\lambda_i\geq 0\)</span> and <span class="math notranslate nohighlight">\(\sum_i\lambda_iy_i=0\)</span>.
We must in addition satisfy the <a class="reference external" href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions">Karush-Kuhn-Tucker</a> (KKT) condition</p>
<div class="math notranslate nohighlight">
\[
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\]</div>
<ol class="arabic simple">
<li><p>If <span class="math notranslate nohighlight">\(\lambda_i &gt; 0\)</span>, then <span class="math notranslate nohighlight">\(y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1\)</span> and we say that <span class="math notranslate nohighlight">\(x_i\)</span> is on the boundary.</p></li>
<li><p>If <span class="math notranslate nohighlight">\(y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)&gt; 1\)</span>, we say <span class="math notranslate nohighlight">\(x_i\)</span> is not on the boundary and we set <span class="math notranslate nohighlight">\(\lambda_i=0\)</span>.</p></li>
</ol>
<p>When <span class="math notranslate nohighlight">\(\lambda_i &gt; 0\)</span>, the vectors <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin <span class="math notranslate nohighlight">\(M\)</span>.</p>
<p>We can rewrite</p>
<div class="math notranslate nohighlight">
\[
\cal{L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
\]</div>
<p>and its constraints in terms of a matrix-vector problem where we minimize w.r.t. <span class="math notranslate nohighlight">\(\lambda\)</span> the following problem</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 &amp; y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 &amp; \dots &amp; \dots &amp; y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 &amp; y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 &amp; \dots &amp; \dots &amp; y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
\dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots \\
\dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots \\
y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 &amp; y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 &amp; \dots &amp; \dots &amp; y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
\end{split}\]</div>
<p>subject to <span class="math notranslate nohighlight">\(\boldsymbol{y}^T\boldsymbol{\lambda}=0\)</span>. Here we defined the vectors <span class="math notranslate nohighlight">\(\boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]\)</span> and
<span class="math notranslate nohighlight">\(\boldsymbol{y}=[y_1,y_2,\dots,y_n]\)</span>.</p>
<p>Solving the above problem, yields the values of <span class="math notranslate nohighlight">\(\lambda_i\)</span>.
To find the coefficients of your hyperplane we need simply to compute</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
\]</div>
<p>With our vector <span class="math notranslate nohighlight">\(\boldsymbol{w}\)</span> we can in turn find the value of the intercept <span class="math notranslate nohighlight">\(b\)</span> (here in two dimensions) via</p>
<div class="math notranslate nohighlight">
\[
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
\]</div>
<p>resulting in</p>
<div class="math notranslate nohighlight">
\[
b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
\]</div>
<p>or if we write it out in terms of the support vectors only, with <span class="math notranslate nohighlight">\(N_s\)</span> being their number, we have</p>
<div class="math notranslate nohighlight">
\[
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right).
\]</div>
<p>With our hyperplane coefficients we can use our classifier to assign any observation by simply using</p>
<div class="math notranslate nohighlight">
\[
y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
\]</div>
<p>Below we discuss how to find the optimal values of <span class="math notranslate nohighlight">\(\lambda_i\)</span>. Before we proceed however, we discuss now the so-called soft classifier.</p>
</section>
<section id="a-soft-classifier">
<h2><span class="section-number">8.3. </span>A soft classifier<a class="headerlink" href="#a-soft-classifier" title="Link to this heading">#</a></h2>
<p>Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.</p>
<p>Suppose now that classes overlap in feature space, as shown in the
figure here. One way to deal with this problem before we define the
so-called <strong>kernel approach</strong>, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.</p>
<p>We introduce thus the so-called <strong>slack</strong> variables <span class="math notranslate nohighlight">\(\boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n]\)</span> and
modify our previous equation</p>
<div class="math notranslate nohighlight">
\[
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
\]</div>
<p>to</p>
<div class="math notranslate nohighlight">
\[
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i,
\]</div>
<p>with the requirement <span class="math notranslate nohighlight">\(\xi_i\geq 0\)</span>. The total violation is now <span class="math notranslate nohighlight">\(\sum_i\xi\)</span>.
The value <span class="math notranslate nohighlight">\(\xi_i\)</span> in the constraint the last constraint corresponds to the amount by which the prediction
<span class="math notranslate nohighlight">\(y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1\)</span> is on the wrong side of its margin. Hence by bounding the sum <span class="math notranslate nohighlight">\(\sum_i \xi_i\)</span>,
we bound the total amount by which predictions fall on the wrong side of their margins.</p>
<p>Misclassifications occur when <span class="math notranslate nohighlight">\(\xi_i &gt; 1\)</span>. Thus bounding the total sum by some value <span class="math notranslate nohighlight">\(C\)</span> bounds in turn the total number of
misclassifications.</p>
<p>This has in turn the consequences that we change our optmization problem to finding the minimum of</p>
<div class="math notranslate nohighlight">
\[
\cal{L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
\]</div>
<p>subject to</p>
<div class="math notranslate nohighlight">
\[
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
\]</div>
<p>with the requirement <span class="math notranslate nohighlight">\(\xi_i\geq 0\)</span>.</p>
<p>Taking the derivatives with respect to <span class="math notranslate nohighlight">\(b\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{w}\)</span> we obtain</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \cal{L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \cal{L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\lambda_i = C-\gamma_i \hspace{0.1cm}\forall i.
\]</div>
<p>Inserting these constraints into the equation for <span class="math notranslate nohighlight">\(\cal{L}\)</span> we obtain the same equation as before</p>
<div class="math notranslate nohighlight">
\[
\cal{L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
\]</div>
<p>but now subject to the constraints <span class="math notranslate nohighlight">\(\lambda_i\geq 0\)</span>, <span class="math notranslate nohighlight">\(\sum_i\lambda_iy_i=0\)</span> and <span class="math notranslate nohighlight">\(0\leq\lambda_i \leq C\)</span>.
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads</p>
<p>5
0</p>
<p>&lt;
&lt;
&lt;
!
!
M
A
T
H
_
B
L
O
C
K</p>
<div class="math notranslate nohighlight">
\[
\gamma_i\xi_i = 0,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
\]</div>
</section>
<section id="kernels-and-non-linearity">
<h2><span class="section-number">8.4. </span>Kernels and non-linearity<a class="headerlink" href="#kernels-and-non-linearity" title="Link to this heading">#</a></h2>
<p>The cases we have studied till now, were all characterized by two classes
with a close to linear separability. The classifiers we have described
so far find linear boundaries in our input feature space. It is
possible to make our procedure more flexible by exploring the feature
space using other basis expansions such as higher-order polynomials,
wavelets, splines etc.</p>
<p>If our feature space is not easy to separate, as shown in the figure
here, we can achieve a better separation by introducing more complex
basis functions. The ideal would be, as shown in the next figure, to, via a specific transformation to
obtain a separation between the classes which is almost linear.</p>
<p>The change of basis, from <span class="math notranslate nohighlight">\(x\rightarrow z=\phi(x)\)</span> leads to the same type of equations to be solved, except that
we need to introduce for example a polynomial transformation to a two-dimensional training set.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
import os
np.random.seed(42)
# To plot pretty figures
import matplotlib
import matplotlib.pyplot as plt
plt.rcParams[&#39;axes.labelsize&#39;] = 14
plt.rcParams[&#39;xtick.labelsize&#39;] = 12
plt.rcParams[&#39;ytick.labelsize&#39;] = 12
from sklearn.svm import SVC
from sklearn import datasets
X1D = np.linspace(-4, 4, 9).reshape(-1, 1)
X2D = np.c_[X1D, X1D**2]
y = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])
plt.figure(figsize=(11, 4))
plt.subplot(121)
plt.grid(True, which=&#39;both&#39;)
plt.axhline(y=0, color=&#39;k&#39;)
plt.plot(X1D[:, 0][y==0], np.zeros(4), &quot;bs&quot;)
plt.plot(X1D[:, 0][y==1], np.zeros(5), &quot;g^&quot;)
plt.gca().get_yaxis().set_ticks([])
plt.xlabel(r&quot;$x_1$&quot;, fontsize=20)
plt.axis([-4.5, 4.5, -0.2, 0.2])
plt.subplot(122)
plt.grid(True, which=&#39;both&#39;)
plt.axhline(y=0, color=&#39;k&#39;)
plt.axvline(x=0, color=&#39;k&#39;)
plt.plot(X2D[:, 0][y==0], X2D[:, 1][y==0], &quot;bs&quot;)
plt.plot(X2D[:, 0][y==1], X2D[:, 1][y==1], &quot;g^&quot;)
plt.xlabel(r&quot;$x_1$&quot;, fontsize=20)
plt.ylabel(r&quot;$x_2$&quot;, fontsize=20, rotation=0)
plt.gca().get_yaxis().set_ticks([0, 4, 8, 12, 16])
plt.plot([-4.5, 4.5], [6.5, 6.5], &quot;r--&quot;, linewidth=3)
plt.axis([-4.5, 4.5, -1, 17])
plt.subplots_adjust(right=1)
plt.show()
</pre></div>
</div>
</div>
</div>
<p>Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with <span class="math notranslate nohighlight">\(x_i\)</span> and <span class="math notranslate nohighlight">\(y_i\)</span> as variables)</p>
<div class="math notranslate nohighlight">
\[
z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
\]</div>
<p>With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)</p>
<div class="math notranslate nohighlight">
\[
\cal{L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{z}_j,
\]</div>
<p>subject to the constraints <span class="math notranslate nohighlight">\(\lambda_i\geq 0\)</span>, <span class="math notranslate nohighlight">\(\sum_i\lambda_iy_i=0\)</span>, and for the support vectors</p>
<div class="math notranslate nohighlight">
\[
y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i,
\]</div>
<p>from which we also find <span class="math notranslate nohighlight">\(b\)</span>.
To compute <span class="math notranslate nohighlight">\(\boldsymbol{z}_i^T\boldsymbol{z}_j\)</span> we define the kernel <span class="math notranslate nohighlight">\(K(\boldsymbol{x}_i,\boldsymbol{x}_j)\)</span> as</p>
<div class="math notranslate nohighlight">
\[
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
\]</div>
<p>For the above example, the kernel reads</p>
<div class="math notranslate nohighlight">
\[\begin{split}
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2.
\end{split}\]</div>
<p>We note that this is nothing but the dot product of the two original
vectors <span class="math notranslate nohighlight">\((\boldsymbol{x}_i^T\boldsymbol{x}_j)^2\)</span>. Instead of thus computing the
product in the Lagrangian of <span class="math notranslate nohighlight">\(\boldsymbol{z}_i^T\boldsymbol{z}_j\)</span> we simply compute
the dot product <span class="math notranslate nohighlight">\((\boldsymbol{x}_i^T\boldsymbol{x}_j)^2\)</span>.</p>
<p>This leads to the so-called
kernel trick and the result leads to the same as if we went through
the trouble of performing the transformation
<span class="math notranslate nohighlight">\(\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j)\)</span> during the SVM calculations.</p>
<p>Using our definition of the kernel We can rewrite again the Lagrangian</p>
<div class="math notranslate nohighlight">
\[
\cal{L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
\]</div>
<p>subject to the constraints <span class="math notranslate nohighlight">\(\lambda_i\geq 0\)</span>, <span class="math notranslate nohighlight">\(\sum_i\lambda_iy_i=0\)</span> in terms of a convex optimization problem</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) &amp; y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) &amp; \dots &amp; \dots &amp; y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) &amp; y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) &amp; \dots &amp; \dots &amp; y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots \\
\dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) &amp; y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) &amp; \dots &amp; \dots &amp; y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
\end{split}\]</div>
<p>subject to <span class="math notranslate nohighlight">\(\boldsymbol{y}^T\boldsymbol{\lambda}=0\)</span>. Here we defined the vectors <span class="math notranslate nohighlight">\(\boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]\)</span> and
<span class="math notranslate nohighlight">\(\boldsymbol{y}=[y_1,y_2,\dots,y_n]\)</span>.
If we add the slack constants this leads to the additional constraint <span class="math notranslate nohighlight">\(0\leq \lambda_i \leq C\)</span>.</p>
<p>We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
&amp;\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
&amp;\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
\end{split}\]</div>
<p>Below we discuss how to solve these equations. Here we note that the matrix <span class="math notranslate nohighlight">\(\boldsymbol{P}\)</span> has matrix elements <span class="math notranslate nohighlight">\(p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j)\)</span>.
Given a kernel <span class="math notranslate nohighlight">\(K\)</span> and the targets <span class="math notranslate nohighlight">\(y_i\)</span> this matrix is easy to set up. The constraint <span class="math notranslate nohighlight">\(\boldsymbol{y}^T\boldsymbol{\lambda}=0\)</span> leads to <span class="math notranslate nohighlight">\(f=0\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{A}=\boldsymbol{y}\)</span>. How to set up the matrix <span class="math notranslate nohighlight">\(\boldsymbol{G}\)</span> is discussed later. Here note that the inequalities <span class="math notranslate nohighlight">\(0\leq \lambda_i \leq C\)</span> can be split up into
<span class="math notranslate nohighlight">\(0\leq \lambda_i\)</span> and <span class="math notranslate nohighlight">\(\lambda_i \leq C\)</span>. These two inequalities define then the matrix <span class="math notranslate nohighlight">\(\boldsymbol{G}\)</span> and the vector <span class="math notranslate nohighlight">\(\boldsymbol{h}\)</span>.</p>
</section>
<section id="different-kernels-and-mercer-s-theorem">
<h2><span class="section-number">8.5. </span>Different kernels and Mercer’s theorem<a class="headerlink" href="#different-kernels-and-mercer-s-theorem" title="Link to this heading">#</a></h2>
<p>There are several popular kernels being used. These are</p>
<ol class="arabic simple">
<li><p>Linear: <span class="math notranslate nohighlight">\(K(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{x}^T\boldsymbol{y}\)</span>,</p></li>
<li><p>Polynomial: <span class="math notranslate nohighlight">\(K(\boldsymbol{x},\boldsymbol{y})=(\boldsymbol{x}^T\boldsymbol{y}+\gamma)^d\)</span>,</p></li>
<li><p>Gaussian Radial Basis Function: <span class="math notranslate nohighlight">\(K(\boldsymbol{x},\boldsymbol{y})=\exp{\left(-\gamma\vert\vert\boldsymbol{x}-\boldsymbol{y}\vert\vert^2\right)}\)</span>,</p></li>
<li><p>Tanh: <span class="math notranslate nohighlight">\(K(\boldsymbol{x},\boldsymbol{y})=\tanh{(\boldsymbol{x}^T\boldsymbol{y}+\gamma)}\)</span>,</p></li>
</ol>
<p>and many other ones.</p>
<p>An important theorem for us is <a class="reference external" href="https://en.wikipedia.org/wiki/Mercer%27s_theorem">Mercer’s
theorem</a>. The
theorem states that if a kernel function <span class="math notranslate nohighlight">\(K\)</span> is symmetric, continuous
and leads to a positive semi-definite matrix <span class="math notranslate nohighlight">\(\boldsymbol{P}\)</span> then there
exists a function <span class="math notranslate nohighlight">\(\phi\)</span> that maps <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{x}_j\)</span> into
another space (possibly with much higher dimensions) such that</p>
<div class="math notranslate nohighlight">
\[
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
\]</div>
<p>So you can use <span class="math notranslate nohighlight">\(K\)</span> as a kernel since you know <span class="math notranslate nohighlight">\(\phi\)</span> exists, even if
you don’t know what <span class="math notranslate nohighlight">\(\phi\)</span> is.</p>
<p>Note that some frequently used kernels (such as the Sigmoid kernel)
don’t respect all of Mercer’s conditions, yet they generally work well
in practice.</p>
</section>
<section id="the-moons-example">
<h2><span class="section-number">8.6. </span>The moons example<a class="headerlink" href="#the-moons-example" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>from __future__ import division, print_function, unicode_literals
import numpy as np
np.random.seed(42)
import matplotlib
import matplotlib.pyplot as plt
plt.rcParams[&#39;axes.labelsize&#39;] = 14
plt.rcParams[&#39;xtick.labelsize&#39;] = 12
plt.rcParams[&#39;ytick.labelsize&#39;] = 12
from sklearn.svm import SVC
from sklearn import datasets
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
from sklearn.datasets import make_moons
X, y = make_moons(n_samples=100, noise=0.15, random_state=42)
def plot_dataset(X, y, axes):
plt.plot(X[:, 0][y==0], X[:, 1][y==0], &quot;bs&quot;)
plt.plot(X[:, 0][y==1], X[:, 1][y==1], &quot;g^&quot;)
plt.axis(axes)
plt.grid(True, which=&#39;both&#39;)
plt.xlabel(r&quot;$x_1$&quot;, fontsize=20)
plt.ylabel(r&quot;$x_2$&quot;, fontsize=20, rotation=0)
plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
plt.show()
from sklearn.datasets import make_moons
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
polynomial_svm_clf = Pipeline([
(&quot;poly_features&quot;, PolynomialFeatures(degree=3)),
(&quot;scaler&quot;, StandardScaler()),
(&quot;svm_clf&quot;, LinearSVC(C=10, loss=&quot;hinge&quot;, random_state=42))
])
polynomial_svm_clf.fit(X, y)
def plot_predictions(clf, axes):
x0s = np.linspace(axes[0], axes[1], 100)
x1s = np.linspace(axes[2], axes[3], 100)
x0, x1 = np.meshgrid(x0s, x1s)
X = np.c_[x0.ravel(), x1.ravel()]
y_pred = clf.predict(X).reshape(x0.shape)
y_decision = clf.decision_function(X).reshape(x0.shape)
plt.contourf(x0, x1, y_pred, cmap=plt.cm.brg, alpha=0.2)
plt.contourf(x0, x1, y_decision, cmap=plt.cm.brg, alpha=0.1)
plot_predictions(polynomial_svm_clf, [-1.5, 2.5, -1, 1.5])
plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
plt.show()
from sklearn.svm import SVC
poly_kernel_svm_clf = Pipeline([
(&quot;scaler&quot;, StandardScaler()),
(&quot;svm_clf&quot;, SVC(kernel=&quot;poly&quot;, degree=3, coef0=1, C=5))
])
poly_kernel_svm_clf.fit(X, y)
poly100_kernel_svm_clf = Pipeline([
(&quot;scaler&quot;, StandardScaler()),
(&quot;svm_clf&quot;, SVC(kernel=&quot;poly&quot;, degree=10, coef0=100, C=5))
])
poly100_kernel_svm_clf.fit(X, y)
plt.figure(figsize=(11, 4))
plt.subplot(121)
plot_predictions(poly_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])
plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
plt.title(r&quot;$d=3, r=1, C=5$&quot;, fontsize=18)
plt.subplot(122)
plot_predictions(poly100_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])
plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
plt.title(r&quot;$d=10, r=100, C=5$&quot;, fontsize=18)
plt.show()
def gaussian_rbf(x, landmark, gamma):
return np.exp(-gamma * np.linalg.norm(x - landmark, axis=1)**2)
gamma = 0.3
x1s = np.linspace(-4.5, 4.5, 200).reshape(-1, 1)
x2s = gaussian_rbf(x1s, -2, gamma)
x3s = gaussian_rbf(x1s, 1, gamma)
XK = np.c_[gaussian_rbf(X1D, -2, gamma), gaussian_rbf(X1D, 1, gamma)]
yk = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])
plt.figure(figsize=(11, 4))
plt.subplot(121)
plt.grid(True, which=&#39;both&#39;)
plt.axhline(y=0, color=&#39;k&#39;)
plt.scatter(x=[-2, 1], y=[0, 0], s=150, alpha=0.5, c=&quot;red&quot;)
plt.plot(X1D[:, 0][yk==0], np.zeros(4), &quot;bs&quot;)
plt.plot(X1D[:, 0][yk==1], np.zeros(5), &quot;g^&quot;)
plt.plot(x1s, x2s, &quot;g--&quot;)
plt.plot(x1s, x3s, &quot;b:&quot;)
plt.gca().get_yaxis().set_ticks([0, 0.25, 0.5, 0.75, 1])
plt.xlabel(r&quot;$x_1$&quot;, fontsize=20)
plt.ylabel(r&quot;Similarity&quot;, fontsize=14)
plt.annotate(r&#39;$\mathbf{x}$&#39;,
xy=(X1D[3, 0], 0),
xytext=(-0.5, 0.20),
ha=&quot;center&quot;,
arrowprops=dict(facecolor=&#39;black&#39;, shrink=0.1),
fontsize=18,
)
plt.text(-2, 0.9, &quot;$x_2$&quot;, ha=&quot;center&quot;, fontsize=20)
plt.text(1, 0.9, &quot;$x_3$&quot;, ha=&quot;center&quot;, fontsize=20)
plt.axis([-4.5, 4.5, -0.1, 1.1])
plt.subplot(122)
plt.grid(True, which=&#39;both&#39;)
plt.axhline(y=0, color=&#39;k&#39;)
plt.axvline(x=0, color=&#39;k&#39;)
plt.plot(XK[:, 0][yk==0], XK[:, 1][yk==0], &quot;bs&quot;)
plt.plot(XK[:, 0][yk==1], XK[:, 1][yk==1], &quot;g^&quot;)
plt.xlabel(r&quot;$x_2$&quot;, fontsize=20)
plt.ylabel(r&quot;$x_3$ &quot;, fontsize=20, rotation=0)
plt.annotate(r&#39;$\phi\left(\mathbf{x}\right)$&#39;,
xy=(XK[3, 0], XK[3, 1]),
xytext=(0.65, 0.50),
ha=&quot;center&quot;,
arrowprops=dict(facecolor=&#39;black&#39;, shrink=0.1),
fontsize=18,
)
plt.plot([-0.1, 1.1], [0.57, -0.1], &quot;r--&quot;, linewidth=3)
plt.axis([-0.1, 1.1, -0.1, 1.1])
plt.subplots_adjust(right=1)
plt.show()
x1_example = X1D[3, 0]
for landmark in (-2, 1):
k = gaussian_rbf(np.array([[x1_example]]), np.array([[landmark]]), gamma)
print(&quot;Phi({}, {}) = {}&quot;.format(x1_example, landmark, k))
rbf_kernel_svm_clf = Pipeline([
(&quot;scaler&quot;, StandardScaler()),
(&quot;svm_clf&quot;, SVC(kernel=&quot;rbf&quot;, gamma=5, C=0.001))
])
rbf_kernel_svm_clf.fit(X, y)
from sklearn.svm import SVC
gamma1, gamma2 = 0.1, 5
C1, C2 = 0.001, 1000
hyperparams = (gamma1, C1), (gamma1, C2), (gamma2, C1), (gamma2, C2)
svm_clfs = []
for gamma, C in hyperparams:
rbf_kernel_svm_clf = Pipeline([
(&quot;scaler&quot;, StandardScaler()),
(&quot;svm_clf&quot;, SVC(kernel=&quot;rbf&quot;, gamma=gamma, C=C))
])
rbf_kernel_svm_clf.fit(X, y)
svm_clfs.append(rbf_kernel_svm_clf)
plt.figure(figsize=(11, 7))
for i, svm_clf in enumerate(svm_clfs):
plt.subplot(221 + i)
plot_predictions(svm_clf, [-1.5, 2.5, -1, 1.5])
plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
gamma, C = hyperparams[i]
plt.title(r&quot;$\gamma = {}, C = {}$&quot;.format(gamma, C), fontsize=16)
plt.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="mathematical-optimization-of-convex-functions">
<h2><span class="section-number">8.7. </span>Mathematical optimization of convex functions<a class="headerlink" href="#mathematical-optimization-of-convex-functions" title="Link to this heading">#</a></h2>
<p>A mathematical (quadratic) optimization problem, or just optimization problem, has the form</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
&amp;\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
&amp;\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
\end{split}\]</div>
<p>subject to some constraints for say a selected set <span class="math notranslate nohighlight">\(i=1,2,\dots, n\)</span>.
In our case we are optimizing with respect to the Lagrangian multipliers <span class="math notranslate nohighlight">\(\lambda_i\)</span>, and the
vector <span class="math notranslate nohighlight">\(\boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n]\)</span> is the optimization variable we are dealing with.</p>
<p>In our case we are particularly interested in a class of optimization problems called convex optmization problems.
In our discussion on gradient descent methods we discussed at length the definition of a convex function.</p>
<p>Convex optimization problems play a central role in applied mathematics and we recommend strongly <a class="reference external" href="http://web.stanford.edu/~boyd/cvxbook/">Boyd and Vandenberghe’s text on the topics</a>.</p>
<p>If we use Python as programming language and wish to venture beyond
<strong>scikit-learn</strong>, <strong>tensorflow</strong> and similar software which makes our
lives so much easier, we need to dive into the wonderful world of
quadratic programming. We can, if we wish, solve the minimization
problem using say standard gradient methods or conjugate gradient
methods. However, these methods tend to exhibit a rather slow
converge. So, welcome to the promised land of quadratic programming.</p>
<p>The functions we need are contained in the quadratic programming package <strong>CVXOPT</strong> and we need to import it together with <strong>numpy</strong> as</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy
import cvxopt
</pre></div>
</div>
</div>
</div>
<p>This will make our life much easier. You don’t need t write your own optimizer.</p>
<p>We remind ourselves about the general problem we want to solve</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
&amp;\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
&amp;\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f.
\end{align*}
\end{split}\]</div>
<p>Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
&amp;\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
&amp;\mathrm{subject to} \\ \nonumber
&amp;x, y \geq 0 \\ \nonumber
&amp;x+3y \geq 15 \\ \nonumber
&amp;2x+5y \leq 100 \\ \nonumber
&amp;3x+4y \leq 80. \\ \nonumber
\end{align*}
\end{split}\]</div>
<p>The minimization problem can be rewritten in terms of vectors and matrices as (with <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> being the unknowns)</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 &amp; 0\\ 0 &amp; 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
\end{split}\]</div>
<p>Similarly, we can now set up the inequalities (we need to change <span class="math notranslate nohighlight">\(\geq\)</span> to <span class="math notranslate nohighlight">\(\leq\)</span> by multiplying with <span class="math notranslate nohighlight">\(-1\)</span> on bot sides) as the following matrix-vector equation</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{bmatrix} -1 &amp; 0 \\ 0 &amp; -1 \\ -1 &amp; -3 \\ 2 &amp; 5 \\ 3 &amp; 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
\end{split}\]</div>
<p>We have collapsed all the inequalities into a single matrix <span class="math notranslate nohighlight">\(\boldsymbol{G}\)</span>. We see also that our matrix</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{P} =\begin{bmatrix} 1 &amp; 0\\ 0 &amp; 0 \end{bmatrix}
\end{split}\]</div>
<p>is clearly positive semi-definite (all eigenvalues larger or equal zero).
Finally, the vector <span class="math notranslate nohighlight">\(\boldsymbol{h}\)</span> is defined as</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
\end{split}\]</div>
<p>Since we don’t have any equalities the matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> is set to zero
The following code solves the equations for us</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Import the necessary packages
import numpy
from cvxopt import matrix
from cvxopt import solvers
P = matrix(numpy.diag([1,0]), tc=’d’)
q = matrix(numpy.array([3,4]), tc=’d’)
G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’)
h = matrix(numpy.array([0,0,-15,100,80]), tc=’d’)
# Construct the QP, invoke solver
sol = solvers.qp(P,q,G,h)
# Extract optimal value and solution
sol[’x’]
sol[’primal objective’]
</pre></div>
</div>
</div>
</div>
<p>We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the <strong>slack</strong> parameter <span class="math notranslate nohighlight">\(C\)</span> we have</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) &amp; y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) &amp; \dots &amp; \dots &amp; y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) &amp; y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) &amp; \dots &amp; \dots &amp; y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots \\
\dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) &amp; y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) &amp; \dots &amp; \dots &amp; y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
\end{split}\]</div>
<p>subject to <span class="math notranslate nohighlight">\(\boldsymbol{y}^T\boldsymbol{\lambda}=0\)</span>. Here we defined the vectors <span class="math notranslate nohighlight">\(\boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]\)</span> and
<span class="math notranslate nohighlight">\(\boldsymbol{y}=[y_1,y_2,\dots,y_n]\)</span>.
With the slack constants this leads to the additional constraint <span class="math notranslate nohighlight">\(0\leq \lambda_i \leq C\)</span>.</p>
<p><strong>code will be added</strong></p>
</section>
</section>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
name: "python3",
path: "./."
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</article>
<footer class="prev-next-footer d-print-none">
<div class="prev-next-area">
<a class="left-prev"
href="chapteroptimization.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
<p class="prev-next-title"><span class="section-number">7. </span>Optimization, the central part of any Machine Learning algortithm</p>
</div>
</a>
<a class="right-next"
href="chapter6.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title"><span class="section-number">9. </span>Decision trees, overarching aims</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
</div>
</footer>
</div>
<div class="bd-sidebar-secondary bd-toc"><div class="sidebar-secondary-items sidebar-secondary__inner">
<div class="sidebar-secondary-item">
<div class="page-toc tocsection onthispage">
<i class="fa-solid fa-list"></i> Contents
</div>
<nav class="bd-toc-nav page-toc">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#hyperplanes-and-all-that">8.1. Hyperplanes and all that</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#the-two-dimensional-case">8.1.1. The two-dimensional case</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#a-better-approach">8.1.2. A better approach</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#a-quick-reminder-on-lagrangian-multipliers">8.2. A quick Reminder on Lagrangian Multipliers</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#a-soft-classifier">8.3. A soft classifier</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#kernels-and-non-linearity">8.4. Kernels and non-linearity</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#different-kernels-and-mercer-s-theorem">8.5. Different kernels and Mercer’s theorem</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-moons-example">8.6. The moons example</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#mathematical-optimization-of-convex-functions">8.7. Mathematical optimization of convex functions</a></li>
</ul>
</nav></div>
</div></div>
</div>
<footer class="bd-footer-content">
<div class="bd-footer-content__inner container">
<div class="footer-item">
<p class="component-author">
By Morten Hjorth-Jensen
</p>
</div>
<div class="footer-item">
<p class="copyright">
© Copyright 2023.
<br/>
</p>
</div>
<div class="footer-item">
</div>
<div class="footer-item">
</div>
</div>
</footer>
</main>
</div>
</div>
<!-- Scripts loaded after <body> so the DOM is not blocked -->
<script src="_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b"></script>
<script src="_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b"></script>
<footer class="bd-footer">
</footer>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink