lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on svm

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2018-11-06 14:28:21 +01:00
parent cb232e1001
commit 81769ba92b
1 changed files with 112 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/SupportVMachines/svm.do.txt
+112 -21
View File
@@ -21,10 +21,13 @@ definition of a _margin_ which separates classes (in case of
classification problems) of variables. It is also used for regression
problems.
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines.
With SVMs we distinguish between hard margin and soft margins. The
latter introduces a so-called softening parameter to be discussed
below. We distinguish also between linear and non-linear
approaches. The latter are the most frequent ones since it is rather
unlikely that we can separate classes easily by say straight lines.
_Note: several figures are missing. They will be added shortly. To run the codes, use the jupyter notebook_
!split
@@ -553,17 +556,17 @@ we need to introduce for example a polynomial transformation to a two-dimensiona
!split
===== The equations =====
Suppose we define a polynomial transformation of degree two (we continue to live in a plane with $x_1$ and $x_2$ as variables)
Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with $x_i$ and $y_i$ as variables)
!bt
\[
z = \phi(x) =\left(1, x_1, x_2, x_1^2, x_2^2, x_1x_2).
z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
\]
!et
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)
!bt
\[
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{z}_i^T\bm{Z}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{z}_j,
\]
!et
subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$, and for the support vectors
@@ -572,30 +575,95 @@ subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$, and for th
y_i(\bm{w}^T\bm{z}_i+b)= 1 \hspace{0.1cm}\forall i,
\]
!et
from which we also find $b$.
from which we also find $b$.
To compute $\bm{z}_i^T\bm{Z}_j$ we define the kerne $K(\bm{x}_i,\bm{x}_j)$ as
!bt
\[
K(\bm{x}_i,\bm{x}_j)=\bm{z}_i^T\bm{Z}_j= \phi(\bm{x}_i)^T\phi(\bm{x}_j).
\]
!et
For the above example, the kernel reads
!bt
\[
K(\bm{x}_i,\bm{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
\]
!et
We note that this is nothing but the dot product of the two original vectors $(\bm{x}_i^T\bm{x}_j)^2$. Instead of thus computing the product in the Lagrangian of $\bm{x}_i^T\bm{z}_j$ we simply compute the dot product $(\bm{x}_i^T\bm{x}_j)^2$. This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation $(\phi(\bm{x}_i)^T\phi(\bm{x}_j)$ during the SVM calculations.
!split
===== Different kernels =====
===== The problem to solve =====
Using our definition of the kernel We can rewrite again the Lagrangian
!bt
\[
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{z}_j,
\]
!et
subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$ in terms of a convex optimization problem
!bt
\[
\frac{1}{2} \bm{\lambda}^T\begin{bmatrix} y_1y_1K(\bm{x}_1,\bm{x}_1) & y_1y_2K(\bm{x}_1,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_1,\bm{x}_n) \\
y_2y_1\bm{x}_2^T\bm{x}_1 & y_2y_2\bm{x}_2^T\bm{x}_2 & \dots & \dots & y_1y_n\bm{x}_2^T\bm{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\bm{x}_n,\bm{x}_1) & y_ny_2K(\bm{x}_n\bm{x}_2) & \dots & \dots & y_ny_nK(\bm{x}_n,\bm{x}_n) \\
\end{bmatrix}\bm{\lambda}-\mathbb{I}\bm{\lambda},
\]
!et
subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and
$\bm{y}=[y_1,y_2,\dots,y_n]$.
If we add the slack constants this leads to the additional constraint $0\leq \lambda_i \leq C$.
We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type
!bt
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\bm{\lambda}^T\bm{P}\bm{\lambda}+\bm{q}^T\bm{\lambda},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \hspace{0.2cm} \bm{A}\bm{\lambda}=f.
\end{align*}
!et
Below we discuss how to solve these equations. Here we note that the matrix $\bm{P} has matrix elements $p_{ij}=y_iy_jK(\bm{x}_i,\bm{x}_j)$.
Given a kernel $K$ and the targets $y_i$ this matrix is easy to set up. The constraint $\bm{y}^T\bm{\lambda}=0$ leads to $f=0$ and $\bm{A}=\bm{y}$. How to set up the matrix $\bm{G}$ is discussed later. Here note that the inequalities $0\leq \lambda_i \leq C$ can be split up into
$0\leq \lambda_i$ and $\lambda_i \leq C$. These two inequalities define then the matrix $\bm{G}$ and the vector $\bm{h}$.
!split
===== Quadratic coefficient matrix =====
===== Different kernels and Mercer's theorem =====
There are several popular kernels being used. These are
o Linear: $K(\bm{x},\bm{y})=\bm{x}^T\bm{y}$,
o Polynomial: $K(\bm{x},\bm{y})=(\bm{x}^T\bm{y}+\gamma)^d$,
o Gaussian Radial Basis Function: $K(\bm{x},\bm{y})=\exp{\left(-\gamma\vert\vert\bm{x}-\bm{y}\vert\vert^2\right)}$,
o Tanh: $K(\bm{x},\bm{y})=\tanh{(\bm{x}^T\bm{y}+\gamma)}$,
and many other ones.
An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
The theorem states that if a kernel function $K$ is symmetric, continuous and leads to a positive definite matrix $\bm{P}$ then
there exists a function $\phi$ that maps $\bm{x}_i$ and $\bm{x}_j$ into another space
(possibly with much higher dimensions) such that
!bt
\[
K(\bm{x}_i,\bm{x}_j)=\phi(\bm{x}_i)^T\phi(\bm{x}_j).
\]
!et
So you can use $K$ as a kernel since you know $\phi$ exists, even if
you don’t know what $\phi$ is.
Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work
well in practice.
!split
===== Mercer's theorem =====
!split
===== Mathematical optimization of convex functions =====
A mathematical optimization problem, or just optimization problem, has the form
A mathematical (quadratic) optimization problem, or just optimization problem, has the form
!bt
\[
\mathrm{minimize}\hspace{0.1cm} f(x),
\]
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\bm{\lambda}^T\bm{P}\bm{\lambda}+\bm{q}^T\bm{\lambda},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \hspace{0.2cm} \bm{A}\bm{\lambda}=f.
\end{align*}
!et
subject to some constraints $g(\lambda_i) \leq b_i$ for say a selected set $i=1,2,\dots, n$.
subject to some constraints for say a selected set $i=1,2,\dots, n$.
In our case we are optimizing with respect to the Lagrangian multipliers $\lambda_i$, and the
vector $\bm{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n]$ is the optimization variable we are dealing with.
and $f(x)$ is our objective function while $g(\lambda_i) \leq b_i$ represents our constraint function.
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
In our disussion on gradient descent methods we discussed at length the definition of a convex function.
@@ -615,19 +683,37 @@ problem using say standard gradient methods or conjugate gradient
methods. However, these methods tend to exhibit a rather slow
converge. So, welcome to the promised land of quadratic programming.
The functions we need are contained in the quadratic programming package _CVXOPT_ and we need to import it
The functions we need are contained in the quadratic programming package _CVXOPT_ and we need to import it together with _numpy_ as
!bc pycod
import numpy
import cvxopt
!ec
Let us first set up the standard form the of quadratic programming (QP) equations by defining the problem as
This will make our life much easier. You don't need t write your own optimizer.
!split
===== A simplex example =====
Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem
!bt
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
&\mathrm{subject to} \\ \nonumber
&x, y \geq 0 \\ \nonumber
&x+3y \geq 15 \\ \nonumber
&2x+5y \leq 100 \\ \nonumber
&3x+4y \leq 80. \\ \nonumber
\end{align*}
!et
The minimization problem can be rewritten in terms of vectors and matrices as (with $x$ and $y$ being the unknowns)
!bt
\[
\mathrm{min}
frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
\]
!et
Similarly, we an now
subject to Gx u. Note that x itself is not provided to the solver, since it is an internal
variable being optimized over. In particular, this means that the solver has no explicit knowledge
of x itself; everything is implicity defined by the supplied parameters. It is essential
@@ -666,3 +752,8 @@ sol = solvers.qp(P,q,G,h)
sol[’x’] # [7.13e-07, 5.00e+00]
sol[’primal objective’]
!ec
!split
===== Multiclass problems and regression with SVMs =====
This material will be added later.