-With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below. -We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines. +With SVMs we distinguish between hard margin and soft margins. The +latter introduces a so-called softening parameter to be discussed +below. We distinguish also between linear and non-linear +approaches. The latter are the most frequent ones since it is rather +unlikely that we can separate classes easily by say straight lines. + +
+Note: several figures are missing. They will be added shortly. To run the codes, use the jupyter notebook
@@ -186,7 +199,7 @@ We distinguish also between linear and non-linear approaches. The latter are the
-Suppose we define a polynomial transformation of degree two (we continue to live in a plane with \( x_1 \) and \( x_2 \) as variables) +Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables) $$ -z = \phi(x) =\left(1, x_1, x_2, x_1^2, x_2^2, x_1x_2). +z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right). $$
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity) $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{Z}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j, $$ subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors @@ -164,6 +171,17 @@ y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i, $$ from which we also find \( b \). +To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as +$$ +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). +$$ + +For the above example, the kernel reads +$$ +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2. +$$ + +We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
@@ -186,6 +204,8 @@ from which we also find \( b \).
+We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type +$$ +\begin{align*} + &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber + &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f. +\end{align*} +$$ + +Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \). +Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into +\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).
@@ -166,6 +204,8 @@ MathJax.Hub.Config({
+There are several popular kernels being used. These are + +
+An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem". +The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then +there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space +(possibly with much higher dimensions) such that +$$ +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). +$$ + +So you can use \( K \) as a kernel since you know \( \phi \) exists, even if +you don’t know what \( \phi \) is. +Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work +well in practice.
@@ -165,6 +198,8 @@ MathJax.Hub.Config({
- + -
+A mathematical (quadratic) optimization problem, or just optimization problem, has the form +$$ +\begin{align*} + &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber + &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f. +\end{align*} +$$ + +subject to some constraints for say a selected set \( i=1,2,\dots, n \). +In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the +vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with. + +
+In our case we are particularly interested in a class of optimization problems called convex optmization problems. +In our disussion on gradient descent methods we discussed at length the definition of a convex function. + +
+Convex optimization problems play a central role in applied mathematics and we recommend strongly Boyd and Vandenberghe's text on the topics.
@@ -164,6 +191,8 @@ MathJax.Hub.Config({
-A mathematical optimization problem, or just optimization problem, has the form -$$ -\mathrm{minimize}\hspace{0.1cm} f(x), -$$ - -subject to some constraints \( g(\lambda_i) \leq b_i \) for say a selected set \( i=1,2,\dots, n \). -In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the -vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with. -and \( f(x) \) is our objective function while \( g(\lambda_i) \leq b_i \) represents our constraint function. +If we use Python as programming language and wish to venture beyond +scikit-learn, tensorflow and similar software which makes our +lives so much easier, we need to dive into the wonderful world of +quadratic programming. We can, if we wish, solve the minimization +problem using say standard gradient methods or conjugate gradient +methods. However, these methods tend to exhibit a rather slow +converge. So, welcome to the promised land of quadratic programming.
-In our case we are particularly interested in a class of optimization problems called convex optmization problems. -In our disussion on gradient descent methods we discussed at length the definition of a convex function. +The functions we need are contained in the quadratic programming package CVXOPT and we need to import it together with numpy as
-Convex optimization problems play a central role in applied mathematics and we recommend strongly Boyd and Vandenberghe's text on the topics. + + +
import numpy
+import cvxopt
++This will make our life much easier. You don't need t write your own optimizer.
@@ -181,6 +191,8 @@ Convex optimization problems play a central role in applied mathematics and we r
+ + + + +
+We remind ourselves about the general problem we want to solve +$$ +\begin{align*} + &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber + &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f. +\end{align*} +$$ + +
+Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem +$$ +\begin{align*} + &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber + &\mathrm{subject to} \\ \nonumber + &x, y \geq 0 \\ \nonumber + &x+3y \geq 15 \\ \nonumber + &2x+5y \leq 100 \\ \nonumber + &3x+4y \leq 80. \\ \nonumber +\end{align*} +$$ + +The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns) +$$ +frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}. +$$ + +Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation +$$ +\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +$$ + +We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix +$$ +\boldsymbol{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} +$$ + +is clearly positive semi-definite (all eigenvalues larger or equal zero). +Finally, the vector \( \boldsymbol{h} \) is defined as +$$ +\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +$$ + +
+Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero +The following code does this for us +
+ + +
# Import the necessary packages
+import numpy
+from cvxopt import matrix
+from cvxopt import solvers
+# Define QP parameters (directly)
+P = matrix([[1.0,0.0],[0.0,0.0]])
+q = matrix([3.0,4.0])
+G = matrix([[-1.0,0.0,-1.0,2.0,3.0],[0.0,-1.0,-3.0,5.0,4.0]])
+h = matrix([0.0,0.0,-15.0,100.0,80.0])
+# Define QP parameters (with NumPy)
+P = matrix(numpy.diag([1,0]), tc=’d’)
+q = matrix(numpy.array([3,4]), tc=’d’)
+G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’)
+h = matrix(numpy.array([0,0,-15,100,80]), tc=’d’)
+# Construct the QP, invoke solver
+sol = solvers.qp(P,q,G,h)
+# Extract optimal value and solution
+sol[’x’]
+sol[’primal objective’]
++
+ +
+ + +
+ + + + +
+We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the slack parameter \( C \) we have +$$ +\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ +y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\ +\dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots \\ +y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ +\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda}, +$$ + +subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \). + +
+code will be added + +
+
+ +
+ + +
+ + + + +
+ +
+ + +-With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below. -We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines. +With SVMs we distinguish between hard margin and soft margins. The +latter introduces a so-called softening parameter to be discussed +below. We distinguish also between linear and non-linear +approaches. The latter are the most frequent ones since it is rather +unlikely that we can separate classes easily by say straight lines. + +
+Note: several figures are missing. They will be added shortly. To run the codes, use the jupyter notebook @@ -829,10 +835,10 @@ we need to introduce for example a polynomial transformation to a two-dimensiona
-Suppose we define a polynomial transformation of degree two (we continue to live in a plane with \( x_1 \) and \( x_2 \) as variables) +Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
$$
-z = \phi(x) =\left(1, x_1, x_2, x_1^2, x_2^2, x_1x_2).
+z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
$$
@@ -840,7 +846,7 @@ $$
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)
$$
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{Z}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
$$
@@ -852,39 +858,117 @@ $$
from which we also find \( b \).
+To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
+
+$$
+K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
+$$
+
+
+For the above example, the kernel reads
+
+$$
+K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
+$$
+
+
+We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
Different kernels
-
+$$
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
+$$
+
+subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
+
+$$
+\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
+y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
+\dots & \dots & \dots & \dots & \dots \\
+\dots & \dots & \dots & \dots & \dots \\
+y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
+\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
+$$
+
-A mathematical optimization problem, or just optimization problem, has the form
+We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type
+There are several popular kernels being used. These are
+
+
+
+and many other ones.
+
+
+An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
+The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then
+there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space
+(possibly with much higher dimensions) such that
+
+A mathematical (quadratic) optimization problem, or just optimization problem, has the form
+
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
@@ -896,7 +980,7 @@ Convex optimization problems play a central role in applied mathematics and we r
If we use Python as programming language and wish to venture beyond
@@ -908,7 +992,8 @@ methods. However, these methods tend to exhibit a rather slow
converge. So, welcome to the promised land of quadratic programming.
-The functions we need are contained in the quadratic programming package CVXOPT and we need to import it
+The functions we need are contained in the quadratic programming package CVXOPT and we need to import it together with numpy as
+
@@ -916,32 +1001,71 @@ The functions we need are contained in the quadratic programming package CVXO
import cvxopt
-Let us first set up the standard form the of quadratic programming (QP) equations by defining the problem as
+This will make our life much easier. You don't need t write your own optimizer.
+
+We remind ourselves about the general problem we want to solve
-subject to Gx u. Note that x itself is not provided to the solver, since it is an internal
-variable being optimized over. In particular, this means that the solver has no explicit knowledge
-of x itself; everything is implicity defined by the supplied parameters. It is essential
-that the same variable order is maintained for the relevant parameters (e.g., qi
-Non-convexity implies the existence of local optima, making it difficult to find global optima.
+Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem
+
-collapsed all inequality constraints into a single G matrix of the standard form.
-Since there are no equality constraints, we do not need to provide the empty A, b. Note
-that even though y
-2 did not appear in the original objective, we had to include it with zero
-coefficients in P because the solver parameters must be defined using the full set of variables.
-Even if certain variables only appear in constraints, they will still need to be expressed with
-zero coefficients in the objective parameters, and vice versa.
-Let us first define the above parameters in Python. CVXOPT supplies its own matrix
-object; all arguments given to its solvers must be in this matrix type. There are two ways
-to do this. The first is to define the matrix directly with (potentially nested) lists:
-from cvxopt import matrix
+Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero
+The following code does this for us
@@ -962,12 +1086,43 @@ h = matrix(numpy.array([0,# Construct the QP, invoke solver
sol = solvers.qp(P,q,G,h)
# Extract optimal value and solution
-sol[’x’] # [7.13e-07, 5.00e+00]
+sol[’x’]
sol[’primal objective’]
+We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the slack parameter \( C \) we have
+
+code will be added
+
-With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
-We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines.
+With SVMs we distinguish between hard margin and soft margins. The
+latter introduces a so-called softening parameter to be discussed
+below. We distinguish also between linear and non-linear
+approaches. The latter are the most frequent ones since it is rather
+unlikely that we can separate classes easily by say straight lines.
+
+
+Note: several figures are missing. They will be added shortly. To run the codes, use the jupyter notebook
-Suppose we define a polynomial transformation of degree two (we continue to live in a plane with \( x_1 \) and \( x_2 \) as variables)
+Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
$$
-z = \phi(x) =\left(1, x_1, x_2, x_1^2, x_2^2, x_1x_2).
+z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
$$
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)
$$
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{Z}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
$$
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors
@@ -689,37 +700,102 @@ y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i,
$$
from which we also find \( b \).
-
-
-
-
-
-
-
-
-A mathematical optimization problem, or just optimization problem, has the form
+To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
$$
-\mathrm{minimize}\hspace{0.1cm} f(x),
+K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
$$
-subject to some constraints \( g(\lambda_i) \leq b_i \) for say a selected set \( i=1,2,\dots, n \).
+For the above example, the kernel reads
+$$
+K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
+$$
+
+We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
+
+
+
+We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type
+$$
+\begin{align*}
+ &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
+ &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
+\end{align*}
+$$
+
+Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \).
+Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into
+\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).
+
+
+
+There are several popular kernels being used. These are
+
+
+An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
+The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then
+there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space
+(possibly with much higher dimensions) such that
+$$
+K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
+$$
+
+So you can use \( K \) as a kernel since you know \( \phi \) exists, even if
+you don’t know what \( \phi \) is.
+Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work
+well in practice.
+
+
+
+A mathematical (quadratic) optimization problem, or just optimization problem, has the form
+$$
+\begin{align*}
+ &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
+ &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
+\end{align*}
+$$
+
+subject to some constraints for say a selected set \( i=1,2,\dots, n \).
In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the
vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with.
-and \( f(x) \) is our objective function while \( g(\lambda_i) \leq b_i \) represents our constraint function.
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
@@ -731,7 +807,7 @@ Convex optimization problems play a central role in applied mathematics and we r
If we use Python as programming language and wish to venture beyond
@@ -743,7 +819,8 @@ methods. However, these methods tend to exhibit a rather slow
converge. So, welcome to the promised land of quadratic programming.
-The functions we need are contained in the quadratic programming package CVXOPT and we need to import it
+The functions we need are contained in the quadratic programming package CVXOPT and we need to import it together with numpy as
+
@@ -751,30 +828,59 @@ The functions we need are contained in the quadratic programming package CVXO
import cvxopt
-Let us first set up the standard form the of quadratic programming (QP) equations by defining the problem as
+This will make our life much easier. You don't need t write your own optimizer.
+
+
+
+We remind ourselves about the general problem we want to solve
$$
-\mathrm{min}
+\begin{align*}
+ &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
+ &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f.
+\end{align*}
$$
-subject to Gx u. Note that x itself is not provided to the solver, since it is an internal
-variable being optimized over. In particular, this means that the solver has no explicit knowledge
-of x itself; everything is implicity defined by the supplied parameters. It is essential
-that the same variable order is maintained for the relevant parameters (e.g., qi
-Non-convexity implies the existence of local optima, making it difficult to find global optima.
+Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem
+$$
+\begin{align*}
+ &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
+ &\mathrm{subject to} \\ \nonumber
+ &x, y \geq 0 \\ \nonumber
+ &x+3y \geq 15 \\ \nonumber
+ &2x+5y \leq 100 \\ \nonumber
+ &3x+4y \leq 80. \\ \nonumber
+\end{align*}
+$$
+
+The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
+$$
+frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
+$$
+
+Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
+$$
+\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
+$$
+
+We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix
+$$
+\boldsymbol{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix}
+$$
+
+is clearly positive semi-definite (all eigenvalues larger or equal zero).
+Finally, the vector \( \boldsymbol{h} \) is defined as
+$$
+\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
+$$
-collapsed all inequality constraints into a single G matrix of the standard form.
-Since there are no equality constraints, we do not need to provide the empty A, b. Note
-that even though y
-2 did not appear in the original objective, we had to include it with zero
-coefficients in P because the solver parameters must be defined using the full set of variables.
-Even if certain variables only appear in constraints, they will still need to be expressed with
-zero coefficients in the objective parameters, and vice versa.
-Let us first define the above parameters in Python. CVXOPT supplies its own matrix
-object; all arguments given to its solvers must be in this matrix type. There are two ways
-to do this. The first is to define the matrix directly with (potentially nested) lists:
-from cvxopt import matrix
+Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero
+The following code does this for us
@@ -795,10 +901,37 @@ h = matrix(numpy.array([0,# Construct the QP, invoke solver
sol = solvers.qp(P,q,G,h)
# Extract optimal value and solution
-sol[’x’] # [7.13e-07, 5.00e+00]
+sol[’x’]
sol[’primal objective’]
+
+We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the slack parameter \( C \) we have
+$$
+\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
+y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
+\dots & \dots & \dots & \dots & \dots \\
+\dots & \dots & \dots & \dots & \dots \\
+y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
+\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
+$$
+
+subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
+\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
+With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
+
+
+code will be added
+
+
+
-With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
-We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines.
+With SVMs we distinguish between hard margin and soft margins. The
+latter introduces a so-called softening parameter to be discussed
+below. We distinguish also between linear and non-linear
+approaches. The latter are the most frequent ones since it is rather
+unlikely that we can separate classes easily by say straight lines.
+
+
+Note: several figures are missing. They will be added shortly. To run the codes, use the jupyter notebook
-Suppose we define a polynomial transformation of degree two (we continue to live in a plane with \( x_1 \) and \( x_2 \) as variables)
+Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
$$
-z = \phi(x) =\left(1, x_1, x_2, x_1^2, x_2^2, x_1x_2).
+z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
$$
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)
$$
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{Z}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
$$
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors
@@ -694,37 +705,102 @@ y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i,
$$
from which we also find \( b \).
-
-
-
-
-
-
-
-
-A mathematical optimization problem, or just optimization problem, has the form
+To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
$$
-\mathrm{minimize}\hspace{0.1cm} f(x),
+K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
$$
-subject to some constraints \( g(\lambda_i) \leq b_i \) for say a selected set \( i=1,2,\dots, n \).
+For the above example, the kernel reads
+$$
+K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
+$$
+
+We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
+
+
+
+We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type
+$$
+\begin{align*}
+ &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
+ &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
+\end{align*}
+$$
+
+Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \).
+Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into
+\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).
+
+
+
+There are several popular kernels being used. These are
+
+
+An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
+The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then
+there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space
+(possibly with much higher dimensions) such that
+$$
+K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
+$$
+
+So you can use \( K \) as a kernel since you know \( \phi \) exists, even if
+you don’t know what \( \phi \) is.
+Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work
+well in practice.
+
+
+
+A mathematical (quadratic) optimization problem, or just optimization problem, has the form
+$$
+\begin{align*}
+ &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
+ &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
+\end{align*}
+$$
+
+subject to some constraints for say a selected set \( i=1,2,\dots, n \).
In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the
vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with.
-and \( f(x) \) is our objective function while \( g(\lambda_i) \leq b_i \) represents our constraint function.
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
@@ -736,7 +812,7 @@ Convex optimization problems play a central role in applied mathematics and we r
If we use Python as programming language and wish to venture beyond
@@ -748,7 +824,8 @@ methods. However, these methods tend to exhibit a rather slow
converge. So, welcome to the promised land of quadratic programming.
-The functions we need are contained in the quadratic programming package CVXOPT and we need to import it
+The functions we need are contained in the quadratic programming package CVXOPT and we need to import it together with numpy as
+
@@ -756,30 +833,59 @@ The functions we need are contained in the quadratic programming package CVXO
import cvxopt
-Let us first set up the standard form the of quadratic programming (QP) equations by defining the problem as
+This will make our life much easier. You don't need t write your own optimizer.
+
+
+
+We remind ourselves about the general problem we want to solve
$$
-\mathrm{min}
+\begin{align*}
+ &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
+ &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f.
+\end{align*}
$$
-subject to Gx u. Note that x itself is not provided to the solver, since it is an internal
-variable being optimized over. In particular, this means that the solver has no explicit knowledge
-of x itself; everything is implicity defined by the supplied parameters. It is essential
-that the same variable order is maintained for the relevant parameters (e.g., qi
-Non-convexity implies the existence of local optima, making it difficult to find global optima.
+Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem
+$$
+\begin{align*}
+ &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
+ &\mathrm{subject to} \\ \nonumber
+ &x, y \geq 0 \\ \nonumber
+ &x+3y \geq 15 \\ \nonumber
+ &2x+5y \leq 100 \\ \nonumber
+ &3x+4y \leq 80. \\ \nonumber
+\end{align*}
+$$
+
+The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
+$$
+frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
+$$
+
+Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
+$$
+\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
+$$
+
+We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix
+$$
+\boldsymbol{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix}
+$$
+
+is clearly positive semi-definite (all eigenvalues larger or equal zero).
+Finally, the vector \( \boldsymbol{h} \) is defined as
+$$
+\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
+$$
-collapsed all inequality constraints into a single G matrix of the standard form.
-Since there are no equality constraints, we do not need to provide the empty A, b. Note
-that even though y
-2 did not appear in the original objective, we had to include it with zero
-coefficients in P because the solver parameters must be defined using the full set of variables.
-Even if certain variables only appear in constraints, they will still need to be expressed with
-zero coefficients in the objective parameters, and vice versa.
-Let us first define the above parameters in Python. CVXOPT supplies its own matrix
-object; all arguments given to its solvers must be in this matrix type. There are two ways
-to do this. The first is to define the matrix directly with (potentially nested) lists:
-from cvxopt import matrix
+Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero
+The following code does this for us
@@ -800,10 +906,37 @@ h = matrix(numpy# Construct the QP, invoke solver
sol = solvers.qp(P,q,G,h)
# Extract optimal value and solution
-sol[’x’] # [7.13e-07, 5.00e+00]
+sol[’x’]
sol[’primal objective’]
+
+We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the slack parameter \( C \) we have
+$$
+\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
+y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
+\dots & \dots & \dots & \dots & \dots \\
+\dots & \dots & \dots & \dots & \dots \\
+y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
+\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
+$$
+
+subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
+\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
+With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
+
+
+code will be added
+
+
+
-Quadratic coefficient matrix
-Mercer's theorem
-Mathematical optimization of convex functions
+subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
+\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
+If we add the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
$$
-\mathrm{minimize}\hspace{0.1cm} f(x),
+\begin{align*}
+ &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
+ &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
+\end{align*}
$$
-subject to some constraints \( g(\lambda_i) \leq b_i \) for say a selected set \( i=1,2,\dots, n \).
+Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \).
+Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into
+\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).
+Different kernels and Mercer's theorem
+
+
+
+
+$$
+K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
+$$
+
+
+So you can use \( K \) as a kernel since you know \( \phi \) exists, even if
+you don’t know what \( \phi \) is.
+Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work
+well in practice.
+Mathematical optimization of convex functions
+
+
+$$
+\begin{align*}
+ &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
+ &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
+\end{align*}
+$$
+
+
+subject to some constraints for say a selected set \( i=1,2,\dots, n \).
In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the
vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with.
-and \( f(x) \) is our objective function while \( g(\lambda_i) \leq b_i \) represents our constraint function.
How do we solve these problems
+How do we solve these problems?
A simplex example
+
+
$$
-\mathrm{min}
+\begin{align*}
+ &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
+ &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f.
+\end{align*}
$$
+$$
+\begin{align*}
+ &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
+ &\mathrm{subject to} \\ \nonumber
+ &x, y \geq 0 \\ \nonumber
+ &x+3y \geq 15 \\ \nonumber
+ &2x+5y \leq 100 \\ \nonumber
+ &3x+4y \leq 80. \\ \nonumber
+\end{align*}
+$$
+
+
+The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
+
+$$
+frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
+$$
+
+
+Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
+
+$$
+\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
+$$
+
+
+We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix
+
+$$
+\boldsymbol{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix}
+$$
+
+
+is clearly positive semi-definite (all eigenvalues larger or equal zero).
+Finally, the vector \( \boldsymbol{h} \) is defined as
+
+$$
+\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
+$$
+
Back to the more realistic cases
+
+
+$$
+\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
+y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
+\dots & \dots & \dots & \dots & \dots \\
+\dots & \dots & \dots & \dots & \dots \\
+y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
+\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
+$$
+
+
+subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
+\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
+With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
+
+Multiclass problems and regression with SVMs
+This material will be added later.
+
@@ -672,15 +683,15 @@ we need to introduce for example a polynomial transformation to a two-dimensiona
The equations
-
-Different kernels
-
-
-
-Quadratic coefficient matrix
-
-Mercer's theorem
-
-
-
-Mathematical optimization of convex functions
-
-
+
+The problem to solve
+Using our definition of the kernel We can rewrite again the Lagrangian
+$$
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
+$$
+
+subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
+$$
+\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
+y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
+\dots & \dots & \dots & \dots & \dots \\
+\dots & \dots & \dots & \dots & \dots \\
+y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
+\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
+$$
+
+subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
+\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
+If we add the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
+
+
+
+Different kernels and Mercer's theorem
+
+
+
+
+and many other ones.
+
+
+
+Mathematical optimization of convex functions
+
+
-How do we solve these problems
+How do we solve these problems?
+
+A simplex example
+
+
+
+Back to the more realistic cases
+
+
+
+Multiclass problems and regression with SVMs
+This material will be added later.
diff --git a/doc/pub/svm/html/svm.html b/doc/pub/svm/html/svm.html
index 6d99d69b8..2ef8ae2da 100644
--- a/doc/pub/svm/html/svm.html
+++ b/doc/pub/svm/html/svm.html
@@ -60,14 +60,19 @@ div { text-align: justify; text-justify: inter-word; }
('Soft optmization problem', 2, None, '___sec15'),
('Kernels and non-linearity', 2, None, '___sec16'),
('The equations', 2, None, '___sec17'),
- ('Different kernels', 2, None, '___sec18'),
- ('Quadratic coefficient matrix', 2, None, '___sec19'),
- ("Mercer's theorem", 2, None, '___sec20'),
+ ('The problem to solve', 2, None, '___sec18'),
+ ("Different kernels and Mercer's theorem", 2, None, '___sec19'),
('Mathematical optimization of convex functions',
2,
None,
- '___sec21'),
- ('How do we solve these problems', 2, None, '___sec22')]}
+ '___sec20'),
+ ('How do we solve these problems?', 2, None, '___sec21'),
+ ('A simplex example', 2, None, '___sec22'),
+ ('Back to the more realistic cases', 2, None, '___sec23'),
+ ('Multiclass problems and regression with SVMs',
+ 2,
+ None,
+ '___sec24')]}
end of tocinfo -->
@@ -136,8 +141,14 @@ classification problems) of variables. It is also used for regression
problems.
@@ -677,15 +688,15 @@ we need to introduce for example a polynomial transformation to a two-dimensiona
The equations
-
-Different kernels
-
-
-
-Quadratic coefficient matrix
-
-Mercer's theorem
-
-
-
-Mathematical optimization of convex functions
-
-
+
+The problem to solve
+Using our definition of the kernel We can rewrite again the Lagrangian
+$$
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
+$$
+
+subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
+$$
+\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
+y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
+\dots & \dots & \dots & \dots & \dots \\
+\dots & \dots & \dots & \dots & \dots \\
+y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
+\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
+$$
+
+subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
+\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
+If we add the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
+
+
+
+Different kernels and Mercer's theorem
+
+
+
+
+and many other ones.
+
+
+
+Mathematical optimization of convex functions
+
+
-How do we solve these problems
+How do we solve these problems?
+
+A simplex example
+
+
+
+Back to the more realistic cases
+
+
+
+Multiclass problems and regression with SVMs
+This material will be added later.
diff --git a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz
index ff6e50e1a..7e410e360 100644
Binary files a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz and b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz differ
diff --git a/doc/pub/svm/ipynb/svm.ipynb b/doc/pub/svm/ipynb/svm.ipynb
index 33ccaa50c..0e100566e 100644
--- a/doc/pub/svm/ipynb/svm.ipynb
+++ b/doc/pub/svm/ipynb/svm.ipynb
@@ -34,10 +34,13 @@
"classification problems) of variables. It is also used for regression\n",
"problems.\n",
"\n",
- "With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.\n",
- "We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines. \n",
- "\n",
+ "With SVMs we distinguish between hard margin and soft margins. The\n",
+ "latter introduces a so-called softening parameter to be discussed\n",
+ "below. We distinguish also between linear and non-linear\n",
+ "approaches. The latter are the most frequent ones since it is rather\n",
+ "unlikely that we can separate classes easily by say straight lines.\n",
"\n",
+ "**Note: several figures are missing. They will be added shortly. To run the codes, use the jupyter notebook**\n",
"\n",
"\n",
"## Hyperplanes and all that\n",
@@ -1080,7 +1083,7 @@
"\n",
"## The equations\n",
"\n",
- "Suppose we define a polynomial transformation of degree two (we continue to live in a plane with $x_1$ and $x_2$ as variables)"
+ "Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with $x_i$ and $y_i$ as variables)"
]
},
{
@@ -1088,7 +1091,7 @@
"metadata": {},
"source": [
"$$\n",
- "z = \\phi(x) =\\left(1, x_1, x_2, x_1^2, x_2^2, x_1x_2).\n",
+ "z = \\phi(x_i) =\\left(x_i^2, y_i^2, \\sqrt{2}x_iy_i\\right).\n",
"$$"
]
},
@@ -1104,7 +1107,7 @@
"metadata": {},
"source": [
"$$\n",
- "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{z}_i^T\\boldsymbol{Z}_j,\n",
+ "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{z}_j,\n",
"$$"
]
},
@@ -1128,18 +1131,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "from which we also find $b$. \n",
- "\n",
- "## Different kernels\n",
- "\n",
- "## Quadratic coefficient matrix\n",
- "\n",
- "\n",
- "## Mercer's theorem\n",
- "\n",
- "## Mathematical optimization of convex functions\n",
- "\n",
- "A mathematical optimization problem, or just optimization problem, has the form"
+ "from which we also find $b$.\n",
+ "To compute $\\boldsymbol{z}_i^T\\boldsymbol{Z}_j$ we define the kerne $K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)$ as"
]
},
{
@@ -1147,7 +1140,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\mathrm{minimize}\\hspace{0.1cm} f(x),\n",
+ "K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=\\boldsymbol{z}_i^T\\boldsymbol{Z}_j= \\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j).\n",
"$$"
]
},
@@ -1155,10 +1148,153 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "subject to some constraints $g(\\lambda_i) \\leq b_i$ for say a selected set $i=1,2,\\dots, n$.\n",
+ "For the above example, the kernel reads"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=[x_i^2, y_i^2, \\sqrt{2}x_iy_i]^T\\begin{bmatrix} x_j^2 \\\\ y_j^2 \\\\ \\sqrt{2}x_jy_j \\end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We note that this is nothing but the dot product of the two original vectors $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. Instead of thus computing the product in the Lagrangian of $\\boldsymbol{x}_i^T\\boldsymbol{z}_j$ we simply compute the dot product $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation $(\\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j)$ during the SVM calculations. \n",
+ "\n",
+ "\n",
+ "## The problem to solve\n",
+ "Using our definition of the kernel We can rewrite again the Lagrangian"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{z}_j,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "subject to the constraints $\\lambda_i\\geq 0$, $\\sum_i\\lambda_iy_i=0$ in terms of a convex optimization problem"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1K(\\boldsymbol{x}_1,\\boldsymbol{x}_1) & y_1y_2K(\\boldsymbol{x}_1,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_1,\\boldsymbol{x}_n) \\\\\n",
+ "y_2y_1\\boldsymbol{x}_2^T\\boldsymbol{x}_1 & y_2y_2\\boldsymbol{x}_2^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_2^T\\boldsymbol{x}_n \\\\\n",
+ "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
+ "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
+ "y_ny_1K(\\boldsymbol{x}_n,\\boldsymbol{x}_1) & y_ny_2K(\\boldsymbol{x}_n\\boldsymbol{x}_2) & \\dots & \\dots & y_ny_nK(\\boldsymbol{x}_n,\\boldsymbol{x}_n) \\\\\n",
+ "\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{I}\\boldsymbol{\\lambda},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "subject to $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$. Here we defined the vectors $\\boldsymbol{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n",
+ "$\\boldsymbol{y}=[y_1,y_2,\\dots,y_n]$. \n",
+ "If we add the slack constants this leads to the additional constraint $0\\leq \\lambda_i \\leq C$.\n",
+ "\n",
+ "We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\begin{align*}\n",
+ " &\\mathrm{min}_{\\lambda}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{\\lambda}^T\\boldsymbol{P}\\boldsymbol{\\lambda}+\\boldsymbol{q}^T\\boldsymbol{\\lambda},\\\\ \\nonumber\n",
+ " &\\mathrm{subject to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n",
+ "\\end{align*}\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Below we discuss how to solve these equations. Here we note that the matrix $\\boldsymbol{P}$ has matrix elements $p_{ij}=y_iy_jK(\\boldsymbol{x}_i,\\boldsymbol{x}_j)$.\n",
+ "Given a kernel $K$ and the targets $y_i$ this matrix is easy to set up. The constraint $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$ leads to $f=0$ and $\\boldsymbol{A}=\\boldsymbol{y}$. How to set up the matrix $\\boldsymbol{G}$ is discussed later. Here note that the inequalities $0\\leq \\lambda_i \\leq C$ can be split up into\n",
+ "$0\\leq \\lambda_i$ and $\\lambda_i \\leq C$. These two inequalities define then the matrix $\\boldsymbol{G}$ and the vector $\\boldsymbol{h}$.\n",
+ "\n",
+ "\n",
+ "## Different kernels and Mercer's theorem\n",
+ "\n",
+ "There are several popular kernels being used. These are\n",
+ "1. Linear: $K(\\boldsymbol{x},\\boldsymbol{y})=\\boldsymbol{x}^T\\boldsymbol{y}$,\n",
+ "\n",
+ "2. Polynomial: $K(\\boldsymbol{x},\\boldsymbol{y})=(\\boldsymbol{x}^T\\boldsymbol{y}+\\gamma)^d$,\n",
+ "\n",
+ "3. Gaussian Radial Basis Function: $K(\\boldsymbol{x},\\boldsymbol{y})=\\exp{\\left(-\\gamma\\vert\\vert\\boldsymbol{x}-\\boldsymbol{y}\\vert\\vert^2\\right)}$,\n",
+ "\n",
+ "4. Tanh: $K(\\boldsymbol{x},\\boldsymbol{y})=\\tanh{(\\boldsymbol{x}^T\\boldsymbol{y}+\\gamma)}$,\n",
+ "\n",
+ "and many other ones.\n",
+ "\n",
+ "An important theorem for us is \"Mercer's theorem\":https://en.wikipedia.org/wiki/Mercer%27s_theorem\". \n",
+ "The theorem states that if a kernel function $K$ is symmetric, continuous and leads to a positive semi-definite matrix $\\boldsymbol{P}$ then\n",
+ "there exists a function $\\phi$ that maps $\\boldsymbol{x}_i$ and $\\boldsymbol{x}_j$ into another space\n",
+ "(possibly with much higher dimensions) such that"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=\\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j).\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "So you can use $K$ as a kernel since you know $\\phi$ exists, even if\n",
+ "you don’t know what $\\phi$ is. \n",
+ "Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work\n",
+ "well in practice.\n",
+ "\n",
+ "\n",
+ "## Mathematical optimization of convex functions\n",
+ "\n",
+ "A mathematical (quadratic) optimization problem, or just optimization problem, has the form"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\begin{align*}\n",
+ " &\\mathrm{min}_{\\lambda}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{\\lambda}^T\\boldsymbol{P}\\boldsymbol{\\lambda}+\\boldsymbol{q}^T\\boldsymbol{\\lambda},\\\\ \\nonumber\n",
+ " &\\mathrm{subject to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n",
+ "\\end{align*}\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "subject to some constraints for say a selected set $i=1,2,\\dots, n$.\n",
"In our case we are optimizing with respect to the Lagrangian multipliers $\\lambda_i$, and the\n",
"vector $\\boldsymbol{\\lambda}=[\\lambda_1, \\lambda_2,\\dots, \\lambda_n]$ is the optimization variable we are dealing with.\n",
- "and $f(x)$ is our objective function while $g(\\lambda_i) \\leq b_i$ represents our constraint function.\n",
"\n",
"In our case we are particularly interested in a class of optimization problems called convex optmization problems. \n",
"In our disussion on gradient descent methods we discussed at length the definition of a convex function. \n",
@@ -1167,7 +1303,7 @@
"\n",
"\n",
"\n",
- "## How do we solve these problems\n",
+ "## How do we solve these problems?\n",
"\n",
"If we use Python as programming language and wish to venture beyond\n",
"**scikit-learn**, **tensorflow** and similar software which makes our\n",
@@ -1177,7 +1313,7 @@
"methods. However, these methods tend to exhibit a rather slow\n",
"converge. So, welcome to the promised land of quadratic programming.\n",
"\n",
- "The functions we need are contained in the quadratic programming package **CVXOPT** and we need to import it"
+ "The functions we need are contained in the quadratic programming package **CVXOPT** and we need to import it together with **numpy** as"
]
},
{
@@ -1196,7 +1332,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Let us first set up the standard form the of quadratic programming (QP) equations by defining the problem as"
+ "This will make our life much easier. You don't need t write your own optimizer.\n",
+ "\n",
+ "\n",
+ "## A simplex example\n",
+ "\n",
+ "We remind ourselves about the general problem we want to solve"
]
},
{
@@ -1204,7 +1345,10 @@
"metadata": {},
"source": [
"$$\n",
- "\\mathrm{min}\n",
+ "\\begin{align*}\n",
+ " &\\mathrm{min}_{x}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{P}\\boldsymbol{x}+\\boldsymbol{q}^T\\boldsymbol{x},\\\\ \\nonumber\n",
+ " &\\mathrm{subject to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{x} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\boldsymbol{A}\\boldsymbol{x}=f.\n",
+ "\\end{align*}\n",
"$$"
]
},
@@ -1212,23 +1356,96 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "subject to Gx u. Note that x itself is not provided to the solver, since it is an internal\n",
- "variable being optimized over. In particular, this means that the solver has no explicit knowledge\n",
- "of x itself; everything is implicity defined by the supplied parameters. It is essential\n",
- "that the same variable order is maintained for the relevant parameters (e.g., qi\n",
- "Non-convexity implies the existence of local optima, making it difficult to find global optima.\n",
- "\n",
- "collapsed all inequality constraints into a single G matrix of the standard form.\n",
- "Since there are no equality constraints, we do not need to provide the empty A, b. Note\n",
- "that even though y\n",
- "2 did not appear in the original objective, we had to include it with zero\n",
- "coefficients in P because the solver parameters must be defined using the full set of variables.\n",
- "Even if certain variables only appear in constraints, they will still need to be expressed with\n",
- "zero coefficients in the objective parameters, and vice versa.\n",
- "Let us first define the above parameters in Python. CVXOPT supplies its own matrix\n",
- "object; all arguments given to its solvers must be in this matrix type. There are two ways\n",
- "to do this. The first is to define the matrix directly with (potentially nested) lists:\n",
- "from cvxopt import matrix"
+ "Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\begin{align*}\n",
+ " &\\mathrm{min}_{x}\\hspace{0.2cm} \\frac{1}{2}x^2+5x+3y \\\\ \\nonumber\n",
+ " &\\mathrm{subject to} \\\\ \\nonumber\n",
+ " &x, y \\geq 0 \\\\ \\nonumber\n",
+ " &x+3y \\geq 15 \\\\ \\nonumber\n",
+ " &2x+5y \\leq 100 \\\\ \\nonumber\n",
+ " &3x+4y \\leq 80. \\\\ \\nonumber\n",
+ "\\end{align*}\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The minimization problem can be rewritten in terms of vectors and matrices as (with $x$ and $y$ being the unknowns)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "frac{1}{2}\\begin{bmatrix} x\\\\ y \\end{bmatrix}^T \\begin{bmatrix} 1 & 0\\\\ 0 & 0 \\end{bmatrix} \\begin{bmatrix} x \\\\ y \\end{bmatrix} + \\begin{bmatrix}3\\\\ 4 \\end{bmatrix}^T \\begin{bmatrix}x \\\\ y \\end{bmatrix}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Similarly, we can now set up the inequalities (we need to change $\\geq$ to $\\leq$ by multiplying with $-1$ on bot sides) as the following matrix-vector equation"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\begin{bmatrix} -1 & 0 \\\\ 0 & -1 \\\\ -1 & -3 \\\\ 2 & 5 \\\\ 3 & 4\\end{matrix}\\begin{bmatrix} x \\\\ y\\end{bmatrix} \\preceq \\begin{bmatrix}0 \\\\ 0\\\\ -15 \\\\ 100 \\\\ 80\\end{bmatrix}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We have collapsed all the inequalities into a single matrix $\\boldsymbol{G}$. We see also that our matrix"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{P} =\\begin{bmatrix} 1 & 0\\\\ 0 & 0 \\end{bmatrix}\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "is clearly positive semi-definite (all eigenvalues larger or equal zero). \n",
+ "Finally, the vector $\\boldsymbol{h}$ is defined as"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{h} = \\begin{bmatrix}0 \\\\ 0\\\\ -15 \\\\ 100 \\\\ 80\\end{bmatrix}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Since we don't have any equalities the matrix $\\boldsymbol{A}$ is set to zero\n",
+ "The following code does this for us"
]
},
{
@@ -1256,9 +1473,47 @@
"# Construct the QP, invoke solver\n",
"sol = solvers.qp(P,q,G,h)\n",
"# Extract optimal value and solution\n",
- "sol[’x’] # [7.13e-07, 5.00e+00]\n",
+ "sol[’x’] \n",
"sol[’primal objective’]"
]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Back to the more realistic cases\n",
+ "\n",
+ "We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the **slack** parameter $C$ we have"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1K(\\boldsymbol{x}_1,\\boldsymbol{x}_1) & y_1y_2K(\\boldsymbol{x}_1,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_1,\\boldsymbol{x}_n) \\\\\n",
+ "y_2y_1\\boldsymbol{x}_2^T\\boldsymbol{x}_1 & y_2y_2\\boldsymbol{x}_2^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_2^T\\boldsymbol{x}_n \\\\\n",
+ "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
+ "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
+ "y_ny_1K(\\boldsymbol{x}_n,\\boldsymbol{x}_1) & y_ny_2K(\\boldsymbol{x}_n\\boldsymbol{x}_2) & \\dots & \\dots & y_ny_nK(\\boldsymbol{x}_n,\\boldsymbol{x}_n) \\\\\n",
+ "\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{I}\\boldsymbol{\\lambda},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "subject to $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$. Here we defined the vectors $\\boldsymbol{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n",
+ "$\\boldsymbol{y}=[y_1,y_2,\\dots,y_n]$. \n",
+ "With the slack constants this leads to the additional constraint $0\\leq \\lambda_i \\leq C$.\n",
+ "\n",
+ "**code will be added**\n",
+ "\n",
+ "\n",
+ "## Multiclass problems and regression with SVMs\n",
+ "This material will be added later."
+ ]
}
],
"metadata": {},
diff --git a/doc/pub/svm/pdf/svm-minted.pdf b/doc/pub/svm/pdf/svm-minted.pdf
index 226be3e02..f9cf6937c 100644
Binary files a/doc/pub/svm/pdf/svm-minted.pdf and b/doc/pub/svm/pdf/svm-minted.pdf differ
diff --git a/doc/src/SupportVMachines/svm.do.txt b/doc/src/SupportVMachines/svm.do.txt
index a645290ed..17b9a8b0b 100644
--- a/doc/src/SupportVMachines/svm.do.txt
+++ b/doc/src/SupportVMachines/svm.do.txt
@@ -621,7 +621,7 @@ We can rewrite this (see the solutions below) in terms of a convex optimization
&\mathrm{subject to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \hspace{0.2cm} \bm{A}\bm{\lambda}=f.
\end{align*}
!et
-Below we discuss how to solve these equations. Here we note that the matrix $\bm{P} has matrix elements $p_{ij}=y_iy_jK(\bm{x}_i,\bm{x}_j)$.
+Below we discuss how to solve these equations. Here we note that the matrix $\bm{P}$ has matrix elements $p_{ij}=y_iy_jK(\bm{x}_i,\bm{x}_j)$.
Given a kernel $K$ and the targets $y_i$ this matrix is easy to set up. The constraint $\bm{y}^T\bm{\lambda}=0$ leads to $f=0$ and $\bm{A}=\bm{y}$. How to set up the matrix $\bm{G}$ is discussed later. Here note that the inequalities $0\leq \lambda_i \leq C$ can be split up into
$0\leq \lambda_i$ and $\lambda_i \leq C$. These two inequalities define then the matrix $\bm{G}$ and the vector $\bm{h}$.
@@ -637,7 +637,7 @@ o Tanh: $K(\bm{x},\bm{y})=\tanh{(\bm{x}^T\bm{y}+\gamma)}$,
and many other ones.
An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
-The theorem states that if a kernel function $K$ is symmetric, continuous and leads to a positive definite matrix $\bm{P}$ then
+The theorem states that if a kernel function $K$ is symmetric, continuous and leads to a positive semi-definite matrix $\bm{P}$ then
there exists a function $\phi$ that maps $\bm{x}_i$ and $\bm{x}_j$ into another space
(possibly with much higher dimensions) such that
!bt
@@ -673,7 +673,7 @@ Convex optimization problems play a central role in applied mathematics and we r
!split
-===== How do we solve these problems =====
+===== How do we solve these problems? =====
If we use Python as programming language and wish to venture beyond
_scikit-learn_, _tensorflow_ and similar software which makes our
@@ -696,10 +696,18 @@ This will make our life much easier. You don't need t write your own optimizer.
!split
===== A simplex example =====
+We remind ourselves about the general problem we want to solve
+!bt
+\begin{align*}
+ &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\bm{x}^T\bm{P}\bm{x}+\bm{q}^T\bm{x},\\ \nonumber
+ &\mathrm{subject to} \hspace{0.2cm} \bm{G}\bm{x} \preceq \bm{h} \hspace{0.2cm} \bm{A}\bm{x}=f.
+\end{align*}
+!et
+
Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem
!bt
\begin{align*}
- &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
+ &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
&\mathrm{subject to} \\ \nonumber
&x, y \geq 0 \\ \nonumber
&x+3y \geq 15 \\ \nonumber
@@ -713,24 +721,29 @@ The minimization problem can be rewritten in terms of vectors and matrices as (w
frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
\]
!et
-Similarly, we an now
-subject to Gx u. Note that x itself is not provided to the solver, since it is an internal
-variable being optimized over. In particular, this means that the solver has no explicit knowledge
-of x itself; everything is implicity defined by the supplied parameters. It is essential
-that the same variable order is maintained for the relevant parameters (e.g., qi
-Non-convexity implies the existence of local optima, making it difficult to find global optima.
+Similarly, we can now set up the inequalities (we need to change $\geq$ to $\leq$ by multiplying with $-1$ on bot sides) as the following matrix-vector equation
+!bt
+\[
+\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
+\]
+!et
+We have collapsed all the inequalities into a single matrix $\bm{G}$. We see also that our matrix
+!bt
+\[
+\bm{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix}
+\]
+!et
+is clearly positive semi-definite (all eigenvalues larger or equal zero).
+Finally, the vector $\bm{h}$ is defined as
+!bt
+\[
+\bm{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
+\]
+!et
-collapsed all inequality constraints into a single G matrix of the standard form.
-Since there are no equality constraints, we do not need to provide the empty A, b. Note
-that even though y
-2 did not appear in the original objective, we had to include it with zero
-coefficients in P because the solver parameters must be defined using the full set of variables.
-Even if certain variables only appear in constraints, they will still need to be expressed with
-zero coefficients in the objective parameters, and vice versa.
-Let us first define the above parameters in Python. CVXOPT supplies its own matrix
-object; all arguments given to its solvers must be in this matrix type. There are two ways
-to do this. The first is to define the matrix directly with (potentially nested) lists:
-from cvxopt import matrix
+
+Since we don't have any equalities the matrix $\bm{A}$ is set to zero
+The following code does this for us
!bc pycod
# Import the necessary packages
import numpy
@@ -749,10 +762,30 @@ h = matrix(numpy.array([0,0,-15,100,80]), tc=’d’)
# Construct the QP, invoke solver
sol = solvers.qp(P,q,G,h)
# Extract optimal value and solution
-sol[’x’] # [7.13e-07, 5.00e+00]
+sol[’x’]
sol[’primal objective’]
!ec
+!split
+===== Back to the more realistic cases =====
+
+We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the _slack_ parameter $C$ we have
+!bt
+\[
+\frac{1}{2} \bm{\lambda}^T\begin{bmatrix} y_1y_1K(\bm{x}_1,\bm{x}_1) & y_1y_2K(\bm{x}_1,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_1,\bm{x}_n) \\
+y_2y_1\bm{x}_2^T\bm{x}_1 & y_2y_2\bm{x}_2^T\bm{x}_2 & \dots & \dots & y_1y_n\bm{x}_2^T\bm{x}_n \\
+\dots & \dots & \dots & \dots & \dots \\
+\dots & \dots & \dots & \dots & \dots \\
+y_ny_1K(\bm{x}_n,\bm{x}_1) & y_ny_2K(\bm{x}_n\bm{x}_2) & \dots & \dots & y_ny_nK(\bm{x}_n,\bm{x}_n) \\
+\end{bmatrix}\bm{\lambda}-\mathbb{I}\bm{\lambda},
+\]
+!et
+subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and
+$\bm{y}=[y_1,y_2,\dots,y_n]$.
+With the slack constants this leads to the additional constraint $0\leq \lambda_i \leq C$.
+
+_code will be added_
+
!split
===== Multiclass problems and regression with SVMs =====