-
diff --git a/doc/pub/svm/html/._svm-bs004.html b/doc/pub/svm/html/._svm-bs004.html index b60f9cb67..749e77be1 100644 --- a/doc/pub/svm/html/._svm-bs004.html +++ b/doc/pub/svm/html/._svm-bs004.html @@ -169,13 +169,13 @@ $$ If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have $$ -b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0, +b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0, $$ if our output \( y_i=1 \). In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if $$ -b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0, +b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0, $$ for the class of observations \( y_i=-1 \), @@ -184,7 +184,7 @@ then \( \boldsymbol{x}_i \) lies on the other side.
Equivalently, for the two classes of observations we have $$ -y_i\left(b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip}\right) > 0. +y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0. $$
diff --git a/doc/pub/svm/html/._svm-bs018.html b/doc/pub/svm/html/._svm-bs018.html index 072d3320b..d79c46dfd 100644 --- a/doc/pub/svm/html/._svm-bs018.html +++ b/doc/pub/svm/html/._svm-bs018.html @@ -171,17 +171,24 @@ y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i, $$ from which we also find \( b \). -To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as +To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as $$ -K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). $$ For the above example, the kernel reads $$ -K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2. +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2. $$ -We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations. +
+We note that this is nothing but the dot product of the two original +vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the +product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute +the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called +kernel trick and the result leads to the same as if we went through +the trouble of performing the transformation +\( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
diff --git a/doc/pub/svm/html/._svm-bs019.html b/doc/pub/svm/html/._svm-bs019.html index a94b303c3..2817079ed 100644 --- a/doc/pub/svm/html/._svm-bs019.html +++ b/doc/pub/svm/html/._svm-bs019.html @@ -160,11 +160,11 @@ $$ subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem $$ \frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ -y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\ +y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ \dots & \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots & \dots \\ y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ -\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda}, +\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, $$ subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and @@ -176,7 +176,7 @@ We can rewrite this (see the solutions below) in terms of a convex optimization $$ \begin{align*} &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber - &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f. + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f. \end{align*} $$ diff --git a/doc/pub/svm/html/._svm-bs020.html b/doc/pub/svm/html/._svm-bs020.html index d6b10f85a..711036856 100644 --- a/doc/pub/svm/html/._svm-bs020.html +++ b/doc/pub/svm/html/._svm-bs020.html @@ -166,7 +166,7 @@ There are several popular kernels being used. These are and many other ones.
-An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem". +An important theorem for us is Mercer's theorem. The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space (possibly with much higher dimensions) such that diff --git a/doc/pub/svm/html/._svm-bs021.html b/doc/pub/svm/html/._svm-bs021.html index 516500727..6bb3920cd 100644 --- a/doc/pub/svm/html/._svm-bs021.html +++ b/doc/pub/svm/html/._svm-bs021.html @@ -158,7 +158,7 @@ A mathematical (quadratic) optimization problem, or just optimization problem, h $$ \begin{align*} &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber - &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f. + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f. \end{align*} $$ @@ -168,7 +168,7 @@ vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the
In our case we are particularly interested in a class of optimization problems called convex optmization problems. -In our disussion on gradient descent methods we discussed at length the definition of a convex function. +In our discussion on gradient descent methods we discussed at length the definition of a convex function.
Convex optimization problems play a central role in applied mathematics and we recommend strongly Boyd and Vandenberghe's text on the topics. diff --git a/doc/pub/svm/html/._svm-bs023.html b/doc/pub/svm/html/._svm-bs023.html index d223dbeb0..a8134b546 100644 --- a/doc/pub/svm/html/._svm-bs023.html +++ b/doc/pub/svm/html/._svm-bs023.html @@ -158,7 +158,7 @@ We remind ourselves about the general problem we want to solve $$ \begin{align*} &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber - &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f. + &\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f. \end{align*} $$ @@ -177,12 +177,12 @@ $$ The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns) $$ -frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}. +\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}. $$ Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation $$ -\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. $$ We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix @@ -198,7 +198,7 @@ $$
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero -The following code does this for us +The following code solves the equations for us
@@ -206,12 +206,6 @@ The following code does this for us import numpy from cvxopt import matrix from cvxopt import solvers -# Define QP parameters (directly) -P = matrix([[1.0,0.0],[0.0,0.0]]) -q = matrix([3.0,4.0]) -G = matrix([[-1.0,0.0,-1.0,2.0,3.0],[0.0,-1.0,-3.0,5.0,4.0]]) -h = matrix([0.0,0.0,-15.0,100.0,80.0]) -# Define QP parameters (with NumPy) P = matrix(numpy.diag([1,0]), tc=’d’) q = matrix(numpy.array([3,4]), tc=’d’) G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’) diff --git a/doc/pub/svm/html/._svm-bs024.html b/doc/pub/svm/html/._svm-bs024.html index 6dcd0583d..aa7fb5b20 100644 --- a/doc/pub/svm/html/._svm-bs024.html +++ b/doc/pub/svm/html/._svm-bs024.html @@ -157,7 +157,7 @@ MathJax.Hub.Config({ We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the slack parameter \( C \) we have $$ \frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ -y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\ +y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ \dots & \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots & \dots \\ y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ diff --git a/doc/pub/svm/html/svm-bs.html b/doc/pub/svm/html/svm-bs.html index e08dfbbfe..d46b684c2 100644 --- a/doc/pub/svm/html/svm-bs.html +++ b/doc/pub/svm/html/svm-bs.html @@ -170,7 +170,7 @@ MathJax.Hub.Config({
-
diff --git a/doc/pub/svm/html/svm-reveal.html b/doc/pub/svm/html/svm-reveal.html index 3916fb6df..e68b85bc0 100644 --- a/doc/pub/svm/html/svm-reveal.html +++ b/doc/pub/svm/html/svm-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({
-
@@ -265,7 +265,7 @@ $$ If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have
$$
-b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
+b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0,
$$
@@ -273,7 +273,7 @@ if our output \( y_i=1 \).
In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if
$$
-b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
+b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0,
$$
@@ -284,7 +284,7 @@ then \( \boldsymbol{x}_i \) lies on the other side.
Equivalently, for the two classes of observations we have
$$
-y_i\left(b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip}\right) > 0.
+y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0.
$$
@@ -858,21 +858,28 @@ $$
from which we also find \( b \).
-To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
+To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
$$
-K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
+K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
$$
For the above example, the kernel reads
$$
-K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
+K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2.
$$
-We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
+
+We note that this is nothing but the dot product of the two original +vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the +product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute +the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called +kernel trick and the result leads to the same as if we went through +the trouble of performing the transformation +\( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations. @@ -889,11 +896,11 @@ subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
-y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
+y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
-\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
+\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
@@ -907,7 +914,7 @@ We can rewrite this (see the solutions below) in terms of a convex optimization
$$
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
- &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
+ &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
$$
@@ -935,7 +942,7 @@ There are several popular kernels being used. These are
and many other ones.
-An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem". +An important theorem for us is Mercer's theorem. The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space (possibly with much higher dimensions) such that @@ -961,7 +968,7 @@ A mathematical (quadratic) optimization problem, or just optimization problem, h $$ \begin{align*} &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber - &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f. + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f. \end{align*} $$
@@ -972,7 +979,7 @@ vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the
In our case we are particularly interested in a class of optimization problems called convex optmization problems. -In our disussion on gradient descent methods we discussed at length the definition of a convex function. +In our discussion on gradient descent methods we discussed at length the definition of a convex function.
Convex optimization problems play a central role in applied mathematics and we recommend strongly Boyd and Vandenberghe's text on the topics. @@ -1014,7 +1021,7 @@ We remind ourselves about the general problem we want to solve $$ \begin{align*} &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber - &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f. + &\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f. \end{align*} $$
@@ -1037,14 +1044,14 @@ $$
The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
$$
-frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
+\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
$$
Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
$$
-\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
+\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
$$
@@ -1065,7 +1072,7 @@ $$
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero -The following code does this for us +The following code solves the equations for us
@@ -1073,12 +1080,6 @@ The following code does this for us import numpy from cvxopt import matrix from cvxopt import solvers -# Define QP parameters (directly) -P = matrix([[1.0,0.0],[0.0,0.0]]) -q = matrix([3.0,4.0]) -G = matrix([[-1.0,0.0,-1.0,2.0,3.0],[0.0,-1.0,-3.0,5.0,4.0]]) -h = matrix([0.0,0.0,-15.0,100.0,80.0]) -# Define QP parameters (with NumPy) P = matrix(numpy.diag([1,0]), tc=’d’) q = matrix(numpy.array([3,4]), tc=’d’) G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’) @@ -1100,7 +1101,7 @@ We are now ready to return to our setup of the optmization problem for a more re
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
-y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
+y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
diff --git a/doc/pub/svm/html/svm-solarized.html b/doc/pub/svm/html/svm-solarized.html
index edb7dc9c0..582cd6470 100644
--- a/doc/pub/svm/html/svm-solarized.html
+++ b/doc/pub/svm/html/svm-solarized.html
@@ -109,7 +109,7 @@ MathJax.Hub.Config({
-
@@ -211,13 +211,13 @@ $$
If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have
$$
-b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
+b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0,
$$
if our output \( y_i=1 \).
In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if
$$
-b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
+b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0,
$$
for the class of observations \( y_i=-1 \),
@@ -226,7 +226,7 @@ then \( \boldsymbol{x}_i \) lies on the other side.
Equivalently, for the two classes of observations we have $$ -y_i\left(b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip}\right) > 0. +y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0. $$
@@ -700,17 +700,24 @@ y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i, $$ from which we also find \( b \). -To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as +To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as $$ -K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). $$ For the above example, the kernel reads $$ -K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2. +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2. $$ -We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations. +
+We note that this is nothing but the dot product of the two original +vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the +product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute +the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called +kernel trick and the result leads to the same as if we went through +the trouble of performing the transformation +\( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
@@ -724,11 +731,11 @@ $$
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
-y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
+y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
-\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
+\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
@@ -740,7 +747,7 @@ We can rewrite this (see the solutions below) in terms of a convex optimization
$$
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
- &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
+ &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
$$
@@ -766,7 +773,7 @@ There are several popular kernels being used. These are
and many other ones.
-An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem". +An important theorem for us is Mercer's theorem. The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space (possibly with much higher dimensions) such that @@ -789,7 +796,7 @@ A mathematical (quadratic) optimization problem, or just optimization problem, h $$ \begin{align*} &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber - &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f. + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f. \end{align*} $$ @@ -799,7 +806,7 @@ vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the
In our case we are particularly interested in a class of optimization problems called convex optmization problems. -In our disussion on gradient descent methods we discussed at length the definition of a convex function. +In our discussion on gradient descent methods we discussed at length the definition of a convex function.
Convex optimization problems play a central role in applied mathematics and we recommend strongly Boyd and Vandenberghe's text on the topics. @@ -840,7 +847,7 @@ We remind ourselves about the general problem we want to solve $$ \begin{align*} &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber - &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f. + &\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f. \end{align*} $$ @@ -859,12 +866,12 @@ $$ The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns) $$ -frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}. +\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}. $$ Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation $$ -\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. $$ We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix @@ -880,7 +887,7 @@ $$
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero -The following code does this for us +The following code solves the equations for us
@@ -888,12 +895,6 @@ The following code does this for us import numpy from cvxopt import matrix from cvxopt import solvers -# Define QP parameters (directly) -P = matrix([[1.0,0.0],[0.0,0.0]]) -q = matrix([3.0,4.0]) -G = matrix([[-1.0,0.0,-1.0,2.0,3.0],[0.0,-1.0,-3.0,5.0,4.0]]) -h = matrix([0.0,0.0,-15.0,100.0,80.0]) -# Define QP parameters (with NumPy) P = matrix(numpy.diag([1,0]), tc=’d’) q = matrix(numpy.array([3,4]), tc=’d’) G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’) @@ -913,7 +914,7 @@ sol[’primal obj We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the slack parameter \( C \) we have $$ \frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ -y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\ +y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ \dots & \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots & \dots \\ y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ diff --git a/doc/pub/svm/html/svm.html b/doc/pub/svm/html/svm.html index 2ef8ae2da..7b4651af8 100644 --- a/doc/pub/svm/html/svm.html +++ b/doc/pub/svm/html/svm.html @@ -114,7 +114,7 @@ MathJax.Hub.Config({
-
@@ -216,13 +216,13 @@ $$
If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have
$$
-b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
+b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0,
$$
if our output \( y_i=1 \).
In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if
$$
-b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
+b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0,
$$
for the class of observations \( y_i=-1 \),
@@ -231,7 +231,7 @@ then \( \boldsymbol{x}_i \) lies on the other side.
Equivalently, for the two classes of observations we have $$ -y_i\left(b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip}\right) > 0. +y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0. $$
@@ -705,17 +705,24 @@ y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i, $$ from which we also find \( b \). -To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as +To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as $$ -K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). $$ For the above example, the kernel reads $$ -K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2. +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2. $$ -We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations. +
+We note that this is nothing but the dot product of the two original +vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the +product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute +the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called +kernel trick and the result leads to the same as if we went through +the trouble of performing the transformation +\( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
@@ -729,11 +736,11 @@ $$
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
-y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
+y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
-\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
+\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
@@ -745,7 +752,7 @@ We can rewrite this (see the solutions below) in terms of a convex optimization
$$
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
- &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
+ &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
$$
@@ -771,7 +778,7 @@ There are several popular kernels being used. These are
and many other ones.
-An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem". +An important theorem for us is Mercer's theorem. The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space (possibly with much higher dimensions) such that @@ -794,7 +801,7 @@ A mathematical (quadratic) optimization problem, or just optimization problem, h $$ \begin{align*} &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber - &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f. + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f. \end{align*} $$ @@ -804,7 +811,7 @@ vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the
In our case we are particularly interested in a class of optimization problems called convex optmization problems. -In our disussion on gradient descent methods we discussed at length the definition of a convex function. +In our discussion on gradient descent methods we discussed at length the definition of a convex function.
Convex optimization problems play a central role in applied mathematics and we recommend strongly Boyd and Vandenberghe's text on the topics. @@ -845,7 +852,7 @@ We remind ourselves about the general problem we want to solve $$ \begin{align*} &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber - &\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f. + &\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f. \end{align*} $$ @@ -864,12 +871,12 @@ $$ The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns) $$ -frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}. +\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}. $$ Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation $$ -\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. $$ We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix @@ -885,7 +892,7 @@ $$
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero -The following code does this for us +The following code solves the equations for us
@@ -893,12 +900,6 @@ The following code does this for us import numpy from cvxopt import matrix from cvxopt import solvers -# Define QP parameters (directly) -P = matrix([[1.0,0.0],[0.0,0.0]]) -q = matrix([3.0,4.0]) -G = matrix([[-1.0,0.0,-1.0,2.0,3.0],[0.0,-1.0,-3.0,5.0,4.0]]) -h = matrix([0.0,0.0,-15.0,100.0,80.0]) -# Define QP parameters (with NumPy) P = matrix(numpy.diag([1,0]), tc=’d’) q = matrix(numpy.array([3,4]), tc=’d’) G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’) @@ -918,7 +919,7 @@ sol[’primal objective’] We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the slack parameter \( C \) we have $$ \frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ -y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\ +y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ \dots & \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots & \dots \\ y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ diff --git a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz index 7e410e360..dc344b8f0 100644 Binary files a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz and b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz differ diff --git a/doc/pub/svm/ipynb/svm.ipynb b/doc/pub/svm/ipynb/svm.ipynb index 0e100566e..cde854a55 100644 --- a/doc/pub/svm/ipynb/svm.ipynb +++ b/doc/pub/svm/ipynb/svm.ipynb @@ -10,7 +10,7 @@ " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Nov 6, 2018**\n", + "Date: **Nov 7, 2018**\n", "\n", "Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", "\n", @@ -144,7 +144,7 @@ "metadata": {}, "source": [ "$$\n", - "b+w_1x_{i1}+w_2x_{i}2+\\dots +w_px_{ip} >0,\n", + "b+w_1x_{i1}+w_2x_{i2}+\\dots +w_px_{ip} >0,\n", "$$" ] }, @@ -161,7 +161,7 @@ "metadata": {}, "source": [ "$$\n", - "b+w_1x_{i1}+w_2x_{i}2+\\dots +w_px_{ip} < 0,\n", + "b+w_1x_{i1}+w_2x_{i2}+\\dots +w_px_{ip} < 0,\n", "$$" ] }, @@ -180,7 +180,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i\\left(b+w_1x_{i1}+w_2x_{i}2+\\dots +w_px_{ip}\\right) > 0.\n", + "y_i\\left(b+w_1x_{i1}+w_2x_{i2}+\\dots +w_px_{ip}\\right) > 0.\n", "$$" ] }, @@ -1132,7 +1132,7 @@ "metadata": {}, "source": [ "from which we also find $b$.\n", - "To compute $\\boldsymbol{z}_i^T\\boldsymbol{Z}_j$ we define the kerne $K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)$ as" + "To compute $\\boldsymbol{z}_i^T\\boldsymbol{z}_j$ we define the kerne $K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)$ as" ] }, { @@ -1140,7 +1140,7 @@ "metadata": {}, "source": [ "$$\n", - "K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=\\boldsymbol{z}_i^T\\boldsymbol{Z}_j= \\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j).\n", + "K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=\\boldsymbol{z}_i^T\\boldsymbol{z}_j= \\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j).\n", "$$" ] }, @@ -1156,7 +1156,7 @@ "metadata": {}, "source": [ "$$\n", - "K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=[x_i^2, y_i^2, \\sqrt{2}x_iy_i]^T\\begin{bmatrix} x_j^2 \\\\ y_j^2 \\\\ \\sqrt{2}x_jy_j \\end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.\n", + "K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=[x_i^2, y_i^2, \\sqrt{2}x_iy_i]^T\\begin{bmatrix} x_j^2 \\\\ y_j^2 \\\\ \\sqrt{2}x_jy_j \\end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2.\n", "$$" ] }, @@ -1164,7 +1164,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We note that this is nothing but the dot product of the two original vectors $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. Instead of thus computing the product in the Lagrangian of $\\boldsymbol{x}_i^T\\boldsymbol{z}_j$ we simply compute the dot product $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation $(\\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j)$ during the SVM calculations. \n", + "We note that this is nothing but the dot product of the two original\n", + "vectors $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. Instead of thus computing the\n", + "product in the Lagrangian of $\\boldsymbol{z}_i^T\\boldsymbol{z}_j$ we simply compute\n", + "the dot product $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. This leads to the so-called\n", + "kernel trick and the result leads to the same as if we went through\n", + "the trouble of performing the transformation\n", + "$(\\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j)$ during the SVM calculations.\n", "\n", "\n", "## The problem to solve\n", @@ -1193,11 +1199,11 @@ "source": [ "$$\n", "\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1K(\\boldsymbol{x}_1,\\boldsymbol{x}_1) & y_1y_2K(\\boldsymbol{x}_1,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_1,\\boldsymbol{x}_n) \\\\\n", - "y_2y_1\\boldsymbol{x}_2^T\\boldsymbol{x}_1 & y_2y_2\\boldsymbol{x}_2^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_2^T\\boldsymbol{x}_n \\\\\n", + "y_2y_1K(\\boldsymbol{x}_2,\\boldsymbol{x}_1) & y_2y_2(\\boldsymbol{x}_2,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_2,\\boldsymbol{x}_n) \\\\\n", "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", "y_ny_1K(\\boldsymbol{x}_n,\\boldsymbol{x}_1) & y_ny_2K(\\boldsymbol{x}_n\\boldsymbol{x}_2) & \\dots & \\dots & y_ny_nK(\\boldsymbol{x}_n,\\boldsymbol{x}_n) \\\\\n", - "\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{I}\\boldsymbol{\\lambda},\n", + "\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{1}\\boldsymbol{\\lambda},\n", "$$" ] }, @@ -1219,7 +1225,7 @@ "$$\n", "\\begin{align*}\n", " &\\mathrm{min}_{\\lambda}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{\\lambda}^T\\boldsymbol{P}\\boldsymbol{\\lambda}+\\boldsymbol{q}^T\\boldsymbol{\\lambda},\\\\ \\nonumber\n", - " &\\mathrm{subject to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n", + " &\\mathrm{subject\\hspace{0.1cm}to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\wedge \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n", "\\end{align*}\n", "$$" ] @@ -1246,7 +1252,7 @@ "\n", "and many other ones.\n", "\n", - "An important theorem for us is \"Mercer's theorem\":https://en.wikipedia.org/wiki/Mercer%27s_theorem\". \n", + "An important theorem for us is [Mercer's theorem](https://en.wikipedia.org/wiki/Mercer%27s_theorem). \n", "The theorem states that if a kernel function $K$ is symmetric, continuous and leads to a positive semi-definite matrix $\\boldsymbol{P}$ then\n", "there exists a function $\\phi$ that maps $\\boldsymbol{x}_i$ and $\\boldsymbol{x}_j$ into another space\n", "(possibly with much higher dimensions) such that" @@ -1283,7 +1289,7 @@ "$$\n", "\\begin{align*}\n", " &\\mathrm{min}_{\\lambda}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{\\lambda}^T\\boldsymbol{P}\\boldsymbol{\\lambda}+\\boldsymbol{q}^T\\boldsymbol{\\lambda},\\\\ \\nonumber\n", - " &\\mathrm{subject to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n", + " &\\mathrm{subject\\hspace{0.1cm}to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\wedge \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n", "\\end{align*}\n", "$$" ] @@ -1297,7 +1303,7 @@ "vector $\\boldsymbol{\\lambda}=[\\lambda_1, \\lambda_2,\\dots, \\lambda_n]$ is the optimization variable we are dealing with.\n", "\n", "In our case we are particularly interested in a class of optimization problems called convex optmization problems. \n", - "In our disussion on gradient descent methods we discussed at length the definition of a convex function. \n", + "In our discussion on gradient descent methods we discussed at length the definition of a convex function. \n", "\n", "Convex optimization problems play a central role in applied mathematics and we recommend strongly [Boyd and Vandenberghe's text on the topics](http://web.stanford.edu/~boyd/cvxbook/).\n", "\n", @@ -1347,7 +1353,7 @@ "$$\n", "\\begin{align*}\n", " &\\mathrm{min}_{x}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{P}\\boldsymbol{x}+\\boldsymbol{q}^T\\boldsymbol{x},\\\\ \\nonumber\n", - " &\\mathrm{subject to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{x} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\boldsymbol{A}\\boldsymbol{x}=f.\n", + " &\\mathrm{subject\\hspace{0.1cm} to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{x} \\preceq \\boldsymbol{h} \\wedge \\boldsymbol{A}\\boldsymbol{x}=f.\n", "\\end{align*}\n", "$$" ] @@ -1387,7 +1393,7 @@ "metadata": {}, "source": [ "$$\n", - "frac{1}{2}\\begin{bmatrix} x\\\\ y \\end{bmatrix}^T \\begin{bmatrix} 1 & 0\\\\ 0 & 0 \\end{bmatrix} \\begin{bmatrix} x \\\\ y \\end{bmatrix} + \\begin{bmatrix}3\\\\ 4 \\end{bmatrix}^T \\begin{bmatrix}x \\\\ y \\end{bmatrix}.\n", + "\\frac{1}{2}\\begin{bmatrix} x\\\\ y \\end{bmatrix}^T \\begin{bmatrix} 1 & 0\\\\ 0 & 0 \\end{bmatrix} \\begin{bmatrix} x \\\\ y \\end{bmatrix} + \\begin{bmatrix}3\\\\ 4 \\end{bmatrix}^T \\begin{bmatrix}x \\\\ y \\end{bmatrix}.\n", "$$" ] }, @@ -1403,7 +1409,7 @@ "metadata": {}, "source": [ "$$\n", - "\\begin{bmatrix} -1 & 0 \\\\ 0 & -1 \\\\ -1 & -3 \\\\ 2 & 5 \\\\ 3 & 4\\end{matrix}\\begin{bmatrix} x \\\\ y\\end{bmatrix} \\preceq \\begin{bmatrix}0 \\\\ 0\\\\ -15 \\\\ 100 \\\\ 80\\end{bmatrix}.\n", + "\\begin{bmatrix} -1 & 0 \\\\ 0 & -1 \\\\ -1 & -3 \\\\ 2 & 5 \\\\ 3 & 4\\end{bmatrix}\\begin{bmatrix} x \\\\ y\\end{bmatrix} \\preceq \\begin{bmatrix}0 \\\\ 0\\\\ -15 \\\\ 100 \\\\ 80\\end{bmatrix}.\n", "$$" ] }, @@ -1445,7 +1451,7 @@ "metadata": {}, "source": [ "Since we don't have any equalities the matrix $\\boldsymbol{A}$ is set to zero\n", - "The following code does this for us" + "The following code solves the equations for us" ] }, { @@ -1460,12 +1466,6 @@ "import numpy\n", "from cvxopt import matrix\n", "from cvxopt import solvers\n", - "# Define QP parameters (directly)\n", - "P = matrix([[1.0,0.0],[0.0,0.0]])\n", - "q = matrix([3.0,4.0])\n", - "G = matrix([[-1.0,0.0,-1.0,2.0,3.0],[0.0,-1.0,-3.0,5.0,4.0]])\n", - "h = matrix([0.0,0.0,-15.0,100.0,80.0])\n", - "# Define QP parameters (with NumPy)\n", "P = matrix(numpy.diag([1,0]), tc=’d’)\n", "q = matrix(numpy.array([3,4]), tc=’d’)\n", "G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’)\n", @@ -1492,7 +1492,7 @@ "source": [ "$$\n", "\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1K(\\boldsymbol{x}_1,\\boldsymbol{x}_1) & y_1y_2K(\\boldsymbol{x}_1,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_1,\\boldsymbol{x}_n) \\\\\n", - "y_2y_1\\boldsymbol{x}_2^T\\boldsymbol{x}_1 & y_2y_2\\boldsymbol{x}_2^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_2^T\\boldsymbol{x}_n \\\\\n", + "y_2y_1K(\\boldsymbol{x}_2,\\boldsymbol{x}_1) & y_2y_2K(\\boldsymbol{x}_2,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_2,\\boldsymbol{x}_n) \\\\\n", "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", "y_ny_1K(\\boldsymbol{x}_n,\\boldsymbol{x}_1) & y_ny_2K(\\boldsymbol{x}_n\\boldsymbol{x}_2) & \\dots & \\dots & y_ny_nK(\\boldsymbol{x}_n,\\boldsymbol{x}_n) \\\\\n", diff --git a/doc/pub/svm/pdf/svm-minted.pdf b/doc/pub/svm/pdf/svm-minted.pdf index f9cf6937c..9cea03dbb 100644 Binary files a/doc/pub/svm/pdf/svm-minted.pdf and b/doc/pub/svm/pdf/svm-minted.pdf differ diff --git a/doc/src/SupportVMachines/svm.do.txt b/doc/src/SupportVMachines/svm.do.txt index 6619b4238..da75a7df4 100644 --- a/doc/src/SupportVMachines/svm.do.txt +++ b/doc/src/SupportVMachines/svm.do.txt @@ -90,14 +90,14 @@ of dimension $n\times p$, where $n$ represents the observations for each feature If the above condition is not met for a given vector $\bm{x}_i$ we have !bt \[ -b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0, +b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0, \] !et if our output $y_i=1$. In this case we say that $\bm{x}_i$ lies on one of the sides of the hyperplane and if !bt \[ -b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0, +b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0, \] !et for the class of observations $y_i=-1$, @@ -106,7 +106,7 @@ then $\bm{x}_i$ lies on the other side. Equivalently, for the two classes of observations we have !bt \[ -y_i\left(b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip}\right) > 0. +y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0. \] !et @@ -576,19 +576,26 @@ y_i(\bm{w}^T\bm{z}_i+b)= 1 \hspace{0.1cm}\forall i, \] !et from which we also find $b$. -To compute $\bm{z}_i^T\bm{Z}_j$ we define the kerne $K(\bm{x}_i,\bm{x}_j)$ as +To compute $\bm{z}_i^T\bm{z}_j$ we define the kerne $K(\bm{x}_i,\bm{x}_j)$ as !bt \[ -K(\bm{x}_i,\bm{x}_j)=\bm{z}_i^T\bm{Z}_j= \phi(\bm{x}_i)^T\phi(\bm{x}_j). +K(\bm{x}_i,\bm{x}_j)=\bm{z}_i^T\bm{z}_j= \phi(\bm{x}_i)^T\phi(\bm{x}_j). \] !et For the above example, the kernel reads !bt \[ -K(\bm{x}_i,\bm{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2. +K(\bm{x}_i,\bm{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2. \] !et -We note that this is nothing but the dot product of the two original vectors $(\bm{x}_i^T\bm{x}_j)^2$. Instead of thus computing the product in the Lagrangian of $\bm{x}_i^T\bm{z}_j$ we simply compute the dot product $(\bm{x}_i^T\bm{x}_j)^2$. This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation $(\phi(\bm{x}_i)^T\phi(\bm{x}_j)$ during the SVM calculations. + +We note that this is nothing but the dot product of the two original +vectors $(\bm{x}_i^T\bm{x}_j)^2$. Instead of thus computing the +product in the Lagrangian of $\bm{z}_i^T\bm{z}_j$ we simply compute +the dot product $(\bm{x}_i^T\bm{x}_j)^2$. This leads to the so-called +kernel trick and the result leads to the same as if we went through +the trouble of performing the transformation +$(\phi(\bm{x}_i)^T\phi(\bm{x}_j)$ during the SVM calculations. !split @@ -603,11 +610,11 @@ subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$ in terms of !bt \[ \frac{1}{2} \bm{\lambda}^T\begin{bmatrix} y_1y_1K(\bm{x}_1,\bm{x}_1) & y_1y_2K(\bm{x}_1,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_1,\bm{x}_n) \\ -y_2y_1\bm{x}_2^T\bm{x}_1 & y_2y_2\bm{x}_2^T\bm{x}_2 & \dots & \dots & y_1y_n\bm{x}_2^T\bm{x}_n \\ +y_2y_1K(\bm{x}_2,\bm{x}_1) & y_2y_2(\bm{x}_2,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_2,\bm{x}_n) \\ \dots & \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots & \dots \\ y_ny_1K(\bm{x}_n,\bm{x}_1) & y_ny_2K(\bm{x}_n\bm{x}_2) & \dots & \dots & y_ny_nK(\bm{x}_n,\bm{x}_n) \\ -\end{bmatrix}\bm{\lambda}-\mathbb{I}\bm{\lambda}, +\end{bmatrix}\bm{\lambda}-\mathbb{1}\bm{\lambda}, \] !et subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and @@ -618,7 +625,7 @@ We can rewrite this (see the solutions below) in terms of a convex optimization !bt \begin{align*} &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\bm{\lambda}^T\bm{P}\bm{\lambda}+\bm{q}^T\bm{\lambda},\\ \nonumber - &\mathrm{subject to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \hspace{0.2cm} \bm{A}\bm{\lambda}=f. + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \hspace{0.2cm} \wedge \bm{A}\bm{\lambda}=f. \end{align*} !et Below we discuss how to solve these equations. Here we note that the matrix $\bm{P}$ has matrix elements $p_{ij}=y_iy_jK(\bm{x}_i,\bm{x}_j)$. @@ -636,7 +643,7 @@ o Gaussian Radial Basis Function: $K(\bm{x},\bm{y})=\exp{\left(-\gamma\vert\vert o Tanh: $K(\bm{x},\bm{y})=\tanh{(\bm{x}^T\bm{y}+\gamma)}$, and many other ones. -An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem". +An important theorem for us is "Mercer's theorem":"https://en.wikipedia.org/wiki/Mercer%27s_theorem". The theorem states that if a kernel function $K$ is symmetric, continuous and leads to a positive semi-definite matrix $\bm{P}$ then there exists a function $\phi$ that maps $\bm{x}_i$ and $\bm{x}_j$ into another space (possibly with much higher dimensions) such that @@ -658,7 +665,7 @@ A mathematical (quadratic) optimization problem, or just optimization problem, h !bt \begin{align*} &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\bm{\lambda}^T\bm{P}\bm{\lambda}+\bm{q}^T\bm{\lambda},\\ \nonumber - &\mathrm{subject to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \hspace{0.2cm} \bm{A}\bm{\lambda}=f. + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \wedge \bm{A}\bm{\lambda}=f. \end{align*} !et subject to some constraints for say a selected set $i=1,2,\dots, n$. @@ -666,7 +673,7 @@ In our case we are optimizing with respect to the Lagrangian multipliers $\lambd vector $\bm{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n]$ is the optimization variable we are dealing with. In our case we are particularly interested in a class of optimization problems called convex optmization problems. -In our disussion on gradient descent methods we discussed at length the definition of a convex function. +In our discussion on gradient descent methods we discussed at length the definition of a convex function. Convex optimization problems play a central role in applied mathematics and we recommend strongly "Boyd and Vandenberghe's text on the topics":"http://web.stanford.edu/~boyd/cvxbook/". @@ -700,7 +707,7 @@ We remind ourselves about the general problem we want to solve !bt \begin{align*} &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\bm{x}^T\bm{P}\bm{x}+\bm{q}^T\bm{x},\\ \nonumber - &\mathrm{subject to} \hspace{0.2cm} \bm{G}\bm{x} \preceq \bm{h} \hspace{0.2cm} \bm{A}\bm{x}=f. + &\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \bm{G}\bm{x} \preceq \bm{h} \wedge \bm{A}\bm{x}=f. \end{align*} !et @@ -718,13 +725,13 @@ Let us show how to perform the optmization using a simple case. Assume we want t The minimization problem can be rewritten in terms of vectors and matrices as (with $x$ and $y$ being the unknowns) !bt \[ -frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}. +\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}. \] !et Similarly, we can now set up the inequalities (we need to change $\geq$ to $\leq$ by multiplying with $-1$ on bot sides) as the following matrix-vector equation !bt \[ -\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. \] !et We have collapsed all the inequalities into a single matrix $\bm{G}$. We see also that our matrix