lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Corrected typos in sim

Browse Source
This commit is contained in:
mhjensen
2018-11-07 07:35:26 +01:00
parent 9570bc4cbe
commit 6ffdb733bd
16 changed files with 153 additions and 142 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/svm/html/._svm-bs000.html
+1 -1
View File
@@ -170,7 +170,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 6, 2018</h4></center> <!-- date -->
<center><h4>Nov 7, 2018</h4></center> <!-- date -->
<br>
<p>
doc/pub/svm/html/._svm-bs004.html
+3 -3
View File
@@ -169,13 +169,13 @@ $$
If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0,
$$
if our output \( y_i=1 \).
In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0,
$$
for the class of observations \( y_i=-1 \),
@@ -184,7 +184,7 @@ then \( \boldsymbol{x}_i \) lies on the other side.
<p>
Equivalently, for the two classes of observations we have
$$
y_i\left(b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip}\right) > 0.
y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0.
$$
<p>
doc/pub/svm/html/._svm-bs018.html
+11 -4
View File
@@ -171,17 +171,24 @@ y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i,
$$
from which we also find \( b \).
To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
$$
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
$$
For the above example, the kernel reads
$$
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2.
$$
We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
<p>
We note that this is nothing but the dot product of the two original
vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the
product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute
the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called
kernel trick and the result leads to the same as if we went through
the trouble of performing the transformation
\( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
<p>
<p>
doc/pub/svm/html/._svm-bs019.html
+3 -3
View File
@@ -160,11 +160,11 @@ $$
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
@@ -176,7 +176,7 @@ We can rewrite this (see the solutions below) in terms of a convex optimization
$$
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
$$
doc/pub/svm/html/._svm-bs020.html
+1 -1
View File
@@ -166,7 +166,7 @@ There are several popular kernels being used. These are
and many other ones.
<p>
An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
An important theorem for us is <a href="https://en.wikipedia.org/wiki/Mercer%27s_theorem" target="_self">Mercer's theorem</a>.
The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then
there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space
(possibly with much higher dimensions) such that
doc/pub/svm/html/._svm-bs021.html
+2 -2
View File
@@ -158,7 +158,7 @@ A mathematical (quadratic) optimization problem, or just optimization problem, h
$$
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
$$
@@ -168,7 +168,7 @@ vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the
<p>
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
In our disussion on gradient descent methods we discussed at length the definition of a convex function.
In our discussion on gradient descent methods we discussed at length the definition of a convex function.
<p>
Convex optimization problems play a central role in applied mathematics and we recommend strongly <a href="http://web.stanford.edu/~boyd/cvxbook/" target="_self">Boyd and Vandenberghe's text on the topics</a>.
doc/pub/svm/html/._svm-bs023.html
+4 -10
View File
@@ -158,7 +158,7 @@ We remind ourselves about the general problem we want to solve
$$
\begin{align*}
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f.
&\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f.
\end{align*}
$$
@@ -177,12 +177,12 @@ $$
The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
$$
frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
$$
Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
$$
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
$$
We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix
@@ -198,7 +198,7 @@ $$
<p>
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero
The following code does this for us
The following code solves the equations for us
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -206,12 +206,6 @@ The following code does this for us
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span> <span style="color: #008000; font-weight: bold">import</span> matrix
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span> <span style="color: #008000; font-weight: bold">import</span> solvers
<span style="color: #408080; font-style: italic"># Define QP parameters (directly)</span>
P <span style="color: #666666">=</span> matrix([[<span style="color: #666666">1.0</span>,<span style="color: #666666">0.0</span>],[<span style="color: #666666">0.0</span>,<span style="color: #666666">0.0</span>]])
q <span style="color: #666666">=</span> matrix([<span style="color: #666666">3.0</span>,<span style="color: #666666">4.0</span>])
G <span style="color: #666666">=</span> matrix([[<span style="color: #666666">-1.0</span>,<span style="color: #666666">0.0</span>,<span style="color: #666666">-1.0</span>,<span style="color: #666666">2.0</span>,<span style="color: #666666">3.0</span>],[<span style="color: #666666">0.0</span>,<span style="color: #666666">-1.0</span>,<span style="color: #666666">-3.0</span>,<span style="color: #666666">5.0</span>,<span style="color: #666666">4.0</span>]])
h <span style="color: #666666">=</span> matrix([<span style="color: #666666">0.0</span>,<span style="color: #666666">0.0</span>,<span style="color: #666666">-15.0</span>,<span style="color: #666666">100.0</span>,<span style="color: #666666">80.0</span>])
<span style="color: #408080; font-style: italic"># Define QP parameters (with NumPy)</span>
P <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>diag([<span style="color: #666666">1</span>,<span style="color: #666666">0</span>]), tc<span style="color: #666666">=</span>’d’)
q <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>array([<span style="color: #666666">3</span>,<span style="color: #666666">4</span>]), tc<span style="color: #666666">=</span>’d’)
G <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>array([[<span style="color: #666666">-1</span>,<span style="color: #666666">0</span>],[<span style="color: #666666">0</span>,<span style="color: #666666">-1</span>],[<span style="color: #666666">-1</span>,<span style="color: #666666">-3</span>],[<span style="color: #666666">2</span>,<span style="color: #666666">5</span>],[<span style="color: #666666">3</span>,<span style="color: #666666">4</span>]]), tc<span style="color: #666666">=</span>’d’)
doc/pub/svm/html/._svm-bs024.html
+1 -1
View File
@@ -157,7 +157,7 @@ MathJax.Hub.Config({
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the <b>slack</b> parameter \( C \) we have
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
doc/pub/svm/html/svm-bs.html
+1 -1
View File
@@ -170,7 +170,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 6, 2018</h4></center> <!-- date -->
<center><h4>Nov 7, 2018</h4></center> <!-- date -->
<br>
<p>
doc/pub/svm/html/svm-reveal.html
+26 -25
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Nov 6, 2018</h4></center> <!-- date -->
<center><h4>Nov 7, 2018</h4></center> <!-- date -->
<br>
<p>
@@ -265,7 +265,7 @@ $$
If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have
<p>&nbsp;<br>
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0,
$$
<p>&nbsp;<br>
@@ -273,7 +273,7 @@ if our output \( y_i=1 \).
In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if
<p>&nbsp;<br>
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0,
$$
<p>&nbsp;<br>
@@ -284,7 +284,7 @@ then \( \boldsymbol{x}_i \) lies on the other side.
Equivalently, for the two classes of observations we have
<p>&nbsp;<br>
$$
y_i\left(b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip}\right) > 0.
y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0.
$$
<p>&nbsp;<br>
@@ -858,21 +858,28 @@ $$
<p>&nbsp;<br>
from which we also find \( b \).
To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
<p>&nbsp;<br>
$$
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
$$
<p>&nbsp;<br>
For the above example, the kernel reads
<p>&nbsp;<br>
$$
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2.
$$
<p>&nbsp;<br>
We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
<p>
We note that this is nothing but the dot product of the two original
vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the
product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute
the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called
kernel trick and the result leads to the same as if we went through
the trouble of performing the transformation
\( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
</section>
@@ -889,11 +896,11 @@ subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in
<p>&nbsp;<br>
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
<p>&nbsp;<br>
@@ -907,7 +914,7 @@ We can rewrite this (see the solutions below) in terms of a convex optimization
$$
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
$$
<p>&nbsp;<br>
@@ -935,7 +942,7 @@ There are several popular kernels being used. These are
and many other ones.
<p>
An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
An important theorem for us is <a href="https://en.wikipedia.org/wiki/Mercer%27s_theorem" target="_blank">Mercer's theorem</a>.
The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then
there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space
(possibly with much higher dimensions) such that
@@ -961,7 +968,7 @@ A mathematical (quadratic) optimization problem, or just optimization problem, h
$$
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
$$
<p>&nbsp;<br>
@@ -972,7 +979,7 @@ vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the
<p>
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
In our disussion on gradient descent methods we discussed at length the definition of a convex function.
In our discussion on gradient descent methods we discussed at length the definition of a convex function.
<p>
Convex optimization problems play a central role in applied mathematics and we recommend strongly <a href="http://web.stanford.edu/~boyd/cvxbook/" target="_blank">Boyd and Vandenberghe's text on the topics</a>.
@@ -1014,7 +1021,7 @@ We remind ourselves about the general problem we want to solve
$$
\begin{align*}
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f.
&\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f.
\end{align*}
$$
<p>&nbsp;<br>
@@ -1037,14 +1044,14 @@ $$
The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
<p>&nbsp;<br>
$$
frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
$$
<p>&nbsp;<br>
Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
<p>&nbsp;<br>
$$
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
$$
<p>&nbsp;<br>
@@ -1065,7 +1072,7 @@ $$
<p>
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero
The following code does this for us
The following code solves the equations for us
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -1073,12 +1080,6 @@ The following code does this for us
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">cvxopt</span> <span style="color: #8B008B; font-weight: bold">import</span> matrix
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">cvxopt</span> <span style="color: #8B008B; font-weight: bold">import</span> solvers
<span style="color: #228B22"># Define QP parameters (directly)</span>
P = matrix([[<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">0.0</span>],[<span style="color: #B452CD">0.0</span>,<span style="color: #B452CD">0.0</span>]])
q = matrix([<span style="color: #B452CD">3.0</span>,<span style="color: #B452CD">4.0</span>])
G = matrix([[-<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">0.0</span>,-<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">2.0</span>,<span style="color: #B452CD">3.0</span>],[<span style="color: #B452CD">0.0</span>,-<span style="color: #B452CD">1.0</span>,-<span style="color: #B452CD">3.0</span>,<span style="color: #B452CD">5.0</span>,<span style="color: #B452CD">4.0</span>]])
h = matrix([<span style="color: #B452CD">0.0</span>,<span style="color: #B452CD">0.0</span>,-<span style="color: #B452CD">15.0</span>,<span style="color: #B452CD">100.0</span>,<span style="color: #B452CD">80.0</span>])
<span style="color: #228B22"># Define QP parameters (with NumPy)</span>
P = matrix(numpy.diag([<span style="color: #B452CD">1</span>,<span style="color: #B452CD">0</span>]), tc=<span style="color: #a61717; background-color: #e3d2d2">’</span>d<span style="color: #a61717; background-color: #e3d2d2">’</span>)
q = matrix(numpy.array([<span style="color: #B452CD">3</span>,<span style="color: #B452CD">4</span>]), tc=<span style="color: #a61717; background-color: #e3d2d2">’</span>d<span style="color: #a61717; background-color: #e3d2d2">’</span>)
G = matrix(numpy.array([[-<span style="color: #B452CD">1</span>,<span style="color: #B452CD">0</span>],[<span style="color: #B452CD">0</span>,-<span style="color: #B452CD">1</span>],[-<span style="color: #B452CD">1</span>,-<span style="color: #B452CD">3</span>],[<span style="color: #B452CD">2</span>,<span style="color: #B452CD">5</span>],[<span style="color: #B452CD">3</span>,<span style="color: #B452CD">4</span>]]), tc=<span style="color: #a61717; background-color: #e3d2d2">’</span>d<span style="color: #a61717; background-color: #e3d2d2">’</span>)
@@ -1100,7 +1101,7 @@ We are now ready to return to our setup of the optmization problem for a more re
<p>&nbsp;<br>
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
doc/pub/svm/html/svm-solarized.html
+26 -25
View File
@@ -109,7 +109,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 6, 2018</h4></center> <!-- date -->
<center><h4>Nov 7, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -211,13 +211,13 @@ $$
If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0,
$$
if our output \( y_i=1 \).
In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0,
$$
for the class of observations \( y_i=-1 \),
@@ -226,7 +226,7 @@ then \( \boldsymbol{x}_i \) lies on the other side.
<p>
Equivalently, for the two classes of observations we have
$$
y_i\left(b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip}\right) > 0.
y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0.
$$
<p>
@@ -700,17 +700,24 @@ y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i,
$$
from which we also find \( b \).
To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
$$
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
$$
For the above example, the kernel reads
$$
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2.
$$
We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
<p>
We note that this is nothing but the dot product of the two original
vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the
product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute
the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called
kernel trick and the result leads to the same as if we went through
the trouble of performing the transformation
\( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -724,11 +731,11 @@ $$
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
@@ -740,7 +747,7 @@ We can rewrite this (see the solutions below) in terms of a convex optimization
$$
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
$$
@@ -766,7 +773,7 @@ There are several popular kernels being used. These are
and many other ones.
<p>
An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
An important theorem for us is <a href="https://en.wikipedia.org/wiki/Mercer%27s_theorem" target="_blank">Mercer's theorem</a>.
The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then
there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space
(possibly with much higher dimensions) such that
@@ -789,7 +796,7 @@ A mathematical (quadratic) optimization problem, or just optimization problem, h
$$
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
$$
@@ -799,7 +806,7 @@ vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the
<p>
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
In our disussion on gradient descent methods we discussed at length the definition of a convex function.
In our discussion on gradient descent methods we discussed at length the definition of a convex function.
<p>
Convex optimization problems play a central role in applied mathematics and we recommend strongly <a href="http://web.stanford.edu/~boyd/cvxbook/" target="_blank">Boyd and Vandenberghe's text on the topics</a>.
@@ -840,7 +847,7 @@ We remind ourselves about the general problem we want to solve
$$
\begin{align*}
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f.
&\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f.
\end{align*}
$$
@@ -859,12 +866,12 @@ $$
The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
$$
frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
$$
Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
$$
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
$$
We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix
@@ -880,7 +887,7 @@ $$
<p>
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero
The following code does this for us
The following code solves the equations for us
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -888,12 +895,6 @@ The following code does this for us
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">cvxopt</span> <span style="color: #8B008B; font-weight: bold">import</span> matrix
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">cvxopt</span> <span style="color: #8B008B; font-weight: bold">import</span> solvers
<span style="color: #228B22"># Define QP parameters (directly)</span>
P = matrix([[<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">0.0</span>],[<span style="color: #B452CD">0.0</span>,<span style="color: #B452CD">0.0</span>]])
q = matrix([<span style="color: #B452CD">3.0</span>,<span style="color: #B452CD">4.0</span>])
G = matrix([[-<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">0.0</span>,-<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">2.0</span>,<span style="color: #B452CD">3.0</span>],[<span style="color: #B452CD">0.0</span>,-<span style="color: #B452CD">1.0</span>,-<span style="color: #B452CD">3.0</span>,<span style="color: #B452CD">5.0</span>,<span style="color: #B452CD">4.0</span>]])
h = matrix([<span style="color: #B452CD">0.0</span>,<span style="color: #B452CD">0.0</span>,-<span style="color: #B452CD">15.0</span>,<span style="color: #B452CD">100.0</span>,<span style="color: #B452CD">80.0</span>])
<span style="color: #228B22"># Define QP parameters (with NumPy)</span>
P = matrix(numpy.diag([<span style="color: #B452CD">1</span>,<span style="color: #B452CD">0</span>]), tc=<span style="color: #a61717; background-color: #e3d2d2">’</span>d<span style="color: #a61717; background-color: #e3d2d2">’</span>)
q = matrix(numpy.array([<span style="color: #B452CD">3</span>,<span style="color: #B452CD">4</span>]), tc=<span style="color: #a61717; background-color: #e3d2d2">’</span>d<span style="color: #a61717; background-color: #e3d2d2">’</span>)
G = matrix(numpy.array([[-<span style="color: #B452CD">1</span>,<span style="color: #B452CD">0</span>],[<span style="color: #B452CD">0</span>,-<span style="color: #B452CD">1</span>],[-<span style="color: #B452CD">1</span>,-<span style="color: #B452CD">3</span>],[<span style="color: #B452CD">2</span>,<span style="color: #B452CD">5</span>],[<span style="color: #B452CD">3</span>,<span style="color: #B452CD">4</span>]]), tc=<span style="color: #a61717; background-color: #e3d2d2">’</span>d<span style="color: #a61717; background-color: #e3d2d2">’</span>)
@@ -913,7 +914,7 @@ sol[<span style="color: #a61717; background-color: #e3d2d2">’</span>primal obj
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the <b>slack</b> parameter \( C \) we have
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
doc/pub/svm/html/svm.html
+26 -25
View File
@@ -114,7 +114,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 6, 2018</h4></center> <!-- date -->
<center><h4>Nov 7, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -216,13 +216,13 @@ $$
If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0,
$$
if our output \( y_i=1 \).
In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0,
$$
for the class of observations \( y_i=-1 \),
@@ -231,7 +231,7 @@ then \( \boldsymbol{x}_i \) lies on the other side.
<p>
Equivalently, for the two classes of observations we have
$$
y_i\left(b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip}\right) > 0.
y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0.
$$
<p>
@@ -705,17 +705,24 @@ y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i,
$$
from which we also find \( b \).
To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
$$
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
$$
For the above example, the kernel reads
$$
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2.
$$
We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
<p>
We note that this is nothing but the dot product of the two original
vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the
product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute
the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called
kernel trick and the result leads to the same as if we went through
the trouble of performing the transformation
\( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -729,11 +736,11 @@ $$
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
@@ -745,7 +752,7 @@ We can rewrite this (see the solutions below) in terms of a convex optimization
$$
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
$$
@@ -771,7 +778,7 @@ There are several popular kernels being used. These are
and many other ones.
<p>
An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
An important theorem for us is <a href="https://en.wikipedia.org/wiki/Mercer%27s_theorem" target="_blank">Mercer's theorem</a>.
The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then
there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space
(possibly with much higher dimensions) such that
@@ -794,7 +801,7 @@ A mathematical (quadratic) optimization problem, or just optimization problem, h
$$
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
$$
@@ -804,7 +811,7 @@ vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the
<p>
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
In our disussion on gradient descent methods we discussed at length the definition of a convex function.
In our discussion on gradient descent methods we discussed at length the definition of a convex function.
<p>
Convex optimization problems play a central role in applied mathematics and we recommend strongly <a href="http://web.stanford.edu/~boyd/cvxbook/" target="_blank">Boyd and Vandenberghe's text on the topics</a>.
@@ -845,7 +852,7 @@ We remind ourselves about the general problem we want to solve
$$
\begin{align*}
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f.
&\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f.
\end{align*}
$$
@@ -864,12 +871,12 @@ $$
The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
$$
frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
$$
Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
$$
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
$$
We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix
@@ -885,7 +892,7 @@ $$
<p>
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero
The following code does this for us
The following code solves the equations for us
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -893,12 +900,6 @@ The following code does this for us
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span> <span style="color: #008000; font-weight: bold">import</span> matrix
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span> <span style="color: #008000; font-weight: bold">import</span> solvers
<span style="color: #408080; font-style: italic"># Define QP parameters (directly)</span>
P <span style="color: #666666">=</span> matrix([[<span style="color: #666666">1.0</span>,<span style="color: #666666">0.0</span>],[<span style="color: #666666">0.0</span>,<span style="color: #666666">0.0</span>]])
q <span style="color: #666666">=</span> matrix([<span style="color: #666666">3.0</span>,<span style="color: #666666">4.0</span>])
G <span style="color: #666666">=</span> matrix([[<span style="color: #666666">-1.0</span>,<span style="color: #666666">0.0</span>,<span style="color: #666666">-1.0</span>,<span style="color: #666666">2.0</span>,<span style="color: #666666">3.0</span>],[<span style="color: #666666">0.0</span>,<span style="color: #666666">-1.0</span>,<span style="color: #666666">-3.0</span>,<span style="color: #666666">5.0</span>,<span style="color: #666666">4.0</span>]])
h <span style="color: #666666">=</span> matrix([<span style="color: #666666">0.0</span>,<span style="color: #666666">0.0</span>,<span style="color: #666666">-15.0</span>,<span style="color: #666666">100.0</span>,<span style="color: #666666">80.0</span>])
<span style="color: #408080; font-style: italic"># Define QP parameters (with NumPy)</span>
P <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>diag([<span style="color: #666666">1</span>,<span style="color: #666666">0</span>]), tc<span style="color: #666666">=</span>’d’)
q <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>array([<span style="color: #666666">3</span>,<span style="color: #666666">4</span>]), tc<span style="color: #666666">=</span>’d’)
G <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>array([[<span style="color: #666666">-1</span>,<span style="color: #666666">0</span>],[<span style="color: #666666">0</span>,<span style="color: #666666">-1</span>],[<span style="color: #666666">-1</span>,<span style="color: #666666">-3</span>],[<span style="color: #666666">2</span>,<span style="color: #666666">5</span>],[<span style="color: #666666">3</span>,<span style="color: #666666">4</span>]]), tc<span style="color: #666666">=</span>’d’)
@@ -918,7 +919,7 @@ sol[’primal objective’]
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the <b>slack</b> parameter \( C \) we have
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
doc/pub/svm/ipynb/ipynb-svm-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/svm/ipynb/svm.ipynb
+25 -25
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Nov 6, 2018**\n",
"Date: **Nov 7, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -144,7 +144,7 @@
"metadata": {},
"source": [
"$$\n",
"b+w_1x_{i1}+w_2x_{i}2+\\dots +w_px_{ip} >0,\n",
"b+w_1x_{i1}+w_2x_{i2}+\\dots +w_px_{ip} >0,\n",
"$$"
]
},
@@ -161,7 +161,7 @@
"metadata": {},
"source": [
"$$\n",
"b+w_1x_{i1}+w_2x_{i}2+\\dots +w_px_{ip} < 0,\n",
"b+w_1x_{i1}+w_2x_{i2}+\\dots +w_px_{ip} < 0,\n",
"$$"
]
},
@@ -180,7 +180,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i\\left(b+w_1x_{i1}+w_2x_{i}2+\\dots +w_px_{ip}\\right) > 0.\n",
"y_i\\left(b+w_1x_{i1}+w_2x_{i2}+\\dots +w_px_{ip}\\right) > 0.\n",
"$$"
]
},
@@ -1132,7 +1132,7 @@
"metadata": {},
"source": [
"from which we also find $b$.\n",
"To compute $\\boldsymbol{z}_i^T\\boldsymbol{Z}_j$ we define the kerne $K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)$ as"
"To compute $\\boldsymbol{z}_i^T\\boldsymbol{z}_j$ we define the kerne $K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)$ as"
]
},
{
@@ -1140,7 +1140,7 @@
"metadata": {},
"source": [
"$$\n",
"K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=\\boldsymbol{z}_i^T\\boldsymbol{Z}_j= \\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j).\n",
"K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=\\boldsymbol{z}_i^T\\boldsymbol{z}_j= \\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j).\n",
"$$"
]
},
@@ -1156,7 +1156,7 @@
"metadata": {},
"source": [
"$$\n",
"K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=[x_i^2, y_i^2, \\sqrt{2}x_iy_i]^T\\begin{bmatrix} x_j^2 \\\\ y_j^2 \\\\ \\sqrt{2}x_jy_j \\end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.\n",
"K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=[x_i^2, y_i^2, \\sqrt{2}x_iy_i]^T\\begin{bmatrix} x_j^2 \\\\ y_j^2 \\\\ \\sqrt{2}x_jy_j \\end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2.\n",
"$$"
]
},
@@ -1164,7 +1164,13 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We note that this is nothing but the dot product of the two original vectors $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. Instead of thus computing the product in the Lagrangian of $\\boldsymbol{x}_i^T\\boldsymbol{z}_j$ we simply compute the dot product $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation $(\\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j)$ during the SVM calculations. \n",
"We note that this is nothing but the dot product of the two original\n",
"vectors $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. Instead of thus computing the\n",
"product in the Lagrangian of $\\boldsymbol{z}_i^T\\boldsymbol{z}_j$ we simply compute\n",
"the dot product $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. This leads to the so-called\n",
"kernel trick and the result leads to the same as if we went through\n",
"the trouble of performing the transformation\n",
"$(\\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j)$ during the SVM calculations.\n",
"\n",
"\n",
"## The problem to solve\n",
@@ -1193,11 +1199,11 @@
"source": [
"$$\n",
"\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1K(\\boldsymbol{x}_1,\\boldsymbol{x}_1) & y_1y_2K(\\boldsymbol{x}_1,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_1,\\boldsymbol{x}_n) \\\\\n",
"y_2y_1\\boldsymbol{x}_2^T\\boldsymbol{x}_1 & y_2y_2\\boldsymbol{x}_2^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_2^T\\boldsymbol{x}_n \\\\\n",
"y_2y_1K(\\boldsymbol{x}_2,\\boldsymbol{x}_1) & y_2y_2(\\boldsymbol{x}_2,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_2,\\boldsymbol{x}_n) \\\\\n",
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
"y_ny_1K(\\boldsymbol{x}_n,\\boldsymbol{x}_1) & y_ny_2K(\\boldsymbol{x}_n\\boldsymbol{x}_2) & \\dots & \\dots & y_ny_nK(\\boldsymbol{x}_n,\\boldsymbol{x}_n) \\\\\n",
"\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{I}\\boldsymbol{\\lambda},\n",
"\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{1}\\boldsymbol{\\lambda},\n",
"$$"
]
},
@@ -1219,7 +1225,7 @@
"$$\n",
"\\begin{align*}\n",
" &\\mathrm{min}_{\\lambda}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{\\lambda}^T\\boldsymbol{P}\\boldsymbol{\\lambda}+\\boldsymbol{q}^T\\boldsymbol{\\lambda},\\\\ \\nonumber\n",
" &\\mathrm{subject to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n",
" &\\mathrm{subject\\hspace{0.1cm}to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\wedge \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n",
"\\end{align*}\n",
"$$"
]
@@ -1246,7 +1252,7 @@
"\n",
"and many other ones.\n",
"\n",
"An important theorem for us is \"Mercer's theorem\":https://en.wikipedia.org/wiki/Mercer%27s_theorem\". \n",
"An important theorem for us is [Mercer's theorem](https://en.wikipedia.org/wiki/Mercer%27s_theorem). \n",
"The theorem states that if a kernel function $K$ is symmetric, continuous and leads to a positive semi-definite matrix $\\boldsymbol{P}$ then\n",
"there exists a function $\\phi$ that maps $\\boldsymbol{x}_i$ and $\\boldsymbol{x}_j$ into another space\n",
"(possibly with much higher dimensions) such that"
@@ -1283,7 +1289,7 @@
"$$\n",
"\\begin{align*}\n",
" &\\mathrm{min}_{\\lambda}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{\\lambda}^T\\boldsymbol{P}\\boldsymbol{\\lambda}+\\boldsymbol{q}^T\\boldsymbol{\\lambda},\\\\ \\nonumber\n",
" &\\mathrm{subject to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n",
" &\\mathrm{subject\\hspace{0.1cm}to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\wedge \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n",
"\\end{align*}\n",
"$$"
]
@@ -1297,7 +1303,7 @@
"vector $\\boldsymbol{\\lambda}=[\\lambda_1, \\lambda_2,\\dots, \\lambda_n]$ is the optimization variable we are dealing with.\n",
"\n",
"In our case we are particularly interested in a class of optimization problems called convex optmization problems. \n",
"In our disussion on gradient descent methods we discussed at length the definition of a convex function. \n",
"In our discussion on gradient descent methods we discussed at length the definition of a convex function. \n",
"\n",
"Convex optimization problems play a central role in applied mathematics and we recommend strongly [Boyd and Vandenberghe's text on the topics](http://web.stanford.edu/~boyd/cvxbook/).\n",
"\n",
@@ -1347,7 +1353,7 @@
"$$\n",
"\\begin{align*}\n",
" &\\mathrm{min}_{x}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{P}\\boldsymbol{x}+\\boldsymbol{q}^T\\boldsymbol{x},\\\\ \\nonumber\n",
" &\\mathrm{subject to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{x} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\boldsymbol{A}\\boldsymbol{x}=f.\n",
" &\\mathrm{subject\\hspace{0.1cm} to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{x} \\preceq \\boldsymbol{h} \\wedge \\boldsymbol{A}\\boldsymbol{x}=f.\n",
"\\end{align*}\n",
"$$"
]
@@ -1387,7 +1393,7 @@
"metadata": {},
"source": [
"$$\n",
"frac{1}{2}\\begin{bmatrix} x\\\\ y \\end{bmatrix}^T \\begin{bmatrix} 1 & 0\\\\ 0 & 0 \\end{bmatrix} \\begin{bmatrix} x \\\\ y \\end{bmatrix} + \\begin{bmatrix}3\\\\ 4 \\end{bmatrix}^T \\begin{bmatrix}x \\\\ y \\end{bmatrix}.\n",
"\\frac{1}{2}\\begin{bmatrix} x\\\\ y \\end{bmatrix}^T \\begin{bmatrix} 1 & 0\\\\ 0 & 0 \\end{bmatrix} \\begin{bmatrix} x \\\\ y \\end{bmatrix} + \\begin{bmatrix}3\\\\ 4 \\end{bmatrix}^T \\begin{bmatrix}x \\\\ y \\end{bmatrix}.\n",
"$$"
]
},
@@ -1403,7 +1409,7 @@
"metadata": {},
"source": [
"$$\n",
"\\begin{bmatrix} -1 & 0 \\\\ 0 & -1 \\\\ -1 & -3 \\\\ 2 & 5 \\\\ 3 & 4\\end{matrix}\\begin{bmatrix} x \\\\ y\\end{bmatrix} \\preceq \\begin{bmatrix}0 \\\\ 0\\\\ -15 \\\\ 100 \\\\ 80\\end{bmatrix}.\n",
"\\begin{bmatrix} -1 & 0 \\\\ 0 & -1 \\\\ -1 & -3 \\\\ 2 & 5 \\\\ 3 & 4\\end{bmatrix}\\begin{bmatrix} x \\\\ y\\end{bmatrix} \\preceq \\begin{bmatrix}0 \\\\ 0\\\\ -15 \\\\ 100 \\\\ 80\\end{bmatrix}.\n",
"$$"
]
},
@@ -1445,7 +1451,7 @@
"metadata": {},
"source": [
"Since we don't have any equalities the matrix $\\boldsymbol{A}$ is set to zero\n",
"The following code does this for us"
"The following code solves the equations for us"
]
},
{
@@ -1460,12 +1466,6 @@
"import numpy\n",
"from cvxopt import matrix\n",
"from cvxopt import solvers\n",
"# Define QP parameters (directly)\n",
"P = matrix([[1.0,0.0],[0.0,0.0]])\n",
"q = matrix([3.0,4.0])\n",
"G = matrix([[-1.0,0.0,-1.0,2.0,3.0],[0.0,-1.0,-3.0,5.0,4.0]])\n",
"h = matrix([0.0,0.0,-15.0,100.0,80.0])\n",
"# Define QP parameters (with NumPy)\n",
"P = matrix(numpy.diag([1,0]), tc=’d’)\n",
"q = matrix(numpy.array([3,4]), tc=’d’)\n",
"G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’)\n",
@@ -1492,7 +1492,7 @@
"source": [
"$$\n",
"\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1K(\\boldsymbol{x}_1,\\boldsymbol{x}_1) & y_1y_2K(\\boldsymbol{x}_1,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_1,\\boldsymbol{x}_n) \\\\\n",
"y_2y_1\\boldsymbol{x}_2^T\\boldsymbol{x}_1 & y_2y_2\\boldsymbol{x}_2^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_2^T\\boldsymbol{x}_n \\\\\n",
"y_2y_1K(\\boldsymbol{x}_2,\\boldsymbol{x}_1) & y_2y_2K(\\boldsymbol{x}_2,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_2,\\boldsymbol{x}_n) \\\\\n",
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
"y_ny_1K(\\boldsymbol{x}_n,\\boldsymbol{x}_1) & y_ny_2K(\\boldsymbol{x}_n\\boldsymbol{x}_2) & \\dots & \\dots & y_ny_nK(\\boldsymbol{x}_n,\\boldsymbol{x}_n) \\\\\n",
doc/pub/svm/pdf/svm-minted.pdf
BIN
View File
Binary file not shown.
doc/src/SupportVMachines/svm.do.txt
+23 -16
View File
@@ -90,14 +90,14 @@ of dimension $n\times p$, where $n$ represents the observations for each feature
If the above condition is not met for a given vector $\bm{x}_i$ we have
!bt
\[
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0,
\]
!et
if our output $y_i=1$.
In this case we say that $\bm{x}_i$ lies on one of the sides of the hyperplane and if
!bt
\[
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0,
\]
!et
for the class of observations $y_i=-1$,
@@ -106,7 +106,7 @@ then $\bm{x}_i$ lies on the other side.
Equivalently, for the two classes of observations we have
!bt
\[
y_i\left(b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip}\right) > 0.
y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0.
\]
!et
@@ -576,19 +576,26 @@ y_i(\bm{w}^T\bm{z}_i+b)= 1 \hspace{0.1cm}\forall i,
\]
!et
from which we also find $b$.
To compute $\bm{z}_i^T\bm{Z}_j$ we define the kerne $K(\bm{x}_i,\bm{x}_j)$ as
To compute $\bm{z}_i^T\bm{z}_j$ we define the kerne $K(\bm{x}_i,\bm{x}_j)$ as
!bt
\[
K(\bm{x}_i,\bm{x}_j)=\bm{z}_i^T\bm{Z}_j= \phi(\bm{x}_i)^T\phi(\bm{x}_j).
K(\bm{x}_i,\bm{x}_j)=\bm{z}_i^T\bm{z}_j= \phi(\bm{x}_i)^T\phi(\bm{x}_j).
\]
!et
For the above example, the kernel reads
!bt
\[
K(\bm{x}_i,\bm{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
K(\bm{x}_i,\bm{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2.
\]
!et
We note that this is nothing but the dot product of the two original vectors $(\bm{x}_i^T\bm{x}_j)^2$. Instead of thus computing the product in the Lagrangian of $\bm{x}_i^T\bm{z}_j$ we simply compute the dot product $(\bm{x}_i^T\bm{x}_j)^2$. This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation $(\phi(\bm{x}_i)^T\phi(\bm{x}_j)$ during the SVM calculations.
We note that this is nothing but the dot product of the two original
vectors $(\bm{x}_i^T\bm{x}_j)^2$. Instead of thus computing the
product in the Lagrangian of $\bm{z}_i^T\bm{z}_j$ we simply compute
the dot product $(\bm{x}_i^T\bm{x}_j)^2$. This leads to the so-called
kernel trick and the result leads to the same as if we went through
the trouble of performing the transformation
$(\phi(\bm{x}_i)^T\phi(\bm{x}_j)$ during the SVM calculations.
!split
@@ -603,11 +610,11 @@ subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$ in terms of
!bt
\[
\frac{1}{2} \bm{\lambda}^T\begin{bmatrix} y_1y_1K(\bm{x}_1,\bm{x}_1) & y_1y_2K(\bm{x}_1,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_1,\bm{x}_n) \\
y_2y_1\bm{x}_2^T\bm{x}_1 & y_2y_2\bm{x}_2^T\bm{x}_2 & \dots & \dots & y_1y_n\bm{x}_2^T\bm{x}_n \\
y_2y_1K(\bm{x}_2,\bm{x}_1) & y_2y_2(\bm{x}_2,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_2,\bm{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\bm{x}_n,\bm{x}_1) & y_ny_2K(\bm{x}_n\bm{x}_2) & \dots & \dots & y_ny_nK(\bm{x}_n,\bm{x}_n) \\
\end{bmatrix}\bm{\lambda}-\mathbb{I}\bm{\lambda},
\end{bmatrix}\bm{\lambda}-\mathbb{1}\bm{\lambda},
\]
!et
subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and
@@ -618,7 +625,7 @@ We can rewrite this (see the solutions below) in terms of a convex optimization
!bt
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\bm{\lambda}^T\bm{P}\bm{\lambda}+\bm{q}^T\bm{\lambda},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \hspace{0.2cm} \bm{A}\bm{\lambda}=f.
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \hspace{0.2cm} \wedge \bm{A}\bm{\lambda}=f.
\end{align*}
!et
Below we discuss how to solve these equations. Here we note that the matrix $\bm{P}$ has matrix elements $p_{ij}=y_iy_jK(\bm{x}_i,\bm{x}_j)$.
@@ -636,7 +643,7 @@ o Gaussian Radial Basis Function: $K(\bm{x},\bm{y})=\exp{\left(-\gamma\vert\vert
o Tanh: $K(\bm{x},\bm{y})=\tanh{(\bm{x}^T\bm{y}+\gamma)}$,
and many other ones.
An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
An important theorem for us is "Mercer's theorem":"https://en.wikipedia.org/wiki/Mercer%27s_theorem".
The theorem states that if a kernel function $K$ is symmetric, continuous and leads to a positive semi-definite matrix $\bm{P}$ then
there exists a function $\phi$ that maps $\bm{x}_i$ and $\bm{x}_j$ into another space
(possibly with much higher dimensions) such that
@@ -658,7 +665,7 @@ A mathematical (quadratic) optimization problem, or just optimization problem, h
!bt
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\bm{\lambda}^T\bm{P}\bm{\lambda}+\bm{q}^T\bm{\lambda},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \hspace{0.2cm} \bm{A}\bm{\lambda}=f.
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \wedge \bm{A}\bm{\lambda}=f.
\end{align*}
!et
subject to some constraints for say a selected set $i=1,2,\dots, n$.
@@ -666,7 +673,7 @@ In our case we are optimizing with respect to the Lagrangian multipliers $\lambd
vector $\bm{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n]$ is the optimization variable we are dealing with.
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
In our disussion on gradient descent methods we discussed at length the definition of a convex function.
In our discussion on gradient descent methods we discussed at length the definition of a convex function.
Convex optimization problems play a central role in applied mathematics and we recommend strongly "Boyd and Vandenberghe's text on the topics":"http://web.stanford.edu/~boyd/cvxbook/".
@@ -700,7 +707,7 @@ We remind ourselves about the general problem we want to solve
!bt
\begin{align*}
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\bm{x}^T\bm{P}\bm{x}+\bm{q}^T\bm{x},\\ \nonumber
&\mathrm{subject to} \hspace{0.2cm} \bm{G}\bm{x} \preceq \bm{h} \hspace{0.2cm} \bm{A}\bm{x}=f.
&\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \bm{G}\bm{x} \preceq \bm{h} \wedge \bm{A}\bm{x}=f.
\end{align*}
!et
@@ -718,13 +725,13 @@ Let us show how to perform the optmization using a simple case. Assume we want t
The minimization problem can be rewritten in terms of vectors and matrices as (with $x$ and $y$ being the unknowns)
!bt
\[
frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
\]
!et
Similarly, we can now set up the inequalities (we need to change $\geq$ to $\leq$ by multiplying with $-1$ on bot sides) as the following matrix-vector equation
!bt
\[
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
\]
!et
We have collapsed all the inequalities into a single matrix $\bm{G}$. We see also that our matrix