lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

replacing hats with boldmath

Browse Source
This commit is contained in:
mhjensen
2018-11-05 21:21:26 +01:00
parent 401104c7ff
commit 4945a43ae7
18 changed files with 387 additions and 343 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/svm/html/._svm-bs003.html
+2 -2
View File
@@ -147,10 +147,10 @@ $$
where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line
\( b+w_1x_1+w_2x_2=0 \).
In two dimensions we define the vectors \( \hat{x} =[x1,x2] \) and \( \hat{w}=[w1,w2] \).
In two dimensions we define the vectors \( \boldmath{x} =[x1,x2] \) and \( \boldmath{w}=[w1,w2] \).
We can then rewrite the above equation as
$$
\hat{w}^T\hat{x}+b=0.
\boldmath{w}^T\boldmath{x}+b=0.
$$
<p>
doc/pub/svm/html/._svm-bs004.html
+6 -6
View File
@@ -139,25 +139,25 @@ b+wx_1+w_2x_2+\dots +w_px_p=0.
$$
If we define a
matrix \( \hat{X}=\left[\hat{x}_1,\hat{x}_2,\dots, \hat{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \hat{X} \),
matrix \( \boldmath{X}=\left[\boldmath{x}_1,\boldmath{x}_2,\dots, \boldmath{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldmath{X} \),
$$
\hat{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\boldmath{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
$$
If the above condition is not met for a given vector \( \hat{x}_i \) we have
If the above condition is not met for a given vector \( \boldmath{x}_i \) we have
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
$$
if our output \( y_i=1 \).
In this case we say that \( \hat{x}_i \) lies on one of the sides of the hyperplane and if
In this case we say that \( \boldmath{x}_i \) lies on one of the sides of the hyperplane and if
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
$$
for the class of observations \( y_i=-1 \),
then \( \hat{x}_i \) lies on the other side.
then \( \boldmath{x}_i \) lies on the other side.
<p>
Equivalently, for the two classes of observations we have
doc/pub/svm/html/._svm-bs006.html
+4 -4
View File
@@ -134,18 +134,18 @@ MathJax.Hub.Config({
<p>
Let us define the function
$$
f(x) = \hat{w}^T\hat{x}+b = 0,
f(x) = \boldmath{w}^T\boldmath{x}+b = 0,
$$
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
<p>
Any point defined by \( \hat{x}_i \) and \( \hat{x}_2 \) on the line \( L \) will satisfy \( \hat{w}^T(\hat{x}_1-\hat{x}_2)=0 \).
Any point defined by \( \boldmath{x}_i \) and \( \boldmath{x}_2 \) on the line \( L \) will satisfy \( \boldmath{w}^T(\boldmath{x}_1-\boldmath{x}_2)=0 \).
<p>
The signed distance \( \delta \) from any point defined by a vector \( \hat{x} \) and a point \( \hat{x}_0 \) on the line \( L \) is then
The signed distance \( \delta \) from any point defined by a vector \( \boldmath{x} \) and a point \( \boldmath{x}_0 \) on the line \( L \) is then
$$
\delta = \frac{1}{\vert\vert \hat{w}\vert\vert}(\hat{w}^T\hat{x}+b).
\delta = \frac{1}{\vert\vert \boldmath{w}\vert\vert}(\boldmath{w}^T\boldmath{x}+b).
$$
<p>
doc/pub/svm/html/._svm-bs007.html
+4 -4
View File
@@ -132,23 +132,23 @@ MathJax.Hub.Config({
<h2 id="___sec6" class="anchor">First attempt at a minimization approach </h2>
<p>
How do we find the parameter \( b \) and the vector \( \hat{w} \)? What we could
How do we find the parameter \( b \) and the vector \( \boldmath{w} \)? What we could
do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
$$
C(\hat{w},b) = -\sum_{i\in M} y_i(\hat{w}^T\hat{x}_i+b).
C(\boldmath{w},b) = -\sum_{i\in M} y_i(\boldmath{w}^T\boldmath{x}_i+b).
$$
<p>
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \hat{w}^T\hat{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldmath{w}^T\boldmath{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
$$
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
$$
and
$$
\frac{\partial C}{\partial \hat{w}} = -\sum_{i\in M} y_ix_i.
\frac{\partial C}{\partial \boldmath{w}} = -\sum_{i\in M} y_ix_i.
$$
<p>
doc/pub/svm/html/._svm-bs008.html
+1 -1
View File
@@ -139,7 +139,7 @@ $$
and
$$
\hat{w} \leftarrow \hat{w} +\eta \frac{\partial C}{\partial \hat{w}},
\boldmath{w} \leftarrow \boldmath{w} +\eta \frac{\partial C}{\partial \boldmath{w}},
$$
where \( \eta \) is our by now well-known learning rate.
doc/pub/svm/html/._svm-bs009.html
+9 -9
View File
@@ -136,11 +136,11 @@ A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).
<p>
Thus, we wish to find a margin \( M \) with \( \hat{w} \) normalized to
\( \vert\vert \hat{w}\vert\vert =1 \) subject to the condition
Thus, we wish to find a margin \( M \) with \( \boldmath{w} \) normalized to
\( \vert\vert \boldmath{w}\vert\vert =1 \) subject to the condition
$$
y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.
@@ -148,22 +148,22 @@ All points are thus at a signed distance from the decision boundary defined by t
<p>
We seek thus the largest value \( M \) defined by
$$
\frac{1}{\vert \vert \hat{w}\vert\vert}y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \boldmath{w}\vert\vert}y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
or just
$$
y_i(\hat{w}^T\hat{x}_i+b) \geq M\vert \vert \hat{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M\vert \vert \boldmath{w}\vert\vert \hspace{0.1cm}\forall i.
$$
If we scale the equation so that \( \vert \vert \hat{w}\vert\vert = 1/M \), we have to find the minimum of
\( \hat{w}^T\hat{w}=\vert \vert \hat{w}\vert\vert \) (the norm) subject to the condition
If we scale the equation so that \( \vert \vert \boldmath{w}\vert\vert = 1/M \), we have to find the minimum of
\( \boldmath{w}^T\boldmath{w}=\vert \vert \boldmath{w}\vert\vert \) (the norm) subject to the condition
$$
y_i(\hat{w}^T\hat{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
$$
<p>
We have thus defined our margin as the invers of the norm of \( \hat{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
We have thus defined our margin as the invers of the norm of \( \boldmath{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
<p>
<p>
doc/pub/svm/html/._svm-bs012.html
+8 -8
View File
@@ -132,40 +132,40 @@ MathJax.Hub.Config({
<h2 id="___sec11" class="anchor">Setting up the problem </h2>
In order to solve the above problem, we define the following Lagrangian function to be minimized
$$
{\cal L}(\lambda,b,\hat{w})=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-1\right],
{\cal L}(\lambda,b,\boldmath{w})=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-1\right],
$$
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i.
\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i.
$$
Inserting these constraints into the equation for \( {\cal L} \) we obtain
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_self">Karush-Kuhn-Tucker</a> (KKT) condition
$$
\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
<ol>
<li> If \( \lambda_i > 0 \), then \( y_i(\hat{w}^T\hat{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\hat{w}^T\hat{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
<li> If \( \lambda_i > 0 \), then \( y_i(\boldmath{w}^T\boldmath{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\boldmath{w}^T\boldmath{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
When \( \lambda_i > 0 \), the vectors \( \hat{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
When \( \lambda_i > 0 \), the vectors \( \boldmath{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
<p>
<p>
doc/pub/svm/html/._svm-bs013.html
+7 -7
View File
@@ -134,21 +134,21 @@ MathJax.Hub.Config({
<p>
We can rewrite
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
$$
\frac{1}{2} \hat{\lambda}^T\begin{bmatrix} y_1y_1\hat{x}_1^T\hat{x}_1 & y_1y_2\hat{x}_1^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_1^T\hat{x}_n \\
y_2y_1\hat{x}_2^T\hat{x}_1 & y_2y_2\hat{x}_2^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_2^T\hat{x}_n \\
\frac{1}{2} \boldmath{\lambda}^T\begin{bmatrix} y_1y_1\boldmath{x}_1^T\boldmath{x}_1 & y_1y_2\boldmath{x}_1^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_1^T\boldmath{x}_n \\
y_2y_1\boldmath{x}_2^T\boldmath{x}_1 & y_2y_2\boldmath{x}_2^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_2^T\boldmath{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1\hat{x}_n^T\hat{x}_1 & y_ny_2\hat{x}_n^T\hat{x}_2 & \dots & \dots & y_ny_n\hat{x}_n^T\hat{x}_n \\
\end{bmatrix}\hat{\lambda}-\mathbb{1}\hat{\lambda},
y_ny_1\boldmath{x}_n^T\boldmath{x}_1 & y_ny_2\boldmath{x}_n^T\boldmath{x}_2 & \dots & \dots & y_ny_n\boldmath{x}_n^T\boldmath{x}_n \\
\end{bmatrix}\boldmath{\lambda}-\mathbb{1}\boldmath{\lambda},
$$
subject to \( \hat{y}^T\hat{\lambda}=0 \). Here we defined the vectors \( \hat{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \hat{y}=[y_1,y_2,\dots,y_n] \).
subject to \( \boldmath{y}^T\boldmath{\lambda}=0 \). Here we defined the vectors \( \boldmath{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldmath{y}=[y_1,y_2,\dots,y_n] \).
<p>
<p>
doc/pub/svm/html/._svm-bs014.html
+10 -5
View File
@@ -135,22 +135,27 @@ MathJax.Hub.Config({
Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
$$
\hat{w}=\sum_{i} \lambda_iy_i\hat{x}_i.
\boldmath{w}=\sum_{i} \lambda_iy_i\boldmath{x}_i.
$$
With our vector \( \hat{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
With our vector \( \boldmath{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
y_i(\hat{w}^T\hat{x}_i+b)=1,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
$$
resulting in
$$
b = \frac{1}{y_i}-\hat{w}^T\hat{x}_i.
b = \frac{1}{y_i}-\boldmath{w}^T\boldmath{x}_i,
$$
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
$$
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldmath{x}_i^T\boldmath{x}_j\right).
$$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
y_i = \mathrm{sign}(\hat{w}^T\hat{x}_i+b).
y_i = \mathrm{sign}(\boldmath{w}^T\boldmath{x}_i+b).
$$
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
doc/pub/svm/html/._svm-bs015.html
+4 -4
View File
@@ -141,20 +141,20 @@ so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
<p>
We introduce thus the so-called <b>slack</b> variables \( \hat{\xi} =[\xi_1,x_2,\dots,x_n] \) and
We introduce thus the so-called <b>slack</b> variables \( \boldmath{\xi} =[\xi_1,x_2,\dots,x_n] \) and
modify our previous equation
$$
y_i(\hat{w}^T\hat{x}_i+b)=1,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
$$
to
$$
y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i,
$$
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
\( y_i(\hat{w}^T\hat{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
\( y_i(\boldmath{w}^T\boldmath{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
<p>
doc/pub/svm/html/._svm-bs016.html
+7 -7
View File
@@ -134,25 +134,25 @@ MathJax.Hub.Config({
<p>
This has in turn the consequences that we change our optmization problem to finding the minimum of
$$
{\cal L}=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
{\cal L}=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
$$
subject to
$$
y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
with the requirement \( \xi_i\geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i,
\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i,
$$
and
@@ -162,13 +162,13 @@ $$
Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \).
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
$$
\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
$$
@@ -177,7 +177,7 @@ $$
and
$$
y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
<p>
doc/pub/svm/html/svm-reveal.html
+64 -57
View File
@@ -225,11 +225,11 @@ $$
where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line
\( b+w_1x_1+w_2x_2=0 \).
In two dimensions we define the vectors \( \hat{x} =[x1,x2] \) and \( \hat{w}=[w1,w2] \).
In two dimensions we define the vectors \( \boldmath{x} =[x1,x2] \) and \( \boldmath{w}=[w1,w2] \).
We can then rewrite the above equation as
<p>&nbsp;<br>
$$
\hat{w}^T\hat{x}+b=0.
\boldmath{w}^T\boldmath{x}+b=0.
$$
<p>&nbsp;<br>
</section>
@@ -248,15 +248,15 @@ $$
<p>&nbsp;<br>
If we define a
matrix \( \hat{X}=\left[\hat{x}_1,\hat{x}_2,\dots, \hat{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \hat{X} \),
matrix \( \boldmath{X}=\left[\boldmath{x}_1,\boldmath{x}_2,\dots, \boldmath{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldmath{X} \),
<p>&nbsp;<br>
$$
\hat{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\boldmath{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
$$
<p>&nbsp;<br>
If the above condition is not met for a given vector \( \hat{x}_i \) we have
If the above condition is not met for a given vector \( \boldmath{x}_i \) we have
<p>&nbsp;<br>
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
@@ -264,7 +264,7 @@ $$
<p>&nbsp;<br>
if our output \( y_i=1 \).
In this case we say that \( \hat{x}_i \) lies on one of the sides of the hyperplane and if
In this case we say that \( \boldmath{x}_i \) lies on one of the sides of the hyperplane and if
<p>&nbsp;<br>
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
@@ -272,7 +272,7 @@ $$
<p>&nbsp;<br>
for the class of observations \( y_i=-1 \),
then \( \hat{x}_i \) lies on the other side.
then \( \boldmath{x}_i \) lies on the other side.
<p>
Equivalently, for the two classes of observations we have
@@ -321,20 +321,20 @@ for our data sample.
Let us define the function
<p>&nbsp;<br>
$$
f(x) = \hat{w}^T\hat{x}+b = 0,
f(x) = \boldmath{w}^T\boldmath{x}+b = 0,
$$
<p>&nbsp;<br>
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
<p>
Any point defined by \( \hat{x}_i \) and \( \hat{x}_2 \) on the line \( L \) will satisfy \( \hat{w}^T(\hat{x}_1-\hat{x}_2)=0 \).
Any point defined by \( \boldmath{x}_i \) and \( \boldmath{x}_2 \) on the line \( L \) will satisfy \( \boldmath{w}^T(\boldmath{x}_1-\boldmath{x}_2)=0 \).
<p>
The signed distance \( \delta \) from any point defined by a vector \( \hat{x} \) and a point \( \hat{x}_0 \) on the line \( L \) is then
The signed distance \( \delta \) from any point defined by a vector \( \boldmath{x} \) and a point \( \boldmath{x}_0 \) on the line \( L \) is then
<p>&nbsp;<br>
$$
\delta = \frac{1}{\vert\vert \hat{w}\vert\vert}(\hat{w}^T\hat{x}+b).
\delta = \frac{1}{\vert\vert \boldmath{w}\vert\vert}(\boldmath{w}^T\boldmath{x}+b).
$$
<p>&nbsp;<br>
</section>
@@ -344,18 +344,18 @@ $$
<h2 id="___sec6">First attempt at a minimization approach </h2>
<p>
How do we find the parameter \( b \) and the vector \( \hat{w} \)? What we could
How do we find the parameter \( b \) and the vector \( \boldmath{w} \)? What we could
do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
<p>&nbsp;<br>
$$
C(\hat{w},b) = -\sum_{i\in M} y_i(\hat{w}^T\hat{x}_i+b).
C(\boldmath{w},b) = -\sum_{i\in M} y_i(\boldmath{w}^T\boldmath{x}_i+b).
$$
<p>&nbsp;<br>
<p>
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \hat{w}^T\hat{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldmath{w}^T\boldmath{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
<p>&nbsp;<br>
$$
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
@@ -365,7 +365,7 @@ $$
and
<p>&nbsp;<br>
$$
\frac{\partial C}{\partial \hat{w}} = -\sum_{i\in M} y_ix_i.
\frac{\partial C}{\partial \boldmath{w}} = -\sum_{i\in M} y_ix_i.
$$
<p>&nbsp;<br>
</section>
@@ -385,7 +385,7 @@ $$
and
<p>&nbsp;<br>
$$
\hat{w} \leftarrow \hat{w} +\eta \frac{\partial C}{\partial \hat{w}},
\boldmath{w} \leftarrow \boldmath{w} +\eta \frac{\partial C}{\partial \boldmath{w}},
$$
<p>&nbsp;<br>
@@ -411,12 +411,12 @@ A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).
<p>
Thus, we wish to find a margin \( M \) with \( \hat{w} \) normalized to
\( \vert\vert \hat{w}\vert\vert =1 \) subject to the condition
Thus, we wish to find a margin \( M \) with \( \boldmath{w} \) normalized to
\( \vert\vert \boldmath{w}\vert\vert =1 \) subject to the condition
<p>&nbsp;<br>
$$
y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
<p>&nbsp;<br>
@@ -426,27 +426,27 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value \( M \) defined by
<p>&nbsp;<br>
$$
\frac{1}{\vert \vert \hat{w}\vert\vert}y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \boldmath{w}\vert\vert}y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
<p>&nbsp;<br>
or just
<p>&nbsp;<br>
$$
y_i(\hat{w}^T\hat{x}_i+b) \geq M\vert \vert \hat{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M\vert \vert \boldmath{w}\vert\vert \hspace{0.1cm}\forall i.
$$
<p>&nbsp;<br>
If we scale the equation so that \( \vert \vert \hat{w}\vert\vert = 1/M \), we have to find the minimum of
\( \hat{w}^T\hat{w}=\vert \vert \hat{w}\vert\vert \) (the norm) subject to the condition
If we scale the equation so that \( \vert \vert \boldmath{w}\vert\vert = 1/M \), we have to find the minimum of
\( \boldmath{w}^T\boldmath{w}=\vert \vert \boldmath{w}\vert\vert \) (the norm) subject to the condition
<p>&nbsp;<br>
$$
y_i(\hat{w}^T\hat{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
$$
<p>&nbsp;<br>
<p>
We have thus defined our margin as the invers of the norm of \( \hat{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
We have thus defined our margin as the invers of the norm of \( \boldmath{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
</section>
@@ -572,14 +572,14 @@ $$
In order to solve the above problem, we define the following Lagrangian function to be minimized
<p>&nbsp;<br>
$$
{\cal L}(\lambda,b,\hat{w})=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-1\right],
{\cal L}(\lambda,b,\boldmath{w})=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-1\right],
$$
<p>&nbsp;<br>
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain
<p>&nbsp;<br>
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -589,14 +589,14 @@ $$
and
<p>&nbsp;<br>
$$
\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i.
\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i.
$$
<p>&nbsp;<br>
Inserting these constraints into the equation for \( {\cal L} \) we obtain
<p>&nbsp;<br>
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
<p>&nbsp;<br>
@@ -604,18 +604,18 @@ subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_blank">Karush-Kuhn-Tucker</a> (KKT) condition
<p>&nbsp;<br>
$$
\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
<p>&nbsp;<br>
<ol>
<p><li> If \( \lambda_i > 0 \), then \( y_i(\hat{w}^T\hat{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<p><li> If \( y_i(\hat{w}^T\hat{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
<p><li> If \( \lambda_i > 0 \), then \( y_i(\boldmath{w}^T\boldmath{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<p><li> If \( y_i(\boldmath{w}^T\boldmath{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
<p>
When \( \lambda_i > 0 \), the vectors \( \hat{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
When \( \lambda_i > 0 \), the vectors \( \boldmath{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
</section>
@@ -626,24 +626,24 @@ When \( \lambda_i > 0 \), the vectors \( \hat{x}_i \) are called support vectors
We can rewrite
<p>&nbsp;<br>
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
<p>&nbsp;<br>
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
<p>&nbsp;<br>
$$
\frac{1}{2} \hat{\lambda}^T\begin{bmatrix} y_1y_1\hat{x}_1^T\hat{x}_1 & y_1y_2\hat{x}_1^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_1^T\hat{x}_n \\
y_2y_1\hat{x}_2^T\hat{x}_1 & y_2y_2\hat{x}_2^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_2^T\hat{x}_n \\
\frac{1}{2} \boldmath{\lambda}^T\begin{bmatrix} y_1y_1\boldmath{x}_1^T\boldmath{x}_1 & y_1y_2\boldmath{x}_1^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_1^T\boldmath{x}_n \\
y_2y_1\boldmath{x}_2^T\boldmath{x}_1 & y_2y_2\boldmath{x}_2^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_2^T\boldmath{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1\hat{x}_n^T\hat{x}_1 & y_ny_2\hat{x}_n^T\hat{x}_2 & \dots & \dots & y_ny_n\hat{x}_n^T\hat{x}_n \\
\end{bmatrix}\hat{\lambda}-\mathbb{1}\hat{\lambda},
y_ny_1\boldmath{x}_n^T\boldmath{x}_1 & y_ny_2\boldmath{x}_n^T\boldmath{x}_2 & \dots & \dots & y_ny_n\boldmath{x}_n^T\boldmath{x}_n \\
\end{bmatrix}\boldmath{\lambda}-\mathbb{1}\boldmath{\lambda},
$$
<p>&nbsp;<br>
subject to \( \hat{y}^T\hat{\lambda}=0 \). Here we defined the vectors \( \hat{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \hat{y}=[y_1,y_2,\dots,y_n] \).
subject to \( \boldmath{y}^T\boldmath{\lambda}=0 \). Here we defined the vectors \( \boldmath{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldmath{y}=[y_1,y_2,\dots,y_n] \).
</section>
@@ -655,28 +655,35 @@ Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
<p>&nbsp;<br>
$$
\hat{w}=\sum_{i} \lambda_iy_i\hat{x}_i.
\boldmath{w}=\sum_{i} \lambda_iy_i\boldmath{x}_i.
$$
<p>&nbsp;<br>
With our vector \( \hat{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
With our vector \( \boldmath{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
<p>&nbsp;<br>
$$
y_i(\hat{w}^T\hat{x}_i+b)=1,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
$$
<p>&nbsp;<br>
resulting in
<p>&nbsp;<br>
$$
b = \frac{1}{y_i}-\hat{w}^T\hat{x}_i.
b = \frac{1}{y_i}-\boldmath{w}^T\boldmath{x}_i,
$$
<p>&nbsp;<br>
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
<p>&nbsp;<br>
$$
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldmath{x}_i^T\boldmath{x}_j\right).
$$
<p>&nbsp;<br>
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
<p>&nbsp;<br>
$$
y_i = \mathrm{sign}(\hat{w}^T\hat{x}_i+b).
y_i = \mathrm{sign}(\boldmath{w}^T\boldmath{x}_i+b).
$$
<p>&nbsp;<br>
@@ -697,24 +704,24 @@ so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
<p>
We introduce thus the so-called <b>slack</b> variables \( \hat{\xi} =[\xi_1,x_2,\dots,x_n] \) and
We introduce thus the so-called <b>slack</b> variables \( \boldmath{\xi} =[\xi_1,x_2,\dots,x_n] \) and
modify our previous equation
<p>&nbsp;<br>
$$
y_i(\hat{w}^T\hat{x}_i+b)=1,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
$$
<p>&nbsp;<br>
to
<p>&nbsp;<br>
$$
y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i,
$$
<p>&nbsp;<br>
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
\( y_i(\hat{w}^T\hat{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
\( y_i(\boldmath{w}^T\boldmath{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
<p>
@@ -730,21 +737,21 @@ misclassifications.
This has in turn the consequences that we change our optmization problem to finding the minimum of
<p>&nbsp;<br>
$$
{\cal L}=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
{\cal L}=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
$$
<p>&nbsp;<br>
subject to
<p>&nbsp;<br>
$$
y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
<p>&nbsp;<br>
with the requirement \( \xi_i\geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain
<p>&nbsp;<br>
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -754,7 +761,7 @@ $$
and
<p>&nbsp;<br>
$$
\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i,
\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i,
$$
<p>&nbsp;<br>
@@ -768,7 +775,7 @@ $$
Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before
<p>&nbsp;<br>
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
<p>&nbsp;<br>
@@ -776,7 +783,7 @@ but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
<p>&nbsp;<br>
$$
\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
<p>&nbsp;<br>
@@ -789,7 +796,7 @@ $$
and
<p>&nbsp;<br>
$$
y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
<p>&nbsp;<br>
</section>
doc/pub/svm/html/svm-solarized.html
+62 -57
View File
@@ -164,10 +164,10 @@ $$
where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line
\( b+w_1x_1+w_2x_2=0 \).
In two dimensions we define the vectors \( \hat{x} =[x1,x2] \) and \( \hat{w}=[w1,w2] \).
In two dimensions we define the vectors \( \boldmath{x} =[x1,x2] \) and \( \boldmath{w}=[w1,w2] \).
We can then rewrite the above equation as
$$
\hat{w}^T\hat{x}+b=0.
\boldmath{w}^T\boldmath{x}+b=0.
$$
<p>
@@ -183,25 +183,25 @@ b+wx_1+w_2x_2+\dots +w_px_p=0.
$$
If we define a
matrix \( \hat{X}=\left[\hat{x}_1,\hat{x}_2,\dots, \hat{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \hat{X} \),
matrix \( \boldmath{X}=\left[\boldmath{x}_1,\boldmath{x}_2,\dots, \boldmath{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldmath{X} \),
$$
\hat{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\boldmath{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
$$
If the above condition is not met for a given vector \( \hat{x}_i \) we have
If the above condition is not met for a given vector \( \boldmath{x}_i \) we have
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
$$
if our output \( y_i=1 \).
In this case we say that \( \hat{x}_i \) lies on one of the sides of the hyperplane and if
In this case we say that \( \boldmath{x}_i \) lies on one of the sides of the hyperplane and if
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
$$
for the class of observations \( y_i=-1 \),
then \( \hat{x}_i \) lies on the other side.
then \( \boldmath{x}_i \) lies on the other side.
<p>
Equivalently, for the two classes of observations we have
@@ -247,18 +247,18 @@ for our data sample.
<p>
Let us define the function
$$
f(x) = \hat{w}^T\hat{x}+b = 0,
f(x) = \boldmath{w}^T\boldmath{x}+b = 0,
$$
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
<p>
Any point defined by \( \hat{x}_i \) and \( \hat{x}_2 \) on the line \( L \) will satisfy \( \hat{w}^T(\hat{x}_1-\hat{x}_2)=0 \).
Any point defined by \( \boldmath{x}_i \) and \( \boldmath{x}_2 \) on the line \( L \) will satisfy \( \boldmath{w}^T(\boldmath{x}_1-\boldmath{x}_2)=0 \).
<p>
The signed distance \( \delta \) from any point defined by a vector \( \hat{x} \) and a point \( \hat{x}_0 \) on the line \( L \) is then
The signed distance \( \delta \) from any point defined by a vector \( \boldmath{x} \) and a point \( \boldmath{x}_0 \) on the line \( L \) is then
$$
\delta = \frac{1}{\vert\vert \hat{w}\vert\vert}(\hat{w}^T\hat{x}+b).
\delta = \frac{1}{\vert\vert \boldmath{w}\vert\vert}(\boldmath{w}^T\boldmath{x}+b).
$$
<p>
@@ -267,23 +267,23 @@ $$
<h2 id="___sec6">First attempt at a minimization approach </h2>
<p>
How do we find the parameter \( b \) and the vector \( \hat{w} \)? What we could
How do we find the parameter \( b \) and the vector \( \boldmath{w} \)? What we could
do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
$$
C(\hat{w},b) = -\sum_{i\in M} y_i(\hat{w}^T\hat{x}_i+b).
C(\boldmath{w},b) = -\sum_{i\in M} y_i(\boldmath{w}^T\boldmath{x}_i+b).
$$
<p>
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \hat{w}^T\hat{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldmath{w}^T\boldmath{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
$$
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
$$
and
$$
\frac{\partial C}{\partial \hat{w}} = -\sum_{i\in M} y_ix_i.
\frac{\partial C}{\partial \boldmath{w}} = -\sum_{i\in M} y_ix_i.
$$
<p>
@@ -299,7 +299,7 @@ $$
and
$$
\hat{w} \leftarrow \hat{w} +\eta \frac{\partial C}{\partial \hat{w}},
\boldmath{w} \leftarrow \boldmath{w} +\eta \frac{\partial C}{\partial \boldmath{w}},
$$
where \( \eta \) is our by now well-known learning rate.
@@ -324,11 +324,11 @@ A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).
<p>
Thus, we wish to find a margin \( M \) with \( \hat{w} \) normalized to
\( \vert\vert \hat{w}\vert\vert =1 \) subject to the condition
Thus, we wish to find a margin \( M \) with \( \boldmath{w} \) normalized to
\( \vert\vert \boldmath{w}\vert\vert =1 \) subject to the condition
$$
y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.
@@ -336,22 +336,22 @@ All points are thus at a signed distance from the decision boundary defined by t
<p>
We seek thus the largest value \( M \) defined by
$$
\frac{1}{\vert \vert \hat{w}\vert\vert}y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \boldmath{w}\vert\vert}y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
or just
$$
y_i(\hat{w}^T\hat{x}_i+b) \geq M\vert \vert \hat{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M\vert \vert \boldmath{w}\vert\vert \hspace{0.1cm}\forall i.
$$
If we scale the equation so that \( \vert \vert \hat{w}\vert\vert = 1/M \), we have to find the minimum of
\( \hat{w}^T\hat{w}=\vert \vert \hat{w}\vert\vert \) (the norm) subject to the condition
If we scale the equation so that \( \vert \vert \boldmath{w}\vert\vert = 1/M \), we have to find the minimum of
\( \boldmath{w}^T\boldmath{w}=\vert \vert \boldmath{w}\vert\vert \) (the norm) subject to the condition
$$
y_i(\hat{w}^T\hat{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
$$
<p>
We have thus defined our margin as the invers of the norm of \( \hat{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
We have thus defined our margin as the invers of the norm of \( \boldmath{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -452,40 +452,40 @@ $$
<h2 id="___sec11">Setting up the problem </h2>
In order to solve the above problem, we define the following Lagrangian function to be minimized
$$
{\cal L}(\lambda,b,\hat{w})=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-1\right],
{\cal L}(\lambda,b,\boldmath{w})=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-1\right],
$$
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i.
\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i.
$$
Inserting these constraints into the equation for \( {\cal L} \) we obtain
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_blank">Karush-Kuhn-Tucker</a> (KKT) condition
$$
\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
<ol>
<li> If \( \lambda_i > 0 \), then \( y_i(\hat{w}^T\hat{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\hat{w}^T\hat{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
<li> If \( \lambda_i > 0 \), then \( y_i(\boldmath{w}^T\boldmath{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\boldmath{w}^T\boldmath{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
When \( \lambda_i > 0 \), the vectors \( \hat{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
When \( \lambda_i > 0 \), the vectors \( \boldmath{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -495,21 +495,21 @@ When \( \lambda_i > 0 \), the vectors \( \hat{x}_i \) are called support vectors
<p>
We can rewrite
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
$$
\frac{1}{2} \hat{\lambda}^T\begin{bmatrix} y_1y_1\hat{x}_1^T\hat{x}_1 & y_1y_2\hat{x}_1^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_1^T\hat{x}_n \\
y_2y_1\hat{x}_2^T\hat{x}_1 & y_2y_2\hat{x}_2^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_2^T\hat{x}_n \\
\frac{1}{2} \boldmath{\lambda}^T\begin{bmatrix} y_1y_1\boldmath{x}_1^T\boldmath{x}_1 & y_1y_2\boldmath{x}_1^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_1^T\boldmath{x}_n \\
y_2y_1\boldmath{x}_2^T\boldmath{x}_1 & y_2y_2\boldmath{x}_2^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_2^T\boldmath{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1\hat{x}_n^T\hat{x}_1 & y_ny_2\hat{x}_n^T\hat{x}_2 & \dots & \dots & y_ny_n\hat{x}_n^T\hat{x}_n \\
\end{bmatrix}\hat{\lambda}-\mathbb{1}\hat{\lambda},
y_ny_1\boldmath{x}_n^T\boldmath{x}_1 & y_ny_2\boldmath{x}_n^T\boldmath{x}_2 & \dots & \dots & y_ny_n\boldmath{x}_n^T\boldmath{x}_n \\
\end{bmatrix}\boldmath{\lambda}-\mathbb{1}\boldmath{\lambda},
$$
subject to \( \hat{y}^T\hat{\lambda}=0 \). Here we defined the vectors \( \hat{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \hat{y}=[y_1,y_2,\dots,y_n] \).
subject to \( \boldmath{y}^T\boldmath{\lambda}=0 \). Here we defined the vectors \( \boldmath{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldmath{y}=[y_1,y_2,\dots,y_n] \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -520,22 +520,27 @@ subject to \( \hat{y}^T\hat{\lambda}=0 \). Here we defined the vectors \( \hat{\
Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
$$
\hat{w}=\sum_{i} \lambda_iy_i\hat{x}_i.
\boldmath{w}=\sum_{i} \lambda_iy_i\boldmath{x}_i.
$$
With our vector \( \hat{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
With our vector \( \boldmath{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
y_i(\hat{w}^T\hat{x}_i+b)=1,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
$$
resulting in
$$
b = \frac{1}{y_i}-\hat{w}^T\hat{x}_i.
b = \frac{1}{y_i}-\boldmath{w}^T\boldmath{x}_i,
$$
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
$$
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldmath{x}_i^T\boldmath{x}_j\right).
$$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
y_i = \mathrm{sign}(\hat{w}^T\hat{x}_i+b).
y_i = \mathrm{sign}(\boldmath{w}^T\boldmath{x}_i+b).
$$
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
@@ -555,20 +560,20 @@ so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
<p>
We introduce thus the so-called <b>slack</b> variables \( \hat{\xi} =[\xi_1,x_2,\dots,x_n] \) and
We introduce thus the so-called <b>slack</b> variables \( \boldmath{\xi} =[\xi_1,x_2,\dots,x_n] \) and
modify our previous equation
$$
y_i(\hat{w}^T\hat{x}_i+b)=1,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
$$
to
$$
y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i,
$$
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
\( y_i(\hat{w}^T\hat{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
\( y_i(\boldmath{w}^T\boldmath{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
<p>
@@ -583,25 +588,25 @@ misclassifications.
<p>
This has in turn the consequences that we change our optmization problem to finding the minimum of
$$
{\cal L}=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
{\cal L}=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
$$
subject to
$$
y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
with the requirement \( \xi_i\geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i,
\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i,
$$
and
@@ -611,13 +616,13 @@ $$
Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \).
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
$$
\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
$$
@@ -626,7 +631,7 @@ $$
and
$$
y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
<p>
doc/pub/svm/html/svm.html
+62 -57
View File
@@ -169,10 +169,10 @@ $$
where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line
\( b+w_1x_1+w_2x_2=0 \).
In two dimensions we define the vectors \( \hat{x} =[x1,x2] \) and \( \hat{w}=[w1,w2] \).
In two dimensions we define the vectors \( \boldmath{x} =[x1,x2] \) and \( \boldmath{w}=[w1,w2] \).
We can then rewrite the above equation as
$$
\hat{w}^T\hat{x}+b=0.
\boldmath{w}^T\boldmath{x}+b=0.
$$
<p>
@@ -188,25 +188,25 @@ b+wx_1+w_2x_2+\dots +w_px_p=0.
$$
If we define a
matrix \( \hat{X}=\left[\hat{x}_1,\hat{x}_2,\dots, \hat{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \hat{X} \),
matrix \( \boldmath{X}=\left[\boldmath{x}_1,\boldmath{x}_2,\dots, \boldmath{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldmath{X} \),
$$
\hat{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\boldmath{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
$$
If the above condition is not met for a given vector \( \hat{x}_i \) we have
If the above condition is not met for a given vector \( \boldmath{x}_i \) we have
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
$$
if our output \( y_i=1 \).
In this case we say that \( \hat{x}_i \) lies on one of the sides of the hyperplane and if
In this case we say that \( \boldmath{x}_i \) lies on one of the sides of the hyperplane and if
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
$$
for the class of observations \( y_i=-1 \),
then \( \hat{x}_i \) lies on the other side.
then \( \boldmath{x}_i \) lies on the other side.
<p>
Equivalently, for the two classes of observations we have
@@ -252,18 +252,18 @@ for our data sample.
<p>
Let us define the function
$$
f(x) = \hat{w}^T\hat{x}+b = 0,
f(x) = \boldmath{w}^T\boldmath{x}+b = 0,
$$
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
<p>
Any point defined by \( \hat{x}_i \) and \( \hat{x}_2 \) on the line \( L \) will satisfy \( \hat{w}^T(\hat{x}_1-\hat{x}_2)=0 \).
Any point defined by \( \boldmath{x}_i \) and \( \boldmath{x}_2 \) on the line \( L \) will satisfy \( \boldmath{w}^T(\boldmath{x}_1-\boldmath{x}_2)=0 \).
<p>
The signed distance \( \delta \) from any point defined by a vector \( \hat{x} \) and a point \( \hat{x}_0 \) on the line \( L \) is then
The signed distance \( \delta \) from any point defined by a vector \( \boldmath{x} \) and a point \( \boldmath{x}_0 \) on the line \( L \) is then
$$
\delta = \frac{1}{\vert\vert \hat{w}\vert\vert}(\hat{w}^T\hat{x}+b).
\delta = \frac{1}{\vert\vert \boldmath{w}\vert\vert}(\boldmath{w}^T\boldmath{x}+b).
$$
<p>
@@ -272,23 +272,23 @@ $$
<h2 id="___sec6">First attempt at a minimization approach </h2>
<p>
How do we find the parameter \( b \) and the vector \( \hat{w} \)? What we could
How do we find the parameter \( b \) and the vector \( \boldmath{w} \)? What we could
do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
$$
C(\hat{w},b) = -\sum_{i\in M} y_i(\hat{w}^T\hat{x}_i+b).
C(\boldmath{w},b) = -\sum_{i\in M} y_i(\boldmath{w}^T\boldmath{x}_i+b).
$$
<p>
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \hat{w}^T\hat{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldmath{w}^T\boldmath{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
$$
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
$$
and
$$
\frac{\partial C}{\partial \hat{w}} = -\sum_{i\in M} y_ix_i.
\frac{\partial C}{\partial \boldmath{w}} = -\sum_{i\in M} y_ix_i.
$$
<p>
@@ -304,7 +304,7 @@ $$
and
$$
\hat{w} \leftarrow \hat{w} +\eta \frac{\partial C}{\partial \hat{w}},
\boldmath{w} \leftarrow \boldmath{w} +\eta \frac{\partial C}{\partial \boldmath{w}},
$$
where \( \eta \) is our by now well-known learning rate.
@@ -329,11 +329,11 @@ A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).
<p>
Thus, we wish to find a margin \( M \) with \( \hat{w} \) normalized to
\( \vert\vert \hat{w}\vert\vert =1 \) subject to the condition
Thus, we wish to find a margin \( M \) with \( \boldmath{w} \) normalized to
\( \vert\vert \boldmath{w}\vert\vert =1 \) subject to the condition
$$
y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.
@@ -341,22 +341,22 @@ All points are thus at a signed distance from the decision boundary defined by t
<p>
We seek thus the largest value \( M \) defined by
$$
\frac{1}{\vert \vert \hat{w}\vert\vert}y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \boldmath{w}\vert\vert}y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
or just
$$
y_i(\hat{w}^T\hat{x}_i+b) \geq M\vert \vert \hat{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M\vert \vert \boldmath{w}\vert\vert \hspace{0.1cm}\forall i.
$$
If we scale the equation so that \( \vert \vert \hat{w}\vert\vert = 1/M \), we have to find the minimum of
\( \hat{w}^T\hat{w}=\vert \vert \hat{w}\vert\vert \) (the norm) subject to the condition
If we scale the equation so that \( \vert \vert \boldmath{w}\vert\vert = 1/M \), we have to find the minimum of
\( \boldmath{w}^T\boldmath{w}=\vert \vert \boldmath{w}\vert\vert \) (the norm) subject to the condition
$$
y_i(\hat{w}^T\hat{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
$$
<p>
We have thus defined our margin as the invers of the norm of \( \hat{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
We have thus defined our margin as the invers of the norm of \( \boldmath{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -457,40 +457,40 @@ $$
<h2 id="___sec11">Setting up the problem </h2>
In order to solve the above problem, we define the following Lagrangian function to be minimized
$$
{\cal L}(\lambda,b,\hat{w})=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-1\right],
{\cal L}(\lambda,b,\boldmath{w})=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-1\right],
$$
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i.
\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i.
$$
Inserting these constraints into the equation for \( {\cal L} \) we obtain
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_blank">Karush-Kuhn-Tucker</a> (KKT) condition
$$
\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
<ol>
<li> If \( \lambda_i > 0 \), then \( y_i(\hat{w}^T\hat{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\hat{w}^T\hat{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
<li> If \( \lambda_i > 0 \), then \( y_i(\boldmath{w}^T\boldmath{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\boldmath{w}^T\boldmath{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
When \( \lambda_i > 0 \), the vectors \( \hat{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
When \( \lambda_i > 0 \), the vectors \( \boldmath{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -500,21 +500,21 @@ When \( \lambda_i > 0 \), the vectors \( \hat{x}_i \) are called support vectors
<p>
We can rewrite
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
$$
\frac{1}{2} \hat{\lambda}^T\begin{bmatrix} y_1y_1\hat{x}_1^T\hat{x}_1 & y_1y_2\hat{x}_1^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_1^T\hat{x}_n \\
y_2y_1\hat{x}_2^T\hat{x}_1 & y_2y_2\hat{x}_2^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_2^T\hat{x}_n \\
\frac{1}{2} \boldmath{\lambda}^T\begin{bmatrix} y_1y_1\boldmath{x}_1^T\boldmath{x}_1 & y_1y_2\boldmath{x}_1^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_1^T\boldmath{x}_n \\
y_2y_1\boldmath{x}_2^T\boldmath{x}_1 & y_2y_2\boldmath{x}_2^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_2^T\boldmath{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1\hat{x}_n^T\hat{x}_1 & y_ny_2\hat{x}_n^T\hat{x}_2 & \dots & \dots & y_ny_n\hat{x}_n^T\hat{x}_n \\
\end{bmatrix}\hat{\lambda}-\mathbb{1}\hat{\lambda},
y_ny_1\boldmath{x}_n^T\boldmath{x}_1 & y_ny_2\boldmath{x}_n^T\boldmath{x}_2 & \dots & \dots & y_ny_n\boldmath{x}_n^T\boldmath{x}_n \\
\end{bmatrix}\boldmath{\lambda}-\mathbb{1}\boldmath{\lambda},
$$
subject to \( \hat{y}^T\hat{\lambda}=0 \). Here we defined the vectors \( \hat{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \hat{y}=[y_1,y_2,\dots,y_n] \).
subject to \( \boldmath{y}^T\boldmath{\lambda}=0 \). Here we defined the vectors \( \boldmath{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldmath{y}=[y_1,y_2,\dots,y_n] \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -525,22 +525,27 @@ subject to \( \hat{y}^T\hat{\lambda}=0 \). Here we defined the vectors \( \hat{\
Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
$$
\hat{w}=\sum_{i} \lambda_iy_i\hat{x}_i.
\boldmath{w}=\sum_{i} \lambda_iy_i\boldmath{x}_i.
$$
With our vector \( \hat{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
With our vector \( \boldmath{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
y_i(\hat{w}^T\hat{x}_i+b)=1,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
$$
resulting in
$$
b = \frac{1}{y_i}-\hat{w}^T\hat{x}_i.
b = \frac{1}{y_i}-\boldmath{w}^T\boldmath{x}_i,
$$
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
$$
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldmath{x}_i^T\boldmath{x}_j\right).
$$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
y_i = \mathrm{sign}(\hat{w}^T\hat{x}_i+b).
y_i = \mathrm{sign}(\boldmath{w}^T\boldmath{x}_i+b).
$$
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
@@ -560,20 +565,20 @@ so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
<p>
We introduce thus the so-called <b>slack</b> variables \( \hat{\xi} =[\xi_1,x_2,\dots,x_n] \) and
We introduce thus the so-called <b>slack</b> variables \( \boldmath{\xi} =[\xi_1,x_2,\dots,x_n] \) and
modify our previous equation
$$
y_i(\hat{w}^T\hat{x}_i+b)=1,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
$$
to
$$
y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i,
$$
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
\( y_i(\hat{w}^T\hat{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
\( y_i(\boldmath{w}^T\boldmath{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
<p>
@@ -588,25 +593,25 @@ misclassifications.
<p>
This has in turn the consequences that we change our optmization problem to finding the minimum of
$$
{\cal L}=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
{\cal L}=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
$$
subject to
$$
y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
with the requirement \( \xi_i\geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i,
\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i,
$$
and
@@ -616,13 +621,13 @@ $$
Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \).
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
$$
\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
$$
@@ -631,7 +636,7 @@ $$
and
$$
y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
<p>
doc/pub/svm/ipynb/ipynb-svm-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/svm/ipynb/svm.ipynb
+74 -58
View File
@@ -79,7 +79,7 @@
"source": [
"where $b$ is the intercept and $w_1$ and $w_2$ define the elements of a vector orthogonal to the line \n",
"$b+w_1x_1+w_2x_2=0$. \n",
"In two dimensions we define the vectors $\\hat{x} =[x1,x2]$ and $\\hat{w}=[w1,w2]$. \n",
"In two dimensions we define the vectors $\\boldmath{x} =[x1,x2]$ and $\\boldmath{w}=[w1,w2]$. \n",
"We can then rewrite the above equation as"
]
},
@@ -88,7 +88,7 @@
"metadata": {},
"source": [
"$$\n",
"\\hat{w}^T\\hat{x}+b=0.\n",
"\\boldmath{w}^T\\boldmath{x}+b=0.\n",
"$$"
]
},
@@ -116,8 +116,8 @@
"metadata": {},
"source": [
"If we define a \n",
"matrix $\\hat{X}=\\left[\\hat{x}_1,\\hat{x}_2,\\dots, \\hat{x}_p\\right]$\n",
"of dimension $n\\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\\hat{X}$,"
"matrix $\\boldmath{X}=\\left[\\boldmath{x}_1,\\boldmath{x}_2,\\dots, \\boldmath{x}_p\\right]$\n",
"of dimension $n\\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\\boldmath{X}$,"
]
},
{
@@ -125,7 +125,7 @@
"metadata": {},
"source": [
"$$\n",
"\\hat{x}_i = \\begin{bmatrix} x_{i1} \\\\ x_{i2} \\\\ \\dots \\\\ \\dots \\\\ x_{ip} \\end{bmatrix}.\n",
"\\boldmath{x}_i = \\begin{bmatrix} x_{i1} \\\\ x_{i2} \\\\ \\dots \\\\ \\dots \\\\ x_{ip} \\end{bmatrix}.\n",
"$$"
]
},
@@ -133,7 +133,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"If the above condition is not met for a given vector $\\hat{x}_i$ we have"
"If the above condition is not met for a given vector $\\boldmath{x}_i$ we have"
]
},
{
@@ -150,7 +150,7 @@
"metadata": {},
"source": [
"if our output $y_i=1$.\n",
"In this case we say that $\\hat{x}_i$ lies on one of the sides of the hyperplane and if"
"In this case we say that $\\boldmath{x}_i$ lies on one of the sides of the hyperplane and if"
]
},
{
@@ -167,7 +167,7 @@
"metadata": {},
"source": [
"for the class of observations $y_i=-1$, \n",
"then $\\hat{x}_i$ lies on the other side. \n",
"then $\\boldmath{x}_i$ lies on the other side. \n",
"\n",
"Equivalently, for the two classes of observations we have"
]
@@ -219,7 +219,7 @@
"metadata": {},
"source": [
"$$\n",
"f(x) = \\hat{w}^T\\hat{x}+b = 0,\n",
"f(x) = \\boldmath{w}^T\\boldmath{x}+b = 0,\n",
"$$"
]
},
@@ -230,9 +230,9 @@
"as the function that determines the line $L$ that separates two classes (our two features), see the figure here. \n",
"\n",
"\n",
"Any point defined by $\\hat{x}_i$ and $\\hat{x}_2$ on the line $L$ will satisfy $\\hat{w}^T(\\hat{x}_1-\\hat{x}_2)=0$. \n",
"Any point defined by $\\boldmath{x}_i$ and $\\boldmath{x}_2$ on the line $L$ will satisfy $\\boldmath{w}^T(\\boldmath{x}_1-\\boldmath{x}_2)=0$. \n",
"\n",
"The signed distance $\\delta$ from any point defined by a vector $\\hat{x}$ and a point $\\hat{x}_0$ on the line $L$ is then"
"The signed distance $\\delta$ from any point defined by a vector $\\boldmath{x}$ and a point $\\boldmath{x}_0$ on the line $L$ is then"
]
},
{
@@ -240,7 +240,7 @@
"metadata": {},
"source": [
"$$\n",
"\\delta = \\frac{1}{\\vert\\vert \\hat{w}\\vert\\vert}(\\hat{w}^T\\hat{x}+b).\n",
"\\delta = \\frac{1}{\\vert\\vert \\boldmath{w}\\vert\\vert}(\\boldmath{w}^T\\boldmath{x}+b).\n",
"$$"
]
},
@@ -250,7 +250,7 @@
"source": [
"## First attempt at a minimization approach\n",
"\n",
"How do we find the parameter $b$ and the vector $\\hat{w}$? What we could\n",
"How do we find the parameter $b$ and the vector $\\boldmath{w}$? What we could\n",
"do is to define a cost function which now contains the set of all\n",
"misclassified points $M$ and attempt to minimize this function"
]
@@ -260,7 +260,7 @@
"metadata": {},
"source": [
"$$\n",
"C(\\hat{w},b) = -\\sum_{i\\in M} y_i(\\hat{w}^T\\hat{x}_i+b).\n",
"C(\\boldmath{w},b) = -\\sum_{i\\in M} y_i(\\boldmath{w}^T\\boldmath{x}_i+b).\n",
"$$"
]
},
@@ -268,7 +268,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We could now for example define all values $y_i =1$ as misclassified in case we have $\\hat{w}^T\\hat{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us"
"We could now for example define all values $y_i =1$ as misclassified in case we have $\\boldmath{w}^T\\boldmath{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us"
]
},
{
@@ -292,7 +292,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial \\hat{w}} = -\\sum_{i\\in M} y_ix_i.\n",
"\\frac{\\partial C}{\\partial \\boldmath{w}} = -\\sum_{i\\in M} y_ix_i.\n",
"$$"
]
},
@@ -326,7 +326,7 @@
"metadata": {},
"source": [
"$$\n",
"\\hat{w} \\leftarrow \\hat{w} +\\eta \\frac{\\partial C}{\\partial \\hat{w}},\n",
"\\boldmath{w} \\leftarrow \\boldmath{w} +\\eta \\frac{\\partial C}{\\partial \\boldmath{w}},\n",
"$$"
]
},
@@ -350,8 +350,8 @@
"A better approach is rather to try to define a large margin between\n",
"the two classes (if they are well separated from the beginning).\n",
"\n",
"Thus, we wish to find a margin $M$ with $\\hat{w}$ normalized to\n",
"$\\vert\\vert \\hat{w}\\vert\\vert =1$ subject to the condition"
"Thus, we wish to find a margin $M$ with $\\boldmath{w}$ normalized to\n",
"$\\vert\\vert \\boldmath{w}\\vert\\vert =1$ subject to the condition"
]
},
{
@@ -359,7 +359,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\hat{w}^T\\hat{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, p.\n",
"y_i(\\boldmath{w}^T\\boldmath{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, p.\n",
"$$"
]
},
@@ -377,7 +377,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{1}{\\vert \\vert \\hat{w}\\vert\\vert}y_i(\\hat{w}^T\\hat{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, n,\n",
"\\frac{1}{\\vert \\vert \\boldmath{w}\\vert\\vert}y_i(\\boldmath{w}^T\\boldmath{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, n,\n",
"$$"
]
},
@@ -393,7 +393,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\hat{w}^T\\hat{x}_i+b) \\geq M\\vert \\vert \\hat{w}\\vert\\vert \\hspace{0.1cm}\\forall i.\n",
"y_i(\\boldmath{w}^T\\boldmath{x}_i+b) \\geq M\\vert \\vert \\boldmath{w}\\vert\\vert \\hspace{0.1cm}\\forall i.\n",
"$$"
]
},
@@ -401,8 +401,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"If we scale the equation so that $\\vert \\vert \\hat{w}\\vert\\vert = 1/M$, we have to find the minimum of \n",
"$\\hat{w}^T\\hat{w}=\\vert \\vert \\hat{w}\\vert\\vert$ (the norm) subject to the condition"
"If we scale the equation so that $\\vert \\vert \\boldmath{w}\\vert\\vert = 1/M$, we have to find the minimum of \n",
"$\\boldmath{w}^T\\boldmath{w}=\\vert \\vert \\boldmath{w}\\vert\\vert$ (the norm) subject to the condition"
]
},
{
@@ -410,7 +410,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\hat{w}^T\\hat{x}_i+b) \\geq 1 \\hspace{0.1cm}\\forall i.\n",
"y_i(\\boldmath{w}^T\\boldmath{x}_i+b) \\geq 1 \\hspace{0.1cm}\\forall i.\n",
"$$"
]
},
@@ -418,7 +418,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We have thus defined our margin as the invers of the norm of $\\hat{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers. \n",
"We have thus defined our margin as the invers of the norm of $\\boldmath{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers. \n",
"\n",
"## A quick reminder on Lagrangian multipliers\n",
"\n",
@@ -644,7 +644,7 @@
"metadata": {},
"source": [
"$$\n",
"{\\cal L}(\\lambda,b,\\hat{w})=\\frac{1}{2}\\hat{w}^T\\hat{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\hat{w}^T\\hat{x}_i+b)-1\\right],\n",
"{\\cal L}(\\lambda,b,\\boldmath{w})=\\frac{1}{2}\\boldmath{w}^T\\boldmath{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\boldmath{w}^T\\boldmath{x}_i+b)-1\\right],\n",
"$$"
]
},
@@ -654,7 +654,7 @@
"source": [
"where $\\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\\lambda_i \\geq 0$.\n",
"\n",
"Taking the derivatives with respect to $b$ and $\\hat{w}$ we obtain"
"Taking the derivatives with respect to $b$ and $\\boldmath{w}$ we obtain"
]
},
{
@@ -678,7 +678,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial {\\cal L}}{\\partial \\hat{w}} = 0 = \\hat{w}-\\sum_{i} \\lambda_iy_i\\hat{x}_i.\n",
"\\frac{\\partial {\\cal L}}{\\partial \\boldmath{w}} = 0 = \\boldmath{w}-\\sum_{i} \\lambda_iy_i\\boldmath{x}_i.\n",
"$$"
]
},
@@ -694,7 +694,7 @@
"metadata": {},
"source": [
"$$\n",
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\hat{x}_i^T\\hat{x}_j,\n",
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldmath{x}_i^T\\boldmath{x}_j,\n",
"$$"
]
},
@@ -711,7 +711,7 @@
"metadata": {},
"source": [
"$$\n",
"\\lambda_i\\left[y_i(\\hat{w}^T\\hat{x}_i+b) -1\\right] \\hspace{0.1cm}\\forall i.\n",
"\\lambda_i\\left[y_i(\\boldmath{w}^T\\boldmath{x}_i+b) -1\\right] \\hspace{0.1cm}\\forall i.\n",
"$$"
]
},
@@ -719,11 +719,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"1. If $\\lambda_i > 0$, then $y_i(\\hat{w}^T\\hat{x}_i+b)=1$ and we say that $x_i$ is on the boundary.\n",
"1. If $\\lambda_i > 0$, then $y_i(\\boldmath{w}^T\\boldmath{x}_i+b)=1$ and we say that $x_i$ is on the boundary.\n",
"\n",
"2. If $y_i(\\hat{w}^T\\hat{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\\lambda_i=0$. \n",
"2. If $y_i(\\boldmath{w}^T\\boldmath{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\\lambda_i=0$. \n",
"\n",
"When $\\lambda_i > 0$, the vectors $\\hat{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. \n",
"When $\\lambda_i > 0$, the vectors $\\boldmath{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. \n",
"\n",
"## The problem to solve\n",
"\n",
@@ -735,7 +735,7 @@
"metadata": {},
"source": [
"$$\n",
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\hat{x}_i^T\\hat{x}_j,\n",
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldmath{x}_i^T\\boldmath{x}_j,\n",
"$$"
]
},
@@ -751,12 +751,12 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{1}{2} \\hat{\\lambda}^T\\begin{bmatrix} y_1y_1\\hat{x}_1^T\\hat{x}_1 & y_1y_2\\hat{x}_1^T\\hat{x}_2 & \\dots & \\dots & y_1y_n\\hat{x}_1^T\\hat{x}_n \\\\\n",
"y_2y_1\\hat{x}_2^T\\hat{x}_1 & y_2y_2\\hat{x}_2^T\\hat{x}_2 & \\dots & \\dots & y_1y_n\\hat{x}_2^T\\hat{x}_n \\\\\n",
"\\frac{1}{2} \\boldmath{\\lambda}^T\\begin{bmatrix} y_1y_1\\boldmath{x}_1^T\\boldmath{x}_1 & y_1y_2\\boldmath{x}_1^T\\boldmath{x}_2 & \\dots & \\dots & y_1y_n\\boldmath{x}_1^T\\boldmath{x}_n \\\\\n",
"y_2y_1\\boldmath{x}_2^T\\boldmath{x}_1 & y_2y_2\\boldmath{x}_2^T\\boldmath{x}_2 & \\dots & \\dots & y_1y_n\\boldmath{x}_2^T\\boldmath{x}_n \\\\\n",
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
"y_ny_1\\hat{x}_n^T\\hat{x}_1 & y_ny_2\\hat{x}_n^T\\hat{x}_2 & \\dots & \\dots & y_ny_n\\hat{x}_n^T\\hat{x}_n \\\\\n",
"\\end{bmatrix}\\hat{\\lambda}-\\mathbb{1}\\hat{\\lambda},\n",
"y_ny_1\\boldmath{x}_n^T\\boldmath{x}_1 & y_ny_2\\boldmath{x}_n^T\\boldmath{x}_2 & \\dots & \\dots & y_ny_n\\boldmath{x}_n^T\\boldmath{x}_n \\\\\n",
"\\end{bmatrix}\\boldmath{\\lambda}-\\mathbb{1}\\boldmath{\\lambda},\n",
"$$"
]
},
@@ -764,8 +764,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"subject to $\\hat{y}^T\\hat{\\lambda}=0$. Here we defined the vectors $\\hat{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n",
"$\\hat{y}=[y_1,y_2,\\dots,y_n]$. \n",
"subject to $\\boldmath{y}^T\\boldmath{\\lambda}=0$. Here we defined the vectors $\\boldmath{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n",
"$\\boldmath{y}=[y_1,y_2,\\dots,y_n]$. \n",
"\n",
"\n",
"## The last steps\n",
@@ -779,7 +779,7 @@
"metadata": {},
"source": [
"$$\n",
"\\hat{w}=\\sum_{i} \\lambda_iy_i\\hat{x}_i.\n",
"\\boldmath{w}=\\sum_{i} \\lambda_iy_i\\boldmath{x}_i.\n",
"$$"
]
},
@@ -787,7 +787,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"With our vector $\\hat{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via"
"With our vector $\\boldmath{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via"
]
},
{
@@ -795,7 +795,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\hat{w}^T\\hat{x}_i+b)=1,\n",
"y_i(\\boldmath{w}^T\\boldmath{x}_i+b)=1,\n",
"$$"
]
},
@@ -811,7 +811,23 @@
"metadata": {},
"source": [
"$$\n",
"b = \\frac{1}{y_i}-\\hat{w}^T\\hat{x}_i.\n",
"b = \\frac{1}{y_i}-\\boldmath{w}^T\\boldmath{x}_i,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"or if we write it out in terms of the support vectors only, with $N_s$ being their number, we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"b = \\frac{1}{N_s}\\sum_{j\\in N_s}\\left(y_j-\\sum_{i=1}^n\\lambda_iy_i\\boldmath{x}_i^T\\boldmath{x}_j\\right).\n",
"$$"
]
},
@@ -827,7 +843,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i = \\mathrm{sign}(\\hat{w}^T\\hat{x}_i+b).\n",
"y_i = \\mathrm{sign}(\\boldmath{w}^T\\boldmath{x}_i+b).\n",
"$$"
]
},
@@ -846,7 +862,7 @@
"so-called **kernel approach**, is to allow a kind of slack in the sense\n",
"that we allow some points to be on the wrong side of the margin.\n",
"\n",
"We introduce thus the so-called **slack** variables $\\hat{\\xi} =[\\xi_1,x_2,\\dots,x_n]$ and \n",
"We introduce thus the so-called **slack** variables $\\boldmath{\\xi} =[\\xi_1,x_2,\\dots,x_n]$ and \n",
"modify our previous equation"
]
},
@@ -855,7 +871,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\hat{w}^T\\hat{x}_i+b)=1,\n",
"y_i(\\boldmath{w}^T\\boldmath{x}_i+b)=1,\n",
"$$"
]
},
@@ -871,7 +887,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\hat{w}^T\\hat{x}_i+b)=1-\\xi_i,\n",
"y_i(\\boldmath{w}^T\\boldmath{x}_i+b)=1-\\xi_i,\n",
"$$"
]
},
@@ -881,7 +897,7 @@
"source": [
"with the requirement $\\xi_i\\geq 0$. The total violation is now $\\sum_i\\xi$. \n",
"The value $\\xi_i$ in the constraint the last constraint corresponds to the amount by which the prediction\n",
"$y_i(\\hat{w}^T\\hat{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\\sum_i \\xi_i$,\n",
"$y_i(\\boldmath{w}^T\\boldmath{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\\sum_i \\xi_i$,\n",
"we bound the total amount by which predictions fall on the wrong side of their margins.\n",
"\n",
"Misclassifications occur when $\\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of\n",
@@ -898,7 +914,7 @@
"metadata": {},
"source": [
"$$\n",
"{\\cal L}=\\frac{1}{2}\\hat{w}^T\\hat{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\hat{w}^T\\hat{x}_i+b)-(1-\\xi_)\\right]+C\\sum_{i=1}^n\\xi_i-\\sum_{i=1}^n\\gamma_i\\xi_i,\n",
"{\\cal L}=\\frac{1}{2}\\boldmath{w}^T\\boldmath{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\boldmath{w}^T\\boldmath{x}_i+b)-(1-\\xi_)\\right]+C\\sum_{i=1}^n\\xi_i-\\sum_{i=1}^n\\gamma_i\\xi_i,\n",
"$$"
]
},
@@ -914,7 +930,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\hat{w}^T\\hat{x}_i+b)=1-\\xi_i \\hspace{0.1cm}\\forall i,\n",
"y_i(\\boldmath{w}^T\\boldmath{x}_i+b)=1-\\xi_i \\hspace{0.1cm}\\forall i,\n",
"$$"
]
},
@@ -924,7 +940,7 @@
"source": [
"with the requirement $\\xi_i\\geq 0$.\n",
"\n",
"Taking the derivatives with respect to $b$ and $\\hat{w}$ we obtain"
"Taking the derivatives with respect to $b$ and $\\boldmath{w}$ we obtain"
]
},
{
@@ -948,7 +964,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial {\\cal L}}{\\partial \\hat{w}} = 0 = \\hat{w}-\\sum_{i} \\lambda_iy_i\\hat{x}_i,\n",
"\\frac{\\partial {\\cal L}}{\\partial \\boldmath{w}} = 0 = \\boldmath{w}-\\sum_{i} \\lambda_iy_i\\boldmath{x}_i,\n",
"$$"
]
},
@@ -980,7 +996,7 @@
"metadata": {},
"source": [
"$$\n",
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\hat{x}_i^T\\hat{x}_j,\n",
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldmath{x}_i^T\\boldmath{x}_j,\n",
"$$"
]
},
@@ -996,8 +1012,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"4\n",
"9\n",
"5\n",
"0\n",
" \n",
"<\n",
"<\n",
@@ -1037,7 +1053,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\hat{w}^T\\hat{x}_i+b) -(1-\\xi_) \\geq 0 \\hspace{0.1cm}\\forall i.\n",
"y_i(\\boldmath{w}^T\\boldmath{x}_i+b) -(1-\\xi_) \\geq 0 \\hspace{0.1cm}\\forall i.\n",
"$$"
]
},
doc/pub/svm/pdf/svm-minted.pdf
BIN
View File
Binary file not shown.
doc/src/SupportVMachines/svm.do.txt
+63 -57
View File
@@ -58,11 +58,11 @@ b+w_1x_1+w_2x_2=0,
!et
where $b$ is the intercept and $w_1$ and $w_2$ define the elements of a vector orthogonal to the line
$b+w_1x_1+w_2x_2=0$.
In two dimensions we define the vectors $\hat{x} =[x1,x2]$ and $\hat{w}=[w1,w2]$.
In two dimensions we define the vectors $\boldmath{x} =[x1,x2]$ and $\boldmath{w}=[w1,w2]$.
We can then rewrite the above equation as
!bt
\[
\hat{w}^T\hat{x}+b=0.
\boldmath{w}^T\boldmath{x}+b=0.
\]
!et
@@ -77,28 +77,28 @@ b+wx_1+w_2x_2+\dots +w_px_p=0.
\]
!et
If we define a
matrix $\hat{X}=\left[\hat{x}_1,\hat{x}_2,\dots, \hat{x}_p\right]$
of dimension $n\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\hat{X}$,
matrix $\boldmath{X}=\left[\boldmath{x}_1,\boldmath{x}_2,\dots, \boldmath{x}_p\right]$
of dimension $n\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\boldmath{X}$,
!bt
\[
\hat{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\boldmath{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\]
!et
If the above condition is not met for a given vector $\hat{x}_i$ we have
If the above condition is not met for a given vector $\boldmath{x}_i$ we have
!bt
\[
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
\]
!et
if our output $y_i=1$.
In this case we say that $\hat{x}_i$ lies on one of the sides of the hyperplane and if
In this case we say that $\boldmath{x}_i$ lies on one of the sides of the hyperplane and if
!bt
\[
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
\]
!et
for the class of observations $y_i=-1$,
then $\hat{x}_i$ lies on the other side.
then $\boldmath{x}_i$ lies on the other side.
Equivalently, for the two classes of observations we have
!bt
@@ -137,35 +137,35 @@ for our data sample.
Let us define the function
!bt
\[
f(x) = \hat{w}^T\hat{x}+b = 0,
f(x) = \boldmath{w}^T\boldmath{x}+b = 0,
\]
!et
as the function that determines the line $L$ that separates two classes (our two features), see the figure here.
Any point defined by $\hat{x}_i$ and $\hat{x}_2$ on the line $L$ will satisfy $\hat{w}^T(\hat{x}_1-\hat{x}_2)=0$.
Any point defined by $\boldmath{x}_i$ and $\boldmath{x}_2$ on the line $L$ will satisfy $\boldmath{w}^T(\boldmath{x}_1-\boldmath{x}_2)=0$.
The signed distance $\delta$ from any point defined by a vector $\hat{x}$ and a point $\hat{x}_0$ on the line $L$ is then
The signed distance $\delta$ from any point defined by a vector $\boldmath{x}$ and a point $\boldmath{x}_0$ on the line $L$ is then
!bt
\[
\delta = \frac{1}{\vert\vert \hat{w}\vert\vert}(\hat{w}^T\hat{x}+b).
\delta = \frac{1}{\vert\vert \boldmath{w}\vert\vert}(\boldmath{w}^T\boldmath{x}+b).
\]
!et
!split
===== First attempt at a minimization approach =====
How do we find the parameter $b$ and the vector $\hat{w}$? What we could
How do we find the parameter $b$ and the vector $\boldmath{w}$? What we could
do is to define a cost function which now contains the set of all
misclassified points $M$ and attempt to minimize this function
!bt
\[
C(\hat{w},b) = -\sum_{i\in M} y_i(\hat{w}^T\hat{x}_i+b).
C(\boldmath{w},b) = -\sum_{i\in M} y_i(\boldmath{w}^T\boldmath{x}_i+b).
\]
!et
We could now for example define all values $y_i =1$ as misclassified in case we have $\hat{w}^T\hat{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us
We could now for example define all values $y_i =1$ as misclassified in case we have $\boldmath{w}^T\boldmath{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us
!bt
\[
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
@@ -174,7 +174,7 @@ We could now for example define all values $y_i =1$ as misclassified in case we
and
!bt
\[
\frac{\partial C}{\partial \hat{w}} = -\sum_{i\in M} y_ix_i.
\frac{\partial C}{\partial \boldmath{w}} = -\sum_{i\in M} y_ix_i.
\]
!et
@@ -190,7 +190,7 @@ b \leftarrow b +\eta \frac{\partial C}{\partial b},
and
!bt
\[
\hat{w} \leftarrow \hat{w} +\eta \frac{\partial C}{\partial \hat{w}},
\boldmath{w} \leftarrow \boldmath{w} +\eta \frac{\partial C}{\partial \boldmath{w}},
\]
!et
where $\eta$ is our by now well-known learning rate.
@@ -210,12 +210,12 @@ at all.
A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).
Thus, we wish to find a margin $M$ with $\hat{w}$ normalized to
$\vert\vert \hat{w}\vert\vert =1$ subject to the condition
Thus, we wish to find a margin $M$ with $\boldmath{w}$ normalized to
$\vert\vert \boldmath{w}\vert\vert =1$ subject to the condition
!bt
\[
y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
\]
!et
All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $b$ and $w_1$ and $w_2$ define this line.
@@ -223,24 +223,24 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value $M$ defined by
!bt
\[
\frac{1}{\vert \vert \hat{w}\vert\vert}y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \boldmath{w}\vert\vert}y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\]
!et
or just
!bt
\[
y_i(\hat{w}^T\hat{x}_i+b) \geq M\vert \vert \hat{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M\vert \vert \boldmath{w}\vert\vert \hspace{0.1cm}\forall i.
\]
!et
If we scale the equation so that $\vert \vert \hat{w}\vert\vert = 1/M$, we have to find the minimum of
$\hat{w}^T\hat{w}=\vert \vert \hat{w}\vert\vert$ (the norm) subject to the condition
If we scale the equation so that $\vert \vert \boldmath{w}\vert\vert = 1/M$, we have to find the minimum of
$\boldmath{w}^T\boldmath{w}=\vert \vert \boldmath{w}\vert\vert$ (the norm) subject to the condition
!bt
\[
y_i(\hat{w}^T\hat{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
\]
!et
We have thus defined our margin as the invers of the norm of $\hat{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers.
We have thus defined our margin as the invers of the norm of $\boldmath{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers.
!split
===== A quick reminder on Lagrangian multipliers =====
@@ -346,12 +346,12 @@ If we have a set of constraints $\phi_k$ we have the equations
In order to solve the above problem, we define the following Lagrangian function to be minimized
!bt
\[
{\cal L}(\lambda,b,\hat{w})=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-1\right],
{\cal L}(\lambda,b,\boldmath{w})=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-1\right],
\]
!et
where $\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\lambda_i \geq 0$.
Taking the derivatives with respect to $b$ and $\hat{w}$ we obtain
Taking the derivatives with respect to $b$ and $\boldmath{w}$ we obtain
!bt
\[
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -360,25 +360,25 @@ Taking the derivatives with respect to $b$ and $\hat{w}$ we obtain
and
!bt
\[
\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i.
\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i.
\]
!et
Inserting these constraints into the equation for ${\cal L}$ we obtain
!bt
\[
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
\]
!et
subject to the constraints $\lambda_i\geq 0$ and $\sum_i\lambda_iy_i=0$.
We must in addition satisfy the "Karush-Kuhn-Tucker":"https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" (KKT) condition
!bt
\[
\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\]
!et
o If $\lambda_i > 0$, then $y_i(\hat{w}^T\hat{x}_i+b)=1$ and we say that $x_i$ is on the boundary.
o If $y_i(\hat{w}^T\hat{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$.
When $\lambda_i > 0$, the vectors $\hat{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$.
o If $\lambda_i > 0$, then $y_i(\boldmath{w}^T\boldmath{x}_i+b)=1$ and we say that $x_i$ is on the boundary.
o If $y_i(\boldmath{w}^T\boldmath{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$.
When $\lambda_i > 0$, the vectors $\boldmath{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$.
!split
===== The problem to solve =====
@@ -386,22 +386,22 @@ When $\lambda_i > 0$, the vectors $\hat{x}_i$ are called support vectors. They a
We can rewrite
!bt
\[
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
\]
!et
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. $\lambda$ the following problem
!bt
\[
\frac{1}{2} \hat{\lambda}^T\begin{bmatrix} y_1y_1\hat{x}_1^T\hat{x}_1 & y_1y_2\hat{x}_1^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_1^T\hat{x}_n \\
y_2y_1\hat{x}_2^T\hat{x}_1 & y_2y_2\hat{x}_2^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_2^T\hat{x}_n \\
\frac{1}{2} \boldmath{\lambda}^T\begin{bmatrix} y_1y_1\boldmath{x}_1^T\boldmath{x}_1 & y_1y_2\boldmath{x}_1^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_1^T\boldmath{x}_n \\
y_2y_1\boldmath{x}_2^T\boldmath{x}_1 & y_2y_2\boldmath{x}_2^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_2^T\boldmath{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1\hat{x}_n^T\hat{x}_1 & y_ny_2\hat{x}_n^T\hat{x}_2 & \dots & \dots & y_ny_n\hat{x}_n^T\hat{x}_n \\
\end{bmatrix}\hat{\lambda}-\mathbb{1}\hat{\lambda},
y_ny_1\boldmath{x}_n^T\boldmath{x}_1 & y_ny_2\boldmath{x}_n^T\boldmath{x}_2 & \dots & \dots & y_ny_n\boldmath{x}_n^T\boldmath{x}_n \\
\end{bmatrix}\boldmath{\lambda}-\mathbb{1}\boldmath{\lambda},
\]
!et
subject to $\hat{y}^T\hat{\lambda}=0$. Here we defined the vectors $\hat{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and
$\hat{y}=[y_1,y_2,\dots,y_n]$.
subject to $\boldmath{y}^T\boldmath{\lambda}=0$. Here we defined the vectors $\boldmath{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and
$\boldmath{y}=[y_1,y_2,\dots,y_n]$.
!split
@@ -411,25 +411,31 @@ Solving the above problem, yields the values of $\lambda_i$.
To find the coefficients of your hyperplane we need simply to compute
!bt
\[
\hat{w}=\sum_{i} \lambda_iy_i\hat{x}_i.
\boldmath{w}=\sum_{i} \lambda_iy_i\boldmath{x}_i.
\]
!et
With our vector $\hat{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via
With our vector $\boldmath{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via
!bt
\[
y_i(\hat{w}^T\hat{x}_i+b)=1,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
\]
!et
resulting in
!bt
\[
b = \frac{1}{y_i}-\hat{w}^T\hat{x}_i.
b = \frac{1}{y_i}-\boldmath{w}^T\boldmath{x}_i,
\]
!et
or if we write it out in terms of the support vectors only, with $N_s$ being their number, we have
!bt
\[
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldmath{x}_i^T\boldmath{x}_j\right).
\]
!et
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
!bt
\[
y_i = \mathrm{sign}(\hat{w}^T\hat{x}_i+b).
y_i = \mathrm{sign}(\boldmath{w}^T\boldmath{x}_i+b).
\]
!et
Below we discuss how to find the optimal values of $\lambda_i$. Before we proceed however, we discuss now the so-called soft classifier.
@@ -444,22 +450,22 @@ figure here. One way to deal with this problem before we define the
so-called _kernel approach_, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
We introduce thus the so-called _slack_ variables $\hat{\xi} =[\xi_1,x_2,\dots,x_n]$ and
We introduce thus the so-called _slack_ variables $\boldmath{\xi} =[\xi_1,x_2,\dots,x_n]$ and
modify our previous equation
!bt
\[
y_i(\hat{w}^T\hat{x}_i+b)=1,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
\]
!et
to
!bt
\[
y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i,
\]
!et
with the requirement $\xi_i\geq 0$. The total violation is now $\sum_i\xi$.
The value $\xi_i$ in the constraint the last constraint corresponds to the amount by which the prediction
$y_i(\hat{w}^T\hat{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$,
$y_i(\boldmath{w}^T\boldmath{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$,
we bound the total amount by which predictions fall on the wrong side of their margins.
Misclassifications occur when $\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of
@@ -472,18 +478,18 @@ misclassifications.
This has in turn the consequences that we change our optmization problem to finding the minimum of
!bt
\[
{\cal L}=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
{\cal L}=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
\]
!et
subject to
!bt
\[
y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
\]
!et
with the requirement $\xi_i\geq 0$.
Taking the derivatives with respect to $b$ and $\hat{w}$ we obtain
Taking the derivatives with respect to $b$ and $\boldmath{w}$ we obtain
!bt
\[
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -492,7 +498,7 @@ Taking the derivatives with respect to $b$ and $\hat{w}$ we obtain
and
!bt
\[
\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i,
\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i,
\]
!et
and
@@ -504,14 +510,14 @@ and
Inserting these constraints into the equation for ${\cal L}$ we obtain the same equation as before
!bt
\[
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
\]
!et
but now subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$ and $0\leq\lambda_i \leq C$.
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
!bt
\[
\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\]
!et
!bt
@@ -522,7 +528,7 @@ We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
and
!bt
\[
y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
\]
!et