lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

mathbf--> bm

Browse Source
This commit is contained in:
mhjensen
2018-11-05 21:27:51 +01:00
parent eaba63cadf
commit 3fe132f4ce
18 changed files with 347 additions and 347 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/svm/html/._svm-bs003.html
+2 -2
View File
@@ -147,10 +147,10 @@ $$
where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line
\( b+w_1x_1+w_2x_2=0 \).
In two dimensions we define the vectors \( \mathbf{x} =[x1,x2] \) and \( \mathbf{w}=[w1,w2] \).
In two dimensions we define the vectors \( \boldsymbol{x} =[x1,x2] \) and \( \boldsymbol{w}=[w1,w2] \).
We can then rewrite the above equation as
$$
\mathbf{w}^T\mathbf{x}+b=0.
\boldsymbol{w}^T\boldsymbol{x}+b=0.
$$
<p>
doc/pub/svm/html/._svm-bs004.html
+6 -6
View File
@@ -139,25 +139,25 @@ b+wx_1+w_2x_2+\dots +w_px_p=0.
$$
If we define a
matrix \( \mathbf{X}=\left[\mathbf{x}_1,\mathbf{x}_2,\dots, \mathbf{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \mathbf{X} \),
matrix \( \boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldsymbol{X} \),
$$
\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
$$
If the above condition is not met for a given vector \( \mathbf{x}_i \) we have
If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
$$
if our output \( y_i=1 \).
In this case we say that \( \mathbf{x}_i \) lies on one of the sides of the hyperplane and if
In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
$$
for the class of observations \( y_i=-1 \),
then \( \mathbf{x}_i \) lies on the other side.
then \( \boldsymbol{x}_i \) lies on the other side.
<p>
Equivalently, for the two classes of observations we have
doc/pub/svm/html/._svm-bs006.html
+4 -4
View File
@@ -134,18 +134,18 @@ MathJax.Hub.Config({
<p>
Let us define the function
$$
f(x) = \mathbf{w}^T\mathbf{x}+b = 0,
f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0,
$$
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
<p>
Any point defined by \( \mathbf{x}_i \) and \( \mathbf{x}_2 \) on the line \( L \) will satisfy \( \mathbf{w}^T(\mathbf{x}_1-\mathbf{x}_2)=0 \).
Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).
<p>
The signed distance \( \delta \) from any point defined by a vector \( \mathbf{x} \) and a point \( \mathbf{x}_0 \) on the line \( L \) is then
The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then
$$
\delta = \frac{1}{\vert\vert \mathbf{w}\vert\vert}(\mathbf{w}^T\mathbf{x}+b).
\delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b).
$$
<p>
doc/pub/svm/html/._svm-bs007.html
+4 -4
View File
@@ -132,23 +132,23 @@ MathJax.Hub.Config({
<h2 id="___sec6" class="anchor">First attempt at a minimization approach </h2>
<p>
How do we find the parameter \( b \) and the vector \( \mathbf{w} \)? What we could
How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could
do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
$$
C(\mathbf{w},b) = -\sum_{i\in M} y_i(\mathbf{w}^T\mathbf{x}_i+b).
C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b).
$$
<p>
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \mathbf{w}^T\mathbf{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
$$
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
$$
and
$$
\frac{\partial C}{\partial \mathbf{w}} = -\sum_{i\in M} y_ix_i.
\frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i.
$$
<p>
doc/pub/svm/html/._svm-bs008.html
+1 -1
View File
@@ -139,7 +139,7 @@ $$
and
$$
\mathbf{w} \leftarrow \mathbf{w} +\eta \frac{\partial C}{\partial \mathbf{w}},
\boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}},
$$
where \( \eta \) is our by now well-known learning rate.
doc/pub/svm/html/._svm-bs009.html
+9 -9
View File
@@ -136,11 +136,11 @@ A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).
<p>
Thus, we wish to find a margin \( M \) with \( \mathbf{w} \) normalized to
\( \vert\vert \mathbf{w}\vert\vert =1 \) subject to the condition
Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to
\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.
@@ -148,22 +148,22 @@ All points are thus at a signed distance from the decision boundary defined by t
<p>
We seek thus the largest value \( M \) defined by
$$
\frac{1}{\vert \vert \mathbf{w}\vert\vert}y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
or just
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M\vert \vert \mathbf{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
$$
If we scale the equation so that \( \vert \vert \mathbf{w}\vert\vert = 1/M \), we have to find the minimum of
\( \mathbf{w}^T\mathbf{w}=\vert \vert \mathbf{w}\vert\vert \) (the norm) subject to the condition
If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of
\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
$$
<p>
We have thus defined our margin as the invers of the norm of \( \mathbf{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
We have thus defined our margin as the invers of the norm of \( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
<p>
<p>
doc/pub/svm/html/._svm-bs012.html
+8 -8
View File
@@ -132,40 +132,40 @@ MathJax.Hub.Config({
<h2 id="___sec11" class="anchor">Setting up the problem </h2>
In order to solve the above problem, we define the following Lagrangian function to be minimized
$$
{\cal L}(\lambda,b,\mathbf{w})=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-1\right],
{\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right],
$$
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i.
\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
$$
Inserting these constraints into the equation for \( {\cal L} \) we obtain
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_self">Karush-Kuhn-Tucker</a> (KKT) condition
$$
\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
<ol>
<li> If \( \lambda_i > 0 \), then \( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\mathbf{w}^T\mathbf{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
<li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
When \( \lambda_i > 0 \), the vectors \( \mathbf{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
<p>
<p>
doc/pub/svm/html/._svm-bs013.html
+7 -7
View File
@@ -134,21 +134,21 @@ MathJax.Hub.Config({
<p>
We can rewrite
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
$$
\frac{1}{2} \mathbf{\lambda}^T\begin{bmatrix} y_1y_1\mathbf{x}_1^T\mathbf{x}_1 & y_1y_2\mathbf{x}_1^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_1^T\mathbf{x}_n \\
y_2y_1\mathbf{x}_2^T\mathbf{x}_1 & y_2y_2\mathbf{x}_2^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_2^T\mathbf{x}_n \\
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1\mathbf{x}_n^T\mathbf{x}_1 & y_ny_2\mathbf{x}_n^T\mathbf{x}_2 & \dots & \dots & y_ny_n\mathbf{x}_n^T\mathbf{x}_n \\
\end{bmatrix}\mathbf{\lambda}-\mathbb{1}\mathbf{\lambda},
y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 & \dots & \dots & y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \( \mathbf{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \mathbf{y}=[y_1,y_2,\dots,y_n] \).
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
<p>
<p>
doc/pub/svm/html/._svm-bs014.html
+6 -6
View File
@@ -135,27 +135,27 @@ MathJax.Hub.Config({
Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
$$
\mathbf{w}=\sum_{i} \lambda_iy_i\mathbf{x}_i.
\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
$$
With our vector \( \mathbf{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
$$
resulting in
$$
b = \frac{1}{y_i}-\mathbf{w}^T\mathbf{x}_i,
b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
$$
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
$$
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\mathbf{x}_i^T\mathbf{x}_j\right).
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right).
$$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
y_i = \mathrm{sign}(\mathbf{w}^T\mathbf{x}_i+b).
y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
$$
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
doc/pub/svm/html/._svm-bs015.html
+4 -4
View File
@@ -141,20 +141,20 @@ so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
<p>
We introduce thus the so-called <b>slack</b> variables \( \mathbf{\xi} =[\xi_1,x_2,\dots,x_n] \) and
We introduce thus the so-called <b>slack</b> variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and
modify our previous equation
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
$$
to
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i,
$$
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
\( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
<p>
doc/pub/svm/html/._svm-bs016.html
+7 -7
View File
@@ -134,25 +134,25 @@ MathJax.Hub.Config({
<p>
This has in turn the consequences that we change our optmization problem to finding the minimum of
$$
{\cal L}=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
{\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
$$
subject to
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
with the requirement \( \xi_i\geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i,
\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i,
$$
and
@@ -162,13 +162,13 @@ $$
Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \).
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
$$
\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
$$
@@ -177,7 +177,7 @@ $$
and
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
<p>
doc/pub/svm/html/svm-reveal.html
+58 -58
View File
@@ -225,11 +225,11 @@ $$
where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line
\( b+w_1x_1+w_2x_2=0 \).
In two dimensions we define the vectors \( \mathbf{x} =[x1,x2] \) and \( \mathbf{w}=[w1,w2] \).
In two dimensions we define the vectors \( \boldsymbol{x} =[x1,x2] \) and \( \boldsymbol{w}=[w1,w2] \).
We can then rewrite the above equation as
<p>&nbsp;<br>
$$
\mathbf{w}^T\mathbf{x}+b=0.
\boldsymbol{w}^T\boldsymbol{x}+b=0.
$$
<p>&nbsp;<br>
</section>
@@ -248,15 +248,15 @@ $$
<p>&nbsp;<br>
If we define a
matrix \( \mathbf{X}=\left[\mathbf{x}_1,\mathbf{x}_2,\dots, \mathbf{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \mathbf{X} \),
matrix \( \boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldsymbol{X} \),
<p>&nbsp;<br>
$$
\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
$$
<p>&nbsp;<br>
If the above condition is not met for a given vector \( \mathbf{x}_i \) we have
If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have
<p>&nbsp;<br>
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
@@ -264,7 +264,7 @@ $$
<p>&nbsp;<br>
if our output \( y_i=1 \).
In this case we say that \( \mathbf{x}_i \) lies on one of the sides of the hyperplane and if
In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if
<p>&nbsp;<br>
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
@@ -272,7 +272,7 @@ $$
<p>&nbsp;<br>
for the class of observations \( y_i=-1 \),
then \( \mathbf{x}_i \) lies on the other side.
then \( \boldsymbol{x}_i \) lies on the other side.
<p>
Equivalently, for the two classes of observations we have
@@ -321,20 +321,20 @@ for our data sample.
Let us define the function
<p>&nbsp;<br>
$$
f(x) = \mathbf{w}^T\mathbf{x}+b = 0,
f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0,
$$
<p>&nbsp;<br>
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
<p>
Any point defined by \( \mathbf{x}_i \) and \( \mathbf{x}_2 \) on the line \( L \) will satisfy \( \mathbf{w}^T(\mathbf{x}_1-\mathbf{x}_2)=0 \).
Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).
<p>
The signed distance \( \delta \) from any point defined by a vector \( \mathbf{x} \) and a point \( \mathbf{x}_0 \) on the line \( L \) is then
The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then
<p>&nbsp;<br>
$$
\delta = \frac{1}{\vert\vert \mathbf{w}\vert\vert}(\mathbf{w}^T\mathbf{x}+b).
\delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b).
$$
<p>&nbsp;<br>
</section>
@@ -344,18 +344,18 @@ $$
<h2 id="___sec6">First attempt at a minimization approach </h2>
<p>
How do we find the parameter \( b \) and the vector \( \mathbf{w} \)? What we could
How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could
do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
<p>&nbsp;<br>
$$
C(\mathbf{w},b) = -\sum_{i\in M} y_i(\mathbf{w}^T\mathbf{x}_i+b).
C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b).
$$
<p>&nbsp;<br>
<p>
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \mathbf{w}^T\mathbf{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
<p>&nbsp;<br>
$$
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
@@ -365,7 +365,7 @@ $$
and
<p>&nbsp;<br>
$$
\frac{\partial C}{\partial \mathbf{w}} = -\sum_{i\in M} y_ix_i.
\frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i.
$$
<p>&nbsp;<br>
</section>
@@ -385,7 +385,7 @@ $$
and
<p>&nbsp;<br>
$$
\mathbf{w} \leftarrow \mathbf{w} +\eta \frac{\partial C}{\partial \mathbf{w}},
\boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}},
$$
<p>&nbsp;<br>
@@ -411,12 +411,12 @@ A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).
<p>
Thus, we wish to find a margin \( M \) with \( \mathbf{w} \) normalized to
\( \vert\vert \mathbf{w}\vert\vert =1 \) subject to the condition
Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to
\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition
<p>&nbsp;<br>
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
<p>&nbsp;<br>
@@ -426,27 +426,27 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value \( M \) defined by
<p>&nbsp;<br>
$$
\frac{1}{\vert \vert \mathbf{w}\vert\vert}y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
<p>&nbsp;<br>
or just
<p>&nbsp;<br>
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M\vert \vert \mathbf{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
$$
<p>&nbsp;<br>
If we scale the equation so that \( \vert \vert \mathbf{w}\vert\vert = 1/M \), we have to find the minimum of
\( \mathbf{w}^T\mathbf{w}=\vert \vert \mathbf{w}\vert\vert \) (the norm) subject to the condition
If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of
\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition
<p>&nbsp;<br>
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
$$
<p>&nbsp;<br>
<p>
We have thus defined our margin as the invers of the norm of \( \mathbf{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
We have thus defined our margin as the invers of the norm of \( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
</section>
@@ -572,14 +572,14 @@ $$
In order to solve the above problem, we define the following Lagrangian function to be minimized
<p>&nbsp;<br>
$$
{\cal L}(\lambda,b,\mathbf{w})=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-1\right],
{\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right],
$$
<p>&nbsp;<br>
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
<p>&nbsp;<br>
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -589,14 +589,14 @@ $$
and
<p>&nbsp;<br>
$$
\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i.
\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
$$
<p>&nbsp;<br>
Inserting these constraints into the equation for \( {\cal L} \) we obtain
<p>&nbsp;<br>
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
<p>&nbsp;<br>
@@ -604,18 +604,18 @@ subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_blank">Karush-Kuhn-Tucker</a> (KKT) condition
<p>&nbsp;<br>
$$
\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
<p>&nbsp;<br>
<ol>
<p><li> If \( \lambda_i > 0 \), then \( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<p><li> If \( y_i(\mathbf{w}^T\mathbf{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
<p><li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<p><li> If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
<p>
When \( \lambda_i > 0 \), the vectors \( \mathbf{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
</section>
@@ -626,24 +626,24 @@ When \( \lambda_i > 0 \), the vectors \( \mathbf{x}_i \) are called support vect
We can rewrite
<p>&nbsp;<br>
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
<p>&nbsp;<br>
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
<p>&nbsp;<br>
$$
\frac{1}{2} \mathbf{\lambda}^T\begin{bmatrix} y_1y_1\mathbf{x}_1^T\mathbf{x}_1 & y_1y_2\mathbf{x}_1^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_1^T\mathbf{x}_n \\
y_2y_1\mathbf{x}_2^T\mathbf{x}_1 & y_2y_2\mathbf{x}_2^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_2^T\mathbf{x}_n \\
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1\mathbf{x}_n^T\mathbf{x}_1 & y_ny_2\mathbf{x}_n^T\mathbf{x}_2 & \dots & \dots & y_ny_n\mathbf{x}_n^T\mathbf{x}_n \\
\end{bmatrix}\mathbf{\lambda}-\mathbb{1}\mathbf{\lambda},
y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 & \dots & \dots & y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
<p>&nbsp;<br>
subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \( \mathbf{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \mathbf{y}=[y_1,y_2,\dots,y_n] \).
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
</section>
@@ -655,35 +655,35 @@ Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
<p>&nbsp;<br>
$$
\mathbf{w}=\sum_{i} \lambda_iy_i\mathbf{x}_i.
\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
$$
<p>&nbsp;<br>
With our vector \( \mathbf{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
<p>&nbsp;<br>
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
$$
<p>&nbsp;<br>
resulting in
<p>&nbsp;<br>
$$
b = \frac{1}{y_i}-\mathbf{w}^T\mathbf{x}_i,
b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
$$
<p>&nbsp;<br>
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
<p>&nbsp;<br>
$$
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\mathbf{x}_i^T\mathbf{x}_j\right).
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right).
$$
<p>&nbsp;<br>
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
<p>&nbsp;<br>
$$
y_i = \mathrm{sign}(\mathbf{w}^T\mathbf{x}_i+b).
y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
$$
<p>&nbsp;<br>
@@ -704,24 +704,24 @@ so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
<p>
We introduce thus the so-called <b>slack</b> variables \( \mathbf{\xi} =[\xi_1,x_2,\dots,x_n] \) and
We introduce thus the so-called <b>slack</b> variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and
modify our previous equation
<p>&nbsp;<br>
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
$$
<p>&nbsp;<br>
to
<p>&nbsp;<br>
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i,
$$
<p>&nbsp;<br>
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
\( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
<p>
@@ -737,21 +737,21 @@ misclassifications.
This has in turn the consequences that we change our optmization problem to finding the minimum of
<p>&nbsp;<br>
$$
{\cal L}=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
{\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
$$
<p>&nbsp;<br>
subject to
<p>&nbsp;<br>
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
<p>&nbsp;<br>
with the requirement \( \xi_i\geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
<p>&nbsp;<br>
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -761,7 +761,7 @@ $$
and
<p>&nbsp;<br>
$$
\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i,
\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i,
$$
<p>&nbsp;<br>
@@ -775,7 +775,7 @@ $$
Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before
<p>&nbsp;<br>
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
<p>&nbsp;<br>
@@ -783,7 +783,7 @@ but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
<p>&nbsp;<br>
$$
\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
<p>&nbsp;<br>
@@ -796,7 +796,7 @@ $$
and
<p>&nbsp;<br>
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
<p>&nbsp;<br>
</section>
doc/pub/svm/html/svm-solarized.html
+58 -58
View File
@@ -164,10 +164,10 @@ $$
where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line
\( b+w_1x_1+w_2x_2=0 \).
In two dimensions we define the vectors \( \mathbf{x} =[x1,x2] \) and \( \mathbf{w}=[w1,w2] \).
In two dimensions we define the vectors \( \boldsymbol{x} =[x1,x2] \) and \( \boldsymbol{w}=[w1,w2] \).
We can then rewrite the above equation as
$$
\mathbf{w}^T\mathbf{x}+b=0.
\boldsymbol{w}^T\boldsymbol{x}+b=0.
$$
<p>
@@ -183,25 +183,25 @@ b+wx_1+w_2x_2+\dots +w_px_p=0.
$$
If we define a
matrix \( \mathbf{X}=\left[\mathbf{x}_1,\mathbf{x}_2,\dots, \mathbf{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \mathbf{X} \),
matrix \( \boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldsymbol{X} \),
$$
\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
$$
If the above condition is not met for a given vector \( \mathbf{x}_i \) we have
If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
$$
if our output \( y_i=1 \).
In this case we say that \( \mathbf{x}_i \) lies on one of the sides of the hyperplane and if
In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
$$
for the class of observations \( y_i=-1 \),
then \( \mathbf{x}_i \) lies on the other side.
then \( \boldsymbol{x}_i \) lies on the other side.
<p>
Equivalently, for the two classes of observations we have
@@ -247,18 +247,18 @@ for our data sample.
<p>
Let us define the function
$$
f(x) = \mathbf{w}^T\mathbf{x}+b = 0,
f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0,
$$
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
<p>
Any point defined by \( \mathbf{x}_i \) and \( \mathbf{x}_2 \) on the line \( L \) will satisfy \( \mathbf{w}^T(\mathbf{x}_1-\mathbf{x}_2)=0 \).
Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).
<p>
The signed distance \( \delta \) from any point defined by a vector \( \mathbf{x} \) and a point \( \mathbf{x}_0 \) on the line \( L \) is then
The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then
$$
\delta = \frac{1}{\vert\vert \mathbf{w}\vert\vert}(\mathbf{w}^T\mathbf{x}+b).
\delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b).
$$
<p>
@@ -267,23 +267,23 @@ $$
<h2 id="___sec6">First attempt at a minimization approach </h2>
<p>
How do we find the parameter \( b \) and the vector \( \mathbf{w} \)? What we could
How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could
do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
$$
C(\mathbf{w},b) = -\sum_{i\in M} y_i(\mathbf{w}^T\mathbf{x}_i+b).
C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b).
$$
<p>
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \mathbf{w}^T\mathbf{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
$$
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
$$
and
$$
\frac{\partial C}{\partial \mathbf{w}} = -\sum_{i\in M} y_ix_i.
\frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i.
$$
<p>
@@ -299,7 +299,7 @@ $$
and
$$
\mathbf{w} \leftarrow \mathbf{w} +\eta \frac{\partial C}{\partial \mathbf{w}},
\boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}},
$$
where \( \eta \) is our by now well-known learning rate.
@@ -324,11 +324,11 @@ A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).
<p>
Thus, we wish to find a margin \( M \) with \( \mathbf{w} \) normalized to
\( \vert\vert \mathbf{w}\vert\vert =1 \) subject to the condition
Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to
\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.
@@ -336,22 +336,22 @@ All points are thus at a signed distance from the decision boundary defined by t
<p>
We seek thus the largest value \( M \) defined by
$$
\frac{1}{\vert \vert \mathbf{w}\vert\vert}y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
or just
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M\vert \vert \mathbf{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
$$
If we scale the equation so that \( \vert \vert \mathbf{w}\vert\vert = 1/M \), we have to find the minimum of
\( \mathbf{w}^T\mathbf{w}=\vert \vert \mathbf{w}\vert\vert \) (the norm) subject to the condition
If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of
\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
$$
<p>
We have thus defined our margin as the invers of the norm of \( \mathbf{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
We have thus defined our margin as the invers of the norm of \( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -452,40 +452,40 @@ $$
<h2 id="___sec11">Setting up the problem </h2>
In order to solve the above problem, we define the following Lagrangian function to be minimized
$$
{\cal L}(\lambda,b,\mathbf{w})=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-1\right],
{\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right],
$$
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i.
\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
$$
Inserting these constraints into the equation for \( {\cal L} \) we obtain
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_blank">Karush-Kuhn-Tucker</a> (KKT) condition
$$
\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
<ol>
<li> If \( \lambda_i > 0 \), then \( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\mathbf{w}^T\mathbf{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
<li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
When \( \lambda_i > 0 \), the vectors \( \mathbf{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -495,21 +495,21 @@ When \( \lambda_i > 0 \), the vectors \( \mathbf{x}_i \) are called support vect
<p>
We can rewrite
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
$$
\frac{1}{2} \mathbf{\lambda}^T\begin{bmatrix} y_1y_1\mathbf{x}_1^T\mathbf{x}_1 & y_1y_2\mathbf{x}_1^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_1^T\mathbf{x}_n \\
y_2y_1\mathbf{x}_2^T\mathbf{x}_1 & y_2y_2\mathbf{x}_2^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_2^T\mathbf{x}_n \\
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1\mathbf{x}_n^T\mathbf{x}_1 & y_ny_2\mathbf{x}_n^T\mathbf{x}_2 & \dots & \dots & y_ny_n\mathbf{x}_n^T\mathbf{x}_n \\
\end{bmatrix}\mathbf{\lambda}-\mathbb{1}\mathbf{\lambda},
y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 & \dots & \dots & y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \( \mathbf{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \mathbf{y}=[y_1,y_2,\dots,y_n] \).
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -520,27 +520,27 @@ subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \(
Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
$$
\mathbf{w}=\sum_{i} \lambda_iy_i\mathbf{x}_i.
\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
$$
With our vector \( \mathbf{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
$$
resulting in
$$
b = \frac{1}{y_i}-\mathbf{w}^T\mathbf{x}_i,
b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
$$
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
$$
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\mathbf{x}_i^T\mathbf{x}_j\right).
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right).
$$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
y_i = \mathrm{sign}(\mathbf{w}^T\mathbf{x}_i+b).
y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
$$
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
@@ -560,20 +560,20 @@ so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
<p>
We introduce thus the so-called <b>slack</b> variables \( \mathbf{\xi} =[\xi_1,x_2,\dots,x_n] \) and
We introduce thus the so-called <b>slack</b> variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and
modify our previous equation
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
$$
to
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i,
$$
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
\( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
<p>
@@ -588,25 +588,25 @@ misclassifications.
<p>
This has in turn the consequences that we change our optmization problem to finding the minimum of
$$
{\cal L}=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
{\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
$$
subject to
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
with the requirement \( \xi_i\geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i,
\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i,
$$
and
@@ -616,13 +616,13 @@ $$
Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \).
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
$$
\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
$$
@@ -631,7 +631,7 @@ $$
and
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
<p>
doc/pub/svm/html/svm.html
+58 -58
View File
@@ -169,10 +169,10 @@ $$
where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line
\( b+w_1x_1+w_2x_2=0 \).
In two dimensions we define the vectors \( \mathbf{x} =[x1,x2] \) and \( \mathbf{w}=[w1,w2] \).
In two dimensions we define the vectors \( \boldsymbol{x} =[x1,x2] \) and \( \boldsymbol{w}=[w1,w2] \).
We can then rewrite the above equation as
$$
\mathbf{w}^T\mathbf{x}+b=0.
\boldsymbol{w}^T\boldsymbol{x}+b=0.
$$
<p>
@@ -188,25 +188,25 @@ b+wx_1+w_2x_2+\dots +w_px_p=0.
$$
If we define a
matrix \( \mathbf{X}=\left[\mathbf{x}_1,\mathbf{x}_2,\dots, \mathbf{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \mathbf{X} \),
matrix \( \boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right] \)
of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldsymbol{X} \),
$$
\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
$$
If the above condition is not met for a given vector \( \mathbf{x}_i \) we have
If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
$$
if our output \( y_i=1 \).
In this case we say that \( \mathbf{x}_i \) lies on one of the sides of the hyperplane and if
In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
$$
for the class of observations \( y_i=-1 \),
then \( \mathbf{x}_i \) lies on the other side.
then \( \boldsymbol{x}_i \) lies on the other side.
<p>
Equivalently, for the two classes of observations we have
@@ -252,18 +252,18 @@ for our data sample.
<p>
Let us define the function
$$
f(x) = \mathbf{w}^T\mathbf{x}+b = 0,
f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0,
$$
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
<p>
Any point defined by \( \mathbf{x}_i \) and \( \mathbf{x}_2 \) on the line \( L \) will satisfy \( \mathbf{w}^T(\mathbf{x}_1-\mathbf{x}_2)=0 \).
Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).
<p>
The signed distance \( \delta \) from any point defined by a vector \( \mathbf{x} \) and a point \( \mathbf{x}_0 \) on the line \( L \) is then
The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then
$$
\delta = \frac{1}{\vert\vert \mathbf{w}\vert\vert}(\mathbf{w}^T\mathbf{x}+b).
\delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b).
$$
<p>
@@ -272,23 +272,23 @@ $$
<h2 id="___sec6">First attempt at a minimization approach </h2>
<p>
How do we find the parameter \( b \) and the vector \( \mathbf{w} \)? What we could
How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could
do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
$$
C(\mathbf{w},b) = -\sum_{i\in M} y_i(\mathbf{w}^T\mathbf{x}_i+b).
C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b).
$$
<p>
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \mathbf{w}^T\mathbf{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
$$
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
$$
and
$$
\frac{\partial C}{\partial \mathbf{w}} = -\sum_{i\in M} y_ix_i.
\frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i.
$$
<p>
@@ -304,7 +304,7 @@ $$
and
$$
\mathbf{w} \leftarrow \mathbf{w} +\eta \frac{\partial C}{\partial \mathbf{w}},
\boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}},
$$
where \( \eta \) is our by now well-known learning rate.
@@ -329,11 +329,11 @@ A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).
<p>
Thus, we wish to find a margin \( M \) with \( \mathbf{w} \) normalized to
\( \vert\vert \mathbf{w}\vert\vert =1 \) subject to the condition
Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to
\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.
@@ -341,22 +341,22 @@ All points are thus at a signed distance from the decision boundary defined by t
<p>
We seek thus the largest value \( M \) defined by
$$
\frac{1}{\vert \vert \mathbf{w}\vert\vert}y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
or just
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M\vert \vert \mathbf{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
$$
If we scale the equation so that \( \vert \vert \mathbf{w}\vert\vert = 1/M \), we have to find the minimum of
\( \mathbf{w}^T\mathbf{w}=\vert \vert \mathbf{w}\vert\vert \) (the norm) subject to the condition
If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of
\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
$$
<p>
We have thus defined our margin as the invers of the norm of \( \mathbf{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
We have thus defined our margin as the invers of the norm of \( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -457,40 +457,40 @@ $$
<h2 id="___sec11">Setting up the problem </h2>
In order to solve the above problem, we define the following Lagrangian function to be minimized
$$
{\cal L}(\lambda,b,\mathbf{w})=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-1\right],
{\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right],
$$
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i.
\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
$$
Inserting these constraints into the equation for \( {\cal L} \) we obtain
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_blank">Karush-Kuhn-Tucker</a> (KKT) condition
$$
\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
<ol>
<li> If \( \lambda_i > 0 \), then \( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\mathbf{w}^T\mathbf{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
<li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
When \( \lambda_i > 0 \), the vectors \( \mathbf{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -500,21 +500,21 @@ When \( \lambda_i > 0 \), the vectors \( \mathbf{x}_i \) are called support vect
<p>
We can rewrite
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
$$
\frac{1}{2} \mathbf{\lambda}^T\begin{bmatrix} y_1y_1\mathbf{x}_1^T\mathbf{x}_1 & y_1y_2\mathbf{x}_1^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_1^T\mathbf{x}_n \\
y_2y_1\mathbf{x}_2^T\mathbf{x}_1 & y_2y_2\mathbf{x}_2^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_2^T\mathbf{x}_n \\
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1\mathbf{x}_n^T\mathbf{x}_1 & y_ny_2\mathbf{x}_n^T\mathbf{x}_2 & \dots & \dots & y_ny_n\mathbf{x}_n^T\mathbf{x}_n \\
\end{bmatrix}\mathbf{\lambda}-\mathbb{1}\mathbf{\lambda},
y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 & \dots & \dots & y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \( \mathbf{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \mathbf{y}=[y_1,y_2,\dots,y_n] \).
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -525,27 +525,27 @@ subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \(
Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
$$
\mathbf{w}=\sum_{i} \lambda_iy_i\mathbf{x}_i.
\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
$$
With our vector \( \mathbf{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
$$
resulting in
$$
b = \frac{1}{y_i}-\mathbf{w}^T\mathbf{x}_i,
b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
$$
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
$$
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\mathbf{x}_i^T\mathbf{x}_j\right).
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right).
$$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
y_i = \mathrm{sign}(\mathbf{w}^T\mathbf{x}_i+b).
y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
$$
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
@@ -565,20 +565,20 @@ so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
<p>
We introduce thus the so-called <b>slack</b> variables \( \mathbf{\xi} =[\xi_1,x_2,\dots,x_n] \) and
We introduce thus the so-called <b>slack</b> variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and
modify our previous equation
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
$$
to
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i,
$$
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
\( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
<p>
@@ -593,25 +593,25 @@ misclassifications.
<p>
This has in turn the consequences that we change our optmization problem to finding the minimum of
$$
{\cal L}=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
{\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
$$
subject to
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
with the requirement \( \xi_i\geq 0 \).
<p>
Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain
Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i,
\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i,
$$
and
@@ -621,13 +621,13 @@ $$
Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \).
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
$$
\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
$$
@@ -636,7 +636,7 @@ $$
and
$$
y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
<p>
doc/pub/svm/ipynb/ipynb-svm-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/svm/ipynb/svm.ipynb
+57 -57
View File
@@ -79,7 +79,7 @@
"source": [
"where $b$ is the intercept and $w_1$ and $w_2$ define the elements of a vector orthogonal to the line \n",
"$b+w_1x_1+w_2x_2=0$. \n",
"In two dimensions we define the vectors $\\mathbf{x} =[x1,x2]$ and $\\mathbf{w}=[w1,w2]$. \n",
"In two dimensions we define the vectors $\\boldsymbol{x} =[x1,x2]$ and $\\boldsymbol{w}=[w1,w2]$. \n",
"We can then rewrite the above equation as"
]
},
@@ -88,7 +88,7 @@
"metadata": {},
"source": [
"$$\n",
"\\mathbf{w}^T\\mathbf{x}+b=0.\n",
"\\boldsymbol{w}^T\\boldsymbol{x}+b=0.\n",
"$$"
]
},
@@ -116,8 +116,8 @@
"metadata": {},
"source": [
"If we define a \n",
"matrix $\\mathbf{X}=\\left[\\mathbf{x}_1,\\mathbf{x}_2,\\dots, \\mathbf{x}_p\\right]$\n",
"of dimension $n\\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\\mathbf{X}$,"
"matrix $\\boldsymbol{X}=\\left[\\boldsymbol{x}_1,\\boldsymbol{x}_2,\\dots, \\boldsymbol{x}_p\\right]$\n",
"of dimension $n\\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\\boldsymbol{X}$,"
]
},
{
@@ -125,7 +125,7 @@
"metadata": {},
"source": [
"$$\n",
"\\mathbf{x}_i = \\begin{bmatrix} x_{i1} \\\\ x_{i2} \\\\ \\dots \\\\ \\dots \\\\ x_{ip} \\end{bmatrix}.\n",
"\\boldsymbol{x}_i = \\begin{bmatrix} x_{i1} \\\\ x_{i2} \\\\ \\dots \\\\ \\dots \\\\ x_{ip} \\end{bmatrix}.\n",
"$$"
]
},
@@ -133,7 +133,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"If the above condition is not met for a given vector $\\mathbf{x}_i$ we have"
"If the above condition is not met for a given vector $\\boldsymbol{x}_i$ we have"
]
},
{
@@ -150,7 +150,7 @@
"metadata": {},
"source": [
"if our output $y_i=1$.\n",
"In this case we say that $\\mathbf{x}_i$ lies on one of the sides of the hyperplane and if"
"In this case we say that $\\boldsymbol{x}_i$ lies on one of the sides of the hyperplane and if"
]
},
{
@@ -167,7 +167,7 @@
"metadata": {},
"source": [
"for the class of observations $y_i=-1$, \n",
"then $\\mathbf{x}_i$ lies on the other side. \n",
"then $\\boldsymbol{x}_i$ lies on the other side. \n",
"\n",
"Equivalently, for the two classes of observations we have"
]
@@ -219,7 +219,7 @@
"metadata": {},
"source": [
"$$\n",
"f(x) = \\mathbf{w}^T\\mathbf{x}+b = 0,\n",
"f(x) = \\boldsymbol{w}^T\\boldsymbol{x}+b = 0,\n",
"$$"
]
},
@@ -230,9 +230,9 @@
"as the function that determines the line $L$ that separates two classes (our two features), see the figure here. \n",
"\n",
"\n",
"Any point defined by $\\mathbf{x}_i$ and $\\mathbf{x}_2$ on the line $L$ will satisfy $\\mathbf{w}^T(\\mathbf{x}_1-\\mathbf{x}_2)=0$. \n",
"Any point defined by $\\boldsymbol{x}_i$ and $\\boldsymbol{x}_2$ on the line $L$ will satisfy $\\boldsymbol{w}^T(\\boldsymbol{x}_1-\\boldsymbol{x}_2)=0$. \n",
"\n",
"The signed distance $\\delta$ from any point defined by a vector $\\mathbf{x}$ and a point $\\mathbf{x}_0$ on the line $L$ is then"
"The signed distance $\\delta$ from any point defined by a vector $\\boldsymbol{x}$ and a point $\\boldsymbol{x}_0$ on the line $L$ is then"
]
},
{
@@ -240,7 +240,7 @@
"metadata": {},
"source": [
"$$\n",
"\\delta = \\frac{1}{\\vert\\vert \\mathbf{w}\\vert\\vert}(\\mathbf{w}^T\\mathbf{x}+b).\n",
"\\delta = \\frac{1}{\\vert\\vert \\boldsymbol{w}\\vert\\vert}(\\boldsymbol{w}^T\\boldsymbol{x}+b).\n",
"$$"
]
},
@@ -250,7 +250,7 @@
"source": [
"## First attempt at a minimization approach\n",
"\n",
"How do we find the parameter $b$ and the vector $\\mathbf{w}$? What we could\n",
"How do we find the parameter $b$ and the vector $\\boldsymbol{w}$? What we could\n",
"do is to define a cost function which now contains the set of all\n",
"misclassified points $M$ and attempt to minimize this function"
]
@@ -260,7 +260,7 @@
"metadata": {},
"source": [
"$$\n",
"C(\\mathbf{w},b) = -\\sum_{i\\in M} y_i(\\mathbf{w}^T\\mathbf{x}_i+b).\n",
"C(\\boldsymbol{w},b) = -\\sum_{i\\in M} y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b).\n",
"$$"
]
},
@@ -268,7 +268,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We could now for example define all values $y_i =1$ as misclassified in case we have $\\mathbf{w}^T\\mathbf{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us"
"We could now for example define all values $y_i =1$ as misclassified in case we have $\\boldsymbol{w}^T\\boldsymbol{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us"
]
},
{
@@ -292,7 +292,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial \\mathbf{w}} = -\\sum_{i\\in M} y_ix_i.\n",
"\\frac{\\partial C}{\\partial \\boldsymbol{w}} = -\\sum_{i\\in M} y_ix_i.\n",
"$$"
]
},
@@ -326,7 +326,7 @@
"metadata": {},
"source": [
"$$\n",
"\\mathbf{w} \\leftarrow \\mathbf{w} +\\eta \\frac{\\partial C}{\\partial \\mathbf{w}},\n",
"\\boldsymbol{w} \\leftarrow \\boldsymbol{w} +\\eta \\frac{\\partial C}{\\partial \\boldsymbol{w}},\n",
"$$"
]
},
@@ -350,8 +350,8 @@
"A better approach is rather to try to define a large margin between\n",
"the two classes (if they are well separated from the beginning).\n",
"\n",
"Thus, we wish to find a margin $M$ with $\\mathbf{w}$ normalized to\n",
"$\\vert\\vert \\mathbf{w}\\vert\\vert =1$ subject to the condition"
"Thus, we wish to find a margin $M$ with $\\boldsymbol{w}$ normalized to\n",
"$\\vert\\vert \\boldsymbol{w}\\vert\\vert =1$ subject to the condition"
]
},
{
@@ -359,7 +359,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\mathbf{w}^T\\mathbf{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, p.\n",
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, p.\n",
"$$"
]
},
@@ -377,7 +377,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{1}{\\vert \\vert \\mathbf{w}\\vert\\vert}y_i(\\mathbf{w}^T\\mathbf{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, n,\n",
"\\frac{1}{\\vert \\vert \\boldsymbol{w}\\vert\\vert}y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, n,\n",
"$$"
]
},
@@ -393,7 +393,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\mathbf{w}^T\\mathbf{x}_i+b) \\geq M\\vert \\vert \\mathbf{w}\\vert\\vert \\hspace{0.1cm}\\forall i.\n",
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M\\vert \\vert \\boldsymbol{w}\\vert\\vert \\hspace{0.1cm}\\forall i.\n",
"$$"
]
},
@@ -401,8 +401,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"If we scale the equation so that $\\vert \\vert \\mathbf{w}\\vert\\vert = 1/M$, we have to find the minimum of \n",
"$\\mathbf{w}^T\\mathbf{w}=\\vert \\vert \\mathbf{w}\\vert\\vert$ (the norm) subject to the condition"
"If we scale the equation so that $\\vert \\vert \\boldsymbol{w}\\vert\\vert = 1/M$, we have to find the minimum of \n",
"$\\boldsymbol{w}^T\\boldsymbol{w}=\\vert \\vert \\boldsymbol{w}\\vert\\vert$ (the norm) subject to the condition"
]
},
{
@@ -410,7 +410,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\mathbf{w}^T\\mathbf{x}_i+b) \\geq 1 \\hspace{0.1cm}\\forall i.\n",
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq 1 \\hspace{0.1cm}\\forall i.\n",
"$$"
]
},
@@ -418,7 +418,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We have thus defined our margin as the invers of the norm of $\\mathbf{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers. \n",
"We have thus defined our margin as the invers of the norm of $\\boldsymbol{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers. \n",
"\n",
"## A quick reminder on Lagrangian multipliers\n",
"\n",
@@ -644,7 +644,7 @@
"metadata": {},
"source": [
"$$\n",
"{\\cal L}(\\lambda,b,\\mathbf{w})=\\frac{1}{2}\\mathbf{w}^T\\mathbf{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\mathbf{w}^T\\mathbf{x}_i+b)-1\\right],\n",
"{\\cal L}(\\lambda,b,\\boldsymbol{w})=\\frac{1}{2}\\boldsymbol{w}^T\\boldsymbol{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)-1\\right],\n",
"$$"
]
},
@@ -654,7 +654,7 @@
"source": [
"where $\\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\\lambda_i \\geq 0$.\n",
"\n",
"Taking the derivatives with respect to $b$ and $\\mathbf{w}$ we obtain"
"Taking the derivatives with respect to $b$ and $\\boldsymbol{w}$ we obtain"
]
},
{
@@ -678,7 +678,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial {\\cal L}}{\\partial \\mathbf{w}} = 0 = \\mathbf{w}-\\sum_{i} \\lambda_iy_i\\mathbf{x}_i.\n",
"\\frac{\\partial {\\cal L}}{\\partial \\boldsymbol{w}} = 0 = \\boldsymbol{w}-\\sum_{i} \\lambda_iy_i\\boldsymbol{x}_i.\n",
"$$"
]
},
@@ -694,7 +694,7 @@
"metadata": {},
"source": [
"$$\n",
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\mathbf{x}_i^T\\mathbf{x}_j,\n",
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{x}_j,\n",
"$$"
]
},
@@ -711,7 +711,7 @@
"metadata": {},
"source": [
"$$\n",
"\\lambda_i\\left[y_i(\\mathbf{w}^T\\mathbf{x}_i+b) -1\\right] \\hspace{0.1cm}\\forall i.\n",
"\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) -1\\right] \\hspace{0.1cm}\\forall i.\n",
"$$"
]
},
@@ -719,11 +719,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"1. If $\\lambda_i > 0$, then $y_i(\\mathbf{w}^T\\mathbf{x}_i+b)=1$ and we say that $x_i$ is on the boundary.\n",
"1. If $\\lambda_i > 0$, then $y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1$ and we say that $x_i$ is on the boundary.\n",
"\n",
"2. If $y_i(\\mathbf{w}^T\\mathbf{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\\lambda_i=0$. \n",
"2. If $y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\\lambda_i=0$. \n",
"\n",
"When $\\lambda_i > 0$, the vectors $\\mathbf{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. \n",
"When $\\lambda_i > 0$, the vectors $\\boldsymbol{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. \n",
"\n",
"## The problem to solve\n",
"\n",
@@ -735,7 +735,7 @@
"metadata": {},
"source": [
"$$\n",
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\mathbf{x}_i^T\\mathbf{x}_j,\n",
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{x}_j,\n",
"$$"
]
},
@@ -751,12 +751,12 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{1}{2} \\mathbf{\\lambda}^T\\begin{bmatrix} y_1y_1\\mathbf{x}_1^T\\mathbf{x}_1 & y_1y_2\\mathbf{x}_1^T\\mathbf{x}_2 & \\dots & \\dots & y_1y_n\\mathbf{x}_1^T\\mathbf{x}_n \\\\\n",
"y_2y_1\\mathbf{x}_2^T\\mathbf{x}_1 & y_2y_2\\mathbf{x}_2^T\\mathbf{x}_2 & \\dots & \\dots & y_1y_n\\mathbf{x}_2^T\\mathbf{x}_n \\\\\n",
"\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1\\boldsymbol{x}_1^T\\boldsymbol{x}_1 & y_1y_2\\boldsymbol{x}_1^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_1^T\\boldsymbol{x}_n \\\\\n",
"y_2y_1\\boldsymbol{x}_2^T\\boldsymbol{x}_1 & y_2y_2\\boldsymbol{x}_2^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_2^T\\boldsymbol{x}_n \\\\\n",
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
"y_ny_1\\mathbf{x}_n^T\\mathbf{x}_1 & y_ny_2\\mathbf{x}_n^T\\mathbf{x}_2 & \\dots & \\dots & y_ny_n\\mathbf{x}_n^T\\mathbf{x}_n \\\\\n",
"\\end{bmatrix}\\mathbf{\\lambda}-\\mathbb{1}\\mathbf{\\lambda},\n",
"y_ny_1\\boldsymbol{x}_n^T\\boldsymbol{x}_1 & y_ny_2\\boldsymbol{x}_n^T\\boldsymbol{x}_2 & \\dots & \\dots & y_ny_n\\boldsymbol{x}_n^T\\boldsymbol{x}_n \\\\\n",
"\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{1}\\boldsymbol{\\lambda},\n",
"$$"
]
},
@@ -764,8 +764,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"subject to $\\mathbf{y}^T\\mathbf{\\lambda}=0$. Here we defined the vectors $\\mathbf{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n",
"$\\mathbf{y}=[y_1,y_2,\\dots,y_n]$. \n",
"subject to $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$. Here we defined the vectors $\\boldsymbol{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n",
"$\\boldsymbol{y}=[y_1,y_2,\\dots,y_n]$. \n",
"\n",
"\n",
"## The last steps\n",
@@ -779,7 +779,7 @@
"metadata": {},
"source": [
"$$\n",
"\\mathbf{w}=\\sum_{i} \\lambda_iy_i\\mathbf{x}_i.\n",
"\\boldsymbol{w}=\\sum_{i} \\lambda_iy_i\\boldsymbol{x}_i.\n",
"$$"
]
},
@@ -787,7 +787,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"With our vector $\\mathbf{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via"
"With our vector $\\boldsymbol{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via"
]
},
{
@@ -795,7 +795,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\mathbf{w}^T\\mathbf{x}_i+b)=1,\n",
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1,\n",
"$$"
]
},
@@ -811,7 +811,7 @@
"metadata": {},
"source": [
"$$\n",
"b = \\frac{1}{y_i}-\\mathbf{w}^T\\mathbf{x}_i,\n",
"b = \\frac{1}{y_i}-\\boldsymbol{w}^T\\boldsymbol{x}_i,\n",
"$$"
]
},
@@ -827,7 +827,7 @@
"metadata": {},
"source": [
"$$\n",
"b = \\frac{1}{N_s}\\sum_{j\\in N_s}\\left(y_j-\\sum_{i=1}^n\\lambda_iy_i\\mathbf{x}_i^T\\mathbf{x}_j\\right).\n",
"b = \\frac{1}{N_s}\\sum_{j\\in N_s}\\left(y_j-\\sum_{i=1}^n\\lambda_iy_i\\boldsymbol{x}_i^T\\boldsymbol{x}_j\\right).\n",
"$$"
]
},
@@ -843,7 +843,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i = \\mathrm{sign}(\\mathbf{w}^T\\mathbf{x}_i+b).\n",
"y_i = \\mathrm{sign}(\\boldsymbol{w}^T\\boldsymbol{x}_i+b).\n",
"$$"
]
},
@@ -862,7 +862,7 @@
"so-called **kernel approach**, is to allow a kind of slack in the sense\n",
"that we allow some points to be on the wrong side of the margin.\n",
"\n",
"We introduce thus the so-called **slack** variables $\\mathbf{\\xi} =[\\xi_1,x_2,\\dots,x_n]$ and \n",
"We introduce thus the so-called **slack** variables $\\boldsymbol{\\xi} =[\\xi_1,x_2,\\dots,x_n]$ and \n",
"modify our previous equation"
]
},
@@ -871,7 +871,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\mathbf{w}^T\\mathbf{x}_i+b)=1,\n",
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1,\n",
"$$"
]
},
@@ -887,7 +887,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\mathbf{w}^T\\mathbf{x}_i+b)=1-\\xi_i,\n",
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1-\\xi_i,\n",
"$$"
]
},
@@ -897,7 +897,7 @@
"source": [
"with the requirement $\\xi_i\\geq 0$. The total violation is now $\\sum_i\\xi$. \n",
"The value $\\xi_i$ in the constraint the last constraint corresponds to the amount by which the prediction\n",
"$y_i(\\mathbf{w}^T\\mathbf{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\\sum_i \\xi_i$,\n",
"$y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\\sum_i \\xi_i$,\n",
"we bound the total amount by which predictions fall on the wrong side of their margins.\n",
"\n",
"Misclassifications occur when $\\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of\n",
@@ -914,7 +914,7 @@
"metadata": {},
"source": [
"$$\n",
"{\\cal L}=\\frac{1}{2}\\mathbf{w}^T\\mathbf{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\mathbf{w}^T\\mathbf{x}_i+b)-(1-\\xi_)\\right]+C\\sum_{i=1}^n\\xi_i-\\sum_{i=1}^n\\gamma_i\\xi_i,\n",
"{\\cal L}=\\frac{1}{2}\\boldsymbol{w}^T\\boldsymbol{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)-(1-\\xi_)\\right]+C\\sum_{i=1}^n\\xi_i-\\sum_{i=1}^n\\gamma_i\\xi_i,\n",
"$$"
]
},
@@ -930,7 +930,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\mathbf{w}^T\\mathbf{x}_i+b)=1-\\xi_i \\hspace{0.1cm}\\forall i,\n",
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1-\\xi_i \\hspace{0.1cm}\\forall i,\n",
"$$"
]
},
@@ -940,7 +940,7 @@
"source": [
"with the requirement $\\xi_i\\geq 0$.\n",
"\n",
"Taking the derivatives with respect to $b$ and $\\mathbf{w}$ we obtain"
"Taking the derivatives with respect to $b$ and $\\boldsymbol{w}$ we obtain"
]
},
{
@@ -964,7 +964,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial {\\cal L}}{\\partial \\mathbf{w}} = 0 = \\mathbf{w}-\\sum_{i} \\lambda_iy_i\\mathbf{x}_i,\n",
"\\frac{\\partial {\\cal L}}{\\partial \\boldsymbol{w}} = 0 = \\boldsymbol{w}-\\sum_{i} \\lambda_iy_i\\boldsymbol{x}_i,\n",
"$$"
]
},
@@ -996,7 +996,7 @@
"metadata": {},
"source": [
"$$\n",
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\mathbf{x}_i^T\\mathbf{x}_j,\n",
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{x}_j,\n",
"$$"
]
},
@@ -1053,7 +1053,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i(\\mathbf{w}^T\\mathbf{x}_i+b) -(1-\\xi_) \\geq 0 \\hspace{0.1cm}\\forall i.\n",
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) -(1-\\xi_) \\geq 0 \\hspace{0.1cm}\\forall i.\n",
"$$"
]
},
doc/pub/svm/pdf/svm-minted.pdf
BIN
View File
Binary file not shown.
doc/src/SupportVMachines/svm.do.txt
+58 -58
View File
@@ -58,11 +58,11 @@ b+w_1x_1+w_2x_2=0,
!et
where $b$ is the intercept and $w_1$ and $w_2$ define the elements of a vector orthogonal to the line
$b+w_1x_1+w_2x_2=0$.
In two dimensions we define the vectors $\mathbf{x} =[x1,x2]$ and $\mathbf{w}=[w1,w2]$.
In two dimensions we define the vectors $\bm{x} =[x1,x2]$ and $\bm{w}=[w1,w2]$.
We can then rewrite the above equation as
!bt
\[
\mathbf{w}^T\mathbf{x}+b=0.
\bm{w}^T\bm{x}+b=0.
\]
!et
@@ -77,28 +77,28 @@ b+wx_1+w_2x_2+\dots +w_px_p=0.
\]
!et
If we define a
matrix $\mathbf{X}=\left[\mathbf{x}_1,\mathbf{x}_2,\dots, \mathbf{x}_p\right]$
of dimension $n\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\mathbf{X}$,
matrix $\bm{X}=\left[\bm{x}_1,\bm{x}_2,\dots, \bm{x}_p\right]$
of dimension $n\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\bm{X}$,
!bt
\[
\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\bm{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\]
!et
If the above condition is not met for a given vector $\mathbf{x}_i$ we have
If the above condition is not met for a given vector $\bm{x}_i$ we have
!bt
\[
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
\]
!et
if our output $y_i=1$.
In this case we say that $\mathbf{x}_i$ lies on one of the sides of the hyperplane and if
In this case we say that $\bm{x}_i$ lies on one of the sides of the hyperplane and if
!bt
\[
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
\]
!et
for the class of observations $y_i=-1$,
then $\mathbf{x}_i$ lies on the other side.
then $\bm{x}_i$ lies on the other side.
Equivalently, for the two classes of observations we have
!bt
@@ -137,35 +137,35 @@ for our data sample.
Let us define the function
!bt
\[
f(x) = \mathbf{w}^T\mathbf{x}+b = 0,
f(x) = \bm{w}^T\bm{x}+b = 0,
\]
!et
as the function that determines the line $L$ that separates two classes (our two features), see the figure here.
Any point defined by $\mathbf{x}_i$ and $\mathbf{x}_2$ on the line $L$ will satisfy $\mathbf{w}^T(\mathbf{x}_1-\mathbf{x}_2)=0$.
Any point defined by $\bm{x}_i$ and $\bm{x}_2$ on the line $L$ will satisfy $\bm{w}^T(\bm{x}_1-\bm{x}_2)=0$.
The signed distance $\delta$ from any point defined by a vector $\mathbf{x}$ and a point $\mathbf{x}_0$ on the line $L$ is then
The signed distance $\delta$ from any point defined by a vector $\bm{x}$ and a point $\bm{x}_0$ on the line $L$ is then
!bt
\[
\delta = \frac{1}{\vert\vert \mathbf{w}\vert\vert}(\mathbf{w}^T\mathbf{x}+b).
\delta = \frac{1}{\vert\vert \bm{w}\vert\vert}(\bm{w}^T\bm{x}+b).
\]
!et
!split
===== First attempt at a minimization approach =====
How do we find the parameter $b$ and the vector $\mathbf{w}$? What we could
How do we find the parameter $b$ and the vector $\bm{w}$? What we could
do is to define a cost function which now contains the set of all
misclassified points $M$ and attempt to minimize this function
!bt
\[
C(\mathbf{w},b) = -\sum_{i\in M} y_i(\mathbf{w}^T\mathbf{x}_i+b).
C(\bm{w},b) = -\sum_{i\in M} y_i(\bm{w}^T\bm{x}_i+b).
\]
!et
We could now for example define all values $y_i =1$ as misclassified in case we have $\mathbf{w}^T\mathbf{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us
We could now for example define all values $y_i =1$ as misclassified in case we have $\bm{w}^T\bm{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us
!bt
\[
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
@@ -174,7 +174,7 @@ We could now for example define all values $y_i =1$ as misclassified in case we
and
!bt
\[
\frac{\partial C}{\partial \mathbf{w}} = -\sum_{i\in M} y_ix_i.
\frac{\partial C}{\partial \bm{w}} = -\sum_{i\in M} y_ix_i.
\]
!et
@@ -190,7 +190,7 @@ b \leftarrow b +\eta \frac{\partial C}{\partial b},
and
!bt
\[
\mathbf{w} \leftarrow \mathbf{w} +\eta \frac{\partial C}{\partial \mathbf{w}},
\bm{w} \leftarrow \bm{w} +\eta \frac{\partial C}{\partial \bm{w}},
\]
!et
where $\eta$ is our by now well-known learning rate.
@@ -210,12 +210,12 @@ at all.
A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).
Thus, we wish to find a margin $M$ with $\mathbf{w}$ normalized to
$\vert\vert \mathbf{w}\vert\vert =1$ subject to the condition
Thus, we wish to find a margin $M$ with $\bm{w}$ normalized to
$\vert\vert \bm{w}\vert\vert =1$ subject to the condition
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
\]
!et
All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $b$ and $w_1$ and $w_2$ define this line.
@@ -223,24 +223,24 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value $M$ defined by
!bt
\[
\frac{1}{\vert \vert \mathbf{w}\vert\vert}y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \bm{w}\vert\vert}y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\]
!et
or just
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M\vert \vert \mathbf{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\bm{w}^T\bm{x}_i+b) \geq M\vert \vert \bm{w}\vert\vert \hspace{0.1cm}\forall i.
\]
!et
If we scale the equation so that $\vert \vert \mathbf{w}\vert\vert = 1/M$, we have to find the minimum of
$\mathbf{w}^T\mathbf{w}=\vert \vert \mathbf{w}\vert\vert$ (the norm) subject to the condition
If we scale the equation so that $\vert \vert \bm{w}\vert\vert = 1/M$, we have to find the minimum of
$\bm{w}^T\bm{w}=\vert \vert \bm{w}\vert\vert$ (the norm) subject to the condition
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\bm{w}^T\bm{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
\]
!et
We have thus defined our margin as the invers of the norm of $\mathbf{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers.
We have thus defined our margin as the invers of the norm of $\bm{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers.
!split
===== A quick reminder on Lagrangian multipliers =====
@@ -346,12 +346,12 @@ If we have a set of constraints $\phi_k$ we have the equations
In order to solve the above problem, we define the following Lagrangian function to be minimized
!bt
\[
{\cal L}(\lambda,b,\mathbf{w})=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-1\right],
{\cal L}(\lambda,b,\bm{w})=\frac{1}{2}\bm{w}^T\bm{w}-\sum_{i=1}^n\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b)-1\right],
\]
!et
where $\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\lambda_i \geq 0$.
Taking the derivatives with respect to $b$ and $\mathbf{w}$ we obtain
Taking the derivatives with respect to $b$ and $\bm{w}$ we obtain
!bt
\[
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -360,25 +360,25 @@ Taking the derivatives with respect to $b$ and $\mathbf{w}$ we obtain
and
!bt
\[
\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i.
\frac{\partial {\cal L}}{\partial \bm{w}} = 0 = \bm{w}-\sum_{i} \lambda_iy_i\bm{x}_i.
\]
!et
Inserting these constraints into the equation for ${\cal L}$ we obtain
!bt
\[
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{x}_j,
\]
!et
subject to the constraints $\lambda_i\geq 0$ and $\sum_i\lambda_iy_i=0$.
We must in addition satisfy the "Karush-Kuhn-Tucker":"https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" (KKT) condition
!bt
\[
\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\]
!et
o If $\lambda_i > 0$, then $y_i(\mathbf{w}^T\mathbf{x}_i+b)=1$ and we say that $x_i$ is on the boundary.
o If $y_i(\mathbf{w}^T\mathbf{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$.
When $\lambda_i > 0$, the vectors $\mathbf{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$.
o If $\lambda_i > 0$, then $y_i(\bm{w}^T\bm{x}_i+b)=1$ and we say that $x_i$ is on the boundary.
o If $y_i(\bm{w}^T\bm{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$.
When $\lambda_i > 0$, the vectors $\bm{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$.
!split
===== The problem to solve =====
@@ -386,22 +386,22 @@ When $\lambda_i > 0$, the vectors $\mathbf{x}_i$ are called support vectors. The
We can rewrite
!bt
\[
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{x}_j,
\]
!et
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. $\lambda$ the following problem
!bt
\[
\frac{1}{2} \mathbf{\lambda}^T\begin{bmatrix} y_1y_1\mathbf{x}_1^T\mathbf{x}_1 & y_1y_2\mathbf{x}_1^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_1^T\mathbf{x}_n \\
y_2y_1\mathbf{x}_2^T\mathbf{x}_1 & y_2y_2\mathbf{x}_2^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_2^T\mathbf{x}_n \\
\frac{1}{2} \bm{\lambda}^T\begin{bmatrix} y_1y_1\bm{x}_1^T\bm{x}_1 & y_1y_2\bm{x}_1^T\bm{x}_2 & \dots & \dots & y_1y_n\bm{x}_1^T\bm{x}_n \\
y_2y_1\bm{x}_2^T\bm{x}_1 & y_2y_2\bm{x}_2^T\bm{x}_2 & \dots & \dots & y_1y_n\bm{x}_2^T\bm{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1\mathbf{x}_n^T\mathbf{x}_1 & y_ny_2\mathbf{x}_n^T\mathbf{x}_2 & \dots & \dots & y_ny_n\mathbf{x}_n^T\mathbf{x}_n \\
\end{bmatrix}\mathbf{\lambda}-\mathbb{1}\mathbf{\lambda},
y_ny_1\bm{x}_n^T\bm{x}_1 & y_ny_2\bm{x}_n^T\bm{x}_2 & \dots & \dots & y_ny_n\bm{x}_n^T\bm{x}_n \\
\end{bmatrix}\bm{\lambda}-\mathbb{1}\bm{\lambda},
\]
!et
subject to $\mathbf{y}^T\mathbf{\lambda}=0$. Here we defined the vectors $\mathbf{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and
$\mathbf{y}=[y_1,y_2,\dots,y_n]$.
subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and
$\bm{y}=[y_1,y_2,\dots,y_n]$.
!split
@@ -411,31 +411,31 @@ Solving the above problem, yields the values of $\lambda_i$.
To find the coefficients of your hyperplane we need simply to compute
!bt
\[
\mathbf{w}=\sum_{i} \lambda_iy_i\mathbf{x}_i.
\bm{w}=\sum_{i} \lambda_iy_i\bm{x}_i.
\]
!et
With our vector $\mathbf{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via
With our vector $\bm{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
y_i(\bm{w}^T\bm{x}_i+b)=1,
\]
!et
resulting in
!bt
\[
b = \frac{1}{y_i}-\mathbf{w}^T\mathbf{x}_i,
b = \frac{1}{y_i}-\bm{w}^T\bm{x}_i,
\]
!et
or if we write it out in terms of the support vectors only, with $N_s$ being their number, we have
!bt
\[
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\mathbf{x}_i^T\mathbf{x}_j\right).
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\bm{x}_i^T\bm{x}_j\right).
\]
!et
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
!bt
\[
y_i = \mathrm{sign}(\mathbf{w}^T\mathbf{x}_i+b).
y_i = \mathrm{sign}(\bm{w}^T\bm{x}_i+b).
\]
!et
Below we discuss how to find the optimal values of $\lambda_i$. Before we proceed however, we discuss now the so-called soft classifier.
@@ -450,22 +450,22 @@ figure here. One way to deal with this problem before we define the
so-called _kernel approach_, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
We introduce thus the so-called _slack_ variables $\mathbf{\xi} =[\xi_1,x_2,\dots,x_n]$ and
We introduce thus the so-called _slack_ variables $\bm{\xi} =[\xi_1,x_2,\dots,x_n]$ and
modify our previous equation
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
y_i(\bm{w}^T\bm{x}_i+b)=1,
\]
!et
to
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i,
y_i(\bm{w}^T\bm{x}_i+b)=1-\xi_i,
\]
!et
with the requirement $\xi_i\geq 0$. The total violation is now $\sum_i\xi$.
The value $\xi_i$ in the constraint the last constraint corresponds to the amount by which the prediction
$y_i(\mathbf{w}^T\mathbf{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$,
$y_i(\bm{w}^T\bm{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$,
we bound the total amount by which predictions fall on the wrong side of their margins.
Misclassifications occur when $\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of
@@ -478,18 +478,18 @@ misclassifications.
This has in turn the consequences that we change our optmization problem to finding the minimum of
!bt
\[
{\cal L}=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
{\cal L}=\frac{1}{2}\bm{w}^T\bm{w}-\sum_{i=1}^n\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
\]
!et
subject to
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\bm{w}^T\bm{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
\]
!et
with the requirement $\xi_i\geq 0$.
Taking the derivatives with respect to $b$ and $\mathbf{w}$ we obtain
Taking the derivatives with respect to $b$ and $\bm{w}$ we obtain
!bt
\[
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -498,7 +498,7 @@ Taking the derivatives with respect to $b$ and $\mathbf{w}$ we obtain
and
!bt
\[
\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i,
\frac{\partial {\cal L}}{\partial \bm{w}} = 0 = \bm{w}-\sum_{i} \lambda_iy_i\bm{x}_i,
\]
!et
and
@@ -510,14 +510,14 @@ and
Inserting these constraints into the equation for ${\cal L}$ we obtain the same equation as before
!bt
\[
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{x}_j,
\]
!et
but now subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$ and $0\leq\lambda_i \leq C$.
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
!bt
\[
\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\]
!et
!bt
@@ -528,7 +528,7 @@ We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
and
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\bm{w}^T\bm{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
\]
!et