lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

mathbf--> bm

Browse Source
This commit is contained in:
mhjensen
2018-11-05 21:27:51 +01:00
parent eaba63cadf
commit 3fe132f4ce
18 changed files with 347 additions and 347 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/SupportVMachines/svm.do.txt
+58 -58
View File
@@ -58,11 +58,11 @@ b+w_1x_1+w_2x_2=0,
!et
where $b$ is the intercept and $w_1$ and $w_2$ define the elements of a vector orthogonal to the line
$b+w_1x_1+w_2x_2=0$.
In two dimensions we define the vectors $\mathbf{x} =[x1,x2]$ and $\mathbf{w}=[w1,w2]$.
In two dimensions we define the vectors $\bm{x} =[x1,x2]$ and $\bm{w}=[w1,w2]$.
We can then rewrite the above equation as
!bt
\[
\mathbf{w}^T\mathbf{x}+b=0.
\bm{w}^T\bm{x}+b=0.
\]
!et
@@ -77,28 +77,28 @@ b+wx_1+w_2x_2+\dots +w_px_p=0.
\]
!et
If we define a
matrix $\mathbf{X}=\left[\mathbf{x}_1,\mathbf{x}_2,\dots, \mathbf{x}_p\right]$
of dimension $n\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\mathbf{X}$,
matrix $\bm{X}=\left[\bm{x}_1,\bm{x}_2,\dots, \bm{x}_p\right]$
of dimension $n\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\bm{X}$,
!bt
\[
\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\bm{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
\]
!et
If the above condition is not met for a given vector $\mathbf{x}_i$ we have
If the above condition is not met for a given vector $\bm{x}_i$ we have
!bt
\[
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
\]
!et
if our output $y_i=1$.
In this case we say that $\mathbf{x}_i$ lies on one of the sides of the hyperplane and if
In this case we say that $\bm{x}_i$ lies on one of the sides of the hyperplane and if
!bt
\[
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
\]
!et
for the class of observations $y_i=-1$,
then $\mathbf{x}_i$ lies on the other side.
then $\bm{x}_i$ lies on the other side.
Equivalently, for the two classes of observations we have
!bt
@@ -137,35 +137,35 @@ for our data sample.
Let us define the function
!bt
\[
f(x) = \mathbf{w}^T\mathbf{x}+b = 0,
f(x) = \bm{w}^T\bm{x}+b = 0,
\]
!et
as the function that determines the line $L$ that separates two classes (our two features), see the figure here.
Any point defined by $\mathbf{x}_i$ and $\mathbf{x}_2$ on the line $L$ will satisfy $\mathbf{w}^T(\mathbf{x}_1-\mathbf{x}_2)=0$.
Any point defined by $\bm{x}_i$ and $\bm{x}_2$ on the line $L$ will satisfy $\bm{w}^T(\bm{x}_1-\bm{x}_2)=0$.
The signed distance $\delta$ from any point defined by a vector $\mathbf{x}$ and a point $\mathbf{x}_0$ on the line $L$ is then
The signed distance $\delta$ from any point defined by a vector $\bm{x}$ and a point $\bm{x}_0$ on the line $L$ is then
!bt
\[
\delta = \frac{1}{\vert\vert \mathbf{w}\vert\vert}(\mathbf{w}^T\mathbf{x}+b).
\delta = \frac{1}{\vert\vert \bm{w}\vert\vert}(\bm{w}^T\bm{x}+b).
\]
!et
!split
===== First attempt at a minimization approach =====
How do we find the parameter $b$ and the vector $\mathbf{w}$? What we could
How do we find the parameter $b$ and the vector $\bm{w}$? What we could
do is to define a cost function which now contains the set of all
misclassified points $M$ and attempt to minimize this function
!bt
\[
C(\mathbf{w},b) = -\sum_{i\in M} y_i(\mathbf{w}^T\mathbf{x}_i+b).
C(\bm{w},b) = -\sum_{i\in M} y_i(\bm{w}^T\bm{x}_i+b).
\]
!et
We could now for example define all values $y_i =1$ as misclassified in case we have $\mathbf{w}^T\mathbf{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us
We could now for example define all values $y_i =1$ as misclassified in case we have $\bm{w}^T\bm{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us
!bt
\[
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
@@ -174,7 +174,7 @@ We could now for example define all values $y_i =1$ as misclassified in case we
and
!bt
\[
\frac{\partial C}{\partial \mathbf{w}} = -\sum_{i\in M} y_ix_i.
\frac{\partial C}{\partial \bm{w}} = -\sum_{i\in M} y_ix_i.
\]
!et
@@ -190,7 +190,7 @@ b \leftarrow b +\eta \frac{\partial C}{\partial b},
and
!bt
\[
\mathbf{w} \leftarrow \mathbf{w} +\eta \frac{\partial C}{\partial \mathbf{w}},
\bm{w} \leftarrow \bm{w} +\eta \frac{\partial C}{\partial \bm{w}},
\]
!et
where $\eta$ is our by now well-known learning rate.
@@ -210,12 +210,12 @@ at all.
A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).
Thus, we wish to find a margin $M$ with $\mathbf{w}$ normalized to
$\vert\vert \mathbf{w}\vert\vert =1$ subject to the condition
Thus, we wish to find a margin $M$ with $\bm{w}$ normalized to
$\vert\vert \bm{w}\vert\vert =1$ subject to the condition
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
\]
!et
All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $b$ and $w_1$ and $w_2$ define this line.
@@ -223,24 +223,24 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value $M$ defined by
!bt
\[
\frac{1}{\vert \vert \mathbf{w}\vert\vert}y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \bm{w}\vert\vert}y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\]
!et
or just
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M\vert \vert \mathbf{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\bm{w}^T\bm{x}_i+b) \geq M\vert \vert \bm{w}\vert\vert \hspace{0.1cm}\forall i.
\]
!et
If we scale the equation so that $\vert \vert \mathbf{w}\vert\vert = 1/M$, we have to find the minimum of
$\mathbf{w}^T\mathbf{w}=\vert \vert \mathbf{w}\vert\vert$ (the norm) subject to the condition
If we scale the equation so that $\vert \vert \bm{w}\vert\vert = 1/M$, we have to find the minimum of
$\bm{w}^T\bm{w}=\vert \vert \bm{w}\vert\vert$ (the norm) subject to the condition
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\bm{w}^T\bm{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
\]
!et
We have thus defined our margin as the invers of the norm of $\mathbf{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers.
We have thus defined our margin as the invers of the norm of $\bm{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers.
!split
===== A quick reminder on Lagrangian multipliers =====
@@ -346,12 +346,12 @@ If we have a set of constraints $\phi_k$ we have the equations
In order to solve the above problem, we define the following Lagrangian function to be minimized
!bt
\[
{\cal L}(\lambda,b,\mathbf{w})=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-1\right],
{\cal L}(\lambda,b,\bm{w})=\frac{1}{2}\bm{w}^T\bm{w}-\sum_{i=1}^n\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b)-1\right],
\]
!et
where $\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\lambda_i \geq 0$.
Taking the derivatives with respect to $b$ and $\mathbf{w}$ we obtain
Taking the derivatives with respect to $b$ and $\bm{w}$ we obtain
!bt
\[
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -360,25 +360,25 @@ Taking the derivatives with respect to $b$ and $\mathbf{w}$ we obtain
and
!bt
\[
\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i.
\frac{\partial {\cal L}}{\partial \bm{w}} = 0 = \bm{w}-\sum_{i} \lambda_iy_i\bm{x}_i.
\]
!et
Inserting these constraints into the equation for ${\cal L}$ we obtain
!bt
\[
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{x}_j,
\]
!et
subject to the constraints $\lambda_i\geq 0$ and $\sum_i\lambda_iy_i=0$.
We must in addition satisfy the "Karush-Kuhn-Tucker":"https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" (KKT) condition
!bt
\[
\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\]
!et
o If $\lambda_i > 0$, then $y_i(\mathbf{w}^T\mathbf{x}_i+b)=1$ and we say that $x_i$ is on the boundary.
o If $y_i(\mathbf{w}^T\mathbf{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$.
When $\lambda_i > 0$, the vectors $\mathbf{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$.
o If $\lambda_i > 0$, then $y_i(\bm{w}^T\bm{x}_i+b)=1$ and we say that $x_i$ is on the boundary.
o If $y_i(\bm{w}^T\bm{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$.
When $\lambda_i > 0$, the vectors $\bm{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$.
!split
===== The problem to solve =====
@@ -386,22 +386,22 @@ When $\lambda_i > 0$, the vectors $\mathbf{x}_i$ are called support vectors. The
We can rewrite
!bt
\[
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{x}_j,
\]
!et
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. $\lambda$ the following problem
!bt
\[
\frac{1}{2} \mathbf{\lambda}^T\begin{bmatrix} y_1y_1\mathbf{x}_1^T\mathbf{x}_1 & y_1y_2\mathbf{x}_1^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_1^T\mathbf{x}_n \\
y_2y_1\mathbf{x}_2^T\mathbf{x}_1 & y_2y_2\mathbf{x}_2^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_2^T\mathbf{x}_n \\
\frac{1}{2} \bm{\lambda}^T\begin{bmatrix} y_1y_1\bm{x}_1^T\bm{x}_1 & y_1y_2\bm{x}_1^T\bm{x}_2 & \dots & \dots & y_1y_n\bm{x}_1^T\bm{x}_n \\
y_2y_1\bm{x}_2^T\bm{x}_1 & y_2y_2\bm{x}_2^T\bm{x}_2 & \dots & \dots & y_1y_n\bm{x}_2^T\bm{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1\mathbf{x}_n^T\mathbf{x}_1 & y_ny_2\mathbf{x}_n^T\mathbf{x}_2 & \dots & \dots & y_ny_n\mathbf{x}_n^T\mathbf{x}_n \\
\end{bmatrix}\mathbf{\lambda}-\mathbb{1}\mathbf{\lambda},
y_ny_1\bm{x}_n^T\bm{x}_1 & y_ny_2\bm{x}_n^T\bm{x}_2 & \dots & \dots & y_ny_n\bm{x}_n^T\bm{x}_n \\
\end{bmatrix}\bm{\lambda}-\mathbb{1}\bm{\lambda},
\]
!et
subject to $\mathbf{y}^T\mathbf{\lambda}=0$. Here we defined the vectors $\mathbf{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and
$\mathbf{y}=[y_1,y_2,\dots,y_n]$.
subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and
$\bm{y}=[y_1,y_2,\dots,y_n]$.
!split
@@ -411,31 +411,31 @@ Solving the above problem, yields the values of $\lambda_i$.
To find the coefficients of your hyperplane we need simply to compute
!bt
\[
\mathbf{w}=\sum_{i} \lambda_iy_i\mathbf{x}_i.
\bm{w}=\sum_{i} \lambda_iy_i\bm{x}_i.
\]
!et
With our vector $\mathbf{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via
With our vector $\bm{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
y_i(\bm{w}^T\bm{x}_i+b)=1,
\]
!et
resulting in
!bt
\[
b = \frac{1}{y_i}-\mathbf{w}^T\mathbf{x}_i,
b = \frac{1}{y_i}-\bm{w}^T\bm{x}_i,
\]
!et
or if we write it out in terms of the support vectors only, with $N_s$ being their number, we have
!bt
\[
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\mathbf{x}_i^T\mathbf{x}_j\right).
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\bm{x}_i^T\bm{x}_j\right).
\]
!et
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
!bt
\[
y_i = \mathrm{sign}(\mathbf{w}^T\mathbf{x}_i+b).
y_i = \mathrm{sign}(\bm{w}^T\bm{x}_i+b).
\]
!et
Below we discuss how to find the optimal values of $\lambda_i$. Before we proceed however, we discuss now the so-called soft classifier.
@@ -450,22 +450,22 @@ figure here. One way to deal with this problem before we define the
so-called _kernel approach_, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
We introduce thus the so-called _slack_ variables $\mathbf{\xi} =[\xi_1,x_2,\dots,x_n]$ and
We introduce thus the so-called _slack_ variables $\bm{\xi} =[\xi_1,x_2,\dots,x_n]$ and
modify our previous equation
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
y_i(\bm{w}^T\bm{x}_i+b)=1,
\]
!et
to
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i,
y_i(\bm{w}^T\bm{x}_i+b)=1-\xi_i,
\]
!et
with the requirement $\xi_i\geq 0$. The total violation is now $\sum_i\xi$.
The value $\xi_i$ in the constraint the last constraint corresponds to the amount by which the prediction
$y_i(\mathbf{w}^T\mathbf{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$,
$y_i(\bm{w}^T\bm{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$,
we bound the total amount by which predictions fall on the wrong side of their margins.
Misclassifications occur when $\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of
@@ -478,18 +478,18 @@ misclassifications.
This has in turn the consequences that we change our optmization problem to finding the minimum of
!bt
\[
{\cal L}=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
{\cal L}=\frac{1}{2}\bm{w}^T\bm{w}-\sum_{i=1}^n\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
\]
!et
subject to
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\bm{w}^T\bm{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
\]
!et
with the requirement $\xi_i\geq 0$.
Taking the derivatives with respect to $b$ and $\mathbf{w}$ we obtain
Taking the derivatives with respect to $b$ and $\bm{w}$ we obtain
!bt
\[
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -498,7 +498,7 @@ Taking the derivatives with respect to $b$ and $\mathbf{w}$ we obtain
and
!bt
\[
\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i,
\frac{\partial {\cal L}}{\partial \bm{w}} = 0 = \bm{w}-\sum_{i} \lambda_iy_i\bm{x}_i,
\]
!et
and
@@ -510,14 +510,14 @@ and
Inserting these constraints into the equation for ${\cal L}$ we obtain the same equation as before
!bt
\[
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{x}_j,
\]
!et
but now subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$ and $0\leq\lambda_i \leq C$.
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
!bt
\[
\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\]
!et
!bt
@@ -528,7 +528,7 @@ We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
and
!bt
\[
y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\bm{w}^T\bm{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
\]
!et