diff --git a/doc/pub/svm/html/._svm-bs004.html b/doc/pub/svm/html/._svm-bs004.html index 3b7fe2b3d..5e5720ac8 100644 --- a/doc/pub/svm/html/._svm-bs004.html +++ b/doc/pub/svm/html/._svm-bs004.html @@ -139,25 +139,25 @@ b+wx_1+w_2x_2+\dots +w_px_p=0. $$ If we define a -matrix \( \hat{X}=\left[\hat{x}_1,\hat{x}_2,\dots, \hat{x}_p\right] \) -of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \hat{X} \), +matrix \( \boldmath{X}=\left[\boldmath{x}_1,\boldmath{x}_2,\dots, \boldmath{x}_p\right] \) +of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldmath{X} \), $$ -\hat{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. +\boldmath{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. $$ -If the above condition is not met for a given vector \( \hat{x}_i \) we have +If the above condition is not met for a given vector \( \boldmath{x}_i \) we have $$ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0, $$ if our output \( y_i=1 \). -In this case we say that \( \hat{x}_i \) lies on one of the sides of the hyperplane and if +In this case we say that \( \boldmath{x}_i \) lies on one of the sides of the hyperplane and if $$ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0, $$ for the class of observations \( y_i=-1 \), -then \( \hat{x}_i \) lies on the other side. +then \( \boldmath{x}_i \) lies on the other side.
Equivalently, for the two classes of observations we have diff --git a/doc/pub/svm/html/._svm-bs006.html b/doc/pub/svm/html/._svm-bs006.html index 1235e3f36..032060832 100644 --- a/doc/pub/svm/html/._svm-bs006.html +++ b/doc/pub/svm/html/._svm-bs006.html @@ -134,18 +134,18 @@ MathJax.Hub.Config({
Let us define the function $$ -f(x) = \hat{w}^T\hat{x}+b = 0, +f(x) = \boldmath{w}^T\boldmath{x}+b = 0, $$ as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
-Any point defined by \( \hat{x}_i \) and \( \hat{x}_2 \) on the line \( L \) will satisfy \( \hat{w}^T(\hat{x}_1-\hat{x}_2)=0 \). +Any point defined by \( \boldmath{x}_i \) and \( \boldmath{x}_2 \) on the line \( L \) will satisfy \( \boldmath{w}^T(\boldmath{x}_1-\boldmath{x}_2)=0 \).
-The signed distance \( \delta \) from any point defined by a vector \( \hat{x} \) and a point \( \hat{x}_0 \) on the line \( L \) is then +The signed distance \( \delta \) from any point defined by a vector \( \boldmath{x} \) and a point \( \boldmath{x}_0 \) on the line \( L \) is then $$ -\delta = \frac{1}{\vert\vert \hat{w}\vert\vert}(\hat{w}^T\hat{x}+b). +\delta = \frac{1}{\vert\vert \boldmath{w}\vert\vert}(\boldmath{w}^T\boldmath{x}+b). $$
diff --git a/doc/pub/svm/html/._svm-bs007.html b/doc/pub/svm/html/._svm-bs007.html index 4f4d81d43..4b594b59f 100644 --- a/doc/pub/svm/html/._svm-bs007.html +++ b/doc/pub/svm/html/._svm-bs007.html @@ -132,23 +132,23 @@ MathJax.Hub.Config({
-How do we find the parameter \( b \) and the vector \( \hat{w} \)? What we could +How do we find the parameter \( b \) and the vector \( \boldmath{w} \)? What we could do is to define a cost function which now contains the set of all misclassified points \( M \) and attempt to minimize this function $$ -C(\hat{w},b) = -\sum_{i\in M} y_i(\hat{w}^T\hat{x}_i+b). +C(\boldmath{w},b) = -\sum_{i\in M} y_i(\boldmath{w}^T\boldmath{x}_i+b). $$
-We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \hat{w}^T\hat{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us +We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldmath{w}^T\boldmath{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us $$ \frac{\partial C}{\partial b} = -\sum_{i\in M} y_i, $$ and $$ -\frac{\partial C}{\partial \hat{w}} = -\sum_{i\in M} y_ix_i. +\frac{\partial C}{\partial \boldmath{w}} = -\sum_{i\in M} y_ix_i. $$
diff --git a/doc/pub/svm/html/._svm-bs008.html b/doc/pub/svm/html/._svm-bs008.html index 8042fa5b6..9890b9a82 100644 --- a/doc/pub/svm/html/._svm-bs008.html +++ b/doc/pub/svm/html/._svm-bs008.html @@ -139,7 +139,7 @@ $$ and $$ -\hat{w} \leftarrow \hat{w} +\eta \frac{\partial C}{\partial \hat{w}}, +\boldmath{w} \leftarrow \boldmath{w} +\eta \frac{\partial C}{\partial \boldmath{w}}, $$ where \( \eta \) is our by now well-known learning rate. diff --git a/doc/pub/svm/html/._svm-bs009.html b/doc/pub/svm/html/._svm-bs009.html index 501fff25e..0e3ba1f0d 100644 --- a/doc/pub/svm/html/._svm-bs009.html +++ b/doc/pub/svm/html/._svm-bs009.html @@ -136,11 +136,11 @@ A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
-Thus, we wish to find a margin \( M \) with \( \hat{w} \) normalized to -\( \vert\vert \hat{w}\vert\vert =1 \) subject to the condition +Thus, we wish to find a margin \( M \) with \( \boldmath{w} \) normalized to +\( \vert\vert \boldmath{w}\vert\vert =1 \) subject to the condition $$ -y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. $$ All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line. @@ -148,22 +148,22 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value \( M \) defined by $$ -\frac{1}{\vert \vert \hat{w}\vert\vert}y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, +\frac{1}{\vert \vert \boldmath{w}\vert\vert}y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, $$ or just $$ -y_i(\hat{w}^T\hat{x}_i+b) \geq M\vert \vert \hat{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M\vert \vert \boldmath{w}\vert\vert \hspace{0.1cm}\forall i. $$ -If we scale the equation so that \( \vert \vert \hat{w}\vert\vert = 1/M \), we have to find the minimum of -\( \hat{w}^T\hat{w}=\vert \vert \hat{w}\vert\vert \) (the norm) subject to the condition +If we scale the equation so that \( \vert \vert \boldmath{w}\vert\vert = 1/M \), we have to find the minimum of +\( \boldmath{w}^T\boldmath{w}=\vert \vert \boldmath{w}\vert\vert \) (the norm) subject to the condition $$ -y_i(\hat{w}^T\hat{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. $$
-We have thus defined our margin as the invers of the norm of \( \hat{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers. +We have thus defined our margin as the invers of the norm of \( \boldmath{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
diff --git a/doc/pub/svm/html/._svm-bs012.html b/doc/pub/svm/html/._svm-bs012.html index 5c23450ec..458c54650 100644 --- a/doc/pub/svm/html/._svm-bs012.html +++ b/doc/pub/svm/html/._svm-bs012.html @@ -132,40 +132,40 @@ MathJax.Hub.Config({
-Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ and $$ -\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i. +\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i. $$ Inserting these constraints into the equation for \( {\cal L} \) we obtain $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j, $$ subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). We must in addition satisfy the Karush-Kuhn-Tucker (KKT) condition $$ -\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -1\right] \hspace{0.1cm}\forall i. +\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -1\right] \hspace{0.1cm}\forall i. $$
diff --git a/doc/pub/svm/html/._svm-bs013.html b/doc/pub/svm/html/._svm-bs013.html index 445734cd8..28d770194 100644 --- a/doc/pub/svm/html/._svm-bs013.html +++ b/doc/pub/svm/html/._svm-bs013.html @@ -134,21 +134,21 @@ MathJax.Hub.Config({
We can rewrite $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j, $$ and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem $$ -\frac{1}{2} \hat{\lambda}^T\begin{bmatrix} y_1y_1\hat{x}_1^T\hat{x}_1 & y_1y_2\hat{x}_1^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_1^T\hat{x}_n \\ -y_2y_1\hat{x}_2^T\hat{x}_1 & y_2y_2\hat{x}_2^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_2^T\hat{x}_n \\ +\frac{1}{2} \boldmath{\lambda}^T\begin{bmatrix} y_1y_1\boldmath{x}_1^T\boldmath{x}_1 & y_1y_2\boldmath{x}_1^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_1^T\boldmath{x}_n \\ +y_2y_1\boldmath{x}_2^T\boldmath{x}_1 & y_2y_2\boldmath{x}_2^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_2^T\boldmath{x}_n \\ \dots & \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots & \dots \\ -y_ny_1\hat{x}_n^T\hat{x}_1 & y_ny_2\hat{x}_n^T\hat{x}_2 & \dots & \dots & y_ny_n\hat{x}_n^T\hat{x}_n \\ -\end{bmatrix}\hat{\lambda}-\mathbb{1}\hat{\lambda}, +y_ny_1\boldmath{x}_n^T\boldmath{x}_1 & y_ny_2\boldmath{x}_n^T\boldmath{x}_2 & \dots & \dots & y_ny_n\boldmath{x}_n^T\boldmath{x}_n \\ +\end{bmatrix}\boldmath{\lambda}-\mathbb{1}\boldmath{\lambda}, $$ -subject to \( \hat{y}^T\hat{\lambda}=0 \). Here we defined the vectors \( \hat{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \hat{y}=[y_1,y_2,\dots,y_n] \). +subject to \( \boldmath{y}^T\boldmath{\lambda}=0 \). Here we defined the vectors \( \boldmath{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldmath{y}=[y_1,y_2,\dots,y_n] \).
diff --git a/doc/pub/svm/html/._svm-bs014.html b/doc/pub/svm/html/._svm-bs014.html index 1386bb634..904f103f0 100644 --- a/doc/pub/svm/html/._svm-bs014.html +++ b/doc/pub/svm/html/._svm-bs014.html @@ -135,22 +135,27 @@ MathJax.Hub.Config({ Solving the above problem, yields the values of \( \lambda_i \). To find the coefficients of your hyperplane we need simply to compute $$ -\hat{w}=\sum_{i} \lambda_iy_i\hat{x}_i. +\boldmath{w}=\sum_{i} \lambda_iy_i\boldmath{x}_i. $$ -With our vector \( \hat{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via +With our vector \( \boldmath{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via $$ -y_i(\hat{w}^T\hat{x}_i+b)=1, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1, $$ resulting in $$ -b = \frac{1}{y_i}-\hat{w}^T\hat{x}_i. +b = \frac{1}{y_i}-\boldmath{w}^T\boldmath{x}_i, +$$ + +or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have +$$ +b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldmath{x}_i^T\boldmath{x}_j\right). $$ With our hyperplane coefficients we can use our classifier to assign any observation by simply using $$ -y_i = \mathrm{sign}(\hat{w}^T\hat{x}_i+b). +y_i = \mathrm{sign}(\boldmath{w}^T\boldmath{x}_i+b). $$ Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier. diff --git a/doc/pub/svm/html/._svm-bs015.html b/doc/pub/svm/html/._svm-bs015.html index 3519f7ca7..bba5a8f03 100644 --- a/doc/pub/svm/html/._svm-bs015.html +++ b/doc/pub/svm/html/._svm-bs015.html @@ -141,20 +141,20 @@ so-called kernel approach, is to allow a kind of slack in the sense that we allow some points to be on the wrong side of the margin.
-We introduce thus the so-called slack variables \( \hat{\xi} =[\xi_1,x_2,\dots,x_n] \) and +We introduce thus the so-called slack variables \( \boldmath{\xi} =[\xi_1,x_2,\dots,x_n] \) and modify our previous equation $$ -y_i(\hat{w}^T\hat{x}_i+b)=1, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1, $$ to $$ -y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i, $$ with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \). The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction -\( y_i(\hat{w}^T\hat{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), +\( y_i(\boldmath{w}^T\boldmath{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), we bound the total amount by which predictions fall on the wrong side of their margins.
diff --git a/doc/pub/svm/html/._svm-bs016.html b/doc/pub/svm/html/._svm-bs016.html index eb9103479..98ec07175 100644 --- a/doc/pub/svm/html/._svm-bs016.html +++ b/doc/pub/svm/html/._svm-bs016.html @@ -134,25 +134,25 @@ MathJax.Hub.Config({
This has in turn the consequences that we change our optmization problem to finding the minimum of $$ -{\cal L}=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, +{\cal L}=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, $$ subject to $$ -y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, $$ with the requirement \( \xi_i\geq 0 \).
-Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ and $$ -\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i, +\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i, $$ and @@ -162,13 +162,13 @@ $$ Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j, $$ but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \). We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads $$ -\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, +\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, $$ $$ @@ -177,7 +177,7 @@ $$ and $$ -y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. +y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. $$
diff --git a/doc/pub/svm/html/svm-reveal.html b/doc/pub/svm/html/svm-reveal.html index 0b0bc008c..534d3378d 100644 --- a/doc/pub/svm/html/svm-reveal.html +++ b/doc/pub/svm/html/svm-reveal.html @@ -225,11 +225,11 @@ $$ where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line \( b+w_1x_1+w_2x_2=0 \). -In two dimensions we define the vectors \( \hat{x} =[x1,x2] \) and \( \hat{w}=[w1,w2] \). +In two dimensions we define the vectors \( \boldmath{x} =[x1,x2] \) and \( \boldmath{w}=[w1,w2] \). We can then rewrite the above equation as
$$
-\hat{w}^T\hat{x}+b=0.
+\boldmath{w}^T\boldmath{x}+b=0.
$$
@@ -248,15 +248,15 @@ $$
If we define a
-matrix \( \hat{X}=\left[\hat{x}_1,\hat{x}_2,\dots, \hat{x}_p\right] \)
-of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \hat{X} \),
+matrix \( \boldmath{X}=\left[\boldmath{x}_1,\boldmath{x}_2,\dots, \boldmath{x}_p\right] \)
+of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldmath{X} \),
$$
-\hat{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
+\boldmath{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
$$
-If the above condition is not met for a given vector \( \hat{x}_i \) we have
+If the above condition is not met for a given vector \( \boldmath{x}_i \) we have
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
@@ -264,7 +264,7 @@ $$
if our output \( y_i=1 \).
-In this case we say that \( \hat{x}_i \) lies on one of the sides of the hyperplane and if
+In this case we say that \( \boldmath{x}_i \) lies on one of the sides of the hyperplane and if
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
@@ -272,7 +272,7 @@ $$
for the class of observations \( y_i=-1 \),
-then \( \hat{x}_i \) lies on the other side.
+then \( \boldmath{x}_i \) lies on the other side.
Equivalently, for the two classes of observations we have @@ -321,20 +321,20 @@ for our data sample. Let us define the function
$$
-f(x) = \hat{w}^T\hat{x}+b = 0,
+f(x) = \boldmath{w}^T\boldmath{x}+b = 0,
$$
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
-Any point defined by \( \hat{x}_i \) and \( \hat{x}_2 \) on the line \( L \) will satisfy \( \hat{w}^T(\hat{x}_1-\hat{x}_2)=0 \). +Any point defined by \( \boldmath{x}_i \) and \( \boldmath{x}_2 \) on the line \( L \) will satisfy \( \boldmath{w}^T(\boldmath{x}_1-\boldmath{x}_2)=0 \).
-The signed distance \( \delta \) from any point defined by a vector \( \hat{x} \) and a point \( \hat{x}_0 \) on the line \( L \) is then +The signed distance \( \delta \) from any point defined by a vector \( \boldmath{x} \) and a point \( \boldmath{x}_0 \) on the line \( L \) is then
$$
-\delta = \frac{1}{\vert\vert \hat{w}\vert\vert}(\hat{w}^T\hat{x}+b).
+\delta = \frac{1}{\vert\vert \boldmath{w}\vert\vert}(\boldmath{w}^T\boldmath{x}+b).
$$
@@ -344,18 +344,18 @@ $$
-How do we find the parameter \( b \) and the vector \( \hat{w} \)? What we could +How do we find the parameter \( b \) and the vector \( \boldmath{w} \)? What we could do is to define a cost function which now contains the set of all misclassified points \( M \) and attempt to minimize this function
$$
-C(\hat{w},b) = -\sum_{i\in M} y_i(\hat{w}^T\hat{x}_i+b).
+C(\boldmath{w},b) = -\sum_{i\in M} y_i(\boldmath{w}^T\boldmath{x}_i+b).
$$
-We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \hat{w}^T\hat{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us +We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldmath{w}^T\boldmath{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
$$
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
@@ -365,7 +365,7 @@ $$
and
$$
-\frac{\partial C}{\partial \hat{w}} = -\sum_{i\in M} y_ix_i.
+\frac{\partial C}{\partial \boldmath{w}} = -\sum_{i\in M} y_ix_i.
$$
@@ -385,7 +385,7 @@ $$
and
$$
-\hat{w} \leftarrow \hat{w} +\eta \frac{\partial C}{\partial \hat{w}},
+\boldmath{w} \leftarrow \boldmath{w} +\eta \frac{\partial C}{\partial \boldmath{w}},
$$
@@ -411,12 +411,12 @@ A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).
-Thus, we wish to find a margin \( M \) with \( \hat{w} \) normalized to -\( \vert\vert \hat{w}\vert\vert =1 \) subject to the condition +Thus, we wish to find a margin \( M \) with \( \boldmath{w} \) normalized to +\( \vert\vert \boldmath{w}\vert\vert =1 \) subject to the condition
$$
-y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
+y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
@@ -426,27 +426,27 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value \( M \) defined by
$$
-\frac{1}{\vert \vert \hat{w}\vert\vert}y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
+\frac{1}{\vert \vert \boldmath{w}\vert\vert}y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
or just
$$
-y_i(\hat{w}^T\hat{x}_i+b) \geq M\vert \vert \hat{w}\vert\vert \hspace{0.1cm}\forall i.
+y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M\vert \vert \boldmath{w}\vert\vert \hspace{0.1cm}\forall i.
$$
-If we scale the equation so that \( \vert \vert \hat{w}\vert\vert = 1/M \), we have to find the minimum of
-\( \hat{w}^T\hat{w}=\vert \vert \hat{w}\vert\vert \) (the norm) subject to the condition
+If we scale the equation so that \( \vert \vert \boldmath{w}\vert\vert = 1/M \), we have to find the minimum of
+\( \boldmath{w}^T\boldmath{w}=\vert \vert \boldmath{w}\vert\vert \) (the norm) subject to the condition
$$
-y_i(\hat{w}^T\hat{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
+y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
$$
-We have thus defined our margin as the invers of the norm of \( \hat{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers. +We have thus defined our margin as the invers of the norm of \( \boldmath{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers. @@ -572,14 +572,14 @@ $$ In order to solve the above problem, we define the following Lagrangian function to be minimized
$$
-{\cal L}(\lambda,b,\hat{w})=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-1\right],
+{\cal L}(\lambda,b,\boldmath{w})=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-1\right],
$$
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
-Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -589,14 +589,14 @@ $$
and
$$
-\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i.
+\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i.
$$
Inserting these constraints into the equation for \( {\cal L} \) we obtain
$$
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
@@ -604,18 +604,18 @@ subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the Karush-Kuhn-Tucker (KKT) condition
$$
-\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
+\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
-When \( \lambda_i > 0 \), the vectors \( \hat{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \). +When \( \lambda_i > 0 \), the vectors \( \boldmath{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \). @@ -626,24 +626,24 @@ When \( \lambda_i > 0 \), the vectors \( \hat{x}_i \) are called support vectors We can rewrite
$$
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
$$
-\frac{1}{2} \hat{\lambda}^T\begin{bmatrix} y_1y_1\hat{x}_1^T\hat{x}_1 & y_1y_2\hat{x}_1^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_1^T\hat{x}_n \\
-y_2y_1\hat{x}_2^T\hat{x}_1 & y_2y_2\hat{x}_2^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_2^T\hat{x}_n \\
+\frac{1}{2} \boldmath{\lambda}^T\begin{bmatrix} y_1y_1\boldmath{x}_1^T\boldmath{x}_1 & y_1y_2\boldmath{x}_1^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_1^T\boldmath{x}_n \\
+y_2y_1\boldmath{x}_2^T\boldmath{x}_1 & y_2y_2\boldmath{x}_2^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_2^T\boldmath{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
-y_ny_1\hat{x}_n^T\hat{x}_1 & y_ny_2\hat{x}_n^T\hat{x}_2 & \dots & \dots & y_ny_n\hat{x}_n^T\hat{x}_n \\
-\end{bmatrix}\hat{\lambda}-\mathbb{1}\hat{\lambda},
+y_ny_1\boldmath{x}_n^T\boldmath{x}_1 & y_ny_2\boldmath{x}_n^T\boldmath{x}_2 & \dots & \dots & y_ny_n\boldmath{x}_n^T\boldmath{x}_n \\
+\end{bmatrix}\boldmath{\lambda}-\mathbb{1}\boldmath{\lambda},
$$
-subject to \( \hat{y}^T\hat{\lambda}=0 \). Here we defined the vectors \( \hat{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
-\( \hat{y}=[y_1,y_2,\dots,y_n] \).
+subject to \( \boldmath{y}^T\boldmath{\lambda}=0 \). Here we defined the vectors \( \boldmath{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
+\( \boldmath{y}=[y_1,y_2,\dots,y_n] \).
@@ -655,28 +655,35 @@ Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
$$
-\hat{w}=\sum_{i} \lambda_iy_i\hat{x}_i.
+\boldmath{w}=\sum_{i} \lambda_iy_i\boldmath{x}_i.
$$
-With our vector \( \hat{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
+With our vector \( \boldmath{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
-y_i(\hat{w}^T\hat{x}_i+b)=1,
+y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
$$
resulting in
$$
-b = \frac{1}{y_i}-\hat{w}^T\hat{x}_i.
+b = \frac{1}{y_i}-\boldmath{w}^T\boldmath{x}_i,
+$$
+
+
+or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
+
+$$
+b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldmath{x}_i^T\boldmath{x}_j\right).
$$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
-y_i = \mathrm{sign}(\hat{w}^T\hat{x}_i+b).
+y_i = \mathrm{sign}(\boldmath{w}^T\boldmath{x}_i+b).
$$
@@ -697,24 +704,24 @@ so-called kernel approach, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
-We introduce thus the so-called slack variables \( \hat{\xi} =[\xi_1,x_2,\dots,x_n] \) and +We introduce thus the so-called slack variables \( \boldmath{\xi} =[\xi_1,x_2,\dots,x_n] \) and modify our previous equation
$$
-y_i(\hat{w}^T\hat{x}_i+b)=1,
+y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
$$
to
$$
-y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i,
+y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i,
$$
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
-\( y_i(\hat{w}^T\hat{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
+\( y_i(\boldmath{w}^T\boldmath{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
@@ -730,21 +737,21 @@ misclassifications. This has in turn the consequences that we change our optmization problem to finding the minimum of
$$
-{\cal L}=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
+{\cal L}=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
$$
subject to
$$
-y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
+y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
with the requirement \( \xi_i\geq 0 \).
-Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -754,7 +761,7 @@ $$
and
$$
-\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i,
+\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i,
$$
@@ -768,7 +775,7 @@ $$
Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before
$$
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j,
$$
@@ -776,7 +783,7 @@ but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
$$
-\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
+\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
@@ -789,7 +796,7 @@ $$
and
$$
-y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
+y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
diff --git a/doc/pub/svm/html/svm-solarized.html b/doc/pub/svm/html/svm-solarized.html
index c13378dd7..6a5149047 100644
--- a/doc/pub/svm/html/svm-solarized.html
+++ b/doc/pub/svm/html/svm-solarized.html
@@ -164,10 +164,10 @@ $$
where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line
\( b+w_1x_1+w_2x_2=0 \).
-In two dimensions we define the vectors \( \hat{x} =[x1,x2] \) and \( \hat{w}=[w1,w2] \).
+In two dimensions we define the vectors \( \boldmath{x} =[x1,x2] \) and \( \boldmath{w}=[w1,w2] \).
We can then rewrite the above equation as
$$
-\hat{w}^T\hat{x}+b=0.
+\boldmath{w}^T\boldmath{x}+b=0.
$$
@@ -183,25 +183,25 @@ b+wx_1+w_2x_2+\dots +w_px_p=0. $$ If we define a -matrix \( \hat{X}=\left[\hat{x}_1,\hat{x}_2,\dots, \hat{x}_p\right] \) -of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \hat{X} \), +matrix \( \boldmath{X}=\left[\boldmath{x}_1,\boldmath{x}_2,\dots, \boldmath{x}_p\right] \) +of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldmath{X} \), $$ -\hat{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. +\boldmath{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. $$ -If the above condition is not met for a given vector \( \hat{x}_i \) we have +If the above condition is not met for a given vector \( \boldmath{x}_i \) we have $$ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0, $$ if our output \( y_i=1 \). -In this case we say that \( \hat{x}_i \) lies on one of the sides of the hyperplane and if +In this case we say that \( \boldmath{x}_i \) lies on one of the sides of the hyperplane and if $$ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0, $$ for the class of observations \( y_i=-1 \), -then \( \hat{x}_i \) lies on the other side. +then \( \boldmath{x}_i \) lies on the other side.
Equivalently, for the two classes of observations we have @@ -247,18 +247,18 @@ for our data sample.
Let us define the function $$ -f(x) = \hat{w}^T\hat{x}+b = 0, +f(x) = \boldmath{w}^T\boldmath{x}+b = 0, $$ as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
-Any point defined by \( \hat{x}_i \) and \( \hat{x}_2 \) on the line \( L \) will satisfy \( \hat{w}^T(\hat{x}_1-\hat{x}_2)=0 \). +Any point defined by \( \boldmath{x}_i \) and \( \boldmath{x}_2 \) on the line \( L \) will satisfy \( \boldmath{w}^T(\boldmath{x}_1-\boldmath{x}_2)=0 \).
-The signed distance \( \delta \) from any point defined by a vector \( \hat{x} \) and a point \( \hat{x}_0 \) on the line \( L \) is then +The signed distance \( \delta \) from any point defined by a vector \( \boldmath{x} \) and a point \( \boldmath{x}_0 \) on the line \( L \) is then $$ -\delta = \frac{1}{\vert\vert \hat{w}\vert\vert}(\hat{w}^T\hat{x}+b). +\delta = \frac{1}{\vert\vert \boldmath{w}\vert\vert}(\boldmath{w}^T\boldmath{x}+b). $$
@@ -267,23 +267,23 @@ $$
-How do we find the parameter \( b \) and the vector \( \hat{w} \)? What we could +How do we find the parameter \( b \) and the vector \( \boldmath{w} \)? What we could do is to define a cost function which now contains the set of all misclassified points \( M \) and attempt to minimize this function $$ -C(\hat{w},b) = -\sum_{i\in M} y_i(\hat{w}^T\hat{x}_i+b). +C(\boldmath{w},b) = -\sum_{i\in M} y_i(\boldmath{w}^T\boldmath{x}_i+b). $$
-We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \hat{w}^T\hat{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us +We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldmath{w}^T\boldmath{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us $$ \frac{\partial C}{\partial b} = -\sum_{i\in M} y_i, $$ and $$ -\frac{\partial C}{\partial \hat{w}} = -\sum_{i\in M} y_ix_i. +\frac{\partial C}{\partial \boldmath{w}} = -\sum_{i\in M} y_ix_i. $$
@@ -299,7 +299,7 @@ $$ and $$ -\hat{w} \leftarrow \hat{w} +\eta \frac{\partial C}{\partial \hat{w}}, +\boldmath{w} \leftarrow \boldmath{w} +\eta \frac{\partial C}{\partial \boldmath{w}}, $$ where \( \eta \) is our by now well-known learning rate. @@ -324,11 +324,11 @@ A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
-Thus, we wish to find a margin \( M \) with \( \hat{w} \) normalized to -\( \vert\vert \hat{w}\vert\vert =1 \) subject to the condition +Thus, we wish to find a margin \( M \) with \( \boldmath{w} \) normalized to +\( \vert\vert \boldmath{w}\vert\vert =1 \) subject to the condition $$ -y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. $$ All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line. @@ -336,22 +336,22 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value \( M \) defined by $$ -\frac{1}{\vert \vert \hat{w}\vert\vert}y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, +\frac{1}{\vert \vert \boldmath{w}\vert\vert}y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, $$ or just $$ -y_i(\hat{w}^T\hat{x}_i+b) \geq M\vert \vert \hat{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M\vert \vert \boldmath{w}\vert\vert \hspace{0.1cm}\forall i. $$ -If we scale the equation so that \( \vert \vert \hat{w}\vert\vert = 1/M \), we have to find the minimum of -\( \hat{w}^T\hat{w}=\vert \vert \hat{w}\vert\vert \) (the norm) subject to the condition +If we scale the equation so that \( \vert \vert \boldmath{w}\vert\vert = 1/M \), we have to find the minimum of +\( \boldmath{w}^T\boldmath{w}=\vert \vert \boldmath{w}\vert\vert \) (the norm) subject to the condition $$ -y_i(\hat{w}^T\hat{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. $$
-We have thus defined our margin as the invers of the norm of \( \hat{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers. +We have thus defined our margin as the invers of the norm of \( \boldmath{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
@@ -452,40 +452,40 @@ $$
-Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ and $$ -\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i. +\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i. $$ Inserting these constraints into the equation for \( {\cal L} \) we obtain $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j, $$ subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). We must in addition satisfy the Karush-Kuhn-Tucker (KKT) condition $$ -\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -1\right] \hspace{0.1cm}\forall i. +\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -1\right] \hspace{0.1cm}\forall i. $$
@@ -495,21 +495,21 @@ When \( \lambda_i > 0 \), the vectors \( \hat{x}_i \) are called support vectors
We can rewrite $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j, $$ and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem $$ -\frac{1}{2} \hat{\lambda}^T\begin{bmatrix} y_1y_1\hat{x}_1^T\hat{x}_1 & y_1y_2\hat{x}_1^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_1^T\hat{x}_n \\ -y_2y_1\hat{x}_2^T\hat{x}_1 & y_2y_2\hat{x}_2^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_2^T\hat{x}_n \\ +\frac{1}{2} \boldmath{\lambda}^T\begin{bmatrix} y_1y_1\boldmath{x}_1^T\boldmath{x}_1 & y_1y_2\boldmath{x}_1^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_1^T\boldmath{x}_n \\ +y_2y_1\boldmath{x}_2^T\boldmath{x}_1 & y_2y_2\boldmath{x}_2^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_2^T\boldmath{x}_n \\ \dots & \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots & \dots \\ -y_ny_1\hat{x}_n^T\hat{x}_1 & y_ny_2\hat{x}_n^T\hat{x}_2 & \dots & \dots & y_ny_n\hat{x}_n^T\hat{x}_n \\ -\end{bmatrix}\hat{\lambda}-\mathbb{1}\hat{\lambda}, +y_ny_1\boldmath{x}_n^T\boldmath{x}_1 & y_ny_2\boldmath{x}_n^T\boldmath{x}_2 & \dots & \dots & y_ny_n\boldmath{x}_n^T\boldmath{x}_n \\ +\end{bmatrix}\boldmath{\lambda}-\mathbb{1}\boldmath{\lambda}, $$ -subject to \( \hat{y}^T\hat{\lambda}=0 \). Here we defined the vectors \( \hat{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \hat{y}=[y_1,y_2,\dots,y_n] \). +subject to \( \boldmath{y}^T\boldmath{\lambda}=0 \). Here we defined the vectors \( \boldmath{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldmath{y}=[y_1,y_2,\dots,y_n] \).
@@ -520,22 +520,27 @@ subject to \( \hat{y}^T\hat{\lambda}=0 \). Here we defined the vectors \( \hat{\
Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
$$
-\hat{w}=\sum_{i} \lambda_iy_i\hat{x}_i.
+\boldmath{w}=\sum_{i} \lambda_iy_i\boldmath{x}_i.
$$
-With our vector \( \hat{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
+With our vector \( \boldmath{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
-y_i(\hat{w}^T\hat{x}_i+b)=1,
+y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
$$
resulting in
$$
-b = \frac{1}{y_i}-\hat{w}^T\hat{x}_i.
+b = \frac{1}{y_i}-\boldmath{w}^T\boldmath{x}_i,
+$$
+
+or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
+$$
+b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldmath{x}_i^T\boldmath{x}_j\right).
$$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
-y_i = \mathrm{sign}(\hat{w}^T\hat{x}_i+b).
+y_i = \mathrm{sign}(\boldmath{w}^T\boldmath{x}_i+b).
$$
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
@@ -555,20 +560,20 @@ so-called kernel approach, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
-We introduce thus the so-called slack variables \( \hat{\xi} =[\xi_1,x_2,\dots,x_n] \) and +We introduce thus the so-called slack variables \( \boldmath{\xi} =[\xi_1,x_2,\dots,x_n] \) and modify our previous equation $$ -y_i(\hat{w}^T\hat{x}_i+b)=1, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1, $$ to $$ -y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i, $$ with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \). The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction -\( y_i(\hat{w}^T\hat{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), +\( y_i(\boldmath{w}^T\boldmath{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), we bound the total amount by which predictions fall on the wrong side of their margins.
@@ -583,25 +588,25 @@ misclassifications.
This has in turn the consequences that we change our optmization problem to finding the minimum of $$ -{\cal L}=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, +{\cal L}=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, $$ subject to $$ -y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, $$ with the requirement \( \xi_i\geq 0 \).
-Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ and $$ -\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i, +\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i, $$ and @@ -611,13 +616,13 @@ $$ Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j, $$ but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \). We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads $$ -\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, +\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, $$ $$ @@ -626,7 +631,7 @@ $$ and $$ -y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. +y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. $$
diff --git a/doc/pub/svm/html/svm.html b/doc/pub/svm/html/svm.html index 03207f3a0..00fdbf97b 100644 --- a/doc/pub/svm/html/svm.html +++ b/doc/pub/svm/html/svm.html @@ -169,10 +169,10 @@ $$ where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line \( b+w_1x_1+w_2x_2=0 \). -In two dimensions we define the vectors \( \hat{x} =[x1,x2] \) and \( \hat{w}=[w1,w2] \). +In two dimensions we define the vectors \( \boldmath{x} =[x1,x2] \) and \( \boldmath{w}=[w1,w2] \). We can then rewrite the above equation as $$ -\hat{w}^T\hat{x}+b=0. +\boldmath{w}^T\boldmath{x}+b=0. $$
@@ -188,25 +188,25 @@ b+wx_1+w_2x_2+\dots +w_px_p=0. $$ If we define a -matrix \( \hat{X}=\left[\hat{x}_1,\hat{x}_2,\dots, \hat{x}_p\right] \) -of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \hat{X} \), +matrix \( \boldmath{X}=\left[\boldmath{x}_1,\boldmath{x}_2,\dots, \boldmath{x}_p\right] \) +of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldmath{X} \), $$ -\hat{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. +\boldmath{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. $$ -If the above condition is not met for a given vector \( \hat{x}_i \) we have +If the above condition is not met for a given vector \( \boldmath{x}_i \) we have $$ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0, $$ if our output \( y_i=1 \). -In this case we say that \( \hat{x}_i \) lies on one of the sides of the hyperplane and if +In this case we say that \( \boldmath{x}_i \) lies on one of the sides of the hyperplane and if $$ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0, $$ for the class of observations \( y_i=-1 \), -then \( \hat{x}_i \) lies on the other side. +then \( \boldmath{x}_i \) lies on the other side.
Equivalently, for the two classes of observations we have @@ -252,18 +252,18 @@ for our data sample.
Let us define the function $$ -f(x) = \hat{w}^T\hat{x}+b = 0, +f(x) = \boldmath{w}^T\boldmath{x}+b = 0, $$ as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
-Any point defined by \( \hat{x}_i \) and \( \hat{x}_2 \) on the line \( L \) will satisfy \( \hat{w}^T(\hat{x}_1-\hat{x}_2)=0 \). +Any point defined by \( \boldmath{x}_i \) and \( \boldmath{x}_2 \) on the line \( L \) will satisfy \( \boldmath{w}^T(\boldmath{x}_1-\boldmath{x}_2)=0 \).
-The signed distance \( \delta \) from any point defined by a vector \( \hat{x} \) and a point \( \hat{x}_0 \) on the line \( L \) is then +The signed distance \( \delta \) from any point defined by a vector \( \boldmath{x} \) and a point \( \boldmath{x}_0 \) on the line \( L \) is then $$ -\delta = \frac{1}{\vert\vert \hat{w}\vert\vert}(\hat{w}^T\hat{x}+b). +\delta = \frac{1}{\vert\vert \boldmath{w}\vert\vert}(\boldmath{w}^T\boldmath{x}+b). $$
@@ -272,23 +272,23 @@ $$
-How do we find the parameter \( b \) and the vector \( \hat{w} \)? What we could +How do we find the parameter \( b \) and the vector \( \boldmath{w} \)? What we could do is to define a cost function which now contains the set of all misclassified points \( M \) and attempt to minimize this function $$ -C(\hat{w},b) = -\sum_{i\in M} y_i(\hat{w}^T\hat{x}_i+b). +C(\boldmath{w},b) = -\sum_{i\in M} y_i(\boldmath{w}^T\boldmath{x}_i+b). $$
-We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \hat{w}^T\hat{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us +We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldmath{w}^T\boldmath{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us $$ \frac{\partial C}{\partial b} = -\sum_{i\in M} y_i, $$ and $$ -\frac{\partial C}{\partial \hat{w}} = -\sum_{i\in M} y_ix_i. +\frac{\partial C}{\partial \boldmath{w}} = -\sum_{i\in M} y_ix_i. $$
@@ -304,7 +304,7 @@ $$ and $$ -\hat{w} \leftarrow \hat{w} +\eta \frac{\partial C}{\partial \hat{w}}, +\boldmath{w} \leftarrow \boldmath{w} +\eta \frac{\partial C}{\partial \boldmath{w}}, $$ where \( \eta \) is our by now well-known learning rate. @@ -329,11 +329,11 @@ A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
-Thus, we wish to find a margin \( M \) with \( \hat{w} \) normalized to -\( \vert\vert \hat{w}\vert\vert =1 \) subject to the condition +Thus, we wish to find a margin \( M \) with \( \boldmath{w} \) normalized to +\( \vert\vert \boldmath{w}\vert\vert =1 \) subject to the condition $$ -y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. $$ All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line. @@ -341,22 +341,22 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value \( M \) defined by $$ -\frac{1}{\vert \vert \hat{w}\vert\vert}y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, +\frac{1}{\vert \vert \boldmath{w}\vert\vert}y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, $$ or just $$ -y_i(\hat{w}^T\hat{x}_i+b) \geq M\vert \vert \hat{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M\vert \vert \boldmath{w}\vert\vert \hspace{0.1cm}\forall i. $$ -If we scale the equation so that \( \vert \vert \hat{w}\vert\vert = 1/M \), we have to find the minimum of -\( \hat{w}^T\hat{w}=\vert \vert \hat{w}\vert\vert \) (the norm) subject to the condition +If we scale the equation so that \( \vert \vert \boldmath{w}\vert\vert = 1/M \), we have to find the minimum of +\( \boldmath{w}^T\boldmath{w}=\vert \vert \boldmath{w}\vert\vert \) (the norm) subject to the condition $$ -y_i(\hat{w}^T\hat{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. $$
-We have thus defined our margin as the invers of the norm of \( \hat{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers. +We have thus defined our margin as the invers of the norm of \( \boldmath{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
@@ -457,40 +457,40 @@ $$
-Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ and $$ -\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i. +\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i. $$ Inserting these constraints into the equation for \( {\cal L} \) we obtain $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j, $$ subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). We must in addition satisfy the Karush-Kuhn-Tucker (KKT) condition $$ -\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -1\right] \hspace{0.1cm}\forall i. +\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -1\right] \hspace{0.1cm}\forall i. $$
@@ -500,21 +500,21 @@ When \( \lambda_i > 0 \), the vectors \( \hat{x}_i \) are called support vectors
We can rewrite $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j, $$ and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem $$ -\frac{1}{2} \hat{\lambda}^T\begin{bmatrix} y_1y_1\hat{x}_1^T\hat{x}_1 & y_1y_2\hat{x}_1^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_1^T\hat{x}_n \\ -y_2y_1\hat{x}_2^T\hat{x}_1 & y_2y_2\hat{x}_2^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_2^T\hat{x}_n \\ +\frac{1}{2} \boldmath{\lambda}^T\begin{bmatrix} y_1y_1\boldmath{x}_1^T\boldmath{x}_1 & y_1y_2\boldmath{x}_1^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_1^T\boldmath{x}_n \\ +y_2y_1\boldmath{x}_2^T\boldmath{x}_1 & y_2y_2\boldmath{x}_2^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_2^T\boldmath{x}_n \\ \dots & \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots & \dots \\ -y_ny_1\hat{x}_n^T\hat{x}_1 & y_ny_2\hat{x}_n^T\hat{x}_2 & \dots & \dots & y_ny_n\hat{x}_n^T\hat{x}_n \\ -\end{bmatrix}\hat{\lambda}-\mathbb{1}\hat{\lambda}, +y_ny_1\boldmath{x}_n^T\boldmath{x}_1 & y_ny_2\boldmath{x}_n^T\boldmath{x}_2 & \dots & \dots & y_ny_n\boldmath{x}_n^T\boldmath{x}_n \\ +\end{bmatrix}\boldmath{\lambda}-\mathbb{1}\boldmath{\lambda}, $$ -subject to \( \hat{y}^T\hat{\lambda}=0 \). Here we defined the vectors \( \hat{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \hat{y}=[y_1,y_2,\dots,y_n] \). +subject to \( \boldmath{y}^T\boldmath{\lambda}=0 \). Here we defined the vectors \( \boldmath{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldmath{y}=[y_1,y_2,\dots,y_n] \).
@@ -525,22 +525,27 @@ subject to \( \hat{y}^T\hat{\lambda}=0 \). Here we defined the vectors \( \hat{\
Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
$$
-\hat{w}=\sum_{i} \lambda_iy_i\hat{x}_i.
+\boldmath{w}=\sum_{i} \lambda_iy_i\boldmath{x}_i.
$$
-With our vector \( \hat{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
+With our vector \( \boldmath{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
-y_i(\hat{w}^T\hat{x}_i+b)=1,
+y_i(\boldmath{w}^T\boldmath{x}_i+b)=1,
$$
resulting in
$$
-b = \frac{1}{y_i}-\hat{w}^T\hat{x}_i.
+b = \frac{1}{y_i}-\boldmath{w}^T\boldmath{x}_i,
+$$
+
+or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
+$$
+b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldmath{x}_i^T\boldmath{x}_j\right).
$$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
-y_i = \mathrm{sign}(\hat{w}^T\hat{x}_i+b).
+y_i = \mathrm{sign}(\boldmath{w}^T\boldmath{x}_i+b).
$$
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
@@ -560,20 +565,20 @@ so-called kernel approach, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
-We introduce thus the so-called slack variables \( \hat{\xi} =[\xi_1,x_2,\dots,x_n] \) and +We introduce thus the so-called slack variables \( \boldmath{\xi} =[\xi_1,x_2,\dots,x_n] \) and modify our previous equation $$ -y_i(\hat{w}^T\hat{x}_i+b)=1, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1, $$ to $$ -y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i, $$ with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \). The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction -\( y_i(\hat{w}^T\hat{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), +\( y_i(\boldmath{w}^T\boldmath{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), we bound the total amount by which predictions fall on the wrong side of their margins.
@@ -588,25 +593,25 @@ misclassifications.
This has in turn the consequences that we change our optmization problem to finding the minimum of $$ -{\cal L}=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, +{\cal L}=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, $$ subject to $$ -y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, $$ with the requirement \( \xi_i\geq 0 \).
-Taking the derivatives with respect to \( b \) and \( \hat{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldmath{w} \) we obtain $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ and $$ -\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i, +\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i, $$ and @@ -616,13 +621,13 @@ $$ Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j, $$ but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \). We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads $$ -\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, +\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, $$ $$ @@ -631,7 +636,7 @@ $$ and $$ -y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. +y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. $$
diff --git a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz index f15bd65e4..43de2c58f 100644 Binary files a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz and b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz differ diff --git a/doc/pub/svm/ipynb/svm.ipynb b/doc/pub/svm/ipynb/svm.ipynb index 00a5696d2..1c9318033 100644 --- a/doc/pub/svm/ipynb/svm.ipynb +++ b/doc/pub/svm/ipynb/svm.ipynb @@ -79,7 +79,7 @@ "source": [ "where $b$ is the intercept and $w_1$ and $w_2$ define the elements of a vector orthogonal to the line \n", "$b+w_1x_1+w_2x_2=0$. \n", - "In two dimensions we define the vectors $\\hat{x} =[x1,x2]$ and $\\hat{w}=[w1,w2]$. \n", + "In two dimensions we define the vectors $\\boldmath{x} =[x1,x2]$ and $\\boldmath{w}=[w1,w2]$. \n", "We can then rewrite the above equation as" ] }, @@ -88,7 +88,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{w}^T\\hat{x}+b=0.\n", + "\\boldmath{w}^T\\boldmath{x}+b=0.\n", "$$" ] }, @@ -116,8 +116,8 @@ "metadata": {}, "source": [ "If we define a \n", - "matrix $\\hat{X}=\\left[\\hat{x}_1,\\hat{x}_2,\\dots, \\hat{x}_p\\right]$\n", - "of dimension $n\\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\\hat{X}$," + "matrix $\\boldmath{X}=\\left[\\boldmath{x}_1,\\boldmath{x}_2,\\dots, \\boldmath{x}_p\\right]$\n", + "of dimension $n\\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\\boldmath{X}$," ] }, { @@ -125,7 +125,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{x}_i = \\begin{bmatrix} x_{i1} \\\\ x_{i2} \\\\ \\dots \\\\ \\dots \\\\ x_{ip} \\end{bmatrix}.\n", + "\\boldmath{x}_i = \\begin{bmatrix} x_{i1} \\\\ x_{i2} \\\\ \\dots \\\\ \\dots \\\\ x_{ip} \\end{bmatrix}.\n", "$$" ] }, @@ -133,7 +133,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If the above condition is not met for a given vector $\\hat{x}_i$ we have" + "If the above condition is not met for a given vector $\\boldmath{x}_i$ we have" ] }, { @@ -150,7 +150,7 @@ "metadata": {}, "source": [ "if our output $y_i=1$.\n", - "In this case we say that $\\hat{x}_i$ lies on one of the sides of the hyperplane and if" + "In this case we say that $\\boldmath{x}_i$ lies on one of the sides of the hyperplane and if" ] }, { @@ -167,7 +167,7 @@ "metadata": {}, "source": [ "for the class of observations $y_i=-1$, \n", - "then $\\hat{x}_i$ lies on the other side. \n", + "then $\\boldmath{x}_i$ lies on the other side. \n", "\n", "Equivalently, for the two classes of observations we have" ] @@ -219,7 +219,7 @@ "metadata": {}, "source": [ "$$\n", - "f(x) = \\hat{w}^T\\hat{x}+b = 0,\n", + "f(x) = \\boldmath{w}^T\\boldmath{x}+b = 0,\n", "$$" ] }, @@ -230,9 +230,9 @@ "as the function that determines the line $L$ that separates two classes (our two features), see the figure here. \n", "\n", "\n", - "Any point defined by $\\hat{x}_i$ and $\\hat{x}_2$ on the line $L$ will satisfy $\\hat{w}^T(\\hat{x}_1-\\hat{x}_2)=0$. \n", + "Any point defined by $\\boldmath{x}_i$ and $\\boldmath{x}_2$ on the line $L$ will satisfy $\\boldmath{w}^T(\\boldmath{x}_1-\\boldmath{x}_2)=0$. \n", "\n", - "The signed distance $\\delta$ from any point defined by a vector $\\hat{x}$ and a point $\\hat{x}_0$ on the line $L$ is then" + "The signed distance $\\delta$ from any point defined by a vector $\\boldmath{x}$ and a point $\\boldmath{x}_0$ on the line $L$ is then" ] }, { @@ -240,7 +240,7 @@ "metadata": {}, "source": [ "$$\n", - "\\delta = \\frac{1}{\\vert\\vert \\hat{w}\\vert\\vert}(\\hat{w}^T\\hat{x}+b).\n", + "\\delta = \\frac{1}{\\vert\\vert \\boldmath{w}\\vert\\vert}(\\boldmath{w}^T\\boldmath{x}+b).\n", "$$" ] }, @@ -250,7 +250,7 @@ "source": [ "## First attempt at a minimization approach\n", "\n", - "How do we find the parameter $b$ and the vector $\\hat{w}$? What we could\n", + "How do we find the parameter $b$ and the vector $\\boldmath{w}$? What we could\n", "do is to define a cost function which now contains the set of all\n", "misclassified points $M$ and attempt to minimize this function" ] @@ -260,7 +260,7 @@ "metadata": {}, "source": [ "$$\n", - "C(\\hat{w},b) = -\\sum_{i\\in M} y_i(\\hat{w}^T\\hat{x}_i+b).\n", + "C(\\boldmath{w},b) = -\\sum_{i\\in M} y_i(\\boldmath{w}^T\\boldmath{x}_i+b).\n", "$$" ] }, @@ -268,7 +268,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We could now for example define all values $y_i =1$ as misclassified in case we have $\\hat{w}^T\\hat{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us" + "We could now for example define all values $y_i =1$ as misclassified in case we have $\\boldmath{w}^T\\boldmath{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us" ] }, { @@ -292,7 +292,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial C}{\\partial \\hat{w}} = -\\sum_{i\\in M} y_ix_i.\n", + "\\frac{\\partial C}{\\partial \\boldmath{w}} = -\\sum_{i\\in M} y_ix_i.\n", "$$" ] }, @@ -326,7 +326,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{w} \\leftarrow \\hat{w} +\\eta \\frac{\\partial C}{\\partial \\hat{w}},\n", + "\\boldmath{w} \\leftarrow \\boldmath{w} +\\eta \\frac{\\partial C}{\\partial \\boldmath{w}},\n", "$$" ] }, @@ -350,8 +350,8 @@ "A better approach is rather to try to define a large margin between\n", "the two classes (if they are well separated from the beginning).\n", "\n", - "Thus, we wish to find a margin $M$ with $\\hat{w}$ normalized to\n", - "$\\vert\\vert \\hat{w}\\vert\\vert =1$ subject to the condition" + "Thus, we wish to find a margin $M$ with $\\boldmath{w}$ normalized to\n", + "$\\vert\\vert \\boldmath{w}\\vert\\vert =1$ subject to the condition" ] }, { @@ -359,7 +359,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\hat{w}^T\\hat{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, p.\n", + "y_i(\\boldmath{w}^T\\boldmath{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, p.\n", "$$" ] }, @@ -377,7 +377,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{1}{\\vert \\vert \\hat{w}\\vert\\vert}y_i(\\hat{w}^T\\hat{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, n,\n", + "\\frac{1}{\\vert \\vert \\boldmath{w}\\vert\\vert}y_i(\\boldmath{w}^T\\boldmath{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, n,\n", "$$" ] }, @@ -393,7 +393,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\hat{w}^T\\hat{x}_i+b) \\geq M\\vert \\vert \\hat{w}\\vert\\vert \\hspace{0.1cm}\\forall i.\n", + "y_i(\\boldmath{w}^T\\boldmath{x}_i+b) \\geq M\\vert \\vert \\boldmath{w}\\vert\\vert \\hspace{0.1cm}\\forall i.\n", "$$" ] }, @@ -401,8 +401,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If we scale the equation so that $\\vert \\vert \\hat{w}\\vert\\vert = 1/M$, we have to find the minimum of \n", - "$\\hat{w}^T\\hat{w}=\\vert \\vert \\hat{w}\\vert\\vert$ (the norm) subject to the condition" + "If we scale the equation so that $\\vert \\vert \\boldmath{w}\\vert\\vert = 1/M$, we have to find the minimum of \n", + "$\\boldmath{w}^T\\boldmath{w}=\\vert \\vert \\boldmath{w}\\vert\\vert$ (the norm) subject to the condition" ] }, { @@ -410,7 +410,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\hat{w}^T\\hat{x}_i+b) \\geq 1 \\hspace{0.1cm}\\forall i.\n", + "y_i(\\boldmath{w}^T\\boldmath{x}_i+b) \\geq 1 \\hspace{0.1cm}\\forall i.\n", "$$" ] }, @@ -418,7 +418,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We have thus defined our margin as the invers of the norm of $\\hat{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers. \n", + "We have thus defined our margin as the invers of the norm of $\\boldmath{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers. \n", "\n", "## A quick reminder on Lagrangian multipliers\n", "\n", @@ -644,7 +644,7 @@ "metadata": {}, "source": [ "$$\n", - "{\\cal L}(\\lambda,b,\\hat{w})=\\frac{1}{2}\\hat{w}^T\\hat{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\hat{w}^T\\hat{x}_i+b)-1\\right],\n", + "{\\cal L}(\\lambda,b,\\boldmath{w})=\\frac{1}{2}\\boldmath{w}^T\\boldmath{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\boldmath{w}^T\\boldmath{x}_i+b)-1\\right],\n", "$$" ] }, @@ -654,7 +654,7 @@ "source": [ "where $\\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\\lambda_i \\geq 0$.\n", "\n", - "Taking the derivatives with respect to $b$ and $\\hat{w}$ we obtain" + "Taking the derivatives with respect to $b$ and $\\boldmath{w}$ we obtain" ] }, { @@ -678,7 +678,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial {\\cal L}}{\\partial \\hat{w}} = 0 = \\hat{w}-\\sum_{i} \\lambda_iy_i\\hat{x}_i.\n", + "\\frac{\\partial {\\cal L}}{\\partial \\boldmath{w}} = 0 = \\boldmath{w}-\\sum_{i} \\lambda_iy_i\\boldmath{x}_i.\n", "$$" ] }, @@ -694,7 +694,7 @@ "metadata": {}, "source": [ "$$\n", - "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\hat{x}_i^T\\hat{x}_j,\n", + "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldmath{x}_i^T\\boldmath{x}_j,\n", "$$" ] }, @@ -711,7 +711,7 @@ "metadata": {}, "source": [ "$$\n", - "\\lambda_i\\left[y_i(\\hat{w}^T\\hat{x}_i+b) -1\\right] \\hspace{0.1cm}\\forall i.\n", + "\\lambda_i\\left[y_i(\\boldmath{w}^T\\boldmath{x}_i+b) -1\\right] \\hspace{0.1cm}\\forall i.\n", "$$" ] }, @@ -719,11 +719,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "1. If $\\lambda_i > 0$, then $y_i(\\hat{w}^T\\hat{x}_i+b)=1$ and we say that $x_i$ is on the boundary.\n", + "1. If $\\lambda_i > 0$, then $y_i(\\boldmath{w}^T\\boldmath{x}_i+b)=1$ and we say that $x_i$ is on the boundary.\n", "\n", - "2. If $y_i(\\hat{w}^T\\hat{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\\lambda_i=0$. \n", + "2. If $y_i(\\boldmath{w}^T\\boldmath{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\\lambda_i=0$. \n", "\n", - "When $\\lambda_i > 0$, the vectors $\\hat{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. \n", + "When $\\lambda_i > 0$, the vectors $\\boldmath{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. \n", "\n", "## The problem to solve\n", "\n", @@ -735,7 +735,7 @@ "metadata": {}, "source": [ "$$\n", - "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\hat{x}_i^T\\hat{x}_j,\n", + "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldmath{x}_i^T\\boldmath{x}_j,\n", "$$" ] }, @@ -751,12 +751,12 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{1}{2} \\hat{\\lambda}^T\\begin{bmatrix} y_1y_1\\hat{x}_1^T\\hat{x}_1 & y_1y_2\\hat{x}_1^T\\hat{x}_2 & \\dots & \\dots & y_1y_n\\hat{x}_1^T\\hat{x}_n \\\\\n", - "y_2y_1\\hat{x}_2^T\\hat{x}_1 & y_2y_2\\hat{x}_2^T\\hat{x}_2 & \\dots & \\dots & y_1y_n\\hat{x}_2^T\\hat{x}_n \\\\\n", + "\\frac{1}{2} \\boldmath{\\lambda}^T\\begin{bmatrix} y_1y_1\\boldmath{x}_1^T\\boldmath{x}_1 & y_1y_2\\boldmath{x}_1^T\\boldmath{x}_2 & \\dots & \\dots & y_1y_n\\boldmath{x}_1^T\\boldmath{x}_n \\\\\n", + "y_2y_1\\boldmath{x}_2^T\\boldmath{x}_1 & y_2y_2\\boldmath{x}_2^T\\boldmath{x}_2 & \\dots & \\dots & y_1y_n\\boldmath{x}_2^T\\boldmath{x}_n \\\\\n", "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", - "y_ny_1\\hat{x}_n^T\\hat{x}_1 & y_ny_2\\hat{x}_n^T\\hat{x}_2 & \\dots & \\dots & y_ny_n\\hat{x}_n^T\\hat{x}_n \\\\\n", - "\\end{bmatrix}\\hat{\\lambda}-\\mathbb{1}\\hat{\\lambda},\n", + "y_ny_1\\boldmath{x}_n^T\\boldmath{x}_1 & y_ny_2\\boldmath{x}_n^T\\boldmath{x}_2 & \\dots & \\dots & y_ny_n\\boldmath{x}_n^T\\boldmath{x}_n \\\\\n", + "\\end{bmatrix}\\boldmath{\\lambda}-\\mathbb{1}\\boldmath{\\lambda},\n", "$$" ] }, @@ -764,8 +764,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "subject to $\\hat{y}^T\\hat{\\lambda}=0$. Here we defined the vectors $\\hat{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n", - "$\\hat{y}=[y_1,y_2,\\dots,y_n]$. \n", + "subject to $\\boldmath{y}^T\\boldmath{\\lambda}=0$. Here we defined the vectors $\\boldmath{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n", + "$\\boldmath{y}=[y_1,y_2,\\dots,y_n]$. \n", "\n", "\n", "## The last steps\n", @@ -779,7 +779,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{w}=\\sum_{i} \\lambda_iy_i\\hat{x}_i.\n", + "\\boldmath{w}=\\sum_{i} \\lambda_iy_i\\boldmath{x}_i.\n", "$$" ] }, @@ -787,7 +787,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "With our vector $\\hat{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via" + "With our vector $\\boldmath{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via" ] }, { @@ -795,7 +795,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\hat{w}^T\\hat{x}_i+b)=1,\n", + "y_i(\\boldmath{w}^T\\boldmath{x}_i+b)=1,\n", "$$" ] }, @@ -811,7 +811,23 @@ "metadata": {}, "source": [ "$$\n", - "b = \\frac{1}{y_i}-\\hat{w}^T\\hat{x}_i.\n", + "b = \\frac{1}{y_i}-\\boldmath{w}^T\\boldmath{x}_i,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "or if we write it out in terms of the support vectors only, with $N_s$ being their number, we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "b = \\frac{1}{N_s}\\sum_{j\\in N_s}\\left(y_j-\\sum_{i=1}^n\\lambda_iy_i\\boldmath{x}_i^T\\boldmath{x}_j\\right).\n", "$$" ] }, @@ -827,7 +843,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i = \\mathrm{sign}(\\hat{w}^T\\hat{x}_i+b).\n", + "y_i = \\mathrm{sign}(\\boldmath{w}^T\\boldmath{x}_i+b).\n", "$$" ] }, @@ -846,7 +862,7 @@ "so-called **kernel approach**, is to allow a kind of slack in the sense\n", "that we allow some points to be on the wrong side of the margin.\n", "\n", - "We introduce thus the so-called **slack** variables $\\hat{\\xi} =[\\xi_1,x_2,\\dots,x_n]$ and \n", + "We introduce thus the so-called **slack** variables $\\boldmath{\\xi} =[\\xi_1,x_2,\\dots,x_n]$ and \n", "modify our previous equation" ] }, @@ -855,7 +871,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\hat{w}^T\\hat{x}_i+b)=1,\n", + "y_i(\\boldmath{w}^T\\boldmath{x}_i+b)=1,\n", "$$" ] }, @@ -871,7 +887,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\hat{w}^T\\hat{x}_i+b)=1-\\xi_i,\n", + "y_i(\\boldmath{w}^T\\boldmath{x}_i+b)=1-\\xi_i,\n", "$$" ] }, @@ -881,7 +897,7 @@ "source": [ "with the requirement $\\xi_i\\geq 0$. The total violation is now $\\sum_i\\xi$. \n", "The value $\\xi_i$ in the constraint the last constraint corresponds to the amount by which the prediction\n", - "$y_i(\\hat{w}^T\\hat{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\\sum_i \\xi_i$,\n", + "$y_i(\\boldmath{w}^T\\boldmath{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\\sum_i \\xi_i$,\n", "we bound the total amount by which predictions fall on the wrong side of their margins.\n", "\n", "Misclassifications occur when $\\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of\n", @@ -898,7 +914,7 @@ "metadata": {}, "source": [ "$$\n", - "{\\cal L}=\\frac{1}{2}\\hat{w}^T\\hat{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\hat{w}^T\\hat{x}_i+b)-(1-\\xi_)\\right]+C\\sum_{i=1}^n\\xi_i-\\sum_{i=1}^n\\gamma_i\\xi_i,\n", + "{\\cal L}=\\frac{1}{2}\\boldmath{w}^T\\boldmath{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\boldmath{w}^T\\boldmath{x}_i+b)-(1-\\xi_)\\right]+C\\sum_{i=1}^n\\xi_i-\\sum_{i=1}^n\\gamma_i\\xi_i,\n", "$$" ] }, @@ -914,7 +930,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\hat{w}^T\\hat{x}_i+b)=1-\\xi_i \\hspace{0.1cm}\\forall i,\n", + "y_i(\\boldmath{w}^T\\boldmath{x}_i+b)=1-\\xi_i \\hspace{0.1cm}\\forall i,\n", "$$" ] }, @@ -924,7 +940,7 @@ "source": [ "with the requirement $\\xi_i\\geq 0$.\n", "\n", - "Taking the derivatives with respect to $b$ and $\\hat{w}$ we obtain" + "Taking the derivatives with respect to $b$ and $\\boldmath{w}$ we obtain" ] }, { @@ -948,7 +964,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial {\\cal L}}{\\partial \\hat{w}} = 0 = \\hat{w}-\\sum_{i} \\lambda_iy_i\\hat{x}_i,\n", + "\\frac{\\partial {\\cal L}}{\\partial \\boldmath{w}} = 0 = \\boldmath{w}-\\sum_{i} \\lambda_iy_i\\boldmath{x}_i,\n", "$$" ] }, @@ -980,7 +996,7 @@ "metadata": {}, "source": [ "$$\n", - "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\hat{x}_i^T\\hat{x}_j,\n", + "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldmath{x}_i^T\\boldmath{x}_j,\n", "$$" ] }, @@ -996,8 +1012,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "4\n", - "9\n", + "5\n", + "0\n", " \n", "<\n", "<\n", @@ -1037,7 +1053,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\hat{w}^T\\hat{x}_i+b) -(1-\\xi_) \\geq 0 \\hspace{0.1cm}\\forall i.\n", + "y_i(\\boldmath{w}^T\\boldmath{x}_i+b) -(1-\\xi_) \\geq 0 \\hspace{0.1cm}\\forall i.\n", "$$" ] }, diff --git a/doc/pub/svm/pdf/svm-minted.pdf b/doc/pub/svm/pdf/svm-minted.pdf index 56320d1ad..00da3c3e4 100644 Binary files a/doc/pub/svm/pdf/svm-minted.pdf and b/doc/pub/svm/pdf/svm-minted.pdf differ diff --git a/doc/src/SupportVMachines/svm.do.txt b/doc/src/SupportVMachines/svm.do.txt index 4bccfb7d5..a7ec380d2 100644 --- a/doc/src/SupportVMachines/svm.do.txt +++ b/doc/src/SupportVMachines/svm.do.txt @@ -58,11 +58,11 @@ b+w_1x_1+w_2x_2=0, !et where $b$ is the intercept and $w_1$ and $w_2$ define the elements of a vector orthogonal to the line $b+w_1x_1+w_2x_2=0$. -In two dimensions we define the vectors $\hat{x} =[x1,x2]$ and $\hat{w}=[w1,w2]$. +In two dimensions we define the vectors $\boldmath{x} =[x1,x2]$ and $\boldmath{w}=[w1,w2]$. We can then rewrite the above equation as !bt \[ -\hat{w}^T\hat{x}+b=0. +\boldmath{w}^T\boldmath{x}+b=0. \] !et @@ -77,28 +77,28 @@ b+wx_1+w_2x_2+\dots +w_px_p=0. \] !et If we define a -matrix $\hat{X}=\left[\hat{x}_1,\hat{x}_2,\dots, \hat{x}_p\right]$ -of dimension $n\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\hat{X}$, +matrix $\boldmath{X}=\left[\boldmath{x}_1,\boldmath{x}_2,\dots, \boldmath{x}_p\right]$ +of dimension $n\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\boldmath{X}$, !bt \[ -\hat{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. +\boldmath{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. \] !et -If the above condition is not met for a given vector $\hat{x}_i$ we have +If the above condition is not met for a given vector $\boldmath{x}_i$ we have !bt \[ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0, \] !et if our output $y_i=1$. -In this case we say that $\hat{x}_i$ lies on one of the sides of the hyperplane and if +In this case we say that $\boldmath{x}_i$ lies on one of the sides of the hyperplane and if !bt \[ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0, \] !et for the class of observations $y_i=-1$, -then $\hat{x}_i$ lies on the other side. +then $\boldmath{x}_i$ lies on the other side. Equivalently, for the two classes of observations we have !bt @@ -137,35 +137,35 @@ for our data sample. Let us define the function !bt \[ -f(x) = \hat{w}^T\hat{x}+b = 0, +f(x) = \boldmath{w}^T\boldmath{x}+b = 0, \] !et as the function that determines the line $L$ that separates two classes (our two features), see the figure here. -Any point defined by $\hat{x}_i$ and $\hat{x}_2$ on the line $L$ will satisfy $\hat{w}^T(\hat{x}_1-\hat{x}_2)=0$. +Any point defined by $\boldmath{x}_i$ and $\boldmath{x}_2$ on the line $L$ will satisfy $\boldmath{w}^T(\boldmath{x}_1-\boldmath{x}_2)=0$. -The signed distance $\delta$ from any point defined by a vector $\hat{x}$ and a point $\hat{x}_0$ on the line $L$ is then +The signed distance $\delta$ from any point defined by a vector $\boldmath{x}$ and a point $\boldmath{x}_0$ on the line $L$ is then !bt \[ -\delta = \frac{1}{\vert\vert \hat{w}\vert\vert}(\hat{w}^T\hat{x}+b). +\delta = \frac{1}{\vert\vert \boldmath{w}\vert\vert}(\boldmath{w}^T\boldmath{x}+b). \] !et !split ===== First attempt at a minimization approach ===== -How do we find the parameter $b$ and the vector $\hat{w}$? What we could +How do we find the parameter $b$ and the vector $\boldmath{w}$? What we could do is to define a cost function which now contains the set of all misclassified points $M$ and attempt to minimize this function !bt \[ -C(\hat{w},b) = -\sum_{i\in M} y_i(\hat{w}^T\hat{x}_i+b). +C(\boldmath{w},b) = -\sum_{i\in M} y_i(\boldmath{w}^T\boldmath{x}_i+b). \] !et -We could now for example define all values $y_i =1$ as misclassified in case we have $\hat{w}^T\hat{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us +We could now for example define all values $y_i =1$ as misclassified in case we have $\boldmath{w}^T\boldmath{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us !bt \[ \frac{\partial C}{\partial b} = -\sum_{i\in M} y_i, @@ -174,7 +174,7 @@ We could now for example define all values $y_i =1$ as misclassified in case we and !bt \[ -\frac{\partial C}{\partial \hat{w}} = -\sum_{i\in M} y_ix_i. +\frac{\partial C}{\partial \boldmath{w}} = -\sum_{i\in M} y_ix_i. \] !et @@ -190,7 +190,7 @@ b \leftarrow b +\eta \frac{\partial C}{\partial b}, and !bt \[ -\hat{w} \leftarrow \hat{w} +\eta \frac{\partial C}{\partial \hat{w}}, +\boldmath{w} \leftarrow \boldmath{w} +\eta \frac{\partial C}{\partial \boldmath{w}}, \] !et where $\eta$ is our by now well-known learning rate. @@ -210,12 +210,12 @@ at all. A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning). -Thus, we wish to find a margin $M$ with $\hat{w}$ normalized to -$\vert\vert \hat{w}\vert\vert =1$ subject to the condition +Thus, we wish to find a margin $M$ with $\boldmath{w}$ normalized to +$\vert\vert \boldmath{w}\vert\vert =1$ subject to the condition !bt \[ -y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. \] !et All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $b$ and $w_1$ and $w_2$ define this line. @@ -223,24 +223,24 @@ All points are thus at a signed distance from the decision boundary defined by t We seek thus the largest value $M$ defined by !bt \[ -\frac{1}{\vert \vert \hat{w}\vert\vert}y_i(\hat{w}^T\hat{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, +\frac{1}{\vert \vert \boldmath{w}\vert\vert}y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, \] !et or just !bt \[ -y_i(\hat{w}^T\hat{x}_i+b) \geq M\vert \vert \hat{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq M\vert \vert \boldmath{w}\vert\vert \hspace{0.1cm}\forall i. \] !et -If we scale the equation so that $\vert \vert \hat{w}\vert\vert = 1/M$, we have to find the minimum of -$\hat{w}^T\hat{w}=\vert \vert \hat{w}\vert\vert$ (the norm) subject to the condition +If we scale the equation so that $\vert \vert \boldmath{w}\vert\vert = 1/M$, we have to find the minimum of +$\boldmath{w}^T\boldmath{w}=\vert \vert \boldmath{w}\vert\vert$ (the norm) subject to the condition !bt \[ -y_i(\hat{w}^T\hat{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +y_i(\boldmath{w}^T\boldmath{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. \] !et -We have thus defined our margin as the invers of the norm of $\hat{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers. +We have thus defined our margin as the invers of the norm of $\boldmath{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers. !split ===== A quick reminder on Lagrangian multipliers ===== @@ -346,12 +346,12 @@ If we have a set of constraints $\phi_k$ we have the equations In order to solve the above problem, we define the following Lagrangian function to be minimized !bt \[ -{\cal L}(\lambda,b,\hat{w})=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-1\right], +{\cal L}(\lambda,b,\boldmath{w})=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-1\right], \] !et where $\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\lambda_i \geq 0$. -Taking the derivatives with respect to $b$ and $\hat{w}$ we obtain +Taking the derivatives with respect to $b$ and $\boldmath{w}$ we obtain !bt \[ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, @@ -360,25 +360,25 @@ Taking the derivatives with respect to $b$ and $\hat{w}$ we obtain and !bt \[ -\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i. +\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i. \] !et Inserting these constraints into the equation for ${\cal L}$ we obtain !bt \[ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j, \] !et subject to the constraints $\lambda_i\geq 0$ and $\sum_i\lambda_iy_i=0$. We must in addition satisfy the "Karush-Kuhn-Tucker":"https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" (KKT) condition !bt \[ -\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -1\right] \hspace{0.1cm}\forall i. +\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -1\right] \hspace{0.1cm}\forall i. \] !et -o If $\lambda_i > 0$, then $y_i(\hat{w}^T\hat{x}_i+b)=1$ and we say that $x_i$ is on the boundary. -o If $y_i(\hat{w}^T\hat{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$. -When $\lambda_i > 0$, the vectors $\hat{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. +o If $\lambda_i > 0$, then $y_i(\boldmath{w}^T\boldmath{x}_i+b)=1$ and we say that $x_i$ is on the boundary. +o If $y_i(\boldmath{w}^T\boldmath{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$. +When $\lambda_i > 0$, the vectors $\boldmath{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. !split ===== The problem to solve ===== @@ -386,22 +386,22 @@ When $\lambda_i > 0$, the vectors $\hat{x}_i$ are called support vectors. They a We can rewrite !bt \[ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j, \] !et and its constraints in terms of a matrix-vector problem where we minimize w.r.t. $\lambda$ the following problem !bt \[ -\frac{1}{2} \hat{\lambda}^T\begin{bmatrix} y_1y_1\hat{x}_1^T\hat{x}_1 & y_1y_2\hat{x}_1^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_1^T\hat{x}_n \\ -y_2y_1\hat{x}_2^T\hat{x}_1 & y_2y_2\hat{x}_2^T\hat{x}_2 & \dots & \dots & y_1y_n\hat{x}_2^T\hat{x}_n \\ +\frac{1}{2} \boldmath{\lambda}^T\begin{bmatrix} y_1y_1\boldmath{x}_1^T\boldmath{x}_1 & y_1y_2\boldmath{x}_1^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_1^T\boldmath{x}_n \\ +y_2y_1\boldmath{x}_2^T\boldmath{x}_1 & y_2y_2\boldmath{x}_2^T\boldmath{x}_2 & \dots & \dots & y_1y_n\boldmath{x}_2^T\boldmath{x}_n \\ \dots & \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots & \dots \\ -y_ny_1\hat{x}_n^T\hat{x}_1 & y_ny_2\hat{x}_n^T\hat{x}_2 & \dots & \dots & y_ny_n\hat{x}_n^T\hat{x}_n \\ -\end{bmatrix}\hat{\lambda}-\mathbb{1}\hat{\lambda}, +y_ny_1\boldmath{x}_n^T\boldmath{x}_1 & y_ny_2\boldmath{x}_n^T\boldmath{x}_2 & \dots & \dots & y_ny_n\boldmath{x}_n^T\boldmath{x}_n \\ +\end{bmatrix}\boldmath{\lambda}-\mathbb{1}\boldmath{\lambda}, \] !et -subject to $\hat{y}^T\hat{\lambda}=0$. Here we defined the vectors $\hat{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and -$\hat{y}=[y_1,y_2,\dots,y_n]$. +subject to $\boldmath{y}^T\boldmath{\lambda}=0$. Here we defined the vectors $\boldmath{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and +$\boldmath{y}=[y_1,y_2,\dots,y_n]$. !split @@ -411,25 +411,31 @@ Solving the above problem, yields the values of $\lambda_i$. To find the coefficients of your hyperplane we need simply to compute !bt \[ -\hat{w}=\sum_{i} \lambda_iy_i\hat{x}_i. +\boldmath{w}=\sum_{i} \lambda_iy_i\boldmath{x}_i. \] !et -With our vector $\hat{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via +With our vector $\boldmath{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via !bt \[ -y_i(\hat{w}^T\hat{x}_i+b)=1, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1, \] !et resulting in !bt \[ -b = \frac{1}{y_i}-\hat{w}^T\hat{x}_i. +b = \frac{1}{y_i}-\boldmath{w}^T\boldmath{x}_i, +\] +!et +or if we write it out in terms of the support vectors only, with $N_s$ being their number, we have +!bt +\[ +b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldmath{x}_i^T\boldmath{x}_j\right). \] !et With our hyperplane coefficients we can use our classifier to assign any observation by simply using !bt \[ -y_i = \mathrm{sign}(\hat{w}^T\hat{x}_i+b). +y_i = \mathrm{sign}(\boldmath{w}^T\boldmath{x}_i+b). \] !et Below we discuss how to find the optimal values of $\lambda_i$. Before we proceed however, we discuss now the so-called soft classifier. @@ -444,22 +450,22 @@ figure here. One way to deal with this problem before we define the so-called _kernel approach_, is to allow a kind of slack in the sense that we allow some points to be on the wrong side of the margin. -We introduce thus the so-called _slack_ variables $\hat{\xi} =[\xi_1,x_2,\dots,x_n]$ and +We introduce thus the so-called _slack_ variables $\boldmath{\xi} =[\xi_1,x_2,\dots,x_n]$ and modify our previous equation !bt \[ -y_i(\hat{w}^T\hat{x}_i+b)=1, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1, \] !et to !bt \[ -y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i, \] !et with the requirement $\xi_i\geq 0$. The total violation is now $\sum_i\xi$. The value $\xi_i$ in the constraint the last constraint corresponds to the amount by which the prediction -$y_i(\hat{w}^T\hat{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$, +$y_i(\boldmath{w}^T\boldmath{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$, we bound the total amount by which predictions fall on the wrong side of their margins. Misclassifications occur when $\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of @@ -472,18 +478,18 @@ misclassifications. This has in turn the consequences that we change our optmization problem to finding the minimum of !bt \[ -{\cal L}=\frac{1}{2}\hat{w}^T\hat{w}-\sum_{i=1}^n\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, +{\cal L}=\frac{1}{2}\boldmath{w}^T\boldmath{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, \] !et subject to !bt \[ -y_i(\hat{w}^T\hat{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, +y_i(\boldmath{w}^T\boldmath{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, \] !et with the requirement $\xi_i\geq 0$. -Taking the derivatives with respect to $b$ and $\hat{w}$ we obtain +Taking the derivatives with respect to $b$ and $\boldmath{w}$ we obtain !bt \[ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, @@ -492,7 +498,7 @@ Taking the derivatives with respect to $b$ and $\hat{w}$ we obtain and !bt \[ -\frac{\partial {\cal L}}{\partial \hat{w}} = 0 = \hat{w}-\sum_{i} \lambda_iy_i\hat{x}_i, +\frac{\partial {\cal L}}{\partial \boldmath{w}} = 0 = \boldmath{w}-\sum_{i} \lambda_iy_i\boldmath{x}_i, \] !et and @@ -504,14 +510,14 @@ and Inserting these constraints into the equation for ${\cal L}$ we obtain the same equation as before !bt \[ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\hat{x}_i^T\hat{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldmath{x}_i^T\boldmath{x}_j, \] !et but now subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$ and $0\leq\lambda_i \leq C$. We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads !bt \[ -\lambda_i\left[y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, +\lambda_i\left[y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, \] !et !bt @@ -522,7 +528,7 @@ We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads and !bt \[ -y_i(\hat{w}^T\hat{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. +y_i(\boldmath{w}^T\boldmath{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. \] !et