diff --git a/doc/pub/svm/html/._svm-bs004.html b/doc/pub/svm/html/._svm-bs004.html index e75fee4e0..37db97a20 100644 --- a/doc/pub/svm/html/._svm-bs004.html +++ b/doc/pub/svm/html/._svm-bs004.html @@ -139,25 +139,25 @@ b+wx_1+w_2x_2+\dots +w_px_p=0. $$ If we define a -matrix \( \mathbf{X}=\left[\mathbf{x}_1,\mathbf{x}_2,\dots, \mathbf{x}_p\right] \) -of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \mathbf{X} \), +matrix \( \boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right] \) +of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldsymbol{X} \), $$ -\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. +\boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. $$ -If the above condition is not met for a given vector \( \mathbf{x}_i \) we have +If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have $$ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0, $$ if our output \( y_i=1 \). -In this case we say that \( \mathbf{x}_i \) lies on one of the sides of the hyperplane and if +In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if $$ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0, $$ for the class of observations \( y_i=-1 \), -then \( \mathbf{x}_i \) lies on the other side. +then \( \boldsymbol{x}_i \) lies on the other side.
Equivalently, for the two classes of observations we have diff --git a/doc/pub/svm/html/._svm-bs006.html b/doc/pub/svm/html/._svm-bs006.html index 62479f519..cce29d74b 100644 --- a/doc/pub/svm/html/._svm-bs006.html +++ b/doc/pub/svm/html/._svm-bs006.html @@ -134,18 +134,18 @@ MathJax.Hub.Config({
Let us define the function $$ -f(x) = \mathbf{w}^T\mathbf{x}+b = 0, +f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0, $$ as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
-Any point defined by \( \mathbf{x}_i \) and \( \mathbf{x}_2 \) on the line \( L \) will satisfy \( \mathbf{w}^T(\mathbf{x}_1-\mathbf{x}_2)=0 \). +Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).
-The signed distance \( \delta \) from any point defined by a vector \( \mathbf{x} \) and a point \( \mathbf{x}_0 \) on the line \( L \) is then +The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then $$ -\delta = \frac{1}{\vert\vert \mathbf{w}\vert\vert}(\mathbf{w}^T\mathbf{x}+b). +\delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b). $$
diff --git a/doc/pub/svm/html/._svm-bs007.html b/doc/pub/svm/html/._svm-bs007.html index 6186224d0..148601449 100644 --- a/doc/pub/svm/html/._svm-bs007.html +++ b/doc/pub/svm/html/._svm-bs007.html @@ -132,23 +132,23 @@ MathJax.Hub.Config({
-How do we find the parameter \( b \) and the vector \( \mathbf{w} \)? What we could +How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could do is to define a cost function which now contains the set of all misclassified points \( M \) and attempt to minimize this function $$ -C(\mathbf{w},b) = -\sum_{i\in M} y_i(\mathbf{w}^T\mathbf{x}_i+b). +C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b). $$
-We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \mathbf{w}^T\mathbf{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us +We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us $$ \frac{\partial C}{\partial b} = -\sum_{i\in M} y_i, $$ and $$ -\frac{\partial C}{\partial \mathbf{w}} = -\sum_{i\in M} y_ix_i. +\frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i. $$
diff --git a/doc/pub/svm/html/._svm-bs008.html b/doc/pub/svm/html/._svm-bs008.html index e06135a53..b650a6265 100644 --- a/doc/pub/svm/html/._svm-bs008.html +++ b/doc/pub/svm/html/._svm-bs008.html @@ -139,7 +139,7 @@ $$ and $$ -\mathbf{w} \leftarrow \mathbf{w} +\eta \frac{\partial C}{\partial \mathbf{w}}, +\boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}}, $$ where \( \eta \) is our by now well-known learning rate. diff --git a/doc/pub/svm/html/._svm-bs009.html b/doc/pub/svm/html/._svm-bs009.html index a2fb4b4c1..e0042ab0d 100644 --- a/doc/pub/svm/html/._svm-bs009.html +++ b/doc/pub/svm/html/._svm-bs009.html @@ -136,11 +136,11 @@ A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
-Thus, we wish to find a margin \( M \) with \( \mathbf{w} \) normalized to -\( \vert\vert \mathbf{w}\vert\vert =1 \) subject to the condition +Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to +\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. $$ All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line. @@ -148,22 +148,22 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value \( M \) defined by $$ -\frac{1}{\vert \vert \mathbf{w}\vert\vert}y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, +\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, $$ or just $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M\vert \vert \mathbf{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i. $$ -If we scale the equation so that \( \vert \vert \mathbf{w}\vert\vert = 1/M \), we have to find the minimum of -\( \mathbf{w}^T\mathbf{w}=\vert \vert \mathbf{w}\vert\vert \) (the norm) subject to the condition +If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of +\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. $$
-We have thus defined our margin as the invers of the norm of \( \mathbf{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers. +We have thus defined our margin as the invers of the norm of \( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
diff --git a/doc/pub/svm/html/._svm-bs012.html b/doc/pub/svm/html/._svm-bs012.html index be48612ae..5274b124e 100644 --- a/doc/pub/svm/html/._svm-bs012.html +++ b/doc/pub/svm/html/._svm-bs012.html @@ -132,40 +132,40 @@ MathJax.Hub.Config({
-Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ and $$ -\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i. +\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i. $$ Inserting these constraints into the equation for \( {\cal L} \) we obtain $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). We must in addition satisfy the Karush-Kuhn-Tucker (KKT) condition $$ -\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -1\right] \hspace{0.1cm}\forall i. +\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i. $$
diff --git a/doc/pub/svm/html/._svm-bs013.html b/doc/pub/svm/html/._svm-bs013.html index 3a0c677f3..230eacbcd 100644 --- a/doc/pub/svm/html/._svm-bs013.html +++ b/doc/pub/svm/html/._svm-bs013.html @@ -134,21 +134,21 @@ MathJax.Hub.Config({
We can rewrite $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem $$ -\frac{1}{2} \mathbf{\lambda}^T\begin{bmatrix} y_1y_1\mathbf{x}_1^T\mathbf{x}_1 & y_1y_2\mathbf{x}_1^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_1^T\mathbf{x}_n \\ -y_2y_1\mathbf{x}_2^T\mathbf{x}_1 & y_2y_2\mathbf{x}_2^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_2^T\mathbf{x}_n \\ +\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\ +y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\ \dots & \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots & \dots \\ -y_ny_1\mathbf{x}_n^T\mathbf{x}_1 & y_ny_2\mathbf{x}_n^T\mathbf{x}_2 & \dots & \dots & y_ny_n\mathbf{x}_n^T\mathbf{x}_n \\ -\end{bmatrix}\mathbf{\lambda}-\mathbb{1}\mathbf{\lambda}, +y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 & \dots & \dots & y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\ +\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, $$ -subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \( \mathbf{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \mathbf{y}=[y_1,y_2,\dots,y_n] \). +subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
diff --git a/doc/pub/svm/html/._svm-bs014.html b/doc/pub/svm/html/._svm-bs014.html index 5a6d01064..690035116 100644 --- a/doc/pub/svm/html/._svm-bs014.html +++ b/doc/pub/svm/html/._svm-bs014.html @@ -135,27 +135,27 @@ MathJax.Hub.Config({ Solving the above problem, yields the values of \( \lambda_i \). To find the coefficients of your hyperplane we need simply to compute $$ -\mathbf{w}=\sum_{i} \lambda_iy_i\mathbf{x}_i. +\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i. $$ -With our vector \( \mathbf{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via +With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1, $$ resulting in $$ -b = \frac{1}{y_i}-\mathbf{w}^T\mathbf{x}_i, +b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i, $$ or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have $$ -b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\mathbf{x}_i^T\mathbf{x}_j\right). +b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right). $$ With our hyperplane coefficients we can use our classifier to assign any observation by simply using $$ -y_i = \mathrm{sign}(\mathbf{w}^T\mathbf{x}_i+b). +y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b). $$ Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier. diff --git a/doc/pub/svm/html/._svm-bs015.html b/doc/pub/svm/html/._svm-bs015.html index af6bc9a7a..6ee62d029 100644 --- a/doc/pub/svm/html/._svm-bs015.html +++ b/doc/pub/svm/html/._svm-bs015.html @@ -141,20 +141,20 @@ so-called kernel approach, is to allow a kind of slack in the sense that we allow some points to be on the wrong side of the margin.
-We introduce thus the so-called slack variables \( \mathbf{\xi} =[\xi_1,x_2,\dots,x_n] \) and +We introduce thus the so-called slack variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and modify our previous equation $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1, $$ to $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i, $$ with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \). The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction -\( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), +\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), we bound the total amount by which predictions fall on the wrong side of their margins.
diff --git a/doc/pub/svm/html/._svm-bs016.html b/doc/pub/svm/html/._svm-bs016.html index 35c382c58..eafe9c5b7 100644 --- a/doc/pub/svm/html/._svm-bs016.html +++ b/doc/pub/svm/html/._svm-bs016.html @@ -134,25 +134,25 @@ MathJax.Hub.Config({
This has in turn the consequences that we change our optmization problem to finding the minimum of $$ -{\cal L}=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, +{\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, $$ subject to $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, $$ with the requirement \( \xi_i\geq 0 \).
-Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ and $$ -\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i, +\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i, $$ and @@ -162,13 +162,13 @@ $$ Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \). We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads $$ -\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, +\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, $$ $$ @@ -177,7 +177,7 @@ $$ and $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. $$
diff --git a/doc/pub/svm/html/svm-reveal.html b/doc/pub/svm/html/svm-reveal.html index 1a58fcbc6..5243d34be 100644 --- a/doc/pub/svm/html/svm-reveal.html +++ b/doc/pub/svm/html/svm-reveal.html @@ -225,11 +225,11 @@ $$ where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line \( b+w_1x_1+w_2x_2=0 \). -In two dimensions we define the vectors \( \mathbf{x} =[x1,x2] \) and \( \mathbf{w}=[w1,w2] \). +In two dimensions we define the vectors \( \boldsymbol{x} =[x1,x2] \) and \( \boldsymbol{w}=[w1,w2] \). We can then rewrite the above equation as
$$
-\mathbf{w}^T\mathbf{x}+b=0.
+\boldsymbol{w}^T\boldsymbol{x}+b=0.
$$
@@ -248,15 +248,15 @@ $$
If we define a
-matrix \( \mathbf{X}=\left[\mathbf{x}_1,\mathbf{x}_2,\dots, \mathbf{x}_p\right] \)
-of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \mathbf{X} \),
+matrix \( \boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right] \)
+of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldsymbol{X} \),
$$
-\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
+\boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
$$
-If the above condition is not met for a given vector \( \mathbf{x}_i \) we have
+If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
@@ -264,7 +264,7 @@ $$
if our output \( y_i=1 \).
-In this case we say that \( \mathbf{x}_i \) lies on one of the sides of the hyperplane and if
+In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if
$$
b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
@@ -272,7 +272,7 @@ $$
for the class of observations \( y_i=-1 \),
-then \( \mathbf{x}_i \) lies on the other side.
+then \( \boldsymbol{x}_i \) lies on the other side.
Equivalently, for the two classes of observations we have @@ -321,20 +321,20 @@ for our data sample. Let us define the function
$$
-f(x) = \mathbf{w}^T\mathbf{x}+b = 0,
+f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0,
$$
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
-Any point defined by \( \mathbf{x}_i \) and \( \mathbf{x}_2 \) on the line \( L \) will satisfy \( \mathbf{w}^T(\mathbf{x}_1-\mathbf{x}_2)=0 \). +Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).
-The signed distance \( \delta \) from any point defined by a vector \( \mathbf{x} \) and a point \( \mathbf{x}_0 \) on the line \( L \) is then +The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then
$$
-\delta = \frac{1}{\vert\vert \mathbf{w}\vert\vert}(\mathbf{w}^T\mathbf{x}+b).
+\delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b).
$$
@@ -344,18 +344,18 @@ $$
-How do we find the parameter \( b \) and the vector \( \mathbf{w} \)? What we could +How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could do is to define a cost function which now contains the set of all misclassified points \( M \) and attempt to minimize this function
$$
-C(\mathbf{w},b) = -\sum_{i\in M} y_i(\mathbf{w}^T\mathbf{x}_i+b).
+C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b).
$$
-We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \mathbf{w}^T\mathbf{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us +We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
$$
\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
@@ -365,7 +365,7 @@ $$
and
$$
-\frac{\partial C}{\partial \mathbf{w}} = -\sum_{i\in M} y_ix_i.
+\frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i.
$$
@@ -385,7 +385,7 @@ $$
and
$$
-\mathbf{w} \leftarrow \mathbf{w} +\eta \frac{\partial C}{\partial \mathbf{w}},
+\boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}},
$$
@@ -411,12 +411,12 @@ A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).
-Thus, we wish to find a margin \( M \) with \( \mathbf{w} \) normalized to -\( \vert\vert \mathbf{w}\vert\vert =1 \) subject to the condition +Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to +\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition
$$
-y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
@@ -426,27 +426,27 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value \( M \) defined by
$$
-\frac{1}{\vert \vert \mathbf{w}\vert\vert}y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
+\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
or just
$$
-y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M\vert \vert \mathbf{w}\vert\vert \hspace{0.1cm}\forall i.
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
$$
-If we scale the equation so that \( \vert \vert \mathbf{w}\vert\vert = 1/M \), we have to find the minimum of
-\( \mathbf{w}^T\mathbf{w}=\vert \vert \mathbf{w}\vert\vert \) (the norm) subject to the condition
+If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of
+\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition
$$
-y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
$$
-We have thus defined our margin as the invers of the norm of \( \mathbf{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers. +We have thus defined our margin as the invers of the norm of \( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers. @@ -572,14 +572,14 @@ $$ In order to solve the above problem, we define the following Lagrangian function to be minimized
$$
-{\cal L}(\lambda,b,\mathbf{w})=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-1\right],
+{\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right],
$$
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
-Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -589,14 +589,14 @@ $$
and
$$
-\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i.
+\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
$$
Inserting these constraints into the equation for \( {\cal L} \) we obtain
$$
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
@@ -604,18 +604,18 @@ subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the Karush-Kuhn-Tucker (KKT) condition
$$
-\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
+\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
-When \( \lambda_i > 0 \), the vectors \( \mathbf{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \). +When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \). @@ -626,24 +626,24 @@ When \( \lambda_i > 0 \), the vectors \( \mathbf{x}_i \) are called support vect We can rewrite
$$
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
$$
-\frac{1}{2} \mathbf{\lambda}^T\begin{bmatrix} y_1y_1\mathbf{x}_1^T\mathbf{x}_1 & y_1y_2\mathbf{x}_1^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_1^T\mathbf{x}_n \\
-y_2y_1\mathbf{x}_2^T\mathbf{x}_1 & y_2y_2\mathbf{x}_2^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_2^T\mathbf{x}_n \\
+\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\
+y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
-y_ny_1\mathbf{x}_n^T\mathbf{x}_1 & y_ny_2\mathbf{x}_n^T\mathbf{x}_2 & \dots & \dots & y_ny_n\mathbf{x}_n^T\mathbf{x}_n \\
-\end{bmatrix}\mathbf{\lambda}-\mathbb{1}\mathbf{\lambda},
+y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 & \dots & \dots & y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\
+\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
-subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \( \mathbf{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
-\( \mathbf{y}=[y_1,y_2,\dots,y_n] \).
+subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
+\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
@@ -655,35 +655,35 @@ Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
$$
-\mathbf{w}=\sum_{i} \lambda_iy_i\mathbf{x}_i.
+\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
$$
-With our vector \( \mathbf{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
+With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
-y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
$$
resulting in
$$
-b = \frac{1}{y_i}-\mathbf{w}^T\mathbf{x}_i,
+b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
$$
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
$$
-b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\mathbf{x}_i^T\mathbf{x}_j\right).
+b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right).
$$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
-y_i = \mathrm{sign}(\mathbf{w}^T\mathbf{x}_i+b).
+y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
$$
@@ -704,24 +704,24 @@ so-called kernel approach, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
-We introduce thus the so-called slack variables \( \mathbf{\xi} =[\xi_1,x_2,\dots,x_n] \) and +We introduce thus the so-called slack variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and modify our previous equation
$$
-y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
$$
to
$$
-y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i,
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i,
$$
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
-\( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
+\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
@@ -737,21 +737,21 @@ misclassifications. This has in turn the consequences that we change our optmization problem to finding the minimum of
$$
-{\cal L}=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
+{\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
$$
subject to
$$
-y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
with the requirement \( \xi_i\geq 0 \).
-Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
$$
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
@@ -761,7 +761,7 @@ $$
and
$$
-\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i,
+\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i,
$$
@@ -775,7 +775,7 @@ $$
Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before
$$
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
$$
@@ -783,7 +783,7 @@ but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
$$
-\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
+\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
@@ -796,7 +796,7 @@ $$
and
$$
-y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
diff --git a/doc/pub/svm/html/svm-solarized.html b/doc/pub/svm/html/svm-solarized.html
index b3cc9ba1f..85221cb67 100644
--- a/doc/pub/svm/html/svm-solarized.html
+++ b/doc/pub/svm/html/svm-solarized.html
@@ -164,10 +164,10 @@ $$
where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line
\( b+w_1x_1+w_2x_2=0 \).
-In two dimensions we define the vectors \( \mathbf{x} =[x1,x2] \) and \( \mathbf{w}=[w1,w2] \).
+In two dimensions we define the vectors \( \boldsymbol{x} =[x1,x2] \) and \( \boldsymbol{w}=[w1,w2] \).
We can then rewrite the above equation as
$$
-\mathbf{w}^T\mathbf{x}+b=0.
+\boldsymbol{w}^T\boldsymbol{x}+b=0.
$$
@@ -183,25 +183,25 @@ b+wx_1+w_2x_2+\dots +w_px_p=0. $$ If we define a -matrix \( \mathbf{X}=\left[\mathbf{x}_1,\mathbf{x}_2,\dots, \mathbf{x}_p\right] \) -of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \mathbf{X} \), +matrix \( \boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right] \) +of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldsymbol{X} \), $$ -\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. +\boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. $$ -If the above condition is not met for a given vector \( \mathbf{x}_i \) we have +If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have $$ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0, $$ if our output \( y_i=1 \). -In this case we say that \( \mathbf{x}_i \) lies on one of the sides of the hyperplane and if +In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if $$ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0, $$ for the class of observations \( y_i=-1 \), -then \( \mathbf{x}_i \) lies on the other side. +then \( \boldsymbol{x}_i \) lies on the other side.
Equivalently, for the two classes of observations we have @@ -247,18 +247,18 @@ for our data sample.
Let us define the function $$ -f(x) = \mathbf{w}^T\mathbf{x}+b = 0, +f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0, $$ as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
-Any point defined by \( \mathbf{x}_i \) and \( \mathbf{x}_2 \) on the line \( L \) will satisfy \( \mathbf{w}^T(\mathbf{x}_1-\mathbf{x}_2)=0 \). +Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).
-The signed distance \( \delta \) from any point defined by a vector \( \mathbf{x} \) and a point \( \mathbf{x}_0 \) on the line \( L \) is then +The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then $$ -\delta = \frac{1}{\vert\vert \mathbf{w}\vert\vert}(\mathbf{w}^T\mathbf{x}+b). +\delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b). $$
@@ -267,23 +267,23 @@ $$
-How do we find the parameter \( b \) and the vector \( \mathbf{w} \)? What we could +How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could do is to define a cost function which now contains the set of all misclassified points \( M \) and attempt to minimize this function $$ -C(\mathbf{w},b) = -\sum_{i\in M} y_i(\mathbf{w}^T\mathbf{x}_i+b). +C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b). $$
-We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \mathbf{w}^T\mathbf{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us +We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us $$ \frac{\partial C}{\partial b} = -\sum_{i\in M} y_i, $$ and $$ -\frac{\partial C}{\partial \mathbf{w}} = -\sum_{i\in M} y_ix_i. +\frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i. $$
@@ -299,7 +299,7 @@ $$ and $$ -\mathbf{w} \leftarrow \mathbf{w} +\eta \frac{\partial C}{\partial \mathbf{w}}, +\boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}}, $$ where \( \eta \) is our by now well-known learning rate. @@ -324,11 +324,11 @@ A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
-Thus, we wish to find a margin \( M \) with \( \mathbf{w} \) normalized to -\( \vert\vert \mathbf{w}\vert\vert =1 \) subject to the condition +Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to +\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. $$ All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line. @@ -336,22 +336,22 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value \( M \) defined by $$ -\frac{1}{\vert \vert \mathbf{w}\vert\vert}y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, +\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, $$ or just $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M\vert \vert \mathbf{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i. $$ -If we scale the equation so that \( \vert \vert \mathbf{w}\vert\vert = 1/M \), we have to find the minimum of -\( \mathbf{w}^T\mathbf{w}=\vert \vert \mathbf{w}\vert\vert \) (the norm) subject to the condition +If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of +\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. $$
-We have thus defined our margin as the invers of the norm of \( \mathbf{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers. +We have thus defined our margin as the invers of the norm of \( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
@@ -452,40 +452,40 @@ $$
-Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ and $$ -\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i. +\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i. $$ Inserting these constraints into the equation for \( {\cal L} \) we obtain $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). We must in addition satisfy the Karush-Kuhn-Tucker (KKT) condition $$ -\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -1\right] \hspace{0.1cm}\forall i. +\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i. $$
@@ -495,21 +495,21 @@ When \( \lambda_i > 0 \), the vectors \( \mathbf{x}_i \) are called support vect
We can rewrite $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem $$ -\frac{1}{2} \mathbf{\lambda}^T\begin{bmatrix} y_1y_1\mathbf{x}_1^T\mathbf{x}_1 & y_1y_2\mathbf{x}_1^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_1^T\mathbf{x}_n \\ -y_2y_1\mathbf{x}_2^T\mathbf{x}_1 & y_2y_2\mathbf{x}_2^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_2^T\mathbf{x}_n \\ +\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\ +y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\ \dots & \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots & \dots \\ -y_ny_1\mathbf{x}_n^T\mathbf{x}_1 & y_ny_2\mathbf{x}_n^T\mathbf{x}_2 & \dots & \dots & y_ny_n\mathbf{x}_n^T\mathbf{x}_n \\ -\end{bmatrix}\mathbf{\lambda}-\mathbb{1}\mathbf{\lambda}, +y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 & \dots & \dots & y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\ +\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, $$ -subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \( \mathbf{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \mathbf{y}=[y_1,y_2,\dots,y_n] \). +subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
@@ -520,27 +520,27 @@ subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \(
Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
$$
-\mathbf{w}=\sum_{i} \lambda_iy_i\mathbf{x}_i.
+\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
$$
-With our vector \( \mathbf{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
+With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
-y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
$$
resulting in
$$
-b = \frac{1}{y_i}-\mathbf{w}^T\mathbf{x}_i,
+b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
$$
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
$$
-b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\mathbf{x}_i^T\mathbf{x}_j\right).
+b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right).
$$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
-y_i = \mathrm{sign}(\mathbf{w}^T\mathbf{x}_i+b).
+y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
$$
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
@@ -560,20 +560,20 @@ so-called kernel approach, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
-We introduce thus the so-called slack variables \( \mathbf{\xi} =[\xi_1,x_2,\dots,x_n] \) and +We introduce thus the so-called slack variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and modify our previous equation $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1, $$ to $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i, $$ with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \). The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction -\( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), +\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), we bound the total amount by which predictions fall on the wrong side of their margins.
@@ -588,25 +588,25 @@ misclassifications.
This has in turn the consequences that we change our optmization problem to finding the minimum of $$ -{\cal L}=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, +{\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, $$ subject to $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, $$ with the requirement \( \xi_i\geq 0 \).
-Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ and $$ -\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i, +\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i, $$ and @@ -616,13 +616,13 @@ $$ Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \). We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads $$ -\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, +\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, $$ $$ @@ -631,7 +631,7 @@ $$ and $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. $$
diff --git a/doc/pub/svm/html/svm.html b/doc/pub/svm/html/svm.html index c2e61d1e2..4d0d9e0e2 100644 --- a/doc/pub/svm/html/svm.html +++ b/doc/pub/svm/html/svm.html @@ -169,10 +169,10 @@ $$ where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line \( b+w_1x_1+w_2x_2=0 \). -In two dimensions we define the vectors \( \mathbf{x} =[x1,x2] \) and \( \mathbf{w}=[w1,w2] \). +In two dimensions we define the vectors \( \boldsymbol{x} =[x1,x2] \) and \( \boldsymbol{w}=[w1,w2] \). We can then rewrite the above equation as $$ -\mathbf{w}^T\mathbf{x}+b=0. +\boldsymbol{w}^T\boldsymbol{x}+b=0. $$
@@ -188,25 +188,25 @@ b+wx_1+w_2x_2+\dots +w_px_p=0. $$ If we define a -matrix \( \mathbf{X}=\left[\mathbf{x}_1,\mathbf{x}_2,\dots, \mathbf{x}_p\right] \) -of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \mathbf{X} \), +matrix \( \boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right] \) +of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldsymbol{X} \), $$ -\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. +\boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. $$ -If the above condition is not met for a given vector \( \mathbf{x}_i \) we have +If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have $$ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0, $$ if our output \( y_i=1 \). -In this case we say that \( \mathbf{x}_i \) lies on one of the sides of the hyperplane and if +In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if $$ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0, $$ for the class of observations \( y_i=-1 \), -then \( \mathbf{x}_i \) lies on the other side. +then \( \boldsymbol{x}_i \) lies on the other side.
Equivalently, for the two classes of observations we have @@ -252,18 +252,18 @@ for our data sample.
Let us define the function $$ -f(x) = \mathbf{w}^T\mathbf{x}+b = 0, +f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0, $$ as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
-Any point defined by \( \mathbf{x}_i \) and \( \mathbf{x}_2 \) on the line \( L \) will satisfy \( \mathbf{w}^T(\mathbf{x}_1-\mathbf{x}_2)=0 \). +Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).
-The signed distance \( \delta \) from any point defined by a vector \( \mathbf{x} \) and a point \( \mathbf{x}_0 \) on the line \( L \) is then +The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then $$ -\delta = \frac{1}{\vert\vert \mathbf{w}\vert\vert}(\mathbf{w}^T\mathbf{x}+b). +\delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b). $$
@@ -272,23 +272,23 @@ $$
-How do we find the parameter \( b \) and the vector \( \mathbf{w} \)? What we could +How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could do is to define a cost function which now contains the set of all misclassified points \( M \) and attempt to minimize this function $$ -C(\mathbf{w},b) = -\sum_{i\in M} y_i(\mathbf{w}^T\mathbf{x}_i+b). +C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b). $$
-We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \mathbf{w}^T\mathbf{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us +We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us $$ \frac{\partial C}{\partial b} = -\sum_{i\in M} y_i, $$ and $$ -\frac{\partial C}{\partial \mathbf{w}} = -\sum_{i\in M} y_ix_i. +\frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i. $$
@@ -304,7 +304,7 @@ $$ and $$ -\mathbf{w} \leftarrow \mathbf{w} +\eta \frac{\partial C}{\partial \mathbf{w}}, +\boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}}, $$ where \( \eta \) is our by now well-known learning rate. @@ -329,11 +329,11 @@ A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
-Thus, we wish to find a margin \( M \) with \( \mathbf{w} \) normalized to -\( \vert\vert \mathbf{w}\vert\vert =1 \) subject to the condition +Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to +\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. $$ All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line. @@ -341,22 +341,22 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value \( M \) defined by $$ -\frac{1}{\vert \vert \mathbf{w}\vert\vert}y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, +\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, $$ or just $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M\vert \vert \mathbf{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i. $$ -If we scale the equation so that \( \vert \vert \mathbf{w}\vert\vert = 1/M \), we have to find the minimum of -\( \mathbf{w}^T\mathbf{w}=\vert \vert \mathbf{w}\vert\vert \) (the norm) subject to the condition +If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of +\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. $$
-We have thus defined our margin as the invers of the norm of \( \mathbf{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers. +We have thus defined our margin as the invers of the norm of \( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
@@ -457,40 +457,40 @@ $$
-Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ and $$ -\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i. +\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i. $$ Inserting these constraints into the equation for \( {\cal L} \) we obtain $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). We must in addition satisfy the Karush-Kuhn-Tucker (KKT) condition $$ -\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -1\right] \hspace{0.1cm}\forall i. +\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i. $$
@@ -500,21 +500,21 @@ When \( \lambda_i > 0 \), the vectors \( \mathbf{x}_i \) are called support vect
We can rewrite $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem $$ -\frac{1}{2} \mathbf{\lambda}^T\begin{bmatrix} y_1y_1\mathbf{x}_1^T\mathbf{x}_1 & y_1y_2\mathbf{x}_1^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_1^T\mathbf{x}_n \\ -y_2y_1\mathbf{x}_2^T\mathbf{x}_1 & y_2y_2\mathbf{x}_2^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_2^T\mathbf{x}_n \\ +\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\ +y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\ \dots & \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots & \dots \\ -y_ny_1\mathbf{x}_n^T\mathbf{x}_1 & y_ny_2\mathbf{x}_n^T\mathbf{x}_2 & \dots & \dots & y_ny_n\mathbf{x}_n^T\mathbf{x}_n \\ -\end{bmatrix}\mathbf{\lambda}-\mathbb{1}\mathbf{\lambda}, +y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 & \dots & \dots & y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\ +\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, $$ -subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \( \mathbf{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \mathbf{y}=[y_1,y_2,\dots,y_n] \). +subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
@@ -525,27 +525,27 @@ subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \(
Solving the above problem, yields the values of \( \lambda_i \).
To find the coefficients of your hyperplane we need simply to compute
$$
-\mathbf{w}=\sum_{i} \lambda_iy_i\mathbf{x}_i.
+\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
$$
-With our vector \( \mathbf{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
+With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
-y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
$$
resulting in
$$
-b = \frac{1}{y_i}-\mathbf{w}^T\mathbf{x}_i,
+b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
$$
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
$$
-b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\mathbf{x}_i^T\mathbf{x}_j\right).
+b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right).
$$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
-y_i = \mathrm{sign}(\mathbf{w}^T\mathbf{x}_i+b).
+y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
$$
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
@@ -565,20 +565,20 @@ so-called kernel approach, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
-We introduce thus the so-called slack variables \( \mathbf{\xi} =[\xi_1,x_2,\dots,x_n] \) and +We introduce thus the so-called slack variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and modify our previous equation $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1, $$ to $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i, $$ with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \). The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction -\( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), +\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), we bound the total amount by which predictions fall on the wrong side of their margins.
@@ -593,25 +593,25 @@ misclassifications.
This has in turn the consequences that we change our optmization problem to finding the minimum of $$ -{\cal L}=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, +{\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, $$ subject to $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, $$ with the requirement \( \xi_i\geq 0 \).
-Taking the derivatives with respect to \( b \) and \( \mathbf{w} \) we obtain +Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ and $$ -\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i, +\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i, $$ and @@ -621,13 +621,13 @@ $$ Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \). We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads $$ -\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, +\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, $$ $$ @@ -636,7 +636,7 @@ $$ and $$ -y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. $$
diff --git a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz index 915b9ae6a..c0c08712c 100644 Binary files a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz and b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz differ diff --git a/doc/pub/svm/ipynb/svm.ipynb b/doc/pub/svm/ipynb/svm.ipynb index 023ab175e..346e18da1 100644 --- a/doc/pub/svm/ipynb/svm.ipynb +++ b/doc/pub/svm/ipynb/svm.ipynb @@ -79,7 +79,7 @@ "source": [ "where $b$ is the intercept and $w_1$ and $w_2$ define the elements of a vector orthogonal to the line \n", "$b+w_1x_1+w_2x_2=0$. \n", - "In two dimensions we define the vectors $\\mathbf{x} =[x1,x2]$ and $\\mathbf{w}=[w1,w2]$. \n", + "In two dimensions we define the vectors $\\boldsymbol{x} =[x1,x2]$ and $\\boldsymbol{w}=[w1,w2]$. \n", "We can then rewrite the above equation as" ] }, @@ -88,7 +88,7 @@ "metadata": {}, "source": [ "$$\n", - "\\mathbf{w}^T\\mathbf{x}+b=0.\n", + "\\boldsymbol{w}^T\\boldsymbol{x}+b=0.\n", "$$" ] }, @@ -116,8 +116,8 @@ "metadata": {}, "source": [ "If we define a \n", - "matrix $\\mathbf{X}=\\left[\\mathbf{x}_1,\\mathbf{x}_2,\\dots, \\mathbf{x}_p\\right]$\n", - "of dimension $n\\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\\mathbf{X}$," + "matrix $\\boldsymbol{X}=\\left[\\boldsymbol{x}_1,\\boldsymbol{x}_2,\\dots, \\boldsymbol{x}_p\\right]$\n", + "of dimension $n\\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\\boldsymbol{X}$," ] }, { @@ -125,7 +125,7 @@ "metadata": {}, "source": [ "$$\n", - "\\mathbf{x}_i = \\begin{bmatrix} x_{i1} \\\\ x_{i2} \\\\ \\dots \\\\ \\dots \\\\ x_{ip} \\end{bmatrix}.\n", + "\\boldsymbol{x}_i = \\begin{bmatrix} x_{i1} \\\\ x_{i2} \\\\ \\dots \\\\ \\dots \\\\ x_{ip} \\end{bmatrix}.\n", "$$" ] }, @@ -133,7 +133,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If the above condition is not met for a given vector $\\mathbf{x}_i$ we have" + "If the above condition is not met for a given vector $\\boldsymbol{x}_i$ we have" ] }, { @@ -150,7 +150,7 @@ "metadata": {}, "source": [ "if our output $y_i=1$.\n", - "In this case we say that $\\mathbf{x}_i$ lies on one of the sides of the hyperplane and if" + "In this case we say that $\\boldsymbol{x}_i$ lies on one of the sides of the hyperplane and if" ] }, { @@ -167,7 +167,7 @@ "metadata": {}, "source": [ "for the class of observations $y_i=-1$, \n", - "then $\\mathbf{x}_i$ lies on the other side. \n", + "then $\\boldsymbol{x}_i$ lies on the other side. \n", "\n", "Equivalently, for the two classes of observations we have" ] @@ -219,7 +219,7 @@ "metadata": {}, "source": [ "$$\n", - "f(x) = \\mathbf{w}^T\\mathbf{x}+b = 0,\n", + "f(x) = \\boldsymbol{w}^T\\boldsymbol{x}+b = 0,\n", "$$" ] }, @@ -230,9 +230,9 @@ "as the function that determines the line $L$ that separates two classes (our two features), see the figure here. \n", "\n", "\n", - "Any point defined by $\\mathbf{x}_i$ and $\\mathbf{x}_2$ on the line $L$ will satisfy $\\mathbf{w}^T(\\mathbf{x}_1-\\mathbf{x}_2)=0$. \n", + "Any point defined by $\\boldsymbol{x}_i$ and $\\boldsymbol{x}_2$ on the line $L$ will satisfy $\\boldsymbol{w}^T(\\boldsymbol{x}_1-\\boldsymbol{x}_2)=0$. \n", "\n", - "The signed distance $\\delta$ from any point defined by a vector $\\mathbf{x}$ and a point $\\mathbf{x}_0$ on the line $L$ is then" + "The signed distance $\\delta$ from any point defined by a vector $\\boldsymbol{x}$ and a point $\\boldsymbol{x}_0$ on the line $L$ is then" ] }, { @@ -240,7 +240,7 @@ "metadata": {}, "source": [ "$$\n", - "\\delta = \\frac{1}{\\vert\\vert \\mathbf{w}\\vert\\vert}(\\mathbf{w}^T\\mathbf{x}+b).\n", + "\\delta = \\frac{1}{\\vert\\vert \\boldsymbol{w}\\vert\\vert}(\\boldsymbol{w}^T\\boldsymbol{x}+b).\n", "$$" ] }, @@ -250,7 +250,7 @@ "source": [ "## First attempt at a minimization approach\n", "\n", - "How do we find the parameter $b$ and the vector $\\mathbf{w}$? What we could\n", + "How do we find the parameter $b$ and the vector $\\boldsymbol{w}$? What we could\n", "do is to define a cost function which now contains the set of all\n", "misclassified points $M$ and attempt to minimize this function" ] @@ -260,7 +260,7 @@ "metadata": {}, "source": [ "$$\n", - "C(\\mathbf{w},b) = -\\sum_{i\\in M} y_i(\\mathbf{w}^T\\mathbf{x}_i+b).\n", + "C(\\boldsymbol{w},b) = -\\sum_{i\\in M} y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b).\n", "$$" ] }, @@ -268,7 +268,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We could now for example define all values $y_i =1$ as misclassified in case we have $\\mathbf{w}^T\\mathbf{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us" + "We could now for example define all values $y_i =1$ as misclassified in case we have $\\boldsymbol{w}^T\\boldsymbol{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us" ] }, { @@ -292,7 +292,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial C}{\\partial \\mathbf{w}} = -\\sum_{i\\in M} y_ix_i.\n", + "\\frac{\\partial C}{\\partial \\boldsymbol{w}} = -\\sum_{i\\in M} y_ix_i.\n", "$$" ] }, @@ -326,7 +326,7 @@ "metadata": {}, "source": [ "$$\n", - "\\mathbf{w} \\leftarrow \\mathbf{w} +\\eta \\frac{\\partial C}{\\partial \\mathbf{w}},\n", + "\\boldsymbol{w} \\leftarrow \\boldsymbol{w} +\\eta \\frac{\\partial C}{\\partial \\boldsymbol{w}},\n", "$$" ] }, @@ -350,8 +350,8 @@ "A better approach is rather to try to define a large margin between\n", "the two classes (if they are well separated from the beginning).\n", "\n", - "Thus, we wish to find a margin $M$ with $\\mathbf{w}$ normalized to\n", - "$\\vert\\vert \\mathbf{w}\\vert\\vert =1$ subject to the condition" + "Thus, we wish to find a margin $M$ with $\\boldsymbol{w}$ normalized to\n", + "$\\vert\\vert \\boldsymbol{w}\\vert\\vert =1$ subject to the condition" ] }, { @@ -359,7 +359,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\mathbf{w}^T\\mathbf{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, p.\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, p.\n", "$$" ] }, @@ -377,7 +377,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{1}{\\vert \\vert \\mathbf{w}\\vert\\vert}y_i(\\mathbf{w}^T\\mathbf{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, n,\n", + "\\frac{1}{\\vert \\vert \\boldsymbol{w}\\vert\\vert}y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, n,\n", "$$" ] }, @@ -393,7 +393,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\mathbf{w}^T\\mathbf{x}_i+b) \\geq M\\vert \\vert \\mathbf{w}\\vert\\vert \\hspace{0.1cm}\\forall i.\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M\\vert \\vert \\boldsymbol{w}\\vert\\vert \\hspace{0.1cm}\\forall i.\n", "$$" ] }, @@ -401,8 +401,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If we scale the equation so that $\\vert \\vert \\mathbf{w}\\vert\\vert = 1/M$, we have to find the minimum of \n", - "$\\mathbf{w}^T\\mathbf{w}=\\vert \\vert \\mathbf{w}\\vert\\vert$ (the norm) subject to the condition" + "If we scale the equation so that $\\vert \\vert \\boldsymbol{w}\\vert\\vert = 1/M$, we have to find the minimum of \n", + "$\\boldsymbol{w}^T\\boldsymbol{w}=\\vert \\vert \\boldsymbol{w}\\vert\\vert$ (the norm) subject to the condition" ] }, { @@ -410,7 +410,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\mathbf{w}^T\\mathbf{x}_i+b) \\geq 1 \\hspace{0.1cm}\\forall i.\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq 1 \\hspace{0.1cm}\\forall i.\n", "$$" ] }, @@ -418,7 +418,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We have thus defined our margin as the invers of the norm of $\\mathbf{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers. \n", + "We have thus defined our margin as the invers of the norm of $\\boldsymbol{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers. \n", "\n", "## A quick reminder on Lagrangian multipliers\n", "\n", @@ -644,7 +644,7 @@ "metadata": {}, "source": [ "$$\n", - "{\\cal L}(\\lambda,b,\\mathbf{w})=\\frac{1}{2}\\mathbf{w}^T\\mathbf{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\mathbf{w}^T\\mathbf{x}_i+b)-1\\right],\n", + "{\\cal L}(\\lambda,b,\\boldsymbol{w})=\\frac{1}{2}\\boldsymbol{w}^T\\boldsymbol{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)-1\\right],\n", "$$" ] }, @@ -654,7 +654,7 @@ "source": [ "where $\\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\\lambda_i \\geq 0$.\n", "\n", - "Taking the derivatives with respect to $b$ and $\\mathbf{w}$ we obtain" + "Taking the derivatives with respect to $b$ and $\\boldsymbol{w}$ we obtain" ] }, { @@ -678,7 +678,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial {\\cal L}}{\\partial \\mathbf{w}} = 0 = \\mathbf{w}-\\sum_{i} \\lambda_iy_i\\mathbf{x}_i.\n", + "\\frac{\\partial {\\cal L}}{\\partial \\boldsymbol{w}} = 0 = \\boldsymbol{w}-\\sum_{i} \\lambda_iy_i\\boldsymbol{x}_i.\n", "$$" ] }, @@ -694,7 +694,7 @@ "metadata": {}, "source": [ "$$\n", - "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\mathbf{x}_i^T\\mathbf{x}_j,\n", + "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{x}_j,\n", "$$" ] }, @@ -711,7 +711,7 @@ "metadata": {}, "source": [ "$$\n", - "\\lambda_i\\left[y_i(\\mathbf{w}^T\\mathbf{x}_i+b) -1\\right] \\hspace{0.1cm}\\forall i.\n", + "\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) -1\\right] \\hspace{0.1cm}\\forall i.\n", "$$" ] }, @@ -719,11 +719,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "1. If $\\lambda_i > 0$, then $y_i(\\mathbf{w}^T\\mathbf{x}_i+b)=1$ and we say that $x_i$ is on the boundary.\n", + "1. If $\\lambda_i > 0$, then $y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1$ and we say that $x_i$ is on the boundary.\n", "\n", - "2. If $y_i(\\mathbf{w}^T\\mathbf{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\\lambda_i=0$. \n", + "2. If $y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\\lambda_i=0$. \n", "\n", - "When $\\lambda_i > 0$, the vectors $\\mathbf{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. \n", + "When $\\lambda_i > 0$, the vectors $\\boldsymbol{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. \n", "\n", "## The problem to solve\n", "\n", @@ -735,7 +735,7 @@ "metadata": {}, "source": [ "$$\n", - "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\mathbf{x}_i^T\\mathbf{x}_j,\n", + "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{x}_j,\n", "$$" ] }, @@ -751,12 +751,12 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{1}{2} \\mathbf{\\lambda}^T\\begin{bmatrix} y_1y_1\\mathbf{x}_1^T\\mathbf{x}_1 & y_1y_2\\mathbf{x}_1^T\\mathbf{x}_2 & \\dots & \\dots & y_1y_n\\mathbf{x}_1^T\\mathbf{x}_n \\\\\n", - "y_2y_1\\mathbf{x}_2^T\\mathbf{x}_1 & y_2y_2\\mathbf{x}_2^T\\mathbf{x}_2 & \\dots & \\dots & y_1y_n\\mathbf{x}_2^T\\mathbf{x}_n \\\\\n", + "\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1\\boldsymbol{x}_1^T\\boldsymbol{x}_1 & y_1y_2\\boldsymbol{x}_1^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_1^T\\boldsymbol{x}_n \\\\\n", + "y_2y_1\\boldsymbol{x}_2^T\\boldsymbol{x}_1 & y_2y_2\\boldsymbol{x}_2^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_2^T\\boldsymbol{x}_n \\\\\n", "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", - "y_ny_1\\mathbf{x}_n^T\\mathbf{x}_1 & y_ny_2\\mathbf{x}_n^T\\mathbf{x}_2 & \\dots & \\dots & y_ny_n\\mathbf{x}_n^T\\mathbf{x}_n \\\\\n", - "\\end{bmatrix}\\mathbf{\\lambda}-\\mathbb{1}\\mathbf{\\lambda},\n", + "y_ny_1\\boldsymbol{x}_n^T\\boldsymbol{x}_1 & y_ny_2\\boldsymbol{x}_n^T\\boldsymbol{x}_2 & \\dots & \\dots & y_ny_n\\boldsymbol{x}_n^T\\boldsymbol{x}_n \\\\\n", + "\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{1}\\boldsymbol{\\lambda},\n", "$$" ] }, @@ -764,8 +764,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "subject to $\\mathbf{y}^T\\mathbf{\\lambda}=0$. Here we defined the vectors $\\mathbf{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n", - "$\\mathbf{y}=[y_1,y_2,\\dots,y_n]$. \n", + "subject to $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$. Here we defined the vectors $\\boldsymbol{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n", + "$\\boldsymbol{y}=[y_1,y_2,\\dots,y_n]$. \n", "\n", "\n", "## The last steps\n", @@ -779,7 +779,7 @@ "metadata": {}, "source": [ "$$\n", - "\\mathbf{w}=\\sum_{i} \\lambda_iy_i\\mathbf{x}_i.\n", + "\\boldsymbol{w}=\\sum_{i} \\lambda_iy_i\\boldsymbol{x}_i.\n", "$$" ] }, @@ -787,7 +787,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "With our vector $\\mathbf{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via" + "With our vector $\\boldsymbol{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via" ] }, { @@ -795,7 +795,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\mathbf{w}^T\\mathbf{x}_i+b)=1,\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1,\n", "$$" ] }, @@ -811,7 +811,7 @@ "metadata": {}, "source": [ "$$\n", - "b = \\frac{1}{y_i}-\\mathbf{w}^T\\mathbf{x}_i,\n", + "b = \\frac{1}{y_i}-\\boldsymbol{w}^T\\boldsymbol{x}_i,\n", "$$" ] }, @@ -827,7 +827,7 @@ "metadata": {}, "source": [ "$$\n", - "b = \\frac{1}{N_s}\\sum_{j\\in N_s}\\left(y_j-\\sum_{i=1}^n\\lambda_iy_i\\mathbf{x}_i^T\\mathbf{x}_j\\right).\n", + "b = \\frac{1}{N_s}\\sum_{j\\in N_s}\\left(y_j-\\sum_{i=1}^n\\lambda_iy_i\\boldsymbol{x}_i^T\\boldsymbol{x}_j\\right).\n", "$$" ] }, @@ -843,7 +843,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i = \\mathrm{sign}(\\mathbf{w}^T\\mathbf{x}_i+b).\n", + "y_i = \\mathrm{sign}(\\boldsymbol{w}^T\\boldsymbol{x}_i+b).\n", "$$" ] }, @@ -862,7 +862,7 @@ "so-called **kernel approach**, is to allow a kind of slack in the sense\n", "that we allow some points to be on the wrong side of the margin.\n", "\n", - "We introduce thus the so-called **slack** variables $\\mathbf{\\xi} =[\\xi_1,x_2,\\dots,x_n]$ and \n", + "We introduce thus the so-called **slack** variables $\\boldsymbol{\\xi} =[\\xi_1,x_2,\\dots,x_n]$ and \n", "modify our previous equation" ] }, @@ -871,7 +871,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\mathbf{w}^T\\mathbf{x}_i+b)=1,\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1,\n", "$$" ] }, @@ -887,7 +887,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\mathbf{w}^T\\mathbf{x}_i+b)=1-\\xi_i,\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1-\\xi_i,\n", "$$" ] }, @@ -897,7 +897,7 @@ "source": [ "with the requirement $\\xi_i\\geq 0$. The total violation is now $\\sum_i\\xi$. \n", "The value $\\xi_i$ in the constraint the last constraint corresponds to the amount by which the prediction\n", - "$y_i(\\mathbf{w}^T\\mathbf{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\\sum_i \\xi_i$,\n", + "$y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\\sum_i \\xi_i$,\n", "we bound the total amount by which predictions fall on the wrong side of their margins.\n", "\n", "Misclassifications occur when $\\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of\n", @@ -914,7 +914,7 @@ "metadata": {}, "source": [ "$$\n", - "{\\cal L}=\\frac{1}{2}\\mathbf{w}^T\\mathbf{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\mathbf{w}^T\\mathbf{x}_i+b)-(1-\\xi_)\\right]+C\\sum_{i=1}^n\\xi_i-\\sum_{i=1}^n\\gamma_i\\xi_i,\n", + "{\\cal L}=\\frac{1}{2}\\boldsymbol{w}^T\\boldsymbol{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)-(1-\\xi_)\\right]+C\\sum_{i=1}^n\\xi_i-\\sum_{i=1}^n\\gamma_i\\xi_i,\n", "$$" ] }, @@ -930,7 +930,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\mathbf{w}^T\\mathbf{x}_i+b)=1-\\xi_i \\hspace{0.1cm}\\forall i,\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1-\\xi_i \\hspace{0.1cm}\\forall i,\n", "$$" ] }, @@ -940,7 +940,7 @@ "source": [ "with the requirement $\\xi_i\\geq 0$.\n", "\n", - "Taking the derivatives with respect to $b$ and $\\mathbf{w}$ we obtain" + "Taking the derivatives with respect to $b$ and $\\boldsymbol{w}$ we obtain" ] }, { @@ -964,7 +964,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial {\\cal L}}{\\partial \\mathbf{w}} = 0 = \\mathbf{w}-\\sum_{i} \\lambda_iy_i\\mathbf{x}_i,\n", + "\\frac{\\partial {\\cal L}}{\\partial \\boldsymbol{w}} = 0 = \\boldsymbol{w}-\\sum_{i} \\lambda_iy_i\\boldsymbol{x}_i,\n", "$$" ] }, @@ -996,7 +996,7 @@ "metadata": {}, "source": [ "$$\n", - "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\mathbf{x}_i^T\\mathbf{x}_j,\n", + "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{x}_j,\n", "$$" ] }, @@ -1053,7 +1053,7 @@ "metadata": {}, "source": [ "$$\n", - "y_i(\\mathbf{w}^T\\mathbf{x}_i+b) -(1-\\xi_) \\geq 0 \\hspace{0.1cm}\\forall i.\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) -(1-\\xi_) \\geq 0 \\hspace{0.1cm}\\forall i.\n", "$$" ] }, diff --git a/doc/pub/svm/pdf/svm-minted.pdf b/doc/pub/svm/pdf/svm-minted.pdf index 31f7e11b3..278550919 100644 Binary files a/doc/pub/svm/pdf/svm-minted.pdf and b/doc/pub/svm/pdf/svm-minted.pdf differ diff --git a/doc/src/SupportVMachines/svm.do.txt b/doc/src/SupportVMachines/svm.do.txt index b5598ce34..ba10dabf7 100644 --- a/doc/src/SupportVMachines/svm.do.txt +++ b/doc/src/SupportVMachines/svm.do.txt @@ -58,11 +58,11 @@ b+w_1x_1+w_2x_2=0, !et where $b$ is the intercept and $w_1$ and $w_2$ define the elements of a vector orthogonal to the line $b+w_1x_1+w_2x_2=0$. -In two dimensions we define the vectors $\mathbf{x} =[x1,x2]$ and $\mathbf{w}=[w1,w2]$. +In two dimensions we define the vectors $\bm{x} =[x1,x2]$ and $\bm{w}=[w1,w2]$. We can then rewrite the above equation as !bt \[ -\mathbf{w}^T\mathbf{x}+b=0. +\bm{w}^T\bm{x}+b=0. \] !et @@ -77,28 +77,28 @@ b+wx_1+w_2x_2+\dots +w_px_p=0. \] !et If we define a -matrix $\mathbf{X}=\left[\mathbf{x}_1,\mathbf{x}_2,\dots, \mathbf{x}_p\right]$ -of dimension $n\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\mathbf{X}$, +matrix $\bm{X}=\left[\bm{x}_1,\bm{x}_2,\dots, \bm{x}_p\right]$ +of dimension $n\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\bm{X}$, !bt \[ -\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. +\bm{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. \] !et -If the above condition is not met for a given vector $\mathbf{x}_i$ we have +If the above condition is not met for a given vector $\bm{x}_i$ we have !bt \[ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0, \] !et if our output $y_i=1$. -In this case we say that $\mathbf{x}_i$ lies on one of the sides of the hyperplane and if +In this case we say that $\bm{x}_i$ lies on one of the sides of the hyperplane and if !bt \[ b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0, \] !et for the class of observations $y_i=-1$, -then $\mathbf{x}_i$ lies on the other side. +then $\bm{x}_i$ lies on the other side. Equivalently, for the two classes of observations we have !bt @@ -137,35 +137,35 @@ for our data sample. Let us define the function !bt \[ -f(x) = \mathbf{w}^T\mathbf{x}+b = 0, +f(x) = \bm{w}^T\bm{x}+b = 0, \] !et as the function that determines the line $L$ that separates two classes (our two features), see the figure here. -Any point defined by $\mathbf{x}_i$ and $\mathbf{x}_2$ on the line $L$ will satisfy $\mathbf{w}^T(\mathbf{x}_1-\mathbf{x}_2)=0$. +Any point defined by $\bm{x}_i$ and $\bm{x}_2$ on the line $L$ will satisfy $\bm{w}^T(\bm{x}_1-\bm{x}_2)=0$. -The signed distance $\delta$ from any point defined by a vector $\mathbf{x}$ and a point $\mathbf{x}_0$ on the line $L$ is then +The signed distance $\delta$ from any point defined by a vector $\bm{x}$ and a point $\bm{x}_0$ on the line $L$ is then !bt \[ -\delta = \frac{1}{\vert\vert \mathbf{w}\vert\vert}(\mathbf{w}^T\mathbf{x}+b). +\delta = \frac{1}{\vert\vert \bm{w}\vert\vert}(\bm{w}^T\bm{x}+b). \] !et !split ===== First attempt at a minimization approach ===== -How do we find the parameter $b$ and the vector $\mathbf{w}$? What we could +How do we find the parameter $b$ and the vector $\bm{w}$? What we could do is to define a cost function which now contains the set of all misclassified points $M$ and attempt to minimize this function !bt \[ -C(\mathbf{w},b) = -\sum_{i\in M} y_i(\mathbf{w}^T\mathbf{x}_i+b). +C(\bm{w},b) = -\sum_{i\in M} y_i(\bm{w}^T\bm{x}_i+b). \] !et -We could now for example define all values $y_i =1$ as misclassified in case we have $\mathbf{w}^T\mathbf{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us +We could now for example define all values $y_i =1$ as misclassified in case we have $\bm{w}^T\bm{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us !bt \[ \frac{\partial C}{\partial b} = -\sum_{i\in M} y_i, @@ -174,7 +174,7 @@ We could now for example define all values $y_i =1$ as misclassified in case we and !bt \[ -\frac{\partial C}{\partial \mathbf{w}} = -\sum_{i\in M} y_ix_i. +\frac{\partial C}{\partial \bm{w}} = -\sum_{i\in M} y_ix_i. \] !et @@ -190,7 +190,7 @@ b \leftarrow b +\eta \frac{\partial C}{\partial b}, and !bt \[ -\mathbf{w} \leftarrow \mathbf{w} +\eta \frac{\partial C}{\partial \mathbf{w}}, +\bm{w} \leftarrow \bm{w} +\eta \frac{\partial C}{\partial \bm{w}}, \] !et where $\eta$ is our by now well-known learning rate. @@ -210,12 +210,12 @@ at all. A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning). -Thus, we wish to find a margin $M$ with $\mathbf{w}$ normalized to -$\vert\vert \mathbf{w}\vert\vert =1$ subject to the condition +Thus, we wish to find a margin $M$ with $\bm{w}$ normalized to +$\vert\vert \bm{w}\vert\vert =1$ subject to the condition !bt \[ -y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. \] !et All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $b$ and $w_1$ and $w_2$ define this line. @@ -223,24 +223,24 @@ All points are thus at a signed distance from the decision boundary defined by t We seek thus the largest value $M$ defined by !bt \[ -\frac{1}{\vert \vert \mathbf{w}\vert\vert}y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, +\frac{1}{\vert \vert \bm{w}\vert\vert}y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, \] !et or just !bt \[ -y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M\vert \vert \mathbf{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\bm{w}^T\bm{x}_i+b) \geq M\vert \vert \bm{w}\vert\vert \hspace{0.1cm}\forall i. \] !et -If we scale the equation so that $\vert \vert \mathbf{w}\vert\vert = 1/M$, we have to find the minimum of -$\mathbf{w}^T\mathbf{w}=\vert \vert \mathbf{w}\vert\vert$ (the norm) subject to the condition +If we scale the equation so that $\vert \vert \bm{w}\vert\vert = 1/M$, we have to find the minimum of +$\bm{w}^T\bm{w}=\vert \vert \bm{w}\vert\vert$ (the norm) subject to the condition !bt \[ -y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +y_i(\bm{w}^T\bm{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. \] !et -We have thus defined our margin as the invers of the norm of $\mathbf{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers. +We have thus defined our margin as the invers of the norm of $\bm{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves about Lagrangian multipliers. !split ===== A quick reminder on Lagrangian multipliers ===== @@ -346,12 +346,12 @@ If we have a set of constraints $\phi_k$ we have the equations In order to solve the above problem, we define the following Lagrangian function to be minimized !bt \[ -{\cal L}(\lambda,b,\mathbf{w})=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-1\right], +{\cal L}(\lambda,b,\bm{w})=\frac{1}{2}\bm{w}^T\bm{w}-\sum_{i=1}^n\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b)-1\right], \] !et where $\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\lambda_i \geq 0$. -Taking the derivatives with respect to $b$ and $\mathbf{w}$ we obtain +Taking the derivatives with respect to $b$ and $\bm{w}$ we obtain !bt \[ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, @@ -360,25 +360,25 @@ Taking the derivatives with respect to $b$ and $\mathbf{w}$ we obtain and !bt \[ -\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i. +\frac{\partial {\cal L}}{\partial \bm{w}} = 0 = \bm{w}-\sum_{i} \lambda_iy_i\bm{x}_i. \] !et Inserting these constraints into the equation for ${\cal L}$ we obtain !bt \[ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{x}_j, \] !et subject to the constraints $\lambda_i\geq 0$ and $\sum_i\lambda_iy_i=0$. We must in addition satisfy the "Karush-Kuhn-Tucker":"https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" (KKT) condition !bt \[ -\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -1\right] \hspace{0.1cm}\forall i. +\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b) -1\right] \hspace{0.1cm}\forall i. \] !et -o If $\lambda_i > 0$, then $y_i(\mathbf{w}^T\mathbf{x}_i+b)=1$ and we say that $x_i$ is on the boundary. -o If $y_i(\mathbf{w}^T\mathbf{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$. -When $\lambda_i > 0$, the vectors $\mathbf{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. +o If $\lambda_i > 0$, then $y_i(\bm{w}^T\bm{x}_i+b)=1$ and we say that $x_i$ is on the boundary. +o If $y_i(\bm{w}^T\bm{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$. +When $\lambda_i > 0$, the vectors $\bm{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. !split ===== The problem to solve ===== @@ -386,22 +386,22 @@ When $\lambda_i > 0$, the vectors $\mathbf{x}_i$ are called support vectors. The We can rewrite !bt \[ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{x}_j, \] !et and its constraints in terms of a matrix-vector problem where we minimize w.r.t. $\lambda$ the following problem !bt \[ -\frac{1}{2} \mathbf{\lambda}^T\begin{bmatrix} y_1y_1\mathbf{x}_1^T\mathbf{x}_1 & y_1y_2\mathbf{x}_1^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_1^T\mathbf{x}_n \\ -y_2y_1\mathbf{x}_2^T\mathbf{x}_1 & y_2y_2\mathbf{x}_2^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_2^T\mathbf{x}_n \\ +\frac{1}{2} \bm{\lambda}^T\begin{bmatrix} y_1y_1\bm{x}_1^T\bm{x}_1 & y_1y_2\bm{x}_1^T\bm{x}_2 & \dots & \dots & y_1y_n\bm{x}_1^T\bm{x}_n \\ +y_2y_1\bm{x}_2^T\bm{x}_1 & y_2y_2\bm{x}_2^T\bm{x}_2 & \dots & \dots & y_1y_n\bm{x}_2^T\bm{x}_n \\ \dots & \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots & \dots \\ -y_ny_1\mathbf{x}_n^T\mathbf{x}_1 & y_ny_2\mathbf{x}_n^T\mathbf{x}_2 & \dots & \dots & y_ny_n\mathbf{x}_n^T\mathbf{x}_n \\ -\end{bmatrix}\mathbf{\lambda}-\mathbb{1}\mathbf{\lambda}, +y_ny_1\bm{x}_n^T\bm{x}_1 & y_ny_2\bm{x}_n^T\bm{x}_2 & \dots & \dots & y_ny_n\bm{x}_n^T\bm{x}_n \\ +\end{bmatrix}\bm{\lambda}-\mathbb{1}\bm{\lambda}, \] !et -subject to $\mathbf{y}^T\mathbf{\lambda}=0$. Here we defined the vectors $\mathbf{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and -$\mathbf{y}=[y_1,y_2,\dots,y_n]$. +subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and +$\bm{y}=[y_1,y_2,\dots,y_n]$. !split @@ -411,31 +411,31 @@ Solving the above problem, yields the values of $\lambda_i$. To find the coefficients of your hyperplane we need simply to compute !bt \[ -\mathbf{w}=\sum_{i} \lambda_iy_i\mathbf{x}_i. +\bm{w}=\sum_{i} \lambda_iy_i\bm{x}_i. \] !et -With our vector $\mathbf{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via +With our vector $\bm{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via !bt \[ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1, +y_i(\bm{w}^T\bm{x}_i+b)=1, \] !et resulting in !bt \[ -b = \frac{1}{y_i}-\mathbf{w}^T\mathbf{x}_i, +b = \frac{1}{y_i}-\bm{w}^T\bm{x}_i, \] !et or if we write it out in terms of the support vectors only, with $N_s$ being their number, we have !bt \[ -b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\mathbf{x}_i^T\mathbf{x}_j\right). +b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\bm{x}_i^T\bm{x}_j\right). \] !et With our hyperplane coefficients we can use our classifier to assign any observation by simply using !bt \[ -y_i = \mathrm{sign}(\mathbf{w}^T\mathbf{x}_i+b). +y_i = \mathrm{sign}(\bm{w}^T\bm{x}_i+b). \] !et Below we discuss how to find the optimal values of $\lambda_i$. Before we proceed however, we discuss now the so-called soft classifier. @@ -450,22 +450,22 @@ figure here. One way to deal with this problem before we define the so-called _kernel approach_, is to allow a kind of slack in the sense that we allow some points to be on the wrong side of the margin. -We introduce thus the so-called _slack_ variables $\mathbf{\xi} =[\xi_1,x_2,\dots,x_n]$ and +We introduce thus the so-called _slack_ variables $\bm{\xi} =[\xi_1,x_2,\dots,x_n]$ and modify our previous equation !bt \[ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1, +y_i(\bm{w}^T\bm{x}_i+b)=1, \] !et to !bt \[ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i, +y_i(\bm{w}^T\bm{x}_i+b)=1-\xi_i, \] !et with the requirement $\xi_i\geq 0$. The total violation is now $\sum_i\xi$. The value $\xi_i$ in the constraint the last constraint corresponds to the amount by which the prediction -$y_i(\mathbf{w}^T\mathbf{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$, +$y_i(\bm{w}^T\bm{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$, we bound the total amount by which predictions fall on the wrong side of their margins. Misclassifications occur when $\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of @@ -478,18 +478,18 @@ misclassifications. This has in turn the consequences that we change our optmization problem to finding the minimum of !bt \[ -{\cal L}=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, +{\cal L}=\frac{1}{2}\bm{w}^T\bm{w}-\sum_{i=1}^n\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, \] !et subject to !bt \[ -y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, +y_i(\bm{w}^T\bm{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, \] !et with the requirement $\xi_i\geq 0$. -Taking the derivatives with respect to $b$ and $\mathbf{w}$ we obtain +Taking the derivatives with respect to $b$ and $\bm{w}$ we obtain !bt \[ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, @@ -498,7 +498,7 @@ Taking the derivatives with respect to $b$ and $\mathbf{w}$ we obtain and !bt \[ -\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i, +\frac{\partial {\cal L}}{\partial \bm{w}} = 0 = \bm{w}-\sum_{i} \lambda_iy_i\bm{x}_i, \] !et and @@ -510,14 +510,14 @@ and Inserting these constraints into the equation for ${\cal L}$ we obtain the same equation as before !bt \[ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{x}_j, \] !et but now subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$ and $0\leq\lambda_i \leq C$. We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads !bt \[ -\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, +\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, \] !et !bt @@ -528,7 +528,7 @@ We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads and !bt \[ -y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. +y_i(\bm{w}^T\bm{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. \] !et