mathbf--> bm

doc/pub/svm/html/svm.html

@@ -169,10 +169,10 @@ $$
 
 where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line 
 \( b+w_1x_1+w_2x_2=0 \). 
-In two dimensions we define the vectors \( \mathbf{x} =[x1,x2] \) and \( \mathbf{w}=[w1,w2] \). 
+In two dimensions we define the vectors \( \boldsymbol{x} =[x1,x2] \) and \( \boldsymbol{w}=[w1,w2] \). 
 We can then rewrite the above equation as 
 $$
-\mathbf{w}^T\mathbf{x}+b=0.
+\boldsymbol{w}^T\boldsymbol{x}+b=0.
 $$
 
 <p>
@@ -188,25 +188,25 @@ b+wx_1+w_2x_2+\dots +w_px_p=0.
 $$
 
 If we define a 
-matrix \( \mathbf{X}=\left[\mathbf{x}_1,\mathbf{x}_2,\dots, \mathbf{x}_p\right] \)
-of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \mathbf{X} \), 
+matrix \( \boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right] \)
+of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldsymbol{X} \), 
 $$
-\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
+\boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}.
 $$
 
-If the above condition is not met for a given vector \( \mathbf{x}_i \) we have 
+If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have 
 $$
 b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} >0,
 $$
 
 if our output \( y_i=1 \).
-In this case we say that \( \mathbf{x}_i \) lies on one of the sides of the hyperplane and if 
+In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if 
 $$
 b+w_1x_{i1}+w_2x_{i}2+\dots +w_px_{ip} < 0,
 $$
 
 for the class of observations \( y_i=-1 \), 
-then \( \mathbf{x}_i \) lies on the other side.
+then \( \boldsymbol{x}_i \) lies on the other side.
 
 <p>
 Equivalently, for the two classes of observations we have 
@@ -252,18 +252,18 @@ for our data sample.
 <p>
 Let us define the function
 $$
-f(x) = \mathbf{w}^T\mathbf{x}+b = 0,
+f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0,
 $$
 
 as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
 
 <p>
-Any point defined by \( \mathbf{x}_i \) and \( \mathbf{x}_2 \) on the line \( L \) will satisfy \( \mathbf{w}^T(\mathbf{x}_1-\mathbf{x}_2)=0 \).
+Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).
 
 <p>
-The signed distance \( \delta \) from any point defined by a vector \( \mathbf{x} \) and a point \( \mathbf{x}_0 \) on the line \( L \) is then
+The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then
 $$
-\delta = \frac{1}{\vert\vert \mathbf{w}\vert\vert}(\mathbf{w}^T\mathbf{x}+b).
+\delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b).
 $$
 
 <p>
@@ -272,23 +272,23 @@ $$
 <h2 id="___sec6">First attempt at a minimization approach </h2>
 
 <p>
-How do we find the parameter \( b \) and the vector \( \mathbf{w} \)? What we could
+How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could
 do is to define a cost function which now contains the set of all
 misclassified points \( M \) and attempt to minimize this function
 
 $$
-C(\mathbf{w},b) = -\sum_{i\in M} y_i(\mathbf{w}^T\mathbf{x}_i+b).
+C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b).
 $$
 
 <p>
-We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \mathbf{w}^T\mathbf{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
+We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
 $$
 \frac{\partial C}{\partial b} = -\sum_{i\in M} y_i,
 $$
 
 and 
 $$
-\frac{\partial C}{\partial \mathbf{w}} = -\sum_{i\in M} y_ix_i.
+\frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i.
 $$
 
 <p>
@@ -304,7 +304,7 @@ $$
 
 and
 $$
-\mathbf{w} \leftarrow \mathbf{w} +\eta \frac{\partial C}{\partial \mathbf{w}},
+\boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}},
 $$
 
 where \( \eta \) is our by now well-known learning rate.
@@ -329,11 +329,11 @@ A better approach is rather to try to define a large margin between
 the two classes (if they are well separated from the beginning).
 
 <p>
-Thus, we wish to find a margin \( M \) with \( \mathbf{w} \) normalized to
-\( \vert\vert \mathbf{w}\vert\vert =1 \) subject to the condition
+Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to
+\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition
 
 $$
-y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. 
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. 
 $$
 
 All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.
@@ -341,22 +341,22 @@ All points are thus at a signed distance from the decision boundary defined by t
 <p>
 We seek thus the largest value \( M \) defined by
 $$
-\frac{1}{\vert \vert \mathbf{w}\vert\vert}y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, 
+\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, 
 $$
 
 or just 
 $$
-y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq M\vert \vert \mathbf{w}\vert\vert \hspace{0.1cm}\forall i. 
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i. 
 $$
 
-If we scale the equation so that \( \vert \vert \mathbf{w}\vert\vert = 1/M \), we have to find the minimum of 
-\( \mathbf{w}^T\mathbf{w}=\vert \vert \mathbf{w}\vert\vert \) (the norm) subject to the condition
+If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of 
+\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition
 $$
-y_i(\mathbf{w}^T\mathbf{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
 $$
 
 <p>
-We have thus defined our margin as the invers of the norm of \( \mathbf{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
+We have thus defined our margin as the invers of the norm of \( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves about Lagrangian multipliers.
 
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -457,40 +457,40 @@ $$
 <h2 id="___sec11">Setting up the problem </h2>
 In order to solve the above problem, we define the following Lagrangian function to be minimized 
 $$
-{\cal L}(\lambda,b,\mathbf{w})=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-1\right],
+{\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right],
 $$
 
 where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
 
 <p>
-Taking the derivatives  with respect to \( b \) and \( \mathbf{w} \) we obtain 
+Taking the derivatives  with respect to \( b \) and \( \boldsymbol{w} \) we obtain 
 $$
 \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
 $$
 
 and 
 $$
-\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i.
+\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
 $$
 
 Inserting these constraints into the equation for \( {\cal L} \) we obtain
 $$
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
 $$
 
 subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). 
 We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_blank">Karush-Kuhn-Tucker</a> (KKT) condition
 $$
-\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
+\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
 $$
 
 
 <ol>
-<li> If \( \lambda_i > 0 \), then \( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
-<li> If \( y_i(\mathbf{w}^T\mathbf{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li> 
+<li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
+<li> If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li> 
 </ol>
 
-When \( \lambda_i > 0 \), the vectors \( \mathbf{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
+When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
 
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -500,21 +500,21 @@ When \( \lambda_i > 0 \), the vectors \( \mathbf{x}_i \) are called support vect
 <p>
 We can rewrite 
 $$
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
 $$
 
 and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
 $$
-\frac{1}{2} \mathbf{\lambda}^T\begin{bmatrix} y_1y_1\mathbf{x}_1^T\mathbf{x}_1 & y_1y_2\mathbf{x}_1^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_1^T\mathbf{x}_n \\
-y_2y_1\mathbf{x}_2^T\mathbf{x}_1 & y_2y_2\mathbf{x}_2^T\mathbf{x}_2 & \dots & \dots & y_1y_n\mathbf{x}_2^T\mathbf{x}_n \\
+\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\
+y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
 \dots & \dots & \dots & \dots & \dots \\
 \dots & \dots & \dots & \dots & \dots \\
-y_ny_1\mathbf{x}_n^T\mathbf{x}_1 & y_ny_2\mathbf{x}_n^T\mathbf{x}_2 & \dots & \dots & y_ny_n\mathbf{x}_n^T\mathbf{x}_n \\
-\end{bmatrix}\mathbf{\lambda}-\mathbb{1}\mathbf{\lambda}, 
+y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 & \dots & \dots & y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\
+\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, 
 $$
 
-subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \( \mathbf{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and 
-\( \mathbf{y}=[y_1,y_2,\dots,y_n] \).
+subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and 
+\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
 
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -525,27 +525,27 @@ subject to \( \mathbf{y}^T\mathbf{\lambda}=0 \). Here we defined the vectors \(
 Solving the above problem, yields the values of \( \lambda_i \).
 To find the coefficients of your hyperplane we need simply to compute 
 $$
-\mathbf{w}=\sum_{i} \lambda_iy_i\mathbf{x}_i.
+\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
 $$
 
-With our vector \( \mathbf{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via 
+With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via 
 $$
-y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
 $$
 
 resulting in
 $$
-b = \frac{1}{y_i}-\mathbf{w}^T\mathbf{x}_i,
+b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
 $$
 
 or if we write it out in terms of the support vectors only, with \( N_s \) being their number,  we have
 $$
-b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\mathbf{x}_i^T\mathbf{x}_j\right).
+b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right).
 $$
 
 With our hyperplane coefficients we can use our classifier to assign any observation by simply using 
 $$
-y_i = \mathrm{sign}(\mathbf{w}^T\mathbf{x}_i+b).
+y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
 $$
 
 Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
@@ -565,20 +565,20 @@ so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
 that we allow some points to be on the wrong side of the margin.
 
 <p>
-We introduce thus the so-called <b>slack</b> variables \( \mathbf{\xi} =[\xi_1,x_2,\dots,x_n] \) and 
+We introduce thus the so-called <b>slack</b> variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and 
 modify our previous equation
 $$
-y_i(\mathbf{w}^T\mathbf{x}_i+b)=1,
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
 $$
 
 to 
 $$
-y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i,
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i,
 $$
 
 with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \). 
 The value \( \xi_i \) in the constraint the last constraint corresponds to the  amount by which the prediction
-\( y_i(\mathbf{w}^T\mathbf{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
+\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
 we bound the total amount by which predictions fall on the wrong side of their margins.
 
 <p>
@@ -593,25 +593,25 @@ misclassifications.
 <p>
 This has in turn the consequences that we change our optmization problem to finding the minimum of 
 $$
-{\cal L}=\frac{1}{2}\mathbf{w}^T\mathbf{w}-\sum_{i=1}^n\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
+{\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
 $$
 
 subject to 
 $$
-y_i(\mathbf{w}^T\mathbf{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
 $$
 
 with the requirement \( \xi_i\geq 0 \).
 
 <p>
-Taking the derivatives  with respect to \( b \) and \( \mathbf{w} \) we obtain 
+Taking the derivatives  with respect to \( b \) and \( \boldsymbol{w} \) we obtain 
 $$
 \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
 $$
 
 and 
 $$
-\frac{\partial {\cal L}}{\partial \mathbf{w}} = 0 = \mathbf{w}-\sum_{i} \lambda_iy_i\mathbf{x}_i, 
+\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i, 
 $$
 
 and
@@ -621,13 +621,13 @@ $$
 
 Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before
 $$
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\mathbf{x}_i^T\mathbf{x}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
 $$
 
 but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \). 
 We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
 $$
-\lambda_i\left[y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
+\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
 $$
 
 $$
@@ -636,7 +636,7 @@ $$
 
 and 
 $$
-y_i(\mathbf{w}^T\mathbf{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
+y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
 $$
 
 <p>