updating sim, not yet ready

doc/src/SupportVMachines/svm.do.txt

@@ -180,8 +180,78 @@ There are however problems with this approach, although it looks pretty straight
 
 !split
 ===== A better approach =====
+
 A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning). 
 
+Thus, we wish to find a margin $M$ with $\hat{\beta}$ normalized to $\vert\vert \hat{\beta}\vert\vert =1$ subject to the condition 
+!bt
+\[
+y_i(\beta_0+\beta_1 x_i) \geq M \forall i=1,2,\dots, n. 
+\]
+!et
+All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $\beta_0$ and $\beta_1$ define this line. 
+
+We seek thus the largest value $M$ defined by
+!bt
+\[
+\frac{1}{\vert \vert \hat{\beta}\vert\vert}y_i(\beta_0+\beta_1 x_) \geq M \forall i=1,2,\dots, n, 
+\]
+!et
+or just 
+!bt
+\[
+y_i(\beta_0+\beta_1 x_i) \geq M\vert \vert \hat{\beta}\vert\vert \forall i=1,2,\dots, n. 
+\]
+!et
+If we scale the equation so that $\vert \vert \hat{\beta}\vert\vert = 1/M$, we have to find the minimum of 
+$\hat{\beta}^T\hat{\beta}$ subject to the condition
+!bt
+\[
+y_i(\beta_0+\beta_1 x_i) \geq 1 \forall i=1,2,\dots, n. 
+\]
+!et
+
+!split
+===== A quick reminder on Lagrangian multipliers =====
+
+Withe the above constraint, we introduce now the calculus of Lagrangian multiplier. 
+In order to solve the above problem, we define the following Lagrangian function to be minimized 
+!bt
+\[
+L(\lambda)=\frac{1}{2}\hat{\beta}^T\hat{\beta}-\sum_{i=1}^n\lambda_i\left[y_i(\beta_0+\beta_1 x_i)-1\right],
+\]
+!et
+where $\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\lambda_i \geq 0$.
+
+Taking the derivatives  with respect to $\beta_0$ and $\beta_1$ we obtain 
+!bt
+\[
+\frac{\partial L}{\partial \beta_0} = -\sum_{i} \lambda_iy_i=0,
+\]
+!et
+and 
+!bt
+\[
+\frac{\partial L}{\partial \beta_1} = \beta_1-\sum_{i} \lambda_iy_ix_i=0.
+\]
+!et
+Inserting these constraints into the equation for $L$ we obtain
+!bt
+\[
+L(\lambda)=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_jx_i^Tx_j,
+\]
+!et
+subject to the constraints $\lambda_i\geq 0$ and $\sum_i\lambda_iy_i=0$. 
+We must in addition satisfy the Koriush-Kuhn-Tucker (KKT) condition
+!bt
+\[
+\lambda_i\left[y_i(\beta_0+\beta_1 x_i) -1\right] \forall i=1,2,\dots, n. 
+\]
+!et
+o If $\lambda_i > 0$, then $y_i(\beta_0+\beta_1 x_i)=1$ and we say that $x_i$ is on the boundary of the slab.
+o If $y_i(\beta_0+\beta_1 x_i)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$. 
+
+
 !split
 ===== Examples with kernels =====