adding more equations to sum part

doc/src/SupportVMachines/svm.do.txt

@@ -115,7 +115,73 @@ Let us define the function
 f(x) = \beta_0+\beta_1x = 0,
 \]
 !et
-as the function that determines the line that separates two classes (our two features). 
+as the function that determines the line $L$ that separates two classes (our two features), see the figur here. 
+
+Define a vector $\hat{\beta}:\left\{\beta_0,\beta_1\right\}$. Let us label the values of $\hat{\beta}$ that satisfy this constraint as $\overline{\beta}$. 
+
+Any two points $x_1$ and $x_2$ on the line $L$ will satisfy $\hat{\beta}(x_1-x_2)=0$. We normalize the solution and define
+!bt
+\[
+\overline{\beta} = \frac{\hat{\beta}}{\vert\vert \hat{\beta}\vert\vert},
+\]
+!et
+which is vector normal to the line $L$. 
+
+The signed distance from a point $x_0$ on $L$ to any point $x$ is then
+!bt
+\[
+\overline{\beta}(x-x_0) = \frac{\beta_1 x + \beta_0}{\vert\vert \hat{\beta}\vert\vert}.
+\]
+!et
+
+!split
+===== First attempt at a minimization approach =====
+
+How do we find the parameters $\beta_0$ and $\beta_0$? What we could
+do is to define a cost function which now contains the set of all
+misclassified points $M$ and attempt to minimize this function
+
+!bt
+\[
+C(\beta_0,\beta_1) = -\sum_{i\in M} y_i(\beta_1x_1+\beta_0).
+\]
+!et
+
+We could now for example define all values $y_i =1$ as misclassified in case we have $\beta_1x_i+\beta_0 < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us
+!bt
+\[
+\frac{\partial C}{\partial \beta_0} = -\sum_{i\in M} y_i,
+\]
+!et
+and 
+!bt
+\[
+\frac{\partial C}{\partial \beta_1} = -\sum_{i\in M} y_ix_i.
+\]
+!et
+
+!split
+===== Solving the equations =====
+
+We can now use the Newton-Raphson method or gradient descent to solve the equations
+!bt
+\[
+\beta_0 \leftarrow \beta_0 +\eta \frac{\partial C}{\partial \beta_0},
+\]
+!et
+and
+!bt
+\[
+\beta_1 \leftarrow \beta_1 +\eta \frac{\partial C}{\partial \beta_1},
+\]
+!et
+where $\eta$ is our by now well-known learning rate. 
+There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.
+
+!split
+===== A better approach =====
+A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning). 
+
 !split
 ===== Examples with kernels =====