A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning). +
+Thus, we wish to find a margin \( M \) with \( \hat{\beta} \) normalized to \( \vert\vert \hat{\beta}\vert\vert =1 \) subject to the condition +$$ +y_i(\beta_0+\beta_1 x_i) \geq M \forall i=1,2,\dots, n. +$$ + +All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( \beta_0 \) and \( \beta_1 \) define this line. + +
+We seek thus the largest value \( M \) defined by +$$ +\frac{1}{\vert \vert \hat{\beta}\vert\vert}y_i(\beta_0+\beta_1 x_) \geq M \forall i=1,2,\dots, n, +$$ + +or just +$$ +y_i(\beta_0+\beta_1 x_i) \geq M\vert \vert \hat{\beta}\vert\vert \forall i=1,2,\dots, n. +$$ + +If we scale the equation so that \( \vert \vert \hat{\beta}\vert\vert = 1/M \), we have to find the minimum of +\( \hat{\beta}^T\hat{\beta} \) subject to the condition +$$ +y_i(\beta_0+\beta_1 x_i) \geq 1 \forall i=1,2,\dots, n. +$$ +
-
+Withe the above constraint, we introduce now the calculus of Lagrangian multiplier. +In order to solve the above problem, we define the following Lagrangian function to be minimized +$$ +L(\lambda)=\frac{1}{2}\hat{\beta}^T\hat{\beta}-\sum_{i=1}^n\lambda_i\left[y_i(\beta_0+\beta_1 x_i)-1\right], +$$ + +where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \). + +
+Taking the derivatives with respect to \( \beta_0 \) and \( \beta_1 \) we obtain +$$ +\frac{\partial L}{\partial \beta_0} = -\sum_{i} \lambda_iy_i=0, +$$ + +and +$$ +\frac{\partial L}{\partial \beta_1} = \beta_1-\sum_{i} \lambda_iy_ix_i=0. +$$ + +Inserting these constraints into the equation for \( L \) we obtain +$$ +L(\lambda)=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_jx_i^Tx_j, +$$ + +subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). +We must in addition satisfy the Koriush-Kuhn-Tucker (KKT) condition +$$ +\lambda_i\left[y_i(\beta_0+\beta_1 x_i) -1\right] \forall i=1,2,\dots, n. +$$ + + +
diff --git a/doc/pub/svm/html/svm-reveal.html b/doc/pub/svm/html/svm-reveal.html
index addb45114..11fba8a0b 100644
--- a/doc/pub/svm/html/svm-reveal.html
+++ b/doc/pub/svm/html/svm-reveal.html
@@ -378,12 +378,99 @@ There are however problems with this approach, although it looks pretty straight
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
+
+
+Thus, we wish to find a margin \( M \) with \( \hat{\beta} \) normalized to \( \vert\vert \hat{\beta}\vert\vert =1 \) subject to the condition
+
+We seek thus the largest value \( M \) defined by
+
+Withe the above constraint, we introduce now the calculus of Lagrangian multiplier.
+In order to solve the above problem, we define the following Lagrangian function to be minimized
+
+Taking the derivatives with respect to \( \beta_0 \) and \( \beta_1 \) we obtain
+
diff --git a/doc/pub/svm/html/svm-solarized.html b/doc/pub/svm/html/svm-solarized.html
index 69804bbc9..04539f684 100644
--- a/doc/pub/svm/html/svm-solarized.html
+++ b/doc/pub/svm/html/svm-solarized.html
@@ -43,7 +43,11 @@ div { text-align: justify; text-justify: inter-word; }
('First attempt at a minimization approach', 2, None, '___sec6'),
('Solving the equations', 2, None, '___sec7'),
('A better approach', 2, None, '___sec8'),
- ('Examples with kernels', 2, None, '___sec9')]}
+ ('A quick reminder on Lagrangian multipliers',
+ 2,
+ None,
+ '___sec9'),
+ ('Examples with kernels', 2, None, '___sec10')]}
end of tocinfo -->
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
+
+Thus, we wish to find a margin \( M \) with \( \hat{\beta} \) normalized to \( \vert\vert \hat{\beta}\vert\vert =1 \) subject to the condition
+$$
+y_i(\beta_0+\beta_1 x_i) \geq M \forall i=1,2,\dots, n.
+$$
+
+All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( \beta_0 \) and \( \beta_1 \) define this line.
+
+
+We seek thus the largest value \( M \) defined by
+$$
+\frac{1}{\vert \vert \hat{\beta}\vert\vert}y_i(\beta_0+\beta_1 x_) \geq M \forall i=1,2,\dots, n,
+$$
+
+or just
+$$
+y_i(\beta_0+\beta_1 x_i) \geq M\vert \vert \hat{\beta}\vert\vert \forall i=1,2,\dots, n.
+$$
+
+If we scale the equation so that \( \vert \vert \hat{\beta}\vert\vert = 1/M \), we have to find the minimum of
+\( \hat{\beta}^T\hat{\beta} \) subject to the condition
+$$
+y_i(\beta_0+\beta_1 x_i) \geq 1 \forall i=1,2,\dots, n.
+$$
+
+Withe the above constraint, we introduce now the calculus of Lagrangian multiplier.
+In order to solve the above problem, we define the following Lagrangian function to be minimized
+$$
+L(\lambda)=\frac{1}{2}\hat{\beta}^T\hat{\beta}-\sum_{i=1}^n\lambda_i\left[y_i(\beta_0+\beta_1 x_i)-1\right],
+$$
+
+where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
+
+
+Taking the derivatives with respect to \( \beta_0 \) and \( \beta_1 \) we obtain
+$$
+\frac{\partial L}{\partial \beta_0} = -\sum_{i} \lambda_iy_i=0,
+$$
+
+and
+$$
+\frac{\partial L}{\partial \beta_1} = \beta_1-\sum_{i} \lambda_iy_ix_i=0.
+$$
+
+Inserting these constraints into the equation for \( L \) we obtain
+$$
+L(\lambda)=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_jx_i^Tx_j,
+$$
+
+subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
+We must in addition satisfy the Koriush-Kuhn-Tucker (KKT) condition
+$$
+\lambda_i\left[y_i(\beta_0+\beta_1 x_i) -1\right] \forall i=1,2,\dots, n.
+$$
+
+
+
diff --git a/doc/pub/svm/html/svm.html b/doc/pub/svm/html/svm.html
index f5fb48306..19369e449 100644
--- a/doc/pub/svm/html/svm.html
+++ b/doc/pub/svm/html/svm.html
@@ -48,7 +48,11 @@ div { text-align: justify; text-justify: inter-word; }
('First attempt at a minimization approach', 2, None, '___sec6'),
('Solving the equations', 2, None, '___sec7'),
('A better approach', 2, None, '___sec8'),
- ('Examples with kernels', 2, None, '___sec9')]}
+ ('A quick reminder on Lagrangian multipliers',
+ 2,
+ None,
+ '___sec9'),
+ ('Examples with kernels', 2, None, '___sec10')]}
end of tocinfo -->
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
+
+Thus, we wish to find a margin \( M \) with \( \hat{\beta} \) normalized to \( \vert\vert \hat{\beta}\vert\vert =1 \) subject to the condition
+$$
+y_i(\beta_0+\beta_1 x_i) \geq M \forall i=1,2,\dots, n.
+$$
+
+All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( \beta_0 \) and \( \beta_1 \) define this line.
+
+
+We seek thus the largest value \( M \) defined by
+$$
+\frac{1}{\vert \vert \hat{\beta}\vert\vert}y_i(\beta_0+\beta_1 x_) \geq M \forall i=1,2,\dots, n,
+$$
+
+or just
+$$
+y_i(\beta_0+\beta_1 x_i) \geq M\vert \vert \hat{\beta}\vert\vert \forall i=1,2,\dots, n.
+$$
+
+If we scale the equation so that \( \vert \vert \hat{\beta}\vert\vert = 1/M \), we have to find the minimum of
+\( \hat{\beta}^T\hat{\beta} \) subject to the condition
+$$
+y_i(\beta_0+\beta_1 x_i) \geq 1 \forall i=1,2,\dots, n.
+$$
+
+Withe the above constraint, we introduce now the calculus of Lagrangian multiplier.
+In order to solve the above problem, we define the following Lagrangian function to be minimized
+$$
+L(\lambda)=\frac{1}{2}\hat{\beta}^T\hat{\beta}-\sum_{i=1}^n\lambda_i\left[y_i(\beta_0+\beta_1 x_i)-1\right],
+$$
+
+where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
+
+
+Taking the derivatives with respect to \( \beta_0 \) and \( \beta_1 \) we obtain
+$$
+\frac{\partial L}{\partial \beta_0} = -\sum_{i} \lambda_iy_i=0,
+$$
+
+and
+$$
+\frac{\partial L}{\partial \beta_1} = \beta_1-\sum_{i} \lambda_iy_ix_i=0.
+$$
+
+Inserting these constraints into the equation for \( L \) we obtain
+$$
+L(\lambda)=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_jx_i^Tx_j,
+$$
+
+subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
+We must in addition satisfy the Koriush-Kuhn-Tucker (KKT) condition
+$$
+\lambda_i\left[y_i(\beta_0+\beta_1 x_i) -1\right] \forall i=1,2,\dots, n.
+$$
+
+
+
diff --git a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz
index 8f47939d0..dccb72d8f 100644
Binary files a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz and b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz differ
diff --git a/doc/pub/svm/ipynb/svm.ipynb b/doc/pub/svm/ipynb/svm.ipynb
index a63b10e6f..a8ef7ff52 100644
--- a/doc/pub/svm/ipynb/svm.ipynb
+++ b/doc/pub/svm/ipynb/svm.ipynb
@@ -300,8 +300,166 @@
"There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.\n",
"\n",
"## A better approach\n",
+ "\n",
"A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning). \n",
"\n",
+ "Thus, we wish to find a margin $M$ with $\\hat{\\beta}$ normalized to $\\vert\\vert \\hat{\\beta}\\vert\\vert =1$ subject to the condition"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "y_i(\\beta_0+\\beta_1 x_i) \\geq M \\forall i=1,2,\\dots, n.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $\\beta_0$ and $\\beta_1$ define this line. \n",
+ "\n",
+ "We seek thus the largest value $M$ defined by"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\frac{1}{\\vert \\vert \\hat{\\beta}\\vert\\vert}y_i(\\beta_0+\\beta_1 x_) \\geq M \\forall i=1,2,\\dots, n,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "or just"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "y_i(\\beta_0+\\beta_1 x_i) \\geq M\\vert \\vert \\hat{\\beta}\\vert\\vert \\forall i=1,2,\\dots, n.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If we scale the equation so that $\\vert \\vert \\hat{\\beta}\\vert\\vert = 1/M$, we have to find the minimum of \n",
+ "$\\hat{\\beta}^T\\hat{\\beta}$ subject to the condition"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "y_i(\\beta_0+\\beta_1 x_i) \\geq 1 \\forall i=1,2,\\dots, n.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## A quick reminder on Lagrangian multipliers\n",
+ "\n",
+ "Withe the above constraint, we introduce now the calculus of Lagrangian multiplier. \n",
+ "In order to solve the above problem, we define the following Lagrangian function to be minimized"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "L(\\lambda)=\\frac{1}{2}\\hat{\\beta}^T\\hat{\\beta}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\beta_0+\\beta_1 x_i)-1\\right],\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "where $\\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\\lambda_i \\geq 0$.\n",
+ "\n",
+ "Taking the derivatives with respect to $\\beta_0$ and $\\beta_1$ we obtain"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\frac{\\partial L}{\\partial \\beta_0} = -\\sum_{i} \\lambda_iy_i=0,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\frac{\\partial L}{\\partial \\beta_1} = \\beta_1-\\sum_{i} \\lambda_iy_ix_i=0.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Inserting these constraints into the equation for $L$ we obtain"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "L(\\lambda)=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_jx_i^Tx_j,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "subject to the constraints $\\lambda_i\\geq 0$ and $\\sum_i\\lambda_iy_i=0$. \n",
+ "We must in addition satisfy the Koriush-Kuhn-Tucker (KKT) condition"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\lambda_i\\left[y_i(\\beta_0+\\beta_1 x_i) -1\\right] \\forall i=1,2,\\dots, n.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "1. If $\\lambda_i > 0$, then $y_i(\\beta_0+\\beta_1 x_i)=1$ and we say that $x_i$ is on the boundary of the slab.\n",
+ "\n",
+ "2. If $y_i(\\beta_0+\\beta_1 x_i)> 1$, we say $x_i$ is not on the boundary and we set $\\lambda_i=0$. \n",
+ "\n",
"## Examples with kernels"
]
},
diff --git a/doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf b/doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf
index ec79cbc31..c7da9db4d 100644
Binary files a/doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf and b/doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf differ
diff --git a/doc/pub/svm/pdf/svm-beamer.pdf b/doc/pub/svm/pdf/svm-beamer.pdf
index 9bade2271..65fe378db 100644
Binary files a/doc/pub/svm/pdf/svm-beamer.pdf and b/doc/pub/svm/pdf/svm-beamer.pdf differ
diff --git a/doc/pub/svm/pdf/svm-minted.pdf b/doc/pub/svm/pdf/svm-minted.pdf
index 3f667c2ab..c4f08c358 100644
Binary files a/doc/pub/svm/pdf/svm-minted.pdf and b/doc/pub/svm/pdf/svm-minted.pdf differ
diff --git a/doc/src/SupportVMachines/svm.do.txt b/doc/src/SupportVMachines/svm.do.txt
index 29fd69633..5db82cef4 100644
--- a/doc/src/SupportVMachines/svm.do.txt
+++ b/doc/src/SupportVMachines/svm.do.txt
@@ -180,8 +180,78 @@ There are however problems with this approach, although it looks pretty straight
!split
===== A better approach =====
+
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
+Thus, we wish to find a margin $M$ with $\hat{\beta}$ normalized to $\vert\vert \hat{\beta}\vert\vert =1$ subject to the condition
+!bt
+\[
+y_i(\beta_0+\beta_1 x_i) \geq M \forall i=1,2,\dots, n.
+\]
+!et
+All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $\beta_0$ and $\beta_1$ define this line.
+
+We seek thus the largest value $M$ defined by
+!bt
+\[
+\frac{1}{\vert \vert \hat{\beta}\vert\vert}y_i(\beta_0+\beta_1 x_) \geq M \forall i=1,2,\dots, n,
+\]
+!et
+or just
+!bt
+\[
+y_i(\beta_0+\beta_1 x_i) \geq M\vert \vert \hat{\beta}\vert\vert \forall i=1,2,\dots, n.
+\]
+!et
+If we scale the equation so that $\vert \vert \hat{\beta}\vert\vert = 1/M$, we have to find the minimum of
+$\hat{\beta}^T\hat{\beta}$ subject to the condition
+!bt
+\[
+y_i(\beta_0+\beta_1 x_i) \geq 1 \forall i=1,2,\dots, n.
+\]
+!et
+
+!split
+===== A quick reminder on Lagrangian multipliers =====
+
+Withe the above constraint, we introduce now the calculus of Lagrangian multiplier.
+In order to solve the above problem, we define the following Lagrangian function to be minimized
+!bt
+\[
+L(\lambda)=\frac{1}{2}\hat{\beta}^T\hat{\beta}-\sum_{i=1}^n\lambda_i\left[y_i(\beta_0+\beta_1 x_i)-1\right],
+\]
+!et
+where $\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\lambda_i \geq 0$.
+
+Taking the derivatives with respect to $\beta_0$ and $\beta_1$ we obtain
+!bt
+\[
+\frac{\partial L}{\partial \beta_0} = -\sum_{i} \lambda_iy_i=0,
+\]
+!et
+and
+!bt
+\[
+\frac{\partial L}{\partial \beta_1} = \beta_1-\sum_{i} \lambda_iy_ix_i=0.
+\]
+!et
+Inserting these constraints into the equation for $L$ we obtain
+!bt
+\[
+L(\lambda)=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_jx_i^Tx_j,
+\]
+!et
+subject to the constraints $\lambda_i\geq 0$ and $\sum_i\lambda_iy_i=0$.
+We must in addition satisfy the Koriush-Kuhn-Tucker (KKT) condition
+!bt
+\[
+\lambda_i\left[y_i(\beta_0+\beta_1 x_i) -1\right] \forall i=1,2,\dots, n.
+\]
+!et
+o If $\lambda_i > 0$, then $y_i(\beta_0+\beta_1 x_i)=1$ and we say that $x_i$ is on the boundary of the slab.
+o If $y_i(\beta_0+\beta_1 x_i)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$.
+
+
!split
===== Examples with kernels =====
A better approach
+
+
+$$
+y_i(\beta_0+\beta_1 x_i) \geq M \forall i=1,2,\dots, n.
+$$
+
+
+All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( \beta_0 \) and \( \beta_1 \) define this line.
+
+
+$$
+\frac{1}{\vert \vert \hat{\beta}\vert\vert}y_i(\beta_0+\beta_1 x_) \geq M \forall i=1,2,\dots, n,
+$$
+
+
+or just
+
+$$
+y_i(\beta_0+\beta_1 x_i) \geq M\vert \vert \hat{\beta}\vert\vert \forall i=1,2,\dots, n.
+$$
+
+
+If we scale the equation so that \( \vert \vert \hat{\beta}\vert\vert = 1/M \), we have to find the minimum of
+\( \hat{\beta}^T\hat{\beta} \) subject to the condition
+
+$$
+y_i(\beta_0+\beta_1 x_i) \geq 1 \forall i=1,2,\dots, n.
+$$
+
Examples with kernels
+A quick reminder on Lagrangian multipliers
+
+
+$$
+L(\lambda)=\frac{1}{2}\hat{\beta}^T\hat{\beta}-\sum_{i=1}^n\lambda_i\left[y_i(\beta_0+\beta_1 x_i)-1\right],
+$$
+
+
+where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
+
+
+$$
+\frac{\partial L}{\partial \beta_0} = -\sum_{i} \lambda_iy_i=0,
+$$
+
+
+and
+
+$$
+\frac{\partial L}{\partial \beta_1} = \beta_1-\sum_{i} \lambda_iy_ix_i=0.
+$$
+
+
+Inserting these constraints into the equation for \( L \) we obtain
+
+$$
+L(\lambda)=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_jx_i^Tx_j,
+$$
+
+
+subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
+We must in addition satisfy the Koriush-Kuhn-Tucker (KKT) condition
+
+$$
+\lambda_i\left[y_i(\beta_0+\beta_1 x_i) -1\right] \forall i=1,2,\dots, n.
+$$
+
+
+
+
+
+Examples with kernels
A better approach
+
+
-Examples with kernels
+A quick reminder on Lagrangian multipliers
+
+
+
+
+
+
+Examples with kernels
A better approach
+
+
-Examples with kernels
+A quick reminder on Lagrangian multipliers
+
+
+
+
+
+
+Examples with kernels