lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating sim, not yet ready

Browse Source
This commit is contained in:
mhjensen
2018-11-04 21:43:08 +01:00
parent efa30bcf9e
commit ae4454f82c
10 changed files with 540 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/svm/html/svm-bs.html
+76 -3
View File
@@ -49,7 +49,11 @@ Automatically generated HTML file from DocOnce source
('First attempt at a minimization approach', 2, None, '___sec6'),
('Solving the equations', 2, None, '___sec7'),
('A better approach', 2, None, '___sec8'),
('Examples with kernels', 2, None, '___sec9')]}
('A quick reminder on Lagrangian multipliers',
2,
None,
'___sec9'),
('Examples with kernels', 2, None, '___sec10')]}
end of tocinfo -->
<body>
@@ -96,7 +100,8 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;">First attempt at a minimization approach</a></li>
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;">Solving the equations</a></li>
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;">A better approach</a></li>
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;">Examples with kernels</a></li>
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;">A quick reminder on Lagrangian multipliers</a></li>
<!-- navigation toc: --> <li><a href="#___sec10" style="font-size: 80%;">Examples with kernels</a></li>
</ul>
</li>
@@ -333,12 +338,80 @@ There are however problems with this approach, although it looks pretty straight
<!-- !split -->
<h2 id="___sec8" class="anchor">A better approach </h2>
<p>
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
<p>
Thus, we wish to find a margin \( M \) with \( \hat{\beta} \) normalized to \( \vert\vert \hat{\beta}\vert\vert =1 \) subject to the condition
$$
y_i(\beta_0+\beta_1 x_i) \geq M \forall i=1,2,\dots, n.
$$
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( \beta_0 \) and \( \beta_1 \) define this line.
<p>
We seek thus the largest value \( M \) defined by
$$
\frac{1}{\vert \vert \hat{\beta}\vert\vert}y_i(\beta_0+\beta_1 x_) \geq M \forall i=1,2,\dots, n,
$$
or just
$$
y_i(\beta_0+\beta_1 x_i) \geq M\vert \vert \hat{\beta}\vert\vert \forall i=1,2,\dots, n.
$$
If we scale the equation so that \( \vert \vert \hat{\beta}\vert\vert = 1/M \), we have to find the minimum of
\( \hat{\beta}^T\hat{\beta} \) subject to the condition
$$
y_i(\beta_0+\beta_1 x_i) \geq 1 \forall i=1,2,\dots, n.
$$
<p>
<!-- !split -->
<h2 id="___sec9" class="anchor">Examples with kernels </h2>
<h2 id="___sec9" class="anchor">A quick reminder on Lagrangian multipliers </h2>
<p>
Withe the above constraint, we introduce now the calculus of Lagrangian multiplier.
In order to solve the above problem, we define the following Lagrangian function to be minimized
$$
L(\lambda)=\frac{1}{2}\hat{\beta}^T\hat{\beta}-\sum_{i=1}^n\lambda_i\left[y_i(\beta_0+\beta_1 x_i)-1\right],
$$
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
<p>
Taking the derivatives with respect to \( \beta_0 \) and \( \beta_1 \) we obtain
$$
\frac{\partial L}{\partial \beta_0} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial L}{\partial \beta_1} = \beta_1-\sum_{i} \lambda_iy_ix_i=0.
$$
Inserting these constraints into the equation for \( L \) we obtain
$$
L(\lambda)=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_jx_i^Tx_j,
$$
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the Koriush-Kuhn-Tucker (KKT) condition
$$
\lambda_i\left[y_i(\beta_0+\beta_1 x_i) -1\right] \forall i=1,2,\dots, n.
$$
<ol>
<li> If \( \lambda_i > 0 \), then \( y_i(\beta_0+\beta_1 x_i)=1 \) and we say that \( x_i \) is on the boundary of the slab.</li>
<li> If \( y_i(\beta_0+\beta_1 x_i)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
<!-- !split -->
<h2 id="___sec10" class="anchor">Examples with kernels </h2>
<p>
doc/pub/svm/html/svm-reveal.html
+88 -1
View File
@@ -378,12 +378,99 @@ There are however problems with this approach, although it looks pretty straight
<section>
<h2 id="___sec8">A better approach </h2>
<p>
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
<p>
Thus, we wish to find a margin \( M \) with \( \hat{\beta} \) normalized to \( \vert\vert \hat{\beta}\vert\vert =1 \) subject to the condition
<p>&nbsp;<br>
$$
y_i(\beta_0+\beta_1 x_i) \geq M \forall i=1,2,\dots, n.
$$
<p>&nbsp;<br>
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( \beta_0 \) and \( \beta_1 \) define this line.
<p>
We seek thus the largest value \( M \) defined by
<p>&nbsp;<br>
$$
\frac{1}{\vert \vert \hat{\beta}\vert\vert}y_i(\beta_0+\beta_1 x_) \geq M \forall i=1,2,\dots, n,
$$
<p>&nbsp;<br>
or just
<p>&nbsp;<br>
$$
y_i(\beta_0+\beta_1 x_i) \geq M\vert \vert \hat{\beta}\vert\vert \forall i=1,2,\dots, n.
$$
<p>&nbsp;<br>
If we scale the equation so that \( \vert \vert \hat{\beta}\vert\vert = 1/M \), we have to find the minimum of
\( \hat{\beta}^T\hat{\beta} \) subject to the condition
<p>&nbsp;<br>
$$
y_i(\beta_0+\beta_1 x_i) \geq 1 \forall i=1,2,\dots, n.
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="___sec9">Examples with kernels </h2>
<h2 id="___sec9">A quick reminder on Lagrangian multipliers </h2>
<p>
Withe the above constraint, we introduce now the calculus of Lagrangian multiplier.
In order to solve the above problem, we define the following Lagrangian function to be minimized
<p>&nbsp;<br>
$$
L(\lambda)=\frac{1}{2}\hat{\beta}^T\hat{\beta}-\sum_{i=1}^n\lambda_i\left[y_i(\beta_0+\beta_1 x_i)-1\right],
$$
<p>&nbsp;<br>
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
<p>
Taking the derivatives with respect to \( \beta_0 \) and \( \beta_1 \) we obtain
<p>&nbsp;<br>
$$
\frac{\partial L}{\partial \beta_0} = -\sum_{i} \lambda_iy_i=0,
$$
<p>&nbsp;<br>
and
<p>&nbsp;<br>
$$
\frac{\partial L}{\partial \beta_1} = \beta_1-\sum_{i} \lambda_iy_ix_i=0.
$$
<p>&nbsp;<br>
Inserting these constraints into the equation for \( L \) we obtain
<p>&nbsp;<br>
$$
L(\lambda)=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_jx_i^Tx_j,
$$
<p>&nbsp;<br>
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the Koriush-Kuhn-Tucker (KKT) condition
<p>&nbsp;<br>
$$
\lambda_i\left[y_i(\beta_0+\beta_1 x_i) -1\right] \forall i=1,2,\dots, n.
$$
<p>&nbsp;<br>
<ol>
<p><li> If \( \lambda_i > 0 \), then \( y_i(\beta_0+\beta_1 x_i)=1 \) and we say that \( x_i \) is on the boundary of the slab.</li>
<p><li> If \( y_i(\beta_0+\beta_1 x_i)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
</section>
<section>
<h2 id="___sec10">Examples with kernels </h2>
<p>
doc/pub/svm/html/svm-solarized.html
+74 -2
View File
@@ -43,7 +43,11 @@ div { text-align: justify; text-justify: inter-word; }
('First attempt at a minimization approach', 2, None, '___sec6'),
('Solving the equations', 2, None, '___sec7'),
('A better approach', 2, None, '___sec8'),
('Examples with kernels', 2, None, '___sec9')]}
('A quick reminder on Lagrangian multipliers',
2,
None,
'___sec9'),
('Examples with kernels', 2, None, '___sec10')]}
end of tocinfo -->
<body>
@@ -285,12 +289,80 @@ There are however problems with this approach, although it looks pretty straight
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">A better approach </h2>
<p>
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
<p>
Thus, we wish to find a margin \( M \) with \( \hat{\beta} \) normalized to \( \vert\vert \hat{\beta}\vert\vert =1 \) subject to the condition
$$
y_i(\beta_0+\beta_1 x_i) \geq M \forall i=1,2,\dots, n.
$$
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( \beta_0 \) and \( \beta_1 \) define this line.
<p>
We seek thus the largest value \( M \) defined by
$$
\frac{1}{\vert \vert \hat{\beta}\vert\vert}y_i(\beta_0+\beta_1 x_) \geq M \forall i=1,2,\dots, n,
$$
or just
$$
y_i(\beta_0+\beta_1 x_i) \geq M\vert \vert \hat{\beta}\vert\vert \forall i=1,2,\dots, n.
$$
If we scale the equation so that \( \vert \vert \hat{\beta}\vert\vert = 1/M \), we have to find the minimum of
\( \hat{\beta}^T\hat{\beta} \) subject to the condition
$$
y_i(\beta_0+\beta_1 x_i) \geq 1 \forall i=1,2,\dots, n.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Examples with kernels </h2>
<h2 id="___sec9">A quick reminder on Lagrangian multipliers </h2>
<p>
Withe the above constraint, we introduce now the calculus of Lagrangian multiplier.
In order to solve the above problem, we define the following Lagrangian function to be minimized
$$
L(\lambda)=\frac{1}{2}\hat{\beta}^T\hat{\beta}-\sum_{i=1}^n\lambda_i\left[y_i(\beta_0+\beta_1 x_i)-1\right],
$$
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
<p>
Taking the derivatives with respect to \( \beta_0 \) and \( \beta_1 \) we obtain
$$
\frac{\partial L}{\partial \beta_0} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial L}{\partial \beta_1} = \beta_1-\sum_{i} \lambda_iy_ix_i=0.
$$
Inserting these constraints into the equation for \( L \) we obtain
$$
L(\lambda)=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_jx_i^Tx_j,
$$
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the Koriush-Kuhn-Tucker (KKT) condition
$$
\lambda_i\left[y_i(\beta_0+\beta_1 x_i) -1\right] \forall i=1,2,\dots, n.
$$
<ol>
<li> If \( \lambda_i > 0 \), then \( y_i(\beta_0+\beta_1 x_i)=1 \) and we say that \( x_i \) is on the boundary of the slab.</li>
<li> If \( y_i(\beta_0+\beta_1 x_i)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Examples with kernels </h2>
<p>
doc/pub/svm/html/svm.html
+74 -2
View File
@@ -48,7 +48,11 @@ div { text-align: justify; text-justify: inter-word; }
('First attempt at a minimization approach', 2, None, '___sec6'),
('Solving the equations', 2, None, '___sec7'),
('A better approach', 2, None, '___sec8'),
('Examples with kernels', 2, None, '___sec9')]}
('A quick reminder on Lagrangian multipliers',
2,
None,
'___sec9'),
('Examples with kernels', 2, None, '___sec10')]}
end of tocinfo -->
<body>
@@ -290,12 +294,80 @@ There are however problems with this approach, although it looks pretty straight
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">A better approach </h2>
<p>
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
<p>
Thus, we wish to find a margin \( M \) with \( \hat{\beta} \) normalized to \( \vert\vert \hat{\beta}\vert\vert =1 \) subject to the condition
$$
y_i(\beta_0+\beta_1 x_i) \geq M \forall i=1,2,\dots, n.
$$
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( \beta_0 \) and \( \beta_1 \) define this line.
<p>
We seek thus the largest value \( M \) defined by
$$
\frac{1}{\vert \vert \hat{\beta}\vert\vert}y_i(\beta_0+\beta_1 x_) \geq M \forall i=1,2,\dots, n,
$$
or just
$$
y_i(\beta_0+\beta_1 x_i) \geq M\vert \vert \hat{\beta}\vert\vert \forall i=1,2,\dots, n.
$$
If we scale the equation so that \( \vert \vert \hat{\beta}\vert\vert = 1/M \), we have to find the minimum of
\( \hat{\beta}^T\hat{\beta} \) subject to the condition
$$
y_i(\beta_0+\beta_1 x_i) \geq 1 \forall i=1,2,\dots, n.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Examples with kernels </h2>
<h2 id="___sec9">A quick reminder on Lagrangian multipliers </h2>
<p>
Withe the above constraint, we introduce now the calculus of Lagrangian multiplier.
In order to solve the above problem, we define the following Lagrangian function to be minimized
$$
L(\lambda)=\frac{1}{2}\hat{\beta}^T\hat{\beta}-\sum_{i=1}^n\lambda_i\left[y_i(\beta_0+\beta_1 x_i)-1\right],
$$
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
<p>
Taking the derivatives with respect to \( \beta_0 \) and \( \beta_1 \) we obtain
$$
\frac{\partial L}{\partial \beta_0} = -\sum_{i} \lambda_iy_i=0,
$$
and
$$
\frac{\partial L}{\partial \beta_1} = \beta_1-\sum_{i} \lambda_iy_ix_i=0.
$$
Inserting these constraints into the equation for \( L \) we obtain
$$
L(\lambda)=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_jx_i^Tx_j,
$$
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the Koriush-Kuhn-Tucker (KKT) condition
$$
\lambda_i\left[y_i(\beta_0+\beta_1 x_i) -1\right] \forall i=1,2,\dots, n.
$$
<ol>
<li> If \( \lambda_i > 0 \), then \( y_i(\beta_0+\beta_1 x_i)=1 \) and we say that \( x_i \) is on the boundary of the slab.</li>
<li> If \( y_i(\beta_0+\beta_1 x_i)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Examples with kernels </h2>
<p>
doc/pub/svm/ipynb/ipynb-svm-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/svm/ipynb/svm.ipynb
+158
View File
@@ -300,8 +300,166 @@
"There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.\n",
"\n",
"## A better approach\n",
"\n",
"A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning). \n",
"\n",
"Thus, we wish to find a margin $M$ with $\\hat{\\beta}$ normalized to $\\vert\\vert \\hat{\\beta}\\vert\\vert =1$ subject to the condition"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"y_i(\\beta_0+\\beta_1 x_i) \\geq M \\forall i=1,2,\\dots, n.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $\\beta_0$ and $\\beta_1$ define this line. \n",
"\n",
"We seek thus the largest value $M$ defined by"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{1}{\\vert \\vert \\hat{\\beta}\\vert\\vert}y_i(\\beta_0+\\beta_1 x_) \\geq M \\forall i=1,2,\\dots, n,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"or just"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"y_i(\\beta_0+\\beta_1 x_i) \\geq M\\vert \\vert \\hat{\\beta}\\vert\\vert \\forall i=1,2,\\dots, n.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we scale the equation so that $\\vert \\vert \\hat{\\beta}\\vert\\vert = 1/M$, we have to find the minimum of \n",
"$\\hat{\\beta}^T\\hat{\\beta}$ subject to the condition"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"y_i(\\beta_0+\\beta_1 x_i) \\geq 1 \\forall i=1,2,\\dots, n.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## A quick reminder on Lagrangian multipliers\n",
"\n",
"Withe the above constraint, we introduce now the calculus of Lagrangian multiplier. \n",
"In order to solve the above problem, we define the following Lagrangian function to be minimized"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"L(\\lambda)=\\frac{1}{2}\\hat{\\beta}^T\\hat{\\beta}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\beta_0+\\beta_1 x_i)-1\\right],\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where $\\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\\lambda_i \\geq 0$.\n",
"\n",
"Taking the derivatives with respect to $\\beta_0$ and $\\beta_1$ we obtain"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial L}{\\partial \\beta_0} = -\\sum_{i} \\lambda_iy_i=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial L}{\\partial \\beta_1} = \\beta_1-\\sum_{i} \\lambda_iy_ix_i=0.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Inserting these constraints into the equation for $L$ we obtain"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"L(\\lambda)=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_jx_i^Tx_j,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"subject to the constraints $\\lambda_i\\geq 0$ and $\\sum_i\\lambda_iy_i=0$. \n",
"We must in addition satisfy the Koriush-Kuhn-Tucker (KKT) condition"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\lambda_i\\left[y_i(\\beta_0+\\beta_1 x_i) -1\\right] \\forall i=1,2,\\dots, n.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. If $\\lambda_i > 0$, then $y_i(\\beta_0+\\beta_1 x_i)=1$ and we say that $x_i$ is on the boundary of the slab.\n",
"\n",
"2. If $y_i(\\beta_0+\\beta_1 x_i)> 1$, we say $x_i$ is not on the boundary and we set $\\lambda_i=0$. \n",
"\n",
"## Examples with kernels"
]
},
doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf
BIN
View File
Binary file not shown.
doc/pub/svm/pdf/svm-beamer.pdf
BIN
View File
Binary file not shown.
doc/pub/svm/pdf/svm-minted.pdf
BIN
View File
Binary file not shown.
doc/src/SupportVMachines/svm.do.txt
+70
View File
@@ -180,8 +180,78 @@ There are however problems with this approach, although it looks pretty straight
!split
===== A better approach =====
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
Thus, we wish to find a margin $M$ with $\hat{\beta}$ normalized to $\vert\vert \hat{\beta}\vert\vert =1$ subject to the condition
!bt
\[
y_i(\beta_0+\beta_1 x_i) \geq M \forall i=1,2,\dots, n.
\]
!et
All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $\beta_0$ and $\beta_1$ define this line.
We seek thus the largest value $M$ defined by
!bt
\[
\frac{1}{\vert \vert \hat{\beta}\vert\vert}y_i(\beta_0+\beta_1 x_) \geq M \forall i=1,2,\dots, n,
\]
!et
or just
!bt
\[
y_i(\beta_0+\beta_1 x_i) \geq M\vert \vert \hat{\beta}\vert\vert \forall i=1,2,\dots, n.
\]
!et
If we scale the equation so that $\vert \vert \hat{\beta}\vert\vert = 1/M$, we have to find the minimum of
$\hat{\beta}^T\hat{\beta}$ subject to the condition
!bt
\[
y_i(\beta_0+\beta_1 x_i) \geq 1 \forall i=1,2,\dots, n.
\]
!et
!split
===== A quick reminder on Lagrangian multipliers =====
Withe the above constraint, we introduce now the calculus of Lagrangian multiplier.
In order to solve the above problem, we define the following Lagrangian function to be minimized
!bt
\[
L(\lambda)=\frac{1}{2}\hat{\beta}^T\hat{\beta}-\sum_{i=1}^n\lambda_i\left[y_i(\beta_0+\beta_1 x_i)-1\right],
\]
!et
where $\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\lambda_i \geq 0$.
Taking the derivatives with respect to $\beta_0$ and $\beta_1$ we obtain
!bt
\[
\frac{\partial L}{\partial \beta_0} = -\sum_{i} \lambda_iy_i=0,
\]
!et
and
!bt
\[
\frac{\partial L}{\partial \beta_1} = \beta_1-\sum_{i} \lambda_iy_ix_i=0.
\]
!et
Inserting these constraints into the equation for $L$ we obtain
!bt
\[
L(\lambda)=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_jx_i^Tx_j,
\]
!et
subject to the constraints $\lambda_i\geq 0$ and $\sum_i\lambda_iy_i=0$.
We must in addition satisfy the Koriush-Kuhn-Tucker (KKT) condition
!bt
\[
\lambda_i\left[y_i(\beta_0+\beta_1 x_i) -1\right] \forall i=1,2,\dots, n.
\]
!et
o If $\lambda_i > 0$, then $y_i(\beta_0+\beta_1 x_i)=1$ and we say that $x_i$ is on the boundary of the slab.
o If $y_i(\beta_0+\beta_1 x_i)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$.
!split
===== Examples with kernels =====