lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

adding more equations to sum part

Browse Source
This commit is contained in:
mhjensen
2018-11-04 09:53:36 +01:00
parent 88561fbaaf
commit efa30bcf9e
11 changed files with 638 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/svm/html/svm-bs.html
+80 -5
View File
@@ -46,7 +46,10 @@ Automatically generated HTML file from DocOnce source
('What is a hyperplane', 2, None, '___sec3'),
('The two-dimensional case', 2, None, '___sec4'),
('Getting into the details', 2, None, '___sec5'),
('Examples with kernels', 2, None, '___sec6')]}
('First attempt at a minimization approach', 2, None, '___sec6'),
('Solving the equations', 2, None, '___sec7'),
('A better approach', 2, None, '___sec8'),
('Examples with kernels', 2, None, '___sec9')]}
end of tocinfo -->
<body>
@@ -90,7 +93,10 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">What is a hyperplane</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">The two-dimensional case</a></li>
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;">Getting into the details</a></li>
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;">Examples with kernels</a></li>
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;">First attempt at a minimization approach</a></li>
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;">Solving the equations</a></li>
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;">A better approach</a></li>
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;">Examples with kernels</a></li>
</ul>
</li>
@@ -124,7 +130,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 3, 2018</h4></center> <!-- date -->
<center><h4>Nov 4, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- potential-jumbotron-button -->
@@ -260,10 +266,79 @@ $$
f(x) = \beta_0+\beta_1x = 0,
$$
as the function that determines the line that separates two classes (our two features).
as the function that determines the line \( L \) that separates two classes (our two features), see the figur here.
<p>
Define a vector \( \hat{\beta}:\left\{\beta_0,\beta_1\right\} \). Let us label the values of \( \hat{\beta} \) that satisfy this constraint as \( \overline{\beta} \).
<p>
Any two points \( x_1 \) and \( x_2 \) on the line \( L \) will satisfy \( \hat{\beta}(x_1-x_2)=0 \). We normalize the solution and define
$$
\overline{\beta} = \frac{\hat{\beta}}{\vert\vert \hat{\beta}\vert\vert},
$$
which is vector normal to the line \( L \).
<p>
The signed distance from a point \( x_0 \) on \( L \) to any point \( x \) is then
$$
\overline{\beta}(x-x_0) = \frac{\beta_1 x + \beta_0}{\vert\vert \hat{\beta}\vert\vert}.
$$
<p>
<!-- !split -->
<h2 id="___sec6" class="anchor">Examples with kernels </h2>
<h2 id="___sec6" class="anchor">First attempt at a minimization approach </h2>
<p>
How do we find the parameters \( \beta_0 \) and \( \beta_0 \)? What we could
do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
$$
C(\beta_0,\beta_1) = -\sum_{i\in M} y_i(\beta_1x_1+\beta_0).
$$
<p>
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \beta_1x_i+\beta_0 < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
$$
\frac{\partial C}{\partial \beta_0} = -\sum_{i\in M} y_i,
$$
and
$$
\frac{\partial C}{\partial \beta_1} = -\sum_{i\in M} y_ix_i.
$$
<p>
<!-- !split -->
<h2 id="___sec7" class="anchor">Solving the equations </h2>
<p>
We can now use the Newton-Raphson method or gradient descent to solve the equations
$$
\beta_0 \leftarrow \beta_0 +\eta \frac{\partial C}{\partial \beta_0},
$$
and
$$
\beta_1 \leftarrow \beta_1 +\eta \frac{\partial C}{\partial \beta_1},
$$
where \( \eta \) is our by now well-known learning rate.
There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.
<p>
<!-- !split -->
<h2 id="___sec8" class="anchor">A better approach </h2>
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
<p>
<!-- !split -->
<h2 id="___sec9" class="anchor">Examples with kernels </h2>
<p>
doc/pub/svm/html/svm-reveal.html
+84 -3
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Nov 3, 2018</h4></center> <!-- date -->
<center><h4>Nov 4, 2018</h4></center> <!-- date -->
<br>
<p>
@@ -297,12 +297,93 @@ f(x) = \beta_0+\beta_1x = 0,
$$
<p>&nbsp;<br>
as the function that determines the line that separates two classes (our two features).
as the function that determines the line \( L \) that separates two classes (our two features), see the figur here.
<p>
Define a vector \( \hat{\beta}:\left\{\beta_0,\beta_1\right\} \). Let us label the values of \( \hat{\beta} \) that satisfy this constraint as \( \overline{\beta} \).
<p>
Any two points \( x_1 \) and \( x_2 \) on the line \( L \) will satisfy \( \hat{\beta}(x_1-x_2)=0 \). We normalize the solution and define
<p>&nbsp;<br>
$$
\overline{\beta} = \frac{\hat{\beta}}{\vert\vert \hat{\beta}\vert\vert},
$$
<p>&nbsp;<br>
which is vector normal to the line \( L \).
<p>
The signed distance from a point \( x_0 \) on \( L \) to any point \( x \) is then
<p>&nbsp;<br>
$$
\overline{\beta}(x-x_0) = \frac{\beta_1 x + \beta_0}{\vert\vert \hat{\beta}\vert\vert}.
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="___sec6">Examples with kernels </h2>
<h2 id="___sec6">First attempt at a minimization approach </h2>
<p>
How do we find the parameters \( \beta_0 \) and \( \beta_0 \)? What we could
do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
<p>&nbsp;<br>
$$
C(\beta_0,\beta_1) = -\sum_{i\in M} y_i(\beta_1x_1+\beta_0).
$$
<p>&nbsp;<br>
<p>
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \beta_1x_i+\beta_0 < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
<p>&nbsp;<br>
$$
\frac{\partial C}{\partial \beta_0} = -\sum_{i\in M} y_i,
$$
<p>&nbsp;<br>
and
<p>&nbsp;<br>
$$
\frac{\partial C}{\partial \beta_1} = -\sum_{i\in M} y_ix_i.
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="___sec7">Solving the equations </h2>
<p>
We can now use the Newton-Raphson method or gradient descent to solve the equations
<p>&nbsp;<br>
$$
\beta_0 \leftarrow \beta_0 +\eta \frac{\partial C}{\partial \beta_0},
$$
<p>&nbsp;<br>
and
<p>&nbsp;<br>
$$
\beta_1 \leftarrow \beta_1 +\eta \frac{\partial C}{\partial \beta_1},
$$
<p>&nbsp;<br>
where \( \eta \) is our by now well-known learning rate.
There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.
</section>
<section>
<h2 id="___sec8">A better approach </h2>
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
</section>
<section>
<h2 id="___sec9">Examples with kernels </h2>
<p>
doc/pub/svm/html/svm-solarized.html
+76 -4
View File
@@ -40,7 +40,10 @@ div { text-align: justify; text-justify: inter-word; }
('What is a hyperplane', 2, None, '___sec3'),
('The two-dimensional case', 2, None, '___sec4'),
('Getting into the details', 2, None, '___sec5'),
('Examples with kernels', 2, None, '___sec6')]}
('First attempt at a minimization approach', 2, None, '___sec6'),
('Solving the equations', 2, None, '___sec7'),
('A better approach', 2, None, '___sec8'),
('Examples with kernels', 2, None, '___sec9')]}
end of tocinfo -->
<body>
@@ -82,7 +85,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 3, 2018</h4></center> <!-- date -->
<center><h4>Nov 4, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -215,10 +218,79 @@ $$
f(x) = \beta_0+\beta_1x = 0,
$$
as the function that determines the line that separates two classes (our two features).
as the function that determines the line \( L \) that separates two classes (our two features), see the figur here.
<p>
Define a vector \( \hat{\beta}:\left\{\beta_0,\beta_1\right\} \). Let us label the values of \( \hat{\beta} \) that satisfy this constraint as \( \overline{\beta} \).
<p>
Any two points \( x_1 \) and \( x_2 \) on the line \( L \) will satisfy \( \hat{\beta}(x_1-x_2)=0 \). We normalize the solution and define
$$
\overline{\beta} = \frac{\hat{\beta}}{\vert\vert \hat{\beta}\vert\vert},
$$
which is vector normal to the line \( L \).
<p>
The signed distance from a point \( x_0 \) on \( L \) to any point \( x \) is then
$$
\overline{\beta}(x-x_0) = \frac{\beta_1 x + \beta_0}{\vert\vert \hat{\beta}\vert\vert}.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Examples with kernels </h2>
<h2 id="___sec6">First attempt at a minimization approach </h2>
<p>
How do we find the parameters \( \beta_0 \) and \( \beta_0 \)? What we could
do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
$$
C(\beta_0,\beta_1) = -\sum_{i\in M} y_i(\beta_1x_1+\beta_0).
$$
<p>
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \beta_1x_i+\beta_0 < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
$$
\frac{\partial C}{\partial \beta_0} = -\sum_{i\in M} y_i,
$$
and
$$
\frac{\partial C}{\partial \beta_1} = -\sum_{i\in M} y_ix_i.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Solving the equations </h2>
<p>
We can now use the Newton-Raphson method or gradient descent to solve the equations
$$
\beta_0 \leftarrow \beta_0 +\eta \frac{\partial C}{\partial \beta_0},
$$
and
$$
\beta_1 \leftarrow \beta_1 +\eta \frac{\partial C}{\partial \beta_1},
$$
where \( \eta \) is our by now well-known learning rate.
There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">A better approach </h2>
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Examples with kernels </h2>
<p>
doc/pub/svm/html/svm.html
+76 -4
View File
@@ -45,7 +45,10 @@ div { text-align: justify; text-justify: inter-word; }
('What is a hyperplane', 2, None, '___sec3'),
('The two-dimensional case', 2, None, '___sec4'),
('Getting into the details', 2, None, '___sec5'),
('Examples with kernels', 2, None, '___sec6')]}
('First attempt at a minimization approach', 2, None, '___sec6'),
('Solving the equations', 2, None, '___sec7'),
('A better approach', 2, None, '___sec8'),
('Examples with kernels', 2, None, '___sec9')]}
end of tocinfo -->
<body>
@@ -87,7 +90,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 3, 2018</h4></center> <!-- date -->
<center><h4>Nov 4, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -220,10 +223,79 @@ $$
f(x) = \beta_0+\beta_1x = 0,
$$
as the function that determines the line that separates two classes (our two features).
as the function that determines the line \( L \) that separates two classes (our two features), see the figur here.
<p>
Define a vector \( \hat{\beta}:\left\{\beta_0,\beta_1\right\} \). Let us label the values of \( \hat{\beta} \) that satisfy this constraint as \( \overline{\beta} \).
<p>
Any two points \( x_1 \) and \( x_2 \) on the line \( L \) will satisfy \( \hat{\beta}(x_1-x_2)=0 \). We normalize the solution and define
$$
\overline{\beta} = \frac{\hat{\beta}}{\vert\vert \hat{\beta}\vert\vert},
$$
which is vector normal to the line \( L \).
<p>
The signed distance from a point \( x_0 \) on \( L \) to any point \( x \) is then
$$
\overline{\beta}(x-x_0) = \frac{\beta_1 x + \beta_0}{\vert\vert \hat{\beta}\vert\vert}.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Examples with kernels </h2>
<h2 id="___sec6">First attempt at a minimization approach </h2>
<p>
How do we find the parameters \( \beta_0 \) and \( \beta_0 \)? What we could
do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
$$
C(\beta_0,\beta_1) = -\sum_{i\in M} y_i(\beta_1x_1+\beta_0).
$$
<p>
We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \beta_1x_i+\beta_0 < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us
$$
\frac{\partial C}{\partial \beta_0} = -\sum_{i\in M} y_i,
$$
and
$$
\frac{\partial C}{\partial \beta_1} = -\sum_{i\in M} y_ix_i.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Solving the equations </h2>
<p>
We can now use the Newton-Raphson method or gradient descent to solve the equations
$$
\beta_0 \leftarrow \beta_0 +\eta \frac{\partial C}{\partial \beta_0},
$$
and
$$
\beta_1 \leftarrow \beta_1 +\eta \frac{\partial C}{\partial \beta_1},
$$
where \( \eta \) is our by now well-known learning rate.
There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">A better approach </h2>
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Examples with kernels </h2>
<p>
doc/pub/svm/ipynb/.ipynb_checkpoints/svm-checkpoint.ipynb
+124
View File
File diff suppressed because one or more lines are too long
doc/pub/svm/ipynb/ipynb-svm-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/svm/ipynb/svm.ipynb
+131 -2
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Nov 3, 2018**\n",
"Date: **Nov 4, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -172,7 +172,136 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"as the function that determines the line that separates two classes (our two features). \n",
"as the function that determines the line $L$ that separates two classes (our two features), see the figur here. \n",
"\n",
"Define a vector $\\hat{\\beta}:\\left\\{\\beta_0,\\beta_1\\right\\}$. Let us label the values of $\\hat{\\beta}$ that satisfy this constraint as $\\overline{\\beta}$. \n",
"\n",
"Any two points $x_1$ and $x_2$ on the line $L$ will satisfy $\\hat{\\beta}(x_1-x_2)=0$. We normalize the solution and define"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\overline{\\beta} = \\frac{\\hat{\\beta}}{\\vert\\vert \\hat{\\beta}\\vert\\vert},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which is vector normal to the line $L$. \n",
"\n",
"The signed distance from a point $x_0$ on $L$ to any point $x$ is then"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\overline{\\beta}(x-x_0) = \\frac{\\beta_1 x + \\beta_0}{\\vert\\vert \\hat{\\beta}\\vert\\vert}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## First attempt at a minimization approach\n",
"\n",
"How do we find the parameters $\\beta_0$ and $\\beta_0$? What we could\n",
"do is to define a cost function which now contains the set of all\n",
"misclassified points $M$ and attempt to minimize this function"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"C(\\beta_0,\\beta_1) = -\\sum_{i\\in M} y_i(\\beta_1x_1+\\beta_0).\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We could now for example define all values $y_i =1$ as misclassified in case we have $\\beta_1x_i+\\beta_0 < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial \\beta_0} = -\\sum_{i\\in M} y_i,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial \\beta_1} = -\\sum_{i\\in M} y_ix_i.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Solving the equations\n",
"\n",
"We can now use the Newton-Raphson method or gradient descent to solve the equations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\beta_0 \\leftarrow \\beta_0 +\\eta \\frac{\\partial C}{\\partial \\beta_0},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\beta_1 \\leftarrow \\beta_1 +\\eta \\frac{\\partial C}{\\partial \\beta_1},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where $\\eta$ is our by now well-known learning rate. \n",
"There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.\n",
"\n",
"## A better approach\n",
"A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning). \n",
"\n",
"## Examples with kernels"
]
},
doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf
BIN
View File
Binary file not shown.
doc/pub/svm/pdf/svm-beamer.pdf
BIN
View File
Binary file not shown.
doc/pub/svm/pdf/svm-minted.pdf
BIN
View File
Binary file not shown.
doc/src/SupportVMachines/svm.do.txt
+67 -1
View File
@@ -115,7 +115,73 @@ Let us define the function
f(x) = \beta_0+\beta_1x = 0,
\]
!et
as the function that determines the line that separates two classes (our two features).
as the function that determines the line $L$ that separates two classes (our two features), see the figur here.
Define a vector $\hat{\beta}:\left\{\beta_0,\beta_1\right\}$. Let us label the values of $\hat{\beta}$ that satisfy this constraint as $\overline{\beta}$.
Any two points $x_1$ and $x_2$ on the line $L$ will satisfy $\hat{\beta}(x_1-x_2)=0$. We normalize the solution and define
!bt
\[
\overline{\beta} = \frac{\hat{\beta}}{\vert\vert \hat{\beta}\vert\vert},
\]
!et
which is vector normal to the line $L$.
The signed distance from a point $x_0$ on $L$ to any point $x$ is then
!bt
\[
\overline{\beta}(x-x_0) = \frac{\beta_1 x + \beta_0}{\vert\vert \hat{\beta}\vert\vert}.
\]
!et
!split
===== First attempt at a minimization approach =====
How do we find the parameters $\beta_0$ and $\beta_0$? What we could
do is to define a cost function which now contains the set of all
misclassified points $M$ and attempt to minimize this function
!bt
\[
C(\beta_0,\beta_1) = -\sum_{i\in M} y_i(\beta_1x_1+\beta_0).
\]
!et
We could now for example define all values $y_i =1$ as misclassified in case we have $\beta_1x_i+\beta_0 < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us
!bt
\[
\frac{\partial C}{\partial \beta_0} = -\sum_{i\in M} y_i,
\]
!et
and
!bt
\[
\frac{\partial C}{\partial \beta_1} = -\sum_{i\in M} y_ix_i.
\]
!et
!split
===== Solving the equations =====
We can now use the Newton-Raphson method or gradient descent to solve the equations
!bt
\[
\beta_0 \leftarrow \beta_0 +\eta \frac{\partial C}{\partial \beta_0},
\]
!et
and
!bt
\[
\beta_1 \leftarrow \beta_1 +\eta \frac{\partial C}{\partial \beta_1},
\]
!et
where $\eta$ is our by now well-known learning rate.
There are however problems with this approach, although it looks pretty straightforward to implement. In case we separate our data into two distinct classes, we may up with many possible lines, as indicated in the figure and shown by running the following program. For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all.
!split
===== A better approach =====
A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning).
!split
===== Examples with kernels =====