lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating sim slides

Browse Source
This commit is contained in:
mhjensen
2018-11-03 22:30:06 +01:00
parent 7940c2bcc4
commit 9928285ec6
10 changed files with 609 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/svm/html/svm-bs.html
+104 -10
View File
@@ -42,7 +42,10 @@ Automatically generated HTML file from DocOnce source
{'highest level': 2,
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
('Strength and weakness', 2, None, '___sec1'),
('Examples with kernels', 2, None, '___sec2')]}
('Hyperplanes and all that', 2, None, '___sec2'),
('What is a hyperplane', 2, None, '___sec3'),
('The two-dimensional case', 2, None, '___sec4'),
('Examples with kernels', 2, None, '___sec5')]}
end of tocinfo -->
<body>
@@ -82,7 +85,10 @@ MathJax.Hub.Config({
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">Support Vector Machines, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Strength and weakness</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Examples with kernels</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Hyperplanes and all that</a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">What is a hyperplane</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">The two-dimensional case</a></li>
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;">Examples with kernels</a></li>
</ul>
</li>
@@ -116,7 +122,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<center><h4>Nov 3, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- potential-jumbotron-button -->
@@ -136,27 +142,115 @@ particularly well suited for classification of complex but small-sized or
medium-sized datasets.
<p>
The basic mathematics relies on the definition of hyperplanes and the definition of a <b>margin</b> which separates
classes (in case of classification problems) of variables. It is also used for regression problems.
The case with two well-separated classes only can be understood in an intuitive way in terms of lines in a two-dimensional space separating the two classes (see figure below).
<p>
The basic mathematics behind the SVM is however less familiar to most of us.
It relies on the definition of hyperplanes and the
definition of a <b>margin</b> which separates classes (in case of
classification problems) of variables. It is also used for regression
problems.
<p>
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
We distringuish also between linearn and non-linear approaches.
<p>
<b>These notes will be updated shortly with more material</b>
We distringuish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines.
<p>
<!-- !split -->
<h2 id="___sec1" class="anchor">Strength and weakness </h2>
<p>
When we implement a linear support vector machine, the main parameter is the constant \( C \). Small values of \( C \) mean simple models.
These models are fast to train and also fast to predict and scale to very large data sets and work well with sparse data. Linear support vector machines make it easy to understand how a prediction is made, however it is often not easy to understand why coefficients are the way they are. These models work also well in higer dimensions.
<p>
<!-- !split -->
<h2 id="___sec2" class="anchor">Examples with kernels </h2>
<h2 id="___sec2" class="anchor">Hyperplanes and all that </h2>
<p>
The theory behind support vector machines (SVM hereafter) is based on
the mathematical description of so-called hyperplanes. Let us start
with a two-dimensional case. This will also allow us to introduce our
first SVM examples. These will be tailored to the case of two specific
classes, as displayed in the figure here.
<p>
We assume here that our data set can be well separated into two
domains, where a straight line does the job in the separating the two
classes. Here the two classes are represented by either crosses or
circles.
<p>
<!-- !split -->
<h2 id="___sec3" class="anchor">What is a hyperplane </h2>
<p>
The aim of the SVM algorithm is to find a hyperplane in an \( n \)-dimensional space, where \( n \) is the number of features that distinctly classifies the data points.
<p>
In an \( n \)-dimensional space, a hyperplane is what we call an affine subspace of dimension of \( n-1 \).
As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is
a two-dimensional subspace, or stated simply, a plane.
<p>
In two dimensions, with the variables \( x_1 \) and \( x_2 \), the hyperplane is defined as
$$
\beta_0+\beta_1x_1+\beta_2x_2=0,
$$
<p>
In an \( n \)-dimensional space we have
$$
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n=0,
$$
<p>
With \( \hat{x}=\left[x_1,x_2,\dots, x_n\right] \), if the above condition is not met and
$$
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n < 0,
$$
we say that \( \hat{x} \) lies on one of the sides of the hyperplane and if
$$
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n>0,
$$
then \( \hat{x} \) lies on the other side.
<p>
<!-- !split -->
<h2 id="___sec4" class="anchor">The two-dimensional case </h2>
<p>
Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional
plane. To separate the two classes of data points, there are many
possible lines (hyperplanes if you prefer a more strict naming)
that could be chosen. Our objective is to find a
plane that has the maximum margin, i.e the maximum distance between
data points of both classes. Maximizing the margin distance provides
some reinforcement so that future data points can be classified with
more confidence.
<p>
What a linear classifier attempts to accomplish is to split the
feature space into two half spaces by placing a hyperplane between the
data points. This hyperplane will be our decision boundary. All
points on one side of the plane will belong to class one and all points
on the other side of the plane will belong to the second class two.
<p>
Unfortunately there are many ways in which we can place a hyperplane
to divide the data. Below is an example of two candidate hyperplanes
for our data sample.
<p>
<!-- !split -->
<h2 id="___sec5" class="anchor">Examples with kernels </h2>
<p>
doc/pub/svm/html/svm-reveal.html
+104 -8
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<center><h4>Nov 3, 2018</h4></center> <!-- date -->
<br>
<p>
@@ -171,27 +171,123 @@ particularly well suited for classification of complex but small-sized or
medium-sized datasets.
<p>
The basic mathematics relies on the definition of hyperplanes and the definition of a <b>margin</b> which separates
classes (in case of classification problems) of variables. It is also used for regression problems.
The case with two well-separated classes only can be understood in an intuitive way in terms of lines in a two-dimensional space separating the two classes (see figure below).
<p>
The basic mathematics behind the SVM is however less familiar to most of us.
It relies on the definition of hyperplanes and the
definition of a <b>margin</b> which separates classes (in case of
classification problems) of variables. It is also used for regression
problems.
<p>
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
We distringuish also between linearn and non-linear approaches.
<p>
<b>These notes will be updated shortly with more material</b>
We distringuish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines.
</section>
<section>
<h2 id="___sec1">Strength and weakness </h2>
<p>
When we implement a linear support vector machine, the main parameter is the constant \( C \). Small values of \( C \) mean simple models.
These models are fast to train and also fast to predict and scale to very large data sets and work well with sparse data. Linear support vector machines make it easy to understand how a prediction is made, however it is often not easy to understand why coefficients are the way they are. These models work also well in higer dimensions.
</section>
<section>
<h2 id="___sec2">Examples with kernels </h2>
<h2 id="___sec2">Hyperplanes and all that </h2>
<p>
The theory behind support vector machines (SVM hereafter) is based on
the mathematical description of so-called hyperplanes. Let us start
with a two-dimensional case. This will also allow us to introduce our
first SVM examples. These will be tailored to the case of two specific
classes, as displayed in the figure here.
<p>
We assume here that our data set can be well separated into two
domains, where a straight line does the job in the separating the two
classes. Here the two classes are represented by either crosses or
circles.
</section>
<section>
<h2 id="___sec3">What is a hyperplane </h2>
<p>
The aim of the SVM algorithm is to find a hyperplane in an \( n \)-dimensional space, where \( n \) is the number of features that distinctly classifies the data points.
<p>
In an \( n \)-dimensional space, a hyperplane is what we call an affine subspace of dimension of \( n-1 \).
As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is
a two-dimensional subspace, or stated simply, a plane.
<p>
In two dimensions, with the variables \( x_1 \) and \( x_2 \), the hyperplane is defined as
<p>&nbsp;<br>
$$
\beta_0+\beta_1x_1+\beta_2x_2=0,
$$
<p>&nbsp;<br>
<p>
In an \( n \)-dimensional space we have
<p>&nbsp;<br>
$$
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n=0,
$$
<p>&nbsp;<br>
<p>
With \( \hat{x}=\left[x_1,x_2,\dots, x_n\right] \), if the above condition is not met and
<p>&nbsp;<br>
$$
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n < 0,
$$
<p>&nbsp;<br>
we say that \( \hat{x} \) lies on one of the sides of the hyperplane and if
<p>&nbsp;<br>
$$
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n>0,
$$
<p>&nbsp;<br>
then \( \hat{x} \) lies on the other side.
</section>
<section>
<h2 id="___sec4">The two-dimensional case </h2>
<p>
Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional
plane. To separate the two classes of data points, there are many
possible lines (hyperplanes if you prefer a more strict naming)
that could be chosen. Our objective is to find a
plane that has the maximum margin, i.e the maximum distance between
data points of both classes. Maximizing the margin distance provides
some reinforcement so that future data points can be classified with
more confidence.
<p>
What a linear classifier attempts to accomplish is to split the
feature space into two half spaces by placing a hyperplane between the
data points. This hyperplane will be our decision boundary. All
points on one side of the plane will belong to class one and all points
on the other side of the plane will belong to the second class two.
<p>
Unfortunately there are many ways in which we can place a hyperplane
to divide the data. Below is an example of two candidate hyperplanes
for our data sample.
</section>
<section>
<h2 id="___sec5">Examples with kernels </h2>
<p>
doc/pub/svm/html/svm-solarized.html
+100 -9
View File
@@ -36,7 +36,10 @@ div { text-align: justify; text-justify: inter-word; }
{'highest level': 2,
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
('Strength and weakness', 2, None, '___sec1'),
('Examples with kernels', 2, None, '___sec2')]}
('Hyperplanes and all that', 2, None, '___sec2'),
('What is a hyperplane', 2, None, '___sec3'),
('The two-dimensional case', 2, None, '___sec4'),
('Examples with kernels', 2, None, '___sec5')]}
end of tocinfo -->
<body>
@@ -78,7 +81,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<center><h4>Nov 3, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -95,27 +98,115 @@ particularly well suited for classification of complex but small-sized or
medium-sized datasets.
<p>
The basic mathematics relies on the definition of hyperplanes and the definition of a <b>margin</b> which separates
classes (in case of classification problems) of variables. It is also used for regression problems.
The case with two well-separated classes only can be understood in an intuitive way in terms of lines in a two-dimensional space separating the two classes (see figure below).
<p>
The basic mathematics behind the SVM is however less familiar to most of us.
It relies on the definition of hyperplanes and the
definition of a <b>margin</b> which separates classes (in case of
classification problems) of variables. It is also used for regression
problems.
<p>
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
We distringuish also between linearn and non-linear approaches.
<p>
<b>These notes will be updated shortly with more material</b>
We distringuish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Strength and weakness </h2>
<p>
When we implement a linear support vector machine, the main parameter is the constant \( C \). Small values of \( C \) mean simple models.
These models are fast to train and also fast to predict and scale to very large data sets and work well with sparse data. Linear support vector machines make it easy to understand how a prediction is made, however it is often not easy to understand why coefficients are the way they are. These models work also well in higer dimensions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Examples with kernels </h2>
<h2 id="___sec2">Hyperplanes and all that </h2>
<p>
The theory behind support vector machines (SVM hereafter) is based on
the mathematical description of so-called hyperplanes. Let us start
with a two-dimensional case. This will also allow us to introduce our
first SVM examples. These will be tailored to the case of two specific
classes, as displayed in the figure here.
<p>
We assume here that our data set can be well separated into two
domains, where a straight line does the job in the separating the two
classes. Here the two classes are represented by either crosses or
circles.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">What is a hyperplane </h2>
<p>
The aim of the SVM algorithm is to find a hyperplane in an \( n \)-dimensional space, where \( n \) is the number of features that distinctly classifies the data points.
<p>
In an \( n \)-dimensional space, a hyperplane is what we call an affine subspace of dimension of \( n-1 \).
As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is
a two-dimensional subspace, or stated simply, a plane.
<p>
In two dimensions, with the variables \( x_1 \) and \( x_2 \), the hyperplane is defined as
$$
\beta_0+\beta_1x_1+\beta_2x_2=0,
$$
<p>
In an \( n \)-dimensional space we have
$$
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n=0,
$$
<p>
With \( \hat{x}=\left[x_1,x_2,\dots, x_n\right] \), if the above condition is not met and
$$
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n < 0,
$$
we say that \( \hat{x} \) lies on one of the sides of the hyperplane and if
$$
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n>0,
$$
then \( \hat{x} \) lies on the other side.
<p>
<!-- !split -->
<h2 id="___sec4">The two-dimensional case </h2>
<p>
Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional
plane. To separate the two classes of data points, there are many
possible lines (hyperplanes if you prefer a more strict naming)
that could be chosen. Our objective is to find a
plane that has the maximum margin, i.e the maximum distance between
data points of both classes. Maximizing the margin distance provides
some reinforcement so that future data points can be classified with
more confidence.
<p>
What a linear classifier attempts to accomplish is to split the
feature space into two half spaces by placing a hyperplane between the
data points. This hyperplane will be our decision boundary. All
points on one side of the plane will belong to class one and all points
on the other side of the plane will belong to the second class two.
<p>
Unfortunately there are many ways in which we can place a hyperplane
to divide the data. Below is an example of two candidate hyperplanes
for our data sample.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Examples with kernels </h2>
<p>
doc/pub/svm/html/svm.html
+100 -9
View File
@@ -41,7 +41,10 @@ div { text-align: justify; text-justify: inter-word; }
{'highest level': 2,
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
('Strength and weakness', 2, None, '___sec1'),
('Examples with kernels', 2, None, '___sec2')]}
('Hyperplanes and all that', 2, None, '___sec2'),
('What is a hyperplane', 2, None, '___sec3'),
('The two-dimensional case', 2, None, '___sec4'),
('Examples with kernels', 2, None, '___sec5')]}
end of tocinfo -->
<body>
@@ -83,7 +86,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 2, 2018</h4></center> <!-- date -->
<center><h4>Nov 3, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -100,27 +103,115 @@ particularly well suited for classification of complex but small-sized or
medium-sized datasets.
<p>
The basic mathematics relies on the definition of hyperplanes and the definition of a <b>margin</b> which separates
classes (in case of classification problems) of variables. It is also used for regression problems.
The case with two well-separated classes only can be understood in an intuitive way in terms of lines in a two-dimensional space separating the two classes (see figure below).
<p>
The basic mathematics behind the SVM is however less familiar to most of us.
It relies on the definition of hyperplanes and the
definition of a <b>margin</b> which separates classes (in case of
classification problems) of variables. It is also used for regression
problems.
<p>
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
We distringuish also between linearn and non-linear approaches.
<p>
<b>These notes will be updated shortly with more material</b>
We distringuish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Strength and weakness </h2>
<p>
When we implement a linear support vector machine, the main parameter is the constant \( C \). Small values of \( C \) mean simple models.
These models are fast to train and also fast to predict and scale to very large data sets and work well with sparse data. Linear support vector machines make it easy to understand how a prediction is made, however it is often not easy to understand why coefficients are the way they are. These models work also well in higer dimensions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Examples with kernels </h2>
<h2 id="___sec2">Hyperplanes and all that </h2>
<p>
The theory behind support vector machines (SVM hereafter) is based on
the mathematical description of so-called hyperplanes. Let us start
with a two-dimensional case. This will also allow us to introduce our
first SVM examples. These will be tailored to the case of two specific
classes, as displayed in the figure here.
<p>
We assume here that our data set can be well separated into two
domains, where a straight line does the job in the separating the two
classes. Here the two classes are represented by either crosses or
circles.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">What is a hyperplane </h2>
<p>
The aim of the SVM algorithm is to find a hyperplane in an \( n \)-dimensional space, where \( n \) is the number of features that distinctly classifies the data points.
<p>
In an \( n \)-dimensional space, a hyperplane is what we call an affine subspace of dimension of \( n-1 \).
As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is
a two-dimensional subspace, or stated simply, a plane.
<p>
In two dimensions, with the variables \( x_1 \) and \( x_2 \), the hyperplane is defined as
$$
\beta_0+\beta_1x_1+\beta_2x_2=0,
$$
<p>
In an \( n \)-dimensional space we have
$$
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n=0,
$$
<p>
With \( \hat{x}=\left[x_1,x_2,\dots, x_n\right] \), if the above condition is not met and
$$
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n < 0,
$$
we say that \( \hat{x} \) lies on one of the sides of the hyperplane and if
$$
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n>0,
$$
then \( \hat{x} \) lies on the other side.
<p>
<!-- !split -->
<h2 id="___sec4">The two-dimensional case </h2>
<p>
Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional
plane. To separate the two classes of data points, there are many
possible lines (hyperplanes if you prefer a more strict naming)
that could be chosen. Our objective is to find a
plane that has the maximum margin, i.e the maximum distance between
data points of both classes. Maximizing the margin distance provides
some reinforcement so that future data points can be classified with
more confidence.
<p>
What a linear classifier attempts to accomplish is to split the
feature space into two half spaces by placing a hyperplane between the
data points. This hyperplane will be our decision boundary. All
points on one side of the plane will belong to class one and all points
on the other side of the plane will belong to the second class two.
<p>
Unfortunately there are many ways in which we can place a hyperplane
to divide the data. Below is an example of two candidate hyperplanes
for our data sample.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Examples with kernels </h2>
<p>
doc/pub/svm/ipynb/ipynb-svm-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/svm/ipynb/svm.ipynb
+119 -5
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Nov 2, 2018**\n",
"Date: **Nov 3, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -26,20 +26,134 @@
"particularly well suited for classification of complex but small-sized or\n",
"medium-sized datasets. \n",
"\n",
"The basic mathematics relies on the definition of hyperplanes and the definition of a **margin** which separates\n",
"classes (in case of classification problems) of variables. It is also used for regression problems.\n",
"The case with two well-separated classes only can be understood in an intuitive way in terms of lines in a two-dimensional space separating the two classes (see figure below). \n",
"\n",
"The basic mathematics behind the SVM is however less familiar to most of us. \n",
"It relies on the definition of hyperplanes and the\n",
"definition of a **margin** which separates classes (in case of\n",
"classification problems) of variables. It is also used for regression\n",
"problems.\n",
"\n",
"With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.\n",
"We distringuish also between linearn and non-linear approaches.\n",
"We distringuish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines. \n",
"\n",
"**These notes will be updated shortly with more material**\n",
"\n",
"\n",
"## Strength and weakness\n",
"\n",
"When we implement a linear support vector machine, the main parameter is the constant $C$. Small values of $C$ mean simple models.\n",
"These models are fast to train and also fast to predict and scale to very large data sets and work well with sparse data. Linear support vector machines make it easy to understand how a prediction is made, however it is often not easy to understand why coefficients are the way they are. These models work also well in higer dimensions. \n",
"\n",
"\n",
"## Hyperplanes and all that\n",
"\n",
"The theory behind support vector machines (SVM hereafter) is based on\n",
"the mathematical description of so-called hyperplanes. Let us start\n",
"with a two-dimensional case. This will also allow us to introduce our\n",
"first SVM examples. These will be tailored to the case of two specific\n",
"classes, as displayed in the figure here.\n",
"\n",
"We assume here that our data set can be well separated into two\n",
"domains, where a straight line does the job in the separating the two\n",
"classes. Here the two classes are represented by either crosses or\n",
"circles.\n",
"\n",
"## What is a hyperplane\n",
"\n",
"The aim of the SVM algorithm is to find a hyperplane in an $n$-dimensional space, where $n$ is the number of features that distinctly classifies the data points. \n",
"\n",
"In an $n$-dimensional space, a hyperplane is what we call an affine subspace of dimension of $n-1$.\n",
"As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is \n",
"a two-dimensional subspace, or stated simply, a plane. \n",
"\n",
"In two dimensions, with the variables $x_1$ and $x_2$, the hyperplane is defined as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\beta_0+\\beta_1x_1+\\beta_2x_2=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In an $n$-dimensional space we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots +\\beta_nx_n=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With $\\hat{x}=\\left[x_1,x_2,\\dots, x_n\\right]$, if the above condition is not met and"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots +\\beta_nx_n<0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"we say that $\\hat{x}$ lies on one of the sides of the hyperplane and if"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots +\\beta_nx_n>0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"then $\\hat{x}$ lies on the other side. \n",
"\n",
"<!-- !split -->\n",
"## The two-dimensional case\n",
"\n",
"Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional\n",
"plane. To separate the two classes of data points, there are many\n",
"possible lines (hyperplanes if you prefer a more strict naming) \n",
"that could be chosen. Our objective is to find a\n",
"plane that has the maximum margin, i.e the maximum distance between\n",
"data points of both classes. Maximizing the margin distance provides\n",
"some reinforcement so that future data points can be classified with\n",
"more confidence.\n",
"\n",
"What a linear classifier attempts to accomplish is to split the\n",
"feature space into two half spaces by placing a hyperplane between the\n",
"data points. This hyperplane will be our decision boundary. All\n",
"points on one side of the plane will belong to class one and all points\n",
"on the other side of the plane will belong to the second class two.\n",
"\n",
"Unfortunately there are many ways in which we can place a hyperplane\n",
"to divide the data. Below is an example of two candidate hyperplanes\n",
"for our data sample.\n",
"\n",
"## Examples with kernels"
]
},
doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf
BIN
View File
Binary file not shown.
doc/pub/svm/pdf/svm-beamer.pdf
BIN
View File
Binary file not shown.
doc/pub/svm/pdf/svm-minted.pdf
BIN
View File
Binary file not shown.
doc/src/SupportVMachines/svm.do.txt
+82 -4
View File
@@ -13,21 +13,99 @@ Machine Learning should have it in their toolbox. SVMs are
particularly well suited for classification of complex but small-sized or
medium-sized datasets.
The basic mathematics relies on the definition of hyperplanes and the definition of a _margin_ which separates
classes (in case of classification problems) of variables. It is also used for regression problems.
The case with two well-separated classes only can be understood in an intuitive way in terms of lines in a two-dimensional space separating the two classes (see figure below).
The basic mathematics behind the SVM is however less familiar to most of us.
It relies on the definition of hyperplanes and the
definition of a _margin_ which separates classes (in case of
classification problems) of variables. It is also used for regression
problems.
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
We distringuish also between linearn and non-linear approaches.
We distringuish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines.
_These notes will be updated shortly with more material_
!split
===== Strength and weakness =====
When we implement a linear support vector machine, the main parameter is the constant $C$. Small values of $C$ mean simple models.
These models are fast to train and also fast to predict and scale to very large data sets and work well with sparse data. Linear support vector machines make it easy to understand how a prediction is made, however it is often not easy to understand why coefficients are the way they are. These models work also well in higer dimensions.
!split
===== Hyperplanes and all that =====
The theory behind support vector machines (SVM hereafter) is based on
the mathematical description of so-called hyperplanes. Let us start
with a two-dimensional case. This will also allow us to introduce our
first SVM examples. These will be tailored to the case of two specific
classes, as displayed in the figure here.
We assume here that our data set can be well separated into two
domains, where a straight line does the job in the separating the two
classes. Here the two classes are represented by either crosses or
circles.
!split
===== What is a hyperplane =====
The aim of the SVM algorithm is to find a hyperplane in an $n$-dimensional space, where $n$ is the number of features that distinctly classifies the data points.
In an $n$-dimensional space, a hyperplane is what we call an affine subspace of dimension of $n-1$.
As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is
a two-dimensional subspace, or stated simply, a plane.
In two dimensions, with the variables $x_1$ and $x_2$, the hyperplane is defined as
!bt
\[
\beta_0+\beta_1x_1+\beta_2x_2=0,
\]
!et
In an $n$-dimensional space we have
!bt
\[
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n=0,
\]
!et
With $\hat{x}=\left[x_1,x_2,\dots, x_n\right]$, if the above condition is not met and
!bt
\[
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n<0,
\]
!et
we say that $\hat{x}$ lies on one of the sides of the hyperplane and if
!bt
\[
\beta_0+\beta_1x_1+\beta_2x_2+\dots +\beta_nx_n>0,
\]
!et
then $\hat{x}$ lies on the other side.
!split
===== The two-dimensional case =====
Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional
plane. To separate the two classes of data points, there are many
possible lines (hyperplanes if you prefer a more strict naming)
that could be chosen. Our objective is to find a
plane that has the maximum margin, i.e the maximum distance between
data points of both classes. Maximizing the margin distance provides
some reinforcement so that future data points can be classified with
more confidence.
What a linear classifier attempts to accomplish is to split the
feature space into two half spaces by placing a hyperplane between the
data points. This hyperplane will be our decision boundary. All
points on one side of the plane will belong to class one and all points
on the other side of the plane will belong to the second class two.
Unfortunately there are many ways in which we can place a hyperplane
to divide the data. Below is an example of two candidate hyperplanes
for our data sample.
!split
===== Examples with kernels =====