lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update of sim slides, more to come

Browse Source
This commit is contained in:
mhjensen
2018-11-01 06:40:07 +01:00
parent 0464d31236
commit 49df21377c
12 changed files with 734 additions and 211 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/Programs/DimRed/covariance.py
+2 -2
View File
@@ -13,8 +13,8 @@ def covariance(x, y, n):
n = 10
x = np.random.normal(size=n)
y = x*np.random.normal(size=n)
z = x*x+y*np.random.normal(size=n)
y = np.random.normal(size=n)
z = x*x*x+y*y +0.5*np.random.normal(size=n)
covxx = covariance(x,x,n)
covxy = covariance(x,y,n)
covxz = covariance(x,z,n)
doc/pub/svm/html/svm-bs.html
+127 -37
View File
@@ -40,10 +40,11 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('Support Vector Machines, overarching aims',
2,
None,
'___sec0')]}
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
('SVMs, basics with scikit-learn', 2, None, '___sec1'),
('SVMs, adding polynomial features', 2, None, '___sec2'),
('SVMs, polynomials and kernels', 2, None, '___sec3'),
('SVMs, regression', 2, None, '___sec4')]}
end of tocinfo -->
<body>
@@ -66,6 +67,10 @@ end of tocinfo -->
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">Support Vector Machines, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">SVMs, basics with scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">SVMs, adding polynomial features</a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">SVMs, polynomials and kernels</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">SVMs, regression</a></li>
</ul>
</li>
@@ -99,7 +104,7 @@ end of tocinfo -->
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>May 30, 2018</h4></center> <!-- date -->
<center><h4>Nov 1, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- potential-jumbotron-button -->
@@ -109,44 +114,129 @@ end of tocinfo -->
<h2 id="___sec0" class="anchor">Support Vector Machines, overarching aims </h2>
<p>
A Support Vector Machine (SVM) is a very powerful and versatile
Machine Learning model, capable of performing linear or nonlinear
classification, regression, and even outlier detection. It is one of
the most popular models in Machine Learning, and anyone interested in
Machine Learning should have it in their toolbox. SVMs are
particularly well suited for classification of complex but small-sized or
medium-sized datasets.
<p>
The basic mathematics relies on the definition of hyperplanes and the definition of a <b>margin</b> which separates
classes (in case of classification problems) of variables. It is also used for regression problems.
<p>
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
We distringuish also between linearn and non-linear approaches.
<p>
<b>These notes will be updated shortly with more material</b>
<p>
<!-- !split -->
<h2 id="___sec1" class="anchor">SVMs, basics with scikit-learn </h2>
<p>
The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM
model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect
Iris-Virginica flowers.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVR
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #408080; font-style: italic"># Generate sample data</span>
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sort(<span style="color: #666666">5*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">40</span>,<span style="color: #666666">1</span>), axis<span style="color: #666666">=0</span>)
y <span style="color: #666666">=</span> X<span style="color: #666666">**3</span>
y<span style="color: #666666">=</span>y<span style="color: #666666">.</span>ravel()
<span style="color: #408080; font-style: italic"># Add noise to targets</span>
X[::<span style="color: #666666">4</span>] <span style="color: #666666">+=3*</span>(<span style="color: #666666">0.5</span> <span style="color: #666666">-</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">1</span>))
y[::<span style="color: #666666">5</span>] <span style="color: #666666">+=</span> <span style="color: #666666">50</span> <span style="color: #666666">*</span> (<span style="color: #666666">0.5</span> <span style="color: #666666">-</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">8</span>))
plt<span style="color: #666666">.</span>plot(X,y, <span style="color: #BA2121">&#39;g^&#39;</span>)
<span style="color: #408080; font-style: italic">#SVR Fit</span>
svr_poly <span style="color: #666666">=</span> SVR(kernel<span style="color: #666666">=</span><span style="color: #BA2121">&#39;poly&#39;</span>, C<span style="color: #666666">=1e3</span>, degree<span style="color: #666666">=3</span>)
y_poly <span style="color: #666666">=</span> svr_poly<span style="color: #666666">.</span>fit(X, y)<span style="color: #666666">.</span>predict(X)
<span style="color: #408080; font-style: italic"># Plots</span>
z <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">0</span>, <span style="color: #666666">5</span>, <span style="color: #666666">0.1</span>)
t <span style="color: #666666">=</span> z<span style="color: #666666">**3</span>
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
plt<span style="color: #666666">.</span>plot(z,z<span style="color: #666666">**3</span>, <span style="color: #BA2121">&#39;r--&#39;</span>, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Cubic Function with No Noise&#39;</span>)
lw <span style="color: #666666">=</span> <span style="color: #666666">2</span>
plt<span style="color: #666666">.</span>scatter(X, y, color<span style="color: #666666">=</span><span style="color: #BA2121">&#39;darkorange&#39;</span>, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Gaussian Cubic Noise&#39;</span>)
plt<span style="color: #666666">.</span>plot(X, y_poly, color<span style="color: #666666">=</span><span style="color: #BA2121">&#39;green&#39;</span>, lw<span style="color: #666666">=</span>lw, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Polynomial model&#39;</span>)
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">&#39;data&#39;</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&#39;target&#39;</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&#39;Cubic Gaussian Distribution&#39;</span>)
plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVC
iris <span style="color: #666666">=</span> datasets<span style="color: #666666">.</span>load_iris()
X <span style="color: #666666">=</span> iris[<span style="color: #BA2121">&quot;data&quot;</span>][:, (<span style="color: #666666">2</span>, <span style="color: #666666">3</span>)] <span style="color: #408080; font-style: italic"># petal length, petal width</span>
y <span style="color: #666666">=</span> (iris[<span style="color: #BA2121">&quot;target&quot;</span>] <span style="color: #666666">==</span> <span style="color: #666666">2</span>)<span style="color: #666666">.</span>astype(np<span style="color: #666666">.</span>float64) <span style="color: #408080; font-style: italic"># Iris-Virginica</span>
svm_clf <span style="color: #666666">=</span> Pipeline((
(<span style="color: #BA2121">&quot;scaler&quot;</span>, StandardScaler()),
(<span style="color: #BA2121">&quot;linear_svc&quot;</span>, LinearSVC(C<span style="color: #666666">=1</span>, loss<span style="color: #666666">=</span><span style="color: #BA2121">&quot;hinge&quot;</span>)),
))
svm_clf<span style="color: #666666">.</span>fit(X_scaled, y)
</pre></div>
<p>
Alternatively, you could use the SVC class, using <b>SVC(kernel="linear", C=1)</b>, but it is much slower,
especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier
class, with <b>SGDClassifier(loss="hinge", alpha=1/(m*C))</b>. This applies regular Stochastic
Gradient Descentto train a linear SVM classifier. It does not converge as fast as the
LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core
training), or to handle online classification tasks.
<p>
<!-- !split -->
<h2 id="___sec2" class="anchor">SVMs, adding polynomial features </h2>
<p>
Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets
are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more
features, such as polynomial features. In some cases this can result in a linearly
separable dataset.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> make_moons
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> PolynomialFeatures
polynomial_svm_clf <span style="color: #666666">=</span> Pipeline((
(<span style="color: #BA2121">&quot;poly_features&quot;</span>, PolynomialFeatures(degree<span style="color: #666666">=3</span>)),
(<span style="color: #BA2121">&quot;scaler&quot;</span>, StandardScaler()),
(<span style="color: #BA2121">&quot;svm_clf&quot;</span>, LinearSVC(C<span style="color: #666666">=10</span>, loss<span style="color: #666666">=</span><span style="color: #BA2121">&quot;hinge&quot;</span>))
))
polynomial_svm_clf<span style="color: #666666">.</span>fit(X, y)
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec3" class="anchor">SVMs, polynomials and kernels </h2>
<p>
Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning
algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets,
and with a high polynomial degree it creates a huge number of features, making the model too slow.
Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the
kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many
polynomial features, even with very high-degree polynomials, without actually having to add them. So
there is no combinatorial explosion of the number of features since you don&#8217;t actually add any features.
This trick is implemented by the SVC class.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
poly_kernel_svm_clf <span style="color: #666666">=</span> Pipeline((
(<span style="color: #BA2121">&quot;scaler&quot;</span>, StandardScaler()),
(<span style="color: #BA2121">&quot;svm_clf&quot;</span>, SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">&quot;poly&quot;</span>, degree<span style="color: #666666">=3</span>, coef0<span style="color: #666666">=1</span>, C<span style="color: #666666">=5</span>))
))
poly_kernel_svm_clf<span style="color: #666666">.</span>fit(X, y)
</pre></div>
<p>
This code trains an SVM classifier using a 3rd-degree polynomial kernel.
<p>
<!-- !split -->
<h2 id="___sec4" class="anchor">SVMs, regression </h2>
<p>
You can use Scikit-Learn&#8217;s LinearSVR class to perform linear SVM Regression. The following code
shows how to use the regression option (more material to come)
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVR
svm_reg <span style="color: #666666">=</span> LinearSVR(epsilon<span style="color: #666666">=1.5</span>)
svm_reg<span style="color: #666666">.</span>fit(X, y)
</pre></div>
<p>
To tackle nonlinear regression tasks, you can use a kernelized SVM model
<!-- ------------------- end of main content --------------- -->
doc/pub/svm/html/svm-reveal.html
+120 -33
View File
@@ -132,7 +132,7 @@ td.padding {
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>May 30, 2018</h4></center> <!-- date -->
<center><h4>Nov 1, 2018</h4></center> <!-- date -->
<br>
<p>
@@ -145,43 +145,130 @@ td.padding {
<section>
<h2 id="___sec0">Support Vector Machines, overarching aims </h2>
<p>
A Support Vector Machine (SVM) is a very powerful and versatile
Machine Learning model, capable of performing linear or nonlinear
classification, regression, and even outlier detection. It is one of
the most popular models in Machine Learning, and anyone interested in
Machine Learning should have it in their toolbox. SVMs are
particularly well suited for classification of complex but small-sized or
medium-sized datasets.
<p>
The basic mathematics relies on the definition of hyperplanes and the definition of a <b>margin</b> which separates
classes (in case of classification problems) of variables. It is also used for regression problems.
<p>
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
We distringuish also between linearn and non-linear approaches.
<p>
<b>These notes will be updated shortly with more material</b>
</section>
<section>
<h2 id="___sec1">SVMs, basics with scikit-learn </h2>
<p>
The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM
model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect
Iris-Virginica flowers.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.svm</span> <span style="color: #8B008B; font-weight: bold">import</span> SVR
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #228B22"># Generate sample data</span>
X = np.sort(<span style="color: #B452CD">5</span>*np.random.rand(<span style="color: #B452CD">40</span>,<span style="color: #B452CD">1</span>), axis=<span style="color: #B452CD">0</span>)
y = X**<span style="color: #B452CD">3</span>
y=y.ravel()
<span style="color: #228B22"># Add noise to targets</span>
X[::<span style="color: #B452CD">4</span>] +=<span style="color: #B452CD">3</span>*(<span style="color: #B452CD">0.5</span> - np.random.rand(<span style="color: #B452CD">1</span>))
y[::<span style="color: #B452CD">5</span>] += <span style="color: #B452CD">50</span> * (<span style="color: #B452CD">0.5</span> - np.random.rand(<span style="color: #B452CD">8</span>))
plt.plot(X,y, <span style="color: #CD5555">&#39;g^&#39;</span>)
<span style="color: #228B22">#SVR Fit</span>
svr_poly = SVR(kernel=<span style="color: #CD5555">&#39;poly&#39;</span>, C=<span style="color: #B452CD">1e3</span>, degree=<span style="color: #B452CD">3</span>)
y_poly = svr_poly.fit(X, y).predict(X)
<span style="color: #228B22"># Plots</span>
z = np.arange(<span style="color: #B452CD">0</span>, <span style="color: #B452CD">5</span>, <span style="color: #B452CD">0.1</span>)
t = z**<span style="color: #B452CD">3</span>
fig = plt.figure()
ax = fig.add_subplot(<span style="color: #B452CD">111</span>)
plt.plot(z,z**<span style="color: #B452CD">3</span>, <span style="color: #CD5555">&#39;r--&#39;</span>, label=<span style="color: #CD5555">&#39;Cubic Function with No Noise&#39;</span>)
lw = <span style="color: #B452CD">2</span>
plt.scatter(X, y, color=<span style="color: #CD5555">&#39;darkorange&#39;</span>, label=<span style="color: #CD5555">&#39;Gaussian Cubic Noise&#39;</span>)
plt.plot(X, y_poly, color=<span style="color: #CD5555">&#39;green&#39;</span>, lw=lw, label=<span style="color: #CD5555">&#39;Polynomial model&#39;</span>)
plt.xlabel(<span style="color: #CD5555">&#39;data&#39;</span>)
plt.ylabel(<span style="color: #CD5555">&#39;target&#39;</span>)
plt.title(<span style="color: #CD5555">&#39;Cubic Gaussian Distribution&#39;</span>)
plt.legend()
plt.show()
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn</span> <span style="color: #8B008B; font-weight: bold">import</span> datasets
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.pipeline</span> <span style="color: #8B008B; font-weight: bold">import</span> Pipeline
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> StandardScaler
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.svm</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearSVC
iris = datasets.load_iris()
X = iris[<span style="color: #CD5555">&quot;data&quot;</span>][:, (<span style="color: #B452CD">2</span>, <span style="color: #B452CD">3</span>)] <span style="color: #228B22"># petal length, petal width</span>
y = (iris[<span style="color: #CD5555">&quot;target&quot;</span>] == <span style="color: #B452CD">2</span>).astype(np.float64) <span style="color: #228B22"># Iris-Virginica</span>
svm_clf = Pipeline((
(<span style="color: #CD5555">&quot;scaler&quot;</span>, StandardScaler()),
(<span style="color: #CD5555">&quot;linear_svc&quot;</span>, LinearSVC(C=<span style="color: #B452CD">1</span>, loss=<span style="color: #CD5555">&quot;hinge&quot;</span>)),
))
svm_clf.fit(X_scaled, y)
</pre></div>
<p>
Alternatively, you could use the SVC class, using <b>SVC(kernel="linear", C=1)</b>, but it is much slower,
especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier
class, with <b>SGDClassifier(loss="hinge", alpha=1/(m*C))</b>. This applies regular Stochastic
Gradient Descentto train a linear SVM classifier. It does not converge as fast as the
LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core
training), or to handle online classification tasks.
</section>
<section>
<h2 id="___sec2">SVMs, adding polynomial features </h2>
<p>
Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets
are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more
features, such as polynomial features. In some cases this can result in a linearly
separable dataset.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.datasets</span> <span style="color: #8B008B; font-weight: bold">import</span> make_moons
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.pipeline</span> <span style="color: #8B008B; font-weight: bold">import</span> Pipeline
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> PolynomialFeatures
polynomial_svm_clf = Pipeline((
(<span style="color: #CD5555">&quot;poly_features&quot;</span>, PolynomialFeatures(degree=<span style="color: #B452CD">3</span>)),
(<span style="color: #CD5555">&quot;scaler&quot;</span>, StandardScaler()),
(<span style="color: #CD5555">&quot;svm_clf&quot;</span>, LinearSVC(C=<span style="color: #B452CD">10</span>, loss=<span style="color: #CD5555">&quot;hinge&quot;</span>))
))
polynomial_svm_clf.fit(X, y)
</pre></div>
</section>
<section>
<h2 id="___sec3">SVMs, polynomials and kernels </h2>
<p>
Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning
algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets,
and with a high polynomial degree it creates a huge number of features, making the model too slow.
Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the
kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many
polynomial features, even with very high-degree polynomials, without actually having to add them. So
there is no combinatorial explosion of the number of features since you don&#8217;t actually add any features.
This trick is implemented by the SVC class.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.svm</span> <span style="color: #8B008B; font-weight: bold">import</span> SVC
poly_kernel_svm_clf = Pipeline((
(<span style="color: #CD5555">&quot;scaler&quot;</span>, StandardScaler()),
(<span style="color: #CD5555">&quot;svm_clf&quot;</span>, SVC(kernel=<span style="color: #CD5555">&quot;poly&quot;</span>, degree=<span style="color: #B452CD">3</span>, coef0=<span style="color: #B452CD">1</span>, C=<span style="color: #B452CD">5</span>))
))
poly_kernel_svm_clf.fit(X, y)
</pre></div>
<p>
This code trains an SVM classifier using a 3rd-degree polynomial kernel.
</section>
<section>
<h2 id="___sec4">SVMs, regression </h2>
<p>
You can use Scikit-Learn&#8217;s LinearSVR class to perform linear SVM Regression. The following code
shows how to use the regression option (more material to come)
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.svm</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearSVR
svm_reg = LinearSVR(epsilon=<span style="color: #B452CD">1.5</span>)
svm_reg.fit(X, y)
</pre></div>
<p>
To tackle nonlinear regression tasks, you can use a kernelized SVM model
</section>
doc/pub/svm/html/svm-solarized.html
+123 -37
View File
@@ -34,10 +34,11 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('Support Vector Machines, overarching aims',
2,
None,
'___sec0')]}
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
('SVMs, basics with scikit-learn', 2, None, '___sec1'),
('SVMs, adding polynomial features', 2, None, '___sec2'),
('SVMs, polynomials and kernels', 2, None, '___sec3'),
('SVMs, regression', 2, None, '___sec4')]}
end of tocinfo -->
<body>
@@ -63,51 +64,136 @@ end of tocinfo -->
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>May 30, 2018</h4></center> <!-- date -->
<center><h4>Nov 1, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Support Vector Machines, overarching aims </h2>
<p>
A Support Vector Machine (SVM) is a very powerful and versatile
Machine Learning model, capable of performing linear or nonlinear
classification, regression, and even outlier detection. It is one of
the most popular models in Machine Learning, and anyone interested in
Machine Learning should have it in their toolbox. SVMs are
particularly well suited for classification of complex but small-sized or
medium-sized datasets.
<p>
The basic mathematics relies on the definition of hyperplanes and the definition of a <b>margin</b> which separates
classes (in case of classification problems) of variables. It is also used for regression problems.
<p>
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
We distringuish also between linearn and non-linear approaches.
<p>
<b>These notes will be updated shortly with more material</b>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">SVMs, basics with scikit-learn </h2>
<p>
The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM
model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect
Iris-Virginica flowers.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.svm</span> <span style="color: #8B008B; font-weight: bold">import</span> SVR
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #228B22"># Generate sample data</span>
X = np.sort(<span style="color: #B452CD">5</span>*np.random.rand(<span style="color: #B452CD">40</span>,<span style="color: #B452CD">1</span>), axis=<span style="color: #B452CD">0</span>)
y = X**<span style="color: #B452CD">3</span>
y=y.ravel()
<span style="color: #228B22"># Add noise to targets</span>
X[::<span style="color: #B452CD">4</span>] +=<span style="color: #B452CD">3</span>*(<span style="color: #B452CD">0.5</span> - np.random.rand(<span style="color: #B452CD">1</span>))
y[::<span style="color: #B452CD">5</span>] += <span style="color: #B452CD">50</span> * (<span style="color: #B452CD">0.5</span> - np.random.rand(<span style="color: #B452CD">8</span>))
plt.plot(X,y, <span style="color: #CD5555">&#39;g^&#39;</span>)
<span style="color: #228B22">#SVR Fit</span>
svr_poly = SVR(kernel=<span style="color: #CD5555">&#39;poly&#39;</span>, C=<span style="color: #B452CD">1e3</span>, degree=<span style="color: #B452CD">3</span>)
y_poly = svr_poly.fit(X, y).predict(X)
<span style="color: #228B22"># Plots</span>
z = np.arange(<span style="color: #B452CD">0</span>, <span style="color: #B452CD">5</span>, <span style="color: #B452CD">0.1</span>)
t = z**<span style="color: #B452CD">3</span>
fig = plt.figure()
ax = fig.add_subplot(<span style="color: #B452CD">111</span>)
plt.plot(z,z**<span style="color: #B452CD">3</span>, <span style="color: #CD5555">&#39;r--&#39;</span>, label=<span style="color: #CD5555">&#39;Cubic Function with No Noise&#39;</span>)
lw = <span style="color: #B452CD">2</span>
plt.scatter(X, y, color=<span style="color: #CD5555">&#39;darkorange&#39;</span>, label=<span style="color: #CD5555">&#39;Gaussian Cubic Noise&#39;</span>)
plt.plot(X, y_poly, color=<span style="color: #CD5555">&#39;green&#39;</span>, lw=lw, label=<span style="color: #CD5555">&#39;Polynomial model&#39;</span>)
plt.xlabel(<span style="color: #CD5555">&#39;data&#39;</span>)
plt.ylabel(<span style="color: #CD5555">&#39;target&#39;</span>)
plt.title(<span style="color: #CD5555">&#39;Cubic Gaussian Distribution&#39;</span>)
plt.legend()
plt.show()
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn</span> <span style="color: #8B008B; font-weight: bold">import</span> datasets
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.pipeline</span> <span style="color: #8B008B; font-weight: bold">import</span> Pipeline
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> StandardScaler
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.svm</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearSVC
iris = datasets.load_iris()
X = iris[<span style="color: #CD5555">&quot;data&quot;</span>][:, (<span style="color: #B452CD">2</span>, <span style="color: #B452CD">3</span>)] <span style="color: #228B22"># petal length, petal width</span>
y = (iris[<span style="color: #CD5555">&quot;target&quot;</span>] == <span style="color: #B452CD">2</span>).astype(np.float64) <span style="color: #228B22"># Iris-Virginica</span>
svm_clf = Pipeline((
(<span style="color: #CD5555">&quot;scaler&quot;</span>, StandardScaler()),
(<span style="color: #CD5555">&quot;linear_svc&quot;</span>, LinearSVC(C=<span style="color: #B452CD">1</span>, loss=<span style="color: #CD5555">&quot;hinge&quot;</span>)),
))
svm_clf.fit(X_scaled, y)
</pre></div>
<p>
Alternatively, you could use the SVC class, using <b>SVC(kernel="linear", C=1)</b>, but it is much slower,
especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier
class, with <b>SGDClassifier(loss="hinge", alpha=1/(m*C))</b>. This applies regular Stochastic
Gradient Descentto train a linear SVM classifier. It does not converge as fast as the
LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core
training), or to handle online classification tasks.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">SVMs, adding polynomial features </h2>
<p>
Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets
are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more
features, such as polynomial features. In some cases this can result in a linearly
separable dataset.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.datasets</span> <span style="color: #8B008B; font-weight: bold">import</span> make_moons
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.pipeline</span> <span style="color: #8B008B; font-weight: bold">import</span> Pipeline
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> PolynomialFeatures
polynomial_svm_clf = Pipeline((
(<span style="color: #CD5555">&quot;poly_features&quot;</span>, PolynomialFeatures(degree=<span style="color: #B452CD">3</span>)),
(<span style="color: #CD5555">&quot;scaler&quot;</span>, StandardScaler()),
(<span style="color: #CD5555">&quot;svm_clf&quot;</span>, LinearSVC(C=<span style="color: #B452CD">10</span>, loss=<span style="color: #CD5555">&quot;hinge&quot;</span>))
))
polynomial_svm_clf.fit(X, y)
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">SVMs, polynomials and kernels </h2>
<p>
Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning
algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets,
and with a high polynomial degree it creates a huge number of features, making the model too slow.
Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the
kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many
polynomial features, even with very high-degree polynomials, without actually having to add them. So
there is no combinatorial explosion of the number of features since you don&#8217;t actually add any features.
This trick is implemented by the SVC class.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.svm</span> <span style="color: #8B008B; font-weight: bold">import</span> SVC
poly_kernel_svm_clf = Pipeline((
(<span style="color: #CD5555">&quot;scaler&quot;</span>, StandardScaler()),
(<span style="color: #CD5555">&quot;svm_clf&quot;</span>, SVC(kernel=<span style="color: #CD5555">&quot;poly&quot;</span>, degree=<span style="color: #B452CD">3</span>, coef0=<span style="color: #B452CD">1</span>, C=<span style="color: #B452CD">5</span>))
))
poly_kernel_svm_clf.fit(X, y)
</pre></div>
<p>
This code trains an SVM classifier using a 3rd-degree polynomial kernel.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">SVMs, regression </h2>
<p>
You can use Scikit-Learn&#8217;s LinearSVR class to perform linear SVM Regression. The following code
shows how to use the regression option (more material to come)
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.svm</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearSVR
svm_reg = LinearSVR(epsilon=<span style="color: #B452CD">1.5</span>)
svm_reg.fit(X, y)
</pre></div>
<p>
To tackle nonlinear regression tasks, you can use a kernelized SVM model
<!-- ------------------- end of main content --------------- -->
doc/pub/svm/html/svm.html
+123 -37
View File
@@ -39,10 +39,11 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('Support Vector Machines, overarching aims',
2,
None,
'___sec0')]}
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
('SVMs, basics with scikit-learn', 2, None, '___sec1'),
('SVMs, adding polynomial features', 2, None, '___sec2'),
('SVMs, polynomials and kernels', 2, None, '___sec3'),
('SVMs, regression', 2, None, '___sec4')]}
end of tocinfo -->
<body>
@@ -68,51 +69,136 @@ end of tocinfo -->
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>May 30, 2018</h4></center> <!-- date -->
<center><h4>Nov 1, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Support Vector Machines, overarching aims </h2>
<p>
A Support Vector Machine (SVM) is a very powerful and versatile
Machine Learning model, capable of performing linear or nonlinear
classification, regression, and even outlier detection. It is one of
the most popular models in Machine Learning, and anyone interested in
Machine Learning should have it in their toolbox. SVMs are
particularly well suited for classification of complex but small-sized or
medium-sized datasets.
<p>
The basic mathematics relies on the definition of hyperplanes and the definition of a <b>margin</b> which separates
classes (in case of classification problems) of variables. It is also used for regression problems.
<p>
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
We distringuish also between linearn and non-linear approaches.
<p>
<b>These notes will be updated shortly with more material</b>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">SVMs, basics with scikit-learn </h2>
<p>
The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM
model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect
Iris-Virginica flowers.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVR
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #408080; font-style: italic"># Generate sample data</span>
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sort(<span style="color: #666666">5*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">40</span>,<span style="color: #666666">1</span>), axis<span style="color: #666666">=0</span>)
y <span style="color: #666666">=</span> X<span style="color: #666666">**3</span>
y<span style="color: #666666">=</span>y<span style="color: #666666">.</span>ravel()
<span style="color: #408080; font-style: italic"># Add noise to targets</span>
X[::<span style="color: #666666">4</span>] <span style="color: #666666">+=3*</span>(<span style="color: #666666">0.5</span> <span style="color: #666666">-</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">1</span>))
y[::<span style="color: #666666">5</span>] <span style="color: #666666">+=</span> <span style="color: #666666">50</span> <span style="color: #666666">*</span> (<span style="color: #666666">0.5</span> <span style="color: #666666">-</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">8</span>))
plt<span style="color: #666666">.</span>plot(X,y, <span style="color: #BA2121">&#39;g^&#39;</span>)
<span style="color: #408080; font-style: italic">#SVR Fit</span>
svr_poly <span style="color: #666666">=</span> SVR(kernel<span style="color: #666666">=</span><span style="color: #BA2121">&#39;poly&#39;</span>, C<span style="color: #666666">=1e3</span>, degree<span style="color: #666666">=3</span>)
y_poly <span style="color: #666666">=</span> svr_poly<span style="color: #666666">.</span>fit(X, y)<span style="color: #666666">.</span>predict(X)
<span style="color: #408080; font-style: italic"># Plots</span>
z <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">0</span>, <span style="color: #666666">5</span>, <span style="color: #666666">0.1</span>)
t <span style="color: #666666">=</span> z<span style="color: #666666">**3</span>
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
plt<span style="color: #666666">.</span>plot(z,z<span style="color: #666666">**3</span>, <span style="color: #BA2121">&#39;r--&#39;</span>, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Cubic Function with No Noise&#39;</span>)
lw <span style="color: #666666">=</span> <span style="color: #666666">2</span>
plt<span style="color: #666666">.</span>scatter(X, y, color<span style="color: #666666">=</span><span style="color: #BA2121">&#39;darkorange&#39;</span>, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Gaussian Cubic Noise&#39;</span>)
plt<span style="color: #666666">.</span>plot(X, y_poly, color<span style="color: #666666">=</span><span style="color: #BA2121">&#39;green&#39;</span>, lw<span style="color: #666666">=</span>lw, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Polynomial model&#39;</span>)
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">&#39;data&#39;</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&#39;target&#39;</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&#39;Cubic Gaussian Distribution&#39;</span>)
plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVC
iris <span style="color: #666666">=</span> datasets<span style="color: #666666">.</span>load_iris()
X <span style="color: #666666">=</span> iris[<span style="color: #BA2121">&quot;data&quot;</span>][:, (<span style="color: #666666">2</span>, <span style="color: #666666">3</span>)] <span style="color: #408080; font-style: italic"># petal length, petal width</span>
y <span style="color: #666666">=</span> (iris[<span style="color: #BA2121">&quot;target&quot;</span>] <span style="color: #666666">==</span> <span style="color: #666666">2</span>)<span style="color: #666666">.</span>astype(np<span style="color: #666666">.</span>float64) <span style="color: #408080; font-style: italic"># Iris-Virginica</span>
svm_clf <span style="color: #666666">=</span> Pipeline((
(<span style="color: #BA2121">&quot;scaler&quot;</span>, StandardScaler()),
(<span style="color: #BA2121">&quot;linear_svc&quot;</span>, LinearSVC(C<span style="color: #666666">=1</span>, loss<span style="color: #666666">=</span><span style="color: #BA2121">&quot;hinge&quot;</span>)),
))
svm_clf<span style="color: #666666">.</span>fit(X_scaled, y)
</pre></div>
<p>
Alternatively, you could use the SVC class, using <b>SVC(kernel="linear", C=1)</b>, but it is much slower,
especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier
class, with <b>SGDClassifier(loss="hinge", alpha=1/(m*C))</b>. This applies regular Stochastic
Gradient Descentto train a linear SVM classifier. It does not converge as fast as the
LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core
training), or to handle online classification tasks.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">SVMs, adding polynomial features </h2>
<p>
Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets
are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more
features, such as polynomial features. In some cases this can result in a linearly
separable dataset.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> make_moons
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> PolynomialFeatures
polynomial_svm_clf <span style="color: #666666">=</span> Pipeline((
(<span style="color: #BA2121">&quot;poly_features&quot;</span>, PolynomialFeatures(degree<span style="color: #666666">=3</span>)),
(<span style="color: #BA2121">&quot;scaler&quot;</span>, StandardScaler()),
(<span style="color: #BA2121">&quot;svm_clf&quot;</span>, LinearSVC(C<span style="color: #666666">=10</span>, loss<span style="color: #666666">=</span><span style="color: #BA2121">&quot;hinge&quot;</span>))
))
polynomial_svm_clf<span style="color: #666666">.</span>fit(X, y)
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">SVMs, polynomials and kernels </h2>
<p>
Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning
algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets,
and with a high polynomial degree it creates a huge number of features, making the model too slow.
Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the
kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many
polynomial features, even with very high-degree polynomials, without actually having to add them. So
there is no combinatorial explosion of the number of features since you don&#8217;t actually add any features.
This trick is implemented by the SVC class.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
poly_kernel_svm_clf <span style="color: #666666">=</span> Pipeline((
(<span style="color: #BA2121">&quot;scaler&quot;</span>, StandardScaler()),
(<span style="color: #BA2121">&quot;svm_clf&quot;</span>, SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">&quot;poly&quot;</span>, degree<span style="color: #666666">=3</span>, coef0<span style="color: #666666">=1</span>, C<span style="color: #666666">=5</span>))
))
poly_kernel_svm_clf<span style="color: #666666">.</span>fit(X, y)
</pre></div>
<p>
This code trains an SVM classifier using a 3rd-degree polynomial kernel.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">SVMs, regression </h2>
<p>
You can use Scikit-Learn&#8217;s LinearSVR class to perform linear SVM Regression. The following code
shows how to use the regression option (more material to come)
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVR
svm_reg <span style="color: #666666">=</span> LinearSVR(epsilon<span style="color: #666666">=1.5</span>)
svm_reg<span style="color: #666666">.</span>fit(X, y)
</pre></div>
<p>
To tackle nonlinear regression tasks, you can use a kernelized SVM model
<!-- ------------------- end of main content --------------- -->
doc/pub/svm/ipynb/ipynb-svm-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/svm/ipynb/svm.ipynb
+133 -31
View File
@@ -10,14 +10,35 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **May 30, 2018**\n",
"Date: **Nov 1, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
"\n",
"\n",
"## Support Vector Machines, overarching aims\n",
"\n",
"## Support Vector Machines, overarching aims"
"A Support Vector Machine (SVM) is a very powerful and versatile\n",
"Machine Learning model, capable of performing linear or nonlinear\n",
"classification, regression, and even outlier detection. It is one of\n",
"the most popular models in Machine Learning, and anyone interested in\n",
"Machine Learning should have it in their toolbox. SVMs are\n",
"particularly well suited for classification of complex but small-sized or\n",
"medium-sized datasets. \n",
"\n",
"The basic mathematics relies on the definition of hyperplanes and the definition of a **margin** which separates\n",
"classes (in case of classification problems) of variables. It is also used for regression problems.\n",
"\n",
"With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.\n",
"We distringuish also between linearn and non-linear approaches.\n",
"\n",
"**These notes will be updated shortly with more material**\n",
"\n",
"## SVMs, basics with scikit-learn\n",
"\n",
"The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM\n",
"model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect\n",
"Iris-Virginica flowers."
]
},
{
@@ -28,41 +49,122 @@
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
"import numpy as np\n",
"from sklearn.svm import SVR\n",
"import matplotlib.pyplot as plt\n",
"from sklearn import datasets\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import StandardScaler\n",
"from sklearn.svm import LinearSVC\n",
"iris = datasets.load_iris()\n",
"X = iris[\"data\"][:, (2, 3)] # petal length, petal width\n",
"y = (iris[\"target\"] == 2).astype(np.float64) # Iris-Virginica\n",
"svm_clf = Pipeline((\n",
"(\"scaler\", StandardScaler()),\n",
"(\"linear_svc\", LinearSVC(C=1, loss=\"hinge\")),\n",
"))\n",
"svm_clf.fit(X_scaled, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternatively, you could use the SVC class, using **SVC(kernel=\"linear\", C=1)**, but it is much slower,\n",
"especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier\n",
"class, with **SGDClassifier(loss=\"hinge\", alpha=1/(m*C))**. This applies regular Stochastic\n",
"Gradient Descentto train a linear SVM classifier. It does not converge as fast as the\n",
"LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core\n",
"training), or to handle online classification tasks.\n",
"\n",
"# Generate sample data\n",
"X = np.sort(5*np.random.rand(40,1), axis=0)\n",
"y = X**3\n",
"y=y.ravel()\n",
"## SVMs, adding polynomial features\n",
"\n",
"# Add noise to targets\n",
"X[::4] +=3*(0.5 - np.random.rand(1))\n",
"y[::5] += 50 * (0.5 - np.random.rand(8))\n",
"Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets\n",
"are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more\n",
"features, such as polynomial features. In some cases this can result in a linearly\n",
"separable dataset."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from sklearn.datasets import make_moons\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import PolynomialFeatures\n",
"polynomial_svm_clf = Pipeline((\n",
"(\"poly_features\", PolynomialFeatures(degree=3)),\n",
"(\"scaler\", StandardScaler()),\n",
"(\"svm_clf\", LinearSVC(C=10, loss=\"hinge\"))\n",
"))\n",
"polynomial_svm_clf.fit(X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## SVMs, polynomials and kernels\n",
"\n",
"plt.plot(X,y, 'g^')\n",
"Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning\n",
"algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets,\n",
"and with a high polynomial degree it creates a huge number of features, making the model too slow.\n",
"Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the\n",
"kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many\n",
"polynomial features, even with very high-degree polynomials, without actually having to add them. So\n",
"there is no combinatorial explosion of the number of features since you don’t actually add any features.\n",
"This trick is implemented by the SVC class."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from sklearn.svm import SVC\n",
"poly_kernel_svm_clf = Pipeline((\n",
"(\"scaler\", StandardScaler()),\n",
"(\"svm_clf\", SVC(kernel=\"poly\", degree=3, coef0=1, C=5))\n",
"))\n",
"poly_kernel_svm_clf.fit(X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This code trains an SVM classifier using a 3rd-degree polynomial kernel.\n",
"\n",
"#SVR Fit\n",
"svr_poly = SVR(kernel='poly', C=1e3, degree=3)\n",
"y_poly = svr_poly.fit(X, y).predict(X)\n",
"\n",
"# Plots\n",
"z = np.arange(0, 5, 0.1)\n",
"t = z**3\n",
"fig = plt.figure()\n",
"ax = fig.add_subplot(111)\n",
"plt.plot(z,z**3, 'r--', label='Cubic Function with No Noise')\n",
"lw = 2\n",
"plt.scatter(X, y, color='darkorange', label='Gaussian Cubic Noise')\n",
"plt.plot(X, y_poly, color='green', lw=lw, label='Polynomial model')\n",
"plt.xlabel('data')\n",
"plt.ylabel('target')\n",
"plt.title('Cubic Gaussian Distribution')\n",
"plt.legend()\n",
"plt.show()"
"## SVMs, regression\n",
"\n",
"You can use Scikit-Learn’s LinearSVR class to perform linear SVM Regression. The following code\n",
"shows how to use the regression option (more material to come)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from sklearn.svm import LinearSVR\n",
"svm_reg = LinearSVR(epsilon=1.5)\n",
"svm_reg.fit(X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To tackle nonlinear regression tasks, you can use a kernelized SVM model"
]
}
],
doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf
BIN
View File
Binary file not shown.
doc/pub/svm/pdf/svm-beamer.pdf
BIN
View File
Binary file not shown.
doc/pub/svm/pdf/svm-minted.pdf
BIN
View File
Binary file not shown.
doc/src/DecisionTrees/DecisionTrees.do.txt
+9 -1
View File
@@ -6,7 +6,15 @@ DATE: today
!split
===== Decision trees, overarching aims =====
!bblock
Add text about decision trees and include about random forests (use Ising model classification)
Decision trees are supervised learning algorithms used for both,
classification and regression tasks where we will concentrate on
classification in this first part of our decision tree tutorial.
Decision trees are assigned to the information based learning
algorithms which use different measures of information gain for
learning. We can use decision trees for issues where we have
continuous but also categorical input and target features.
!eblock
!split
doc/src/SupportVMachines/svm.do.txt
+97 -33
View File
@@ -2,42 +2,106 @@ TITLE: Data Analysis and Machine Learning: Support Vector Machines
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
DATE: today
!split
===== Support Vector Machines, overarching aims =====
A Support Vector Machine (SVM) is a very powerful and versatile
Machine Learning model, capable of performing linear or nonlinear
classification, regression, and even outlier detection. It is one of
the most popular models in Machine Learning, and anyone interested in
Machine Learning should have it in their toolbox. SVMs are
particularly well suited for classification of complex but small-sized or
medium-sized datasets.
The basic mathematics relies on the definition of hyperplanes and the definition of a _margin_ which separates
classes (in case of classification problems) of variables. It is also used for regression problems.
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
We distringuish also between linearn and non-linear approaches.
_These notes will be updated shortly with more material_
!split
===== SVMs, basics with scikit-learn =====
The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM
model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect
Iris-Virginica flowers.
!bc pycod
import numpy as np
from sklearn.svm import SVR
import matplotlib.pyplot as plt
# Generate sample data
X = np.sort(5*np.random.rand(40,1), axis=0)
y = X**3
y=y.ravel()
# Add noise to targets
X[::4] +=3*(0.5 - np.random.rand(1))
y[::5] += 50 * (0.5 - np.random.rand(8))
plt.plot(X,y, 'g^')
#SVR Fit
svr_poly = SVR(kernel='poly', C=1e3, degree=3)
y_poly = svr_poly.fit(X, y).predict(X)
# Plots
z = np.arange(0, 5, 0.1)
t = z**3
fig = plt.figure()
ax = fig.add_subplot(111)
plt.plot(z,z**3, 'r--', label='Cubic Function with No Noise')
lw = 2
plt.scatter(X, y, color='darkorange', label='Gaussian Cubic Noise')
plt.plot(X, y_poly, color='green', lw=lw, label='Polynomial model')
plt.xlabel('data')
plt.ylabel('target')
plt.title('Cubic Gaussian Distribution')
plt.legend()
plt.show()
from sklearn import datasets
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
iris = datasets.load_iris()
X = iris["data"][:, (2, 3)] # petal length, petal width
y = (iris["target"] == 2).astype(np.float64) # Iris-Virginica
svm_clf = Pipeline((
("scaler", StandardScaler()),
("linear_svc", LinearSVC(C=1, loss="hinge")),
))
svm_clf.fit(X_scaled, y)
!ec
Alternatively, you could use the SVC class, using _SVC(kernel="linear", C=1)_, but it is much slower,
especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier
class, with _SGDClassifier(loss="hinge", alpha=1/(m*C))_. This applies regular Stochastic
Gradient Descentto train a linear SVM classifier. It does not converge as fast as the
LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core
training), or to handle online classification tasks.
!split
===== SVMs, adding polynomial features =====
Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets
are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more
features, such as polynomial features. In some cases this can result in a linearly
separable dataset.
!bc pycod
from sklearn.datasets import make_moons
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
polynomial_svm_clf = Pipeline((
("poly_features", PolynomialFeatures(degree=3)),
("scaler", StandardScaler()),
("svm_clf", LinearSVC(C=10, loss="hinge"))
))
polynomial_svm_clf.fit(X, y)
!ec
!split
===== SVMs, polynomials and kernels =====
Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning
algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets,
and with a high polynomial degree it creates a huge number of features, making the model too slow.
Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the
kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many
polynomial features, even with very high-degree polynomials, without actually having to add them. So
there is no combinatorial explosion of the number of features since you don’t actually add any features.
This trick is implemented by the SVC class.
!bc pycod
from sklearn.svm import SVC
poly_kernel_svm_clf = Pipeline((
("scaler", StandardScaler()),
("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5))
))
poly_kernel_svm_clf.fit(X, y)
!ec
This code trains an SVM classifier using a 3rd-degree polynomial kernel.
!split
===== SVMs, regression =====
You can use Scikit-Learn’s LinearSVR class to perform linear SVM Regression. The following code
shows how to use the regression option (more material to come)
!bc pycod
from sklearn.svm import LinearSVR
svm_reg = LinearSVR(epsilon=1.5)
svm_reg.fit(X, y)
!ec
To tackle nonlinear regression tasks, you can use a kernelized SVM model