lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating week 46

Browse Source
This commit is contained in:
mhjensen
2020-11-08 22:48:11 +01:00
parent 3e13bb0d64
commit 489c98a82f
34 changed files with 2834 additions and 2713 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week46/html/week46-reveal.html
+56 -40
View File
@@ -3,9 +3,9 @@
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 46: Support Vector Machines">
<meta name="description" content="Week 46: Gradient Boosting Summary and Support Vector Machines">
<title>Week 46: Support Vector Machines</title>
<title>Week 46: Gradient Boosting Summary and Support Vector Machines</title>
@@ -132,7 +132,7 @@ MathJax.Hub.Config({
<center><h1 style="text-align: center;">Week 46: Support Vector Machines</h1></center> <!-- document title -->
<center><h1 style="text-align: center;">Week 46: Gradient Boosting Summary and Support Vector Machines</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<center><h4>Nov 8, 2020</h4></center> <!-- date -->
<br>
<p>
@@ -159,7 +159,23 @@ MathJax.Hub.Config({
<section>
<h2 id="___sec0">Support Vector Machines, overarching aims </h2>
<h2 id="___sec0">Overview of week 46 </h2>
<ul>
<p><li> <b>Thursday</b>: Summary of Gradient Boosting and further examples of applications.</li>
<p><li> <b>Friday</b>: Support Vector Machines, classification and regression</li>
</ul>
<p>
Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) of Hastie et al contains also a good discussion.
<p>
<a href="https://www.youtube.com/watch?v=efR1C6CvhmE&ab_channel=StatQuestwithJoshStarmer" target="_blank">Overview of Support Vector Machines</a>. see also <a href="https://www.youtube.com/watch?v=N1vOgolbjSc&ab_channel=AliceZhao" target="_blank">this video</a>.
</section>
<section>
<h2 id="___sec1">Support Vector Machines, overarching aims </h2>
<p>
A Support Vector Machine (SVM) is a very powerful and versatile
@@ -192,7 +208,7 @@ unlikely that we can separate classes easily by say straight lines.
<section>
<h2 id="___sec1">Hyperplanes and all that </h2>
<h2 id="___sec2">Hyperplanes and all that </h2>
<p>
The theory behind support vector machines (SVM hereafter) is based on
@@ -245,9 +261,9 @@ lin_clf.fit(X_scaled, y)
svm_clf.fit(X_scaled, y)
sgd_clf.fit(X_scaled, y)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;LinearSVC: &quot;</span>, lin_clf.intercept_, lin_clf.coef_)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;SVC: &quot;</span>, svm_clf.intercept_, svm_clf.coef_)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;SGDClassifier(alpha={:.5f}):&quot;</span>.format(sgd_clf.alpha), sgd_clf.intercept_, sgd_clf.coef_)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;LinearSVC: &quot;</span>, lin_clf.intercept_, lin_clf.coef_)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;SVC: &quot;</span>, svm_clf.intercept_, svm_clf.coef_)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;SGDClassifier(alpha={:.5f}):&quot;</span>.format(sgd_clf.alpha), sgd_clf.intercept_, sgd_clf.coef_)
<span style="color: #228B22"># Compute the slope and bias of each decision boundary</span>
w1 = -lin_clf.coef_[<span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span>]/lin_clf.coef_[<span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>]
@@ -280,7 +296,7 @@ plt.show()
<section>
<h2 id="___sec2">What is a hyperplane? </h2>
<h2 id="___sec3">What is a hyperplane? </h2>
<p>
The aim of the SVM algorithm is to find a hyperplane in a
@@ -315,7 +331,7 @@ $$
<section>
<h2 id="___sec3">A \( p \)-dimensional space of features </h2>
<h2 id="___sec4">A \( p \)-dimensional space of features </h2>
<p>
We limit ourselves to two classes of outputs \( y_i \) and assign these classes the values \( y_i = \pm 1 \).
@@ -367,7 +383,7 @@ When we try to separate hyperplanes, if it exists, we can use it to construct a
<section>
<h2 id="___sec4">The two-dimensional case </h2>
<h2 id="___sec5">The two-dimensional case </h2>
<p>
Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional
@@ -394,7 +410,7 @@ for our data sample.
<section>
<h2 id="___sec5">Getting into the details </h2>
<h2 id="___sec6">Getting into the details </h2>
<p>
Let us define the function
@@ -420,7 +436,7 @@ $$
<section>
<h2 id="___sec6">First attempt at a minimization approach </h2>
<h2 id="___sec7">First attempt at a minimization approach </h2>
<p>
How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could
@@ -451,7 +467,7 @@ $$
<section>
<h2 id="___sec7">Solving the equations </h2>
<h2 id="___sec8">Solving the equations </h2>
<p>
We can now use the Newton-Raphson method or different variants of the gradient descent family (from plain gradient descent to various stochastic gradient descent approaches) to solve the equations
@@ -473,7 +489,7 @@ where \( \eta \) is our by now well-known learning rate.
<section>
<h2 id="___sec8">Code Example </h2>
<h2 id="___sec9">Code Example </h2>
<p>
The equations we discussed above can be coded rather easily (the
@@ -488,7 +504,7 @@ regression). We are going to set up a simple case with two classes only and we w
<section>
<h2 id="___sec9">Problems with the Simpler Approach </h2>
<h2 id="___sec10">Problems with the Simpler Approach </h2>
<p>
There are however problems with this approach, although it looks
@@ -504,7 +520,7 @@ at all.
<section>
<h2 id="___sec10">A better approach </h2>
<h2 id="___sec11">A better approach </h2>
<p>
A better approach is rather to try to define a large margin between
@@ -554,7 +570,7 @@ about Lagrangian multipliers.
<section>
<h2 id="___sec11">A quick Reminder on Lagrangian Multipliers </h2>
<h2 id="___sec12">A quick Reminder on Lagrangian Multipliers </h2>
<p>
Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an
@@ -616,7 +632,7 @@ Then \( dz \) is no longer arbitrary.
<section>
<h2 id="___sec12">Adding the Multiplier </h2>
<h2 id="___sec13">Adding the Multiplier </h2>
<p>
However, we can add to
@@ -671,7 +687,7 @@ $$
<section>
<h2 id="___sec13">Setting up the Problem </h2>
<h2 id="___sec14">Setting up the Problem </h2>
In order to solve the above problem, we define the following Lagrangian function to be minimized
<p>&nbsp;<br>
$$
@@ -723,7 +739,7 @@ When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support
<section>
<h2 id="___sec14">The problem to solve </h2>
<h2 id="___sec15">The problem to solve </h2>
<p>
We can rewrite
@@ -751,7 +767,7 @@ subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vec
<section>
<h2 id="___sec15">The last steps </h2>
<h2 id="___sec16">The last steps </h2>
<p>
Solving the above problem, yields the values of \( \lambda_i \).
@@ -795,7 +811,7 @@ Below we discuss how to find the optimal values of \( \lambda_i \). Before we pr
<section>
<h2 id="___sec16">A soft classifier </h2>
<h2 id="___sec17">A soft classifier </h2>
<p>
Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
@@ -834,7 +850,7 @@ misclassifications.
<section>
<h2 id="___sec17">Soft optmization problem </h2>
<h2 id="___sec18">Soft optmization problem </h2>
<p>
This has in turn the consequences that we change our optmization problem to finding the minimum of
@@ -906,7 +922,7 @@ $$
<section>
<h2 id="___sec18">Kernels and non-linearity </h2>
<h2 id="___sec19">Kernels and non-linearity </h2>
<p>
The cases we have studied till now, were all characterized by two classes
@@ -954,7 +970,7 @@ y = np.array([<span style="color: #B452CD">0</span>, <span style="color: #B452CD
plt.figure(figsize=(<span style="color: #B452CD">11</span>, <span style="color: #B452CD">4</span>))
plt.subplot(<span style="color: #B452CD">121</span>)
plt.grid(<span style="color: #658b00">True</span>, which=<span style="color: #CD5555">&#39;both&#39;</span>)
plt.grid(<span style="color: #8B008B; font-weight: bold">True</span>, which=<span style="color: #CD5555">&#39;both&#39;</span>)
plt.axhline(y=<span style="color: #B452CD">0</span>, color=<span style="color: #CD5555">&#39;k&#39;</span>)
plt.plot(X1D[:, <span style="color: #B452CD">0</span>][y==<span style="color: #B452CD">0</span>], np.zeros(<span style="color: #B452CD">4</span>), <span style="color: #CD5555">&quot;bs&quot;</span>)
plt.plot(X1D[:, <span style="color: #B452CD">0</span>][y==<span style="color: #B452CD">1</span>], np.zeros(<span style="color: #B452CD">5</span>), <span style="color: #CD5555">&quot;g^&quot;</span>)
@@ -963,7 +979,7 @@ plt.xlabel(<span style="color: #CD5555">r&quot;$x_1$&quot;</span>, fontsize=<spa
plt.axis([-<span style="color: #B452CD">4.5</span>, <span style="color: #B452CD">4.5</span>, -<span style="color: #B452CD">0.2</span>, <span style="color: #B452CD">0.2</span>])
plt.subplot(<span style="color: #B452CD">122</span>)
plt.grid(<span style="color: #658b00">True</span>, which=<span style="color: #CD5555">&#39;both&#39;</span>)
plt.grid(<span style="color: #8B008B; font-weight: bold">True</span>, which=<span style="color: #CD5555">&#39;both&#39;</span>)
plt.axhline(y=<span style="color: #B452CD">0</span>, color=<span style="color: #CD5555">&#39;k&#39;</span>)
plt.axvline(x=<span style="color: #B452CD">0</span>, color=<span style="color: #CD5555">&#39;k&#39;</span>)
plt.plot(X2D[:, <span style="color: #B452CD">0</span>][y==<span style="color: #B452CD">0</span>], X2D[:, <span style="color: #B452CD">1</span>][y==<span style="color: #B452CD">0</span>], <span style="color: #CD5555">&quot;bs&quot;</span>)
@@ -980,7 +996,7 @@ plt.show()
<section>
<h2 id="___sec19">The equations </h2>
<h2 id="___sec20">The equations </h2>
<p>
Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
@@ -1035,7 +1051,7 @@ the trouble of performing the transformation
<section>
<h2 id="___sec20">The problem to solve </h2>
<h2 id="___sec21">The problem to solve </h2>
Using our definition of the kernel We can rewrite again the Lagrangian
<p>&nbsp;<br>
$$
@@ -1077,7 +1093,7 @@ Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up.
<section>
<h2 id="___sec21">Different kernels and Mercer's theorem </h2>
<h2 id="___sec22">Different kernels and Mercer's theorem </h2>
<p>
There are several popular kernels being used. These are
@@ -1118,7 +1134,7 @@ in practice.
<section>
<h2 id="___sec22">The moons example </h2>
<h2 id="___sec23">The moons example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -1151,7 +1167,7 @@ X, y = make_moons(n_samples=<span style="color: #B452CD">100</span>, noise=<span
plt.plot(X[:, <span style="color: #B452CD">0</span>][y==<span style="color: #B452CD">0</span>], X[:, <span style="color: #B452CD">1</span>][y==<span style="color: #B452CD">0</span>], <span style="color: #CD5555">&quot;bs&quot;</span>)
plt.plot(X[:, <span style="color: #B452CD">0</span>][y==<span style="color: #B452CD">1</span>], X[:, <span style="color: #B452CD">1</span>][y==<span style="color: #B452CD">1</span>], <span style="color: #CD5555">&quot;g^&quot;</span>)
plt.axis(axes)
plt.grid(<span style="color: #658b00">True</span>, which=<span style="color: #CD5555">&#39;both&#39;</span>)
plt.grid(<span style="color: #8B008B; font-weight: bold">True</span>, which=<span style="color: #CD5555">&#39;both&#39;</span>)
plt.xlabel(<span style="color: #CD5555">r&quot;$x_1$&quot;</span>, fontsize=<span style="color: #B452CD">20</span>)
plt.ylabel(<span style="color: #CD5555">r&quot;$x_2$&quot;</span>, fontsize=<span style="color: #B452CD">20</span>, rotation=<span style="color: #B452CD">0</span>)
@@ -1229,7 +1245,7 @@ yk = np.array([<span style="color: #B452CD">0</span>, <span style="color: #B452C
plt.figure(figsize=(<span style="color: #B452CD">11</span>, <span style="color: #B452CD">4</span>))
plt.subplot(<span style="color: #B452CD">121</span>)
plt.grid(<span style="color: #658b00">True</span>, which=<span style="color: #CD5555">&#39;both&#39;</span>)
plt.grid(<span style="color: #8B008B; font-weight: bold">True</span>, which=<span style="color: #CD5555">&#39;both&#39;</span>)
plt.axhline(y=<span style="color: #B452CD">0</span>, color=<span style="color: #CD5555">&#39;k&#39;</span>)
plt.scatter(x=[-<span style="color: #B452CD">2</span>, <span style="color: #B452CD">1</span>], y=[<span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span>], s=<span style="color: #B452CD">150</span>, alpha=<span style="color: #B452CD">0.5</span>, c=<span style="color: #CD5555">&quot;red&quot;</span>)
plt.plot(X1D[:, <span style="color: #B452CD">0</span>][yk==<span style="color: #B452CD">0</span>], np.zeros(<span style="color: #B452CD">4</span>), <span style="color: #CD5555">&quot;bs&quot;</span>)
@@ -1251,7 +1267,7 @@ plt.text(<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0.9
plt.axis([-<span style="color: #B452CD">4.5</span>, <span style="color: #B452CD">4.5</span>, -<span style="color: #B452CD">0.1</span>, <span style="color: #B452CD">1.1</span>])
plt.subplot(<span style="color: #B452CD">122</span>)
plt.grid(<span style="color: #658b00">True</span>, which=<span style="color: #CD5555">&#39;both&#39;</span>)
plt.grid(<span style="color: #8B008B; font-weight: bold">True</span>, which=<span style="color: #CD5555">&#39;both&#39;</span>)
plt.axhline(y=<span style="color: #B452CD">0</span>, color=<span style="color: #CD5555">&#39;k&#39;</span>)
plt.axvline(x=<span style="color: #B452CD">0</span>, color=<span style="color: #CD5555">&#39;k&#39;</span>)
plt.plot(XK[:, <span style="color: #B452CD">0</span>][yk==<span style="color: #B452CD">0</span>], XK[:, <span style="color: #B452CD">1</span>][yk==<span style="color: #B452CD">0</span>], <span style="color: #CD5555">&quot;bs&quot;</span>)
@@ -1276,7 +1292,7 @@ plt.show()
x1_example = X1D[<span style="color: #B452CD">3</span>, <span style="color: #B452CD">0</span>]
<span style="color: #8B008B; font-weight: bold">for</span> landmark <span style="color: #8B008B">in</span> (-<span style="color: #B452CD">2</span>, <span style="color: #B452CD">1</span>):
k = gaussian_rbf(np.array([[x1_example]]), np.array([[landmark]]), gamma)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Phi({}, {}) = {}&quot;</span>.format(x1_example, landmark, k))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Phi({}, {}) = {}&quot;</span>.format(x1_example, landmark, k))
rbf_kernel_svm_clf = Pipeline([
(<span style="color: #CD5555">&quot;scaler&quot;</span>, StandardScaler()),
@@ -1315,7 +1331,7 @@ plt.show()
<section>
<h2 id="___sec23">Mathematical optimization of convex functions </h2>
<h2 id="___sec24">Mathematical optimization of convex functions </h2>
<p>
A mathematical (quadratic) optimization problem, or just optimization problem, has the form
@@ -1342,7 +1358,7 @@ Convex optimization problems play a central role in applied mathematics and we r
<section>
<h2 id="___sec24">How do we solve these problems? </h2>
<h2 id="___sec25">How do we solve these problems? </h2>
<p>
If we use Python as programming language and wish to venture beyond
@@ -1368,7 +1384,7 @@ This will make our life much easier. You don't need t write your own optimizer.
<section>
<h2 id="___sec25">A simple example </h2>
<h2 id="___sec26">A simple example </h2>
<p>
We remind ourselves about the general problem we want to solve
@@ -1449,7 +1465,7 @@ sol[<span style="color: #a61717; background-color: #e3d2d2">’</span>primal obj
<section>
<h2 id="___sec26">Back to the more realistic cases </h2>
<h2 id="___sec27">Back to the more realistic cases </h2>
<p>
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the <b>slack</b> parameter \( C \) we have