lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating week 46

Browse Source
This commit is contained in:
mhjensen
2020-11-08 22:48:11 +01:00
parent 3e13bb0d64
commit 489c98a82f
34 changed files with 2834 additions and 2713 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week46/html/week46.html
+86 -70
View File
@@ -7,9 +7,9 @@ Automatically generated HTML file from DocOnce source
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 46: Support Vector Machines">
<meta name="description" content="Week 46: Gradient Boosting Summary and Support Vector Machines">
<title>Week 46: Support Vector Machines</title>
<title>Week 46: Gradient Boosting Summary and Support Vector Machines</title>
<style type="text/css">
@@ -40,39 +40,40 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
('Hyperplanes and all that', 2, None, '___sec1'),
('What is a hyperplane?', 2, None, '___sec2'),
('A $p$-dimensional space of features', 2, None, '___sec3'),
('The two-dimensional case', 2, None, '___sec4'),
('Getting into the details', 2, None, '___sec5'),
('First attempt at a minimization approach', 2, None, '___sec6'),
('Solving the equations', 2, None, '___sec7'),
('Code Example', 2, None, '___sec8'),
('Problems with the Simpler Approach', 2, None, '___sec9'),
('A better approach', 2, None, '___sec10'),
'sections': [('Overview of week 46', 2, None, '___sec0'),
('Support Vector Machines, overarching aims', 2, None, '___sec1'),
('Hyperplanes and all that', 2, None, '___sec2'),
('What is a hyperplane?', 2, None, '___sec3'),
('A $p$-dimensional space of features', 2, None, '___sec4'),
('The two-dimensional case', 2, None, '___sec5'),
('Getting into the details', 2, None, '___sec6'),
('First attempt at a minimization approach', 2, None, '___sec7'),
('Solving the equations', 2, None, '___sec8'),
('Code Example', 2, None, '___sec9'),
('Problems with the Simpler Approach', 2, None, '___sec10'),
('A better approach', 2, None, '___sec11'),
('A quick Reminder on Lagrangian Multipliers',
2,
None,
'___sec11'),
('Adding the Multiplier', 2, None, '___sec12'),
('Setting up the Problem', 2, None, '___sec13'),
('The problem to solve', 2, None, '___sec14'),
('The last steps', 2, None, '___sec15'),
('A soft classifier', 2, None, '___sec16'),
('Soft optmization problem', 2, None, '___sec17'),
('Kernels and non-linearity', 2, None, '___sec18'),
('The equations', 2, None, '___sec19'),
('The problem to solve', 2, None, '___sec20'),
("Different kernels and Mercer's theorem", 2, None, '___sec21'),
('The moons example', 2, None, '___sec22'),
'___sec12'),
('Adding the Multiplier', 2, None, '___sec13'),
('Setting up the Problem', 2, None, '___sec14'),
('The problem to solve', 2, None, '___sec15'),
('The last steps', 2, None, '___sec16'),
('A soft classifier', 2, None, '___sec17'),
('Soft optmization problem', 2, None, '___sec18'),
('Kernels and non-linearity', 2, None, '___sec19'),
('The equations', 2, None, '___sec20'),
('The problem to solve', 2, None, '___sec21'),
("Different kernels and Mercer's theorem", 2, None, '___sec22'),
('The moons example', 2, None, '___sec23'),
('Mathematical optimization of convex functions',
2,
None,
'___sec23'),
('How do we solve these problems?', 2, None, '___sec24'),
('A simple example', 2, None, '___sec25'),
('Back to the more realistic cases', 2, None, '___sec26')]}
'___sec24'),
('How do we solve these problems?', 2, None, '___sec25'),
('A simple example', 2, None, '___sec26'),
('Back to the more realistic cases', 2, None, '___sec27')]}
end of tocinfo -->
<body>
@@ -98,7 +99,7 @@ MathJax.Hub.Config({
<center><h1>Week 46: Support Vector Machines</h1></center> <!-- document title -->
<center><h1>Week 46: Gradient Boosting Summary and Support Vector Machines</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -114,12 +115,27 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<center><h4>Nov 8, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Support Vector Machines, overarching aims </h2>
<h2 id="___sec0">Overview of week 46 </h2>
<ul>
<li> <b>Thursday</b>: Summary of Gradient Boosting and further examples of applications.</li>
<li> <b>Friday</b>: Support Vector Machines, classification and regression</li>
</ul>
Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) of Hastie et al contains also a good discussion.
<p>
<a href="https://www.youtube.com/watch?v=efR1C6CvhmE&ab_channel=StatQuestwithJoshStarmer" target="_blank">Overview of Support Vector Machines</a>. see also <a href="https://www.youtube.com/watch?v=N1vOgolbjSc&ab_channel=AliceZhao" target="_blank">this video</a>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Support Vector Machines, overarching aims </h2>
<p>
A Support Vector Machine (SVM) is a very powerful and versatile
@@ -152,7 +168,7 @@ unlikely that we can separate classes easily by say straight lines.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Hyperplanes and all that </h2>
<h2 id="___sec2">Hyperplanes and all that </h2>
<p>
The theory behind support vector machines (SVM hereafter) is based on
@@ -205,9 +221,9 @@ lin_clf<span style="color: #666666">.</span>fit(X_scaled, y)
svm_clf<span style="color: #666666">.</span>fit(X_scaled, y)
sgd_clf<span style="color: #666666">.</span>fit(X_scaled, y)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;LinearSVC: &quot;</span>, lin_clf<span style="color: #666666">.</span>intercept_, lin_clf<span style="color: #666666">.</span>coef_)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;SVC: &quot;</span>, svm_clf<span style="color: #666666">.</span>intercept_, svm_clf<span style="color: #666666">.</span>coef_)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;SGDClassifier(alpha={:.5f}):&quot;</span><span style="color: #666666">.</span>format(sgd_clf<span style="color: #666666">.</span>alpha), sgd_clf<span style="color: #666666">.</span>intercept_, sgd_clf<span style="color: #666666">.</span>coef_)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;LinearSVC: &quot;</span>, lin_clf<span style="color: #666666">.</span>intercept_, lin_clf<span style="color: #666666">.</span>coef_)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;SVC: &quot;</span>, svm_clf<span style="color: #666666">.</span>intercept_, svm_clf<span style="color: #666666">.</span>coef_)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;SGDClassifier(alpha=</span><span style="color: #BB6688; font-weight: bold">{:.5f}</span><span style="color: #BA2121">):&quot;</span><span style="color: #666666">.</span>format(sgd_clf<span style="color: #666666">.</span>alpha), sgd_clf<span style="color: #666666">.</span>intercept_, sgd_clf<span style="color: #666666">.</span>coef_)
<span style="color: #408080; font-style: italic"># Compute the slope and bias of each decision boundary</span>
w1 <span style="color: #666666">=</span> <span style="color: #666666">-</span>lin_clf<span style="color: #666666">.</span>coef_[<span style="color: #666666">0</span>, <span style="color: #666666">0</span>]<span style="color: #666666">/</span>lin_clf<span style="color: #666666">.</span>coef_[<span style="color: #666666">0</span>, <span style="color: #666666">1</span>]
@@ -239,7 +255,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">What is a hyperplane? </h2>
<h2 id="___sec3">What is a hyperplane? </h2>
<p>
The aim of the SVM algorithm is to find a hyperplane in a
@@ -270,7 +286,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">A \( p \)-dimensional space of features </h2>
<h2 id="___sec4">A \( p \)-dimensional space of features </h2>
<p>
We limit ourselves to two classes of outputs \( y_i \) and assign these classes the values \( y_i = \pm 1 \).
@@ -312,7 +328,7 @@ When we try to separate hyperplanes, if it exists, we can use it to construct a
<p>
<!-- !split -->
<h2 id="___sec4">The two-dimensional case </h2>
<h2 id="___sec5">The two-dimensional case </h2>
<p>
Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional
@@ -339,7 +355,7 @@ for our data sample.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Getting into the details </h2>
<h2 id="___sec6">Getting into the details </h2>
<p>
Let us define the function
@@ -361,7 +377,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">First attempt at a minimization approach </h2>
<h2 id="___sec7">First attempt at a minimization approach </h2>
<p>
How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could
@@ -386,7 +402,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Solving the equations </h2>
<h2 id="___sec8">Solving the equations </h2>
<p>
We can now use the Newton-Raphson method or different variants of the gradient descent family (from plain gradient descent to various stochastic gradient descent approaches) to solve the equations
@@ -404,7 +420,7 @@ where \( \eta \) is our by now well-known learning rate.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Code Example </h2>
<h2 id="___sec9">Code Example </h2>
<p>
The equations we discussed above can be coded rather easily (the
@@ -418,7 +434,7 @@ regression). We are going to set up a simple case with two classes only and we w
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Problems with the Simpler Approach </h2>
<h2 id="___sec10">Problems with the Simpler Approach </h2>
<p>
There are however problems with this approach, although it looks
@@ -434,7 +450,7 @@ at all.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">A better approach </h2>
<h2 id="___sec11">A better approach </h2>
<p>
A better approach is rather to try to define a large margin between
@@ -476,7 +492,7 @@ about Lagrangian multipliers.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">A quick Reminder on Lagrangian Multipliers </h2>
<h2 id="___sec12">A quick Reminder on Lagrangian Multipliers </h2>
<p>
Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an
@@ -526,7 +542,7 @@ Then \( dz \) is no longer arbitrary.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Adding the Multiplier </h2>
<h2 id="___sec13">Adding the Multiplier </h2>
<p>
However, we can add to
@@ -569,7 +585,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Setting up the Problem </h2>
<h2 id="___sec14">Setting up the Problem </h2>
In order to solve the above problem, we define the following Lagrangian function to be minimized
$$
{\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right],
@@ -610,7 +626,7 @@ When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">The problem to solve </h2>
<h2 id="___sec15">The problem to solve </h2>
<p>
We can rewrite
@@ -634,7 +650,7 @@ subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vec
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">The last steps </h2>
<h2 id="___sec16">The last steps </h2>
<p>
Solving the above problem, yields the values of \( \lambda_i \).
@@ -668,7 +684,7 @@ Below we discuss how to find the optimal values of \( \lambda_i \). Before we pr
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">A soft classifier </h2>
<h2 id="___sec17">A soft classifier </h2>
<p>
Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
@@ -703,7 +719,7 @@ misclassifications.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">Soft optmization problem </h2>
<h2 id="___sec18">Soft optmization problem </h2>
<p>
This has in turn the consequences that we change our optmization problem to finding the minimum of
@@ -757,7 +773,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec18">Kernels and non-linearity </h2>
<h2 id="___sec19">Kernels and non-linearity </h2>
<p>
The cases we have studied till now, were all characterized by two classes
@@ -805,7 +821,7 @@ y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>a
plt<span style="color: #666666">.</span>figure(figsize<span style="color: #666666">=</span>(<span style="color: #666666">11</span>, <span style="color: #666666">4</span>))
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">121</span>)
plt<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">&#39;both&#39;</span>)
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">&#39;both&#39;</span>)
plt<span style="color: #666666">.</span>axhline(y<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">&#39;k&#39;</span>)
plt<span style="color: #666666">.</span>plot(X1D[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==0</span>], np<span style="color: #666666">.</span>zeros(<span style="color: #666666">4</span>), <span style="color: #BA2121">&quot;bs&quot;</span>)
plt<span style="color: #666666">.</span>plot(X1D[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==1</span>], np<span style="color: #666666">.</span>zeros(<span style="color: #666666">5</span>), <span style="color: #BA2121">&quot;g^&quot;</span>)
@@ -814,7 +830,7 @@ plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r&qu
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">-4.5</span>, <span style="color: #666666">4.5</span>, <span style="color: #666666">-0.2</span>, <span style="color: #666666">0.2</span>])
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">122</span>)
plt<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">&#39;both&#39;</span>)
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">&#39;both&#39;</span>)
plt<span style="color: #666666">.</span>axhline(y<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">&#39;k&#39;</span>)
plt<span style="color: #666666">.</span>axvline(x<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">&#39;k&#39;</span>)
plt<span style="color: #666666">.</span>plot(X2D[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==0</span>], X2D[:, <span style="color: #666666">1</span>][y<span style="color: #666666">==0</span>], <span style="color: #BA2121">&quot;bs&quot;</span>)
@@ -830,7 +846,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">The equations </h2>
<h2 id="___sec20">The equations </h2>
<p>
Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
@@ -875,7 +891,7 @@ the trouble of performing the transformation
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec20">The problem to solve </h2>
<h2 id="___sec21">The problem to solve </h2>
Using our definition of the kernel We can rewrite again the Lagrangian
$$
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
@@ -911,7 +927,7 @@ Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec21">Different kernels and Mercer's theorem </h2>
<h2 id="___sec22">Different kernels and Mercer's theorem </h2>
<p>
There are several popular kernels being used. These are
@@ -949,7 +965,7 @@ in practice.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec22">The moons example </h2>
<h2 id="___sec23">The moons example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -982,7 +998,7 @@ X, y <span style="color: #666666">=</span> make_moons(n_samples<span style="colo
plt<span style="color: #666666">.</span>plot(X[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==0</span>], X[:, <span style="color: #666666">1</span>][y<span style="color: #666666">==0</span>], <span style="color: #BA2121">&quot;bs&quot;</span>)
plt<span style="color: #666666">.</span>plot(X[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==1</span>], X[:, <span style="color: #666666">1</span>][y<span style="color: #666666">==1</span>], <span style="color: #BA2121">&quot;g^&quot;</span>)
plt<span style="color: #666666">.</span>axis(axes)
plt<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">&#39;both&#39;</span>)
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">&#39;both&#39;</span>)
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r&quot;$x_1$&quot;</span>, fontsize<span style="color: #666666">=20</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">r&quot;$x_2$&quot;</span>, fontsize<span style="color: #666666">=20</span>, rotation<span style="color: #666666">=0</span>)
@@ -1060,7 +1076,7 @@ yk <span style="color: #666666">=</span> np<span style="color: #666666">.</span>
plt<span style="color: #666666">.</span>figure(figsize<span style="color: #666666">=</span>(<span style="color: #666666">11</span>, <span style="color: #666666">4</span>))
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">121</span>)
plt<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">&#39;both&#39;</span>)
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">&#39;both&#39;</span>)
plt<span style="color: #666666">.</span>axhline(y<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">&#39;k&#39;</span>)
plt<span style="color: #666666">.</span>scatter(x<span style="color: #666666">=</span>[<span style="color: #666666">-2</span>, <span style="color: #666666">1</span>], y<span style="color: #666666">=</span>[<span style="color: #666666">0</span>, <span style="color: #666666">0</span>], s<span style="color: #666666">=150</span>, alpha<span style="color: #666666">=0.5</span>, c<span style="color: #666666">=</span><span style="color: #BA2121">&quot;red&quot;</span>)
plt<span style="color: #666666">.</span>plot(X1D[:, <span style="color: #666666">0</span>][yk<span style="color: #666666">==0</span>], np<span style="color: #666666">.</span>zeros(<span style="color: #666666">4</span>), <span style="color: #BA2121">&quot;bs&quot;</span>)
@@ -1070,7 +1086,7 @@ plt<span style="color: #666666">.</span>plot(x1s, x3s, <span style="color: #BA21
plt<span style="color: #666666">.</span>gca()<span style="color: #666666">.</span>get_yaxis()<span style="color: #666666">.</span>set_ticks([<span style="color: #666666">0</span>, <span style="color: #666666">0.25</span>, <span style="color: #666666">0.5</span>, <span style="color: #666666">0.75</span>, <span style="color: #666666">1</span>])
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r&quot;$x_1$&quot;</span>, fontsize<span style="color: #666666">=20</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">r&quot;Similarity&quot;</span>, fontsize<span style="color: #666666">=14</span>)
plt<span style="color: #666666">.</span>annotate(<span style="color: #BA2121">r&#39;$\mathbf{x}$&#39;</span>,
plt<span style="color: #666666">.</span>annotate(<span style="color: #BA2121">r&#39;$\mathbf</span><span style="color: #BB6688; font-weight: bold">{x}</span><span style="color: #BA2121">$&#39;</span>,
xy<span style="color: #666666">=</span>(X1D[<span style="color: #666666">3</span>, <span style="color: #666666">0</span>], <span style="color: #666666">0</span>),
xytext<span style="color: #666666">=</span>(<span style="color: #666666">-0.5</span>, <span style="color: #666666">0.20</span>),
ha<span style="color: #666666">=</span><span style="color: #BA2121">&quot;center&quot;</span>,
@@ -1082,14 +1098,14 @@ plt<span style="color: #666666">.</span>text(<span style="color: #666666">1</spa
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">-4.5</span>, <span style="color: #666666">4.5</span>, <span style="color: #666666">-0.1</span>, <span style="color: #666666">1.1</span>])
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">122</span>)
plt<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">&#39;both&#39;</span>)
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">&#39;both&#39;</span>)
plt<span style="color: #666666">.</span>axhline(y<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">&#39;k&#39;</span>)
plt<span style="color: #666666">.</span>axvline(x<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">&#39;k&#39;</span>)
plt<span style="color: #666666">.</span>plot(XK[:, <span style="color: #666666">0</span>][yk<span style="color: #666666">==0</span>], XK[:, <span style="color: #666666">1</span>][yk<span style="color: #666666">==0</span>], <span style="color: #BA2121">&quot;bs&quot;</span>)
plt<span style="color: #666666">.</span>plot(XK[:, <span style="color: #666666">0</span>][yk<span style="color: #666666">==1</span>], XK[:, <span style="color: #666666">1</span>][yk<span style="color: #666666">==1</span>], <span style="color: #BA2121">&quot;g^&quot;</span>)
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r&quot;$x_2$&quot;</span>, fontsize<span style="color: #666666">=20</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">r&quot;$x_3$ &quot;</span>, fontsize<span style="color: #666666">=20</span>, rotation<span style="color: #666666">=0</span>)
plt<span style="color: #666666">.</span>annotate(<span style="color: #BA2121">r&#39;$\phi\left(\mathbf{x}\right)$&#39;</span>,
plt<span style="color: #666666">.</span>annotate(<span style="color: #BA2121">r&#39;$\phi\left(\mathbf</span><span style="color: #BB6688; font-weight: bold">{x}</span><span style="color: #BA2121">\right)$&#39;</span>,
xy<span style="color: #666666">=</span>(XK[<span style="color: #666666">3</span>, <span style="color: #666666">0</span>], XK[<span style="color: #666666">3</span>, <span style="color: #666666">1</span>]),
xytext<span style="color: #666666">=</span>(<span style="color: #666666">0.65</span>, <span style="color: #666666">0.50</span>),
ha<span style="color: #666666">=</span><span style="color: #BA2121">&quot;center&quot;</span>,
@@ -1107,7 +1123,7 @@ plt<span style="color: #666666">.</span>show()
x1_example <span style="color: #666666">=</span> X1D[<span style="color: #666666">3</span>, <span style="color: #666666">0</span>]
<span style="color: #008000; font-weight: bold">for</span> landmark <span style="color: #AA22FF; font-weight: bold">in</span> (<span style="color: #666666">-2</span>, <span style="color: #666666">1</span>):
k <span style="color: #666666">=</span> gaussian_rbf(np<span style="color: #666666">.</span>array([[x1_example]]), np<span style="color: #666666">.</span>array([[landmark]]), gamma)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Phi({}, {}) = {}&quot;</span><span style="color: #666666">.</span>format(x1_example, landmark, k))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Phi(</span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">, </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">) = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">.</span>format(x1_example, landmark, k))
rbf_kernel_svm_clf <span style="color: #666666">=</span> Pipeline([
(<span style="color: #BA2121">&quot;scaler&quot;</span>, StandardScaler()),
@@ -1138,14 +1154,14 @@ plt<span style="color: #666666">.</span>figure(figsize<span style="color: #66666
plot_predictions(svm_clf, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
plot_dataset(X, y, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
gamma, C <span style="color: #666666">=</span> hyperparams[i]
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">r&quot;$\gamma = {}, C = {}$&quot;</span><span style="color: #666666">.</span>format(gamma, C), fontsize<span style="color: #666666">=16</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">r&quot;$\gamma = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">, C = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">$&quot;</span><span style="color: #666666">.</span>format(gamma, C), fontsize<span style="color: #666666">=16</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">Mathematical optimization of convex functions </h2>
<h2 id="___sec24">Mathematical optimization of convex functions </h2>
<p>
A mathematical (quadratic) optimization problem, or just optimization problem, has the form
@@ -1170,7 +1186,7 @@ Convex optimization problems play a central role in applied mathematics and we r
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">How do we solve these problems? </h2>
<h2 id="___sec25">How do we solve these problems? </h2>
<p>
If we use Python as programming language and wish to venture beyond
@@ -1196,7 +1212,7 @@ This will make our life much easier. You don't need t write your own optimizer.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">A simple example </h2>
<h2 id="___sec26">A simple example </h2>
<p>
We remind ourselves about the general problem we want to solve
@@ -1264,7 +1280,7 @@ sol[’primal objective’]
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">Back to the more realistic cases </h2>
<h2 id="___sec27">Back to the more realistic cases </h2>
<p>
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the <b>slack</b> parameter \( C \) we have