updating week 46

2020-11-09 23:21:39 +01:00
parent 30d8508b04
commit 40023e523f
36 changed files with 2809 additions and 2659 deletions
@@ -41,39 +41,41 @@ div { text-align: justify; text-justify: inter-word; }
 <!-- tocinfo
 {'highest level': 2,
 'sections': [('Overview of week 46', 2, None, '___sec0'),
-              ('Support Vector Machines, overarching aims', 2, None, '___sec1'),
-              ('Hyperplanes and all that', 2, None, '___sec2'),
-              ('What is a hyperplane?', 2, None, '___sec3'),
-              ('A $p$-dimensional space of features', 2, None, '___sec4'),
-              ('The two-dimensional case', 2, None, '___sec5'),
-              ('Getting into the details', 2, None, '___sec6'),
-              ('First attempt at a minimization approach', 2, None, '___sec7'),
-              ('Solving the equations', 2, None, '___sec8'),
-              ('Code Example', 2, None, '___sec9'),
-              ('Problems with the Simpler Approach', 2, None, '___sec10'),
-              ('A better approach', 2, None, '___sec11'),
+              ('Thursday', 2, None, '___sec1'),
+              ('Friday', 2, None, '___sec2'),
+              ('Support Vector Machines, overarching aims', 2, None, '___sec3'),
+              ('Hyperplanes and all that', 2, None, '___sec4'),
+              ('What is a hyperplane?', 2, None, '___sec5'),
+              ('A $p$-dimensional space of features', 2, None, '___sec6'),
+              ('The two-dimensional case', 2, None, '___sec7'),
+              ('Getting into the details', 2, None, '___sec8'),
+              ('First attempt at a minimization approach', 2, None, '___sec9'),
+              ('Solving the equations', 2, None, '___sec10'),
+              ('Code Example', 2, None, '___sec11'),
+              ('Problems with the Simpler Approach', 2, None, '___sec12'),
+              ('A better approach', 2, None, '___sec13'),
              ('A quick Reminder on Lagrangian Multipliers',
               2,
               None,
-               '___sec12'),
-              ('Adding the Multiplier', 2, None, '___sec13'),
-              ('Setting up the Problem', 2, None, '___sec14'),
-              ('The problem to solve', 2, None, '___sec15'),
-              ('The last steps', 2, None, '___sec16'),
-              ('A soft classifier', 2, None, '___sec17'),
-              ('Soft optmization problem', 2, None, '___sec18'),
-              ('Kernels and non-linearity', 2, None, '___sec19'),
-              ('The equations', 2, None, '___sec20'),
-              ('The problem to solve', 2, None, '___sec21'),
-              ("Different kernels and Mercer's theorem", 2, None, '___sec22'),
-              ('The moons example', 2, None, '___sec23'),
+               '___sec14'),
+              ('Adding the Multiplier', 2, None, '___sec15'),
+              ('Setting up the Problem', 2, None, '___sec16'),
+              ('The problem to solve', 2, None, '___sec17'),
+              ('The last steps', 2, None, '___sec18'),
+              ('A soft classifier', 2, None, '___sec19'),
+              ('Soft optmization problem', 2, None, '___sec20'),
+              ('Kernels and non-linearity', 2, None, '___sec21'),
+              ('The equations', 2, None, '___sec22'),
+              ('The problem to solve', 2, None, '___sec23'),
+              ("Different kernels and Mercer's theorem", 2, None, '___sec24'),
+              ('The moons example', 2, None, '___sec25'),
              ('Mathematical optimization of convex functions',
               2,
               None,
-               '___sec24'),
-              ('How do we solve these problems?', 2, None, '___sec25'),
-              ('A simple example', 2, None, '___sec26'),
-              ('Back to the more realistic cases', 2, None, '___sec27')]}
+               '___sec26'),
+              ('How do we solve these problems?', 2, None, '___sec27'),
+              ('A simple example', 2, None, '___sec28'),
+              ('Back to the more realistic cases', 2, None, '___sec29')]}
 end of tocinfo -->

 <body>
@@ -115,7 +117,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Nov 8, 2020</h4></center> <!-- date -->
+<center><h4>Nov 9, 2020</h4></center> <!-- date -->
 <br>
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -123,8 +125,8 @@ MathJax.Hub.Config({
 <h2 id="___sec0">Overview of week 46 </h2>

 <ul>
-<li> <b>Thursday</b>: Summary of Gradient Boosting and further examples of applications.</li>
-<li> <b>Friday</b>: Support Vector Machines, classification and regression</li>
+<li> <b>Thursday</b>: Summary of Gradient Boosting and further examples of applications, from the physical sciences to the social sciences.</li>
+<li> <b>Friday</b>: Support Vector Machines, classification and regression.</li>
 </ul>

 Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones)  of Hastie et al contains also a good discussion.
@@ -135,7 +137,26 @@ Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones)  o
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec1">Support Vector Machines, overarching aims  </h2>
+<h2 id="___sec1">Thursday </h2>
+
+<p>
+The first lecture on Thursday is devoted to a summary from last week, with additional examples. This material is included in the lectures from week 45, see also Hastie <em>et al.</em> chapter 10.1-10.10.
+
+<p>
+The second letter will be given by John M. Aiken, who recently defended his thesis on machine learning, and in particular using boosting methods, to data from the social sciences and science education.
+
+<p>
+<!-- !split --><br><br><br><br><br><br><br><br><br><br>
+
+<h2 id="___sec2">Friday </h2>
+
+<p>
+We start with a new topic and second-last topic covered in this course, support vector machines (SVM).
+
+<p>
+<!-- !split --><br><br><br><br><br><br><br><br><br><br>
+
+<h2 id="___sec3">Support Vector Machines, overarching aims  </h2>

 <p>
 A Support Vector Machine (SVM) is a very powerful and versatile
@@ -168,7 +189,7 @@ unlikely that we can separate classes easily by say straight lines.
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec2">Hyperplanes and all that </h2>
+<h2 id="___sec4">Hyperplanes and all that </h2>

 <p>
 The theory behind support vector machines (SVM hereafter) is based on
@@ -255,7 +276,7 @@ plt<span style="color: #666666">.</span>show()
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec3">What is a hyperplane? </h2>
+<h2 id="___sec5">What is a hyperplane? </h2>

 <p>
 The aim of the SVM algorithm is to find a hyperplane in a
@@ -286,7 +307,7 @@ $$
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec4">A \( p \)-dimensional space of features </h2>
+<h2 id="___sec6">A \( p \)-dimensional space of features </h2>

 <p>
 We limit ourselves to two classes of outputs \( y_i \) and assign these classes the values \( y_i = \pm 1 \). 
@@ -328,7 +349,7 @@ When we try to separate hyperplanes, if it exists, we can use it to construct a
 <p>
 <!-- !split  -->

-<h2 id="___sec5">The two-dimensional case </h2>
+<h2 id="___sec7">The two-dimensional case </h2>

 <p>
 Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional
@@ -355,7 +376,7 @@ for our data sample.
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec6">Getting into the details </h2>
+<h2 id="___sec8">Getting into the details </h2>

 <p>
 Let us define the function
@@ -377,7 +398,7 @@ $$
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec7">First attempt at a minimization approach </h2>
+<h2 id="___sec9">First attempt at a minimization approach </h2>

 <p>
 How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could
@@ -402,7 +423,7 @@ $$
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec8">Solving the equations </h2>
+<h2 id="___sec10">Solving the equations </h2>

 <p>
 We can now use the Newton-Raphson method or different variants of the gradient descent family (from plain gradient descent to various stochastic gradient descent approaches) to solve the equations
@@ -420,7 +441,7 @@ where \( \eta \) is our by now well-known learning rate.
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec9">Code Example </h2>
+<h2 id="___sec11">Code Example </h2>

 <p>
 The equations we discussed above can be coded rather easily (the
@@ -434,7 +455,7 @@ regression). We are going to set up a simple case with two classes only and we w
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec10">Problems with the Simpler Approach </h2>
+<h2 id="___sec12">Problems with the Simpler Approach </h2>

 <p>
 There are however problems with this approach, although it looks
@@ -450,7 +471,7 @@ at all.
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec11">A better approach </h2>
+<h2 id="___sec13">A better approach </h2>

 <p>
 A better approach is rather to try to define a large margin between
@@ -492,7 +513,7 @@ about Lagrangian multipliers.
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec12">A quick Reminder on Lagrangian Multipliers </h2>
+<h2 id="___sec14">A quick Reminder on Lagrangian Multipliers </h2>

 <p>
 Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an
@@ -542,7 +563,7 @@ Then \( dz \) is no longer arbitrary.
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec13">Adding the Multiplier </h2>
+<h2 id="___sec15">Adding the Multiplier </h2>

 <p>
 However, we can add to
@@ -585,7 +606,7 @@ $$
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec14">Setting up the Problem </h2>
+<h2 id="___sec16">Setting up the Problem </h2>
 In order to solve the above problem, we define the following Lagrangian function to be minimized 
 $$
 {\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right],
@@ -626,7 +647,7 @@ When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec15">The problem to solve </h2>
+<h2 id="___sec17">The problem to solve </h2>

 <p>
 We can rewrite 
@@ -650,7 +671,7 @@ subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vec
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec16">The last steps </h2>
+<h2 id="___sec18">The last steps </h2>

 <p>
 Solving the above problem, yields the values of \( \lambda_i \).
@@ -684,7 +705,7 @@ Below we discuss how to find the optimal values of \( \lambda_i \). Before we pr
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec17">A soft classifier </h2>
+<h2 id="___sec19">A soft classifier </h2>

 <p>
 Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
@@ -719,7 +740,7 @@ misclassifications.
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec18">Soft optmization problem </h2>
+<h2 id="___sec20">Soft optmization problem </h2>

 <p>
 This has in turn the consequences that we change our optmization problem to finding the minimum of 
@@ -773,7 +794,7 @@ $$
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec19">Kernels and non-linearity </h2>
+<h2 id="___sec21">Kernels and non-linearity </h2>

 <p>
 The cases we have studied till now, were all characterized by two classes
@@ -846,7 +867,7 @@ plt<span style="color: #666666">.</span>show()
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec20">The equations </h2>
+<h2 id="___sec22">The equations </h2>

 <p>
 Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
@@ -891,7 +912,7 @@ the trouble of performing the transformation
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec21">The problem to solve </h2>
+<h2 id="___sec23">The problem to solve </h2>
 Using our definition of the kernel We can rewrite again the Lagrangian
 $$
 {\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
@@ -927,7 +948,7 @@ Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up.
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec22">Different kernels and Mercer's theorem </h2>
+<h2 id="___sec24">Different kernels and Mercer's theorem </h2>

 <p>
 There are several popular kernels being used. These are
@@ -965,7 +986,7 @@ in practice.
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec23">The moons example </h2>
+<h2 id="___sec25">The moons example </h2>
 <p>

 <!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -1161,7 +1182,7 @@ plt<span style="color: #666666">.</span>show()
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec24">Mathematical optimization of convex functions </h2>
+<h2 id="___sec26">Mathematical optimization of convex functions </h2>

 <p>
 A mathematical (quadratic) optimization problem, or just optimization problem, has the form
@@ -1186,7 +1207,7 @@ Convex optimization problems play a central role in applied mathematics and we r
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec25">How do we solve these problems? </h2>
+<h2 id="___sec27">How do we solve these problems? </h2>

 <p>
 If we use Python as programming language and wish to venture beyond
@@ -1212,7 +1233,7 @@ This will make our life much easier. You don't need t write your own optimizer.
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec26">A simple example </h2>
+<h2 id="___sec28">A simple example </h2>

 <p>
 We remind ourselves about the general problem we want to solve
@@ -1280,7 +1301,7 @@ sol[’primal objective’]
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec27">Back to the more realistic cases </h2>
+<h2 id="___sec29">Back to the more realistic cases </h2>

 <p>
 We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the <b>slack</b> parameter \( C \) we have