diff --git a/doc/pub/week47/html/week47-bs.html b/doc/pub/week47/html/week47-bs.html index fd20117c5..79d233cbd 100644 --- a/doc/pub/week47/html/week47-bs.html +++ b/doc/pub/week47/html/week47-bs.html @@ -1,31 +1,28 @@ - + - - -Week 47: Support Vector Machines - + +Week 47: Support Vector Machines and Summary of Course + - - - @@ -84,8 +236,6 @@ MathJax.Hub.Config({ - -
@@ -94,35 +244,88 @@ MathJax.Hub.Config({ - Week 47: Support Vector Machines + Week 47: Support Vector Machines and Summary of Course
-
-
-

 

 

 

- - - -
-

Week 47: Support Vector Machines

+
+

Week 47: Support Vector Machines and Summary of Course

+
-

-

Morten Hjorth-Jensen [1, 2]
- -

+

+[1] Department of Physics, University of Oslo +
+
+[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
+
+
+

Nov 20, 2021

+
+
-
[1] Department of Physics, University of Oslo
-
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-
-

-

Nov 26, 2020

-
-

Read »

@@ -181,29 +380,22 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 22
    • +
  • 76
  • »
    • -
    - - -
    - © 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week47/html/week47-reveal.html b/doc/pub/week47/html/week47-reveal.html index 3268afb24..15194cc29 100644 --- a/doc/pub/week47/html/week47-reveal.html +++ b/doc/pub/week47/html/week47-reveal.html @@ -1,19 +1,18 @@ + - + + - + - + +Week 47: Support Vector Machines and Summary of Course -Week 47: Support Vector Machines - - - - - - - - + @@ -55,36 +54,81 @@ document.getElementsByTagName( 'head' )[0].appendChild( link ); - - - @@ -78,133 +263,93 @@ MathJax.Hub.Config({ - - +
    +

    Week 47: Support Vector Machines and Summary of Course

    +
    - - -

    Week 47: Support Vector Machines

    - -

    -

    Morten Hjorth-Jensen [1, 2]
    - -

    +

    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 20, 2021

    +
    +
    -
    [1] Department of Physics, University of Oslo
    -
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    -
    -

    -

    Nov 26, 2020

    -
    -











    - -

    Overview of week 47

    +

    Overview of week 47

    +

    Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) of Hastie et al contains also a good discussion.

    -Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) of Hastie et al contains also a good discussion. - -

    -See overview video on Support Vector Machines. See also this video. - -

    -









    - -

    Thursday

    - -

    -We start with our final topic this semester, Support Vector Machines - -

    -









    - -

    Friday

    - -

    -Friday's lecture is split in two parts. The first lecture is deveoted to a presentation of possible data sets and projects for project 3. The second lecture continues with support vector machines. It may happen that we will need parts of the second lecture as well for the project ideas. - -

    -









    - -

    Workshop plan Friday (first lecture)

    - -

    -Here are the various projects that will be presented during the first lecture (and possibly parts of the second lecture as well). Titles are tentative. Talks are approx 5-10 mins with roughly 5 minutes for discussions. - -

    +

    See overview video on Support Vector Machines. See also this video.











    +

    Support Vector Machines, overarching aims

    -

    Support Vector Machines, overarching aims

    - -

    -A Support Vector Machine (SVM) is a very powerful and versatile +

    A Support Vector Machine (SVM) is a very powerful and versatile Machine Learning method, capable of performing linear or nonlinear classification, regression, and even outlier detection. It is one of the most popular models in Machine Learning, and anyone interested in Machine Learning should have it in their toolbox. SVMs are particularly well suited for classification of complex but small-sized or -medium-sized datasets. +medium-sized datasets. +

    -

    -The case with two well-separated classes only can be understood in an +

    The case with two well-separated classes only can be understood in an intuitive way in terms of lines in a two-dimensional space separating -the two classes. +the two classes (see figure below). +

    -

    -The basic mathematics behind the SVM is however less familiar to most of us. +

    The basic mathematics behind the SVM is however less familiar to most of us. It relies on the definition of hyperplanes and the definition of a margin which separates classes (in case of classification problems) of variables. It is also used for regression problems. +

    -

    -With SVMs we distinguish between hard margin and soft margins. The +

    With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below. We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines. +

    -











    +

    Hyperplanes and all that

    -

    Hyperplanes and all that

    - -

    -The theory behind support vector machines (SVM hereafter) is based on +

    The theory behind support vector machines (SVM hereafter) is based on the mathematical description of so-called hyperplanes. Let us start with a two-dimensional case. This will also allow us to introduce our first SVM examples. These will be tailored to the case of two specific classes, as displayed in the figure here based on the usage of the petal data. +

    -

    -We assume here that our data set can be well separated into two +

    We assume here that our data set can be well separated into two domains, where a straight line does the job in the separating the two classes. Here the two classes are represented by either squares or circles. -

    +

    -
    from sklearn import datasets
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from sklearn import datasets
 from sklearn.svm import SVC, LinearSVC
 from sklearn.linear_model import SGDClassifier
 from sklearn.preprocessing import StandardScaler
@@ -270,387 +415,395 @@ plt.legend(loc="upper center", fon
 plt.axis([0, 5.5, 0, 2])
 
 plt.show()
-
    
-
    
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    What is a hyperplane?

    -

    What is a hyperplane?

    - -

    -The aim of the SVM algorithm is to find a hyperplane in a +

    The aim of the SVM algorithm is to find a hyperplane in a \( p \)-dimensional space, where \( p \) is the number of features that distinctly classifies the data points. +

    -

    -In a \( p \)-dimensional space, a hyperplane is what we call an affine subspace of dimension of \( p-1 \). +

    In a \( p \)-dimensional space, a hyperplane is what we call an affine subspace of dimension of \( p-1 \). As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is -a two-dimensional subspace, or stated simply, a plane. +a two-dimensional subspace, or stated simply, a plane. +

    -

    -In two dimensions, with the variables \( x_1 \) and \( x_2 \), the hyperplane is defined as +

    In two dimensions, with the variables \( x_1 \) and \( x_2 \), the hyperplane is defined as

    $$ b+w_1x_1+w_2x_2=0, $$ -

    -where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line +

    where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line \( b+w_1x_1+w_2x_2=0 \). In two dimensions we define the vectors \( \boldsymbol{x} =[x1,x2] \) and \( \boldsymbol{w}=[w1,w2] \). We can then rewrite the above equation as +

    $$ \boldsymbol{x}^T\boldsymbol{w}+b=0. $$ -

    +









    +

    A \( p \)-dimensional space of features

    -

    A \( p \)-dimensional space of features

    - -

    -We limit ourselves to two classes of outputs \( y_i \) and assign these classes the values \( y_i = \pm 1 \). -In a \( p \)-dimensional space of say \( p \) features we have a hyperplane defined as +

    We limit ourselves to two classes of outputs \( y_i \) and assign these classes the values \( y_i = \pm 1 \). +In a \( p \)-dimensional space of say \( p \) features we have a hyperplane defines as +

    $$ b+wx_1+w_2x_2+\dots +w_px_p=0. $$ -If we define a +

    If we define a matrix \( \boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right] \) of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldsymbol{X} \), +

    $$ \boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. $$ -If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have +

    If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have

    $$ b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0, $$ -if our output \( y_i=1 \). +

    if our output \( y_i=1 \). In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if +

    $$ b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0, $$ -for the class of observations \( y_i=-1 \), -then \( \boldsymbol{x}_i \) lies on the other side. +

    for the class of observations \( y_i=-1 \), +then \( \boldsymbol{x}_i \) lies on the other side. +

    -

    -Equivalently, for the two classes of observations we have +

    Equivalently, for the two classes of observations we have

    $$ y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0. $$ -

    -When we try to separate hyperplanes, if it exists, we can use it to construct a natural classifier: a test observation is assigned a given class depending on which side of the hyperplane it is located. +

    When we try to separate hyperplanes, if it exists, we can use it to construct a natural classifier: a test observation is assigned a given class depending on which side of the hyperplane it is located.

    -

    +

    The two-dimensional case

    -

    The two-dimensional case

    - -

    -Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional +

    Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional plane. To separate the two classes of data points, there are many possible lines (hyperplanes if you prefer a more strict naming) that could be chosen. Our objective is to find a plane that has the maximum margin, i.e the maximum distance between data points of both classes. Maximizing the margin distance provides some reinforcement so that future data points can be classified with -more confidence. Figure 12.1 of Hastie et al is a good illustration. +more confidence. +

    -

    -What a linear classifier attempts to accomplish is to split the +

    What a linear classifier attempts to accomplish is to split the feature space into two half spaces by placing a hyperplane between the data points. This hyperplane will be our decision boundary. All points on one side of the plane will belong to class one and all points on the other side of the plane will belong to the second class two. +

    -

    -Unfortunately there are many ways in which we can place a hyperplane +

    Unfortunately there are many ways in which we can place a hyperplane to divide the data. Below is an example of two candidate hyperplanes for our data sample. +

    -











    +

    Getting into the details

    -

    Getting into the details

    - -

    -Let us define the function +

    Let us define the function

    $$ -f(x) = \boldsymbol{x}^T\boldsymbol{w}+b = 0, +f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0, $$ -as the function that determines the line \( L \) that separates two classes (our two features), see Figure 12.1 of Hastie et al. +

    as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.

    -

    -Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{x}^T(\boldsymbol{w}_1-\boldsymbol{x}_2)=0 \). +

    Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).

    -

    -The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then +

    The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then

    $$ \delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b). $$ -

    +









    +

    First attempt at a minimization approach

    -

    First attempt at a minimization approach

    - -

    -How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could +

    How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could do is to define a cost function which now contains the set of all misclassified points \( M \) and attempt to minimize this function +

    $$ -C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b). +C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b). $$ -

    -We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us +

    We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us

    $$ \frac{\partial C}{\partial b} = -\sum_{i\in M} y_i, $$ -and +

    and

    $$ \frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i. $$ -

    +









    +

    Solving the equations

    -

    Solving the equations

    - -

    -We can now use the Newton-Raphson method or different variants of the gradient descent family (from plain gradient descent to various stochastic gradient descent approaches) to solve the equations +

    We can now use the Newton-Raphson method or different variants of the gradient descent family (from plain gradient descent to various stochastic gradient descent approaches) to solve the equations

    $$ b \leftarrow b +\eta \frac{\partial C}{\partial b}, $$ -and +

    and

    $$ \boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}}, $$ -where \( \eta \) is our by now well-known learning rate. +

    where \( \eta \) is our by now well-known learning rate.

    -











    +

    Code Example

    -

    Can we code this?

    +

    The equations we discussed above can be coded rather easily (the +framework is similar to what we developed for logistic +regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way. +

    -

    -The equations we discussed above can be coded rather easily (the -framework is similar to what we developed for logistic regression). We -can set up a simple case with two classes only and we want to find a -line which separates them the best possible way. + +

    +
    +
    +
    +
    + 
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    -

    -There are however problems with this approach, although it looks -pretty straightforward to implement. When running a code for such a -case we can easily end up with many diffeent lines which separate the -two classes. -

    -For small +









    +

    Problems with the Simpler Approach

    + +

    There are however problems with this approach, although it looks +pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes. +

    + +

    For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all. +

    -











    +

    A better approach

    -

    A better approach

    - -

    -A better approach is rather to try to define a large margin between +

    A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning). +

    -

    -Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to +

    Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to \( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition +

    $$ -y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. $$ -All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line. +

    All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.

    -

    -We seek thus the largest value \( M \) defined by +

    We seek thus the largest value \( M \) defined by

    $$ -\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, +\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, $$ -or just +

    or just

    $$ -y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i. $$ -If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of +

    If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of \( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition +

    $$ y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. $$ -

    -We have thus defined our margin as the inverse of the norm of +

    We have thus defined our margin as the invers of the norm of \( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves -about Lagrangian multipliers and optimzation problems. +about Lagrangian multipliers. +

    -











    +

    A quick Reminder on Lagrangian Multipliers

    -

    A quick Reminder on Lagrangian Multipliers

    - -

    -Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an +

    Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an extreme we have +

    $$ df=0. $$ -A necessary and sufficient condition is +

    A necessary and sufficient condition is

    $$ \frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0, $$ -due to +

    due to

    $$ df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz. $$ -In many problems the variables \( x,y,z \) are often subject to constraints (such as those above for the margin) +

    In many problems the variables \( x,y,z \) are often subject to constraints (such as those above for the margin) so that they are no longer all independent. It is possible at least in principle to use each constraint to eliminate one variable and to proceed with a new and smaller set of independent varables. +

    -

    -The use of so-called Lagrangian multipliers is an alternative technique when the elimination +

    The use of so-called Lagrangian multipliers is an alternative technique when the elimination of variables is incovenient or undesirable. Assume that we have an equation of constraint on the variables \( x,y,z \) +

    $$ \phi(x,y,z) = 0, $$ - resulting in +

    resulting in

    $$ d\phi = \frac{\partial \phi}{\partial x}dx+\frac{\partial \phi}{\partial y}dy+\frac{\partial \phi}{\partial z}dz =0. $$ -Now we cannot set anymore +

    Now we cannot set anymore

    $$ \frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0, $$ -if \( df=0 \) is wanted +

    if \( df=0 \) is wanted because there are now only two independent variables! Assume \( x \) and \( y \) are the independent variables. Then \( dz \) is no longer arbitrary. +

    -











    +

    Adding the Multiplier

    -

    Adding the Multiplier

    - -

    -However, we can add to +

    However, we can add to

    $$ df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz, $$ -a multiplum of \( d\phi \), viz. \( \lambda d\phi \), resulting in +

    a multiplum of \( d\phi \), viz. \( \lambda d\phi \), resulting in

    $$ df+\lambda d\phi = (\frac{\partial f}{\partial z}+\lambda \frac{\partial \phi}{\partial x})dx+(\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y})dy+ (\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z})dz =0. $$ -Our multiplier is chosen so that +

    Our multiplier is chosen so that

    $$ \frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z} =0. $$ -

    -We need to remember that we took \( dx \) and \( dy \) to be arbitrary and thus we must have +

    We need to remember that we took \( dx \) and \( dy \) to be arbitrary and thus we must have

    $$ \frac{\partial f}{\partial x}+\lambda\frac{\partial \phi}{\partial x} =0, $$ -and +

    and

    $$ \frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y} =0. $$ -When all these equations are satisfied, \( df=0 \). We have four unknowns, \( x,y,z \) and +

    When all these equations are satisfied, \( df=0 \). We have four unknowns, \( x,y,z \) and \( \lambda \). Actually we want only \( x,y,z \), \( \lambda \) needs not to be determined, it is therefore often called Lagrange's undetermined multiplier. If we have a set of constraints \( \phi_k \) we have the equations +

    $$ \frac{\partial f}{\partial x_i}+\sum_k\lambda_k\frac{\partial \phi_k}{\partial x_i} =0. $$ -

    -









    -

    Setting up the Problem

    -In order to solve the above problem, we define the following Lagrangian function to be minimized +









    +

    Setting up the Problem

    +

    In order to solve the above problem, we define the following Lagrangian function to be minimized

    $$ {\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right], $$ -where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \). +

    where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).

    -

    -Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain +

    Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain

    $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ -and +

    and

    $$ \frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i. $$ -Inserting these constraints into the equation for \( {\cal L} \) we obtain +

    Inserting these constraints into the equation for \( {\cal L} \) we obtain

    $$ {\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ -subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). +

    subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). We must in addition satisfy the Karush-Kuhn-Tucker (KKT) condition +

    $$ -\lambda_i\left[y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) -1\right] \hspace{0.1cm}\forall i. +\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i. $$ -
      -
    1. If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)=1 \) and we say that \( x_i \) is on the boundary.
      2. -
    2. If \( y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).
      3. +
    3. If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.
      4. +
    4. If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).
    +

    When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).

    -When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \). - -

    -The support vectors (the points that define the margin \( M \)) are the quantities we keep in order to make predictions. - -











    +

    The problem to solve

    -

    The problem to solve

    - -

    -We can rewrite +

    We can rewrite

    $$ {\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ -and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem +

    and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem

    $$ \frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\ y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\ @@ -660,145 +813,1503 @@ y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x \end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, $$ -subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +

    -











    +

    The last steps

    -

    The last steps

    - -

    -Solving the above problem, yields the values of \( \lambda_i \). +

    Solving the above problem, yields the values of \( \lambda_i \). To find the coefficients of your hyperplane we need simply to compute +

    $$ \boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i. $$ -With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via +

    With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via

    $$ -y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)=1, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1, $$ -resulting in +

    resulting in

    $$ -b = \frac{1}{y_i}-\boldsymbol{x}_1^T\boldsymbol{w}, +b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i, $$ -or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have +

    or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have

    $$ b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right). $$ -With our hyperplane coefficients we can use our classifier to assign any observation by simply using +

    With our hyperplane coefficients we can use our classifier to assign any observation by simply using

    $$ -y_i = \mathrm{sign}(\boldsymbol{x}_i^T\boldsymbol{w}+b). +y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b). $$ -Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier. +

    Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.

    -











    +

    A soft classifier

    -

    A soft classifier

    +

    Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.

    -

    -Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined. - -

    -Suppose now that the two classes overlap in feature space, as shown in Figure 12.1 of -Hastie et al. - -

    -One way to deal with this problem before we define the +

    Suppose now that classes overlap in feature space, as shown in the +figure here. One way to deal with this problem before we define the so-called kernel approach, is to allow a kind of slack in the sense that we allow some points to be on the wrong side of the margin. +

    -

    -We introduce thus the so-called slack variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and +

    We introduce thus the so-called slack variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and modify our previous equation +

    $$ -y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b)=1, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1, $$ -to +

    to

    $$ -y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b)=1-\xi_i, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i, $$ -with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \). +

    with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \). The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction -\( y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), +\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), we bound the total amount by which predictions fall on the wrong side of their margins. +

    -

    -Misclassifications occur when \( \xi_i > 1 \). Thus bounding the total sum by some value \( C \) bounds in turn the total number of +

    Misclassifications occur when \( \xi_i > 1 \). Thus bounding the total sum by some value \( C \) bounds in turn the total number of misclassifications. +

    -











    +

    Soft optmization problem

    -

    Soft optmization problem

    - -

    -This has in turn the consequences that we change our optmization problem to finding the minimum of +

    This has in turn the consequences that we change our optmization problem to finding the minimum of

    $$ {\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, $$ -subject to +

    subject to

    $$ -y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b)=1-\xi_i \hspace{0.1cm}\forall i, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, $$ -with the requirement \( \xi_i\geq 0 \). +

    with the requirement \( \xi_i\geq 0 \).

    -

    -Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain +

    Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain

    $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ -and +

    and

    $$ \frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i, $$ -and +

    and

    $$ \lambda_i = C-\gamma_i \hspace{0.1cm}\forall i. $$ -Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before +

    Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before

    $$ {\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ -but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \). +

    but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \). We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads +

    $$ -\lambda_i\left[y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, +\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, $$ $$ \gamma_i\xi_i = 0, $$ -and +

    and

    $$ -y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. $$ +









    +

    Kernels and non-linearity

    + +

    The cases we have studied till now, were all characterized by two classes +with a close to linear separability. The classifiers we have described +so far find linear boundaries in our input feature space. It is +possible to make our procedure more flexible by exploring the feature +space using other basis expansions such as higher-order polynomials, +wavelets, splines etc. +

    + +

    If our feature space is not easy to separate, as shown in the figure +here, we can achieve a better separation by introducing more complex +basis functions. The ideal would be, as shown in the next figure, to, via a specific transformation to +obtain a separation between the classes which is almost linear. +

    + +

    The change of basis, from \( x\rightarrow z=\phi(x) \) leads to the same type of equations to be solved, except that +we need to introduce for example a polynomial transformation to a two-dimensional training set. +

    + + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import os
+
+np.random.seed(42)
+
+# To plot pretty figures
+import matplotlib
+import matplotlib.pyplot as plt
+plt.rcParams['axes.labelsize'] = 14
+plt.rcParams['xtick.labelsize'] = 12
+plt.rcParams['ytick.labelsize'] = 12
+
+
+from sklearn.svm import SVC
+from sklearn import datasets
+
+
+
+X1D = np.linspace(-4, 4, 9).reshape(-1, 1)
+X2D = np.c_[X1D, X1D**2]
+y = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])
+
+plt.figure(figsize=(11, 4))
+
+plt.subplot(121)
+plt.grid(True, which='both')
+plt.axhline(y=0, color='k')
+plt.plot(X1D[:, 0][y==0], np.zeros(4), "bs")
+plt.plot(X1D[:, 0][y==1], np.zeros(5), "g^")
+plt.gca().get_yaxis().set_ticks([])
+plt.xlabel(r"$x_1$", fontsize=20)
+plt.axis([-4.5, 4.5, -0.2, 0.2])
+
+plt.subplot(122)
+plt.grid(True, which='both')
+plt.axhline(y=0, color='k')
+plt.axvline(x=0, color='k')
+plt.plot(X2D[:, 0][y==0], X2D[:, 1][y==0], "bs")
+plt.plot(X2D[:, 0][y==1], X2D[:, 1][y==1], "g^")
+plt.xlabel(r"$x_1$", fontsize=20)
+plt.ylabel(r"$x_2$", fontsize=20, rotation=0)
+plt.gca().get_yaxis().set_ticks([0, 4, 8, 12, 16])
+plt.plot([-4.5, 4.5], [6.5, 6.5], "r--", linewidth=3)
+plt.axis([-4.5, 4.5, -1, 17])
+plt.subplots_adjust(right=1)
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    The equations

    + +

    Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)

    +$$ +z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right). +$$ + +

    With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)

    +$$ +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{z}_j, +$$ + +

    subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors

    +$$ +y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i, +$$ + +

    from which we also find \( b \). +To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kernel \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as +

    +$$ +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). +$$ + +

    For the above example, the kernel reads

    +$$ +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2. +$$ + +

    We note that this is nothing but the dot product of the two original +vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the +product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute +the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). +

    + +

    This leads to the so-called +kernel trick and the result leads to the same as if we went through +the trouble of performing the transformation +\( \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations. +

    + +









    +

    The problem to solve

    +

    Using our definition of the kernel We can rewrite again the Lagrangian

    +$$ +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j, +$$ + +

    subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem

    +$$ +\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ +y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ +\dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots \\ +y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ +\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, +$$ + +

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +If we add the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \). +

    + +

    We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type

    +$$ +\begin{align*} + &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f. +\end{align*} +$$ + +

    Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \). +Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into +\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \). +

    + +









    +

    Different kernels and Mercer's theorem

    + +

    There are several popular kernels being used. These are

    +
      +
    1. Linear: \( K(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{x}^T\boldsymbol{y} \),
      2. +
    2. Polynomial: \( K(\boldsymbol{x},\boldsymbol{y})=(\boldsymbol{x}^T\boldsymbol{y}+\gamma)^d \),
      3. +
    3. Gaussian Radial Basis Function: \( K(\boldsymbol{x},\boldsymbol{y})=\exp{\left(-\gamma\vert\vert\boldsymbol{x}-\boldsymbol{y}\vert\vert^2\right)} \),
      4. +
    4. Tanh: \( K(\boldsymbol{x},\boldsymbol{y})=\tanh{(\boldsymbol{x}^T\boldsymbol{y}+\gamma)} \),
      5. +
    +

    and many other ones.

    + +

    An important theorem for us is Mercer's +theorem. The +theorem states that if a kernel function \( K \) is symmetric, continuous +and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there +exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into +another space (possibly with much higher dimensions) such that +

    + +$$ +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). +$$ + +

    So you can use \( K \) as a kernel since you know \( \phi \) exists, even if +you don’t know what \( \phi \) is. +

    + +

    Note that some frequently used kernels (such as the Sigmoid kernel) +don’t respect all of Mercer’s conditions, yet they generally work well +in practice. +

    + +









    +

    The moons example

    + + +
    +
    +
    +
    +
    + 
    from __future__ import division, print_function, unicode_literals
+
+import numpy as np
+np.random.seed(42)
+
+import matplotlib
+import matplotlib.pyplot as plt
+plt.rcParams['axes.labelsize'] = 14
+plt.rcParams['xtick.labelsize'] = 12
+plt.rcParams['ytick.labelsize'] = 12
+
+
+from sklearn.svm import SVC
+from sklearn import datasets
+
+
+
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler
+from sklearn.svm import LinearSVC
+
+
+from sklearn.datasets import make_moons
+X, y = make_moons(n_samples=100, noise=0.15, random_state=42)
+
+def plot_dataset(X, y, axes):
+    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "bs")
+    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "g^")
+    plt.axis(axes)
+    plt.grid(True, which='both')
+    plt.xlabel(r"$x_1$", fontsize=20)
+    plt.ylabel(r"$x_2$", fontsize=20, rotation=0)
+
+plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+plt.show()
+
+from sklearn.datasets import make_moons
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import PolynomialFeatures
+
+polynomial_svm_clf = Pipeline([
+        ("poly_features", PolynomialFeatures(degree=3)),
+        ("scaler", StandardScaler()),
+        ("svm_clf", LinearSVC(C=10, loss="hinge", random_state=42))
+    ])
+
+polynomial_svm_clf.fit(X, y)
+
+def plot_predictions(clf, axes):
+    x0s = np.linspace(axes[0], axes[1], 100)
+    x1s = np.linspace(axes[2], axes[3], 100)
+    x0, x1 = np.meshgrid(x0s, x1s)
+    X = np.c_[x0.ravel(), x1.ravel()]
+    y_pred = clf.predict(X).reshape(x0.shape)
+    y_decision = clf.decision_function(X).reshape(x0.shape)
+    plt.contourf(x0, x1, y_pred, cmap=plt.cm.brg, alpha=0.2)
+    plt.contourf(x0, x1, y_decision, cmap=plt.cm.brg, alpha=0.1)
+
+plot_predictions(polynomial_svm_clf, [-1.5, 2.5, -1, 1.5])
+plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+
+plt.show()
+
+
+from sklearn.svm import SVC
+
+poly_kernel_svm_clf = Pipeline([
+        ("scaler", StandardScaler()),
+        ("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5))
+    ])
+poly_kernel_svm_clf.fit(X, y)
+
+poly100_kernel_svm_clf = Pipeline([
+        ("scaler", StandardScaler()),
+        ("svm_clf", SVC(kernel="poly", degree=10, coef0=100, C=5))
+    ])
+poly100_kernel_svm_clf.fit(X, y)
+
+plt.figure(figsize=(11, 4))
+
+plt.subplot(121)
+plot_predictions(poly_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])
+plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+plt.title(r"$d=3, r=1, C=5$", fontsize=18)
+
+plt.subplot(122)
+plot_predictions(poly100_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])
+plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+plt.title(r"$d=10, r=100, C=5$", fontsize=18)
+
+plt.show()
+
+def gaussian_rbf(x, landmark, gamma):
+    return np.exp(-gamma * np.linalg.norm(x - landmark, axis=1)**2)
+
+gamma = 0.3
+
+x1s = np.linspace(-4.5, 4.5, 200).reshape(-1, 1)
+x2s = gaussian_rbf(x1s, -2, gamma)
+x3s = gaussian_rbf(x1s, 1, gamma)
+
+XK = np.c_[gaussian_rbf(X1D, -2, gamma), gaussian_rbf(X1D, 1, gamma)]
+yk = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])
+
+plt.figure(figsize=(11, 4))
+
+plt.subplot(121)
+plt.grid(True, which='both')
+plt.axhline(y=0, color='k')
+plt.scatter(x=[-2, 1], y=[0, 0], s=150, alpha=0.5, c="red")
+plt.plot(X1D[:, 0][yk==0], np.zeros(4), "bs")
+plt.plot(X1D[:, 0][yk==1], np.zeros(5), "g^")
+plt.plot(x1s, x2s, "g--")
+plt.plot(x1s, x3s, "b:")
+plt.gca().get_yaxis().set_ticks([0, 0.25, 0.5, 0.75, 1])
+plt.xlabel(r"$x_1$", fontsize=20)
+plt.ylabel(r"Similarity", fontsize=14)
+plt.annotate(r'$\mathbf{x}$',
+             xy=(X1D[3, 0], 0),
+             xytext=(-0.5, 0.20),
+             ha="center",
+             arrowprops=dict(facecolor='black', shrink=0.1),
+             fontsize=18,
+            )
+plt.text(-2, 0.9, "$x_2$", ha="center", fontsize=20)
+plt.text(1, 0.9, "$x_3$", ha="center", fontsize=20)
+plt.axis([-4.5, 4.5, -0.1, 1.1])
+
+plt.subplot(122)
+plt.grid(True, which='both')
+plt.axhline(y=0, color='k')
+plt.axvline(x=0, color='k')
+plt.plot(XK[:, 0][yk==0], XK[:, 1][yk==0], "bs")
+plt.plot(XK[:, 0][yk==1], XK[:, 1][yk==1], "g^")
+plt.xlabel(r"$x_2$", fontsize=20)
+plt.ylabel(r"$x_3$  ", fontsize=20, rotation=0)
+plt.annotate(r'$\phi\left(\mathbf{x}\right)$',
+             xy=(XK[3, 0], XK[3, 1]),
+             xytext=(0.65, 0.50),
+             ha="center",
+             arrowprops=dict(facecolor='black', shrink=0.1),
+             fontsize=18,
+            )
+plt.plot([-0.1, 1.1], [0.57, -0.1], "r--", linewidth=3)
+plt.axis([-0.1, 1.1, -0.1, 1.1])
+    
+plt.subplots_adjust(right=1)
+
+plt.show()
+
+
+x1_example = X1D[3, 0]
+for landmark in (-2, 1):
+    k = gaussian_rbf(np.array([[x1_example]]), np.array([[landmark]]), gamma)
+    print("Phi({}, {}) = {}".format(x1_example, landmark, k))
+
+rbf_kernel_svm_clf = Pipeline([
+        ("scaler", StandardScaler()),
+        ("svm_clf", SVC(kernel="rbf", gamma=5, C=0.001))
+    ])
+rbf_kernel_svm_clf.fit(X, y)
+
+
+from sklearn.svm import SVC
+
+gamma1, gamma2 = 0.1, 5
+C1, C2 = 0.001, 1000
+hyperparams = (gamma1, C1), (gamma1, C2), (gamma2, C1), (gamma2, C2)
+
+svm_clfs = []
+for gamma, C in hyperparams:
+    rbf_kernel_svm_clf = Pipeline([
+            ("scaler", StandardScaler()),
+            ("svm_clf", SVC(kernel="rbf", gamma=gamma, C=C))
+        ])
+    rbf_kernel_svm_clf.fit(X, y)
+    svm_clfs.append(rbf_kernel_svm_clf)
+
+plt.figure(figsize=(11, 7))
+
+for i, svm_clf in enumerate(svm_clfs):
+    plt.subplot(221 + i)
+    plot_predictions(svm_clf, [-1.5, 2.5, -1, 1.5])
+    plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+    gamma, C = hyperparams[i]
+    plt.title(r"$\gamma = {}, C = {}$".format(gamma, C), fontsize=16)
+
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Mathematical optimization of convex functions

    + +

    A mathematical (quadratic) optimization problem, or just optimization problem, has the form

    +$$ +\begin{align*} + &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f. +\end{align*} +$$ + +

    subject to some constraints for say a selected set \( i=1,2,\dots, n \). +In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the +vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with. +

    + +

    In our case we are particularly interested in a class of optimization problems called convex optmization problems. +In our discussion on gradient descent methods we discussed at length the definition of a convex function. +

    + +

    Convex optimization problems play a central role in applied mathematics and we recommend strongly Boyd and Vandenberghe's text on the topics.

    + +









    +

    How do we solve these problems?

    + +

    If we use Python as programming language and wish to venture beyond +scikit-learn, tensorflow and similar software which makes our +lives so much easier, we need to dive into the wonderful world of +quadratic programming. We can, if we wish, solve the minimization +problem using say standard gradient methods or conjugate gradient +methods. However, these methods tend to exhibit a rather slow +converge. So, welcome to the promised land of quadratic programming. +

    + +

    The functions we need are contained in the quadratic programming package CVXOPT and we need to import it together with numpy as

    + + + +
    +
    +
    +
    +
    + 
    import numpy
+import cvxopt
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    This will make our life much easier. You don't need t write your own optimizer.

    + +









    +

    A simple example

    + +

    We remind ourselves about the general problem we want to solve

    +$$ +\begin{align*} + &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber + &\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f. +\end{align*} +$$ + +

    Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem

    +$$ +\begin{align*} + &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber + &\mathrm{subject to} \\ \nonumber + &x, y \geq 0 \\ \nonumber + &x+3y \geq 15 \\ \nonumber + &2x+5y \leq 100 \\ \nonumber + &3x+4y \leq 80. \\ \nonumber +\end{align*} +$$ + +

    The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)

    +$$ +\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}. +$$ + +

    Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation

    +$$ +\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +$$ + +

    We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix

    +$$ +\boldsymbol{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} +$$ + +

    is clearly positive semi-definite (all eigenvalues larger or equal zero). +Finally, the vector \( \boldsymbol{h} \) is defined as +

    +$$ +\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +$$ + +

    Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero +The following code solves the equations for us +

    + + +
    +
    +
    +
    +
    + 
    # Import the necessary packages
+import numpy
+from cvxopt import matrix
+from cvxopt import solvers
+P = matrix(numpy.diag([1,0]), tc=’d’)
+q = matrix(numpy.array([3,4]), tc=’d’)
+G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’)
+h = matrix(numpy.array([0,0,-15,100,80]), tc=’d’)
+# Construct the QP, invoke solver
+sol = solvers.qp(P,q,G,h)
+# Extract optimal value and solution
+sol[’x’] 
+sol[’primal objective’]
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Back to the more realistic cases

    + +

    We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the slack parameter \( C \) we have

    +$$ +\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ +y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ +\dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots \\ +y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ +\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda}, +$$ + +

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \). +

    + +code will be added + +









    +

    Summary of course

    + +









    +

    What? Me worry? No final exam in this course!

    +

    +
    +

    +
    +

    + +









    +

    Topics we have covered this year

    + +

    The course has two central parts

    + +
      +
    1. Statistical analysis and optimization of data
      2. +
    2. Machine learning
      3. +
    +









    +

    Statistical analysis and optimization of data

    + +

    The following topics have been discussed:

    +
      +
    1. Basic concepts, expectation values, variance, covariance, correlation functions and errors;
      2. +
    2. Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions;
      3. +
    3. Central elements from linear algebra, matrix inversion and SVD
      4. +
    4. Gradient methods for data optimization
      5. +
    5. Estimation of errors using cross-validation, bootstrapping and jackknife methods;
      6. +
    6. Practical optimization using Singular-value decomposition and least squares for parameterizing data.
      7. +
    7. Principal Component Analysis to reduce the number of features.
      8. +
    +









    +

    Machine learning

    + +

    The following topics will be covered

    +
      +
    1. Linear methods for regression and classification: +
        +
      1. Ordinary Least Squares
        2. +
      2. Ridge regression
        3. +
      3. Lasso regression
        4. +
      4. Logistic regression
        5. +
      +
    2. Neural networks and deep learning: +
        +
      1. Feed Forward Neural Networks
        2. +
      2. Convolutional Neural Networks
        3. +
      3. Recurrent Neural Networks
        4. +
      +
    3. Decisions trees and ensemble methods: +
        +
      1. Decision trees
        2. +
      2. Bagging and voting
        3. +
      3. Random forests
        4. +
      4. Boosting and gradient boosting
        5. +
      +
    4. Support vector machines +
        +
      1. Binary classification and multiclass classification
        2. +
      2. Kernel methods
        3. +
      3. Regression
        4. +
      +
    +









    +

    Learning outcomes and overarching aims of this course

    + +

    The course introduces a variety of central algorithms and methods +essential for studies of data analysis and machine learning. The +course is project based and through the various projects, normally +three, you will be exposed to fundamental research problems +in these fields, with the aim to reproduce state of the art scientific +results. The students will learn to develop and structure large codes +for studying these systems, get acquainted with computing facilities +and learn to handle large scientific projects. A good scientific and +ethical conduct is emphasized throughout the course. +

    + + +









    +

    Perspective on Machine Learning

    + +
      +
    1. Rapidly emerging application area
      2. +
    2. Experiment AND theory are evolving in many many fields. Still many low-hanging fruits.
      3. +
    3. Requires education/retraining for more widespread adoption
      4. +
    4. A lot of “word-of-mouth” development methods
      5. +
    +

    Huge amounts of data sets require automation, classical analysis tools often inadequate. +High energy physics hit this wall in the 90’s. +In 2009 single top quark production was determined via Boosted decision trees, Bayesian +Neural Networks, etc. +

    + +









    +

    Machine Learning Research

    + +

    Where to find recent results:

    +
      +
    1. Conference proceedings, arXiv and blog posts!
      2. +
    2. NIPS: Neural Information Processing Systems
      3. +
    3. ICLR: International Conference on Learning Representations
      4. +
    4. ICML: International Conference on Machine Learning
      5. +
    5. Journal of Machine Learning Research
      6. +
    6. Follow ML on ArXiv
      7. +
    +









    +

    Starting your Machine Learning Project

    + +
      +
    1. Identify problem type: classification, regression
      2. +
    2. Consider your data carefully
      3. +
    3. Choose a simple model that fits 1. and 2.
      4. +
    4. Consider your data carefully again! Think of data representation more carefully.
      5. +
    5. Based on your results, feedback loop to earliest possible point
      6. +
    +









    +

    Choose a Model and Algorithm

    + +
      +
    1. Supervised?
      2. +
    2. Start with the simplest model that fits your problem
      3. +
    3. Start with minimal processing of data
      4. +
    +









    +

    Preparing Your Data

    + +
      +
    1. Shuffle your data
      2. +
    2. Mean center your data
      3. + +
    3. Normalize the variance
      4. + +
    4. Whitening
      5. + +
    5. When to do train/test split?
      6. +
    +









    +

    Which Activation and Weights to Choose in Neural Networks

    + +
      +
    1. RELU? ELU?
      2. +
    2. Sigmoid or Tanh?
      3. +
    3. Set all weights to 0?
      4. + +
    4. Set all weights to random values?
      5. + +
    +









    +

    Optimization Methods and Hyperparameters

    +
      +
    1. Stochastic gradient descent +
        +
      1. Stochastic gradient descent + momentum
        2. +
      +
    2. State-of-the-art approaches:
      3. + +
    +

    Which regularization and hyperparameters? \( L_1 \) or \( L_2 \), soft +classifiers, depths of trees and many other. Need to explore a large +set of hyperparameters and regularization methods. +

    + +









    +

    Resampling

    + +

    When do we resample?

    + +
      +
    1. Bootstrap
      2. +
    2. Cross-validation
      3. +
    3. Jackknife and many other
      4. +
    +









    +

    Other courses on Data science and Machine Learning at UiO

    + +

    The link here https://www.mn.uio.no/english/research/about/centre-focus/innovation/data-science/studies/ gives an excellent overview of courses on Machine learning at UiO.

    + +
      +
    1. STK2100 Machine learning and statistical methods for prediction and classification.
      2. +
    2. IN3050/IN4050 Introduction to Artificial Intelligence and Machine Learning. Introductory course in machine learning and AI with an algorithmic approach.
      3. +
    3. STK-INF3000/4000 Selected Topics in Data Science. The course provides insight into selected contemporary relevant topics within Data Science.
      4. +
    4. IN4080 Natural Language Processing. Probabilistic and machine learning techniques applied to natural language processing.
      5. +
    5. STK-IN4300 – Statistical learning methods in Data Science. An advanced introduction to statistical and machine learning. For students with a good mathematics and statistics background.
      6. +
    6. IN-STK5000 Adaptive Methods for Data-Based Decision Making. Methods for adaptive collection and processing of data based on machine learning techniques.
      7. +
    7. IN5400/INF5860 – Machine Learning for Image Analysis. An introduction to deep learning with particular emphasis on applications within Image analysis, but useful for other application areas too.
      8. +
    8. TEK5040 – Dyp læring for autonome systemer. The course addresses advanced algorithms and architectures for deep learning with neural networks. The course provides an introduction to how deep-learning techniques can be used in the construction of key parts of advanced autonomous systems that exist in physical environments and cyber environments.
      9. +
    +









    +

    Additional courses of interest

    + +
      +
    1. STK4051 Computational Statistics
      2. +
    2. STK4021 Applied Bayesian Analysis and Numerical Methods
      3. +
    +









    +

    What's the future like?

    + +

    Based on multi-layer nonlinear neural networks, deep learning can +learn directly from raw data, automatically extract and abstract +features from layer to layer, and then achieve the goal of regression, +classification, or ranking. Deep learning has made breakthroughs in +computer vision, speech processing and natural language, and reached +or even surpassed human level. The success of deep learning is mainly +due to the three factors: big data, big model, and big computing. +

    + +

    In the past few decades, many different architectures of deep neural +networks have been proposed, such as +

    +
      +
    1. Convolutional neural networks, which are mostly used in image and video data processing, and have also been applied to sequential data such as text processing;
      2. +
    2. Recurrent neural networks, which can process sequential data of variable length and have been widely used in natural language understanding and speech processing;
      3. +
    3. Encoder-decoder framework, which is mostly used for image or sequence generation, such as machine translation, text summarization, and image captioning.
      4. +
    +









    +

    Types of Machine Learning, a repetition

    + +
    + +

    +

    The approaches to machine learning are many, but are often split into two main categories. +In supervised learning we know the answer to a problem, +and let the computer deduce the logic behind it. On the other hand, unsupervised learning +is a method for finding patterns and relationship in data sets without any prior knowledge of the system. +Some authours also operate with a third category, namely reinforcement learning. This is a paradigm +of learning inspired by behavioural psychology, where learning is achieved by trial-and-error, +solely from rewards and punishment. +

    + +

    Another way to categorize machine learning tasks is to consider the desired output of a system. +Some of the most common tasks are: +

    + + +
    + + +









    +

    Why Boltzmann machines?

    + +

    What is known as restricted Boltzmann Machines (RMB) have received a lot of attention lately. +One of the major reasons is that they can be stacked layer-wise to build deep neural networks that capture complicated statistics. +

    + +

    The original RBMs had just one visible layer and a hidden layer, but recently so-called Gaussian-binary RBMs have gained quite some popularity in imaging since they are capable of modeling continuous data that are common to natural images.

    + +

    Furthermore, they have been used to solve complicated quantum mechanical many-particle problems or classical statistical physics problems like the Ising and Potts classes of models.

    + +









    +

    Boltzmann Machines

    + +

    Why use a generative model rather than the more well known discriminative deep neural networks (DNN)?

    + + +









    +

    Some similarities and differences from DNNs

    + +
      +
    1. Both use gradient-descent based learning procedures for minimizing cost functions
      2. +
    2. Energy based models don't use backpropagation and automatic differentiation for computing gradients, instead turning to Markov Chain Monte Carlo methods.
      3. +
    3. DNNs often have several hidden layers. A restricted Boltzmann machine has only one hidden layer, however several RBMs can be stacked to make up Deep Belief Networks, of which they constitute the building blocks.
      4. +
    +

    History: The RBM was developed by amongst others Geoffrey Hinton, called by some the "Godfather of Deep Learning", working with the University of Toronto and Google.

    + +









    +

    Boltzmann machines (BM)

    + +
    + +

    +

    A BM is what we would call an undirected probabilistic graphical model +with stochastic continuous or discrete units. +

    +
    + +
    + +

    +

    It is interpreted as a stochastic recurrent neural network where the +state of each unit(neurons/nodes) depends on the units it is connected +to. The weights in the network represent thus the strength of the +interaction between various units/nodes. +

    +
    + +
    + +

    +

    It turns into a Hopfield network if we choose deterministic rather +than stochastic units. In contrast to a Hopfield network, a BM is a +so-called generative model. It allows us to generate new samples from +the learned distribution. +

    +
    + + +









    +

    A standard BM setup

    + +
    + +

    +

    A standard BM network is divided into a set of observable and visible units \( \hat{x} \) and a set of unknown hidden units/nodes \( \hat{h} \).

    +
    + + +
    + +

    +

    Additionally there can be bias nodes for the hidden and visible layers. These biases are normally set to \( 1 \).

    +
    + + +
    + +

    +

    BMs are stackable, meaning they cwe can train a BM which serves as input to another BM. We can construct deep networks for learning complex PDFs. The layers can be trained one after another, a feature which makes them popular in deep learning

    +
    + + +

    However, they are often hard to train. This leads to the introduction of so-called restricted BMs, or RBMS. +Here we take away all lateral connections between nodes in the visible layer as well as connections between nodes in the hidden layer. The network is illustrated in the figure below. +

    + +









    +

    The structure of the RBM network

    + +

    +
    +

    +
    +

    + +









    +

    The network

    + +The network layers: +
      +
    1. A function \( \mathbf{x} \) that represents the visible layer, a vector of \( M \) elements (nodes). This layer represents both what the RBM might be given as training input, and what we want it to be able to reconstruct. This might for example be given by the pixels of an image or coefficients representing speech, or the coordinates of a quantum mechanical state function.
      2. +
    2. The function \( \mathbf{h} \) represents the hidden, or latent, layer. A vector of \( N \) elements (nodes). Also called "feature detectors".
      3. +
    +









    +

    Goals

    + +

    The goal of the hidden layer is to increase the model's expressive +power. We encode complex interactions between visible variables by +introducing additional, hidden variables that interact with visible +degrees of freedom in a simple manner, yet still reproduce the complex +correlations between visible degrees in the data once marginalized +over (integrated out). +

    + +The network parameters, to be optimized/learned: +
      +
    1. \( \mathbf{a} \) represents the visible bias, a vector of same length as \( \mathbf{x} \).
      2. +
    2. \( \mathbf{b} \) represents the hidden bias, a vector of same lenght as \( \mathbf{h} \).
      3. +
    3. \( W \) represents the interaction weights, a matrix of size \( M\times N \).
      4. +
    +









    +

    Joint distribution

    + +

    The restricted Boltzmann machine is described by a Boltzmann distribution

    +$$ +\begin{align} + P_{rbm}(\mathbf{x},\mathbf{h}) = \frac{1}{Z} e^{-\frac{1}{T_0}E(\mathbf{x},\mathbf{h})}, +\label{_auto1} +\end{align} +$$ + +

    where \( Z \) is the normalization constant or partition function, defined as

    +$$ +\begin{align} + Z = \int \int e^{-\frac{1}{T_0}E(\mathbf{x},\mathbf{h})} d\mathbf{x} d\mathbf{h}. +\label{_auto2} +\end{align} +$$ + +

    It is common to ignore \( T_0 \) by setting it to one.

    + +









    +

    Network Elements, the energy function

    + +

    The function \( E(\mathbf{x},\mathbf{h}) \) gives the energy of a +configuration (pair of vectors) \( (\mathbf{x}, \mathbf{h}) \). The lower +the energy of a configuration, the higher the probability of it. This +function also depends on the parameters \( \mathbf{a} \), \( \mathbf{b} \) and +\( W \). Thus, when we adjust them during the learning procedure, we are +adjusting the energy function to best fit our problem. +

    + +

    An expression for the energy function is

    +$$ +E(\hat{x},\hat{h}) = -\sum_{ia}^{NA}b_i^a \alpha_i^a(x_i)-\sum_{jd}^{MD}c_j^d \beta_j^d(h_j)-\sum_{ijad}^{NAMD}b_i^a \alpha_i^a(x_i)c_j^d \beta_j^d(h_j)w_{ij}^{ad}. +$$ + +

    Here \( \beta_j^d(h_j) \) and \( \alpha_i^a(x_j) \) are so-called transfer functions that map a given input value to a desired feature value. The labels \( a \) and \( d \) denote that there can be multiple transfer functions per variable. The first sum depends only on the visible units. The second on the hidden ones. Note that there is no connection between nodes in a layer.

    + +

    The quantities \( b \) and \( c \) can be interpreted as the visible and hidden biases, respectively.

    + +

    The connection between the nodes in the two layers is given by the weights \( w_{ij} \).

    + +









    +

    Defining different types of RBMs

    +

    There are different variants of RBMs, and the differences lie in the types of visible and hidden units we choose as well as in the implementation of the energy function \( E(\mathbf{x},\mathbf{h}) \).

    + +
    +Binary-Binary RBM: +

    + +

    RBMs were first developed using binary units in both the visible and hidden layer. The corresponding energy function is defined as follows:

    +$$ +\begin{align} + E(\mathbf{x}, \mathbf{h}) = - \sum_i^M x_i a_i- \sum_j^N b_j h_j - \sum_{i,j}^{M,N} x_i w_{ij} h_j, +\label{_auto3} +\end{align} +$$ + +

    where the binary values taken on by the nodes are most commonly 0 and 1.

    +
    + +
    +Gaussian-Binary RBM: +

    + +

    Another varient is the RBM where the visible units are Gaussian while the hidden units remain binary:

    +$$ +\begin{align} + E(\mathbf{x}, \mathbf{h}) = \sum_i^M \frac{(x_i - a_i)^2}{2\sigma_i^2} - \sum_j^N b_j h_j - \sum_{i,j}^{M,N} \frac{x_i w_{ij} h_j}{\sigma_i^2}. +\label{_auto4} +\end{align} +$$ +
    + + +









    +

    More about RBMs

    +
      +
    1. Useful when we model continuous data (i.e., we wish \( \mathbf{x} \) to be continuous)
      2. +
    2. Requires a smaller learning rate, since there's no upper bound to the value a component might take in the reconstruction
      3. +
    +

    Other types of units include:

    +
      +
    1. Softmax and multinomial units
      2. +
    2. Gaussian visible and hidden units
      3. +
    3. Binomial units
      4. +
    4. Rectified linear units
      5. +
    +

    To read more, see Lectures on Boltzmann machines in Physics.

    + +









    +

    Autoencoders: Overarching view

    + +

    Autoencoders are artificial neural networks capable of learning +efficient representations of the input data (these representations are called codings) without +any supervision (i.e., the training set is unlabeled). These codings +typically have a much lower dimensionality than the input data, making +autoencoders useful for dimensionality reduction. +

    + +

    More importantly, autoencoders act as powerful feature detectors, and +they can be used for unsupervised pretraining of deep neural networks. +

    + +

    Lastly, they are capable of randomly generating new data that looks +very similar to the training data; this is called a generative +model. For example, you could train an autoencoder on pictures of +faces, and it would then be able to generate new faces. Surprisingly, +autoencoders work by simply learning to copy their inputs to their +outputs. This may sound like a trivial task, but we will see that +constraining the network in various ways can make it rather +difficult. For example, you can limit the size of the internal +representation, or you can add noise to the inputs and train the +network to recover the original inputs. These constraints prevent the +autoencoder from trivially copying the inputs directly to the outputs, +which forces it to learn efficient ways of representing the data. In +short, the codings are byproducts of the autoencoder’s attempt to +learn the identity function under some constraints. +

    + +Video on autoencoders + +

    See also A. Geron's textbook, chapter 15.

    + +









    +

    Bayesian Machine Learning

    + +

    This is an important topic if we aim at extracting a probability +distribution. This gives us also a confidence interval and error +estimates. +

    + +

    Bayesian machine learning allows us to encode our prior beliefs about +what those models should look like, independent of what the data tells +us. This is especially useful when we don’t have a ton of data to +confidently learn our model. +

    + +Video on Bayesian deep learning + +

    See also the slides here.

    + +









    +

    Reinforcement Learning

    + +

    Reinforcement Learning (RL) is one of the most exciting fields of +Machine Learning today, and also one of the oldest. It has been around +since the 1950s, producing many interesting applications over the +years. +

    + +

    It studies +how agents take actions based on trial and error, so as to maximize +some notion of cumulative reward in a dynamic system or +environment. Due to its generality, the problem has also been studied +in many other disciplines, such as game theory, control theory, +operations research, information theory, multi-agent systems, swarm +intelligence, statistics, and genetic algorithms. +

    + +

    In March 2016, AlphaGo, a computer program that plays the board game +Go, beat Lee Sedol in a five-game match. This was the first time a +computer Go program had beaten a 9-dan (highest rank) professional +without handicaps. AlphaGo is based on deep convolutional neural +networks and reinforcement learning. AlphaGo’s victory was a major +milestone in artificial intelligence and it has also made +reinforcement learning a hot research area in the field of machine +learning. +

    + +

    Lecture on Reinforcement Learning.

    + +

    See also A. Geron's textbook, chapter 16.

    + +









    +

    Transfer learning

    + +

    The goal of transfer learning is to transfer the model or knowledge +obtained from a source task to the target task, in order to resolve +the issues of insufficient training data in the target task. The +rationality of doing so lies in that usually the source and target +tasks have inter-correlations, and therefore either the features, +samples, or models in the source task might provide useful information +for us to better solve the target task. Transfer learning is a hot +research topic in recent years, with many problems still waiting to be studied. +

    + +

    Lecture on transfer learning.

    + +









    +

    Adversarial learning

    + +

    The conventional deep generative model has a potential problem: the +model tends to generate extreme instances to maximize the +probabilistic likelihood, which will hurt its performance. Adversarial +learning utilizes the adversarial behaviors (e.g., generating +adversarial instances or training an adversarial model) to enhance the +robustness of the model and improve the quality of the generated +data. In recent years, one of the most promising unsupervised learning +technologies, generative adversarial networks (GAN), has already been +successfully applied to image, speech, and text. +

    + +

    Lecture on adversial learning.

    + +









    +

    Dual learning

    + +

    Dual learning is a new learning paradigm, the basic idea of which is +to use the primal-dual structure between machine learning tasks to +obtain effective feedback/regularization, and guide and strengthen the +learning process, thus reducing the requirement of large-scale labeled +data for deep learning. The idea of dual learning has been applied to +many problems in machine learning, including machine translation, +image style conversion, question answering and generation, image +classification and generation, text classification and generation, +image-to-text, and text-to-image. +

    + +









    +

    Distributed machine learning

    + +

    Distributed computation will speed up machine learning algorithms, +significantly improve their efficiency, and thus enlarge their +application. When distributed meets machine learning, more than just +implementing the machine learning algorithms in parallel is required. +

    + +









    +

    Meta learning

    + +

    Meta learning is an emerging research direction in machine +learning. Roughly speaking, meta learning concerns learning how to +learn, and focuses on the understanding and adaptation of the learning +itself, instead of just completing a specific learning task. That is, +a meta learner needs to be able to evaluate its own learning methods +and adjust its own learning methods according to specific learning +tasks. +

    + +









    +

    The Challenges Facing Machine Learning

    + +

    While there has been much progress in machine learning, there are also challenges.

    + +

    For example, the mainstream machine learning technologies are +black-box approaches, making us concerned about their potential +risks. To tackle this challenge, we may want to make machine learning +more explainable and controllable. As another example, the +computational complexity of machine learning algorithms is usually +very high and we may want to invent lightweight algorithms or +implementations. Furthermore, in many domains such as physics, +chemistry, biology, and social sciences, people usually seek elegantly +simple equations (e.g., the Schrödinger equation) to uncover the +underlying laws behind various phenomena. In the field of machine +learning, can we reveal simple laws instead of designing more complex +models for data fitting? Although there are many challenges, we are +still very optimistic about the future of machine learning. As we look +forward to the future, here are what we think the research hotspots in +the next ten years will be. +

    + +

    See the article on Discovery of Physics From Data: Universal Laws and Discrepancies

    + +









    +

    Explainable machine learning

    + +

    Machine learning, especially deep learning, evolves rapidly. The +ability gap between machine and human on many complex cognitive tasks +becomes narrower and narrower. However, we are still in the very early +stage in terms of explaining why those effective models work and how +they work. +

    + +

    What is missing: the gap between correlation and causation. Standard Machine Learning is based on what e have called a frequentist approach.

    + +

    Most +machine learning techniques, especially the statistical ones, depend +highly on correlations in data sets to make predictions and analyses. In +contrast, rational humans tend to reply on clear and trustworthy +causality relations obtained via logical reasoning on real and clear +facts. It is one of the core goals of explainable machine learning to +transition from solving problems by data correlation to solving +problems by logical reasoning. +

    + +Bayesian Machine Learning is one of the exciting research directions in this field. + +









    +

    Quantum machine learning

    + +

    Quantum machine learning is an emerging interdisciplinary research +area at the intersection of quantum computing and machine learning. +

    + +

    Quantum computers use effects such as quantum coherence and quantum +entanglement to process information, which is fundamentally different +from classical computers. Quantum algorithms have surpassed the best +classical algorithms in several problems (e.g., searching for an +unsorted database, inverting a sparse matrix), which we call quantum +acceleration. +

    + +

    When quantum computing meets machine learning, it can be a mutually +beneficial and reinforcing process, as it allows us to take advantage +of quantum computing to improve the performance of classical machine +learning algorithms. In addition, we can also use the machine learning +algorithms (on classic computers) to analyze and improve quantum +computing systems. +

    + +

    Lecture on Quantum ML.

    + +

    Read interview with Maria Schuld on her work on Quantum Machine Learning. See also her recent textbook.

    + +









    +

    Quantum machine learning algorithms based on linear algebra

    + +

    Many quantum machine learning algorithms are based on variants of +quantum algorithms for solving linear equations, which can efficiently +solve N-variable linear equations with complexity of O(log2 N) under +certain conditions. The quantum matrix inversion algorithm can +accelerate many machine learning methods, such as least square linear +regression, least square version of support vector machine, Gaussian +process, and more. The training of these algorithms can be simplified +to solve linear equations. The key bottleneck of this type of quantum +machine learning algorithms is data input—that is, how to initialize +the quantum system with the entire data set. Although efficient +data-input algorithms exist for certain situations, how to efficiently +input data into a quantum system is as yet unknown for most cases. +

    + +









    +

    Quantum reinforcement learning

    + +

    In quantum reinforcement learning, a quantum agent interacts with the +classical environment to obtain rewards from the environment, so as to +adjust and improve its behavioral strategies. In some cases, it +achieves quantum acceleration by the quantum processing capabilities +of the agent or the possibility of exploring the environment through +quantum superposition. Such algorithms have been proposed in +superconducting circuits and systems of trapped ions. +

    + +









    +

    Quantum deep learning

    + +

    Dedicated quantum information processors, such as quantum annealers +and programmable photonic circuits, are well suited for building deep +quantum networks. The simplest deep quantum network is the Boltzmann +machine. The classical Boltzmann machine consists of bits with tunable +interactions and is trained by adjusting the interaction of these bits +so that the distribution of its expression conforms to the statistics +of the data. To quantize the Boltzmann machine, the neural network can +simply be represented as a set of interacting quantum spins that +correspond to an adjustable Ising model. Then, by initializing the +input neurons in the Boltzmann machine to a fixed state and allowing +the system to heat up, we can read out the output qubits to get the +result. +

    + +









    +

    Social machine learning

    + +

    Machine learning aims to imitate how humans +learn. While we have developed successful machine learning algorithms, +until now we have ignored one important fact: humans are social. Each +of us is one part of the total society and it is difficult for us to +live, learn, and improve ourselves, alone and isolated. Therefore, we +should design machines with social properties. Can we let machines +evolve by imitating human society so as to achieve more effective, +intelligent, interpretable “social machine learning”? +

    + +

    And much more.

    + +









    +

    The last words?

    + +

    Early computer scientist Alan Kay said, The best way to predict the +future is to create it. Therefore, all machine learning +practitioners, whether scholars or engineers, professors or students, +need to work together to advance these important research +topics. Together, we will not just predict the future, but create it. +

    + +









    +

    Best wishes to you all and thanks so much for your heroic efforts this semester

    + +

    +
    +

    +
    +

    + - -
    - © 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week47/html/week47.html b/doc/pub/week47/html/week47.html index 660134bba..4990fa4c8 100644 --- a/doc/pub/week47/html/week47.html +++ b/doc/pub/week47/html/week47.html @@ -1,69 +1,326 @@ - + - - -Week 47: Support Vector Machines - - + +Week 47: Support Vector Machines and Summary of Course - - @@ -83,133 +340,93 @@ MathJax.Hub.Config({ - - +
    +

    Week 47: Support Vector Machines and Summary of Course

    +
    - - -

    Week 47: Support Vector Machines

    - -

    -

    Morten Hjorth-Jensen [1, 2]
    - -

    +

    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 20, 2021

    +
    +
    -
    [1] Department of Physics, University of Oslo
    -
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    -
    -

    -

    Nov 26, 2020

    -
    -











    - -

    Overview of week 47

    +

    Overview of week 47

    +

    Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) of Hastie et al contains also a good discussion.

    -Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) of Hastie et al contains also a good discussion. - -

    -See overview video on Support Vector Machines. See also this video. - -

    -









    - -

    Thursday

    - -

    -We start with our final topic this semester, Support Vector Machines - -

    -









    - -

    Friday

    - -

    -Friday's lecture is split in two parts. The first lecture is deveoted to a presentation of possible data sets and projects for project 3. The second lecture continues with support vector machines. It may happen that we will need parts of the second lecture as well for the project ideas. - -

    -









    - -

    Workshop plan Friday (first lecture)

    - -

    -Here are the various projects that will be presented during the first lecture (and possibly parts of the second lecture as well). Titles are tentative. Talks are approx 5-10 mins with roughly 5 minutes for discussions. - -

    +

    See overview video on Support Vector Machines. See also this video.











    +

    Support Vector Machines, overarching aims

    -

    Support Vector Machines, overarching aims

    - -

    -A Support Vector Machine (SVM) is a very powerful and versatile +

    A Support Vector Machine (SVM) is a very powerful and versatile Machine Learning method, capable of performing linear or nonlinear classification, regression, and even outlier detection. It is one of the most popular models in Machine Learning, and anyone interested in Machine Learning should have it in their toolbox. SVMs are particularly well suited for classification of complex but small-sized or -medium-sized datasets. +medium-sized datasets. +

    -

    -The case with two well-separated classes only can be understood in an +

    The case with two well-separated classes only can be understood in an intuitive way in terms of lines in a two-dimensional space separating -the two classes. +the two classes (see figure below). +

    -

    -The basic mathematics behind the SVM is however less familiar to most of us. +

    The basic mathematics behind the SVM is however less familiar to most of us. It relies on the definition of hyperplanes and the definition of a margin which separates classes (in case of classification problems) of variables. It is also used for regression problems. +

    -

    -With SVMs we distinguish between hard margin and soft margins. The +

    With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below. We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines. +

    -











    +

    Hyperplanes and all that

    -

    Hyperplanes and all that

    - -

    -The theory behind support vector machines (SVM hereafter) is based on +

    The theory behind support vector machines (SVM hereafter) is based on the mathematical description of so-called hyperplanes. Let us start with a two-dimensional case. This will also allow us to introduce our first SVM examples. These will be tailored to the case of two specific classes, as displayed in the figure here based on the usage of the petal data. +

    -

    -We assume here that our data set can be well separated into two +

    We assume here that our data set can be well separated into two domains, where a straight line does the job in the separating the two classes. Here the two classes are represented by either squares or circles. -

    +

    -
    from sklearn import datasets
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from sklearn import datasets
 from sklearn.svm import SVC, LinearSVC
 from sklearn.linear_model import SGDClassifier
 from sklearn.preprocessing import StandardScaler
@@ -275,387 +492,395 @@ plt.legend(loc=
 plt.axis([0, 5.5, 0, 2])
 
 plt.show()
-
    
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    What is a hyperplane?

    -

    What is a hyperplane?

    - -

    -The aim of the SVM algorithm is to find a hyperplane in a +

    The aim of the SVM algorithm is to find a hyperplane in a \( p \)-dimensional space, where \( p \) is the number of features that distinctly classifies the data points. +

    -

    -In a \( p \)-dimensional space, a hyperplane is what we call an affine subspace of dimension of \( p-1 \). +

    In a \( p \)-dimensional space, a hyperplane is what we call an affine subspace of dimension of \( p-1 \). As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is -a two-dimensional subspace, or stated simply, a plane. +a two-dimensional subspace, or stated simply, a plane. +

    -

    -In two dimensions, with the variables \( x_1 \) and \( x_2 \), the hyperplane is defined as +

    In two dimensions, with the variables \( x_1 \) and \( x_2 \), the hyperplane is defined as

    $$ b+w_1x_1+w_2x_2=0, $$ -

    -where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line +

    where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line \( b+w_1x_1+w_2x_2=0 \). In two dimensions we define the vectors \( \boldsymbol{x} =[x1,x2] \) and \( \boldsymbol{w}=[w1,w2] \). We can then rewrite the above equation as +

    $$ \boldsymbol{x}^T\boldsymbol{w}+b=0. $$ -

    +









    +

    A \( p \)-dimensional space of features

    -

    A \( p \)-dimensional space of features

    - -

    -We limit ourselves to two classes of outputs \( y_i \) and assign these classes the values \( y_i = \pm 1 \). -In a \( p \)-dimensional space of say \( p \) features we have a hyperplane defined as +

    We limit ourselves to two classes of outputs \( y_i \) and assign these classes the values \( y_i = \pm 1 \). +In a \( p \)-dimensional space of say \( p \) features we have a hyperplane defines as +

    $$ b+wx_1+w_2x_2+\dots +w_px_p=0. $$ -If we define a +

    If we define a matrix \( \boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right] \) of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldsymbol{X} \), +

    $$ \boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. $$ -If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have +

    If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have

    $$ b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0, $$ -if our output \( y_i=1 \). +

    if our output \( y_i=1 \). In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if +

    $$ b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0, $$ -for the class of observations \( y_i=-1 \), -then \( \boldsymbol{x}_i \) lies on the other side. +

    for the class of observations \( y_i=-1 \), +then \( \boldsymbol{x}_i \) lies on the other side. +

    -

    -Equivalently, for the two classes of observations we have +

    Equivalently, for the two classes of observations we have

    $$ y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0. $$ -

    -When we try to separate hyperplanes, if it exists, we can use it to construct a natural classifier: a test observation is assigned a given class depending on which side of the hyperplane it is located. +

    When we try to separate hyperplanes, if it exists, we can use it to construct a natural classifier: a test observation is assigned a given class depending on which side of the hyperplane it is located.

    -

    +

    The two-dimensional case

    -

    The two-dimensional case

    - -

    -Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional +

    Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional plane. To separate the two classes of data points, there are many possible lines (hyperplanes if you prefer a more strict naming) that could be chosen. Our objective is to find a plane that has the maximum margin, i.e the maximum distance between data points of both classes. Maximizing the margin distance provides some reinforcement so that future data points can be classified with -more confidence. Figure 12.1 of Hastie et al is a good illustration. +more confidence. +

    -

    -What a linear classifier attempts to accomplish is to split the +

    What a linear classifier attempts to accomplish is to split the feature space into two half spaces by placing a hyperplane between the data points. This hyperplane will be our decision boundary. All points on one side of the plane will belong to class one and all points on the other side of the plane will belong to the second class two. +

    -

    -Unfortunately there are many ways in which we can place a hyperplane +

    Unfortunately there are many ways in which we can place a hyperplane to divide the data. Below is an example of two candidate hyperplanes for our data sample. +

    -











    +

    Getting into the details

    -

    Getting into the details

    - -

    -Let us define the function +

    Let us define the function

    $$ -f(x) = \boldsymbol{x}^T\boldsymbol{w}+b = 0, +f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0, $$ -as the function that determines the line \( L \) that separates two classes (our two features), see Figure 12.1 of Hastie et al. +

    as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.

    -

    -Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{x}^T(\boldsymbol{w}_1-\boldsymbol{x}_2)=0 \). +

    Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).

    -

    -The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then +

    The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then

    $$ \delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b). $$ -

    +









    +

    First attempt at a minimization approach

    -

    First attempt at a minimization approach

    - -

    -How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could +

    How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could do is to define a cost function which now contains the set of all misclassified points \( M \) and attempt to minimize this function +

    $$ -C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b). +C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b). $$ -

    -We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us +

    We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us

    $$ \frac{\partial C}{\partial b} = -\sum_{i\in M} y_i, $$ -and +

    and

    $$ \frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i. $$ -

    +









    +

    Solving the equations

    -

    Solving the equations

    - -

    -We can now use the Newton-Raphson method or different variants of the gradient descent family (from plain gradient descent to various stochastic gradient descent approaches) to solve the equations +

    We can now use the Newton-Raphson method or different variants of the gradient descent family (from plain gradient descent to various stochastic gradient descent approaches) to solve the equations

    $$ b \leftarrow b +\eta \frac{\partial C}{\partial b}, $$ -and +

    and

    $$ \boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}}, $$ -where \( \eta \) is our by now well-known learning rate. +

    where \( \eta \) is our by now well-known learning rate.

    -











    +

    Code Example

    -

    Can we code this?

    +

    The equations we discussed above can be coded rather easily (the +framework is similar to what we developed for logistic +regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way. +

    -

    -The equations we discussed above can be coded rather easily (the -framework is similar to what we developed for logistic regression). We -can set up a simple case with two classes only and we want to find a -line which separates them the best possible way. + +

    +
    +
    +
    +
    + 
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    -

    -There are however problems with this approach, although it looks -pretty straightforward to implement. When running a code for such a -case we can easily end up with many diffeent lines which separate the -two classes. -

    -For small +









    +

    Problems with the Simpler Approach

    + +

    There are however problems with this approach, although it looks +pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes. +

    + +

    For small gaps between the entries, we may also end up needing many iterations before the solutions converge and if the data cannot be separated properly into two distinct classes, we may not experience a converge at all. +

    -











    +

    A better approach

    -

    A better approach

    - -

    -A better approach is rather to try to define a large margin between +

    A better approach is rather to try to define a large margin between the two classes (if they are well separated from the beginning). +

    -

    -Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to +

    Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to \( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition +

    $$ -y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. $$ -All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line. +

    All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.

    -

    -We seek thus the largest value \( M \) defined by +

    We seek thus the largest value \( M \) defined by

    $$ -\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, +\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, $$ -or just +

    or just

    $$ -y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i. $$ -If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of +

    If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of \( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition +

    $$ y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. $$ -

    -We have thus defined our margin as the inverse of the norm of +

    We have thus defined our margin as the invers of the norm of \( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as possible margin \( M \). Before we proceed, we need to remind ourselves -about Lagrangian multipliers and optimzation problems. +about Lagrangian multipliers. +

    -











    +

    A quick Reminder on Lagrangian Multipliers

    -

    A quick Reminder on Lagrangian Multipliers

    - -

    -Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an +

    Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an extreme we have +

    $$ df=0. $$ -A necessary and sufficient condition is +

    A necessary and sufficient condition is

    $$ \frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0, $$ -due to +

    due to

    $$ df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz. $$ -In many problems the variables \( x,y,z \) are often subject to constraints (such as those above for the margin) +

    In many problems the variables \( x,y,z \) are often subject to constraints (such as those above for the margin) so that they are no longer all independent. It is possible at least in principle to use each constraint to eliminate one variable and to proceed with a new and smaller set of independent varables. +

    -

    -The use of so-called Lagrangian multipliers is an alternative technique when the elimination +

    The use of so-called Lagrangian multipliers is an alternative technique when the elimination of variables is incovenient or undesirable. Assume that we have an equation of constraint on the variables \( x,y,z \) +

    $$ \phi(x,y,z) = 0, $$ - resulting in +

    resulting in

    $$ d\phi = \frac{\partial \phi}{\partial x}dx+\frac{\partial \phi}{\partial y}dy+\frac{\partial \phi}{\partial z}dz =0. $$ -Now we cannot set anymore +

    Now we cannot set anymore

    $$ \frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0, $$ -if \( df=0 \) is wanted +

    if \( df=0 \) is wanted because there are now only two independent variables! Assume \( x \) and \( y \) are the independent variables. Then \( dz \) is no longer arbitrary. +

    -











    +

    Adding the Multiplier

    -

    Adding the Multiplier

    - -

    -However, we can add to +

    However, we can add to

    $$ df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz, $$ -a multiplum of \( d\phi \), viz. \( \lambda d\phi \), resulting in +

    a multiplum of \( d\phi \), viz. \( \lambda d\phi \), resulting in

    $$ df+\lambda d\phi = (\frac{\partial f}{\partial z}+\lambda \frac{\partial \phi}{\partial x})dx+(\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y})dy+ (\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z})dz =0. $$ -Our multiplier is chosen so that +

    Our multiplier is chosen so that

    $$ \frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z} =0. $$ -

    -We need to remember that we took \( dx \) and \( dy \) to be arbitrary and thus we must have +

    We need to remember that we took \( dx \) and \( dy \) to be arbitrary and thus we must have

    $$ \frac{\partial f}{\partial x}+\lambda\frac{\partial \phi}{\partial x} =0, $$ -and +

    and

    $$ \frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y} =0. $$ -When all these equations are satisfied, \( df=0 \). We have four unknowns, \( x,y,z \) and +

    When all these equations are satisfied, \( df=0 \). We have four unknowns, \( x,y,z \) and \( \lambda \). Actually we want only \( x,y,z \), \( \lambda \) needs not to be determined, it is therefore often called Lagrange's undetermined multiplier. If we have a set of constraints \( \phi_k \) we have the equations +

    $$ \frac{\partial f}{\partial x_i}+\sum_k\lambda_k\frac{\partial \phi_k}{\partial x_i} =0. $$ -

    -









    -

    Setting up the Problem

    -In order to solve the above problem, we define the following Lagrangian function to be minimized +









    +

    Setting up the Problem

    +

    In order to solve the above problem, we define the following Lagrangian function to be minimized

    $$ {\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right], $$ -where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \). +

    where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).

    -

    -Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain +

    Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain

    $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ -and +

    and

    $$ \frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i. $$ -Inserting these constraints into the equation for \( {\cal L} \) we obtain +

    Inserting these constraints into the equation for \( {\cal L} \) we obtain

    $$ {\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ -subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). +

    subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). We must in addition satisfy the Karush-Kuhn-Tucker (KKT) condition +

    $$ -\lambda_i\left[y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) -1\right] \hspace{0.1cm}\forall i. +\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i. $$ -
      -
    1. If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)=1 \) and we say that \( x_i \) is on the boundary.
      2. -
    2. If \( y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).
      3. +
    3. If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.
      4. +
    4. If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).
    +

    When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).

    -When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \). - -

    -The support vectors (the points that define the margin \( M \)) are the quantities we keep in order to make predictions. - -











    +

    The problem to solve

    -

    The problem to solve

    - -

    -We can rewrite +

    We can rewrite

    $$ {\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ -and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem +

    and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem

    $$ \frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\ y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\ @@ -665,145 +890,1503 @@ y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x \end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, $$ -subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +

    -











    +

    The last steps

    -

    The last steps

    - -

    -Solving the above problem, yields the values of \( \lambda_i \). +

    Solving the above problem, yields the values of \( \lambda_i \). To find the coefficients of your hyperplane we need simply to compute +

    $$ \boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i. $$ -With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via +

    With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via

    $$ -y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)=1, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1, $$ -resulting in +

    resulting in

    $$ -b = \frac{1}{y_i}-\boldsymbol{x}_1^T\boldsymbol{w}, +b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i, $$ -or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have +

    or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have

    $$ b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right). $$ -With our hyperplane coefficients we can use our classifier to assign any observation by simply using +

    With our hyperplane coefficients we can use our classifier to assign any observation by simply using

    $$ -y_i = \mathrm{sign}(\boldsymbol{x}_i^T\boldsymbol{w}+b). +y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b). $$ -Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier. +

    Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.

    -











    +

    A soft classifier

    -

    A soft classifier

    +

    Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.

    -

    -Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined. - -

    -Suppose now that the two classes overlap in feature space, as shown in Figure 12.1 of -Hastie et al. - -

    -One way to deal with this problem before we define the +

    Suppose now that classes overlap in feature space, as shown in the +figure here. One way to deal with this problem before we define the so-called kernel approach, is to allow a kind of slack in the sense that we allow some points to be on the wrong side of the margin. +

    -

    -We introduce thus the so-called slack variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and +

    We introduce thus the so-called slack variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and modify our previous equation +

    $$ -y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b)=1, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1, $$ -to +

    to

    $$ -y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b)=1-\xi_i, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i, $$ -with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \). +

    with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \). The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction -\( y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), +\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), we bound the total amount by which predictions fall on the wrong side of their margins. +

    -

    -Misclassifications occur when \( \xi_i > 1 \). Thus bounding the total sum by some value \( C \) bounds in turn the total number of +

    Misclassifications occur when \( \xi_i > 1 \). Thus bounding the total sum by some value \( C \) bounds in turn the total number of misclassifications. +

    -











    +

    Soft optmization problem

    -

    Soft optmization problem

    - -

    -This has in turn the consequences that we change our optmization problem to finding the minimum of +

    This has in turn the consequences that we change our optmization problem to finding the minimum of

    $$ {\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, $$ -subject to +

    subject to

    $$ -y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b)=1-\xi_i \hspace{0.1cm}\forall i, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, $$ -with the requirement \( \xi_i\geq 0 \). +

    with the requirement \( \xi_i\geq 0 \).

    -

    -Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain +

    Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain

    $$ \frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, $$ -and +

    and

    $$ \frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i, $$ -and +

    and

    $$ \lambda_i = C-\gamma_i \hspace{0.1cm}\forall i. $$ -Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before +

    Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before

    $$ {\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ -but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \). +

    but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \). We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads +

    $$ -\lambda_i\left[y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, +\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, $$ $$ \gamma_i\xi_i = 0, $$ -and +

    and

    $$ -y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. $$ +









    +

    Kernels and non-linearity

    + +

    The cases we have studied till now, were all characterized by two classes +with a close to linear separability. The classifiers we have described +so far find linear boundaries in our input feature space. It is +possible to make our procedure more flexible by exploring the feature +space using other basis expansions such as higher-order polynomials, +wavelets, splines etc. +

    + +

    If our feature space is not easy to separate, as shown in the figure +here, we can achieve a better separation by introducing more complex +basis functions. The ideal would be, as shown in the next figure, to, via a specific transformation to +obtain a separation between the classes which is almost linear. +

    + +

    The change of basis, from \( x\rightarrow z=\phi(x) \) leads to the same type of equations to be solved, except that +we need to introduce for example a polynomial transformation to a two-dimensional training set. +

    + + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import os
+
+np.random.seed(42)
+
+# To plot pretty figures
+import matplotlib
+import matplotlib.pyplot as plt
+plt.rcParams['axes.labelsize'] = 14
+plt.rcParams['xtick.labelsize'] = 12
+plt.rcParams['ytick.labelsize'] = 12
+
+
+from sklearn.svm import SVC
+from sklearn import datasets
+
+
+
+X1D = np.linspace(-4, 4, 9).reshape(-1, 1)
+X2D = np.c_[X1D, X1D**2]
+y = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])
+
+plt.figure(figsize=(11, 4))
+
+plt.subplot(121)
+plt.grid(True, which='both')
+plt.axhline(y=0, color='k')
+plt.plot(X1D[:, 0][y==0], np.zeros(4), "bs")
+plt.plot(X1D[:, 0][y==1], np.zeros(5), "g^")
+plt.gca().get_yaxis().set_ticks([])
+plt.xlabel(r"$x_1$", fontsize=20)
+plt.axis([-4.5, 4.5, -0.2, 0.2])
+
+plt.subplot(122)
+plt.grid(True, which='both')
+plt.axhline(y=0, color='k')
+plt.axvline(x=0, color='k')
+plt.plot(X2D[:, 0][y==0], X2D[:, 1][y==0], "bs")
+plt.plot(X2D[:, 0][y==1], X2D[:, 1][y==1], "g^")
+plt.xlabel(r"$x_1$", fontsize=20)
+plt.ylabel(r"$x_2$", fontsize=20, rotation=0)
+plt.gca().get_yaxis().set_ticks([0, 4, 8, 12, 16])
+plt.plot([-4.5, 4.5], [6.5, 6.5], "r--", linewidth=3)
+plt.axis([-4.5, 4.5, -1, 17])
+plt.subplots_adjust(right=1)
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    The equations

    + +

    Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)

    +$$ +z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right). +$$ + +

    With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)

    +$$ +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{z}_j, +$$ + +

    subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors

    +$$ +y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i, +$$ + +

    from which we also find \( b \). +To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kernel \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as +

    +$$ +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). +$$ + +

    For the above example, the kernel reads

    +$$ +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2. +$$ + +

    We note that this is nothing but the dot product of the two original +vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the +product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute +the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). +

    + +

    This leads to the so-called +kernel trick and the result leads to the same as if we went through +the trouble of performing the transformation +\( \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations. +

    + +









    +

    The problem to solve

    +

    Using our definition of the kernel We can rewrite again the Lagrangian

    +$$ +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j, +$$ + +

    subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem

    +$$ +\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ +y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ +\dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots \\ +y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ +\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, +$$ + +

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +If we add the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \). +

    + +

    We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type

    +$$ +\begin{align*} + &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f. +\end{align*} +$$ + +

    Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \). +Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into +\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \). +

    + +









    +

    Different kernels and Mercer's theorem

    + +

    There are several popular kernels being used. These are

    +
      +
    1. Linear: \( K(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{x}^T\boldsymbol{y} \),
      2. +
    2. Polynomial: \( K(\boldsymbol{x},\boldsymbol{y})=(\boldsymbol{x}^T\boldsymbol{y}+\gamma)^d \),
      3. +
    3. Gaussian Radial Basis Function: \( K(\boldsymbol{x},\boldsymbol{y})=\exp{\left(-\gamma\vert\vert\boldsymbol{x}-\boldsymbol{y}\vert\vert^2\right)} \),
      4. +
    4. Tanh: \( K(\boldsymbol{x},\boldsymbol{y})=\tanh{(\boldsymbol{x}^T\boldsymbol{y}+\gamma)} \),
      5. +
    +

    and many other ones.

    + +

    An important theorem for us is Mercer's +theorem. The +theorem states that if a kernel function \( K \) is symmetric, continuous +and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there +exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into +another space (possibly with much higher dimensions) such that +

    + +$$ +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). +$$ + +

    So you can use \( K \) as a kernel since you know \( \phi \) exists, even if +you don’t know what \( \phi \) is. +

    + +

    Note that some frequently used kernels (such as the Sigmoid kernel) +don’t respect all of Mercer’s conditions, yet they generally work well +in practice. +

    + +









    +

    The moons example

    + + +
    +
    +
    +
    +
    + 
    from __future__ import division, print_function, unicode_literals
+
+import numpy as np
+np.random.seed(42)
+
+import matplotlib
+import matplotlib.pyplot as plt
+plt.rcParams['axes.labelsize'] = 14
+plt.rcParams['xtick.labelsize'] = 12
+plt.rcParams['ytick.labelsize'] = 12
+
+
+from sklearn.svm import SVC
+from sklearn import datasets
+
+
+
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler
+from sklearn.svm import LinearSVC
+
+
+from sklearn.datasets import make_moons
+X, y = make_moons(n_samples=100, noise=0.15, random_state=42)
+
+def plot_dataset(X, y, axes):
+    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "bs")
+    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "g^")
+    plt.axis(axes)
+    plt.grid(True, which='both')
+    plt.xlabel(r"$x_1$", fontsize=20)
+    plt.ylabel(r"$x_2$", fontsize=20, rotation=0)
+
+plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+plt.show()
+
+from sklearn.datasets import make_moons
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import PolynomialFeatures
+
+polynomial_svm_clf = Pipeline([
+        ("poly_features", PolynomialFeatures(degree=3)),
+        ("scaler", StandardScaler()),
+        ("svm_clf", LinearSVC(C=10, loss="hinge", random_state=42))
+    ])
+
+polynomial_svm_clf.fit(X, y)
+
+def plot_predictions(clf, axes):
+    x0s = np.linspace(axes[0], axes[1], 100)
+    x1s = np.linspace(axes[2], axes[3], 100)
+    x0, x1 = np.meshgrid(x0s, x1s)
+    X = np.c_[x0.ravel(), x1.ravel()]
+    y_pred = clf.predict(X).reshape(x0.shape)
+    y_decision = clf.decision_function(X).reshape(x0.shape)
+    plt.contourf(x0, x1, y_pred, cmap=plt.cm.brg, alpha=0.2)
+    plt.contourf(x0, x1, y_decision, cmap=plt.cm.brg, alpha=0.1)
+
+plot_predictions(polynomial_svm_clf, [-1.5, 2.5, -1, 1.5])
+plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+
+plt.show()
+
+
+from sklearn.svm import SVC
+
+poly_kernel_svm_clf = Pipeline([
+        ("scaler", StandardScaler()),
+        ("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5))
+    ])
+poly_kernel_svm_clf.fit(X, y)
+
+poly100_kernel_svm_clf = Pipeline([
+        ("scaler", StandardScaler()),
+        ("svm_clf", SVC(kernel="poly", degree=10, coef0=100, C=5))
+    ])
+poly100_kernel_svm_clf.fit(X, y)
+
+plt.figure(figsize=(11, 4))
+
+plt.subplot(121)
+plot_predictions(poly_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])
+plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+plt.title(r"$d=3, r=1, C=5$", fontsize=18)
+
+plt.subplot(122)
+plot_predictions(poly100_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])
+plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+plt.title(r"$d=10, r=100, C=5$", fontsize=18)
+
+plt.show()
+
+def gaussian_rbf(x, landmark, gamma):
+    return np.exp(-gamma * np.linalg.norm(x - landmark, axis=1)**2)
+
+gamma = 0.3
+
+x1s = np.linspace(-4.5, 4.5, 200).reshape(-1, 1)
+x2s = gaussian_rbf(x1s, -2, gamma)
+x3s = gaussian_rbf(x1s, 1, gamma)
+
+XK = np.c_[gaussian_rbf(X1D, -2, gamma), gaussian_rbf(X1D, 1, gamma)]
+yk = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])
+
+plt.figure(figsize=(11, 4))
+
+plt.subplot(121)
+plt.grid(True, which='both')
+plt.axhline(y=0, color='k')
+plt.scatter(x=[-2, 1], y=[0, 0], s=150, alpha=0.5, c="red")
+plt.plot(X1D[:, 0][yk==0], np.zeros(4), "bs")
+plt.plot(X1D[:, 0][yk==1], np.zeros(5), "g^")
+plt.plot(x1s, x2s, "g--")
+plt.plot(x1s, x3s, "b:")
+plt.gca().get_yaxis().set_ticks([0, 0.25, 0.5, 0.75, 1])
+plt.xlabel(r"$x_1$", fontsize=20)
+plt.ylabel(r"Similarity", fontsize=14)
+plt.annotate(r'$\mathbf{x}$',
+             xy=(X1D[3, 0], 0),
+             xytext=(-0.5, 0.20),
+             ha="center",
+             arrowprops=dict(facecolor='black', shrink=0.1),
+             fontsize=18,
+            )
+plt.text(-2, 0.9, "$x_2$", ha="center", fontsize=20)
+plt.text(1, 0.9, "$x_3$", ha="center", fontsize=20)
+plt.axis([-4.5, 4.5, -0.1, 1.1])
+
+plt.subplot(122)
+plt.grid(True, which='both')
+plt.axhline(y=0, color='k')
+plt.axvline(x=0, color='k')
+plt.plot(XK[:, 0][yk==0], XK[:, 1][yk==0], "bs")
+plt.plot(XK[:, 0][yk==1], XK[:, 1][yk==1], "g^")
+plt.xlabel(r"$x_2$", fontsize=20)
+plt.ylabel(r"$x_3$  ", fontsize=20, rotation=0)
+plt.annotate(r'$\phi\left(\mathbf{x}\right)$',
+             xy=(XK[3, 0], XK[3, 1]),
+             xytext=(0.65, 0.50),
+             ha="center",
+             arrowprops=dict(facecolor='black', shrink=0.1),
+             fontsize=18,
+            )
+plt.plot([-0.1, 1.1], [0.57, -0.1], "r--", linewidth=3)
+plt.axis([-0.1, 1.1, -0.1, 1.1])
+    
+plt.subplots_adjust(right=1)
+
+plt.show()
+
+
+x1_example = X1D[3, 0]
+for landmark in (-2, 1):
+    k = gaussian_rbf(np.array([[x1_example]]), np.array([[landmark]]), gamma)
+    print("Phi({}, {}) = {}".format(x1_example, landmark, k))
+
+rbf_kernel_svm_clf = Pipeline([
+        ("scaler", StandardScaler()),
+        ("svm_clf", SVC(kernel="rbf", gamma=5, C=0.001))
+    ])
+rbf_kernel_svm_clf.fit(X, y)
+
+
+from sklearn.svm import SVC
+
+gamma1, gamma2 = 0.1, 5
+C1, C2 = 0.001, 1000
+hyperparams = (gamma1, C1), (gamma1, C2), (gamma2, C1), (gamma2, C2)
+
+svm_clfs = []
+for gamma, C in hyperparams:
+    rbf_kernel_svm_clf = Pipeline([
+            ("scaler", StandardScaler()),
+            ("svm_clf", SVC(kernel="rbf", gamma=gamma, C=C))
+        ])
+    rbf_kernel_svm_clf.fit(X, y)
+    svm_clfs.append(rbf_kernel_svm_clf)
+
+plt.figure(figsize=(11, 7))
+
+for i, svm_clf in enumerate(svm_clfs):
+    plt.subplot(221 + i)
+    plot_predictions(svm_clf, [-1.5, 2.5, -1, 1.5])
+    plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+    gamma, C = hyperparams[i]
+    plt.title(r"$\gamma = {}, C = {}$".format(gamma, C), fontsize=16)
+
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Mathematical optimization of convex functions

    + +

    A mathematical (quadratic) optimization problem, or just optimization problem, has the form

    +$$ +\begin{align*} + &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f. +\end{align*} +$$ + +

    subject to some constraints for say a selected set \( i=1,2,\dots, n \). +In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the +vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with. +

    + +

    In our case we are particularly interested in a class of optimization problems called convex optmization problems. +In our discussion on gradient descent methods we discussed at length the definition of a convex function. +

    + +

    Convex optimization problems play a central role in applied mathematics and we recommend strongly Boyd and Vandenberghe's text on the topics.

    + +









    +

    How do we solve these problems?

    + +

    If we use Python as programming language and wish to venture beyond +scikit-learn, tensorflow and similar software which makes our +lives so much easier, we need to dive into the wonderful world of +quadratic programming. We can, if we wish, solve the minimization +problem using say standard gradient methods or conjugate gradient +methods. However, these methods tend to exhibit a rather slow +converge. So, welcome to the promised land of quadratic programming. +

    + +

    The functions we need are contained in the quadratic programming package CVXOPT and we need to import it together with numpy as

    + + + +
    +
    +
    +
    +
    + 
    import numpy
+import cvxopt
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    This will make our life much easier. You don't need t write your own optimizer.

    + +









    +

    A simple example

    + +

    We remind ourselves about the general problem we want to solve

    +$$ +\begin{align*} + &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber + &\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f. +\end{align*} +$$ + +

    Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem

    +$$ +\begin{align*} + &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber + &\mathrm{subject to} \\ \nonumber + &x, y \geq 0 \\ \nonumber + &x+3y \geq 15 \\ \nonumber + &2x+5y \leq 100 \\ \nonumber + &3x+4y \leq 80. \\ \nonumber +\end{align*} +$$ + +

    The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)

    +$$ +\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}. +$$ + +

    Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation

    +$$ +\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +$$ + +

    We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix

    +$$ +\boldsymbol{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} +$$ + +

    is clearly positive semi-definite (all eigenvalues larger or equal zero). +Finally, the vector \( \boldsymbol{h} \) is defined as +

    +$$ +\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +$$ + +

    Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero +The following code solves the equations for us +

    + + +
    +
    +
    +
    +
    + 
    # Import the necessary packages
+import numpy
+from cvxopt import matrix
+from cvxopt import solvers
+P = matrix(numpy.diag([1,0]), tc=’d’)
+q = matrix(numpy.array([3,4]), tc=’d’)
+G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’)
+h = matrix(numpy.array([0,0,-15,100,80]), tc=’d’)
+# Construct the QP, invoke solver
+sol = solvers.qp(P,q,G,h)
+# Extract optimal value and solution
+sol[’x’] 
+sol[’primal objective’]
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Back to the more realistic cases

    + +

    We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the slack parameter \( C \) we have

    +$$ +\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ +y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ +\dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots \\ +y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ +\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda}, +$$ + +

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \). +

    + +code will be added + +









    +

    Summary of course

    + +









    +

    What? Me worry? No final exam in this course!

    +

    +
    +

    +
    +

    + +









    +

    Topics we have covered this year

    + +

    The course has two central parts

    + +
      +
    1. Statistical analysis and optimization of data
      2. +
    2. Machine learning
      3. +
    +









    +

    Statistical analysis and optimization of data

    + +

    The following topics have been discussed:

    +
      +
    1. Basic concepts, expectation values, variance, covariance, correlation functions and errors;
      2. +
    2. Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions;
      3. +
    3. Central elements from linear algebra, matrix inversion and SVD
      4. +
    4. Gradient methods for data optimization
      5. +
    5. Estimation of errors using cross-validation, bootstrapping and jackknife methods;
      6. +
    6. Practical optimization using Singular-value decomposition and least squares for parameterizing data.
      7. +
    7. Principal Component Analysis to reduce the number of features.
      8. +
    +









    +

    Machine learning

    + +

    The following topics will be covered

    +
      +
    1. Linear methods for regression and classification: +
        +
      1. Ordinary Least Squares
        2. +
      2. Ridge regression
        3. +
      3. Lasso regression
        4. +
      4. Logistic regression
        5. +
      +
    2. Neural networks and deep learning: +
        +
      1. Feed Forward Neural Networks
        2. +
      2. Convolutional Neural Networks
        3. +
      3. Recurrent Neural Networks
        4. +
      +
    3. Decisions trees and ensemble methods: +
        +
      1. Decision trees
        2. +
      2. Bagging and voting
        3. +
      3. Random forests
        4. +
      4. Boosting and gradient boosting
        5. +
      +
    4. Support vector machines +
        +
      1. Binary classification and multiclass classification
        2. +
      2. Kernel methods
        3. +
      3. Regression
        4. +
      +
    +









    +

    Learning outcomes and overarching aims of this course

    + +

    The course introduces a variety of central algorithms and methods +essential for studies of data analysis and machine learning. The +course is project based and through the various projects, normally +three, you will be exposed to fundamental research problems +in these fields, with the aim to reproduce state of the art scientific +results. The students will learn to develop and structure large codes +for studying these systems, get acquainted with computing facilities +and learn to handle large scientific projects. A good scientific and +ethical conduct is emphasized throughout the course. +

    + + +









    +

    Perspective on Machine Learning

    + +
      +
    1. Rapidly emerging application area
      2. +
    2. Experiment AND theory are evolving in many many fields. Still many low-hanging fruits.
      3. +
    3. Requires education/retraining for more widespread adoption
      4. +
    4. A lot of “word-of-mouth” development methods
      5. +
    +

    Huge amounts of data sets require automation, classical analysis tools often inadequate. +High energy physics hit this wall in the 90’s. +In 2009 single top quark production was determined via Boosted decision trees, Bayesian +Neural Networks, etc. +

    + +









    +

    Machine Learning Research

    + +

    Where to find recent results:

    +
      +
    1. Conference proceedings, arXiv and blog posts!
      2. +
    2. NIPS: Neural Information Processing Systems
      3. +
    3. ICLR: International Conference on Learning Representations
      4. +
    4. ICML: International Conference on Machine Learning
      5. +
    5. Journal of Machine Learning Research
      6. +
    6. Follow ML on ArXiv
      7. +
    +









    +

    Starting your Machine Learning Project

    + +
      +
    1. Identify problem type: classification, regression
      2. +
    2. Consider your data carefully
      3. +
    3. Choose a simple model that fits 1. and 2.
      4. +
    4. Consider your data carefully again! Think of data representation more carefully.
      5. +
    5. Based on your results, feedback loop to earliest possible point
      6. +
    +









    +

    Choose a Model and Algorithm

    + +
      +
    1. Supervised?
      2. +
    2. Start with the simplest model that fits your problem
      3. +
    3. Start with minimal processing of data
      4. +
    +









    +

    Preparing Your Data

    + +
      +
    1. Shuffle your data
      2. +
    2. Mean center your data
      3. + +
    3. Normalize the variance
      4. + +
    4. Whitening
      5. + +
    5. When to do train/test split?
      6. +
    +









    +

    Which Activation and Weights to Choose in Neural Networks

    + +
      +
    1. RELU? ELU?
      2. +
    2. Sigmoid or Tanh?
      3. +
    3. Set all weights to 0?
      4. + +
    4. Set all weights to random values?
      5. + +
    +









    +

    Optimization Methods and Hyperparameters

    +
      +
    1. Stochastic gradient descent +
        +
      1. Stochastic gradient descent + momentum
        2. +
      +
    2. State-of-the-art approaches:
      3. + +
    +

    Which regularization and hyperparameters? \( L_1 \) or \( L_2 \), soft +classifiers, depths of trees and many other. Need to explore a large +set of hyperparameters and regularization methods. +

    + +









    +

    Resampling

    + +

    When do we resample?

    + +
      +
    1. Bootstrap
      2. +
    2. Cross-validation
      3. +
    3. Jackknife and many other
      4. +
    +









    +

    Other courses on Data science and Machine Learning at UiO

    + +

    The link here https://www.mn.uio.no/english/research/about/centre-focus/innovation/data-science/studies/ gives an excellent overview of courses on Machine learning at UiO.

    + +
      +
    1. STK2100 Machine learning and statistical methods for prediction and classification.
      2. +
    2. IN3050/IN4050 Introduction to Artificial Intelligence and Machine Learning. Introductory course in machine learning and AI with an algorithmic approach.
      3. +
    3. STK-INF3000/4000 Selected Topics in Data Science. The course provides insight into selected contemporary relevant topics within Data Science.
      4. +
    4. IN4080 Natural Language Processing. Probabilistic and machine learning techniques applied to natural language processing.
      5. +
    5. STK-IN4300 – Statistical learning methods in Data Science. An advanced introduction to statistical and machine learning. For students with a good mathematics and statistics background.
      6. +
    6. IN-STK5000 Adaptive Methods for Data-Based Decision Making. Methods for adaptive collection and processing of data based on machine learning techniques.
      7. +
    7. IN5400/INF5860 – Machine Learning for Image Analysis. An introduction to deep learning with particular emphasis on applications within Image analysis, but useful for other application areas too.
      8. +
    8. TEK5040 – Dyp læring for autonome systemer. The course addresses advanced algorithms and architectures for deep learning with neural networks. The course provides an introduction to how deep-learning techniques can be used in the construction of key parts of advanced autonomous systems that exist in physical environments and cyber environments.
      9. +
    +









    +

    Additional courses of interest

    + +
      +
    1. STK4051 Computational Statistics
      2. +
    2. STK4021 Applied Bayesian Analysis and Numerical Methods
      3. +
    +









    +

    What's the future like?

    + +

    Based on multi-layer nonlinear neural networks, deep learning can +learn directly from raw data, automatically extract and abstract +features from layer to layer, and then achieve the goal of regression, +classification, or ranking. Deep learning has made breakthroughs in +computer vision, speech processing and natural language, and reached +or even surpassed human level. The success of deep learning is mainly +due to the three factors: big data, big model, and big computing. +

    + +

    In the past few decades, many different architectures of deep neural +networks have been proposed, such as +

    +
      +
    1. Convolutional neural networks, which are mostly used in image and video data processing, and have also been applied to sequential data such as text processing;
      2. +
    2. Recurrent neural networks, which can process sequential data of variable length and have been widely used in natural language understanding and speech processing;
      3. +
    3. Encoder-decoder framework, which is mostly used for image or sequence generation, such as machine translation, text summarization, and image captioning.
      4. +
    +









    +

    Types of Machine Learning, a repetition

    + +
    + +

    +

    The approaches to machine learning are many, but are often split into two main categories. +In supervised learning we know the answer to a problem, +and let the computer deduce the logic behind it. On the other hand, unsupervised learning +is a method for finding patterns and relationship in data sets without any prior knowledge of the system. +Some authours also operate with a third category, namely reinforcement learning. This is a paradigm +of learning inspired by behavioural psychology, where learning is achieved by trial-and-error, +solely from rewards and punishment. +

    + +

    Another way to categorize machine learning tasks is to consider the desired output of a system. +Some of the most common tasks are: +

    + + +
    + + +









    +

    Why Boltzmann machines?

    + +

    What is known as restricted Boltzmann Machines (RMB) have received a lot of attention lately. +One of the major reasons is that they can be stacked layer-wise to build deep neural networks that capture complicated statistics. +

    + +

    The original RBMs had just one visible layer and a hidden layer, but recently so-called Gaussian-binary RBMs have gained quite some popularity in imaging since they are capable of modeling continuous data that are common to natural images.

    + +

    Furthermore, they have been used to solve complicated quantum mechanical many-particle problems or classical statistical physics problems like the Ising and Potts classes of models.

    + +









    +

    Boltzmann Machines

    + +

    Why use a generative model rather than the more well known discriminative deep neural networks (DNN)?

    + + +









    +

    Some similarities and differences from DNNs

    + +
      +
    1. Both use gradient-descent based learning procedures for minimizing cost functions
      2. +
    2. Energy based models don't use backpropagation and automatic differentiation for computing gradients, instead turning to Markov Chain Monte Carlo methods.
      3. +
    3. DNNs often have several hidden layers. A restricted Boltzmann machine has only one hidden layer, however several RBMs can be stacked to make up Deep Belief Networks, of which they constitute the building blocks.
      4. +
    +

    History: The RBM was developed by amongst others Geoffrey Hinton, called by some the "Godfather of Deep Learning", working with the University of Toronto and Google.

    + +









    +

    Boltzmann machines (BM)

    + +
    + +

    +

    A BM is what we would call an undirected probabilistic graphical model +with stochastic continuous or discrete units. +

    +
    + +
    + +

    +

    It is interpreted as a stochastic recurrent neural network where the +state of each unit(neurons/nodes) depends on the units it is connected +to. The weights in the network represent thus the strength of the +interaction between various units/nodes. +

    +
    + +
    + +

    +

    It turns into a Hopfield network if we choose deterministic rather +than stochastic units. In contrast to a Hopfield network, a BM is a +so-called generative model. It allows us to generate new samples from +the learned distribution. +

    +
    + + +









    +

    A standard BM setup

    + +
    + +

    +

    A standard BM network is divided into a set of observable and visible units \( \hat{x} \) and a set of unknown hidden units/nodes \( \hat{h} \).

    +
    + + +
    + +

    +

    Additionally there can be bias nodes for the hidden and visible layers. These biases are normally set to \( 1 \).

    +
    + + +
    + +

    +

    BMs are stackable, meaning they cwe can train a BM which serves as input to another BM. We can construct deep networks for learning complex PDFs. The layers can be trained one after another, a feature which makes them popular in deep learning

    +
    + + +

    However, they are often hard to train. This leads to the introduction of so-called restricted BMs, or RBMS. +Here we take away all lateral connections between nodes in the visible layer as well as connections between nodes in the hidden layer. The network is illustrated in the figure below. +

    + +









    +

    The structure of the RBM network

    + +

    +
    +

    +
    +

    + +









    +

    The network

    + +The network layers: +
      +
    1. A function \( \mathbf{x} \) that represents the visible layer, a vector of \( M \) elements (nodes). This layer represents both what the RBM might be given as training input, and what we want it to be able to reconstruct. This might for example be given by the pixels of an image or coefficients representing speech, or the coordinates of a quantum mechanical state function.
      2. +
    2. The function \( \mathbf{h} \) represents the hidden, or latent, layer. A vector of \( N \) elements (nodes). Also called "feature detectors".
      3. +
    +









    +

    Goals

    + +

    The goal of the hidden layer is to increase the model's expressive +power. We encode complex interactions between visible variables by +introducing additional, hidden variables that interact with visible +degrees of freedom in a simple manner, yet still reproduce the complex +correlations between visible degrees in the data once marginalized +over (integrated out). +

    + +The network parameters, to be optimized/learned: +
      +
    1. \( \mathbf{a} \) represents the visible bias, a vector of same length as \( \mathbf{x} \).
      2. +
    2. \( \mathbf{b} \) represents the hidden bias, a vector of same lenght as \( \mathbf{h} \).
      3. +
    3. \( W \) represents the interaction weights, a matrix of size \( M\times N \).
      4. +
    +









    +

    Joint distribution

    + +

    The restricted Boltzmann machine is described by a Boltzmann distribution

    +$$ +\begin{align} + P_{rbm}(\mathbf{x},\mathbf{h}) = \frac{1}{Z} e^{-\frac{1}{T_0}E(\mathbf{x},\mathbf{h})}, +\label{_auto1} +\end{align} +$$ + +

    where \( Z \) is the normalization constant or partition function, defined as

    +$$ +\begin{align} + Z = \int \int e^{-\frac{1}{T_0}E(\mathbf{x},\mathbf{h})} d\mathbf{x} d\mathbf{h}. +\label{_auto2} +\end{align} +$$ + +

    It is common to ignore \( T_0 \) by setting it to one.

    + +









    +

    Network Elements, the energy function

    + +

    The function \( E(\mathbf{x},\mathbf{h}) \) gives the energy of a +configuration (pair of vectors) \( (\mathbf{x}, \mathbf{h}) \). The lower +the energy of a configuration, the higher the probability of it. This +function also depends on the parameters \( \mathbf{a} \), \( \mathbf{b} \) and +\( W \). Thus, when we adjust them during the learning procedure, we are +adjusting the energy function to best fit our problem. +

    + +

    An expression for the energy function is

    +$$ +E(\hat{x},\hat{h}) = -\sum_{ia}^{NA}b_i^a \alpha_i^a(x_i)-\sum_{jd}^{MD}c_j^d \beta_j^d(h_j)-\sum_{ijad}^{NAMD}b_i^a \alpha_i^a(x_i)c_j^d \beta_j^d(h_j)w_{ij}^{ad}. +$$ + +

    Here \( \beta_j^d(h_j) \) and \( \alpha_i^a(x_j) \) are so-called transfer functions that map a given input value to a desired feature value. The labels \( a \) and \( d \) denote that there can be multiple transfer functions per variable. The first sum depends only on the visible units. The second on the hidden ones. Note that there is no connection between nodes in a layer.

    + +

    The quantities \( b \) and \( c \) can be interpreted as the visible and hidden biases, respectively.

    + +

    The connection between the nodes in the two layers is given by the weights \( w_{ij} \).

    + +









    +

    Defining different types of RBMs

    +

    There are different variants of RBMs, and the differences lie in the types of visible and hidden units we choose as well as in the implementation of the energy function \( E(\mathbf{x},\mathbf{h}) \).

    + +
    +Binary-Binary RBM: +

    + +

    RBMs were first developed using binary units in both the visible and hidden layer. The corresponding energy function is defined as follows:

    +$$ +\begin{align} + E(\mathbf{x}, \mathbf{h}) = - \sum_i^M x_i a_i- \sum_j^N b_j h_j - \sum_{i,j}^{M,N} x_i w_{ij} h_j, +\label{_auto3} +\end{align} +$$ + +

    where the binary values taken on by the nodes are most commonly 0 and 1.

    +
    + +
    +Gaussian-Binary RBM: +

    + +

    Another varient is the RBM where the visible units are Gaussian while the hidden units remain binary:

    +$$ +\begin{align} + E(\mathbf{x}, \mathbf{h}) = \sum_i^M \frac{(x_i - a_i)^2}{2\sigma_i^2} - \sum_j^N b_j h_j - \sum_{i,j}^{M,N} \frac{x_i w_{ij} h_j}{\sigma_i^2}. +\label{_auto4} +\end{align} +$$ +
    + + +









    +

    More about RBMs

    +
      +
    1. Useful when we model continuous data (i.e., we wish \( \mathbf{x} \) to be continuous)
      2. +
    2. Requires a smaller learning rate, since there's no upper bound to the value a component might take in the reconstruction
      3. +
    +

    Other types of units include:

    +
      +
    1. Softmax and multinomial units
      2. +
    2. Gaussian visible and hidden units
      3. +
    3. Binomial units
      4. +
    4. Rectified linear units
      5. +
    +

    To read more, see Lectures on Boltzmann machines in Physics.

    + +









    +

    Autoencoders: Overarching view

    + +

    Autoencoders are artificial neural networks capable of learning +efficient representations of the input data (these representations are called codings) without +any supervision (i.e., the training set is unlabeled). These codings +typically have a much lower dimensionality than the input data, making +autoencoders useful for dimensionality reduction. +

    + +

    More importantly, autoencoders act as powerful feature detectors, and +they can be used for unsupervised pretraining of deep neural networks. +

    + +

    Lastly, they are capable of randomly generating new data that looks +very similar to the training data; this is called a generative +model. For example, you could train an autoencoder on pictures of +faces, and it would then be able to generate new faces. Surprisingly, +autoencoders work by simply learning to copy their inputs to their +outputs. This may sound like a trivial task, but we will see that +constraining the network in various ways can make it rather +difficult. For example, you can limit the size of the internal +representation, or you can add noise to the inputs and train the +network to recover the original inputs. These constraints prevent the +autoencoder from trivially copying the inputs directly to the outputs, +which forces it to learn efficient ways of representing the data. In +short, the codings are byproducts of the autoencoder’s attempt to +learn the identity function under some constraints. +

    + +Video on autoencoders + +

    See also A. Geron's textbook, chapter 15.

    + +









    +

    Bayesian Machine Learning

    + +

    This is an important topic if we aim at extracting a probability +distribution. This gives us also a confidence interval and error +estimates. +

    + +

    Bayesian machine learning allows us to encode our prior beliefs about +what those models should look like, independent of what the data tells +us. This is especially useful when we don’t have a ton of data to +confidently learn our model. +

    + +Video on Bayesian deep learning + +

    See also the slides here.

    + +









    +

    Reinforcement Learning

    + +

    Reinforcement Learning (RL) is one of the most exciting fields of +Machine Learning today, and also one of the oldest. It has been around +since the 1950s, producing many interesting applications over the +years. +

    + +

    It studies +how agents take actions based on trial and error, so as to maximize +some notion of cumulative reward in a dynamic system or +environment. Due to its generality, the problem has also been studied +in many other disciplines, such as game theory, control theory, +operations research, information theory, multi-agent systems, swarm +intelligence, statistics, and genetic algorithms. +

    + +

    In March 2016, AlphaGo, a computer program that plays the board game +Go, beat Lee Sedol in a five-game match. This was the first time a +computer Go program had beaten a 9-dan (highest rank) professional +without handicaps. AlphaGo is based on deep convolutional neural +networks and reinforcement learning. AlphaGo’s victory was a major +milestone in artificial intelligence and it has also made +reinforcement learning a hot research area in the field of machine +learning. +

    + +

    Lecture on Reinforcement Learning.

    + +

    See also A. Geron's textbook, chapter 16.

    + +









    +

    Transfer learning

    + +

    The goal of transfer learning is to transfer the model or knowledge +obtained from a source task to the target task, in order to resolve +the issues of insufficient training data in the target task. The +rationality of doing so lies in that usually the source and target +tasks have inter-correlations, and therefore either the features, +samples, or models in the source task might provide useful information +for us to better solve the target task. Transfer learning is a hot +research topic in recent years, with many problems still waiting to be studied. +

    + +

    Lecture on transfer learning.

    + +









    +

    Adversarial learning

    + +

    The conventional deep generative model has a potential problem: the +model tends to generate extreme instances to maximize the +probabilistic likelihood, which will hurt its performance. Adversarial +learning utilizes the adversarial behaviors (e.g., generating +adversarial instances or training an adversarial model) to enhance the +robustness of the model and improve the quality of the generated +data. In recent years, one of the most promising unsupervised learning +technologies, generative adversarial networks (GAN), has already been +successfully applied to image, speech, and text. +

    + +

    Lecture on adversial learning.

    + +









    +

    Dual learning

    + +

    Dual learning is a new learning paradigm, the basic idea of which is +to use the primal-dual structure between machine learning tasks to +obtain effective feedback/regularization, and guide and strengthen the +learning process, thus reducing the requirement of large-scale labeled +data for deep learning. The idea of dual learning has been applied to +many problems in machine learning, including machine translation, +image style conversion, question answering and generation, image +classification and generation, text classification and generation, +image-to-text, and text-to-image. +

    + +









    +

    Distributed machine learning

    + +

    Distributed computation will speed up machine learning algorithms, +significantly improve their efficiency, and thus enlarge their +application. When distributed meets machine learning, more than just +implementing the machine learning algorithms in parallel is required. +

    + +









    +

    Meta learning

    + +

    Meta learning is an emerging research direction in machine +learning. Roughly speaking, meta learning concerns learning how to +learn, and focuses on the understanding and adaptation of the learning +itself, instead of just completing a specific learning task. That is, +a meta learner needs to be able to evaluate its own learning methods +and adjust its own learning methods according to specific learning +tasks. +

    + +









    +

    The Challenges Facing Machine Learning

    + +

    While there has been much progress in machine learning, there are also challenges.

    + +

    For example, the mainstream machine learning technologies are +black-box approaches, making us concerned about their potential +risks. To tackle this challenge, we may want to make machine learning +more explainable and controllable. As another example, the +computational complexity of machine learning algorithms is usually +very high and we may want to invent lightweight algorithms or +implementations. Furthermore, in many domains such as physics, +chemistry, biology, and social sciences, people usually seek elegantly +simple equations (e.g., the Schrödinger equation) to uncover the +underlying laws behind various phenomena. In the field of machine +learning, can we reveal simple laws instead of designing more complex +models for data fitting? Although there are many challenges, we are +still very optimistic about the future of machine learning. As we look +forward to the future, here are what we think the research hotspots in +the next ten years will be. +

    + +

    See the article on Discovery of Physics From Data: Universal Laws and Discrepancies

    + +









    +

    Explainable machine learning

    + +

    Machine learning, especially deep learning, evolves rapidly. The +ability gap between machine and human on many complex cognitive tasks +becomes narrower and narrower. However, we are still in the very early +stage in terms of explaining why those effective models work and how +they work. +

    + +

    What is missing: the gap between correlation and causation. Standard Machine Learning is based on what e have called a frequentist approach.

    + +

    Most +machine learning techniques, especially the statistical ones, depend +highly on correlations in data sets to make predictions and analyses. In +contrast, rational humans tend to reply on clear and trustworthy +causality relations obtained via logical reasoning on real and clear +facts. It is one of the core goals of explainable machine learning to +transition from solving problems by data correlation to solving +problems by logical reasoning. +

    + +Bayesian Machine Learning is one of the exciting research directions in this field. + +









    +

    Quantum machine learning

    + +

    Quantum machine learning is an emerging interdisciplinary research +area at the intersection of quantum computing and machine learning. +

    + +

    Quantum computers use effects such as quantum coherence and quantum +entanglement to process information, which is fundamentally different +from classical computers. Quantum algorithms have surpassed the best +classical algorithms in several problems (e.g., searching for an +unsorted database, inverting a sparse matrix), which we call quantum +acceleration. +

    + +

    When quantum computing meets machine learning, it can be a mutually +beneficial and reinforcing process, as it allows us to take advantage +of quantum computing to improve the performance of classical machine +learning algorithms. In addition, we can also use the machine learning +algorithms (on classic computers) to analyze and improve quantum +computing systems. +

    + +

    Lecture on Quantum ML.

    + +

    Read interview with Maria Schuld on her work on Quantum Machine Learning. See also her recent textbook.

    + +









    +

    Quantum machine learning algorithms based on linear algebra

    + +

    Many quantum machine learning algorithms are based on variants of +quantum algorithms for solving linear equations, which can efficiently +solve N-variable linear equations with complexity of O(log2 N) under +certain conditions. The quantum matrix inversion algorithm can +accelerate many machine learning methods, such as least square linear +regression, least square version of support vector machine, Gaussian +process, and more. The training of these algorithms can be simplified +to solve linear equations. The key bottleneck of this type of quantum +machine learning algorithms is data input—that is, how to initialize +the quantum system with the entire data set. Although efficient +data-input algorithms exist for certain situations, how to efficiently +input data into a quantum system is as yet unknown for most cases. +

    + +









    +

    Quantum reinforcement learning

    + +

    In quantum reinforcement learning, a quantum agent interacts with the +classical environment to obtain rewards from the environment, so as to +adjust and improve its behavioral strategies. In some cases, it +achieves quantum acceleration by the quantum processing capabilities +of the agent or the possibility of exploring the environment through +quantum superposition. Such algorithms have been proposed in +superconducting circuits and systems of trapped ions. +

    + +









    +

    Quantum deep learning

    + +

    Dedicated quantum information processors, such as quantum annealers +and programmable photonic circuits, are well suited for building deep +quantum networks. The simplest deep quantum network is the Boltzmann +machine. The classical Boltzmann machine consists of bits with tunable +interactions and is trained by adjusting the interaction of these bits +so that the distribution of its expression conforms to the statistics +of the data. To quantize the Boltzmann machine, the neural network can +simply be represented as a set of interacting quantum spins that +correspond to an adjustable Ising model. Then, by initializing the +input neurons in the Boltzmann machine to a fixed state and allowing +the system to heat up, we can read out the output qubits to get the +result. +

    + +









    +

    Social machine learning

    + +

    Machine learning aims to imitate how humans +learn. While we have developed successful machine learning algorithms, +until now we have ignored one important fact: humans are social. Each +of us is one part of the total society and it is difficult for us to +live, learn, and improve ourselves, alone and isolated. Therefore, we +should design machines with social properties. Can we let machines +evolve by imitating human society so as to achieve more effective, +intelligent, interpretable “social machine learning”? +

    + +

    And much more.

    + +









    +

    The last words?

    + +

    Early computer scientist Alan Kay said, The best way to predict the +future is to create it. Therefore, all machine learning +practitioners, whether scholars or engineers, professors or students, +need to work together to advance these important research +topics. Together, we will not just predict the future, but create it. +

    + +









    +

    Best wishes to you all and thanks so much for your heroic efforts this semester

    + +

    +
    +

    +
    +

    + - -
    - © 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week47/ipynb/ipynb-week47-src.tar.gz b/doc/pub/week47/ipynb/ipynb-week47-src.tar.gz index caaf2e5c8..d97115ccc 100644 Binary files a/doc/pub/week47/ipynb/ipynb-week47-src.tar.gz and b/doc/pub/week47/ipynb/ipynb-week47-src.tar.gz differ diff --git a/doc/pub/week47/ipynb/week47.ipynb b/doc/pub/week47/ipynb/week47.ipynb index 8a476bec3..8b1006405 100644 --- a/doc/pub/week47/ipynb/week47.ipynb +++ b/doc/pub/week47/ipynb/week47.ipynb @@ -2,59 +2,56 @@ "cells": [ { "cell_type": "markdown", - "metadata": {}, + "id": "14292f55", + "metadata": { + "editable": true + }, "source": [ - "\n", - "# Week 47: Support Vector Machines\n", - "\n", - " \n", + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "6b80e5f6", + "metadata": { + "editable": true + }, + "source": [ + "# Week 47: Support Vector Machines and Summary of Course\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Nov 26, 2020**\n", - "\n", - "Copyright 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", - "\n", - "\n", + "Date: **Nov 20, 2021**\n", "\n", + "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license" + ] + }, + { + "cell_type": "markdown", + "id": "1bc032ad", + "metadata": { + "editable": true + }, + "source": [ "## Overview of week 47\n", "\n", - "* **Thursday**: Support Vector Machines, classification and regression. [Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember19.mp4?vrtx=view-as-webpage)\n", + "* **Thursday**: Support Vector Machines, classification and regression. \n", "\n", - "* **Friday**: Workshop on project 3. [Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember20.mp4?vrtx=view-as-webpage)\n", + "* **Friday**: Support Vector Machines and Summary of Course\n", "\n", "Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) of Hastie et al contains also a good discussion.\n", "\n", - "[See overview video on Support Vector Machines](https://www.youtube.com/watch?v=efR1C6CvhmE&ab_channel=StatQuestwithJoshStarmer). See also [this video](https://www.youtube.com/watch?v=N1vOgolbjSc&ab_channel=AliceZhao).\n", - "\n", - "\n", - "## Thursday\n", - "\n", - "We start with our final topic this semester, Support Vector Machines\n", - "\n", - "## Friday\n", - "\n", - "Friday's lecture is split in two parts. The first lecture is deveoted to a presentation of possible data sets and projects for project 3. The second lecture continues with support vector machines. It may happen that we will need parts of the second lecture as well for the project ideas.\n", - "\n", - "## Workshop plan Friday (first lecture)\n", - "\n", - "Here are the various projects that will be presented during the first lecture (and possibly parts of the second lecture as well). *Titles are tentative*. Talks are approx 5-10 mins with roughly 5 minutes for discussions.\n", - "\n", - "* Maria Emine Nylund: **Lego Bricks Classifier**\n", - "\n", - "* Fabio Rodrigues Pereira: **Financial Machine Learning**\n", - "\n", - "* Markus Borud Pettersen: **Machine Learning and Brain Grid Cells**\n", - "\n", - "* Jing Sun and Endrias Getachew Asgedom: **Machine learning-based approaches to denoising microseismic data**\n", - "\n", - "* Felicia Jacobsen: **Analysis of Breast Cancer Data**\n", - "\n", - "* Simon Elias Schrader: **Predicting atomization energies of molecules**\n", - "\n", - "* Varvara Bazilova and Sergio Andres Diaz Mesa: **Glacier Mapping and Machine Learning**\n", - "\n", - "* Gert Werner Kluge, Hanna Alida Fossen Hardersen and Sushma Sharma Adhikari: **Gamma ray signals stemming from dark matter in the galactic center**\n", - "\n", + "[See overview video on Support Vector Machines](https://www.youtube.com/watch?v=efR1C6CvhmE&ab_channel=StatQuestwithJoshStarmer). See also [this video](https://www.youtube.com/watch?v=N1vOgolbjSc&ab_channel=AliceZhao)." + ] + }, + { + "cell_type": "markdown", + "id": "847c9d3a", + "metadata": { + "editable": true + }, + "source": [ "## Support Vector Machines, overarching aims\n", "\n", "A Support Vector Machine (SVM) is a very powerful and versatile\n", @@ -67,7 +64,7 @@ "\n", "The case with two well-separated classes only can be understood in an\n", "intuitive way in terms of lines in a two-dimensional space separating\n", - "the two classes.\n", + "the two classes (see figure below).\n", "\n", "The basic mathematics behind the SVM is however less familiar to most of us. \n", "It relies on the definition of hyperplanes and the\n", @@ -79,8 +76,16 @@ "latter introduces a so-called softening parameter to be discussed\n", "below. We distinguish also between linear and non-linear\n", "approaches. The latter are the most frequent ones since it is rather\n", - "unlikely that we can separate classes easily by say straight lines.\n", - "\n", + "unlikely that we can separate classes easily by say straight lines." + ] + }, + { + "cell_type": "markdown", + "id": "3b791677", + "metadata": { + "editable": true + }, + "source": [ "## Hyperplanes and all that\n", "\n", "The theory behind support vector machines (SVM hereafter) is based on\n", @@ -98,8 +103,10 @@ { "cell_type": "code", "execution_count": 1, + "id": "ad80a795", "metadata": { - "collapsed": false + "collapsed": false, + "editable": true }, "outputs": [], "source": [ @@ -175,7 +182,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2d93325e", + "metadata": { + "editable": true + }, "source": [ "## What is a hyperplane?\n", "\n", @@ -192,7 +202,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "217d510d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "b+w_1x_1+w_2x_2=0,\n", @@ -201,7 +214,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f0887e40", + "metadata": { + "editable": true + }, "source": [ "where $b$ is the intercept and $w_1$ and $w_2$ define the elements of a vector orthogonal to the line \n", "$b+w_1x_1+w_2x_2=0$. \n", @@ -211,7 +227,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e1e43dfe", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}^T\\boldsymbol{w}+b=0.\n", @@ -220,17 +239,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e1cf4d6f", + "metadata": { + "editable": true + }, "source": [ "## A $p$-dimensional space of features\n", "\n", "We limit ourselves to two classes of outputs $y_i$ and assign these classes the values $y_i = \\pm 1$. \n", - "In a $p$-dimensional space of say $p$ features we have a hyperplane defined as" + "In a $p$-dimensional space of say $p$ features we have a hyperplane defines as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "92f49457", + "metadata": { + "editable": true + }, "source": [ "$$\n", "b+wx_1+w_2x_2+\\dots +w_px_p=0.\n", @@ -239,7 +264,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ea725e8b", + "metadata": { + "editable": true + }, "source": [ "If we define a \n", "matrix $\\boldsymbol{X}=\\left[\\boldsymbol{x}_1,\\boldsymbol{x}_2,\\dots, \\boldsymbol{x}_p\\right]$\n", @@ -248,7 +276,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a8319348", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_i = \\begin{bmatrix} x_{i1} \\\\ x_{i2} \\\\ \\dots \\\\ \\dots \\\\ x_{ip} \\end{bmatrix}.\n", @@ -257,14 +288,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "86169d95", + "metadata": { + "editable": true + }, "source": [ "If the above condition is not met for a given vector $\\boldsymbol{x}_i$ we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "f8a6e5de", + "metadata": { + "editable": true + }, "source": [ "$$\n", "b+w_1x_{i1}+w_2x_{i2}+\\dots +w_px_{ip} >0,\n", @@ -273,7 +310,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "dca365c8", + "metadata": { + "editable": true + }, "source": [ "if our output $y_i=1$.\n", "In this case we say that $\\boldsymbol{x}_i$ lies on one of the sides of the hyperplane and if" @@ -281,7 +321,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1a9d1fc9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "b+w_1x_{i1}+w_2x_{i2}+\\dots +w_px_{ip} < 0,\n", @@ -290,7 +333,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2c6da3c8", + "metadata": { + "editable": true + }, "source": [ "for the class of observations $y_i=-1$, \n", "then $\\boldsymbol{x}_i$ lies on the other side. \n", @@ -300,7 +346,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f75d02dd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y_i\\left(b+w_1x_{i1}+w_2x_{i2}+\\dots +w_px_{ip}\\right) > 0.\n", @@ -309,11 +358,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ca647eb8", + "metadata": { + "editable": true + }, + "source": [ + "When we try to separate hyperplanes, if it exists, we can use it to construct a natural classifier: a test observation is assigned a given class depending on which side of the hyperplane it is located." + ] + }, + { + "cell_type": "markdown", + "id": "d1a0cf9d", + "metadata": { + "editable": true + }, "source": [ - "When we try to separate hyperplanes, if it exists, we can use it to construct a natural classifier: a test observation is assigned a given class depending on which side of the hyperplane it is located.\n", - "\n", - "\n", "## The two-dimensional case\n", "\n", "Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional\n", @@ -323,7 +382,7 @@ "plane that has the maximum margin, i.e the maximum distance between\n", "data points of both classes. Maximizing the margin distance provides\n", "some reinforcement so that future data points can be classified with\n", - "more confidence. Figure 12.1 of Hastie et al is a good illustration.\n", + "more confidence.\n", "\n", "What a linear classifier attempts to accomplish is to split the\n", "feature space into two half spaces by placing a hyperplane between the\n", @@ -333,8 +392,16 @@ "\n", "Unfortunately there are many ways in which we can place a hyperplane\n", "to divide the data. Below is an example of two candidate hyperplanes\n", - "for our data sample.\n", - "\n", + "for our data sample." + ] + }, + { + "cell_type": "markdown", + "id": "5dc7f973", + "metadata": { + "editable": true + }, + "source": [ "## Getting into the details\n", "\n", "Let us define the function" @@ -342,28 +409,36 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "52aeefde", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "f(x) = \\boldsymbol{x}^T\\boldsymbol{w}+b = 0,\n", + "f(x) = \\boldsymbol{w}^T\\boldsymbol{x}+b = 0,\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "a7415eca", + "metadata": { + "editable": true + }, "source": [ - "as the function that determines the line $L$ that separates two classes (our two features), see Figure 12.1 of [Hastie et al](https://www.springer.com/gp/book/9780387848570).\n", + "as the function that determines the line $L$ that separates two classes (our two features), see the figure here. \n", "\n", - "\n", - "Any point defined by $\\boldsymbol{x}_i$ and $\\boldsymbol{x}_2$ on the line $L$ will satisfy $\\boldsymbol{x}^T(\\boldsymbol{w}_1-\\boldsymbol{x}_2)=0$. \n", + "Any point defined by $\\boldsymbol{x}_i$ and $\\boldsymbol{x}_2$ on the line $L$ will satisfy $\\boldsymbol{w}^T(\\boldsymbol{x}_1-\\boldsymbol{x}_2)=0$. \n", "\n", "The signed distance $\\delta$ from any point defined by a vector $\\boldsymbol{x}$ and a point $\\boldsymbol{x}_0$ on the line $L$ is then" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "7d38c251", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\delta = \\frac{1}{\\vert\\vert \\boldsymbol{w}\\vert\\vert}(\\boldsymbol{w}^T\\boldsymbol{x}+b).\n", @@ -372,7 +447,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "495c531b", + "metadata": { + "editable": true + }, "source": [ "## First attempt at a minimization approach\n", "\n", @@ -383,23 +461,32 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "bc38eb64", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "C(\\boldsymbol{w},b) = -\\sum_{i\\in M} y_i(\\boldsymbol{x}^T\\boldsymbol{w}_i+b).\n", + "C(\\boldsymbol{w},b) = -\\sum_{i\\in M} y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b).\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "dcfc773e", + "metadata": { + "editable": true + }, "source": [ "We could now for example define all values $y_i =1$ as misclassified in case we have $\\boldsymbol{w}^T\\boldsymbol{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "eb09ba6c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C}{\\partial b} = -\\sum_{i\\in M} y_i,\n", @@ -408,14 +495,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "df2c2a65", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "76d329d2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C}{\\partial \\boldsymbol{w}} = -\\sum_{i\\in M} y_ix_i.\n", @@ -424,7 +517,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ff04dcd7", + "metadata": { + "editable": true + }, "source": [ "## Solving the equations\n", "\n", @@ -433,7 +529,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1b1507e4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "b \\leftarrow b +\\eta \\frac{\\partial C}{\\partial b},\n", @@ -442,14 +541,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "070a2947", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "4ec9d1a9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{w} \\leftarrow \\boldsymbol{w} +\\eta \\frac{\\partial C}{\\partial \\boldsymbol{w}},\n", @@ -458,31 +563,54 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f3dd50d5", + "metadata": { + "editable": true + }, "source": [ - "where $\\eta$ is our by now well-known learning rate. \n", - "\n", - "\n", - "## Can we code this?\n", + "where $\\eta$ is our by now well-known learning rate." + ] + }, + { + "cell_type": "markdown", + "id": "fb099fc3", + "metadata": { + "editable": true + }, + "source": [ + "## Code Example\n", "\n", "The equations we discussed above can be coded rather easily (the\n", - "framework is similar to what we developed for logistic regression). We\n", - "can set up a simple case with two classes only and we want to find a\n", - "line which separates them the best possible way.\n", - "\n", + "framework is similar to what we developed for logistic\n", + "regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way." + ] + }, + { + "cell_type": "markdown", + "id": "8a729395", + "metadata": { + "editable": true + }, + "source": [ + "## Problems with the Simpler Approach\n", "\n", "There are however problems with this approach, although it looks\n", - "pretty straightforward to implement. When running a code for such a\n", - "case we can easily end up with many diffeent lines which separate the\n", - "two classes.\n", - "\n", + "pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes.\n", "\n", "For small\n", "gaps between the entries, we may also end up needing many iterations\n", "before the solutions converge and if the data cannot be separated\n", "properly into two distinct classes, we may not experience a converge\n", - "at all.\n", - "\n", + "at all." + ] + }, + { + "cell_type": "markdown", + "id": "405322b0", + "metadata": { + "editable": true + }, + "source": [ "## A better approach\n", "\n", "A better approach is rather to try to define a large margin between\n", @@ -494,16 +622,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8c7d68fe", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "y_i(\\boldsymbol{x}^T\\boldsymbol{w}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, p.\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, p.\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "b75a185a", + "metadata": { + "editable": true + }, "source": [ "All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $b$ and $w_1$ and $w_2$ define this line. \n", "\n", @@ -512,32 +646,44 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "45e6204f", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "\\frac{1}{\\vert \\vert \\boldsymbol{w}\\vert\\vert}y_i(\\boldsymbol{x}^T\\boldsymbol{w}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, n,\n", + "\\frac{1}{\\vert \\vert \\boldsymbol{w}\\vert\\vert}y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, n,\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "20514a04", + "metadata": { + "editable": true + }, "source": [ "or just" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "b60eeebc", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "y_i(\\boldsymbol{x}^T\\boldsymbol{w}_i+b) \\geq M\\vert \\vert \\boldsymbol{w}\\vert\\vert \\hspace{0.1cm}\\forall i.\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M\\vert \\vert \\boldsymbol{w}\\vert\\vert \\hspace{0.1cm}\\forall i.\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "20ccb807", + "metadata": { + "editable": true + }, "source": [ "If we scale the equation so that $\\vert \\vert \\boldsymbol{w}\\vert\\vert = 1/M$, we have to find the minimum of \n", "$\\boldsymbol{w}^T\\boldsymbol{w}=\\vert \\vert \\boldsymbol{w}\\vert\\vert$ (the norm) subject to the condition" @@ -545,7 +691,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d8110b1c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq 1 \\hspace{0.1cm}\\forall i.\n", @@ -554,13 +703,24 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c839f0aa", + "metadata": { + "editable": true + }, "source": [ - "We have thus defined our margin as the inverse of the norm of\n", + "We have thus defined our margin as the invers of the norm of\n", "$\\boldsymbol{w}$. We want to minimize the norm in order to have a as large as\n", "possible margin $M$. Before we proceed, we need to remind ourselves\n", - "about Lagrangian multipliers and optimzation problems.\n", - "\n", + "about Lagrangian multipliers." + ] + }, + { + "cell_type": "markdown", + "id": "1208bf5a", + "metadata": { + "editable": true + }, + "source": [ "## A quick Reminder on Lagrangian Multipliers\n", "\n", "Consider a function of three independent variables $f(x,y,z)$ . For the function $f$ to be an\n", @@ -569,7 +729,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a61cd44c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "df=0.\n", @@ -578,14 +741,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8f721aa8", + "metadata": { + "editable": true + }, "source": [ "A necessary and sufficient condition is" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "f0e46bbc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial f}{\\partial x} =\\frac{\\partial f}{\\partial y}=\\frac{\\partial f}{\\partial z}=0,\n", @@ -594,14 +763,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "474e989f", + "metadata": { + "editable": true + }, "source": [ "due to" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "05307bfe", + "metadata": { + "editable": true + }, "source": [ "$$\n", "df = \\frac{\\partial f}{\\partial x}dx+\\frac{\\partial f}{\\partial y}dy+\\frac{\\partial f}{\\partial z}dz.\n", @@ -610,7 +785,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d3809acd", + "metadata": { + "editable": true + }, "source": [ "In many problems the variables $x,y,z$ are often subject to constraints (such as those above for the margin)\n", "so that they are no longer all independent. It is possible at least in principle to use each \n", @@ -624,7 +802,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "349f6a53", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\phi(x,y,z) = 0,\n", @@ -633,14 +814,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "52c50006", + "metadata": { + "editable": true + }, "source": [ "resulting in" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "2e24de43", + "metadata": { + "editable": true + }, "source": [ "$$\n", "d\\phi = \\frac{\\partial \\phi}{\\partial x}dx+\\frac{\\partial \\phi}{\\partial y}dy+\\frac{\\partial \\phi}{\\partial z}dz =0.\n", @@ -649,14 +836,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "9d41d3eb", + "metadata": { + "editable": true + }, "source": [ "Now we cannot set anymore" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "e12b5bf5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial f}{\\partial x} =\\frac{\\partial f}{\\partial y}=\\frac{\\partial f}{\\partial z}=0,\n", @@ -665,13 +858,24 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "004c70bc", + "metadata": { + "editable": true + }, "source": [ "if $df=0$ is wanted\n", "because there are now only two independent variables! Assume $x$ and $y$ are the independent \n", "variables.\n", - "Then $dz$ is no longer arbitrary.\n", - "\n", + "Then $dz$ is no longer arbitrary." + ] + }, + { + "cell_type": "markdown", + "id": "ba0b9524", + "metadata": { + "editable": true + }, + "source": [ "## Adding the Multiplier\n", "\n", "However, we can add to" @@ -679,7 +883,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1aca26a8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "df = \\frac{\\partial f}{\\partial x}dx+\\frac{\\partial f}{\\partial y}dy+\\frac{\\partial f}{\\partial z}dz,\n", @@ -688,14 +895,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3eee29ae", + "metadata": { + "editable": true + }, "source": [ "a multiplum of $d\\phi$, viz. $\\lambda d\\phi$, resulting in" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "138716a6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "df+\\lambda d\\phi = (\\frac{\\partial f}{\\partial z}+\\lambda\n", @@ -706,14 +919,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c0b94d06", + "metadata": { + "editable": true + }, "source": [ "Our multiplier is chosen so that" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "42eee105", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial f}{\\partial z}+\\lambda\\frac{\\partial \\phi}{\\partial z} =0.\n", @@ -722,14 +941,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6d3fa055", + "metadata": { + "editable": true + }, "source": [ "We need to remember that we took $dx$ and $dy$ to be arbitrary and thus we must have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "1a7c9969", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial f}{\\partial x}+\\lambda\\frac{\\partial \\phi}{\\partial x} =0,\n", @@ -738,14 +963,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "4c58c3a9", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "5053ea84", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial f}{\\partial y}+\\lambda\\frac{\\partial \\phi}{\\partial y} =0.\n", @@ -754,7 +985,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3503e4c8", + "metadata": { + "editable": true + }, "source": [ "When all these equations are satisfied, $df=0$. We have four unknowns, $x,y,z$ and\n", "$\\lambda$. Actually we want only $x,y,z$, $\\lambda$ needs not to be determined, \n", @@ -765,7 +999,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2e922854", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial f}{\\partial x_i}+\\sum_k\\lambda_k\\frac{\\partial \\phi_k}{\\partial x_i} =0.\n", @@ -774,7 +1011,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f7a79c79", + "metadata": { + "editable": true + }, "source": [ "## Setting up the Problem\n", "In order to solve the above problem, we define the following Lagrangian function to be minimized" @@ -782,7 +1022,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "deb1367f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\cal L}(\\lambda,b,\\boldsymbol{w})=\\frac{1}{2}\\boldsymbol{w}^T\\boldsymbol{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)-1\\right],\n", @@ -791,7 +1034,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b8eb196d", + "metadata": { + "editable": true + }, "source": [ "where $\\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\\lambda_i \\geq 0$.\n", "\n", @@ -800,7 +1046,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "664560f2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial {\\cal L}}{\\partial b} = -\\sum_{i} \\lambda_iy_i=0,\n", @@ -809,14 +1058,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a96c3b22", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "a209b96e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial {\\cal L}}{\\partial \\boldsymbol{w}} = 0 = \\boldsymbol{w}-\\sum_{i} \\lambda_iy_i\\boldsymbol{x}_i.\n", @@ -825,14 +1080,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e2dadc8e", + "metadata": { + "editable": true + }, "source": [ "Inserting these constraints into the equation for ${\\cal L}$ we obtain" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "f1da3fb6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{x}_j,\n", @@ -841,7 +1102,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a16adda0", + "metadata": { + "editable": true + }, "source": [ "subject to the constraints $\\lambda_i\\geq 0$ and $\\sum_i\\lambda_iy_i=0$. \n", "We must in addition satisfy the [Karush-Kuhn-Tucker](https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions) (KKT) condition" @@ -849,25 +1113,37 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "10c4420c", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "\\lambda_i\\left[y_i(\\boldsymbol{x}^T\\boldsymbol{w}_i+b) -1\\right] \\hspace{0.1cm}\\forall i.\n", + "\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) -1\\right] \\hspace{0.1cm}\\forall i.\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "72f7eaa5", + "metadata": { + "editable": true + }, "source": [ - "1. If $\\lambda_i > 0$, then $y_i(\\boldsymbol{x}^T\\boldsymbol{w}_i+b)=1$ and we say that $x_i$ is on the boundary.\n", + "1. If $\\lambda_i > 0$, then $y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1$ and we say that $x_i$ is on the boundary.\n", "\n", - "2. If $y_i(\\boldsymbol{x}^T\\boldsymbol{w}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\\lambda_i=0$. \n", - "\n", - "When $\\lambda_i > 0$, the vectors $\\boldsymbol{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. \n", - "\n", - "The support vectors (the points that define the margin $M$) are the quantities we keep in order to make predictions.\n", + "2. If $y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\\lambda_i=0$. \n", "\n", + "When $\\lambda_i > 0$, the vectors $\\boldsymbol{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$." + ] + }, + { + "cell_type": "markdown", + "id": "cf7449bd", + "metadata": { + "editable": true + }, + "source": [ "## The problem to solve\n", "\n", "We can rewrite" @@ -875,7 +1151,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "97f21dda", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{x}_j,\n", @@ -884,14 +1163,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a62e22f8", + "metadata": { + "editable": true + }, "source": [ "and its constraints in terms of a matrix-vector problem where we minimize w.r.t. $\\lambda$ the following problem" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "bcea5e1b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1\\boldsymbol{x}_1^T\\boldsymbol{x}_1 & y_1y_2\\boldsymbol{x}_1^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_1^T\\boldsymbol{x}_n \\\\\n", @@ -905,12 +1190,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f8fe188e", + "metadata": { + "editable": true + }, "source": [ "subject to $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$. Here we defined the vectors $\\boldsymbol{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n", - "$\\boldsymbol{y}=[y_1,y_2,\\dots,y_n]$. \n", - "\n", - "\n", + "$\\boldsymbol{y}=[y_1,y_2,\\dots,y_n]$." + ] + }, + { + "cell_type": "markdown", + "id": "413e240a", + "metadata": { + "editable": true + }, + "source": [ "## The last steps\n", "\n", "Solving the above problem, yields the values of $\\lambda_i$.\n", @@ -919,7 +1214,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "027fbb8f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{w}=\\sum_{i} \\lambda_iy_i\\boldsymbol{x}_i.\n", @@ -928,46 +1226,64 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7a5c313c", + "metadata": { + "editable": true + }, "source": [ "With our vector $\\boldsymbol{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "43f98f65", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "y_i(\\boldsymbol{x}^T\\boldsymbol{w}_i+b)=1,\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1,\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "3d0d0732", + "metadata": { + "editable": true + }, "source": [ "resulting in" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "7e7af264", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "b = \\frac{1}{y_i}-\\boldsymbol{x}_1^T\\boldsymbol{w},\n", + "b = \\frac{1}{y_i}-\\boldsymbol{w}^T\\boldsymbol{x}_i,\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "0f0966e4", + "metadata": { + "editable": true + }, "source": [ "or if we write it out in terms of the support vectors only, with $N_s$ being their number, we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "8c4d65a2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "b = \\frac{1}{N_s}\\sum_{j\\in N_s}\\left(y_j-\\sum_{i=1}^n\\lambda_iy_i\\boldsymbol{x}_i^T\\boldsymbol{x}_j\\right).\n", @@ -976,34 +1292,49 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "36956b4c", + "metadata": { + "editable": true + }, "source": [ "With our hyperplane coefficients we can use our classifier to assign any observation by simply using" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "34f2ad9e", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "y_i = \\mathrm{sign}(\\boldsymbol{x}_i^T\\boldsymbol{w}+b).\n", + "y_i = \\mathrm{sign}(\\boldsymbol{w}^T\\boldsymbol{x}_i+b).\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "f3b5969e", + "metadata": { + "editable": true + }, + "source": [ + "Below we discuss how to find the optimal values of $\\lambda_i$. Before we proceed however, we discuss now the so-called soft classifier." + ] + }, + { + "cell_type": "markdown", + "id": "2b535b11", + "metadata": { + "editable": true + }, "source": [ - "Below we discuss how to find the optimal values of $\\lambda_i$. Before we proceed however, we discuss now the so-called soft classifier. \n", - "\n", "## A soft classifier\n", "\n", "Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.\n", "\n", - "Suppose now that the two classes overlap in feature space, as shown in Figure 12.1 of\n", - "[Hastie et al](https://www.springer.com/gp/book/9780387848570).\n", - "\n", - "One way to deal with this problem before we define the\n", + "Suppose now that classes overlap in feature space, as shown in the\n", + "figure here. One way to deal with this problem before we define the\n", "so-called **kernel approach**, is to allow a kind of slack in the sense\n", "that we allow some points to be on the wrong side of the margin.\n", "\n", @@ -1013,50 +1344,72 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "82086b67", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "y_i(\\boldsymbol{x}_i^T\\boldsymbol{w}+b)=1,\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1,\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "43c2849b", + "metadata": { + "editable": true + }, "source": [ "to" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "79fa1055", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "y_i(\\boldsymbol{x}_1^T\\boldsymbol{w}+b)=1-\\xi_i,\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1-\\xi_i,\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "5a877772", + "metadata": { + "editable": true + }, "source": [ "with the requirement $\\xi_i\\geq 0$. The total violation is now $\\sum_i\\xi$. \n", "The value $\\xi_i$ in the constraint the last constraint corresponds to the amount by which the prediction\n", - "$y_i(\\boldsymbol{x}_i^T\\boldsymbol{w}+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\\sum_i \\xi_i$,\n", + "$y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\\sum_i \\xi_i$,\n", "we bound the total amount by which predictions fall on the wrong side of their margins.\n", "\n", "Misclassifications occur when $\\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of\n", - "misclassifications.\n", - "\n", + "misclassifications." + ] + }, + { + "cell_type": "markdown", + "id": "068ef297", + "metadata": { + "editable": true + }, + "source": [ "## Soft optmization problem\n", "\n", - "\n", "This has in turn the consequences that we change our optmization problem to finding the minimum of" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "0997965a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\cal L}=\\frac{1}{2}\\boldsymbol{w}^T\\boldsymbol{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)-(1-\\xi_)\\right]+C\\sum_{i=1}^n\\xi_i-\\sum_{i=1}^n\\gamma_i\\xi_i,\n", @@ -1065,23 +1418,32 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ae19263c", + "metadata": { + "editable": true + }, "source": [ "subject to" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "2730e8d2", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "y_i(\\boldsymbol{x}_1^T\\boldsymbol{w}+b)=1-\\xi_i \\hspace{0.1cm}\\forall i,\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1-\\xi_i \\hspace{0.1cm}\\forall i,\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "47573da0", + "metadata": { + "editable": true + }, "source": [ "with the requirement $\\xi_i\\geq 0$.\n", "\n", @@ -1090,7 +1452,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "03eb71f2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial {\\cal L}}{\\partial b} = -\\sum_{i} \\lambda_iy_i=0,\n", @@ -1099,14 +1464,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0b617712", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "c403b04e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial {\\cal L}}{\\partial \\boldsymbol{w}} = 0 = \\boldsymbol{w}-\\sum_{i} \\lambda_iy_i\\boldsymbol{x}_i,\n", @@ -1115,14 +1486,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b5480b2f", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "413d1c9e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\lambda_i = C-\\gamma_i \\hspace{0.1cm}\\forall i.\n", @@ -1131,14 +1508,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c27914e9", + "metadata": { + "editable": true + }, "source": [ "Inserting these constraints into the equation for ${\\cal L}$ we obtain the same equation as before" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "ef7189f9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{x}_j,\n", @@ -1147,7 +1530,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "765b9571", + "metadata": { + "editable": true + }, "source": [ "but now subject to the constraints $\\lambda_i\\geq 0$, $\\sum_i\\lambda_iy_i=0$ and $0\\leq\\lambda_i \\leq C$. \n", "We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads" @@ -1155,31 +1541,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "43cdb6a6", + "metadata": { + "editable": true + }, "source": [ - "5\n", - "0\n", - " \n", - "<\n", - "<\n", - "<\n", - "!\n", - "!\n", - "M\n", - "A\n", - "T\n", - "H\n", - "_\n", - "B\n", - "L\n", - "O\n", - "C\n", - "K" + "$$\n", + "\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) -(1-\\xi_)\\right]=0 \\hspace{0.1cm}\\forall i,\n", + "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "eccc3467", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\gamma_i\\xi_i = 0,\n", @@ -1188,22 +1565,2087 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2d5cec95", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "d0b092ba", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "y_i(\\boldsymbol{x}_i^T\\boldsymbol{w}+b) -(1-\\xi_) \\geq 0 \\hspace{0.1cm}\\forall i.\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) -(1-\\xi_) \\geq 0 \\hspace{0.1cm}\\forall i.\n", "$$" ] + }, + { + "cell_type": "markdown", + "id": "8f23fbe6", + "metadata": { + "editable": true + }, + "source": [ + "## Kernels and non-linearity\n", + "\n", + "The cases we have studied till now, were all characterized by two classes\n", + "with a close to linear separability. The classifiers we have described\n", + "so far find linear boundaries in our input feature space. It is\n", + "possible to make our procedure more flexible by exploring the feature\n", + "space using other basis expansions such as higher-order polynomials,\n", + "wavelets, splines etc.\n", + "\n", + "If our feature space is not easy to separate, as shown in the figure\n", + "here, we can achieve a better separation by introducing more complex\n", + "basis functions. The ideal would be, as shown in the next figure, to, via a specific transformation to \n", + "obtain a separation between the classes which is almost linear. \n", + "\n", + "The change of basis, from $x\\rightarrow z=\\phi(x)$ leads to the same type of equations to be solved, except that\n", + "we need to introduce for example a polynomial transformation to a two-dimensional training set." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "7837b7a7", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import os\n", + "\n", + "np.random.seed(42)\n", + "\n", + "# To plot pretty figures\n", + "import matplotlib\n", + "import matplotlib.pyplot as plt\n", + "plt.rcParams['axes.labelsize'] = 14\n", + "plt.rcParams['xtick.labelsize'] = 12\n", + "plt.rcParams['ytick.labelsize'] = 12\n", + "\n", + "\n", + "from sklearn.svm import SVC\n", + "from sklearn import datasets\n", + "\n", + "\n", + "\n", + "X1D = np.linspace(-4, 4, 9).reshape(-1, 1)\n", + "X2D = np.c_[X1D, X1D**2]\n", + "y = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])\n", + "\n", + "plt.figure(figsize=(11, 4))\n", + "\n", + "plt.subplot(121)\n", + "plt.grid(True, which='both')\n", + "plt.axhline(y=0, color='k')\n", + "plt.plot(X1D[:, 0][y==0], np.zeros(4), \"bs\")\n", + "plt.plot(X1D[:, 0][y==1], np.zeros(5), \"g^\")\n", + "plt.gca().get_yaxis().set_ticks([])\n", + "plt.xlabel(r\"$x_1$\", fontsize=20)\n", + "plt.axis([-4.5, 4.5, -0.2, 0.2])\n", + "\n", + "plt.subplot(122)\n", + "plt.grid(True, which='both')\n", + "plt.axhline(y=0, color='k')\n", + "plt.axvline(x=0, color='k')\n", + "plt.plot(X2D[:, 0][y==0], X2D[:, 1][y==0], \"bs\")\n", + "plt.plot(X2D[:, 0][y==1], X2D[:, 1][y==1], \"g^\")\n", + "plt.xlabel(r\"$x_1$\", fontsize=20)\n", + "plt.ylabel(r\"$x_2$\", fontsize=20, rotation=0)\n", + "plt.gca().get_yaxis().set_ticks([0, 4, 8, 12, 16])\n", + "plt.plot([-4.5, 4.5], [6.5, 6.5], \"r--\", linewidth=3)\n", + "plt.axis([-4.5, 4.5, -1, 17])\n", + "plt.subplots_adjust(right=1)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "6a67b779", + "metadata": { + "editable": true + }, + "source": [ + "## The equations\n", + "\n", + "Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with $x_i$ and $y_i$ as variables)" + ] + }, + { + "cell_type": "markdown", + "id": "606baee1", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z = \\phi(x_i) =\\left(x_i^2, y_i^2, \\sqrt{2}x_iy_i\\right).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "17baaf79", + "metadata": { + "editable": true + }, + "source": [ + "With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)" + ] + }, + { + "cell_type": "markdown", + "id": "9bd93358", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{z}_i^T\\boldsymbol{z}_j,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c7abd31e", + "metadata": { + "editable": true + }, + "source": [ + "subject to the constraints $\\lambda_i\\geq 0$, $\\sum_i\\lambda_iy_i=0$, and for the support vectors" + ] + }, + { + "cell_type": "markdown", + "id": "91603b28", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{z}_i+b)= 1 \\hspace{0.1cm}\\forall i,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1ace16e1", + "metadata": { + "editable": true + }, + "source": [ + "from which we also find $b$.\n", + "To compute $\\boldsymbol{z}_i^T\\boldsymbol{z}_j$ we define the kernel $K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)$ as" + ] + }, + { + "cell_type": "markdown", + "id": "2cec32a6", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=\\boldsymbol{z}_i^T\\boldsymbol{z}_j= \\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f7763bbe", + "metadata": { + "editable": true + }, + "source": [ + "For the above example, the kernel reads" + ] + }, + { + "cell_type": "markdown", + "id": "2daa66b9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=[x_i^2, y_i^2, \\sqrt{2}x_iy_i]^T\\begin{bmatrix} x_j^2 \\\\ y_j^2 \\\\ \\sqrt{2}x_jy_j \\end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e5237877", + "metadata": { + "editable": true + }, + "source": [ + "We note that this is nothing but the dot product of the two original\n", + "vectors $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. Instead of thus computing the\n", + "product in the Lagrangian of $\\boldsymbol{z}_i^T\\boldsymbol{z}_j$ we simply compute\n", + "the dot product $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$.\n", + "\n", + "This leads to the so-called\n", + "kernel trick and the result leads to the same as if we went through\n", + "the trouble of performing the transformation\n", + "$\\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j)$ during the SVM calculations." + ] + }, + { + "cell_type": "markdown", + "id": "15354451", + "metadata": { + "editable": true + }, + "source": [ + "## The problem to solve\n", + "Using our definition of the kernel We can rewrite again the Lagrangian" + ] + }, + { + "cell_type": "markdown", + "id": "93851a18", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{z}_j,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1ec72034", + "metadata": { + "editable": true + }, + "source": [ + "subject to the constraints $\\lambda_i\\geq 0$, $\\sum_i\\lambda_iy_i=0$ in terms of a convex optimization problem" + ] + }, + { + "cell_type": "markdown", + "id": "ca61eb83", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1K(\\boldsymbol{x}_1,\\boldsymbol{x}_1) & y_1y_2K(\\boldsymbol{x}_1,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_1,\\boldsymbol{x}_n) \\\\\n", + "y_2y_1K(\\boldsymbol{x}_2,\\boldsymbol{x}_1) & y_2y_2(\\boldsymbol{x}_2,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_2,\\boldsymbol{x}_n) \\\\\n", + "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", + "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", + "y_ny_1K(\\boldsymbol{x}_n,\\boldsymbol{x}_1) & y_ny_2K(\\boldsymbol{x}_n\\boldsymbol{x}_2) & \\dots & \\dots & y_ny_nK(\\boldsymbol{x}_n,\\boldsymbol{x}_n) \\\\\n", + "\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{1}\\boldsymbol{\\lambda},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ccfe3f37", + "metadata": { + "editable": true + }, + "source": [ + "subject to $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$. Here we defined the vectors $\\boldsymbol{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n", + "$\\boldsymbol{y}=[y_1,y_2,\\dots,y_n]$. \n", + "If we add the slack constants this leads to the additional constraint $0\\leq \\lambda_i \\leq C$.\n", + "\n", + "We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type" + ] + }, + { + "cell_type": "markdown", + "id": "002e0aea", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\begin{align*}\n", + " &\\mathrm{min}_{\\lambda}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{\\lambda}^T\\boldsymbol{P}\\boldsymbol{\\lambda}+\\boldsymbol{q}^T\\boldsymbol{\\lambda},\\\\ \\nonumber\n", + " &\\mathrm{subject\\hspace{0.1cm}to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\wedge \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "31d3979b", + "metadata": { + "editable": true + }, + "source": [ + "Below we discuss how to solve these equations. Here we note that the matrix $\\boldsymbol{P}$ has matrix elements $p_{ij}=y_iy_jK(\\boldsymbol{x}_i,\\boldsymbol{x}_j)$.\n", + "Given a kernel $K$ and the targets $y_i$ this matrix is easy to set up. The constraint $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$ leads to $f=0$ and $\\boldsymbol{A}=\\boldsymbol{y}$. How to set up the matrix $\\boldsymbol{G}$ is discussed later. Here note that the inequalities $0\\leq \\lambda_i \\leq C$ can be split up into\n", + "$0\\leq \\lambda_i$ and $\\lambda_i \\leq C$. These two inequalities define then the matrix $\\boldsymbol{G}$ and the vector $\\boldsymbol{h}$." + ] + }, + { + "cell_type": "markdown", + "id": "9cfedef9", + "metadata": { + "editable": true + }, + "source": [ + "## Different kernels and Mercer's theorem\n", + "\n", + "There are several popular kernels being used. These are\n", + "1. Linear: $K(\\boldsymbol{x},\\boldsymbol{y})=\\boldsymbol{x}^T\\boldsymbol{y}$,\n", + "\n", + "2. Polynomial: $K(\\boldsymbol{x},\\boldsymbol{y})=(\\boldsymbol{x}^T\\boldsymbol{y}+\\gamma)^d$,\n", + "\n", + "3. Gaussian Radial Basis Function: $K(\\boldsymbol{x},\\boldsymbol{y})=\\exp{\\left(-\\gamma\\vert\\vert\\boldsymbol{x}-\\boldsymbol{y}\\vert\\vert^2\\right)}$,\n", + "\n", + "4. Tanh: $K(\\boldsymbol{x},\\boldsymbol{y})=\\tanh{(\\boldsymbol{x}^T\\boldsymbol{y}+\\gamma)}$,\n", + "\n", + "and many other ones.\n", + "\n", + "An important theorem for us is [Mercer's\n", + "theorem](https://en.wikipedia.org/wiki/Mercer%27s_theorem). The\n", + "theorem states that if a kernel function $K$ is symmetric, continuous\n", + "and leads to a positive semi-definite matrix $\\boldsymbol{P}$ then there\n", + "exists a function $\\phi$ that maps $\\boldsymbol{x}_i$ and $\\boldsymbol{x}_j$ into\n", + "another space (possibly with much higher dimensions) such that" + ] + }, + { + "cell_type": "markdown", + "id": "9dc1f9b2", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=\\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "bdbdf563", + "metadata": { + "editable": true + }, + "source": [ + "So you can use $K$ as a kernel since you know $\\phi$ exists, even if\n", + "you don’t know what $\\phi$ is. \n", + "\n", + "Note that some frequently used kernels (such as the Sigmoid kernel)\n", + "don’t respect all of Mercer’s conditions, yet they generally work well\n", + "in practice." + ] + }, + { + "cell_type": "markdown", + "id": "650dc998", + "metadata": { + "editable": true + }, + "source": [ + "## The moons example" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2d562b0d", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from __future__ import division, print_function, unicode_literals\n", + "\n", + "import numpy as np\n", + "np.random.seed(42)\n", + "\n", + "import matplotlib\n", + "import matplotlib.pyplot as plt\n", + "plt.rcParams['axes.labelsize'] = 14\n", + "plt.rcParams['xtick.labelsize'] = 12\n", + "plt.rcParams['ytick.labelsize'] = 12\n", + "\n", + "\n", + "from sklearn.svm import SVC\n", + "from sklearn import datasets\n", + "\n", + "\n", + "\n", + "from sklearn.pipeline import Pipeline\n", + "from sklearn.preprocessing import StandardScaler\n", + "from sklearn.svm import LinearSVC\n", + "\n", + "\n", + "from sklearn.datasets import make_moons\n", + "X, y = make_moons(n_samples=100, noise=0.15, random_state=42)\n", + "\n", + "def plot_dataset(X, y, axes):\n", + " plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"bs\")\n", + " plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"g^\")\n", + " plt.axis(axes)\n", + " plt.grid(True, which='both')\n", + " plt.xlabel(r\"$x_1$\", fontsize=20)\n", + " plt.ylabel(r\"$x_2$\", fontsize=20, rotation=0)\n", + "\n", + "plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n", + "plt.show()\n", + "\n", + "from sklearn.datasets import make_moons\n", + "from sklearn.pipeline import Pipeline\n", + "from sklearn.preprocessing import PolynomialFeatures\n", + "\n", + "polynomial_svm_clf = Pipeline([\n", + " (\"poly_features\", PolynomialFeatures(degree=3)),\n", + " (\"scaler\", StandardScaler()),\n", + " (\"svm_clf\", LinearSVC(C=10, loss=\"hinge\", random_state=42))\n", + " ])\n", + "\n", + "polynomial_svm_clf.fit(X, y)\n", + "\n", + "def plot_predictions(clf, axes):\n", + " x0s = np.linspace(axes[0], axes[1], 100)\n", + " x1s = np.linspace(axes[2], axes[3], 100)\n", + " x0, x1 = np.meshgrid(x0s, x1s)\n", + " X = np.c_[x0.ravel(), x1.ravel()]\n", + " y_pred = clf.predict(X).reshape(x0.shape)\n", + " y_decision = clf.decision_function(X).reshape(x0.shape)\n", + " plt.contourf(x0, x1, y_pred, cmap=plt.cm.brg, alpha=0.2)\n", + " plt.contourf(x0, x1, y_decision, cmap=plt.cm.brg, alpha=0.1)\n", + "\n", + "plot_predictions(polynomial_svm_clf, [-1.5, 2.5, -1, 1.5])\n", + "plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n", + "\n", + "plt.show()\n", + "\n", + "\n", + "from sklearn.svm import SVC\n", + "\n", + "poly_kernel_svm_clf = Pipeline([\n", + " (\"scaler\", StandardScaler()),\n", + " (\"svm_clf\", SVC(kernel=\"poly\", degree=3, coef0=1, C=5))\n", + " ])\n", + "poly_kernel_svm_clf.fit(X, y)\n", + "\n", + "poly100_kernel_svm_clf = Pipeline([\n", + " (\"scaler\", StandardScaler()),\n", + " (\"svm_clf\", SVC(kernel=\"poly\", degree=10, coef0=100, C=5))\n", + " ])\n", + "poly100_kernel_svm_clf.fit(X, y)\n", + "\n", + "plt.figure(figsize=(11, 4))\n", + "\n", + "plt.subplot(121)\n", + "plot_predictions(poly_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])\n", + "plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n", + "plt.title(r\"$d=3, r=1, C=5$\", fontsize=18)\n", + "\n", + "plt.subplot(122)\n", + "plot_predictions(poly100_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])\n", + "plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n", + "plt.title(r\"$d=10, r=100, C=5$\", fontsize=18)\n", + "\n", + "plt.show()\n", + "\n", + "def gaussian_rbf(x, landmark, gamma):\n", + " return np.exp(-gamma * np.linalg.norm(x - landmark, axis=1)**2)\n", + "\n", + "gamma = 0.3\n", + "\n", + "x1s = np.linspace(-4.5, 4.5, 200).reshape(-1, 1)\n", + "x2s = gaussian_rbf(x1s, -2, gamma)\n", + "x3s = gaussian_rbf(x1s, 1, gamma)\n", + "\n", + "XK = np.c_[gaussian_rbf(X1D, -2, gamma), gaussian_rbf(X1D, 1, gamma)]\n", + "yk = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])\n", + "\n", + "plt.figure(figsize=(11, 4))\n", + "\n", + "plt.subplot(121)\n", + "plt.grid(True, which='both')\n", + "plt.axhline(y=0, color='k')\n", + "plt.scatter(x=[-2, 1], y=[0, 0], s=150, alpha=0.5, c=\"red\")\n", + "plt.plot(X1D[:, 0][yk==0], np.zeros(4), \"bs\")\n", + "plt.plot(X1D[:, 0][yk==1], np.zeros(5), \"g^\")\n", + "plt.plot(x1s, x2s, \"g--\")\n", + "plt.plot(x1s, x3s, \"b:\")\n", + "plt.gca().get_yaxis().set_ticks([0, 0.25, 0.5, 0.75, 1])\n", + "plt.xlabel(r\"$x_1$\", fontsize=20)\n", + "plt.ylabel(r\"Similarity\", fontsize=14)\n", + "plt.annotate(r'$\\mathbf{x}$',\n", + " xy=(X1D[3, 0], 0),\n", + " xytext=(-0.5, 0.20),\n", + " ha=\"center\",\n", + " arrowprops=dict(facecolor='black', shrink=0.1),\n", + " fontsize=18,\n", + " )\n", + "plt.text(-2, 0.9, \"$x_2$\", ha=\"center\", fontsize=20)\n", + "plt.text(1, 0.9, \"$x_3$\", ha=\"center\", fontsize=20)\n", + "plt.axis([-4.5, 4.5, -0.1, 1.1])\n", + "\n", + "plt.subplot(122)\n", + "plt.grid(True, which='both')\n", + "plt.axhline(y=0, color='k')\n", + "plt.axvline(x=0, color='k')\n", + "plt.plot(XK[:, 0][yk==0], XK[:, 1][yk==0], \"bs\")\n", + "plt.plot(XK[:, 0][yk==1], XK[:, 1][yk==1], \"g^\")\n", + "plt.xlabel(r\"$x_2$\", fontsize=20)\n", + "plt.ylabel(r\"$x_3$ \", fontsize=20, rotation=0)\n", + "plt.annotate(r'$\\phi\\left(\\mathbf{x}\\right)$',\n", + " xy=(XK[3, 0], XK[3, 1]),\n", + " xytext=(0.65, 0.50),\n", + " ha=\"center\",\n", + " arrowprops=dict(facecolor='black', shrink=0.1),\n", + " fontsize=18,\n", + " )\n", + "plt.plot([-0.1, 1.1], [0.57, -0.1], \"r--\", linewidth=3)\n", + "plt.axis([-0.1, 1.1, -0.1, 1.1])\n", + " \n", + "plt.subplots_adjust(right=1)\n", + "\n", + "plt.show()\n", + "\n", + "\n", + "x1_example = X1D[3, 0]\n", + "for landmark in (-2, 1):\n", + " k = gaussian_rbf(np.array([[x1_example]]), np.array([[landmark]]), gamma)\n", + " print(\"Phi({}, {}) = {}\".format(x1_example, landmark, k))\n", + "\n", + "rbf_kernel_svm_clf = Pipeline([\n", + " (\"scaler\", StandardScaler()),\n", + " (\"svm_clf\", SVC(kernel=\"rbf\", gamma=5, C=0.001))\n", + " ])\n", + "rbf_kernel_svm_clf.fit(X, y)\n", + "\n", + "\n", + "from sklearn.svm import SVC\n", + "\n", + "gamma1, gamma2 = 0.1, 5\n", + "C1, C2 = 0.001, 1000\n", + "hyperparams = (gamma1, C1), (gamma1, C2), (gamma2, C1), (gamma2, C2)\n", + "\n", + "svm_clfs = []\n", + "for gamma, C in hyperparams:\n", + " rbf_kernel_svm_clf = Pipeline([\n", + " (\"scaler\", StandardScaler()),\n", + " (\"svm_clf\", SVC(kernel=\"rbf\", gamma=gamma, C=C))\n", + " ])\n", + " rbf_kernel_svm_clf.fit(X, y)\n", + " svm_clfs.append(rbf_kernel_svm_clf)\n", + "\n", + "plt.figure(figsize=(11, 7))\n", + "\n", + "for i, svm_clf in enumerate(svm_clfs):\n", + " plt.subplot(221 + i)\n", + " plot_predictions(svm_clf, [-1.5, 2.5, -1, 1.5])\n", + " plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n", + " gamma, C = hyperparams[i]\n", + " plt.title(r\"$\\gamma = {}, C = {}$\".format(gamma, C), fontsize=16)\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "3106d683", + "metadata": { + "editable": true + }, + "source": [ + "## Mathematical optimization of convex functions\n", + "\n", + "A mathematical (quadratic) optimization problem, or just optimization problem, has the form" + ] + }, + { + "cell_type": "markdown", + "id": "a714dce8", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\begin{align*}\n", + " &\\mathrm{min}_{\\lambda}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{\\lambda}^T\\boldsymbol{P}\\boldsymbol{\\lambda}+\\boldsymbol{q}^T\\boldsymbol{\\lambda},\\\\ \\nonumber\n", + " &\\mathrm{subject\\hspace{0.1cm}to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\wedge \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "01cb9a05", + "metadata": { + "editable": true + }, + "source": [ + "subject to some constraints for say a selected set $i=1,2,\\dots, n$.\n", + "In our case we are optimizing with respect to the Lagrangian multipliers $\\lambda_i$, and the\n", + "vector $\\boldsymbol{\\lambda}=[\\lambda_1, \\lambda_2,\\dots, \\lambda_n]$ is the optimization variable we are dealing with.\n", + "\n", + "In our case we are particularly interested in a class of optimization problems called convex optmization problems. \n", + "In our discussion on gradient descent methods we discussed at length the definition of a convex function. \n", + "\n", + "Convex optimization problems play a central role in applied mathematics and we recommend strongly [Boyd and Vandenberghe's text on the topics](http://web.stanford.edu/~boyd/cvxbook/)." + ] + }, + { + "cell_type": "markdown", + "id": "a4526963", + "metadata": { + "editable": true + }, + "source": [ + "## How do we solve these problems?\n", + "\n", + "If we use Python as programming language and wish to venture beyond\n", + "**scikit-learn**, **tensorflow** and similar software which makes our\n", + "lives so much easier, we need to dive into the wonderful world of\n", + "quadratic programming. We can, if we wish, solve the minimization\n", + "problem using say standard gradient methods or conjugate gradient\n", + "methods. However, these methods tend to exhibit a rather slow\n", + "converge. So, welcome to the promised land of quadratic programming.\n", + "\n", + "The functions we need are contained in the quadratic programming package **CVXOPT** and we need to import it together with **numpy** as" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "681ebb1e", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import numpy\n", + "import cvxopt" + ] + }, + { + "cell_type": "markdown", + "id": "261d23c2", + "metadata": { + "editable": true + }, + "source": [ + "This will make our life much easier. You don't need t write your own optimizer." + ] + }, + { + "cell_type": "markdown", + "id": "c8baeba1", + "metadata": { + "editable": true + }, + "source": [ + "## A simple example\n", + "\n", + "We remind ourselves about the general problem we want to solve" + ] + }, + { + "cell_type": "markdown", + "id": "0b803837", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\begin{align*}\n", + " &\\mathrm{min}_{x}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{P}\\boldsymbol{x}+\\boldsymbol{q}^T\\boldsymbol{x},\\\\ \\nonumber\n", + " &\\mathrm{subject\\hspace{0.1cm} to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{x} \\preceq \\boldsymbol{h} \\wedge \\boldsymbol{A}\\boldsymbol{x}=f.\n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "9246d7f2", + "metadata": { + "editable": true + }, + "source": [ + "Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem" + ] + }, + { + "cell_type": "markdown", + "id": "290a112b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\begin{align*}\n", + " &\\mathrm{min}_{x}\\hspace{0.2cm} \\frac{1}{2}x^2+5x+3y \\\\ \\nonumber\n", + " &\\mathrm{subject to} \\\\ \\nonumber\n", + " &x, y \\geq 0 \\\\ \\nonumber\n", + " &x+3y \\geq 15 \\\\ \\nonumber\n", + " &2x+5y \\leq 100 \\\\ \\nonumber\n", + " &3x+4y \\leq 80. \\\\ \\nonumber\n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "9d9a7d1a", + "metadata": { + "editable": true + }, + "source": [ + "The minimization problem can be rewritten in terms of vectors and matrices as (with $x$ and $y$ being the unknowns)" + ] + }, + { + "cell_type": "markdown", + "id": "a860c2bc", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{1}{2}\\begin{bmatrix} x\\\\ y \\end{bmatrix}^T \\begin{bmatrix} 1 & 0\\\\ 0 & 0 \\end{bmatrix} \\begin{bmatrix} x \\\\ y \\end{bmatrix} + \\begin{bmatrix}3\\\\ 4 \\end{bmatrix}^T \\begin{bmatrix}x \\\\ y \\end{bmatrix}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "7c4ecc73", + "metadata": { + "editable": true + }, + "source": [ + "Similarly, we can now set up the inequalities (we need to change $\\geq$ to $\\leq$ by multiplying with $-1$ on bot sides) as the following matrix-vector equation" + ] + }, + { + "cell_type": "markdown", + "id": "d502bbef", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\begin{bmatrix} -1 & 0 \\\\ 0 & -1 \\\\ -1 & -3 \\\\ 2 & 5 \\\\ 3 & 4\\end{bmatrix}\\begin{bmatrix} x \\\\ y\\end{bmatrix} \\preceq \\begin{bmatrix}0 \\\\ 0\\\\ -15 \\\\ 100 \\\\ 80\\end{bmatrix}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "357f011c", + "metadata": { + "editable": true + }, + "source": [ + "We have collapsed all the inequalities into a single matrix $\\boldsymbol{G}$. We see also that our matrix" + ] + }, + { + "cell_type": "markdown", + "id": "427b5285", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{P} =\\begin{bmatrix} 1 & 0\\\\ 0 & 0 \\end{bmatrix}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "798b0a8e", + "metadata": { + "editable": true + }, + "source": [ + "is clearly positive semi-definite (all eigenvalues larger or equal zero). \n", + "Finally, the vector $\\boldsymbol{h}$ is defined as" + ] + }, + { + "cell_type": "markdown", + "id": "4dc10f2a", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{h} = \\begin{bmatrix}0 \\\\ 0\\\\ -15 \\\\ 100 \\\\ 80\\end{bmatrix}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "47872a2d", + "metadata": { + "editable": true + }, + "source": [ + "Since we don't have any equalities the matrix $\\boldsymbol{A}$ is set to zero\n", + "The following code solves the equations for us" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "544472ca", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# Import the necessary packages\n", + "import numpy\n", + "from cvxopt import matrix\n", + "from cvxopt import solvers\n", + "P = matrix(numpy.diag([1,0]), tc=’d’)\n", + "q = matrix(numpy.array([3,4]), tc=’d’)\n", + "G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’)\n", + "h = matrix(numpy.array([0,0,-15,100,80]), tc=’d’)\n", + "# Construct the QP, invoke solver\n", + "sol = solvers.qp(P,q,G,h)\n", + "# Extract optimal value and solution\n", + "sol[’x’] \n", + "sol[’primal objective’]" + ] + }, + { + "cell_type": "markdown", + "id": "27bcddcf", + "metadata": { + "editable": true + }, + "source": [ + "## Back to the more realistic cases\n", + "\n", + "We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the **slack** parameter $C$ we have" + ] + }, + { + "cell_type": "markdown", + "id": "3df03759", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1K(\\boldsymbol{x}_1,\\boldsymbol{x}_1) & y_1y_2K(\\boldsymbol{x}_1,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_1,\\boldsymbol{x}_n) \\\\\n", + "y_2y_1K(\\boldsymbol{x}_2,\\boldsymbol{x}_1) & y_2y_2K(\\boldsymbol{x}_2,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_2,\\boldsymbol{x}_n) \\\\\n", + "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", + "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", + "y_ny_1K(\\boldsymbol{x}_n,\\boldsymbol{x}_1) & y_ny_2K(\\boldsymbol{x}_n\\boldsymbol{x}_2) & \\dots & \\dots & y_ny_nK(\\boldsymbol{x}_n,\\boldsymbol{x}_n) \\\\\n", + "\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{I}\\boldsymbol{\\lambda},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "8bc769b8", + "metadata": { + "editable": true + }, + "source": [ + "subject to $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$. Here we defined the vectors $\\boldsymbol{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n", + "$\\boldsymbol{y}=[y_1,y_2,\\dots,y_n]$. \n", + "With the slack constants this leads to the additional constraint $0\\leq \\lambda_i \\leq C$.\n", + "\n", + "**code will be added**" + ] + }, + { + "cell_type": "markdown", + "id": "b08e57ec", + "metadata": { + "editable": true + }, + "source": [ + "## Summary of course" + ] + }, + { + "cell_type": "markdown", + "id": "1fbe6a9f", + "metadata": { + "editable": true + }, + "source": [ + "## What? Me worry? No final exam in this course!\n", + "\n", + "\n", + "\n", + "

    Figure 1:

    \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "357b665a", + "metadata": { + "editable": true + }, + "source": [ + "## Topics we have covered this year\n", + "\n", + "The course has two central parts\n", + "\n", + "1. Statistical analysis and optimization of data\n", + "\n", + "2. Machine learning" + ] + }, + { + "cell_type": "markdown", + "id": "036ae39a", + "metadata": { + "editable": true + }, + "source": [ + "## Statistical analysis and optimization of data\n", + "\n", + "The following topics have been discussed:\n", + "1. Basic concepts, expectation values, variance, covariance, correlation functions and errors;\n", + "\n", + "2. Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions;\n", + "\n", + "3. Central elements from linear algebra, matrix inversion and SVD\n", + "\n", + "4. Gradient methods for data optimization\n", + "\n", + "5. Estimation of errors using cross-validation, bootstrapping and jackknife methods;\n", + "\n", + "6. Practical optimization using Singular-value decomposition and least squares for parameterizing data.\n", + "\n", + "7. Principal Component Analysis to reduce the number of features." + ] + }, + { + "cell_type": "markdown", + "id": "df774b11", + "metadata": { + "editable": true + }, + "source": [ + "## Machine learning\n", + "\n", + "The following topics will be covered\n", + "1. Linear methods for regression and classification:\n", + "\n", + "a. Ordinary Least Squares\n", + "\n", + "b. Ridge regression\n", + "\n", + "c. Lasso regression\n", + "\n", + "d. Logistic regression\n", + "\n", + "5. Neural networks and deep learning:\n", + "\n", + "a. Feed Forward Neural Networks\n", + "\n", + "b. Convolutional Neural Networks\n", + "\n", + "c. Recurrent Neural Networks\n", + "\n", + "4. Decisions trees and ensemble methods:\n", + "\n", + "a. Decision trees\n", + "\n", + "b. Bagging and voting\n", + "\n", + "c. Random forests\n", + "\n", + "d. Boosting and gradient boosting\n", + "\n", + "5. Support vector machines\n", + "\n", + "a. Binary classification and multiclass classification\n", + "\n", + "b. Kernel methods\n", + "\n", + "c. Regression" + ] + }, + { + "cell_type": "markdown", + "id": "46e8bc30", + "metadata": { + "editable": true + }, + "source": [ + "## Learning outcomes and overarching aims of this course\n", + "\n", + "The course introduces a variety of central algorithms and methods\n", + "essential for studies of data analysis and machine learning. The\n", + "course is project based and through the various projects, normally\n", + "three, you will be exposed to fundamental research problems\n", + "in these fields, with the aim to reproduce state of the art scientific\n", + "results. The students will learn to develop and structure large codes\n", + "for studying these systems, get acquainted with computing facilities\n", + "and learn to handle large scientific projects. A good scientific and\n", + "ethical conduct is emphasized throughout the course. \n", + "\n", + "* Understand linear methods for regression and classification;\n", + "\n", + "* Learn about neural network;\n", + "\n", + "* Learn about bagging, boosting and trees\n", + "\n", + "* Support vector machines\n", + "\n", + "* Learn about basic data analysis;\n", + "\n", + "* Be capable of extending the acquired knowledge to other systems and cases;\n", + "\n", + "* Have an understanding of central algorithms used in data analysis and machine learning;\n", + "\n", + "* Work on numerical projects to illustrate the theory. The projects play a central role and you are expected to know modern programming languages like Python or C++." + ] + }, + { + "cell_type": "markdown", + "id": "5c906b4a", + "metadata": { + "editable": true + }, + "source": [ + "## Perspective on Machine Learning\n", + "\n", + "1. Rapidly emerging application area\n", + "\n", + "2. Experiment AND theory are evolving in many many fields. Still many low-hanging fruits.\n", + "\n", + "3. Requires education/retraining for more widespread adoption\n", + "\n", + "4. A lot of “word-of-mouth” development methods\n", + "\n", + "Huge amounts of data sets require automation, classical analysis tools often inadequate. \n", + "High energy physics hit this wall in the 90’s.\n", + "In 2009 single top quark production was determined via [Boosted decision trees, Bayesian\n", + "Neural Networks, etc.](https://arxiv.org/pdf/0903.0850.pdf)" + ] + }, + { + "cell_type": "markdown", + "id": "90eb24f6", + "metadata": { + "editable": true + }, + "source": [ + "## Machine Learning Research\n", + "\n", + "Where to find recent results:\n", + "1. Conference proceedings, arXiv and blog posts!\n", + "\n", + "2. **NIPS**: [Neural Information Processing Systems](https://papers.nips.cc)\n", + "\n", + "3. **ICLR**: [International Conference on Learning Representations](https://openreview.net/group?id=ICLR.cc/2018/Conference#accepted-oral-papers)\n", + "\n", + "4. **ICML**: International Conference on Machine Learning\n", + "\n", + "5. [Journal of Machine Learning Research](http://www.jmlr.org/papers/v19/) \n", + "\n", + "6. [Follow ML on ArXiv](https://arxiv.org/list/cs.LG/recent)" + ] + }, + { + "cell_type": "markdown", + "id": "2d06d0ec", + "metadata": { + "editable": true + }, + "source": [ + "## Starting your Machine Learning Project\n", + "\n", + "1. Identify problem type: classification, regression\n", + "\n", + "2. Consider your data carefully\n", + "\n", + "3. Choose a simple model that fits 1. and 2.\n", + "\n", + "4. Consider your data carefully again! Think of data representation more carefully.\n", + "\n", + "5. Based on your results, feedback loop to earliest possible point" + ] + }, + { + "cell_type": "markdown", + "id": "3328ed06", + "metadata": { + "editable": true + }, + "source": [ + "## Choose a Model and Algorithm\n", + "\n", + "1. Supervised?\n", + "\n", + "2. Start with the simplest model that fits your problem\n", + "\n", + "3. Start with minimal processing of data" + ] + }, + { + "cell_type": "markdown", + "id": "063cba9c", + "metadata": { + "editable": true + }, + "source": [ + "## Preparing Your Data\n", + "\n", + "1. Shuffle your data\n", + "\n", + "2. Mean center your data\n", + "\n", + " * Why?\n", + "\n", + "3. Normalize the variance\n", + "\n", + " * Why?\n", + "\n", + "4. **Whitening**\n", + "\n", + " * Decorrelates data\n", + "\n", + " * Can be hit or miss\n", + "\n", + "5. When to do train/test split?" + ] + }, + { + "cell_type": "markdown", + "id": "654a24c6", + "metadata": { + "editable": true + }, + "source": [ + "## Which Activation and Weights to Choose in Neural Networks\n", + "\n", + "1. RELU? ELU?\n", + "\n", + "2. Sigmoid or Tanh?\n", + "\n", + "3. Set all weights to 0?\n", + "\n", + " * Terrible idea\n", + "\n", + "4. Set all weights to random values?\n", + "\n", + " * Small random values" + ] + }, + { + "cell_type": "markdown", + "id": "0b522f71", + "metadata": { + "editable": true + }, + "source": [ + "## Optimization Methods and Hyperparameters\n", + "1. Stochastic gradient descent\n", + "\n", + "a. Stochastic gradient descent + momentum\n", + "\n", + "2. State-of-the-art approaches:\n", + "\n", + " * RMSProp\n", + "\n", + " * Adam\n", + "\n", + " * and more\n", + "\n", + "Which regularization and hyperparameters? $L_1$ or $L_2$, soft\n", + "classifiers, depths of trees and many other. Need to explore a large\n", + "set of hyperparameters and regularization methods." + ] + }, + { + "cell_type": "markdown", + "id": "a0696241", + "metadata": { + "editable": true + }, + "source": [ + "## Resampling\n", + "\n", + "When do we resample?\n", + "\n", + "1. [Bootstrap](https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A)\n", + "\n", + "2. [Cross-validation](https://www.youtube.com/watch?v=fSytzGwwBVw&ab_channel=StatQuestwithJoshStarmer)\n", + "\n", + "3. Jackknife and many other" + ] + }, + { + "cell_type": "markdown", + "id": "f0de0167", + "metadata": { + "editable": true + }, + "source": [ + "## Other courses on Data science and Machine Learning at UiO\n", + "\n", + "The link here gives an excellent overview of courses on Machine learning at UiO.\n", + "\n", + "1. [STK2100 Machine learning and statistical methods for prediction and classification](http://www.uio.no/studier/emner/matnat/math/STK2100/index-eng.html). \n", + "\n", + "2. [IN3050/IN4050 Introduction to Artificial Intelligence and Machine Learning](https://www.uio.no/studier/emner/matnat/ifi/IN3050/index-eng.html). Introductory course in machine learning and AI with an algorithmic approach. \n", + "\n", + "3. [STK-INF3000/4000 Selected Topics in Data Science](http://www.uio.no/studier/emner/matnat/math/STK-INF3000/index-eng.html). The course provides insight into selected contemporary relevant topics within Data Science. \n", + "\n", + "4. [IN4080 Natural Language Processing](https://www.uio.no/studier/emner/matnat/ifi/IN4080/index.html). Probabilistic and machine learning techniques applied to natural language processing. \n", + "\n", + "5. [STK-IN4300 – Statistical learning methods in Data Science](https://www.uio.no/studier/emner/matnat/math/STK-IN4300/index-eng.html). An advanced introduction to statistical and machine learning. For students with a good mathematics and statistics background.\n", + "\n", + "6. [IN-STK5000 Adaptive Methods for Data-Based Decision Making](https://www.uio.no/studier/emner/matnat/ifi/IN-STK5000/index-eng.html). Methods for adaptive collection and processing of data based on machine learning techniques. \n", + "\n", + "7. [IN5400/INF5860 – Machine Learning for Image Analysis](https://www.uio.no/studier/emner/matnat/ifi/IN5400/). An introduction to deep learning with particular emphasis on applications within Image analysis, but useful for other application areas too.\n", + "\n", + "8. [TEK5040 – Dyp læring for autonome systemer](https://www.uio.no/studier/emner/matnat/its/TEK5040/). The course addresses advanced algorithms and architectures for deep learning with neural networks. The course provides an introduction to how deep-learning techniques can be used in the construction of key parts of advanced autonomous systems that exist in physical environments and cyber environments." + ] + }, + { + "cell_type": "markdown", + "id": "b0896b77", + "metadata": { + "editable": true + }, + "source": [ + "## Additional courses of interest\n", + "\n", + "1. [STK4051 Computational Statistics](https://www.uio.no/studier/emner/matnat/math/STK4051/index-eng.html)\n", + "\n", + "2. [STK4021 Applied Bayesian Analysis and Numerical Methods](https://www.uio.no/studier/emner/matnat/math/STK4021/index-eng.html)" + ] + }, + { + "cell_type": "markdown", + "id": "95af654b", + "metadata": { + "editable": true + }, + "source": [ + "## What's the future like?\n", + "\n", + "Based on multi-layer nonlinear neural networks, deep learning can\n", + "learn directly from raw data, automatically extract and abstract\n", + "features from layer to layer, and then achieve the goal of regression,\n", + "classification, or ranking. Deep learning has made breakthroughs in\n", + "computer vision, speech processing and natural language, and reached\n", + "or even surpassed human level. The success of deep learning is mainly\n", + "due to the three factors: big data, big model, and big computing.\n", + "\n", + "In the past few decades, many different architectures of deep neural\n", + "networks have been proposed, such as\n", + "1. Convolutional neural networks, which are mostly used in image and video data processing, and have also been applied to sequential data such as text processing;\n", + "\n", + "2. Recurrent neural networks, which can process sequential data of variable length and have been widely used in natural language understanding and speech processing;\n", + "\n", + "3. Encoder-decoder framework, which is mostly used for image or sequence generation, such as machine translation, text summarization, and image captioning." + ] + }, + { + "cell_type": "markdown", + "id": "509ed8ed", + "metadata": { + "editable": true + }, + "source": [ + "## Types of Machine Learning, a repetition\n", + "\n", + "The approaches to machine learning are many, but are often split into two main categories. \n", + "In *supervised learning* we know the answer to a problem,\n", + "and let the computer deduce the logic behind it. On the other hand, *unsupervised learning*\n", + "is a method for finding patterns and relationship in data sets without any prior knowledge of the system.\n", + "Some authours also operate with a third category, namely *reinforcement learning*. This is a paradigm \n", + "of learning inspired by behavioural psychology, where learning is achieved by trial-and-error, \n", + "solely from rewards and punishment.\n", + "\n", + "Another way to categorize machine learning tasks is to consider the desired output of a system.\n", + "Some of the most common tasks are:\n", + "\n", + " * Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.\n", + "\n", + " * Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.\n", + "\n", + " * Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.\n", + "\n", + " * Other unsupervised learning algortihms like **Boltzmann machines**" + ] + }, + { + "cell_type": "markdown", + "id": "2abc1c1f", + "metadata": { + "editable": true + }, + "source": [ + "## Why Boltzmann machines?\n", + "\n", + "What is known as restricted Boltzmann Machines (RMB) have received a lot of attention lately. \n", + "One of the major reasons is that they can be stacked layer-wise to build deep neural networks that capture complicated statistics.\n", + "\n", + "The original RBMs had just one visible layer and a hidden layer, but recently so-called Gaussian-binary RBMs have gained quite some popularity in imaging since they are capable of modeling continuous data that are common to natural images. \n", + "\n", + "Furthermore, they have been used to solve complicated [quantum mechanical many-particle problems or classical statistical physics problems like the Ising and Potts classes of models](https://journals.aps.org/rmp/abstract/10.1103/RevModPhys.91.045002)." + ] + }, + { + "cell_type": "markdown", + "id": "dd46cd6a", + "metadata": { + "editable": true + }, + "source": [ + "## Boltzmann Machines\n", + "\n", + "Why use a generative model rather than the more well known discriminative deep neural networks (DNN)? \n", + "\n", + "* Discriminitave methods have several limitations: They are mainly supervised learning methods, thus requiring labeled data. And there are tasks they cannot accomplish, like drawing new examples from an unknown probability distribution.\n", + "\n", + "* A generative model can learn to represent and sample from a probability distribution. The core idea is to learn a parametric model of the probability distribution from which the training data was drawn. As an example\n", + "\n", + "a. A model for images could learn to draw new examples of cats and dogs, given a training dataset of images of cats and dogs.\n", + "\n", + "b. Generate a sample of an ordered or disordered phase, having been given samples of such phases.\n", + "\n", + "c. Model the trial function for [Monte Carlo calculations](https://journals.aps.org/rmp/abstract/10.1103/RevModPhys.91.045002)." + ] + }, + { + "cell_type": "markdown", + "id": "aa90be3c", + "metadata": { + "editable": true + }, + "source": [ + "## Some similarities and differences from DNNs\n", + "\n", + "1. Both use gradient-descent based learning procedures for minimizing cost functions\n", + "\n", + "2. Energy based models don't use backpropagation and automatic differentiation for computing gradients, instead turning to Markov Chain Monte Carlo methods.\n", + "\n", + "3. DNNs often have several hidden layers. A restricted Boltzmann machine has only one hidden layer, however several RBMs can be stacked to make up Deep Belief Networks, of which they constitute the building blocks.\n", + "\n", + "History: The RBM was developed by amongst others [Geoffrey Hinton](https://en.wikipedia.org/wiki/Geoffrey_Hinton), called by some the \"Godfather of Deep Learning\", working with the University of Toronto and Google." + ] + }, + { + "cell_type": "markdown", + "id": "9baeb6fc", + "metadata": { + "editable": true + }, + "source": [ + "## Boltzmann machines (BM)\n", + "\n", + "A BM is what we would call an undirected probabilistic graphical model\n", + "with stochastic continuous or discrete units.\n", + "\n", + "It is interpreted as a stochastic recurrent neural network where the\n", + "state of each unit(neurons/nodes) depends on the units it is connected\n", + "to. The weights in the network represent thus the strength of the\n", + "interaction between various units/nodes.\n", + "\n", + "It turns into a Hopfield network if we choose deterministic rather\n", + "than stochastic units. In contrast to a Hopfield network, a BM is a\n", + "so-called generative model. It allows us to generate new samples from\n", + "the learned distribution." + ] + }, + { + "cell_type": "markdown", + "id": "6e922377", + "metadata": { + "editable": true + }, + "source": [ + "## A standard BM setup\n", + "\n", + "A standard BM network is divided into a set of observable and visible units $\\hat{x}$ and a set of unknown hidden units/nodes $\\hat{h}$.\n", + "\n", + "Additionally there can be bias nodes for the hidden and visible layers. These biases are normally set to $1$.\n", + "\n", + "BMs are stackable, meaning they cwe can train a BM which serves as input to another BM. We can construct deep networks for learning complex PDFs. The layers can be trained one after another, a feature which makes them popular in deep learning\n", + "\n", + "However, they are often hard to train. This leads to the introduction of so-called restricted BMs, or RBMS.\n", + "Here we take away all lateral connections between nodes in the visible layer as well as connections between nodes in the hidden layer. The network is illustrated in the figure below." + ] + }, + { + "cell_type": "markdown", + "id": "9467df93", + "metadata": { + "editable": true + }, + "source": [ + "## The structure of the RBM network\n", + "\n", + "\n", + "\n", + "\n", + "

    Figure 1:

    \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "08d89ce1", + "metadata": { + "editable": true + }, + "source": [ + "## The network\n", + "\n", + "**The network layers**:\n", + "1. A function $\\mathbf{x}$ that represents the visible layer, a vector of $M$ elements (nodes). This layer represents both what the RBM might be given as training input, and what we want it to be able to reconstruct. This might for example be given by the pixels of an image or coefficients representing speech, or the coordinates of a quantum mechanical state function.\n", + "\n", + "2. The function $\\mathbf{h}$ represents the hidden, or latent, layer. A vector of $N$ elements (nodes). Also called \"feature detectors\"." + ] + }, + { + "cell_type": "markdown", + "id": "7d7d4fed", + "metadata": { + "editable": true + }, + "source": [ + "## Goals\n", + "\n", + "The goal of the hidden layer is to increase the model's expressive\n", + "power. We encode complex interactions between visible variables by\n", + "introducing additional, hidden variables that interact with visible\n", + "degrees of freedom in a simple manner, yet still reproduce the complex\n", + "correlations between visible degrees in the data once marginalized\n", + "over (integrated out).\n", + "\n", + "**The network parameters, to be optimized/learned**:\n", + "1. $\\mathbf{a}$ represents the visible bias, a vector of same length as $\\mathbf{x}$.\n", + "\n", + "2. $\\mathbf{b}$ represents the hidden bias, a vector of same lenght as $\\mathbf{h}$.\n", + "\n", + "3. $W$ represents the interaction weights, a matrix of size $M\\times N$." + ] + }, + { + "cell_type": "markdown", + "id": "7429c840", + "metadata": { + "editable": true + }, + "source": [ + "## Joint distribution\n", + "\n", + "The restricted Boltzmann machine is described by a Boltzmann distribution" + ] + }, + { + "cell_type": "markdown", + "id": "f5debabb", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "
    \n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\tP_{rbm}(\\mathbf{x},\\mathbf{h}) = \\frac{1}{Z} e^{-\\frac{1}{T_0}E(\\mathbf{x},\\mathbf{h})},\n", + "\\label{_auto1} \\tag{1}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "6e43b075", + "metadata": { + "editable": true + }, + "source": [ + "where $Z$ is the normalization constant or partition function, defined as" + ] + }, + { + "cell_type": "markdown", + "id": "a9d075f0", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "
    \n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\tZ = \\int \\int e^{-\\frac{1}{T_0}E(\\mathbf{x},\\mathbf{h})} d\\mathbf{x} d\\mathbf{h}.\n", + "\\label{_auto2} \\tag{2}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "bd793fc2", + "metadata": { + "editable": true + }, + "source": [ + "It is common to ignore $T_0$ by setting it to one." + ] + }, + { + "cell_type": "markdown", + "id": "a8976f06", + "metadata": { + "editable": true + }, + "source": [ + "## Network Elements, the energy function\n", + "\n", + "The function $E(\\mathbf{x},\\mathbf{h})$ gives the **energy** of a\n", + "configuration (pair of vectors) $(\\mathbf{x}, \\mathbf{h})$. The lower\n", + "the energy of a configuration, the higher the probability of it. This\n", + "function also depends on the parameters $\\mathbf{a}$, $\\mathbf{b}$ and\n", + "$W$. Thus, when we adjust them during the learning procedure, we are\n", + "adjusting the energy function to best fit our problem.\n", + "\n", + "An expression for the energy function is" + ] + }, + { + "cell_type": "markdown", + "id": "1c83ae78", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "E(\\hat{x},\\hat{h}) = -\\sum_{ia}^{NA}b_i^a \\alpha_i^a(x_i)-\\sum_{jd}^{MD}c_j^d \\beta_j^d(h_j)-\\sum_{ijad}^{NAMD}b_i^a \\alpha_i^a(x_i)c_j^d \\beta_j^d(h_j)w_{ij}^{ad}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "12dff191", + "metadata": { + "editable": true + }, + "source": [ + "Here $\\beta_j^d(h_j)$ and $\\alpha_i^a(x_j)$ are so-called transfer functions that map a given input value to a desired feature value. The labels $a$ and $d$ denote that there can be multiple transfer functions per variable. The first sum depends only on the visible units. The second on the hidden ones. **Note** that there is no connection between nodes in a layer.\n", + "\n", + "The quantities $b$ and $c$ can be interpreted as the visible and hidden biases, respectively.\n", + "\n", + "The connection between the nodes in the two layers is given by the weights $w_{ij}$." + ] + }, + { + "cell_type": "markdown", + "id": "78bac9f8", + "metadata": { + "editable": true + }, + "source": [ + "## Defining different types of RBMs\n", + "There are different variants of RBMs, and the differences lie in the types of visible and hidden units we choose as well as in the implementation of the energy function $E(\\mathbf{x},\\mathbf{h})$. \n", + "\n", + "**Binary-Binary RBM:**\n", + "\n", + "RBMs were first developed using binary units in both the visible and hidden layer. The corresponding energy function is defined as follows:" + ] + }, + { + "cell_type": "markdown", + "id": "dffac8d3", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "
    \n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\tE(\\mathbf{x}, \\mathbf{h}) = - \\sum_i^M x_i a_i- \\sum_j^N b_j h_j - \\sum_{i,j}^{M,N} x_i w_{ij} h_j,\n", + "\\label{_auto3} \\tag{3}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e170b4b2", + "metadata": { + "editable": true + }, + "source": [ + "where the binary values taken on by the nodes are most commonly 0 and 1.\n", + "\n", + "**Gaussian-Binary RBM:**\n", + "\n", + "Another varient is the RBM where the visible units are Gaussian while the hidden units remain binary:" + ] + }, + { + "cell_type": "markdown", + "id": "fdc03484", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "
    \n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\tE(\\mathbf{x}, \\mathbf{h}) = \\sum_i^M \\frac{(x_i - a_i)^2}{2\\sigma_i^2} - \\sum_j^N b_j h_j - \\sum_{i,j}^{M,N} \\frac{x_i w_{ij} h_j}{\\sigma_i^2}. \n", + "\\label{_auto4} \\tag{4}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "6079489b", + "metadata": { + "editable": true + }, + "source": [ + "## More about RBMs\n", + "1. Useful when we model continuous data (i.e., we wish $\\mathbf{x}$ to be continuous)\n", + "\n", + "2. Requires a smaller learning rate, since there's no upper bound to the value a component might take in the reconstruction\n", + "\n", + "Other types of units include:\n", + "1. Softmax and multinomial units\n", + "\n", + "2. Gaussian visible and hidden units\n", + "\n", + "3. Binomial units\n", + "\n", + "4. Rectified linear units\n", + "\n", + "To read more, see [Lectures on Boltzmann machines in Physics](https://github.com/CompPhysics/ComputationalPhysics2/blob/gh-pages/doc/pub/notebook2/ipynb/notebook2.ipynb)." + ] + }, + { + "cell_type": "markdown", + "id": "f167d850", + "metadata": { + "editable": true + }, + "source": [ + "## Autoencoders: Overarching view\n", + "\n", + "Autoencoders are artificial neural networks capable of learning\n", + "efficient representations of the input data (these representations are called codings) without\n", + "any supervision (i.e., the training set is unlabeled). These codings\n", + "typically have a much lower dimensionality than the input data, making\n", + "autoencoders useful for dimensionality reduction. \n", + "\n", + "More importantly, autoencoders act as powerful feature detectors, and\n", + "they can be used for unsupervised pretraining of deep neural networks.\n", + "\n", + "Lastly, they are capable of randomly generating new data that looks\n", + "very similar to the training data; this is called a generative\n", + "model. For example, you could train an autoencoder on pictures of\n", + "faces, and it would then be able to generate new faces. Surprisingly,\n", + "autoencoders work by simply learning to copy their inputs to their\n", + "outputs. This may sound like a trivial task, but we will see that\n", + "constraining the network in various ways can make it rather\n", + "difficult. For example, you can limit the size of the internal\n", + "representation, or you can add noise to the inputs and train the\n", + "network to recover the original inputs. These constraints prevent the\n", + "autoencoder from trivially copying the inputs directly to the outputs,\n", + "which forces it to learn efficient ways of representing the data. In\n", + "short, the codings are byproducts of the autoencoder’s attempt to\n", + "learn the identity function under some constraints.\n", + "\n", + "[Video on autoencoders](https://www.coursera.org/lecture/building-deep-learning-models-with-tensorflow/autoencoders-1U4L3)\n", + "\n", + "See also A. Geron's textbook, chapter 15." + ] + }, + { + "cell_type": "markdown", + "id": "92f837ab", + "metadata": { + "editable": true + }, + "source": [ + "## Bayesian Machine Learning\n", + "\n", + "This is an important topic if we aim at extracting a probability\n", + "distribution. This gives us also a confidence interval and error\n", + "estimates.\n", + "\n", + "Bayesian machine learning allows us to encode our prior beliefs about\n", + "what those models should look like, independent of what the data tells\n", + "us. This is especially useful when we don’t have a ton of data to\n", + "confidently learn our model.\n", + "\n", + "[Video on Bayesian deep learning](https://www.youtube.com/watch?v=E1qhGw8QxqY&ab_channel=AndrewGordonWilson)\n", + "\n", + "See also the [slides here](https://github.com/CompPhysics/MachineLearning/blob/master/doc/Articles/lec03.pdf)." + ] + }, + { + "cell_type": "markdown", + "id": "35de7d79", + "metadata": { + "editable": true + }, + "source": [ + "## Reinforcement Learning\n", + "\n", + "Reinforcement Learning (RL) is one of the most exciting fields of\n", + "Machine Learning today, and also one of the oldest. It has been around\n", + "since the 1950s, producing many interesting applications over the\n", + "years.\n", + "\n", + "It studies\n", + "how agents take actions based on trial and error, so as to maximize\n", + "some notion of cumulative reward in a dynamic system or\n", + "environment. Due to its generality, the problem has also been studied\n", + "in many other disciplines, such as game theory, control theory,\n", + "operations research, information theory, multi-agent systems, swarm\n", + "intelligence, statistics, and genetic algorithms.\n", + "\n", + "In March 2016, AlphaGo, a computer program that plays the board game\n", + "Go, beat Lee Sedol in a five-game match. This was the first time a\n", + "computer Go program had beaten a 9-dan (highest rank) professional\n", + "without handicaps. AlphaGo is based on deep convolutional neural\n", + "networks and reinforcement learning. AlphaGo’s victory was a major\n", + "milestone in artificial intelligence and it has also made\n", + "reinforcement learning a hot research area in the field of machine\n", + "learning.\n", + "\n", + "[Lecture on Reinforcement Learning](https://www.youtube.com/watch?v=FgzM3zpZ55o&ab_channel=stanfordonline).\n", + "\n", + "See also A. Geron's textbook, chapter 16." + ] + }, + { + "cell_type": "markdown", + "id": "2341b6e9", + "metadata": { + "editable": true + }, + "source": [ + "## Transfer learning\n", + "\n", + "The goal of transfer learning is to transfer the model or knowledge\n", + "obtained from a source task to the target task, in order to resolve\n", + "the issues of insufficient training data in the target task. The\n", + "rationality of doing so lies in that usually the source and target\n", + "tasks have inter-correlations, and therefore either the features,\n", + "samples, or models in the source task might provide useful information\n", + "for us to better solve the target task. Transfer learning is a hot\n", + "research topic in recent years, with many problems still waiting to be studied.\n", + "\n", + "[Lecture on transfer learning](https://www.ias.edu/video/machinelearning/2020/0331-SamoryKpotufe)." + ] + }, + { + "cell_type": "markdown", + "id": "074e8484", + "metadata": { + "editable": true + }, + "source": [ + "## Adversarial learning\n", + "\n", + "The conventional deep generative model has a potential problem: the\n", + "model tends to generate extreme instances to maximize the\n", + "probabilistic likelihood, which will hurt its performance. Adversarial\n", + "learning utilizes the adversarial behaviors (e.g., generating\n", + "adversarial instances or training an adversarial model) to enhance the\n", + "robustness of the model and improve the quality of the generated\n", + "data. In recent years, one of the most promising unsupervised learning\n", + "technologies, generative adversarial networks (GAN), has already been\n", + "successfully applied to image, speech, and text.\n", + "\n", + "[Lecture on adversial learning](https://www.youtube.com/watch?v=CIfsB_EYsVI&ab_channel=StanfordUniversitySchoolofEngineering)." + ] + }, + { + "cell_type": "markdown", + "id": "c17c3b86", + "metadata": { + "editable": true + }, + "source": [ + "## Dual learning\n", + "\n", + "Dual learning is a new learning paradigm, the basic idea of which is\n", + "to use the primal-dual structure between machine learning tasks to\n", + "obtain effective feedback/regularization, and guide and strengthen the\n", + "learning process, thus reducing the requirement of large-scale labeled\n", + "data for deep learning. The idea of dual learning has been applied to\n", + "many problems in machine learning, including machine translation,\n", + "image style conversion, question answering and generation, image\n", + "classification and generation, text classification and generation,\n", + "image-to-text, and text-to-image." + ] + }, + { + "cell_type": "markdown", + "id": "49a1a37d", + "metadata": { + "editable": true + }, + "source": [ + "## Distributed machine learning\n", + "\n", + "Distributed computation will speed up machine learning algorithms,\n", + "significantly improve their efficiency, and thus enlarge their\n", + "application. When distributed meets machine learning, more than just\n", + "implementing the machine learning algorithms in parallel is required." + ] + }, + { + "cell_type": "markdown", + "id": "a185bd43", + "metadata": { + "editable": true + }, + "source": [ + "## Meta learning\n", + "\n", + "Meta learning is an emerging research direction in machine\n", + "learning. Roughly speaking, meta learning concerns learning how to\n", + "learn, and focuses on the understanding and adaptation of the learning\n", + "itself, instead of just completing a specific learning task. That is,\n", + "a meta learner needs to be able to evaluate its own learning methods\n", + "and adjust its own learning methods according to specific learning\n", + "tasks." + ] + }, + { + "cell_type": "markdown", + "id": "53273d78", + "metadata": { + "editable": true + }, + "source": [ + "## The Challenges Facing Machine Learning\n", + "\n", + "While there has been much progress in machine learning, there are also challenges.\n", + "\n", + "For example, the mainstream machine learning technologies are\n", + "black-box approaches, making us concerned about their potential\n", + "risks. To tackle this challenge, we may want to make machine learning\n", + "more explainable and controllable. As another example, the\n", + "computational complexity of machine learning algorithms is usually\n", + "very high and we may want to invent lightweight algorithms or\n", + "implementations. Furthermore, in many domains such as physics,\n", + "chemistry, biology, and social sciences, people usually seek elegantly\n", + "simple equations (e.g., the Schrödinger equation) to uncover the\n", + "underlying laws behind various phenomena. In the field of machine\n", + "learning, can we reveal simple laws instead of designing more complex\n", + "models for data fitting? Although there are many challenges, we are\n", + "still very optimistic about the future of machine learning. As we look\n", + "forward to the future, here are what we think the research hotspots in\n", + "the next ten years will be.\n", + "\n", + "See the article on [Discovery of Physics From Data: Universal Laws and Discrepancies](https://www.frontiersin.org/articles/10.3389/frai.2020.00025/full)" + ] + }, + { + "cell_type": "markdown", + "id": "db13f810", + "metadata": { + "editable": true + }, + "source": [ + "## Explainable machine learning\n", + "\n", + "Machine learning, especially deep learning, evolves rapidly. The\n", + "ability gap between machine and human on many complex cognitive tasks\n", + "becomes narrower and narrower. However, we are still in the very early\n", + "stage in terms of explaining why those effective models work and how\n", + "they work.\n", + "\n", + "**What is missing: the gap between correlation and causation**. Standard Machine Learning is based on what e have called a frequentist approach. \n", + "\n", + "Most\n", + "machine learning techniques, especially the statistical ones, depend\n", + "highly on correlations in data sets to make predictions and analyses. In\n", + "contrast, rational humans tend to reply on clear and trustworthy\n", + "causality relations obtained via logical reasoning on real and clear\n", + "facts. It is one of the core goals of explainable machine learning to\n", + "transition from solving problems by data correlation to solving\n", + "problems by logical reasoning.\n", + "\n", + "**Bayesian Machine Learning is one of the exciting research directions in this field**." + ] + }, + { + "cell_type": "markdown", + "id": "753fa5bb", + "metadata": { + "editable": true + }, + "source": [ + "## Quantum machine learning\n", + "\n", + "Quantum machine learning is an emerging interdisciplinary research\n", + "area at the intersection of quantum computing and machine learning.\n", + "\n", + "Quantum computers use effects such as quantum coherence and quantum\n", + "entanglement to process information, which is fundamentally different\n", + "from classical computers. Quantum algorithms have surpassed the best\n", + "classical algorithms in several problems (e.g., searching for an\n", + "unsorted database, inverting a sparse matrix), which we call quantum\n", + "acceleration.\n", + "\n", + "When quantum computing meets machine learning, it can be a mutually\n", + "beneficial and reinforcing process, as it allows us to take advantage\n", + "of quantum computing to improve the performance of classical machine\n", + "learning algorithms. In addition, we can also use the machine learning\n", + "algorithms (on classic computers) to analyze and improve quantum\n", + "computing systems.\n", + "\n", + "[Lecture on Quantum ML](https://www.youtube.com/watch?v=Xh9pUu3-WxM&ab_channel=InstituteforPure%26AppliedMathematics%28IPAM%29).\n", + "\n", + "[Read interview with Maria Schuld on her work on Quantum Machine Learning](https://physics.aps.org/articles/v13/179?utm_campaign=weekly&utm_medium=email&utm_source=emailalert). See also [her recent textbook](https://www.springer.com/gp/book/9783319964232)." + ] + }, + { + "cell_type": "markdown", + "id": "bebca9bb", + "metadata": { + "editable": true + }, + "source": [ + "## Quantum machine learning algorithms based on linear algebra\n", + "\n", + "Many quantum machine learning algorithms are based on variants of\n", + "quantum algorithms for solving linear equations, which can efficiently\n", + "solve N-variable linear equations with complexity of O(log2 N) under\n", + "certain conditions. The quantum matrix inversion algorithm can\n", + "accelerate many machine learning methods, such as least square linear\n", + "regression, least square version of support vector machine, Gaussian\n", + "process, and more. The training of these algorithms can be simplified\n", + "to solve linear equations. The key bottleneck of this type of quantum\n", + "machine learning algorithms is data input—that is, how to initialize\n", + "the quantum system with the entire data set. Although efficient\n", + "data-input algorithms exist for certain situations, how to efficiently\n", + "input data into a quantum system is as yet unknown for most cases." + ] + }, + { + "cell_type": "markdown", + "id": "2e40473f", + "metadata": { + "editable": true + }, + "source": [ + "## Quantum reinforcement learning\n", + "\n", + "In quantum reinforcement learning, a quantum agent interacts with the\n", + "classical environment to obtain rewards from the environment, so as to\n", + "adjust and improve its behavioral strategies. In some cases, it\n", + "achieves quantum acceleration by the quantum processing capabilities\n", + "of the agent or the possibility of exploring the environment through\n", + "quantum superposition. Such algorithms have been proposed in\n", + "superconducting circuits and systems of trapped ions." + ] + }, + { + "cell_type": "markdown", + "id": "5b9ab189", + "metadata": { + "editable": true + }, + "source": [ + "## Quantum deep learning\n", + "\n", + "Dedicated quantum information processors, such as quantum annealers\n", + "and programmable photonic circuits, are well suited for building deep\n", + "quantum networks. The simplest deep quantum network is the Boltzmann\n", + "machine. The classical Boltzmann machine consists of bits with tunable\n", + "interactions and is trained by adjusting the interaction of these bits\n", + "so that the distribution of its expression conforms to the statistics\n", + "of the data. To quantize the Boltzmann machine, the neural network can\n", + "simply be represented as a set of interacting quantum spins that\n", + "correspond to an adjustable Ising model. Then, by initializing the\n", + "input neurons in the Boltzmann machine to a fixed state and allowing\n", + "the system to heat up, we can read out the output qubits to get the\n", + "result." + ] + }, + { + "cell_type": "markdown", + "id": "c152d5d8", + "metadata": { + "editable": true + }, + "source": [ + "## Social machine learning\n", + "\n", + "Machine learning aims to imitate how humans\n", + "learn. While we have developed successful machine learning algorithms,\n", + "until now we have ignored one important fact: humans are social. Each\n", + "of us is one part of the total society and it is difficult for us to\n", + "live, learn, and improve ourselves, alone and isolated. Therefore, we\n", + "should design machines with social properties. Can we let machines\n", + "evolve by imitating human society so as to achieve more effective,\n", + "intelligent, interpretable “social machine learning”?\n", + "\n", + "And much more." + ] + }, + { + "cell_type": "markdown", + "id": "e00bb23f", + "metadata": { + "editable": true + }, + "source": [ + "## The last words?\n", + "\n", + "Early computer scientist Alan Kay said, **The best way to predict the\n", + "future is to create it**. Therefore, all machine learning\n", + "practitioners, whether scholars or engineers, professors or students,\n", + "need to work together to advance these important research\n", + "topics. Together, we will not just predict the future, but create it." + ] + }, + { + "cell_type": "markdown", + "id": "c6bb101b", + "metadata": { + "editable": true + }, + "source": [ + "## Best wishes to you all and thanks so much for your heroic efforts this semester\n", + "\n", + "\n", + "\n", + "\n", + "

    Figure 1:

    \n", + "" + ] } ], "metadata": {}, "nbformat": 4, - "nbformat_minor": 4 + "nbformat_minor": 5 } diff --git a/doc/src/week47/figures/Nebbdyr2.pdf b/doc/src/week47/figures/Nebbdyr2.pdf new file mode 100755 index 000000000..b830726fb Binary files /dev/null and b/doc/src/week47/figures/Nebbdyr2.pdf differ diff --git a/doc/src/week47/figures/Nebbdyr2.png b/doc/src/week47/figures/Nebbdyr2.png new file mode 100644 index 000000000..a21b0cb99 Binary files /dev/null and b/doc/src/week47/figures/Nebbdyr2.png differ diff --git a/doc/src/week47/figures/RBM.pdf b/doc/src/week47/figures/RBM.pdf new file mode 100644 index 000000000..50bb768e1 --- /dev/null +++ b/doc/src/week47/figures/RBM.pdf @@ -0,0 +1,1623 @@ +%PDF-1.5 % +1 0 obj <>/OCGs[22 0 R]>>/Type/Catalog>> endobj 47 0 obj <>stream + + + + + application/pdf + + + RBM + + + + + Adobe Illustrator CS3 + 2018-02-16T10:55:58-05:00 + 2018-02-16T10:55:58-05:00 + 2018-02-16T10:55:58-05:00 + + + + 256 + 76 + JPEG + /9j/4AAQSkZJRgABAgEASABIAAD/7QAsUGhvdG9zaG9wIDMuMAA4QklNA+0AAAAAABAASAAAAAEA AQBIAAAAAQAB/+4ADkFkb2JlAGTAAAAAAf/bAIQABgQEBAUEBgUFBgkGBQYJCwgGBggLDAoKCwoK DBAMDAwMDAwQDA4PEA8ODBMTFBQTExwbGxscHx8fHx8fHx8fHwEHBwcNDA0YEBAYGhURFRofHx8f Hx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f/8AAEQgATAEAAwER AAIRAQMRAf/EAaIAAAAHAQEBAQEAAAAAAAAAAAQFAwIGAQAHCAkKCwEAAgIDAQEBAQEAAAAAAAAA AQACAwQFBgcICQoLEAACAQMDAgQCBgcDBAIGAnMBAgMRBAAFIRIxQVEGE2EicYEUMpGhBxWxQiPB UtHhMxZi8CRygvElQzRTkqKyY3PCNUQnk6OzNhdUZHTD0uIIJoMJChgZhJRFRqS0VtNVKBry4/PE 1OT0ZXWFlaW1xdXl9WZ2hpamtsbW5vY3R1dnd4eXp7fH1+f3OEhYaHiImKi4yNjo+Ck5SVlpeYmZ qbnJ2en5KjpKWmp6ipqqusra6voRAAICAQIDBQUEBQYECAMDbQEAAhEDBCESMUEFURNhIgZxgZEy obHwFMHR4SNCFVJicvEzJDRDghaSUyWiY7LCB3PSNeJEgxdUkwgJChgZJjZFGidkdFU38qOzwygp 0+PzhJSktMTU5PRldYWVpbXF1eX1RlZmdoaWprbG1ub2R1dnd4eXp7fH1+f3OEhYaHiImKi4yNjo +DlJWWl5iZmpucnZ6fkqOkpaanqKmqq6ytrq+v/aAAwDAQACEQMRAD8A9U4q7FXYq7FXYq7FXYq7 FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXiN2n5tKdMcz30t tJeagGghhuUmih/xBahPrMvqusitZmQw0jTjFy+0tTiq+xvPzF/xDNqtzFqUMtxa6da6xcR2PI2o Se+eaKyT0WW4WNmiHPi54uWrtsVQGhaj+cmleV49PtLOW1TT9DUWFs9rPJM8qWCMCB9Vkj9ZbslO ElwtVH90T8RVZnqV951ttFvRcvf3TaZ5ksEiuoLZ/rFxpQltJrhxFaRgyoBJKjcE3VSDU1wKkmt+ efzBt9R1PUrOC/h8u2ME2ov9ds/SPo2U0ErRKJbW39MXFq0oUNM8leojZSpVQ7eZPzvW3vZTCfUG myXdnCtrJKzySWizoqelaNCZYrl/S4vcAsq/3ZPxEq9ltIZYbWKKadrmWNFWS4kCK0jAULsIwiAn /JAGBVXFXYq7FXyZd/8AORP5iy68mpwzwwWsey6UIw1uUJBo5P7xiafaDA+FM6UdmYhGuve4/iF9 ReWtVfWPLulas8YhfUbO3u2iB5BDPEshUE0rTlTOeyw4ZmPcabwbCV/mT5qu/KnknUtftIY7i5sx F6UUteBaWZIatxINB6laA5ZpcIyZBE9USNB4L+Xn56+fZ/OtjZancpqFnrN7BbSwyIqCH15Fi5Ql AvHjXoag/M1zc6ns/EMZIFGIao5Db6fznm95X+fP5l695M0/TLfRVjS51b6wDeSLzaEQen/dofhL N6vVqgU6eGx7O0scpJl0prnKmO/kP+bnmrzBrR8ua46XqrbyTw37DjOPTKji/H4XHxdaV9zl/aOi hCPHHbyRCZOz3bNO2vnj87Pzl85aT5uu/LeizJp1vp5gZrqNQ80rSQpNRi4ZVUc6UA37mhpm80Gh xygJy3tpnMg09E/JP8wtW86+W7m61WKJLyxnFu00IKrKOCsGKGvFt96bewzB1+mjimBHkWcJWHoe YLN2KsZ/MnVtQ0nyVqV5ps722p0ihsJo0SVxcTzJDEBG6yK9XkA48d+2+Ko7yvr8Ws2MziC5t57K d7K7ivBEJhNCByJ9B5Ijy5A/C1Pl0xVifnrzP5gjOuPpF++nW/lpbMz+lFDNJcz3TKzRMJkl4xpC y/YAYs3UAfEVZf5d8wwa3bXMsdtPZy2dzJZ3NtciMSJLFQn+6eVCCrAghj1wKhbrVNRsPOFlZTSi bTNZilW2QqA8FzbIJCAwHxJLHyPxbqy+B2VT/FXYq7FXYq7FXYq7FXYq7FUNqOmWGpWxtb+BLm2L I7QyCqFo2DryXoQGUGh2xVE4q7FXYq7FXYq+ZvPv5WeWtH87BtOsLm50GJFkv7GOfgRKQ8rRxyMr sEEKhzXelfiA3G0j2rkEa2vva/DFvcPJfm/ylqFha6ZpTGya1iS3h0u5BjmRIkAVFDE8+K0+yx98 1siSSTzLYjfO9r5fu/K99a6/G02lzKqywIWEkj81MSR8CGMhlC8QOpyWLJKEhKPMIIt8/wD5feUv LmgeZV1rzBY3TWNtdRtp0olEi2rmRvq8lwsaI0is0TBXXbkpqu4zOzdpZJx4aAvmwEAH0xbXVtdW 8dzbSpPbyqGimjYMjKehVhUEZrWx5F+eL+WPMNmmkx2smpa3Y+q0FzbycEtdlMwZjVHY8UBSmxpU qSK5Om1M8RuPVjKILf5EeTPK+jpNcGOX/FcavHdmdtliEhjb0FWilOcfFyakMKeFbNVrZ5djtFEY gPUdZ17R9FtvrOp3SW0ZqEDVLuQKkIi1Zz7KMwmb52/May0bzv5mXVrHTLjT4Kr+k9UZqtLDFFzM vobqvC3i5LRtxxrSozP02vniHDziwlAF7x5J8j+X/J+kfo7RUcRSN6s88jl5JZKAc2Oy9B0UAZj5 9RLLK5MgKZBlCXYqlnmHQYtbso7Z7mezeCeK6gubb0jIksLckYCaOaI0PZkOKsA8xaL5g0y7n8ve UmupJbzQ71VlN0I3W+uZw31+Z2eNmlryHNAWDMo2XcFU3vvI2o62srz3M2lW2t29sNbto3ha6jns 94XSRkuYSWFElXpsOLdeSqB1DQdZ8va3aQ6PqF9qc2sy6ldXMNxNbQD1pYI41k/dpaqUibi1AGcC pUHfFU28paD5kmTQL/zL+6utF08Wsdu0guJpLt41iuLqeUFlJYRngAx2YljU0UKzPFXYq7FXYq7F XYq7FXYq7FXYq7FXYq7FXYq7FWCzQPceX9V8wRJ6lxFqkupQDqXisGFqVHtLbQMB/rYVSzXvKGi2 0kczMsOh3hQ2Opqqutm7nkkUytRXtGJ/dk7xMaKy1BChJY9a120jXVr+UNZ2UU03l+0vmleC5YIR 9YtrlgzmSi1hhuAzcW2beoVThb3y7psujWOrp9SibT20fWbTUB6dY0j9WOblUpIgdHXmjEAv1ril Ipf0vpizraXFzp2jXXI6eJFeJ9X+NeMFEBe2uHWqiRVV5R8RWtTihNIbnQ7C0t7bUwmlarc39sl9 YygRra2dq7XUcMPaSJjBQyIT6jtvuQuKqOq394dWuxYh9Gd5/wBJaHd3EDteyPcII5oba0G7rLJG WcS0py5cehCq2x00390Vv4jL5slkMA0i7c3Bh4UZru9kIAlhUMHSOMLEdgAW3CrL9P8AL+nx+YF0 oA3Fvp+myNeSSbvNPqctHllbu7LbPXwDeFMUpp5LuJm0KOzuGL3elu+n3LHqzWzcEc/8ZI+L/TgV PcVdirsVdirsVdirsVdirsVdirsVdirzrzW3nxvzEs7bRri9tNFu4rGG7u4YVmhiFNSeZl9ZJYUf kluGbjX7IPUDFWKWOufnKmlXl5Kl8NQvJdPlljntnEdrbyacGma2SKzu35fXFMbxiORlFKqOXqYV RWs+Zvzgs7e9uY7e8uL2KT/Q7W2s/wDQpBBZ2srqK2c90wmmnmVeXpD4Kc0YUKqYyXPnu8Gg3U95 q9vci+lTX7CC0CW8Jks5vSiiYWwea39cIvq83C8uRZSAVVSfTtf/ADjTybFJ6V+fMMWmym5sprLk kfp2h+qzRSyQlp7qWYL6sTSNQk/CtN1WT6BrH5gf4n0+z1QXVxp4m1C1mK2ggBjt7idLe8uZWg9F lliSLikUsbVPIK6H4Qr0bFXYq7FUFreorpmjX2otuLO3lnp4+mhan00xVT8vaZ+jvL+n6dJ8T29t HFMTvycIA5P+s1ScVYjo8FzrET+Unr+hdDme11Ofl/vRFG5NtaKR29Hh63/A9zhVAebriTRrW18o 3sTanp2pXVuNKUMrTtBDMkj2bhyKkcQiSMacW3IIqVUD5r8r+YLXy7cXutzfX9Ptomls7Bp2J06R W5xMZH4/WwKBGL/EBXhXFCCtLbzR5xie90ZGn8vwCRNLGqzM0kMtFBmgkRhK0yMG9My/ZH7VSQFU LazXGqea00i/juLjzbaxzrFdXVwLcxyO8bwzL6bMkYhjVqxw8uRP7S1IVT3V21byZqll5g14yavI krQzazGQgeCaMgW5tyeMBWUKylfhbfkQcUski8tau0a+ZYrhG81Sj1SA9bV7dgCtjtt6QX7Ljfn8 femKoryRfpqsutavwaKW4vfq7W0lPUhFrCkRikAJowk9Q/TgVXt/9x/nW5h6Qa3bLcx+H1m0pFL9 LQvF/wACcVZDirsVeYat5787aZcX0dzcaVEljr1lpkrPaT1+oagkZiuF/wBL+Jw0vE9vhY9qYqzL SNV1V9Q1Aag8Uti18bPSDa2s4YLHFzkNzJzmQUkV4+ZCLVfFwuKo3X9f0vQNKm1XVJHisLehmlji lnKhiFBKQrI9Knc02xVLtU822CabqrWdytrfWKiKNtQgmih+tTRerBGBL9X9UtyX4Y3rvTY4qs8i +ddP80aRbyxyqdTS0tLjUrdY5YlRruL1FMfqj442KsFZWYbHeoxVkmKuxV2KuxV2KuxV2KuxV2Ku xV2KuxV2KuxVIfOP76xs9N76nfW1uR4xo/1iYfTDA+KrvMGqXZni0TSWA1a9Us09AwtbcGj3DDx/ ZjU9W9gcVYob220LzD+ivLDQCO4tha3t5cljb29za85jLLJsJpmid2deQaqgsRhVLn06z1zX0tNI iGtS/VJZNR1zUCyxO87COGWIoAzJGElEaw0UHowIJxQsttMi1LzGNI806j+kYdMd2bU7hikNz9XR W+rRxmkMbxhq3BBZmApWnKirei6Vf6nqWqweVboaPBFI8w1NQ4+t212WeBBAQo4o4kVJ/tBQOFR0 VXaTpPlfU7fWtJ1aGLR74TlrIljIENhHxkuIriQL6xWcytJy+Kh+PFUPpema75itrqO4mFybe3dL HR715lcQXMQRb+2uJBzPMhvTEquVHwl1JOKpl5f8wJY6Wl1pkq21xDE76p5YvCYozJb1Fy1g5BC8 XRqqlU8QhrilEeV0MWj22raHV9ctol/xHo7AxSXDSEyuCj8aSK7sYZKUcbVodlU88w6jZ3WhWHmi wf1ItLnS9L0IYQbw3asDuCsTvVT0Zd8CsqBBAINQdwRirsVYX5r/ACzs/MWr3d7PeNBBe6e1lPbI lSZ1WZLe6D8hRoUupaCm5I3+HFUs8wWmr6Dd+Sbe2n1C7eG4vDrF5axXP1eUzW0jSTXccHOBQ95I jr6v2d6GgY4qpeU9a8w+ZY7PS/MFpeyaXqmgRwahNLZmGJ78qwvH5+mgVWV1Cfsv1QFRXFWtS8r+ Yre68taBHqN5PcTR6jLquvJDcC1ll+qiCFbyCGVYyDyUorv/ALr2IqcKodvL3mXyMbXU11dL6Gb9 C6RcW9pp0gmNpp5cOyr6t27s8cshZUTl047jcKzDyHqmvahpt4dct7i3vYr67VFuIRDS3ad2tlQh VWThAUBcV37luWKslxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxVhfnXzHZaRrlhcXRBXT7a4uYoSwX nczcYIF5HZRw9Ysx+yoJwqxjR9Q1/X/rMWjRSzLeSc9W1irWoncfCIkmILQ28Q+FVjDSkdfTJLFQ nms+SFtfLwvLtxey6OY7220yJBFYRrbuJJY47f4ubPGHXnIWap7YpQOo+cLePVNVXTpz9e1SaGzs 5YF9WRLS2iFZIoxXnI00sqxL0JBY/CrUUJrD5COr6TFaaypsdPt1rpukwOGMMlDS4uJd/WnqxJH2 K1ryrXFKHOvy6V5osm1vjDfJBLY38qjjHcQAGa1u4x2HON0Za/Cz06EHFVDRNKj82aPZaXeR10a2 /wBL1eQVDT6hcMZ3gjcUZVheUmRlNeXw/wA2KovX9O13TEhuGle8SwJbTtaVC91bCnxRXsaCtxbu NnZBy/a41HLFWPrrGl69ZaxojQxs2q3dnLDbtxlEUl3KsN2YnGzKvAzq69Q9R7KGUax5L1SN47rS Lx5prUEWn1h6XUK/yRXZDl08Y7hXVvEYpYr/AIwfRtUnh12ye0GoApq9myenFchhwNzAgLqs1DSR VdkkH2WL0GKGe+QtRW88tW8frC4ksS1k86kMJBAeMctR19WLhJ/ssCWQ4q7FXEBgQRUHYg9KYqp2 1tb2tvFbW0SQW0CLHDDGoRERBxVVVaBVUCgAxVUxVTntba4CC4iSYROssYkUNxkQ1V1r0ZT0OKqm KuxV2KuxV2KseT8wfJ0kmmImpITrECXOnHhLxeGRWdWZuPGOqxuaOQfhPgcVQ97+Z3km0ktlOqQy w3M0UH1yN0NtH68VxLHI85ZY+B+pSrVWNGFDiqOt/Ofl6W+vrN7uO2exuYbP1J5I445pri3juYxA xb4+STr71xVFf4h0QNqSveRxDSHSLUnlPpJC0kSTLyd+K7xyqag038cVcPMvlwxGUarZmIFQZBcR cavGZl35U+KJS4/yd+mKr7XX9Cu5YYbTUbW4muYhcW8cU0btJCSQJECsSyVU/ENtsVR2KuxVIPPn meTyz5VvdYhgFzdRenDaW7Hir3FxKsEIY9l9SQcvbFXmvnfyb5k0yJPPfmS8tPMx0qNW1bSDbNbw /VVcs4tmWQ8jFzZlEqty7+GFXstsIBbxfV1VYOC+kqDioSnw0G1BTArFPPWu69FqGjeWvL5hg1XX muCb+6QyxW1raKjTyCKo9R/3qqik033xVivlbyzP+XfnHTbG5eDVbDzGZLW21MxGK6trqKJp/SI5 yKYpUjb7NPiG+2FXrOBXmHm/S9R/MHzRfeWbeaHTtK8t+g17qDRGa5ku7qITJFD8UYSNYmVnNTUk Cm2Kpv8Al5e6rp9/qXknVRBJcaFFbzWV9bIYkuLS6MgRpIyz8JVeJg++/XFWcYq8W0Dyjrvm28P5 haLd2egNNJKdBtEtjMrwq7xie6JkC8pt3oibbb8t8KvR/IPmi48yeXI768gW11KCaey1K2jJZEur WVoZeBO/EsnJa9jgVZ+Y97pun+S9Uv8AUNPj1SK2i5RWUoBV5mIjiFT9n43HxDpirA7Tyx56/Lfy vd6/a6la3sVty1HWtAWB0hMKLWYW1w8ksqvHGvwlqhqb4VeuWlzFdWsN1CaxTossZO1VcBh+BwKp apqlhpVhNqGoTC3tLcBpZWBNKkKAAoLMWYgAAVJ2GKpDN+ZPlJPL1xrkd2ZLeBp4Ut3jkguJbm2Q vJbxwzrFI0o4kFeO1DXoaKrYfPmnWS3dnrrums6ZDDcapb2dnezRxpdycIBEyxP6+7cKx1qVY0Wh CqonU/NAstes7USW9xYTTpp96I3BuLS8mQy2/qqCfglWgAIDVZTuDsqyHFXYq7FXYq7FXYq84/5U xpwMxN6J1k1CW8jhuYBLEto9tdQpYEB0ZoUfUJH+1Uj4duuKqcX5PX0c1tdnX/Wv7aS1ljkubZp4 62i36InGW4aRowup8VDyMwEYqzE1xVCD8g9NQwiPU3kgiWOH6vKLhUeBbSztJEc2tzaSMzfUAwq3 AcqFGoMVZHfeR9fk1HzDPp+uRWVr5gkguHT6o73EMtvBDAOE63MY4MtvvSMOKni6mhxVg8P5GNca odMnvbtdMsbYyHUPjjE+oSXc1xDJH++ec+hBcvDI3qAsCPiqDQqzzyd+XsHlvUn1BbhZZZbJLOVE WfiZBd3N3LMHubi7mrK938QaQ7itd6AKy/FXYqlfmjy7YeZNAvNEvi6214gUyRnjJG6sHjkQ9mjd VZfcYqw+68geedbgTRvNHmaC98uI6G5itbP6vdXqRsGEdzL6joqkqOfpqOXTbFXogAAoNgMVY35y 8oTa6dPvtOv20rXtHkeXTNQVBKo9VeMsUsRKh4pVADCoOwIOKpfo3kzzHP5htPMHm/VoNRu9NSRd KsbGBre1geVeEk55vK8krIeIqaKK064qzTFWG675N8wJ5hn8x+UtVh07Ub6KOHVLS9ga4tLn0QRD IQjxOkiKSvIHcdsVRnk/yfc6Nc6jquq6gdW8wau0Zvr30xDGscClYYIIgW4Rx8j3JJNScVZNirz6 LyH530U3On+U/MdvY6BdSyTQ217Zm6msTMxkkFq4ljVl5sSqyKaYqyryn5ZsfLWg22j2bySxwc2k uJjylmmlcySyyHuzuxY4qr+YdCsNf0S90bUFY2d9E0UvA8XFejKezKaMPfFWFXH5f+fdWsBoHmHz TFeeW/hW6MFmYdQu4VI/czzeq0ahgKOyJVt+lcVehxxxxRrHGoSNAFRRsAAKAAYqk3mvy9da3bWK W14tlNY3sN8jyQ/WEYw1orR84+7clPLZgDv0xVgDfl9q318eVH1C9azbS9SuZNWWOdLaa/vr0TRN Ogd7dpYW5v2JDClAMKsv/wAN6jq+p6Pr2oH9GXltGYdW0xOM8dx6UyzQcZQRxEc8fqIeNSrFWAPQ KkPljS/MWr29mL+C7trSPWrrVrj9ISTNLGkM8i2VpCJyZOPERylvsAfCu5PEq9JwK7FXYq7FXYq7 FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7F XYq7FX//2Q== + + + + + + uuid:B356723E9A14E81188E88C6EC4CB6E13 + uuid:5da937a6-fc9b-4248-a315-698c69e6ff3a + + uuid:B256723E9A14E81188E88C6EC4CB6E13 + uuid:B156723E9A14E81188E88C6EC4CB6E13 + + + + Print + + + 1 + False + False + + 460.000000 + 180.000000 + Points + + + + + MyriadPro-Regular + Myriad Pro + Regular + Open Type + Version 2.007;PS 002.000;Core 1.0.38;makeotf.lib1.7.9032 + False + MyriadPro-Regular.otf + + + TimesNewRomanPS-ItalicMT + Times New Roman + Italic + Open Type + Version 5.05 + False + Times New Roman Italic.ttf + + + TimesNewRomanPSMT + Times New Roman + Regular + Open Type + Version 5.05 + False + Times New Roman.ttf + + + Symbol + Symbol + Regular + TrueType + 11.1d1e1 + False + Symbol.ttf + + + + + + Cyan + Magenta + Black + + + + + + Default Swatch Group + 0 + + + + White + CMYK + PROCESS + 0.000000 + 0.000000 + 0.000000 + 0.000000 + + + Black + CMYK + PROCESS + 0.000000 + 0.000000 + 0.000000 + 100.000000 + + + CMYK Red + CMYK + PROCESS + 0.000000 + 100.000000 + 100.000000 + 0.000000 + + + CMYK Yellow + CMYK + PROCESS + 0.000000 + 0.000000 + 100.000000 + 0.000000 + + + CMYK Green + CMYK + PROCESS + 100.000000 + 0.000000 + 100.000000 + 0.000000 + + + CMYK Cyan + CMYK + PROCESS + 100.000000 + 0.000000 + 0.000000 + 0.000000 + + + CMYK Blue + CMYK + PROCESS + 100.000000 + 100.000000 + 0.000000 + 0.000000 + + + CMYK Magenta + CMYK + PROCESS + 0.000000 + 100.000000 + 0.000000 + 0.000000 + + + C=15 M=100 Y=90 K=10 + CMYK + PROCESS + 14.999998 + 100.000000 + 90.000000 + 10.000002 + + + C=0 M=90 Y=85 K=0 + CMYK + PROCESS + 0.000000 + 90.000000 + 85.000000 + 0.000000 + + + C=0 M=80 Y=95 K=0 + CMYK + PROCESS + 0.000000 + 80.000000 + 95.000000 + 0.000000 + + + C=0 M=50 Y=100 K=0 + CMYK + PROCESS + 0.000000 + 50.000000 + 100.000000 + 0.000000 + + + C=0 M=35 Y=85 K=0 + CMYK + PROCESS + 0.000000 + 35.000004 + 85.000000 + 0.000000 + + + C=5 M=0 Y=90 K=0 + CMYK + PROCESS + 5.000001 + 0.000000 + 90.000000 + 0.000000 + + + C=20 M=0 Y=100 K=0 + CMYK + PROCESS + 19.999998 + 0.000000 + 100.000000 + 0.000000 + + + C=50 M=0 Y=100 K=0 + CMYK + PROCESS + 50.000000 + 0.000000 + 100.000000 + 0.000000 + + + C=75 M=0 Y=100 K=0 + CMYK + PROCESS + 75.000000 + 0.000000 + 100.000000 + 0.000000 + + + C=85 M=10 Y=100 K=10 + CMYK + PROCESS + 85.000000 + 10.000002 + 100.000000 + 10.000002 + + + C=90 M=30 Y=95 K=30 + CMYK + PROCESS + 90.000000 + 30.000002 + 95.000000 + 30.000002 + + + C=75 M=0 Y=75 K=0 + CMYK + PROCESS + 75.000000 + 0.000000 + 75.000000 + 0.000000 + + + C=80 M=10 Y=45 K=0 + CMYK + PROCESS + 80.000000 + 10.000002 + 45.000000 + 0.000000 + + + C=70 M=15 Y=0 K=0 + CMYK + PROCESS + 70.000000 + 14.999998 + 0.000000 + 0.000000 + + + C=85 M=50 Y=0 K=0 + CMYK + PROCESS + 85.000000 + 50.000000 + 0.000000 + 0.000000 + + + C=100 M=95 Y=5 K=0 + CMYK + PROCESS + 100.000000 + 95.000000 + 5.000001 + 0.000000 + + + C=100 M=100 Y=25 K=25 + CMYK + PROCESS + 100.000000 + 100.000000 + 25.000000 + 25.000000 + + + C=75 M=100 Y=0 K=0 + CMYK + PROCESS + 75.000000 + 100.000000 + 0.000000 + 0.000000 + + + C=50 M=100 Y=0 K=0 + CMYK + PROCESS + 50.000000 + 100.000000 + 0.000000 + 0.000000 + + + C=35 M=100 Y=35 K=10 + CMYK + PROCESS + 35.000004 + 100.000000 + 35.000004 + 10.000002 + + + C=10 M=100 Y=50 K=0 + CMYK + PROCESS + 10.000002 + 100.000000 + 50.000000 + 0.000000 + + + C=0 M=95 Y=20 K=0 + CMYK + PROCESS + 0.000000 + 95.000000 + 19.999998 + 0.000000 + + + C=25 M=25 Y=40 K=0 + CMYK + PROCESS + 25.000000 + 25.000000 + 39.999996 + 0.000000 + + + C=40 M=45 Y=50 K=5 + CMYK + PROCESS + 39.999996 + 45.000000 + 50.000000 + 5.000001 + + + C=50 M=50 Y=60 K=25 + CMYK + PROCESS + 50.000000 + 50.000000 + 60.000004 + 25.000000 + + + C=55 M=60 Y=65 K=40 + CMYK + PROCESS + 55.000000 + 60.000004 + 65.000000 + 39.999996 + + + C=25 M=40 Y=65 K=0 + CMYK + PROCESS + 25.000000 + 39.999996 + 65.000000 + 0.000000 + + + C=30 M=50 Y=75 K=10 + CMYK + PROCESS + 30.000002 + 50.000000 + 75.000000 + 10.000002 + + + C=35 M=60 Y=80 K=25 + CMYK + PROCESS + 35.000004 + 60.000004 + 80.000000 + 25.000000 + + + C=40 M=65 Y=90 K=35 + CMYK + PROCESS + 39.999996 + 65.000000 + 90.000000 + 35.000004 + + + C=40 M=70 Y=100 K=50 + CMYK + PROCESS + 39.999996 + 70.000000 + 100.000000 + 50.000000 + + + C=50 M=70 Y=80 K=70 + CMYK + PROCESS + 50.000000 + 70.000000 + 80.000000 + 70.000000 + + + + + + Print Color Group + 1 + + + + C=0 M=30 Y=70 K=0 + CMYK + PROCESS + 0.000000 + 30.000002 + 70.000000 + 0.000000 + + + C=5 M=70 Y=90 K=0 + CMYK + PROCESS + 5.000001 + 70.000000 + 90.000000 + 0.000000 + + + C=5 M=90 Y=75 K=0 + CMYK + PROCESS + 5.000001 + 90.000000 + 75.000000 + 0.000000 + + + C=30 M=0 Y=95 K=0 + CMYK + PROCESS + 30.000002 + 0.000000 + 95.000000 + 0.000000 + + + C=60 M=5 Y=95 K=0 + CMYK + PROCESS + 60.000004 + 5.000001 + 95.000000 + 0.000000 + + + C=30 M=0 Y=10 K=0 + CMYK + PROCESS + 30.000002 + 0.000000 + 10.000002 + 0.000000 + + + C=60 M=10 Y=5 K=0 + CMYK + PROCESS + 60.000004 + 10.000002 + 5.000001 + 0.000000 + + + C=80 M=5 Y=10 K=0 + CMYK + PROCESS + 80.000000 + 5.000001 + 10.000002 + 0.000000 + + + + + + Grayscale + 1 + + + + K=100 + GRAY + PROCESS + 255 + + + K=90 + GRAY + PROCESS + 229 + + + K=80 + GRAY + PROCESS + 203 + + + K=70 + GRAY + PROCESS + 178 + + + K=60 + GRAY + PROCESS + 152 + + + K=50 + GRAY + PROCESS + 127 + + + K=40 + GRAY + PROCESS + 101 + + + K=30 + GRAY + PROCESS + 76 + + + K=20 + GRAY + PROCESS + 50 + + + K=10 + GRAY + PROCESS + 25 + + + K=5 + GRAY + PROCESS + 12 + + + + + + + + + Adobe PDF library 8.00 + + + + + + + + + + + + + + + + + + + + + + + + + endstream endobj 2 0 obj <> endobj 22 0 obj <> endobj 23 0 obj [/View/Design] endobj 24 0 obj <>>> endobj 21 0 obj [22 0 R] endobj 5 0 obj <>/ArtBox[8.69482 21.6172 456.34 158.043]/MediaBox[0.0 0.0 460.0 180.0]/Thumb 46 0 R/TrimBox[0.0 0.0 460.0 180.0]/Resources<>/ProcSet[/PDF/Text]/Properties<>/ExtGState<>>>/Type/Page/LastModified(D:20180216105558-05'00')>> endobj 37 0 obj <>stream +HVn0+(p_8Aנ"4خ8.ECRekUԈ`â3~i?a**(-nTKsEAiHnOEAVs|s.dd>stream +8;Yhp_%?bm#ip8bpdkAMOg@0PcqbY3 +0=YAsHN9fb@po~> endstream endobj 25 0 obj <> endobj 28 0 obj <> endobj 30 0 obj <> endobj 26 0 obj <> endobj 32 0 obj <> endobj 33 0 obj [34 0 R] endobj 43 0 obj <>stream +H\j0 ~ +Cq]C`Mv,8(!o? L`-ݴז|v`簰Eq%8oN&*-p +s.xTϦ;qBJP@]Ax3ةus|7 h,UUHPH +w]^]<]7NPvaC~o endstream endobj 34 0 obj <> endobj 35 0 obj <> endobj 36 0 obj <> endobj 42 0 obj <>stream +HV TAv eEf|QQE"EEQQ+*1"iRi<#xlRI[k9=솠'm{g DxfW$زג" v|f^K_TkV =qfbvt[#඙x,^Z{~90xyk԰axNުu;*4.]t"/m]k7LeiyYe Vt"~Wʬׁ@8|u03ſ:ZH]Ѱ|9G"RhkcZA$^`עP.6?hD3>Gגe}W@s+|OsDќS4iv+9Es^<9h)iN*ScE_Bx`B)Qd͍ E\'UUTeJ{Ϲv)2~ WPnT#*x>No_C~2V +`Sb6P4NgpbKD?WEY DxRK,R MdoWɢpօodow;W[7SmOz{zZߥ2u><ѹ@E@;б6+:h.Bs>1گ9,]v[{p򻆻%kg2)Go,g;*\ǹ5Џw:F",xT!Ld!;ma)!Н&|RL4+Qp*oBCCU Ìs|pKAjM)+ ӹ9V' 2]o=\ƄR1tJtWp5;WY/%l%OZFJ(9US@jsMDNEP$>$6jPB0fщa}v %ԩo`QDי<毎 +Red&Lpk)p%9։mB=Û +%\:sdH. c! X(9C΃Wg[xzX:Hzp%haԓˢWXz:ޛNܴ޾ȱ|-#W%SESMt0,VT<`u6.hf-vYE +5:hCͳPl*` Cm|֦ + +ߔCbę\fRM$L6HL8Sh4MP'0웚`HT~Q +FQ[bTb܌$%nUI@h33bIYVKAL6/yYZ }d2ణRX-d!e!W@&l'mE&-d1HgkN4ۚMb]}Hiݰd @9/A%[j. /^ĺMP,AqM(QQJ~ H8B*YB_f?Ve n*VTuVtmnAV7>xV[lj-_C1P4^n"?DjjF#GOQ0Eյvu꽣齭F>iF{F3AQQz1~2~WqUz%i$|x0P|GJ0 1Fټ;> HC2X CQ#ebuLgDqS;L[Ās=ܻ;H>mfi>2ST.XX@ Sln=zi欲2D5Ĺmo:^3k21ol4)XX`wdaiϝ̚SU_Zc--[~=ab{tԒ _wyƓw? /Nt(-w`]ߔ7(]n{ELIC-4t]C=XLrdIHI$$BtQ棕]M>Pw;x+-])8Ev峐bq~RoN*(n l26m)}]'> + (+X8I~"GB ȃ[r D+P)ȃP&@QdW( ʳa ][\!3U~ +&*0 BVG6߃T$=4$7 6{✡Yń1O +j$ǩ~XOc=//=LȾbuW%{|6{|,\Rd!`śxͻr+t<p|F YL,>+O 0>m={y"gң@Bs,"e>3&`}uqI_|sB19E89@-H98!0g3)e]> ({B}fzH1'8 p,8A' TS8gb)i,`9XqVB|a?q .ep5k\ ?؄͸[؊_6"n/&|Qa5܌ La'B-LA3ve|+܅$>On܃{ږU543T۶ŭmGLkd@3fy:pV2sjMc+0IUZVM괣)J\M$Tc֩鴪ԫ*<eV=Аce +u1iMz"j&s%ObzGUKxr\3)]Mg:y4ËffzB6qX#K2~8Fbne+%@rSTRIBbq9+ŌLG )rj\KVڲԩFViۃN&t^QUQ?tδvA1D٫HA" +UDT@d}&vo j\5\1u) (+ƅ?09ns޻6$L+N]%pV-~cvk@Za:alBY$[U%[J5xf7фW e&AQVK[4B2[nEˍD236=1i#Ûm-7zs v4# ͦ8V_-=9V?51HS|٠$*0'j +˚|&`.ixQVקJG:+8ql.[⍹lf +^}N5 f1I;nfr j9OLeڐR)d1]LZi4["v[3HIoU(\MhH3ls=!đ9ĐץdU*7S],%"m -;cs +?P͡Tվna2Of$ FWMvlöRY*S -̳T~^RVvTf쎋eg &e-a?Ro/n"]@Տԭ^ۓׁA\In<>RC$!&"$ '#%!&"$?ߓߑߒZ5Klx!_P|OGKCME)Xhh9_ޠףZ^*_||?O9Y||||‘dr_-uC~<P@.y G%?K plj1Q>"kŨAmAN\ìGxяvvYMmH۽lsy/y[&wQ;86ZzmԴ?8iۥ 11plַr[(LޤyDFZn gYzcnd鵢kXgK&bLd+e|d;x1 k9uq[G(^} +lZb"jѷU[^LvZjhɐ+u}SԡeR-,_ [-pؒ~ŐfYJHl /"ȅBj '#s&ggzrfχ ?|O|YKNlR>j52pG2QZ7$YprY S 0QI[._*8bWVηQemڶoWYBEd,$rX!ܣ{d߭> endobj 39 0 obj <>stream +H|R}TUaYiw<;c@ +Yd~TGbowMbDS@az Pt?*DA"XVbt3bNhSi&# j}*g2R:C:f-ҬޒpK ±` İeV b%EV`; + ^Y^@6A/ zyov;pɾ!+]qbK̓@H'pW*D>CRLC$! 95 N-K= Џz.ujFЬ Qm٥ +2!>}ʼn*N QvUnC+Vmݜ:J^tyd[nW4^chogSU5 +mJ9 ۋT))L?^=|,FAzcN8)&Զ$A+n|@n%bN\cD`Q&1Q _hZ]Gu}֫s^Agżֵ6\>eD&ܥyCڇk z{o 7u{ePCA(~BmAoy-# \Z Sk4! EXXp?z鈪LHy(n,QwarPEٗ~\蟗~9Nm^x``-Ue5#:`Q(h:>A/SmX&M;LXK7UyP%Q}YprR Ws+H:dȬ7k?t=f>"iy)s:]0uҿ9wR%T}_2 ˢXt m +zuF@PQ5ŒFI~z?sepLA#崙Hf 0C`bYa)5Ǿb},mIA>3Q]aIJjʪU=g}TWU,wPAuiĽ,MP ſD6^IrO\ !٩t^"o"Vh8A᧪o$hؘUk3k&2%3=uk7K;}t%ߡZE|U*c]cA+t#a`d[P>בS욘WkeUb> rl.❻Tys +rwG՗wj癡n ~0B@' 7n__5E v4ǧ)?(lvTI*Pl;^PP%-,{哞ϋ? > endstream endobj 31 0 obj <> endobj 41 0 obj <>stream +H{TU}省;<""A$j*(X1&iQ[Vk(( YڬDc|5(/]ѥF.7c%Ī1_$jkm>s&]}䏻o͜3sPㆎ=pFx{43eeԞB +/}Xw`p ufbIrgL,s@Z< 21;+c);_l݉NTnW4}ۡS@JZ =6@I*eL/lZAqyYoUeg N+sv{ + +R#r\<0cǛ/U6D, )CArpmK>3j1ox-FcQQ[~*,|k#2}>yhcóS L i"Z%Js F X*@# $y)r^JVA2%Owі%=}` <i3f«Dn) ``T@+7#cHk*SoTz_|?)_fVoQܪP>`}^*)Cˬ}ʗ[(oUQ?V:A!.S)@џ(_fQo}|uTEH6;?"Rf:dNh25IguJ9\j.9MN9\D.*"4˚MsM&rMDnj"4zMAJa##ۚM&&ҠSD5􎹯|w3ft躛 7nFw܌}fDFh kکnn!=_D6 Z{ΰ~ մFgt6gaƋͻ[~WU~'Tꈫ>uQW}};2uU'BE(Dd1XbN6dvN%#@,aܞNꌫκU\U󮺠=:2:ZgVJcX-ev-[JkmMl+3?C w mvQn^_ݵ)]OfoS)gl91VkO_zu Ulfk>>6 oݫ-l>V7"p(E|0W=]5\)E{ղAZ4T} =s:Gi3a38 {`/|_yoR\% +KVc܅ j3X*U.6O+|6|!/Koy1_W|-=_7ͼ]|7+~8?k~_.o#L)Li>3Œ2Xov2]fL1Sͧ~s 32yy{wH&M,COFh+L2Y}e9X#(.qr̖e.ބ+GlCk v5]i0 8` +8gsps!|ƅ%BV/W +Õq-f׃7hq3D>A|OR>g(9 +|_`W#_vew.u|~@R?GnW@(EUIHD"uE( +)Hh!b2A1)Ȥ8 z |SOBRoPrI6$5$ Jl̦U!Dԡ*Oh)U4ٰab܍;UqTiwB\t+W-W.˅ʥ2'c/djekl;e1UeUW^(*uA8H01seeeWk5P5j[NRw˪C}PrMk &ZsRkk)m̪X+YXmV5:úКfr@[ jkoh6(/|UOze^KC&z }У8=QO7L~\/7V6`0!TMy<֞vmbo +z^'d@V^~`Caa7q2#H-XP)93BrU"X$" X$dsjvTrF,zK}@(iJEJn Zplf-%r UYr ,ᦽ[X)=YXv]rk^m=Rr-UMi*t\x-KiEQ>nwej>vIV%]P7V=^=| + sK{ s.p.fn۰|w%~Fiy +4(7 y!yEiI$jW,kofnUhW?kjj +vvpwp6s:;::mΚNtS4KAJ$JB]JtʠLʢlʡ\jE g1Suԉ:S>uԍSH=K &mkE~ԟ +h A4 +PFõniQ,=Gi q4&DDyz2Jh:͠i&D/+*͢4ށUޅ5֙S>ll!lhz*T{jڦ'jjxu&/z\|#Ьcْ|70?I~U~Rc(rxP UvSzBFժ:UBPP8bb{Î<a}Pɽ8d/+8'<\M-q +ŵ fin)K܃_8GO4\Dh&"!)h!"I"]dV"[䊶f+u4EtRt=DOK}D_Ob(P!b&b3JT_p+gX1Q :AD:1N';$;)Nq;NO3)t91yg3M]Ա1. +ʮQmma&۶n`veG uvSTy=O=!UJ uRRՏIU?s+w:`>
֗گZoX,fK RgJCP>煼Ȯe=aS(ְlkW8ڧ:t NI:Y):Ut3@=HօHgl=Du fb.T_=kW}Uߩ}p~p18P'$p . +:܀p n; p/q?'Ddq} qƁj4#:c{ZxORFZF+ѵZUds5u5t9[:[8p6vlp:NtsxwE^Î}]vȱұ#qL.q)9=wo߲5kq>_ɿQu4}֚eS@#kqh49mTevw{&Fj_Zy1_\>> y'}b*s[_}+l6-wWu9JM!n8X k`B&xa$TDp}Xk1<' Q8 3t +ǡcpNI8w;3܃Y3~Lh/~a A0.Q0FX 8! a7&b*4-8q.(DBF\) 5Zq.x+p%՘kp-7&܌[p+n\q]-zci܍{}ˀcY{,q:{q`"VŸ be `<ϥ|!_JG$Qyt.UIw!=DKo#}2P`"Ce 2RFh#ce 2QI#e|#SeJLkں=:[ tC!2Sd̖92WI$|IPRom6C09<,ZBBIEb%J+VU~**c]TUQUR +Rʦj +QX*PpUUUSUjJo4Utf{ܖ"-R ݵR[)m.7kug<ͷW㵼9s$Gq4oȱ7x;wx?|>$䓫rMcy"qSVF)\P^GT +S*JŨ8TJS*K5ԈSjJͨ9ԊZSjK=uԉ:SJݨ;ԋzSK? 4O`BCi 4FhCci4O><Wȿ%/|57&|=_!?)?A)~/7{Hl"Ml'ld6M_B@6M |4]cw6 |Hdt6H$HdHHTHi}` #~Bd? Ȇȁȅȃȇ((6̈́"(b((R(2(r(/9*b@P QUQ Q5Q QuQ  !1)9Z%Z5ڠ-ڡ=:#: tFtE7tGD/FE?i4fLEiͥy4_^ZHh1- + WZ[z$k%LIDz(G33L0431",^~a&DhcbV5QsҜ悹d掹o1MyF9^zk BDA>',_>gB c#>sȍ!Bn /W~O=OW?P?O¼:/܋{^mb^mzۼozw;zǼ w;zt@x{ﱔ8DDBB"c)hK$b*viZ(N-E% Z?Os>_{}5)2L9mΘslΙ_L)3漹`. +Si.*Sm.+檹fWsfn掹+32恓;Ns99bs)Q겺:92_?[ta!c ( zuF/˝:UC0T9(hǨl3{61ߩԫsN^:X8ȒnX!=d,.pppn6snp#Mt[(7mqcX7&)`禺wooo:v8;{}N_V?\uDUqU*T'IUT:.BUKѕbTCtnFgxՎ-$;^1ҎO?hs!1{Av&db[VlK? ]e ]%.3_GhzJZ1:Vq⎽k7ݢ.~͘tt:D$}U_qЃO)zHK:I$Y*%`M*<xyTډ {±9nJ[r6@V7%9 +0`(BXE ^4THD{zb +w߉X]/G&I +bx%#Hw>D4F."0+Q8=$-!Q42o"cD{ħ6Id{e^~4XJۑx1B]1y-Hi{ :T:>(FƑl# +=Cn7?᳟c7{@ D*` mgH lٯ7@7)Wyhadkw$pL0M mp,֣Qʸ}iEٯ{C^%B0 dlz<`4OfM/m} N3J۱(,Ig d/+d)aJLk|S#4a \yv9ȕBZsF|{MR-Wvq6{rq +WJi̺we9N>#{5P͈Hpثb.TuN]b`l@;':V?* tOՁ*o7mt Ycn&>c&!\oˌo,ăhZ7-}2H%b |Y eCo1rnSI|'{SKrEn)&p޵gώt dxnryjFVźS/|"Q&Ę.foE li'p~ wȩõ/}վ~??Ɵ/{©V9O/_ۧuXOK$z^OqrSYYDo2Df:qxlQU V%Ҭd|5ؓYdu\V9y? ev*D)W Pƪ^<21̴Ƹj`WK.2Pm)ǥi^Qd4u)VE:\+8CHB5$uS꽾^*qoX> +X"i=&'%z'F<ۨX3kňRĪX~N 1P-m>jF-QSPfK++{'i7k=|;I 60zE4l4dBe'НX0g'*j/3LG'"[x]Ho#JmG,eQcs gQw+;'ɲel|,[RtJlײƶ'N\82Ο&LJ?&!Li@p8)( C3 0 _h tf! K;ٮ 3n~o=Q{ߡmtL/"ģ"4o{iOvr7ԟpy܄94S>'/5gze$}DFZsB xPդ_"ڐQ" D RfrH~h4R$rIk(C#0Dz)+4 l!XB~R@T,dOl{[s$8l #me,e>^,OȒ HL~}7.Fr&1! [ႊ/n IFE}\a;ȴ[B&j&# `W,/R6 "FƬcM +&c` +Z'E߈9!#D•z_7*Lo-"HyDmcTP@N "O!oDxZMPhT % 8Pa =G}e|8nԭ/9sN:UBcW7Qy42J `a%+ÞPKc"<5uq{W^;~uldp`OXz~/||>|!no1v; 7ܤX(R)h?_'𘻣hoA:WJا?_oJD0|]CX~>kߋdc0,j 8_m$ܢ"x3(O鴰.<8PDhñ4)TvU{Ibyyaۡ7[?@t=lΦ ,"L;Ul $:]O4Ӕj>fcls=ǝO;NNg.3Br1<2˾_T37 u' %zWT~`}|*::HBwO88{#H{=zW"_!tjyoCD0kDYHċ,,vM ) +-rY{`(8՚p*hVUŧ-H尧wޒ_b`b)]Z۵e]9fmfknhx C{yN R}PP8)/ЗX'Лu9FcFgwfRhܘ6Mjǵ[[;8S&vj`vr_qKS>MUaO"ސtkF2VjjYַ1vE%v =MMs$"=4=*1˵D'6U0^~t/D{[yV-ѣMqj,!D5Y<&sL:O!?Ȫ2W*o> L38;a=l$lfL4kY7H6o0,,n9yCtB?R-tN>M2.Dfkw;=#}L)cfSmek,u6,DlUYk3>2M}A!-f䌖㙵>0//jBWpƇp,D"fe5)=bMM[%9-b$%1. rPWh3\ >)slcW>'AvHKxZwEA|Fҿa\@g G=Ӟfڗ%ҬK3o)q@|7ЋsPzE-+ԞQNcADWGyHVd\fEWSk7A?r';?BlzgJNJv׵3L 9M|L,W۷PuRyTp5J~YlzJXpL8Qxq|:]xL.zx.Z)*]jlED¡oz@mQ 06;}g ?TXS k-rTX'O_Kd 'G9"$Y}h#<ǀf_q}ԣq>Oć[J`+`OۀG}{ +8LXy»t_&{ā! ׇɵXs\C&q5z{1mG87q{yD" yޠ9n.ҵ +c{k#ocL/} +5 + b?#]h㜧}q]?lŜU.½F~"\%]0[eRIM׼&?`-acr// +Y\ZQ@ƅ'cYM q_]W? bCPbcLH!HB@ +cf h[:2UjKPa>:6CJm)N)lgݟMB̹>ݳ眅=is\a ̏ݹH{9|xn-#ў)lS k;ȶyl,Eyn{m֡# #>|?uԏH h;Dp*g{؟}??2M4gww%X5ׄ̑ʹFwlq`i +X2Ѷ_ Ao3?d Sn5'|>BaYAwztlQF}⨟ dbphBx{bx~ ;m-iiw H}J_*QwDYҾׄ]@9\>-;g)ƥvi0${1X_Z}C۫-iۨ~{ 8 +0Λc*5T ϝr3j^I8,R'$h&,TnB h??Oۊ|"Wd=$V Km2nGsBѠX͞KJqHc} d_P2e>{%\4?ǽF_ڏS6qs)M1/EyNKfSRR0F*qV5ݙTB9^:4 R[~9bTe5#@;l35(Xx4U@?ngdf0< -JxVamԏ \}_~qo0c7Z~cbl7%#<"2ߋlX7 r;l%Ruhx 緕{Kc~G>|F}Xo!m1sl@;4BehB~4H. *iSs/R_\^3 ՗57?džh] rNպFd;яy8ROA(j'pw{k9[@稥TDYKm%jM쇸׊.~ u+ jZQGd@N`ض`}gη Yv lcyG By\0W+ێ( kNޡUvZiFvxCD^aο;iH~D+q\:sJΣ9?snֲ_8)9w9xGރU&nD}s@*:>s/1(q NA{@6 y4[;nb.4F^CL- +eU !w3v,)k]he ʩU݌v)wl=.[y)thm3w=r'ljGUQnC/L7Z'Z(e~`'9 _tVWm(T٬N ӝ=vg>S(i0Ϝ5=IDΕ9X9`r$=^F9ej/)~$ |X2ݧmb1*.M < ϣd7៳P}9שXUW .Xw=$26ѭnIM:r˟Gi=.IF`چ!YKQixok2eVQ%u*K:N 7;P0^1|2߱7wBYf~7-Q_k ~cNDcyWۨ`AWrZU#bV>?}{v)Ո }uars)Y ne|pB +.`7[oPI5r^?@}xM\'Èo!=Csś'MN\8pnUTWGT8%Q&8=YҽKrA&!]MASqblNtOFxc|rAϸ u%S6F@?5 kO۠$|c +C8/FN16˲,6a} YAN_WztUcGWf٠> +ؠzt6ϾlP?ouKOC>SEti?o/ܰ|$_>} +T^-Bu6JKAi!"2(l2/A錐vdaL2cf$q`6d6a1@}[r|~s=os=ZpG Qw4]$k!y|_Չ ׂyZ+UuV}tR7t>اyaf]UjgaU|QBzw%V!V@>Ð.36cK{Q.*^"*Er_#J` 4LytC.C۔soӍ;RKě;8o^T8=6c옳ZMu.|xY9o.g^_H`*vNq~l,(e*Kb*r>Y8kPn6^r}C9lȲވ@ebsYR6jtMguL5i +I>3zm7&(z{;OXG[=&{{?wTCӺ gg t1·aM#^0+'[ǎz/{yWu`XpU˳U=$l֣Ӡo\bny}|ކMz~la';CYѩne vbS~syqH-ZUi`R-]_ٔnξJtnoyEo֠ckSo22 S:k wН@.#53tj'&(j58J納|6')W8K!c-i#0皦Dc4/{Gڽxz{L[5wy&kx|9&pswX'buZI[ h$D`|s,x9 +܏X{}U:ž1 nؒ Aa^>/e*B ޶klo;}1S}V&+`t$Sc;  kT8? nnOC[)Q&{8NG`hۜ-{AeΞ0I( -XT8}]ƺL3 +pn򠋜XS.zཙx +g%XKs"ȿ2x@{v@H3!; [aU¢/l5Y`eLha_?3cem)AgΧqSYX~U3 i?${(cԂ-H~ewm=ݵTý'4F(ƻQ1ڹLb~VLzcE3B ڻ5]mRd",ԙ#ODc̗u,ݣ5hcdx}y]i챗!GڽT+HYNVp" +W3_v} +~i|ˡV_O1P =uw(>z^C݂Wr݂+@ڀxL;Jg[hޟ$nE` +Z-R7ľ[ُXɱk. 4n=$ XYKN퇳QIָ}[hEfApy%uzY?n1#%l1A\~ױgR 84Y@# p99uvQ>}޷8k*ybNupњ b;҄z*>=uRk~ݹJHh/w=ro:FLI6My6[Ly|nabQ1[Ad2Rn[Ja3Y78!'vߋqrpQ72p^nhwɋ*6&:󞧛Y˰znO`sa?{/o9ʿÍI݅tFr6tz5l}y\~(*/%UxOQN#89 `0 `0 `0 `0 `0 `0 Q *y +Ȧt+ OC~˾EbߦZ}F2MVBJr Zk]~lp-&kߦݧ}7Vƣ,q۬x/i?A=BgPMdNl +39+/_f2xG's|5Ws|5wce*_͡*_͡*_!w?+#e;w$)uG[ʌOLa/g_"\K_~"6DX~V./R Ōx t?̓LMoRɩAj|R O/z-@\R`JE<_,E4!)Մ9h/I￢Ei+CzE4#(o D >ͲBSײ2cXb2S2摠fώ+%R EXMuyݞQ#nA4 I*% ?𭈝69vn섊qn}OoN=dS9*6N9>ӏC7;:y`P ΙjG!w -;PT͒ꮜ*ST&R^u\T9W*K5@2Hv XjJ-Dj˥z*ՕzrR ,nDMrUͥtU﷑rӌ;QotkZwLw^Wo=of UfEYF!9v܏񘀉 x1<ɘ'0ObӘg,f`&fa6XK ˱+2^*װ:zlFl›x o؂xlv.Ƨ ^~Aa|5#83~ 7 TrćcdINS| ,sù|8 /E%\e\\ɗ +Wq5UƵ\s7r[|]nn{|ps?Gp'wq7?g_prK~ů w?G8m9 +I_Ķ&6'Ķm۶m۶m۶m;~EUKk[{GgWwO/!8Ȉ舁؈ Iɐ)i/2"2# "#r"r?A^C~@ABaAQCq@IBiAYCyT@ETBeTAUTCu@MBmA]C}4@C4Bc4AS4 - +m ] = }111C1 1#1 +1c111 1S1 ӃY9yXEX%XeXXUX5XuA#؀؄؂Į# >!1 )9\%\\5\ -=<#<<3< +;|'||7|/e0g )!aQ1qF Iɘ)i阞֒Y٘9;1yXXEvSY"h8XeXXXXUXX5XXuXX ؘؐMؔ؜-ؚؒmؖ؞ؙؑ]ؕ؝=ؓ؛}ؗ؟88C88#88c8888S88388s88 K˸+k븞[۸;{xxGxx'xxgxxxxWxx7xxwxxO/o_?W\AdPHRhQXSxEPDERdEQTEStPLRlQ\?JDJ$JdJJTJ4JtJ WIEYMٕC9K() + +JJʨʩ**jjꨮꩾZZڨک::zzhihiFhFihi&h&ihifhfihihihiVhVihi6h6ihivhvihi鰎討NN..鲮誮nn鱞詞^^>>鳾諾~~诃9iȎ⨎ᘎ؎㸎 ЉIɜ)ҩi3:3;:;s:s?q^s~pAraqQsqpIriqYsyWpEWreWqUWsupMrmq]s}7pC7rc7qS7sspKrkq[s{wpGwrgwqWwswpOroq_s@`PpHhXxODOdOTOtLl\|/B/b/R/rJjZzoFofoVovNn^~AaQqIiYy_E_e_U_uMm]}?C?c?S?sKk[{GgWwOo m6m۶moٶm۶m?Cp@HBh@FXCxD@DDBdDATDCt@LBlA\C|$@B$Bb$AR$Cr@JBjAZCzd@FdBfdAVdCv@NBnA^C~@ + +JJ ʢʣ** +jj ZZ ڢڣ:: zzcbcbcF"A񘀉ɘ阁٘XXXXXXXX؀؄؂؆؁؅؃؇88888888󸀋˸븁۸xxxxxxxx`  P 0I p H(hX8xLDL$LdLLTL4LtL L,l\<| `AbaaQcq`IbiaYcyV`EVbeVaUVcu`Mbma]c}6`C6bc6aS6cs`Kbka[c{v`GvbgvaWvcw`Oboa_c@`PpH28c8888S88388s88 K˸+k븞[۸;{xxGxx'xxgxxxxWxx7xxwxxO/o_?LB!JF%YaNAIEQMC1KGqO@ HDILɕB)JFiNAIEYMٕC9KGyOU@*B*"*b**R*2*r* +J*jZ:zjFj&jfjjVj6jvjN.n^>~A!a@iFkjk&j&kjkfjfkjkjkjkVjVkjk6j6kjkvjvkj눎꘎NN..늮ꚮnn뉞Ꙟ^^>>닾꛾~~ꟃ9C8C9e;9#8#989c8c9898989S8S989383989s8s989 8]ȅ]E]]%]ʥ]e]]]ɕ]U]]5]˵]u] ȍM-ʭmɝ]=˽}==ȃ=C==# hXxODOdOTOtLl\|/B/b/R/rJjZzoFofoVovNn^~AaQqIiYy_E_e_U_uMm]}?C?c?S?sKk[{GgWwOo\u|Yǁ6 7ܡw%)IqHK ŭ6Y  ++{?;vNVt@GtBtF*.;+;z'za'FE? ]+v` P pHhX B +A I)ijPC=fb0s00 KX&, +*:͈#-H" +i8pdClD6a3`O셽~@qa8GHq q8'Dq +Ni8gLqy8B\q .eWJ\q u7F܄q nmwN܅q}A<cxOI< sx/E +^kxoM{xC| >g_K| w?G ~oO;XN 3+XIvܑ]ٍك=ً;7/?p w ]srsGrGs rwbUxNDNdNTNtXZ`9ٜù\\\¥l26q9Wp%Wq5p-qlf la)Z\VLӡ 70,p#۸{r/}/< Cx(G(|O)>g,|/%W*_|o-w.#~O)?_+~o-?'_+/U:HG$eR.>K@:KT +"EvtCzJ/IzK+ ]dWM*d2JF+KPB*'J 2Q&d"SeLH̐:2Kd̑2OY(d,(ˤI +Y)d.Pp`&RBE; +GK u[tk!piAv^ÚU%B!|?֜57~@M03Fut`NG:NE +sx!omj.|n͗>; +UN' bUX&s%Ѱf +ʗkZNW(*ZkT+U-Thf[2F”3L%L;oLβ9+6F,$l.fD]anFڳJg̜f6lJV!SYbJcmsS:,p՜3+l>U æ=e3\憂ѰM'nRn!gw϶bp2k69_*2 j5kCz>4NsaWiJ>#U:>ьjVkj͘f:344Z?Z?Z?Z?Z?Z?~vn)nQ浟ϥ+Dk-m{oTw|2bV Gnǭl6+,gcs[/Zz.eMal +L{O8rOfR?m9ь{KU +c WQbi#^l"j)[lsmc5:TשfXT3eM?kX7pHSpf[nj6)o7-;a *KWՠTU Z.WUfk2km8 +'uRZ04.-seR\]U"Z"R%JɵFō6FuިTFтhSmmeXڴ+3:E, rP%nWkoW[Bm*)aY3gzƭ0 $v'd !%aF_hh@ۻ3;8WfCyHQ/)]|-ѳoTgմ'}v{;M =YbnɯVir|KWwSq:fOb:OǛ2r3CK14^ {hz?/Inn_}IO ᬝΪ~9w> 'ٌ8t_}'sO~? sO_ $wv2}/]:σ4gA| ~sdeOoϖc}4)~ +~/[_ |/p#7p9 $p#7p#aM>׮]h@hBkA[C{t@b}qg\=Wϸ z3o??;Rw_כ+7ͼ^&yc>m |fNPA#n'pP_/B}P_/K +'Z{t@'~7ebH=~O=E})џO?F4Z AW_|:||UUR*3Ԯl_1Ϊu f]R)śRWétQ5x=f݇}^H b uZx(959W蒞;7 +55CPc39Ѧ&VI`2LϗTu X:KJKPn\(CJP*tD4,]~'uh.h>{:Y(xD;uRu,ce>3qJG^*sNeΩK^*R*w*,R*R*R.KҺ..R.EbѺX.KسYa,ȳp^-Cf +56`;Nh;qN։:WtVQq}cklckt6M޺yjΦt6MgvvMg'XPt ا `0505x>tbyrBѦG3333xBe+^t```111x}}}gҧC0鬝xm鬪UbWUXU*vUŮbWU쪊]UaUŮUª)ZSXPlb7 aq.ő8?1e:kMG1#̈?fT?fYWAN#QH4%*jFe2D Q(C!e222222222222222222222*eTʨQ)RFJ2*eTʨQ+VFZ2jeʨQ+QFF2e4h(QFSni5 +FQQQ222222222222N}N}N}N}|sx{s~\~_~ߒvkvkvy}u<}<^/mvmvOÇ*Gɟfw~8}=.]IRc;5Sc;5Sc;5Sc;5Sc;5Sc;5Sc;5Sc;5}Fm/O]Շp[1mvmvmvmvmvmvu5{7l֨C=PA#r}6m[-~ o[-~ ow;~w;~{=~{=~??G#o7:逎Y}ԡ Mh(|_'mͬa:jZ^n?,O7c빽TƣMhnn<6yG|;!ji\v_ki_vsxkxu{w^|~v}vw=^_<|ݾ15O]FGۿFW_ G7ћVVzkzG|T(w0jlmzk[Φw6lݎ o {#?UwKz<@0]dEoO%\ zآX<$KlꮅT;Vq1t1}asQEu76MMMݝMWwWwWfRIW6Wu_}UfQE+cgj/m\ܨ:5"P7dx]7=gҳuægۥg=.=K'K'7:]:ɍkkflښM[њM[ik6mƑ7mͦٴ5Ѵ5fl\S]!\`\`a6l׆ڰ)ĮصvbvbNډNډ];k'v:k'vĮ3Yݳgu:/)ԵSSrZNrZN]˩k9uʩk9u-)ԵӠpZ8I?tҿ⮓~P_Ckh} A} 55Z_CkP_Ckh} 1 %00WuΧO*C?Ud+x董G%zQN>+W<_仒J+仒䵒J^+yep3 n7fp3g3 ~_/ ~_/ >b0,b0,b0,b0,Va >Lc04Lc04Lc04Lc>c0<c0<c0<1`y 1`y 1`ylx^`y 1`y 1`al3llllo8|cC e0PC e0PC e0PC ex```759%5'Y ˝%5[}k]Czǿ[cb<|AsKk9ĺg``kn%_Bn%+!^ o k]`kl}Bby_cgR}vv v X]]݈uy ޯr!^ b̻:3v ;[c%.w``kl#b3mork!3v u9%5gb;vv v wK]T/ K*o>_})SɧO%J>|*TQɣG%JsVpqq N<;w9NNN~:uӟM799NOOOO;w?_s\ɀ       v,KQ1 Ö- +V=fa&6=Ϻ QU#r#a+GX9šѼ4BJ7Nzҋ}ѻq;nǵP;nǵ3Q;nΩvNsS\;AvPTjU;A=pk{'/y| w~ݿ[gZBWZFi=Eh:3?3?3?3_ ~/_ ~// / / / ++++_+~ůW_+~ůWoooo~o 7~o 7[onzvۭ[o^몟$?I~$I'O$?I~$I'O$?I~$I'O$?I~$Ivov<~&M7o~;~''{➸'{➸'{^{^{^{^xxx88/ |/ }p_///q'q'q'q'rtoint}Zon)RN:.~/~29yT2Oл鼵UCTyќ/ ^Z/woi=E#or7δЅҍ6EŏPC! 0PC! 0ӳN=E0PC! 0P8[G!pQG![8RK!,RK!,RK!,RK!,RK!,RK!,RK!,RK!,RK!,R8[O!<SO!<[80TS!L0TS!L[UU8o!lV[!lV[!lVak}֙ zmv}0mfou~/~|o2MqO36sLLL ]J+h;7si1B-aZq6Fm1+;p}+LZdxJItI z.hqE7XtE7XnnF v[qz]ǣ.2 +nSTJ7m-VǦM:6hjw_fZ4ƏV+Əƫt W11Zhux2ݧU>^>Z?S<=շ1Sh}lcqSX}~V?S9f13l)c31z3㹆E̋ڷY7oFkۨa/z[mcu;WocuTWoկT?<>ՏhT_wS*Q7׏T?ޛ6oT?ޥgn]jOG}i؛iMnvs_Ӱ4a7 iMnCc 7O;;;;~w;~w;''''_(QjE.rtu4?;{{k~6F㬆YK;qw- }h_yvnOGQCr NuպtW+;xeFj˞I) ^FqnXݰa%`ikyVƈ%[kj^-===Q 'X&mv xWk/ɘ.*[櫨QEǫhCH@/&/Yk [kl/\h5ʚ ׸W Hl֛dkp+ǭKյ>W ȣ,.' pp 2 3ΐ6 aČx1-W>gisG-҂Gn枆ZFg[n͊v^딏NN<;IM/bf0Y:+3O+̯uk9YRx*}K+OKK Z{-Z[-h($9e+^m*bHQ_$?.ٯG/qpvUHK0=/Ef_pu?WQ%Qrb_(ffg".?fhf_f,N?Fwyd-;mG~@t z:IB} ? Aʝ"u.ǍfiC߃Mבˬ/5!7!#O,#kAR^SJZ_̇c_dju-bשFe:nj"K&D4bZ@r*)4~w3tM\8ulNΉr?C}WokBl VW ѤwBz,_:>M~],;N_­L_wh_anȈb|+2nR\y~s>GA6Q(r:=j*iwڜ82ˋz5),{mmQS_Ng+=(F:q,6ㆳ+)J@Pdo60*6 +:5 x$aDc$7v>1Uc9e@HcvB4 #HHKrFiZ0lX4U^ z6 lwe\Hڅ L)%r +Ta?d'szՄdDNI|>}"$UC@W/Sݚ/<>y(S.b'b!oi.-t/fqM%^m}FhwqK7wpwkH}Q[iDzr)\s0ٰI䚌?I+.rEI}Rn4&,c(5C*HSAЦM\A4E"*\nm?@?@ "]:tpsusqps*{.p^rL!z+act}~gYVAj +9QIuː=دg~*Qm05uD##c'E[iGvbrf^PXMTΘ͹X̹苡 +j,>eM<~2End4 I#mOKv9NUP-cЯ|,w`J˱4c(c[tg{Jm$sm؇Z+  rkMfZr>C'"e%.8-wd%8vC%kð Jn3d?  endstream endobj 29 0 obj <> endobj 40 0 obj <>stream +H XWǫ_% 3ㅠ( #D.adDT4qn(1jbL-E'|Z8Ҳ%8nqQqv?X-W~"Z2+S\9'xziM UX>wI{|Q8hst){vIbޜI%Х(t*$>Ќ9 %< rB0R ! + ]Yj>L! KЮ$3_>ҭWGKf>6{@!V"0Rq +R&>D@b0O@h}] -t tz 6ٔf]ML tt#!Μa΀ )8;AbhcaLa!>Pa'pWx$HET+I.3 ]a#}I_2[/}ltY5pT0['z6HיHߢwtG u?gAɬGf;KbAˬGޞ7;f^+=dr2#NߘVj|u.t.TйPNJ:Ju"K'B5"Ԥfщ:jщЖ~:t"4@'B["+`lFDhNZu"4X'BDh[ щЧCC9'ڊ +c|K8iljD:0"9NH$'ҙD8 p\[P ‰tDb8nH,'ҝDzp"=9xN$yIDD$s2يI|p>}9~sLie2syssDnDs"N$ʉ cDs"#8tNd$'2`DFs"c88NyNd<#2ȉdr"Y|Ldيp2LNwsA[[`*nw%y!0ڜ{%Z͓ +60Ur +6*d~\ pyB pO(]aTddrLn#4#SLPhUE +\9WTtRFE3f6*cT4Ϩ¨%FE +QB~iCH$kK?V;~?Z\qZX!I(ԺY -"Pd괡ꬡ1E7+<` zC50%@awk03O-D\AfD:o uP uP uP3E*w@{r 3\ѮC{=֒O|a%7YL*T这lRM$aw"MҺ2z>w=^Jb/va_eju[dY60qdS)zNZfb~i](eSwS9C쵐C$`u4UTP+qBֱUTfUKJD"6d>;a}(S(GtqS]؇=x"p_>j6럟 Z[CMf~ZY۴ y*&+vGxl׾CNt=GI;O~ |vsS 6|DQnj 3`Rܼ)Ei%Kg̜U6{ s_,W.ݫ_%/}co|kYUλ7mǭ;vӇ޻o寇||'ǎ8yp)Kd + +BL2l4X!iRJ[9l^ua?=dON318y[ ISS* JU&(eM;[ebOO//ovr8dGG'@G#=p +*Q-QC5RU"B].Tom^uZTϨkDg3řv: őYQ5ƸƤލ}Ӛ›Nƒj1B*seA<+ޓ|4Nқ6@.E[SU [2mdE(\&/ɷ +(%ZQRTK}ď tG'Z $vBjwjV[|hK-EQE}IHvjľ{jbD":~wzN3s{}WktYN"FRs`.EtYk;]]n]eK#. lVCKˬ6Ys)ߓ=hrNU3)Dvl~B$-Ira3̨n`׳}vǮm״kv}OB@^1a*!>kv>H09qL Qv l!q!qk₅[/=ndWf27tv9Hr؎;1h9ccQsnM٬p w sY,L777+MV~TaO$]`|8'm _7Ⲹ$b |+BWUCDexh^ CpZ# Ic8a Ƌb$"bib?PCOhY&!. X `5 Z.^/`3l ~v.-25 >P + +EHP Ʊa8GCI]RP@YpJdW⎸ UO8 g,p.oP*x%p" +*C +L޹p]^܇`Cxq#1<'$x9<L)d2^;^xo--Se:Љ`c&>>h#1>dLTLç Li8g ę̝ly300_c!A@1&Q~*`BT\b܄[p!էԐQcjBM5ԊZSjK=unzj[?$X?:+КiYV5ǚkͳ[ }D unԝzPOܤf6FDi mm+mvnC{iE~:@QtP\tHܤM1t1:N'$t9qK'n?<]]X^sSI7&ݢu R25Ft{AwԝGnXw2$!& UdFH5J67FUOj&Ijij +PXJw=tOK?џ:U~?ӟ/nGUTگ(uPRUQGQuLZR  Op*q:gSx![VLg%X'^+:7)qkY9\eؕݬvg;= Df  [/h+ʮeͳ;^h'e][H^} ܑd+Ò(G4I y %I#'%ERqpi۳|ȗ2_,p.% Ws +Eyr;sYW+y5_iZmLVϬAYaY"&lʕ5|-Y0Y!Yz?J4)?~gjq2%%e yNr9Os-zs=MS9ݝT՛X\GI?xޤ8﫣oscwE?Q3ԗ#qN/zS*H0TS *I4!Wr#w O"o!_*K~TSHTjK=uԉ:SJݨ; + +ԋzSK? 4Jh84FSKhOh*}Jh:͠4fKh>-(ZHh1- +ZIh5H(45MMW3L5KVs\5Ow&(P-RT-S +RRVSQmR*FmV[VMmW;NKV{TOqfʳ:.꒺8sUuM]W7MuKVwT*E42P=Rz9sii*C +ՠ:ԀP jC >4Ch +͠9 +ZCh =t :C +ݠ;  zC ? 0 +`80 +FC މ7)܃8C8{r/}/< Cx('$|%(bE$\/.G>%R@ +J!),ERBJJ))-eU]OS>?g9}^_%}Y_џod}WuNi:]g~/T?uR_Kg 2hc 19LN8Lny߸<k)h +¦)j⦄)iJҦbiva?0͸i13LyST4 0bzzbYrX+`EJX`UձZX`]6F`SlͱV`[lvN`Wݱa0`(^`_qA8PqQ80Xg?x8#q|.©OT1S`V}D5i4Ae@Q41Rc"B 5RI4QSO73h瞙sg㱼7&~S rmKۭUhVUi{ZOۯӚF;>>ii/fVCZ]];.Y!>ُ'' ☨E8!NShggQ]G>C}q)ȺMZչd +/e2χG$u \4le;|ƓUi ]ui4 r/A@b-)͓[o+ ԡm}g|Vw.uRyHy:,Vk:XX^q_|69x^@R._)Xclxu)I35TG,Yȥ\*vJ+;V+ڑK1`4H[t.;ȫx3!KR=:J!:,[̷ʀv]aM2l,x vtH-Kh#9z~ϣ,meoir58a98pR_/D~ܴFBn7hvYQ +;TЇ1|Z;?0Rաj.-W_Zcgnn?Vv7gZd-޶[ +6 |i"'!\LzimY;Cs}qó7:ݧVNivq;x zBb/2)Q!K$8%Ϥk#9YqOK4QH0}xd#Qۉ_t"&DTDT314Wybi"hMoRw^b J[jxI=(ZyAE{ebkD!MU3)ԗonV/Vm5d%1S\ +J*c2DJ ?\ve;i3/WJ 6^p"Wx:۪"q\]Y#)qhHLdݢȠﬕJi²6)m:uT)yʤMۺ[4KaM"{œC"#:"I2GX6AI#kfʅ3(^E@)lJl*sPM1ݗB~Xȥ b)+_؜1ɣG1<)a C =p@Sۧw=ݻuҹShǐiv[H2=1(c„8y=:?=e+SrLy2JGѨXe4CYn_J3ruV`o;`tV]f˓UK5RRe`L13ƼJv$s`" nF_SFfܮNgn\ɩ3&̎1% J5mm9~mT 鞘3yܦ)%_rwNuzmOu^*|+[[c~#C~U*oO<o] +O9`5SK 9֫+yH{<&.2CkVtyҾGr,#9/1(0:$qP$O"$,Ñʜ 71T7t{S uϹq|7/;o$qNJnJ%$kEEv. >sdG m.GJܟfr^#'y͂{ io[ۋۇil/og=Rg@=5mg"._ˍŭ}t9Q崮J\L>KרW5A)HC+~ڗJ'rnbi=K;J:g'\&U,x)57tjG|FU鎃paj1 S {)L-]߄ !ER R¹:1T8eU +B#:.30zR2^^0ZRUJ120idfUg~qkm^imF-0gQv  qѓ3.f.amD(H)sQ~!tR~>48|*}Nx@p?|6AժgE-*]pee| KY.V J[}#'w ?{/C6'!I|Fg{+rI …¢xaM+uaD܅a$Lz: hQTEmGI'1hoo5FE#F6;*j5F)OF|vlpQOel~}Ƕ}|Gʭږ<=rz9F.%Ouv +o֫UѼ~n|_W!w(Fjp +~c961׍7YH +B,RhX*)Ab=,vp]HK*9*ggXn*?TJ >¾R,f7HUB:lOOrhNM缜y*U9+^ROTf6?Oo&Ӓͯ)_\g(~7#໐=$;I*!tdBQcH'y( )AJ$^:fAu}?;/V rp꾋6.K g:չK;9+b}o܋O1ľOj vJQ3hEe`>v91cfy:}Okߌqv! (=} B16AjTs3{ [[ صSqM<`h({Ӏ5&yPk xP_c&vaJka|utwaIav٭e.:蟇y%k|Y9 Snv [CkRi_Px€}ȐB~ _㵰Dah5u3%r#*`ttj@AӋlsyLY.Y[yYo> PgC3)]0ϐQsقJYSq,O-Ŝnw:2:TXSGSF/筆 N nGZآǍd]1Ὤsq:IbB`>?2n||Ḃ ӰN!w5p=YhoQAnћF.?5=hp p|j }ιbK DTH CJ)E1 RA,PJQ@26($tlVN[T(RpW9p(7k}c}ַ}|; xX$^ [|g@|KƘgN@!!gߋĬ}VbKbƷ^l4ˍ/9%< J>[Qw'ii%4HbV֌MD珼OnmOS"j.7߉ V/~%@2Ūé4XޱUB%tg u÷AΘ>ZZ ƪ3;Zumi묱.7H +V[o`i똭:zArD[W kFe(Z#n.V9J;џlW4kܦr$T9xh&Y?^pozvZrhr4._@9v5>q(>_4ӨG0.Vr?oZ[q‘sxCA_f?Vp-w~ + x,N.Qi棁4>BtrU5=T:.>|ߒ&hc5̠w(oa*8砯 ER{5,^%9QMD?BՏK8ú [OKxڧ ObZ|,>To~g"Uq VA%W{q"w+U=G02;j|:7N!9h>)'l2^v_L3Xs=r[ +qάa?*JTy[pw]d}$7DC;$stZ edeMR[-RɋV{ej>OD#R +]C>pi.y~2'hD< x~pqci^h#^'RϠ@>שgY (ڭ{ݴ Ïwz_QZH iTIuOio@-Z4 SOatP +.R%ѯizX}hkjTjJ!/"qy} f"ͻ_or矜[0Hq7^ל R:oKt<[D{.Kxx +.ߍV[L!ߏ'̻gnvk5|;7rVq,{ho}k؉m +Xt5K@)d6x[Xs49Ϸn~^n>>Zp5' ¥g¥g"/#SR4Wj5;R5[pu/Os 1kJ +cU==h-Rkͨ[no,'ڹZg9qC}ZSXəfG\c8k9AU W;!v:dMFbb@$%Pry,pKs qkxiXS;s]V[p__}bu&uy{xPT+@4}dbʁSJcQ _1#5,uv ez rFܝ?PSx uJM{ȑ3Zw9"1vjb?iAN>s9JDJ|em>B\dk +̇)1Cac|# :(vuq\#.Hefd͈1F/_€R- EeF|zrLh[Z 0~ eÚa̲F +sv b~A?:~h܍QROqu;N7/;g߂7AMx$?j : +zOx+̀"Ww7 wd/*x_CWf? lCyAsӠ#y-W#݀Ȝbw?=_z;9XЍKs3;b9~;8? pЖ\0Qց/<x.[Ԏ7;hXֳ UR&o:Ph=W.Z[1>^rC,c=4[爎]BZ7)|El*OMä9Y,tEfŸPDԟ/r˫/\|3X^9]ϹW)F:f]io56|z'Z2ӽ3[rgwtoOkyw9؁o1tm@=0D|ֹMSW3q; y DϖVd-9~;A{)VsG[K_׶*׍ٚb=m\3u-B~v̈́5w= ktyB׶s1Ƌ/+~pM;j<o02/$C}=P%*t|wņZLwi+gȽ@#''Q1v%}ϸhpo/t >4t5!t> +Tus]qD.M^MoOChPߓ6}漡 Q5 '?ae'͝G 3[s.( ypJ +3EFȱܐ6J+ kBf7 s'rnf OS >{M^>*|O:_w=;Gy{ـ ҁn2,ZT8&`8CLٶdҬ[i߯x:YjT-a@o5uD':8[M}~ ߣ|CG7f0u=)Iezn7^Bn|aoXT'[v;P+rY8A34p/X we]S*wAT˘ k92h+g-z[Ý,Π,jIXy`=z@oeTxuOρ?oC8/^^PD^rgCsN_i?QG /sYV*{/`# @ @ @ @ @"\#lH"=\<=\¯C7Q~[t<1)J;40ܥhK4aο:kο:3O1Fr #OO?=2p#˅GyFERVSS w)38~jO?un8 +OOazR嫄#/8i#NZw᭹|s|mmn[p63h8N1':y8)NW4|4|4D3Q7̀,Kv8R PN/`e'Oe7i2c$|QS!l!J(SoΔR!PCbʿލԍժݛFIMcg7Fef:Pz퓢5S6ڛbyK +Pߕ$կiZ"ux*6^KBr ]$BLbORܯb)ݹ&Li{r<Ԭ)ïzZ˧!F*2@ ,Cd 2BF(-cd'(dL2M2Sfd̕yP>cD>ϐB]bRY*d|++dղF)elD*ͲE6.;dݲGr@!9,GH#'䤜Q蜜 rQ.e"W_˯7BZCz-wܓ@_< ȈLȌ,ȊlȎȉ\xyQ +Uk(xo$J!QeQQ&*2*:j&j6.>-4D#4F4E34G D+FE;Gw=`Cp@8z"&z!ND 7<‡>A_ĢcbcbcFbFc b]11 1S1 131 `6`.a>`!>GS|a1K| b[JjwXuX ؈ [ ۱; {~>888'pp .. +W[O=<_O(I!LLLôL¬Wyd>2 Rg!f,X%d_,R eiaYcyV`EJ*jZ:z| ولMٌق-ينmَفo#;3ve7vgF;4Ξ^Qtb4{Mcؗ¡ñ|}NDNdNTNtΐٜY9s9 !?3~/_+~o˸rWrWs ZznF~M-m]=#q?'!1<_O4ѠQӴ-RKhitZz-QˤeֲhYlZv-?ŗmֲm۶m-kqi-۶mݞ{~ALBlA\C|$@B$Bb$AR$Cr@JBjAZCzd@FdBfdAVdCv@Noȍ<ȋ|ȏ( +OFE1G D)FE9GTD%TFTE5TG D-FE=G4D#4F4E34G D+FE;GtD'tFtE7tGD/FE?  0 (8L$LL4L ,ż_XEX%XeXXkǫkה7`#6a3:_{ކ؁؅؃؇88888888󸀋˸븁۸xxxxxxxx ÈaQ1׸1>0!110)19S0%S150-1=30#3130+1;s0'sZZna^c~`ϭЯfe1g d)fe9gVd%VfVe5Vg d-fe=g6d#6f6e36g d+fe;gvd'vfve7vgd/fe? 0(8N$NN4N ,<\\\¥\\k븞p7r7s Vm]=}>닾꛾~:C:aZvu8wGt$GvGu4Gw t,vu>>>>>>>˾⫾᛾۾㻾~~~~~~~~A d*@@  A b)D тA f+ A a?&;*J~2{ *$--Êp0 }MBF( NL'p]s_>hS46h3g,耎 APsg؋W؇[|~OW_ &,`!=$"6e36g Cy !>̅\\¥\|1>'$Wp%*>g,j>5|/%W*_|o-w[.#n6n.~O)?^~/%>;~#7?'G`5Q +W%ZtБ:JGtNЉ:I'Z֩:M ә:Kg:|+buRP!uU7u׹:OTe*WUR=KGJ.R?]Kt.!aBWj,U+lըVQ4BurTJjRJ+4J5Fc5N5AWi5VzMdM Q7fݢ[uvݡiS5Cwifnݣٚ{5W~=5Oia-"--2-#zTq='B+Vi=gVy zQ/eZW^zSomڠڤڢwPiivhvic}Os}/iF6ʂɘ1<PaE6hL*|~@ Pǀaq~K+YIFoiunV(M&v,a+{#y&>TmTvQe븨r7< +"L[}㸋*~OgwE)]mw3[ +{LAX֓m\]E({}Ч̰Ӫ<v*y,'<X阓랎[+[ƚg;'Rvڑ=:X(JoNJ&|M噊F3a$1*FՉLTF7?NET&,p&7Gfc cö'ȦwvFqmr2LG}>ÀaKw2lhf?`b/+l {kh6.t2g^v՞m,"1>4ϦͰ-sܤw&MV2ΤɨF;+u\>~sK͵k+,k۶Yʸ9 F4ѸZHZWq~}IEct!88{OVNz?zy^Uv'6Ic6i{KXڳ.ޔYAe[}߭oOۻ,'TT6UuvMz"ʪ*o7XU[ l}RvrJ-P g?-Jy\LiODDiWdߓ*eߧWLtD.ZY??]u;6=yxƤͬgwǻzmf7)۴JW}ݗ'>i&5r*ZC=ڢЈI +W +\+p2W +\+= +_೵VjW +_+|W +_W+ +~Ϧk* +~װ +~oo7 |7 |pk5.N`j5n [ío"~$ٛn}1Ep,cX"8Ep,5G@/(.X<<g."E/f"n+bl۾p+e嵮]/s/"/r"eE-bW9wSz)ExY-Emg|my+r"{zWsܾ_;?[W"E8E慸)b[Įλ"^~W|my} /s]^9n+⦈ٮ$̝",I66 A '@ *ǟoSKUr;~|=_{;GLnmv^vov~^oy}{ov~6{^~v~gt.=w~zi襡^7Ӄ&=镣B }4GC }4GC }4GC }4GC_/ }4GC }4Gc?C 3?C 3o 7~o 7~sC =7sC =7sC =7sC =7cC =5BO=!BO=!^i|͵W+W^zEW^zEW^V"+B///¯zmݹ;nws~7nwsݦuƟg|߇fƟ}_ڌ?Ϗ_o_?n>},Ygb={,Xw5w5w5o5o7{nܼ^i3o3of~vvvvv_n~sѺ>c# hB_6Ӄ&=_qWד= }]q=ݓWw]q=9ݓ=9ݓ7\OPuOPuOPuOPuOP w=$Ll?<<|Kwovnu:БmӉt+Nu?????#?#?#?#~~~~~~~~~o 7~o 7~w;~w;?O?O?O?O?3?3?3?3bbbbbbbbbbzV׳gu=++CWů_*~UUWůФ'UWů_*TSOş?*~TQGŏ?*~TQ/Gŏ?*~TQGŏ?*~TQGŏ?*~TQ?)SM7o)~S|K\'O?)~RI'O?)~RI'O?)~RI'O?)~R ނ-x ނo)~SMZ^?c~}8?:<>oxpk~xT?:V`.0{W ~*rn߹_p~}m/[(nsr?\qWtyӉoȜ"s)2Ȝ"s)2Ȝ"s)2Ȝ"s)2Ȝ3/B"t)B"t)B"t)B"t)B"t֫56 +;];x`Xs="~J 5+3a9u sOu>}| )KۋO.me= FQsQ}Y\d$J.2'J9> +KGdT 1۹lNWmRĸ[tiR<s~SPf,8JQ+cҺmV;77Y#Oӡf9sȆ܈Ow֧sТH:J`ٗ YTcO0 d٩LD +τo=?鴝gfo*KLjWWIIIm*V#l׊޻б^6+ SVz +OzVXI"J!6 BT qJ Q_I~ +OguCŚ,,ii1$~;^)mV-I %<~3aeO>I+8R^4[=$.?bգc+^5z+{x\+~HC@lH;\|ǾfokꓤW`3ex;0,x_걔GvK>ظ>J %0}wX6%s <Ɇ4[gLG'$BOeH-ȁH%Dr͑MHOdE3.Gۢh8*Q]Qov2+1hCҏ)FcUC]VDf*HcT{&g)JV%$?MmnKH +.Ɓ*_S{Nbn4 /1|[`*YX|s˲8~l3?̿~kݯU&&sk)j:;9W^(y嘀<Wwq:Ϋ 1>j$!hpkϧ1 A;K1zV 4+Y$'tLKtIK]ɹ$(FaWwq?|\p?ls~3((ʎ()VK{}\;EeGOt_ K2U-Utdl,}>nw0{/v_ rAoa C + +"Yp/>Ib( 8n~x ?#9z#OTѝNz]W3Ay~Rq!&{&vv @M`}ڥl@J sAh3t{dVmc 5m?07n2)d,hČbAD endstream endobj 44 0 obj [/Indexed/DeviceRGB 255 45 0 R] endobj 45 0 obj <>stream +8;X]O>EqN@%''O_@%e@?J;%+8(9e>X=MR6S?i^YgA3=].HDXF.R$lIL@"pJ+EP(%0 +b]6ajmNZn*!='OQZeQ^Y*,=]?C.B+\Ulg9dhD*"iC[;*=3`oP1[!S^)?1)IZ4dup` +E1r!/,*0[*9.aFIR2&b-C#soRZ7Dl%MLY\.?d>Mn +6%Q2oYfNRF$$+ON<+]RUJmC0InDZ4OTs0S!saG>GGKUlQ*Q?45:CI&4J'_2j$XKrcYp0n+Xl_nU*O( +l[$6Nn+Z_Nq0]s7hs]`XX1nZ8&94a\~> endstream endobj 6 0 obj <> endobj 7 0 obj <> endobj 8 0 obj <>stream +%!PS-Adobe-3.0 %%Creator: Adobe Illustrator(R) 13.0 %%AI8_CreatorVersion: 13.0.0 %%For: (pmehta) () %%Title: (RBM.ai) %%CreationDate: 2/16/18 10:55 AM %%BoundingBox: 174 237 697 389 %%HiResBoundingBox: 174.6948 237.6172 696.6953 388.9453 %%DocumentProcessColors: Cyan Magenta Black %AI5_FileFormat 9.0 %AI12_BuildNumber: 406 %AI3_ColorUsage: Color %AI7_ImageSettings: 0 %%CMYKProcessColor: 1 1 1 1 ([Registration]) %AI3_TemplateBox: 396.5 305.5 396.5 305.5 %AI3_TileBox: 108 -50 684 684 %AI3_DocumentPreview: None %AI5_ArtSize: 460 180 %AI5_RulerUnits: 2 %AI9_ColorModel: 2 %AI5_ArtFlags: 0 0 0 1 0 0 1 0 0 %AI5_TargetResolution: 800 %AI5_NumLayers: 1 %AI9_OpenToView: -46 612 1.31 1162 814 18 0 0 50 76 1 0 1 1 1 0 1 %AI5_OpenViewLayers: 7 %%PageOrigin:94 -376 %AI7_GridSettings: 72 8 72 8 1 0 0.8 0.8 0.8 0.9 0.9 0.9 %AI9_Flatten: 1 %AI12_CMSettings: 00.MS %%EndComments endstream endobj 18 0 obj <>stream +HWْ TPAT +vq[1}~ ݻ9Jr\2!0P̖ZptqqYbRw`+fS$'ϛ*E^v')ciʆ&wl"򗈠̣tn3\ޙv?'^_ 45|b-C[\ +\_@|/S-O3Džk#FL 0F+՞ 2%tqP|y퀢k8z*PGq@XGB݉BuaH~Pv~(}Owv%VvjUڳCєX( +7RwTT]~>;KU]gWn-+asNDBeLGBV6^6WG2:~ .\t(JBTڋ8kPAUn@( #1)!=#zfDDɹ0e +ĵއ"iMP͏s";\L6Zin&j@M ;7㢎-bL`Wƣ_$`@=n=z\ v .Wb\Я^L%yr{5S-o_A‘Vl!_9"!%{ywHm%pr\,B n@J,ł +_A*s] ɍSowa|fl@ZY+?cQewu[e8Bm -E&fĊj3 aP`D!Kcr/q?ܢ#=HO.I_!`6S''ZEtdpƅjump+YV -pi~a +jDe^ay](s9)zYv=[JoX*ɟ9HbLP O((jn{zZJU"bb8H"({i7,@ OT3cnH|rHS|u˰9w=;4B4!IūEy[?gw)Ӯ}lpMKe,hLeR%ܰGK L8hIMy p?n<,"=SO(<Aԯ.?euhV:uNR&ܰI,=\S%; h4H}T4c?y478fu8 bv3 =tO)Vh2|=G, K4oCɕDa){GL3oaްlbyR0*Mpspv~}w|syVO1 6IJފv6&ϬY@,QoQh wyJE\.̳G6:resq=,>\m?ם+j;O(hT5^}ik3)g;(+!xXq)F >hoL]i! bv$˵t&!(qi酶I=9orLm,EOUþj_ GQчRxNd(6YgܕPbTh)g'w^XTآ}?1#|1uuH`QC9RPY ~sQ/LexXP<Ͼ:$kn;6-PLm`j&ձEº_iˏ7(=OG;7}ȋgp?DiM J&ϹA%E$|5V4)W*5i=}3=&r,A*_u-|u8Wlha +|"K_=kNUF~7YNG4Ѩ?vI,:}JAOO5u +) +hVZJ7#_=N߫-,hID,U{+C +2Qt򻻨i p:M1;$"BѰSy{jyzc3oFF/m'zMR|{xh#2L&(gq +"3SO2Wj-,CC\AA҄g5lCPAaE{!ߺ7(0TjF{$^Bɶ`p9:,rZZٛJz]&7CfVK}9?%";Yt gЖgJp& + + 8KEKlU '[@p +$k;PiZ.PrDpSq;:ird5diW1@Jٴj7>A%Fgy-Zu NE`ݖ"=Nڵ_MC-p˂mL x$n +m +aKȪ͟/oj;; t5( Shԇ*1~Kge(ӷg,T : 58PSnufu!z\uZ\3YP=WiˍlXxs~W4`o섽^$!- MB姀*mPR< +.mSy;/9ߘ ޼ua:bCBqңtIS[cxnUU-ӯ*vgx-Y>RO.ܖn)NM]w-پ+oX~E4-Y`wM2fݻ-nXx8?wLO=Ƴ-]OipXSMwZ(KJc=`kGɸOAmb9Z +鯼C^[i+Q I@X QJ-PGZVA?3Lfg&{l֏I3)ikha1 +D0^PHXH8=a`O :& obZ-".|WK8o'I_>yD,0O϶iuG>D R"e夀hp`.7!<]@t}ˌԇ;~eNaWRW}oR}4Gu@j{[83g-ul Q-͊qYi~Jmr"sKVe+V`vßC;QqqiJJs´qaiRդ37+qJ6ك!C{i0;{HjX_pcCQ8tT{sP +2KJ*]}gj@Vzdwl*]+ܤ Wϛ7" 2^*t=yo=/auO?^AA,һ5[ +#;Ё(Q5Iů}zQKqd#*/ +zv$jSp0br2_MSKQ=l48vdҝ}0Q>,_D)+pNf29wh>2R`ٚ༜\Eu[`Js h01 8M$5U1d+h(h-А + +&sJ⹴죹pQtffǤq`%BlXԞ[,8͜H؁eXcQ<9m!wVP7gw*TNxf{\o/:~͊$2.͢`V*`V:-U" <5?h=ݱ8sN!0]b8,(*$ p(]j8N8^ lj+F'OY Q0i&W[,/)җVps'Z[Q1g]aT Ł[l +\Zı85BJVb_U Vr*Rr*iTn?&n>_%2*(fd14rzэ);JVr(cK4ɞtC_xyPJ +2dX9YBZܕ +G Jlǝ,9m +PÈP?`mӄ 8/*S$8f_Tq'wF_؝6K +HiZ(w#4=^3B>7|>?~tƣZݮ~Y_Nà>~؟rBofPo3/؝MكYd5^rNjdwV|YȟO;O~4Jr^<899z̜{ݢTѳ{iP???G N+aZ1]ģ1a$ɣcYNV:&RvGwV O'/7GPV(bK/EQ^q5y|G}Ժk~r}-G*sD +cP(wwWն1M)^U겳ufboo`J+)kk^DȜ*a-s||I ]zFLnzR +L)p+[b0:|YHT:} +nr-$M>ɜ\ +L64&SdL f۩rS4=VRƺx}fls=m?sGa^dwTRvU0=1SiB,.ZIi$)Rd6 Ha2uU ͭKTh,+S&Xr~LQmԘ4U8CMW7/N A悺7N\EWFwWb(QB~+)LJ p.RcJ|d%U>}u/WH:w7K*=]*` !^я=]rgXģ>)5hǞ.K^q/] +Ԗo Rt3E*nV)t(}קbqC5%J,qwqnvlܱe\8~%9Mn92LRUt ޡ}g\A4`i:* %x )Lƣ.XJ*RW[)zw܏I;g@n^ڂ&Vc8+pͿa!tPd,=Tgdr%2̒G.C.#6*\-J)_qݝd6X+)~%K6ޚ3,oZHJW?\5C*'nLR7)>79_>UD̑&g^w2ZRζ?R#' bh  +tvpf`ک>6eO6}8XW'OҒqb@rJl cf:M~^ßY^BgXVu:O _H@ 8ci]hÎoeYQ0mGpň1j/8*[iCh PKhz $אZB 畔 pY`nHJb (:6{Yk^Tvʁ ^8ZXJN].ie0nP¿0LbU%p0p_F){0 +p(8<{؝"ͱhzל7ڷ'C-^9X*'|pЖ0% &RJ^@+؋oM?[rjJܛC yh.ۅ hX#IsBDO.N*&#":{T'&x8cU$zy.(ҭ [c!қjf,(7`{eVa%xS"g\eځml~!Ԫ#fr<G(ރY+jk-o^òbdLk!&:NC)0ø]JԗoV1$SDX[I}K"L\ +.H` +ROyg4LB+BPԿ}vR>v,+eeEqdh!01{0uE4:^ުM~5Z~9w5,oMLхkq )Sn55SzK,n콆Ȋ;_RV,,RV]CKTYH?a Qw uE#@hB".%&2FxB/sfR5Q}G%Vjr+?qWx8hKߕNGe +@aOb"$!±r[~>KVnP|Os9%hI*F7/˺Xb_Ʋ3)EOku/c*~$˛H z9KPe90=`HiO^j|3XD4*cJUOz9yϥ/^U|~3Ӻi` #ŢܧL5cM˄?Vq%HOz&d⼻6ͳ@8sis%sBq=K}8iZ{4u LRB䚼{: xNqǠcnPHs.ϱ i[iI4[_W)ȅx;-oSE#G\Ըi=uj. V|=]lܣOv#u# 9WZJ|"H $ + +AgfrLf~_;Uwu35oUuaYݱm9H1ZPlk :(o:f@EJxj"nW 14hּ['%SR ~Tݷ^>з҆ )]NjfSlj83f)b+ Rb8OFJyųtg\_I{)h)A(K&IFu(mR=iC4pI/үX &_vb][#[{KcgbҰ9{Vh+n'\V71 ճ:=q!M[~o4˭oJU|W{HA|P'Ԧb0LX[ ɿZ$dP`‹W29[jޟa^KOXۋxߋ(I._+ۀ΂pgd^3:["E#0*5'\vd(noSrԻo[KGqqC?3ƾvքoȽ6bI6Gͅ7%qJFoAUJ]AZSE}awP֕١àk~ɦ,9KF(P1\N+⯾@3sQH5z^z1>X{e3 k,Uԇl%L>zR1"ug"渱X-~؀jKz96"dvƑ\NAӤP?-X&;ވm ހMNMAVAIYT=9di*z ETO6[TcԾ9,CN8 k 292(OqwΤR hߗ}Hf <6s(N8(cMq#Pr+]'_"'-%ܧ*?'BD0SR9.閩lmv[-qWwҝh~Erϼ _ZEAEepkXA0CTB]M2:QZR`WG8}u +QU {n߃ +JɁO1mYr +2\ + 1S\M9,YIFfUf +U,4gaj`O/8;I:jeO\"`/^J/Q3K/~>uQCws 5'txiu8fZ\0xc`$/0'WAgV8^}bZ#8tL={k_Ha3V2g xxǏDQ G^`'1Gʼncջ- +!`zzOS#7ѩ!էeη ,/gg<c7w饽XFoŊWŨ +Cn E%l`gG21Ąӵ;鯎 +nA2ޚYS\.m'N#eDS eZ;΂0"PT֐XB+WG8 [ ZCSɱejz+q9~ s^ +Kڣ``PH,`GmwoWNJ7eHEFI{sl#ew!t`OՏ.u"}j` {z|hvx(?/Ǜ + y*^1>bq"yA촤Ftu/Ȭp:Yw\iG:zد#-!WT5زy3 hʴv }j&Q/"0B}55Xr+[iR;,1J( +zxH\=K{0)sLax0^K]1[ihmdcCL8D\>¸8ǀ @ {[tB}>xV2 N$F-|]i~8~h8].dO%M6uqq +1C?"ils^ ?^e\^*6ɮSL~\,EI[)ͣiT{(g],?b9߳wk!>6)+v63+nN|voMKo&ډ8N!<Ȳ,ț2#?]y76 ?,mϷ}t}B~]f|\,x7k +'_wɔ~2B:$.J6 w[zzE k-9l6\c^~RO>™t`Z_e T WXbi_U,77'/&zS]K@F(u/Fz"b7 &|Ɩ8[?j&h,1)reon*~6M9.zx"^ Y endstream endobj 19 0 obj <>stream +HW}_L~TeTT4_Ҭ|I+3+3sN}cڙfvfB[T"Iox!RGϋrXj\NQu<.dVғjRvOO:]mD0ZN?1Ėx1 "/~n$Ar!NqgL8gaIc+]_3?Mf񃀔n+@ g6_ved) n% nʖ`)B;|tSL.J +-TbG)ul6RcŤ l°3ؘM4ԺXi%8? Q)(k̓c&) &^F\$Wsċd\FRK-֫]$gOMo@IT(SV]\<'ϟ7PH$RY濪B^'8pػ` Y;N%{O(>rR`6YmF{XM5hx =Y?fCuk©1Nx*2NiUMVѿF Snp"z̵GVu^1z~WLFEj(8#yn*\Y" -%9Vsy6HXGdE!퉨qm.98eȈZQ&b>8 Uy@ ]6~fna-^b´C‚@S3Z}F92}9(ݥQv5S_RwEUOoA}W1r!aR.Y#Bzb} uâ+o70 sV^b0-`)8Հ&.)"CkHv,ֽ"Cuj! ڕbۍi7ws+P5(bԶ/GkMw4xe-ZT4mD[4$0(#h mKbxCءzx8p xqX8I([Kr *.xoΨ>NrA9,5*LoC8v:6*8>y= iB :4 pʯDI9nB 'aTf7G:{ !ic1t%0 e[Cns א 򷈑ݻufG7-qUZcR/y"fs_~0$B#?6:!ͷ$~E|1W"~E>0vlNٯrRii w;ݼ{ą8&odO{F#͕òSI3y@,U)q{wt6;&,j*?xBd f\~"f -9qh\0nb;yv#l+L/K*\:\u(vKc7bj|ؔ$N]v*E3{d}UjhB5pO u=f۴u0MmDG`Rr \I/8tJGtBRxtd`AГ td?ޫt-qf _ td {B"" +( sY 233Nz7dϹc wv=]@"[@SQɗ\st@8_1U,8VtrŠ៥XG>~q.{X:9|SNaɁ_d0Y].INg KY-KW1HI gޫ0 ͵zQpPC@sƦ?D-^d0sp^ g6K&^@o™6xvyTze%q&>*/=E ‘R^|D,%Ym[bS<.,zI4Vr-x +虍GaKtp'$Ṳlq:OƔCQo F4IdxyQІU:>8|$wx6PFҒHEXܿ)QI!I$&=޼0{m)[;n/pP +I) 3up:f'-ʞYTK=۾fI2)2g΅`ǣAx%Sꐽag ȷ(r'/lD?$ҍ̎|HgYN~“扶k7],v@c~Z$>և*<$~[P`,$]٭IJ5aL׎}mȕ hUW֝zGfV|xNř_kRQ[ikB0ko5V Q8mj}U؋3_n 3@S(ƭXh_xym$*vFt8:Azn[0JiG1_6#irP;7czZu"G~VBSG{^[}%QbRL i_s.w,|AhSY0cF1=+АGoZk } w5ݹV֬ή2U/7Чλݖl_ +M}NF#sIinY4 +K)hK`vTpɜJU/?wIBȥ%7fCp3$(8`t}@y#+,-\0Y'!<1m(uY`vYr$3|z;m(Vl|4ajZo4E{GaJ>G,ØƸa6V}u(Hgѡ]1DG{齌ܬtl^V' ›Ž!9hgM]bЛ\7uSV8ɻ Ďr##\R4֙}{T%UWj8%9})lk`Q-n ƽy&Tg"fimc F[E![]>Bw}O:t[oˮg)hdK h?򛻸ZrfJ0up~޵0&XsF'k9Zg׸s +Jp)h\dѤ99 O-=1bê^ fwbL#E,bKT% Tmf42jVlV,t/'U/8Y4XS~hU4uf0׷Jj VDvРvd1F-fODCM`y[Y͞FvA٪Z=ͣ.KbN(2lv" SZN+Dޕ`+¼y]c)C,+>M4yyv6oPb),? uiSC+Y)ef]< +adž2֐cAv#Nݢߤ;lyF.pJW8-br̙}j>:Bfa`!◝ nn|_tBa]z*8pI>h]nU!±=~…CrE\@p E(8Z"0~/Y:A¦xq6NGۚ, QN@I{͚eDV3Qkbz-V1qY#(TU)aV. aaq;oj5k%p1z rދ:}^$mlw.&*Ns~zͮ+Km\g' +FJ ^䀢OKDIUI&[;Nce]~0loxGAR B >] 0m$43o؋{env5dlf'-+28PW JL;P2g";kdO6q]&#/, 0.AU~1l|!̳: * kEu+,_w^7(q0YhCP`9*D8Fߏ!,lba"Cc l}ޚ@0 WqǢomwK Ȗ2:H3^fGA7p̾&]V{yI""bM* ՃkRfS)0C%onpc(v\UJ}k 1Qy߲\KbuV[]/ +-G`O]96M\BtFrʴmp;@Yb 0 eߘr(Q$C'Cc+ݼބtEo͸xK$L/cF (`GϾ_}M%((IHn_#ufRr}N:Vr{i3Mgˊ#g_} H5a:2Eu)`7]p֌kEݞX¿$CVXYˇ5hYGB Ҝ j[hj꓀rdzSa}X!s%.[},0dU.;el9X4fJ.;B s۠鲛q-fkr__3_mGlT?DlD>,RWϋoW.,j*_ى;#r +1a; +ȕ&7 ^.a9f$RiL>abODHL,kO`o8ܷ\Q`2`K9NTw*y00߇XYK4"  Іuvla_y|roD8In8i= jhTEg„l7o>7}m$j"U"8DfvUȊ" @,hnN?4hWNzklD*q|]SSk:h*?O {vTCNeH,ᓨ+ U.! EHuXc8/cnBv[8^*cl/YǒM.wFHڪ1)i,z8w ΕI,?4Qz̚0`6(}Wq.vgCk r 07XX6RQo1\ޣEys%Gl_$L戙.~%c\L}g]u*PïC2Pձ7TcD% +{}_H@wդ0hgyh&@2he 44wy0%^Q0 t >t$bˆŒq.]Zm%GHgIw vaKV3+NKTV-zBn29zI K)@a;8[A%*y.l`,@QB+"B!ڟ_M^,:p +Vpp?Kvg?ݏԯF*W9OgdB AݘеTTCA{Ǐ^.{R xK+-ǚk&j9QOڗzBQuZ]WDD"xM +dK+ +x%v6ن/l]N㸌 k9SW4 6|wSl(z+ ڿÕj)đݝR\ E *nH&D1G(\c&шGvv0,Y]Hr%1+(yarT[ ˒WdH%e%8C0Č\%c [* @?GK^x<ӜޜՈD@oe6R @$;pg >ǁb~g1CIėQ{б5 +A.MrbrM?],`/Rw.+N3{<#LuөzM ucZ5TZ֬i띟k43Fk1f@e j%[6T8$lQM0uߤ=r;gX/EsPKA\Zrē%]Fg1v[>$Lc%.b=]I +$QD%VyRsXN9+"vg[olThe \^HdW4iq/ 8'$U6n/W߲r=7S"9x.8g4ZD8'~~bU #\`o6 kVwfk]O{wDO0e0I&ԶJbuEZ% + +{df=xا.5]Nb$vU`\a:G?,~c9.fmñWÀ9o>>P.}<ē,{(e &ZlҾxo{/6,Իtq[Lַvy䏗9eGT\zӗy:ߝd<9h.fEL/.0MS$UoWopX̚͡4o6m7@GnD Fq\ 7 SP;ۓ:9i~+,zs +%+%؋dŸPf־Zd9ci (>KD䒣WgbrIگjr?bd$kZ|]KSʖ\hɍ"7lɱ/,-z¼%YД%5\XlYSy6ށ&̕ ֋6/!Vو@ƻV|[x ɒݴPKw<2o`bUѮy} +~CߗOwN>pB[eYck.fߪFw|QLO-~j'ulm\/eTʮ7 +fbח+hS<UڰZo"Dnu/{c40,QO?$uh(q,ʤ3`zy>ڃG*Y 1ERlh 0T&pN ÖA +>0-|1(^d-&6?!mr|JI L/dyh5暔!qԢRy&e^Hdu'r+bGQ.2Gy|E!'( +K! '(C+A+{ε4sz+U V؜]2 {7 IL2hz% z>NԾ6z'<[h4h}~cxL\\hSip`v8NhZ̕L)"- 7=?=zPp"i6"m[i6iȜ-eKJ9PڈU~(hN4=iHS d4iNKs06X@d]1)1Owj+g"S2 @)|ڼRL,qx)rSM˞",WfW6Zh?FyXmwA:a߈{=\- kD+)X\۩. endstream endobj 9 0 obj <>stream +%%BoundingBox: 174 237 697 389 %%HiResBoundingBox: 174.6948 237.6172 696.6953 388.9453 %AI7_Thumbnail: 128 40 8 %%BeginData: 5498 Hex Bytes %0000330000660000990000CC0033000033330033660033990033CC0033FF %0066000066330066660066990066CC0066FF009900009933009966009999 %0099CC0099FF00CC0000CC3300CC6600CC9900CCCC00CCFF00FF3300FF66 %00FF9900FFCC3300003300333300663300993300CC3300FF333300333333 %3333663333993333CC3333FF3366003366333366663366993366CC3366FF %3399003399333399663399993399CC3399FF33CC0033CC3333CC6633CC99 %33CCCC33CCFF33FF0033FF3333FF6633FF9933FFCC33FFFF660000660033 %6600666600996600CC6600FF6633006633336633666633996633CC6633FF %6666006666336666666666996666CC6666FF669900669933669966669999 %6699CC6699FF66CC0066CC3366CC6666CC9966CCCC66CCFF66FF0066FF33 %66FF6666FF9966FFCC66FFFF9900009900339900669900999900CC9900FF %9933009933339933669933999933CC9933FF996600996633996666996699 %9966CC9966FF9999009999339999669999999999CC9999FF99CC0099CC33 %99CC6699CC9999CCCC99CCFF99FF0099FF3399FF6699FF9999FFCC99FFFF %CC0000CC0033CC0066CC0099CC00CCCC00FFCC3300CC3333CC3366CC3399 %CC33CCCC33FFCC6600CC6633CC6666CC6699CC66CCCC66FFCC9900CC9933 %CC9966CC9999CC99CCCC99FFCCCC00CCCC33CCCC66CCCC99CCCCCCCCCCFF %CCFF00CCFF33CCFF66CCFF99CCFFCCCCFFFFFF0033FF0066FF0099FF00CC %FF3300FF3333FF3366FF3399FF33CCFF33FFFF6600FF6633FF6666FF6699 %FF66CCFF66FFFF9900FF9933FF9966FF9999FF99CCFF99FFFFCC00FFCC33 %FFCC66FFCC99FFCCCCFFCCFFFFFF33FFFF66FFFF99FFFFCC110000001100 %000011111111220000002200000022222222440000004400000044444444 %550000005500000055555555770000007700000077777777880000008800 %000088888888AA000000AA000000AAAAAAAABB000000BB000000BBBBBBBB %DD000000DD000000DDDDDDDDEE000000EE000000EEEEEEEE0000000000FF %00FF0000FFFFFF0000FF00FFFFFF00FFFFFF %524C45FDFCFFFDFCFFFD0BFFA8FF7DFFFFA8FD06FFA8FD70FF527D527D7D %527D7D52847D7E52A8A87DFF7D7DA8FD057DFD0FFFA853297EFD08FF5329 %7EFD07FFA854297EFD08FF53537EFD0EFF7DA8FD05FF527DFD05FF7DA8FD %15FF7D7D52527D7D527D7D527D527D7DA87DFF52525253527D52FD10FF53 %2929297EFD06FFFD04297EFD06FF532929297EFD06FFFD042978FD0DFF52 %A8FD04FF7DA852A8FD04FFA87DFD15FF7DFF53FD057D52FD047DA8A85252 %7D7D7D527D7D7DA8FD0FFF2829282953FD05FFA82906292878FD06FF2829 %062953FD05FFA82928290654FD0CFFA85252A8FFFFA87DFF7D52A8FD04FF %52FD28FF7DFD14FFFD04297EFD06FFFD042953FD06FFFD042978FD06FFFD %042953FD0CFFA87DA87DFFFFFF7DFF7DA87DFD04FF7DA8FD26FFA8A8FD14 %FF53292953A8FD06FF7E282929A9FD06FF7E292853A8FD06FF7E292929A9 %FD0CFFA87D7DA87D7DA87D7D52527D7DA87DFF7DFD3EFF7D527DFD08FF7D %28A8FD08FF5253FD08FF7D5253FD0FFFA8FF847D52A8A87DA8FFA8FD047D %FD3FFF7D525252A8A8FD04FFA8527D527DFD05FFA85252527DFD05FFA852 %52527DFD11FFA852A87DFFA8FFFFFF52A87DFD40FF7DA87D7D527DA8FFFF %A97DA87DA8527DA8FFFFA853A8A8A87DA8FFFFA87E527D7DA87DFD11FFA8 %FD08FFA8FD41FFFD04A87D7E7D7D7DA87DFF7EA8A8FF537D7D7E7D7DFD04 %A87D7E7D7D52A87DA87DA97DFD5BFFA8A8FF7DFF7DA8A884527E7DA8FF7E %FFA87D7D7DA8FFA9A8FF7DA8527D7DFF7DFFA8A8FFA8FD5BFF7DAFFF7DA8 %FF7DFF7D7DA87D53A87DFF7D52A87D52FF7EA87D597EA852FFA8A8FFA8A8 %FF7DA8FD5AFFA8FFFFA8A8FFFF52A8FFA87DFFA87D52A8FF7D7DFF7D7D7D %FFA87DA8FF7D7DFFFF7DFFFFA884FD33FFFD05A8FD04FFA8FFFFA87DA8A8 %FFA8FFA8FFA8FD11FF7DFFFFFF7DFFFF7D7DFFFF7D7DFF7D527DFF7D7DA8 %7D527DA87D7DFFFF7D7DFFAF84FFFFA87DFD33FF7D52527D52A8527D527E %527DA8537D527D84537D7D7DFD10FFA8A8FFFFFFA8A884A8A87DFFA8A852 %A884A8527D7D7D7EA87D527DA8FF7EA8A87DFF7DFFFFFF7DFD0AFFA87D7E %7DA87DFD09FF7DA8FD18FF7D27A8527D5252527D7D2752FF527E27A8537D %527D52FD10FFA8A8FFFFFFA87D7DFFFFA8527D7DFFFD087D59A8A8A8527D %7DFFFFA852A8FFFFFFA87DFD09FFFD047DA87DA8FFFFFFA8FFA8FFFFA852 %FD18FFFD05A8FFA8FFA8A87EA8A8FFA8A87DFFFD04A8FD10FF7DFD04FFA8 %52FFFFFFA8527DFFFFFF525284FFFFA8277DFFFFFFA852FD04FF7D7DFFFF %FFA8A8FD0AFF7DA859FFA8FD05FF52A87DFF7D7D52FD3CFFAFA8FD04FF7D %7DFFFFA87D537EA8A8597D7D7DA8FFFD047DA8FF7D7D7DA8FFFF7D7DA8FF %FFFF7DFD0AFF7D53537DFF7DFD04FF7E7DA8FD047DA8FD3CFF7DFFFFFF7D %FFA8A8A87EA8A87E527D7DA8A8A87D52A8A8FF7D7D527E84A87EA8A8A8FF %7DA8FFFFA8A8FD09FFFD047DFF597DFD04A852FF7D527D527DA8A8A8FD38 %FFA8A8FFFF7D84FFFF52A8FFA852A8A87D7DFFA87D7DA8527DFFFF527DA8 %7D53FFA87D7DFFA87DA8FFA87DFD0AFFA8FFA8FF52FD047DFFA8FF52A8A8 %A87D7D52A8FD38FFA8A8FFA87DFFFF7D7DA8A852A8FFFF52A8FFA87DFFFF %A87DA8FFA87DFFFF7D7DFFA87DA8FFA8A8FFFF7DFD0EFFFD047DA8FFFFA8 %7DFFFFFFA852A87EFD38FF7DFFA87DFFA87DA87D7DA8A8A8FF7DFF7D52A8 %A8A8FF7DFF7D52A8FF7DFF7DA8A87D7DA87DFF7DA8FFA87EFD0EFFA97DFD %09FFA8A8FD3AFFA8FFA8A8A87DA87DA8A8FF7DFF7DA8A8A87DA9A8A8FFA8 %FFA87D7EA8A87DFF7DFF7DA87DA87DFF7DA8AFA8FD54FFA87DA87DA8527D %7DFFFF84A8A8A87D7D7DFFFF7DA87DFF7DA87DFFA87D7DA87D7EA8A8FFA8 %7D7D52A87DA87DFD54FFA8A87D7D52A8FD04FFA8A87D7D7DA8FD04FF7DA8 %A8A87DFD04FFA87D7D7DA8A8FFFFFFA8A8527D7DA8A8FD53FFA852527DA9 %FD06FF525252FD07FF527D7DFD07FF7D5252FD06FFAF7D5252A8FD52FFA8 %5252A8FD07FF7D4C52FD08FF7DF87DFD08FF52277DFD07FFA85252A8FD50 %FF7D277D527DFD06FF7D527D537DFD06FF7D5284527DFD06FF7D527D5252 %FD06FFA8527D527DFD0CFF7DFD04FFA87DFD1BFFA8FFFFA87EA8FF7DFFFF %FFA8FD15FF52FFFFFF52FD06FF52A8FFFF52A8FD05FF52FFFFFF52FD06FF %52A8FFFF52A8FD05FF52FFFFFF52A8FD06FFA8A8A8FF7DA87EFF7EFFFF7D %A8FD18FF52A82E7D7D7D5252FD047DA87DFFA8527EFD057DFD0BFFA852FF %FFFF52A8FD05FF52FFFFFF52A8FD04FFA852FFFFFF7DA8FD05FF52FFFFFF %52A8FD04FFA852FFFFFF7D7DFD05FFA87D52A8A87DFF7D7D7DA8A87E7DFD %18FF7D527D7D7D527D7D7D52527DA852FF7D525252277D52A8FD0CFF52A8 %FFA852FD06FF527DFF7D52FD06FF5284FFA828FD06FF597DFFA852A8FD05 %FF527DFFA852FD06FFA8FD057DFFA87DA884A8A87DFD18FF7D847DA8FD07 %7DA8A8FD057D52A87DA8A8FD0CFFA8525252A8FD06FFA8525252A8FD06FF %A9525252A8FD06FFA852525284FD07FF525252A8FD06FF7D7D59A87DA87D %FF52FF7DA87DA8FD29FF7DFD13FFA8FD0AFFA9FD0AFFA8FD15FFA8FD0BFF %7D7DFF7EFFFFFF7D7DA8FD2AFFA8FD4BFFA8A8FD05FF84AFFDFCFFFD1DFF %FF %%EndData endstream endobj 10 0 obj <>stream +Hko?ؠH(R"EEM{y v. +hG==znwHڲopz`K7g-e="7o^/}zޠޟI^՞GU^Z%FO:ߞjY}r}0C8ԗtS[V6>8IL~QK &"n/ p3DhrYݎw}Ðvu~ZZmVOxT z;ժ汫jp?wjװW% h]2#誨j yi $y*G fʆaX@X̻Aǫ0 =o$:_q{u9Ӽͮ|t5='p>:1c?]7]>e?._>OʄpJ]QVݜ|u Kouhk{DM)agve!D3s{.f֌]4P;Hm_mRDy3w}Uݏ|܄nnCQ'I.`5o˴kLs2!-6[ +O; +aF=Zm>jdY(=QeU +j=݌P<0K(Ǣ6UհgճhW:f֪Px;۝v3o\X#TۭQ jo4.65#کD`@N|XCz{@[iDp +Kݨg|]X 9j55ӗhFQza҃d}jUc_A`ڗnlպٌǕjgv>Z#!Ŷ35Җҍ}B[iWGC9?o^SYOI+C +6/v`+M\٣]MI'UF>;\9i^LW1ݞ;q<8S5TL CN]"MCljm;D{ijZM#C~Fkr2]s8^ʶ]wfvW/nW%z)n!3nZ>77ǻK5S~c^L`'u^8Ӿt<~we3Cխ?LCU6>LSі<sḘ96ZX61h쌄v}WNkhh@*Â" XH@ Qk\ z/jDa hZQg'zһvW޺wj|Y:cù$CQ;[BC4a}u GS5WX~eճ&nkwVm]_Z3|i  +p ~Ssk^ϖGy<<,ɮX';yֵ^<ÐLo?I,6o?۝89c`T4A,| c 4,N[wǧr.klJ~~%9ܬϿ'w-driT0>>^A%~R_yEQEVERB V"*‚",O$y,y9qdEgYfI&3q/3,Œd8 ",M$Hy,i)Iq$E'Y&I"$NXB( $d!sT&RJ!%TF2DbB"HBp &D("ySp9GB@Y$J)L1RYʢV(E,7Ui[h ZvcMƯ3Ȯ6.4}^i̍Z8|Fim񳑨„mL-=y%}SfjO)4&}eL S1+S^a'-KZ3qDUJaH.\Ɇ|M9amu֤)'<ĈK@d i-j~1O T'P!Pسg^)lMR1C auxxĸ1UXsSc_w뾼]]KyP[sȢTSq ɮPebrݞ;RϬ\ycM z]YGwTx5,^ʽpUKo*#h 8JE*_(`^ܑ0/2B篺²^ΰN?g_<7s)2;,mMZRu ǜŏٷ?tD¤ 'dc-`.>% I@4yy(+>;]uLtH-x#S DHdiJ%xvw%O ೴ו}@z,[ȴܰ! ~F@bvm a?P+8eS ++ːأ"+ 7DG B[G,ongT$> +?c=[,3ۚҥoXo[ʜ"x +Sg"{` alXn:U#J +Mg-]t[Ň(X A2DZ=7<:υ>nQ8BvsTҵ=JdD>}Z+tvSe`՜&ZDs (ٮ+b<(v{O4 B.>M*}c@4NeE[hV>ˍclL,[WּLW?/m1|"T] UԢK Q&u~~U13HGI+u~#| y y}ctB2䲑N|th9nߥtt$]+餋O8kH_IG?I'm\N:H'sBH'F:.HwC>d#]ҥąt|GߎC[m#|t bȺ ]VoI@wO%oSɑth-K+#餭t ;=MpVԉou2 u9utD}ʆ:: u'N>)x@,d긟nCDH G! +qEQ'Y܆:su2~AR6ѯGԱmcބ ud=NXچ:uCݢ?v 4Vԉ*?˦玣]:RVA,@Dc p"YSLMUQ(Sw\OO3Q!':,Pu! EM:هYHW 9|ZgR1vCXrqCr9BǬvG!q2U2Nrq8l5#$OyQ) +Fچq2mANb[`%$*2Nf䢌+C44eKuYz!'ksCNrK9ɬ_$$˻+ +c8,PN|ߔ((YZ!2.R3IJ9ɵE!0K7rb6M9r R/r;QU=H9zH9ȌQ+e?ܔRܔ;^ +9 b/q6qsCέ(䰡kr/r|T1 Y. +#r#y9riqA.S\1{ĕGqEUfN\J#+]\p⊯Xq)rdي+&YqE⊿.`+\4#qӋd'y*##p̜Jv*yJv*JvZ)2*ъ+.YqE6Z)2slyW̬ss#̛rfĕ?chӊ+6.GF\m+Wd+λŕWM\93lŕً+FV\"M\s<3#V\%⊙"o*#'V\wq+ŕg5⚃WM\1lhŕf5S\s*ىkdUWɇ̉d'9*#']\eUWɇ̊k.N\wq̊+\Nq̊d'Ȼ;V\9:]\sc2oʙWd+]\V\y?b*Ɋk^\6q޴*ىC\ǭ-F\7q̈+#dWƊ+W䯉kʇbdي+.2slU&2qlyהoS+^\92lyW9q1F\wqŕYqjq|̜Jv!2*ъC\e5U/qWN\%*3'v̜J*W̬"[qMW̬Jv⊼+fV\9@9]\F\xq̈+ƈ+VdWWQ+rwqN2⚊WnM\9"[qņ5e/|2oʙׅG\!/Wf#_XOqWayW9qE*yWJvwqՉ+bŕʑWd+̛bfي+f₭rьJ>56/vJr)2s*ى]\eUWɇUWɇ̊D+bfوk̉+W]\1j!WcĕyWΌg✻bvZqȈkl^\vq4lyX0ʼ+gF\2{qȊ+W䯉뀷@7s>n@\t;5ME]VqMMQ׹S\C]w [$]FK:IAKdA:Qu.]VAO":uPIYI759 xF$݀VWOIۍIq q]XSMhIqa2r*ٙ+>5o1xs|̬q7W̬2s4W̬Jv押+c՜͕se̕3c\c< i~n:F݀.:uց>nP[nőﶴ*I7II7NmvNOMNOdA7hO֬rNnvJ譋9D]@+5 u<\vVo=wHFp;^gT +lwE<~7ZѨ cQ">_ڽd^M6PdH@ME9,uKjǣF`EFlAfýD-BfK6 ]l\6%BffYL|2{<;fG*|2^RJLJ UG7͑/CD~"2dvѓːٕ|2|nelqO[Tɗox\wzv^f3]|l>jOǓ!w] s8asD_̦ <I)J?< +; @ (C hD:2›R!hnZPEo_݇w?//ѫ߼|O>?_~};y߾:`h}w?~\AN0~?W߼ӯKӑڞ'xm8r90ȗӺU7ɉtdٿJt6zEHL3qX4iN __8T/VIr ś2 [CNq,C[רJ\@P\!{Q8nq]͓-S!)yiVSq嶛^ik k-kV.V>Rb]Gm/壬~JVYRq|}:9/w{3ESɗJ>BKKU*9R4\lW%UzN^wN~|_ʹRb,ٖcKfZh5)α/$IRި g=g"YY/\Idpm{{x%丫檿<9Caxr4'_qO<>v<8h@qusʼHGR+ϒyF=uҔl뤕 +Sɶ[ʫ>:jRjhh=0衛o )]Fgl/tYm{NI&qͺ;NYxeptҐICC曇fVN2rG8iv5+QNDtMcV5^_}xi~#mTxiB5J/{uir41_Q;8i~qV7vVkkiyEVZinVgo~8>JיV=4V55륡1m}BB/XiN/c9U>#V5V+ Ҭh\bT=xi֊Tas+ zOVfLAq8ӏz7JSșn^/f{4<غ&bOb?r:(.Grc´ҵ\R$_Ve)` B딗`)Yb?´;KrT.L),za|A3aڤtriU4vͽ]>KR؛'RD)$J}zVnCR(Ks^.H)=~@ʚ usRؔ)jH!-AJ49J9ڃQzc4W.?'g8˃Q/uIcl$FdEVkR-IQ`{3y~~0ʦU(SPh}/8Ӏ(um/D>QX6 vk/C*PjuPSk!K-6 G-fp !kmM8&RȐoƕF⍒>ZC'ތ㍟iu{oXoso97޸F1IQ[Vn]pU䯷f8K1 D06hUm[hn; M70gmtCIN7hoУ>%\;#fh_7XZ'\G4pAW+Z5 oU7JqFS#XQ[&[|g}楞x #viaБ+lrCVo$MKFyi{ћU 8#l=c$O@I8ׁ$פ'\3HC8GU 纶$uNѠ6K8oQȲ$e!jm!bdj7\2!]{x +:!ȸ=r̘slIk%[I9jP蔣V +Slz:|toatM!ZVtʉC|@y 栃m9СvLN'iIYp:s:%^r֓sI"$Pm^tmvI5ұmA'mQ~@b :$AGm"X8N]A1m&3ktwL٠i+IǿsNnl3IifIXra&w^;Qƪ:5[* :G˲Dt^+A:u9Du$\ֺ~E/PUoZ.:mt97頺"(K*jf۬teܴJ-M#9oZgjX>'ky㪯 II:y%Iǚ3&lt|&7W޸(5'4]D76,}ӪN^IWҀ +i]ex3t~IW.t +e7* AUlWzA[xAǀ2o`u-7BG +5-7*St</Pr#kqH-㍬@A:~&MrHǒYzr9M~(d>c#-9rgr{9ʥ7Bjode7 :'˺Gku&\$E/|z3r3+ɬp%ws&1U|168B+`MKAjts%=1}\AsȸEs ]b:RLPRD$8m\n+nzjeRYK9pe-:ms5,L.$ 攃VRK&ir/k"̡"Vb9Tt&{ù-9Ёd\;9\;1fsZ〓s5s K49Gݪ>cM@sugb}"2S.a_y\֤7d.}8Ǎ,ᒜs~bN:^a[0)0wx{0Ɠbr51VCڋ94hz1Gq1G`t0GU}9uʺ.j/ }KKormQnC Ss+U EAʴ7n:hқ\*q+9\77RZY :j5'uo=AǟV%Ƶ[&7Bw]opst* :}+[{wʍL<ݤC&rVcyɉel n:n3dtԣ t`<脚z@GYn!Vi[)ι۾sh_AP?@:#|MYC1T7xc{A* +dgpx,MəQfQ>`ɩ.=j^MCC~anN3^# }0bHДOX>y@@ȗ#ǎK@Y*ruE*|E̞zNGb!?\t+ Os} %+=t=0^/ Z<rp#9Jyryp:.SlșjN? 81?y6/8,QP`BN@04 4 te_ӗח"۷/O{y/?>|O߾C_l ?~?ˏ_RGw6ə +#\B3:/ 5 C@bbgcdgf%w~MǗ +ȋ[`4mTtj9ZOURz[:F{B'XȄ jA[u\SU:*z%kP]1JJmHΫ +eAW628oH)$h)XSt'1brnzX&DKЖ5k:l8uK#kN/QNT4yM ۑsaTvsHn]SjqLE2Ii}.猨3 ]X9&j̉}NMYs@\kP{mqS-SNLjO{ +EʴX rܒ[YKsV.(\#eij@b2A6Kj==I =4/.)]~4/kNU&*" UQZ^rͱ m[nD?o&^|2]t2xLـVwW(Ӿ(zvPɺr6g?(FFb,4ʇ=vF0)eN7O 𔆏(<.#7[x[G::օ72ěv! o)7M8G"O &OyX [wZvވc9/DNeۦ#zp ٜpҶy"x9Y'c-F'q + +qԬ' 9:kqB(V7(k(82/)qzǯq9㈐fN|ZGt1NӪ3+bӚHAgr:esA9eCN9U@2g/)ڦdsȉBi\dSZ]3sku${bt{.cÝSǟ^tٚ[r6tzw9C\Cw.ӱC;xhz6z]*ءn1/˞Ojm B=}nzjSm|{mgڗ<ءl{VǮ?!{6)vI;wt&+Ŏ2,֣ք{g'ӷimON{v5ƅjqrv\vNogۮwH#iǝ4qNw^[I;ݝvaŎzCo^; ND ;y.کF&\;rX{ɷv ;Xt#ygڱ'{v8յc]fN1kǺHJɵ\H|Li]st˞ڡZi_ڱ7VDWֻw׎lq[<< jOKǮP\[ҭjH9vavtxh׮>C.vx}` ]Qyߣ $[97r)-a`Ǵ#j]#w %kg̲ʹJ]#aڡ+W\;NWPzZTԨ[ eFk:^HP҈IW'=9//TAhT:; u3S'v]F:Z$=[˔֡u%UleҭcHw;;3:v赳"Ųn;v9ŲFwP Rl\bc3brر-{mFe=tyHX]!'Xݱ+:vGre͂jzTڡƣs'j!bWKώ* ;;ԏz*ɝR[zp/vqcWq)=)e/9v-cވN)ԡ96B,c#RzIKԱ7%BJ2sկN߲c!V{=B,"uCz\;Ҏ;iUvt՜a9{mk j9ò'uˎ eZNS-R}3UJDXDպ|a.$ػRǺԡ6:vxPϵCǺZȖ;ǺαV?0w4Ȕ9NUws'Tw;9ޚ2:zؐ9]2Q\g[rr焛_qpΡ~ #s]$1 `9C9K';;z}9~sdE~EYpjs2JGә'cبɓfs7:=:aβ_c-e&+[,*t^iV(ty@#A^{=P~9$ЛқC'W:zUtܞ[KJ PV|).FRމ:=)jiY:d^[Ǻq봾|[:czR6gcĸsZ2\Ut~Ǽ;0Gʼnғ`Έx ;Řrxm[[-FrZ웻->VdX9U;pZ/ƤGdX}3;ev9w NpD=|p<N`w#wt4 +]aQ?2w)չ+ -2,Y +o 3"[ᝦcYaȰ 'Ȱ°eCyzCȰkwWwq9jE嗖3,ױnϰ>^ȵyO2X[ZR]? %Z,Gjq_#m'pdu2H"Z@ɩfz'\J]bj**W?ծ{11q+-=evS#^ӧW?'u$sŴǛy'9 =vN{*qiO,9؜ҫ>u·mr#Hun +UY#XGAEHSY5WjCBƮ;i?/~?_~?ˏ^g·rBܶ?"@HiYAvUc~G~_[<=履%___ +=Ǖw@SL뼥* 961T)eGC4zif_bH@e>)GK=#FO𢡶RШ\::Al5e 9DO)LexMeiFac igBGԘiCPť"myA;f +fBioj.YHm7B+GulT/A?B ӚfhJ݇iqj1 +cF9I^Z@c6Y$.tSciỴ{ZtMۙ<1Fѵw[#VCԼϩi)"`;kӘzN9"bzTTnVRmϘ9- +sS޵yNj:fS{Z'TbtޒtBQ_iZI:2tQMkrL![,}&8itZI5IG]VruVjcJt|ĢeW5kX5t# K1!=I9[wI:.cWiU=mWfDLnt%.T q} /qd+أ}@WptqHWE:tˏ]yF3l*bгFv>Htt(Vz +*q!1*M:V=3HYmYHtvJ1?UDzry~s8q:̀8/1皐sʺv;VefDzB۲2[ȲgUE۳*׭@'"ЉD4*]g^K :99g9H`iq9T^~0}]Cs=>$0=Ḇ(&G7qan; ,cܔ+عXmA尾RP[VPY<3C9IE3(*,x(W`4n}Q堗GW.̵bNAnsWrz#qq]YZ:׃|8ǘE /ε-39œ|#(ǹRnI9lO<\~ycΘ{ybΞF9@9Z`of,Ok/Q9s#0GY8i>{Qǔ*tL1{:Wvit=sn'˹2Ԟ) Sz+/ve9ӹrȎfL˹jc\)={@XǸR6t0.A=ǸBo'jBk E:r6H9 ++s]CiBձEW6]Cw[ :$2z<}h9qлϖq;W\{t$qB_qށFazZ0ذc\?sseUot͊7WJJVބ,pB5M ż}ΩR$`LimI_q 8 + :Wpe +fgpscWOcJ +)M:.$mCM;t[4sL91lwNa+jm9֝]Ҟ2^%ʺ[/cN86˘3@s8l9Եȱe7#hĚ:mqn=}#ubJ֡E{r5D7uMqUqx0c]sQj>08Cfw8ԩlU[܍Pk\f' cWơ.ơn"0c#%qPԇǿO8ᬭ̊2ٻ8( c ʼnCV/Jqֽqr%i-[:r+S䄛:~Ez DZ`BEn`k<ӏёcl+rvN;ӱ95%G_z {stBctDFr2ga"'S9]xQ8TWuCRjs\%VgNa1'9}gN7%YuΤiB^&;: q紶Kvsk\ӹn&vQgn|&ť#OrJڻ: A[l/P[(ݨ㾬[q:EL/ܴ4TPROt5ʭ:nТSg&5:u[I/+̤q ;u\gɨdl8:>n)uU(N^qVnR/VnE]c-˺$ʙ[16rYU9-NH=cji+j-+*[ ;ؕ[|4PPN0ʭ5:a8'gYgnձr+?4[ʭO.K5N鰐JJ^4Nn;t/r+wF :Fs:~Qбuq)5eAG : b+>;m_89H; ؁{l61aPdۡ]awmwyq=U0; q=UiRtx1N8=wv8 +bKЯmMC>ZkLh:z6dyX4G+_y[囪gYYY7zړmaOe_qFN"YL+ʛQ3JxוDI_^C +'Ya>`Y'$]TJ3޲Spˁfէ4C=bi%X4Q>stream +Htͪ &ʉv$J't^Ihm™J,/:l^-[,K-Y/M}}Pa5^Kh$r׷9EzS #"͐sfMR3hN1t.תCBI^_ŜM&Ѩ%% cH$tDZ}V[b؆`flF.u63xHxɜל6ͯѻk*>07,wan@{!0rt)Xca !-QvE4\ZBz1|Q\wpx<"fM}j03#Ҟ) 4pLj vecgr4pq6ؔBO5gRΘ>Lcd"-]jZgm6me +OyB<gfu$#3G$>5uSJ8HF-f_O__~}Ͼ7~>懏?忿黿_;}1(L(y(%υQF89(lQʜr8[QYY,u)|mmB[1YJ)zLe)v :a1S>0ŚLQ8 +k~ fH4킫)lQ4B)D2Q +\fR 'Q +Jb] ЬH +w„Z\ERH].,,@:6R&H]/-raM]0=ln_{/jV91΅XFa]>(QQK{Qԓú\$ɼh@UzaSP(V@CAԳɁkUunN!b; u&C0'$ k6ۄl"dM8C g+| 6c%O*ԅ7Z :cd oaj VxslyǛoo~gBE y6|wml̫p63Bt$F*6.=e͢uJ;`b1-Ao +G@-M7h z nQtzE77Ջneݠ*ͳk UxsM.e|>6\ԋovvV7זmS-wb,c7|{4M7ϐ2 M+Y"T.Erڀ4^uStB'pa^ p!U,E$E8!!"=}'uuN6@ )/�fBҒb4qn|䠩+ȹ r%g@nj,ʅߵrfQ5(6 + ʹVM9+t9\M砻dZmt (ҩW`S:Ȃ@~st"J tb9Xl(AF7ýlC9œs +f7`bt:h t5s[B*@OkAwfY.й6 tnSA1/6 t1&]KYg7ӴH*qo@J>tq2t!bv3KA:}H'"]d&0@>}6[`˲]D taU ~p#tF.tE:<*[CILzUHM:o7ҹܤϮ 7yZY5ӭqw^6Oc(|A|7[AHgRsTݤsϻ"oQ49hwߨm?)qM|5tM:_;{$'IGaYee=98v:8dTp7929Iw8y'w)9riY5R[@hYghw6yF`͹pKg NDYqq`{0Emޘd a +JAk^us򕞘>{9φFs CO0~AglߔgWyGPs0wrϤ,;{rnrȲr0x?焽Pr3ÊrWPzZfT`N`ޘ-3(0M\x^;`8S9'CsOE9yZwι&ic6\koqmcxZs919:0K5FdN0uRQ2^8|%8`7$ssxno0ongC`2oQWau*9Hooys&zuQe"!j{MW-  DQѨ(,9g_? V}홟@9לsUch\ :JCLfJ)whseLVNAÔߓε&q33Йηsyqli*_cߠV :~Y.@|Y9Zö]<+~*1u9Y#4q}J&c\BmAo$[@jDtu\}(vB/w+tMMt71@X;r+1P_Fa94ؘFa^W 23q te\-Tq.r+,6:qtJ!ͻ 3c\18@gWnE6(γ޾,3[M;sV&s&k ^s6M:ژ3^޴C.Q)s#HAj= G]8Ї`ui3G]]2˱L{i25` 1ֽq#1"(nNcm{'cMx2.+уq5{X7A/!;/KƋb3^Ch`1q}1/:Y!P A_844x"k0./=Ӏcͯ9\d QKirU9{w-9{A',$# G"9.O^CLH9;RrL)bIy(gWE9Ⱥ)YA'~aYފ,}Lqa*ŝqQӃslsi5~sulfRs]}@vH:ܗO%P؝z1tǷޞ4PǢc[H_m:iPi絡EkBtizK$zVH؀۷yu3@<|Fu|;uhncǷNޘ&[uUToحB +F&ǷBkvXڎoQaY +ǷǷR_eu 謃nݬC[:Vި9i|[* ^V[e,!"&JtLxqIn'KNPC5|>]Hˣ"P6|eH=&В !%a1>מ=U>aB=<Î@%Yֺ}Ï?O_^~Лo޾|O>?x?{>__y/IЗ7o_~~ĸ)y.O?W@,-ԕMg4u=` CLQ,5$䣮tKti]'{B-GyXp*eiFMCQ7YҴʸ}G,ό%ϰDK):نLg*!(ɘ9}=SqA.2 [=e^J6д&/΋7sbX mmHm!~#p/7 ;ehq5Ǧ6dC޳ӑNs#fQ֘nKfG&d´{YVh3`eOi1v5@8S 6oep{Qr>\1fd?c:tk9梦dcZ#ژ=O{L`~b;iQUgK%\&p+[D*;;~Z9Px(:{n"fh^-ނjl'=IN Xj=qtS:֛xVb7M79{ 4!7@,Wik׊[8АO _(>Fib fzP +JQyYRA) +tz1#6K3*9,iҼj!xJ))cY$ +6OO7S/˚o"5s5{Wx937~UYS/*k鶀 +iiDUl8z͋ʦ +e\8UPkS%p +=^TΠ)$\4E 8ֶ.Rr=aX#`jo1)SU$`J)d,tJRr 򩹨'Jͪ|$(!~CJ-Kk^02rMRncc8I=ER&GSP$QʤF)B6I4$E,RfR,7H:  %(iU(9 +>цQ.1&F9ZP:"A=;"Ubb8Jc#.X~O8r!BIqИ⨇%Xw8Gj%G"$lTB7HmmCv% @ݒ@udn:4Nd:s%o>"}!P)ܘh?*¡}X + pY#y/WJƊǼ93w +,)!8I)Ʊֺq?pm[1=ghşbfw׭8,<8rqIX g3HGyHt9r%rr\9"E989WHwQε +y\9S9vUΰ?E9n1Ϙi]9z*Z_rs:,α&18t2H:㱌CǏh:jE:l4|.s11"kVJl6R:]Rǟ^|zZ8{tj[[O_팇rjvd2^`Z^;XrluȾ>/CmYOy^VCMmIbξ!G;֢}s웓.v3fѢü&vC`ǚ ѱc v~?#g%vymXj .\;?Nnю+ɟvIv!}w'ݫ＾v;vc F/ءcrFbgYkr>UWX?rNrnyڱ6귱7j׷d1XǍtƛ^:wN7ud{$vyVWWbTʛ_=72o~S6Wӆu¶Q(Y?ǖGQlcWp|N); C7T4GF4eAJ/{QHHщyt<}=WyO:B3{__'tYDֶ }i*q!f;E:.7aW:4_\]Pg}Wǩsg%uSmq:sx4^gon]Xhݺ4:fֱŊӖZZŊuiݺku̱;:uXo.`3:PK+q!u8?IlUo<>s?Tjj:=[P8lTVG7^:3+urץN$_:^g:dNqeZڱ72~;3&vʇܷ%vue!cB}S,2*zS,DOy:e8I-GŎyYX jC,SX԰)&v#vYA׵CwuE;Լ _aܗJqNe!y;7Cړ:uuMyC5:6뷶a;5{C,7z˅C?N=wȰYNZXu;xZgXnNӞ1q?3,k{Z7z6cypꜢ?m_y7zn2)ԱvZ;9PKXry f9АtYIHdѐ).UwO'i:Ǽ.αsF3cs#^\/9ijcJdM+Cg~ǥ92K琯aΜ,ݤ0Z4`9/2ng9N!GCx_9r~E}{82 vaHf2'$k'zEwNi:5̄:N8gL蘥k-އ5:fZ{~G\.dwB< +?oFKС^t<ٔ_EN?:ԦXO3J[OHet*Qz_:n3s\ +ޅ:?)1ҺfunQ:LI<%uءc֡=niwa7\X2R;dY1v8ӢKPCfX9&w߀qǃ9OD#qcnTrǫ;ܶ ѹCƶ-hr[oELPgX sUwءI6:= cfrC®pBљRf]auJWaK5fXٹkho5B3,9w:âvlVx<|;z3gy3,Np +d`͉e'vox}3C,hC;=Oα;ba72Ӻo|:l3DV U>נdh=9We Ӈ~[F>~Uj^j͇nn:W'obl`qE[KT? 滭lDW,ur+lHzrwlj1*=MΦ -~c%l\פ2WٚxK=>2DI݅3yL~j塠 HװahPE>$Ltm_?|o?|}o}ÏW?|_雏?O~G_?}_o|$ao|_+#{K8A4Bs:1K5  Ī!kpY5G3"#|7 u)Qk4mтWHI\JI}|D 5oӪtJ45eo-gOπʐ]$$j\gT"azvqQÅ J+G<+Zcѵ*X( ׸ $@^^FN|^ JA]z bB7ZS4] ^*\,,y ee= +W A4w +dK!$NEi668Lj!3tQ2+$zR5F{cιn-҆rJ 8=f(yq>cy7[@ɟz*:3 gXݨKA7JX]tc(lCA%6:Y~Am|6zMd6hLms mm#hv5ٴa-{BZZXCWv&(OkenCHu/c5Y udc"[gRd~૒lhSڠG!BnFo_7(gZdn@dol YA6h؇he]lc,l{cmz`u6Il1bbhK6mxglÎ٢la:6"._؆ۺ(k6e\۔uܠ{p nVMFtbłnp~|(ݤur]tkXmc2ƛn潫F]ݞQt` +8l)tk,J> 8>2x pҥ]ӴnD8gvjE.261,УXȱpȱ/I.IﯠB-&r0:*b6eC m^ǡ)Oc6UoiM8HKY:8b3hRp;qʷzwɚqԫ6:~r@NTYOU-OU,πo ' +]Ӭ3Onfb2}Adv'}Y .A y!ܠ;⠛#\ı&Hw戣v#Ӯ|2݄K;[p:᠇C8,preM >p!\Fzǂ W &\9"qWp=K!6*^jxEbFj@t !J8hK۸q]ClqK$y0]OGl14UUn8jrqŸ9 qa|z(V~KHt-"DECp3[ 3v9z1zM#E8^nzV q7㸋i`NeF7Ay$=}DŽFAtDn݈Ý6J=#T8v>q҃8B8Nat^0zOfk 1qrԻG !V8ncvU 9Ub?n~UVUރ rvUM{*eK+ '&vAN ǬRr\y[끜n<̶r|_A9.G} ,7O@Ne6Y֏lx̀OlX9%j=n]G 1{*VUx9k{c>N2J=/ımJB5BcݎY}1mV޺rz= '=nX=f:cV9g^Y.u̪sDrVE9l ߀.^IY}8a*cVJ0NPǫJw )$}?81n4K:VcUOaJkBcj`7sɁ8hY(A1F:7ߌG靹F}|x0M: ou\SQO/c[s$7Du.kCo17%!G7^>XwCqA>"l`ے]xCi7a zՋoR,=\7y|5 = t o䷚ox[޲o7P>c +3*8jKco@5`58|̸X8^JgXst9 puspvN)"rZAg+qЩq!r *!NڙS{2#'fIN˂q{7㐟@!6J&&6nCWa-(mc{SRn6(FA}aeê 6樓)0Y'0ǢMBa:~cn907xUnOBcf=搡(}07 v**9a*9vb![v-X=^^d\GU;ݏN +sos n sڂs8^UڎW~猍FWhtAn!V9/[99mbxU3U{W.y*O3{(!?@rHVëF@ 'e&|*y} kme#ȑ*cX@zuAܐeځ)L>CcU)*{qH}N.۽s]b.7435@jh}cX}PR,'X@u3v ׼hꄣRrӬ#18|e]y Tżċ C}ګü/}S,H=bZc@>V_jg,;ݼ{7v^xaOJj/uOb<<|@.Ǿp?vWb%K +navoy6I{nEnJ"ؙM=}cb^s 1/!ʫ3+A`pU'AWcsS^6xs_TҪp>$e&mA  }$#mgr;K~f9'8D?j  3%w]'7_|?>>'o뇏곿헿?>?K~_~_ᣏ? +n3)2<Hr) $FxQE`gFʹ,=E1^pj]3}k.Y/M& .^LJ05l9VP:Mt, 7yVO/ʷѾyF5KnU=cd t7L&cBV ^PIMr SvZ;!tNԕ3DWbȆ:6D )[u!Z4=sZ{7w4hxkmb 6d:QJKQfFyS;j.LKթk(15MV~8t;\]=Lt֪3'槦dh1-Zs 6C{eL{PPKc^Cz +xTeƎ[=JDl2ԉGjeTY@ezX.bU2N'Y5vҒX?Woiy wAW|iFDDoA3Dw!)~hmtvM:<ɟ_͏ RBO RȖ R<H!kR&msxQCR<4,)ciOt4:Mr)iJ+n7NҎSY mrigѯ~18~X)v] ՘eo*sTi#@c_66hg[ϫdsAC)h8 )m7u9Q*Vn>>cĒUnZy#6$c1TbunadZ\c0 +NHG!޲܆anI/e*~M-]+cdjX7x$N֡6 ?D2^+Qw]tZPܭJB:d}y^B:p%iut'q}PkcՠقM7?ֱL H:f]uY)1ނ:fYЪuu8^+c'X ZǼ[ZuZ*yh {M!MuɝgVԖ\+ +3(|ql]̭+KQW&+8ic>CY*Mo +VZQ=Ԋڬ: ~uS[WٳQ:8`n]壯:v{kYWV\>uuO9n]~B:x+/ϲtUOJ]C$wұ7vC:mVm^?xkY\Al9orՙ(yrUNZP'tZUɕy&^JҡtY *67l{!=pxAFs؞~sby-]o֞s- sxkj_&琷# JYrt6vH麹+k|teUٍtNtctOKe^yëroxޭv~VV1>`Ox9KP-ѥs)dz +999eo59-{(O9u?5kNGּ xO)~B9~~&WVKO#e`~q'F 9ޭ}I92*r|_{>qc7r+ۿNP@n8B~#r\Ƙq)5c *':9ZfbRN0k\l9h>c_T;csgbjlĜ(;/9Vs|,9VOㆰ6:ѲN@wNGYI֬ΏLiRC:OI)aEZ:Ht MnUyz'7 O_^ivdX7#Ž̚<0\q3vKnbzu[^yjY-޳&i7bH.+;M !{KOܥcWpjOtM+h/]ozE{e׮8[^Q;Z+7=4̬GWWo]e1˅tڡ6ʛ^cO$cǟӱG^KHӫJMQ?e77y6:lVW +4P݉emۛ^& iQG[\D'QǃR\TQ'lQʝ/?P {\I>>5b}ƾ;π-Ncׄ9Nö-]ŏ~|;giN;b^Kv5rzpNK;yic\ ܜɜg@`S+@Aȁ&ROȌf`Yv|f}xoz뗯>~w_?!;[{%Ǐz_׿#9{^1j^e\ረ++0ɵAz+ɆBV QA)y NCx𾵶Xp);M~[7:ȅ cQBgeŐxx1DF~ψycaʡ,uǏ|tb?螳s&5cWo+e1-D7/~as 㜚m잢t'[#9F>'Æb=J!lnA 4f8M*S;@s"׃B@[u@פ]:kN-r0UG.{n, [Cgwl~wwRTSEs=.瀋Ojsw>gU{1^ewߒū9=G.J];![o;B_qFN4(o%)]侅O2 dYcUSu>Vbky~Oeĕ>UYMh@/lŒ6V×hp5ѣŴHWEr~_ ;ARif)tMy:$EF$ͼl'雤iiJNAS4U Mi{"fm&)+|KJ8ղq +؟8-7r8Erv<)bx8X<(iEy +}"6WkS-` +[0.uK!`)b,%QJI=QX[3P*;QJO}1t*(V +9I) R.\^AJf?@' 嘜ZTstAHG cE)tdGhUovz׋̎)h :Rffmns(bm2mRg;1nRQBNUmRSZIVY1fbui-Vqm7 +a[E2A8x7(oQW p;67B࠱B QO=c!:Fj%G7~]|MUoe'p̀[cz5m72enеz 1nP +q bߠ/;ߠ}buz+"g@CzMf :]^oߠ$Vcc mnᄣ_ނpM8S/1_7 +ar8N"Sd!7[ =ԫvNݲ;!P 9 +Ʃq9Sq='y0Nr\㨓3`\|B;ǘL8ݬGmnՃq٘+GâI9Ʋt樧ʪ !易.sQucxu-8Gd61:mAwS7Chu,`]eEy{˜8PWۮu[%Pm'j ttɷIبzn"]|+u6{ իy1nB]:u]Fb-T9ꨇ@u+@cUסns6uFNu +c*=Xǝd7txuLk`Sto)lSzZ Q;:&4L:P%P'SQzNnIbXT:-GN:`u|ΐ{`b3ݳR#Ousy`cy`1_W Q~vv:ݳRYm ݬ,?欓.gӪE AF!uVYkkYJ}.ָXggY1ohYu ݸ1u7v9R:MH t̤vjb )[tԫ:v6q9P'ݹ&ٹ*:IN(:IU\\gI:goe :zfޠ6k'i 4Aݥoб(u^ː]]yZ}99^_sssC9|ys;V0ڜo*ss.cj :i鬠7ies sA\9=y7`J9@Xitй6&bMH=XyO!= 7|z(Ue"7JA,@*n#U3c\Bmמy<'ul'$c͢l7sHG]t t>+W'5HX Hk57t谯:Bqf'4PU/q#jtz%@uܠC&& \3/tR:bkݠ3G:e.Љ}rБDRoqt܈t{'22tt1jWYz:_ˬ:<_+y'Nÿ5,GWNXڏT_uѰuUcԽtGK {Eܾ尯{DBHkQYfb5Aw:hܚbblrͰedPX[e:7 uݺ@bU{P(܁~WP_kjQG]uuTB|@GrqQ)50Q~W}t]:3N쫆jW\¾rɼnʍqJYiuLQþfN3/YsqQS1W+cRN2ý67:qЩP)jtNuy ti3AGKvAsQsНx;BhdžS:唣79rh3fS9]s`SvQ(ܑRi瑞r_2rkrIᔃ.\8E9X1N9h#ANb|A7r-E9V).8SrH\!M! raoCr=r-r}̡0s5 s2r% s% sa|aN]cN(r s6g<12sH/!Vsz# su1's{Y.1?tty:yAh1P(TyNK,IgN:%Qt$N:Iz/$Rqt,bhRtDrilAnc!\ 벇ye8N +ל;mR X߰ +FWh۽v2aGMm^vMl^1;VPrIE{]3ѱv};l6Iaqإ DvH A;?ꨶpY[N}HiG2½p]oX3+90vkl ;>JC{P ʇUa4oʍ+37A77{%` G3牺I騛sWvQQG +:jT̅:J2ucq*++\I7¯"$Dƅ:~"p:3krYڙ5m{sGo}):=."ϥ:ǘ]ƾXWhJ4u~8fht?~ e=tPs針i.:=lHzwzb&:=pN]ξ^v lq5X@y9?O_^~Лo߾|O>?O~?|7;/_~yo~|y?~Q?~ۗG-WAC?}tV@M;:3XC];뼣 vNɞ8Nqi 4̻0 D[uWmGΡܘ$x'#(<W%:~%*g. 0H2c?0W"Kɼ=eUv]m'@agLl;K2xs%f^{6m07Q?īJڀ)v0h%=W<>Ķ ({ls)*;Zl0YtD{mאacc 9.O~2ɺi {/TW)چcT3{?c)zUtaLMkà[#7^ҤT +wåȓCPܡTԝ8x%`㲷7֘ՙ=Ml!ΔUၚ=2Y +I?s;/^W>wѐ3Sgw+hk],^+EYZNK,攬OgfΟ~4i ëlB^+<q%R"m;R + .jTǰ|s.oqR endstream endobj 12 0 obj <>stream +Ht˪. O6mMdc8ss81!`N~UZ'fZRRjw_?~Ç?_{g>}__Ͼ/߽w?o?~?M_2D=ھ{?|WΆ/~G_?^ވA/`ԡ˂Wh7cED5ڲF6s*-fVlpx=^֛15zԯ(v}B+k'zX뵵\0hS97R16)5Wy#FmCg/`vV_ޤLF 2 /.|<ܜz4 :xڵSmy-.AfٽmG βU/bjsdTTkoW'+r6JcS&)&*c'VJ=%6Z>- 'Ћ.\NȵM%tQxL N5z3m_{Jԥ3'+ȜWl)YL V *U &LU\sӚ Pqco> $aâE5h +hћɃX8gb0apx.u.]ƨv!QC=|,e*YK-r.= w]HV"h.u8vQ>vQ;i =xȖ#/ zGЃ14s|`!OTozE=hmKbMLǭ:MZ_cCO1"5oeY;Ajzc2Gc\#Ẩɑ߁Md,[K,q-.mas{0Ynn⻪F.ϼAnѓoA ugU /"{ԥZYӳ殺mՅzV9n?{ȰG]أǯ=j=iy[MwUfh哪=tnRl?Z/{従7%` &>W/E?xT06}RpZeSEJA=6=ˮk%XrbR9lWg4&Y]ؓy`^6_㻲.lPXz ~@4b z}v&7(H-C\2c,mӓ:7dR/tB/]ڦ.򿥭!zx]^t6 %/!:#q20zlA{~C_̃<̝WXe/,pv>Er9o)mzԸHԣgu׷&y[Ev}FŶԃM=Et#7i>Lg>.>y#ԃaM7`܈ԃQ5uQV(pa o<0QGIM=zoz)ud,pw}K= \00 J@DBY!3Q~֪߻wwukɱ +{H9`N<'v{ DRaz,v{1@L{+q_&$|(ILce"̓|nmuc(q;; &8xx]܀!Dcd +\'<"ݬWF<eq~K%mØ/xr L0 mȓtvCԑY?xڿyHyژ'u0Ob&e*:y@Od==Ʀ=ez(.0z*LW!no˥Ǝ=6UAT,<C\p^؟i[1*iMVb;-$7>`[K#&T!&X0), =~YrOL/VwSv{L2-wJ{(%l̓z,?zQwޢ3J=\v+J=Rxc\(aXzB8G*z u[" |=hRꡪKuj=#ZHwaЦ* +\)8ȇi }Q}0~ʀ{Uҏ] + fǩ +PtjyuEV( ǜb"H:ɩAd~gʮ?{v@"P}ō8,5".2@H& q^QHNSr QZC +BƂ)04TAH4m4Wޜn?bo7o{wz˯8GG=fO7I_)v&smCMZ0eS 77@hs8>l2Ӹa |axdd, -xXzĵTR:qbClXS7,1"CAEZX8G!w+B:pNuF/NJV\cH~Bsh *2y^i 2_6:i9_uL.qCFP0rpWtərsXBϙ9;!1-g.b?9)񬖜[w{'Ssn)sE9C G\F hfe zaP']*[XMWF:$kDБ-i9ѤyjoG<1-i,"H'V4Z ,Iç-k' [Eջ4p֮&M8CŎVfK{Ҥf&DlPVXt,!|nH2!煝Q=1jׇ+p񼆨=pO3Bg^XIAs[׊HMZhܹR"ꃼ_!>?lDV.'h (uOz6aT*#m̹`1OObNLqLuM jz&i9C/zڨg[ڨg`bGBgkK64!ʹAc=:M`kiJjisbkڠvfHB~KyM斶NJz($`M$k˛H$В7fϛh$lY{F*27`ڞ83<m픘'`\ ]ˎ"^ՔUKE#,ͤy=`[w8o"#.1}[Rz(s 8y~*ngZpk6gb3'¾*Ln Ba$( &U~Ny3_Fa˅1`-#w|47'b 7MΙ[dhs۫f}"GJ}4W6ik]>D4fXM-%si 5iLG#`h:Ū5xfy*u8VQn:Mv%|2ӭ!k-]># N|Y8>6ML(75 xlYViۡLu"b0ޝ rV:@ 2/3*WNJHxHp dXc.mGa+(HnV$;R0X9?3]{0m}ĠP"mNEV)UuMi +wLhCSGakwZ RN ,O(`B()E3W Fp1YFDX頶,-vj칒mNjVS5K&7n1ymdfS[e3?zjr< +N2N' rINٛD%?'Νj2N@\|H ڶI˳=li qΗF/#e%+=ʆC6=#NбKnOlk{jf $Mva\KP2N\l D'VR M<d{\W/?f`[l2e1PJZ.eYWHmJ\`Z= {M^/{ Y ,=Z"I Asvht-"N[{[NA}87>)l>֐ +`+/e [1Б;A1M #,c.)R 6E28=&<Gιzt~Z,.L%{Zl\6(fmGgJ*tM\~ :[uZIe|:˛ +^W/=WU4A 9;AsL5N4~ j2L<CbL1$&=0ǁ;cb`T~"*KAw~Vo۬o0yq۞PSP l9+[ck!P&p0/'jm m(PLp,;8|Mm{jmȄ֝S~!0U&f1OpL։``j>6nm:fOlh RK'\[g"~ s?hSP:\ =Lu>VMܲ> kD`$3+e_R q1^+PF80Ƃ +c,ӂ%+sAO zzзm~2ng8g!krë3HAIu?\u>~ +D-rzO!X6KgH2[r"%!EꅒIS}?bمڠK$]}{ 50`h3G 57R809'7i&`/=F:h"vK- P6I01LE k!p!u(bx=eW5;A \UHsŁv#wս2k|^ .zildy h'jk}׈vJg5:OP]% no)hbf Rj ,ilG؞r$xaH8uPt`'e'EBmAם,,g'Թk6)/,SBt[;w@S^Kz2"ܶs1a2]KO]ApeR߹~F}>Ӌ՗W?|s_?_}ӻ7Go?(/~[_埀~ok5?>|d7$,Q 6Ï/Ꭷ< =7hC_fm:;N`0oi]4 7l  dV6VsSxu', C5^YǪ06S(`] 4y0CxDb`u[|?{E6;e |Ai d ZR&RiAћPlDՙJ5RGnGL_ p(񄂮x3=@A!,OUK# y1+뱔/lll `3% #ѧOwABu+3###-4Rl{}o1oʞSX< 'FQ5:W-Ck+#jxuO3WyZe4Je-X5!Pڛ5̞`Ir%Pp{)/EnFu³=Yp &hptجww`JL{;5d\58K}F L=F/,wb?G@Ò}Ί-Hf]˰Pm' iUb{M +0 $SވR8, +Z3S7' ]l Uh zYHF.ޢ3hjE=0լ;Fp[WGh.r +M@q * =6Yl [A%tK) +fGp>A4E [42ŏd9p%\Wڏ1 +,Z`V"F@Jtˎ::#:}Ny<7~i*'7^&H+cW؃$- Ub> 8<6{jmxj W:!Ԁ?+23̪ngڠ;e^ */l`n&;JHՉigu.CH-qjMgP*O[;\CjSS &-@;ve2{[hvNS' +wU"UhiXO f)f) ;Xͽ.3C](NO3="Y5`fBl> uެ\747hxZD*t*]2{x̩*csqb0"~ +fGac,+DFllCаNіo b ^JY^~[Z4l]&:%HM(9ܮSJ_Z$)mQܲ%RKRԧi}YJɅTҽE@;gHdH}ƛٺFu`4L`$T+%?-'P5]u.h9иaΥWDi\*(i!aU:}ҝ f֑dvk 7vO4Cb N:l@쌵;un0^5Wz+Ѥ7& +O/llOl.l膇v E%644 CGUo;TF2r&a04Jq쯚Kp9&! (^XtViݣ^)2ϊ$9ڰ[.^ͼv?\n@zvTs٠Bw{-)4m4yS0oh#8Ee'|xyFK"Ik\>Q\| m.q,e8 =)O{krqaoH 3a`| hy<Cej]'Cr^'cj_gd\ڑl\pzt]O<:LОMk 1ALQBA5L TvF$t߽W__}߼~ǟ?}O|5wM[%ݾi;o@->Y{mI-?"6WgDkND0:k$ofFח7AnOV0fC_ebyR"Ek0}ȥ˴z{Vl/X2~|v\Oܲ-\Ai?ӚKdڂ%hWeYf4Y`QHjw0.H8q7nM5; Ĩ"`d+*j8Um"0T/_ϦRgfcȭur5Qr 0zKnU5PLwE @!IpV69؃v0R|;A^Pʷ6 f.R@DXF;Қ7cH\w]-ϬHnm6PXU$﬉+w 0 +cKuGVq_f5p‹O,xyd)5[W¸҇S]R^k?$O#XwVOg`Z*$pg9/@zgkRvP,[~7 t  q֘c}\I+Y$@D؂on^MK`$PxV%Ww3E8[MyBp]YMT lp;WӉ'?$ + ~g;  u:J"1bP2)c?vs?]>VLYf]1*16 +׬yb`JoDfc zS\ ܒ+9@:4o0pp$#smYu_\iftPLj OX\NH-?wB&0&qfڬa#T?{fn5L!-˺n?]EMk=>V@: Zܧu2?n(Tw`*N-R #_Թ a1'i`YŔ6ĵY}.n+َD`,3k[IƋ]|޵.Uݎ" +?zu#t//-H؉G/bD|{GUs.E+_j_]]QLɴ--Bp7B-2^;a3Eٶ',_T/%z7dxFrZVlYVIGM1&S/D1M + 7ܨz^ˌw##t?iъwN[j*CdgRxqlTa0\^ ,OC!Zy&ʉ]O{Fh~UvI QOT9}$0/{]V "?T+:)_Cȓ\_NxTgą"S kS^jXcJ`[euvZ~,ñ5}p+DFyglS[~^nO-X2t5pKca.! `}XG|?%X_W`v? d^Q-dE5jyQQ6=l9`խ.<ģkл`T-L72qp7<`Y;O-j{pz(̓q#}d ܽ*#]wE7.*ck9`lfo)-\!^ycBLaK ^6zӵbƈwplCkjWefg[Bʮl]a!dթR>;PN[;Đ> q7KYkpҖo@BoIa!eɥz+@ B#8e꨹pVG?Gf*mҀkd}ғuv !F XY%o+ 3B%b ~dA2;ʕo {X#3:'3ȍ~x^KbV퀘=p`ՊyLF(*86VV  ;Zu}t\Zcz;aWh.v8ո:23Zqo2nI;k} i#$L,iH=l;dm "(VXҢțx@8s?B`,1>0_㭧Bi%e9 +w"\okQP,g M/MbjvJ; ol'2&o$4 h; / }ŵ4^nɡ!)o^p| -2ڢpF!`vM/T>jGj0}k#.i#՚|B6|l\7uC8*~rHeܲL)d +WW:xm4YOV}wR-d#yF|G:B ]w^91{>GZmƨ[.Ϳ{7b2~ǺAegTjWЊ) +/0Fsb⪜v$-U"km?l]ހHR؁TU +c(Zـ1jMa۷<|8-y(u;mͶHuy?OU#]ǏR+~˗/?O?~_O_7F__O^g d?_᏿Lߨt AQ[dSAIX8}'o^9yЩ4+k6 @JVx?Km}o#=M^2q\MUUj" ]pW'$ϴIēT0Љ,-?CA8 ȣd z +i diCΚ@ o=:Γ+ŷ2ة :`Ή<{|y~`c5̀2B!0|%{m tNM1:bӲn+91jnhOX/J\i* ±b)T#ɡ3ۿYk(x%"Wri]={/?}. CL]-In+o7`= oEȬ=;pr0IdKe|OjH>Al|,(;Yy۬6@}[+ P8

    Co*K_ ,65rv$Kqp럜2iΙ%y: ԏ[ { !sނi `G[}WAMu3y⳵Ȥb=3,_o 8(*΁ O5dK@׭w~*Ķ;+S bĵv,q'[zkNQ ϡĚų6JMPrg}_6٥4sЯN-TF8DZI[gI >UF\eL6v>%hҜz0p츔ImVƔ߿>e1+'}W uA=mVش׎%vܟ"&O.! {<g,bټ4SvSYb J><HNpMeZ)ǥy|<ERVWh2735Ig'Mjsn#X h-5#4^P`Ќbsn2p5 prluUvPA|AQmq+p+Ksx_S7'DETh wx\W!goIJb|@z'-EB?wi$d;UHpD y(v `& :;}G4x?}Xi0SMhvN&}[oLJ^!/mar⪓2(#NM ą5!}\v?#W#γs%86ىwq5 ^@U7QpKK[9i|NvC'KP잻 ;f6iEC`ӡxH6[Ź2z/oC-h[ UMCA!?D=<𫹝m) !I*uY5~Xb ysesӓo{tF[ߝpE2`|&f)O0<]ίG xB ^,'Pr̃kv0uR/ۖ W2ޥ}p +^'p\nE2Qa,]ItǮR].&jHy0{1@+gWe@Gw&=J;r1Qn;wӿ@*`Ƹٵ +\|_^ #c|AVKwS3B7w)t1t]?{vzGæ!3smLJXz6;o7zsf8 dܠ0r;MŪ $ +q^G -s ~ؼ0mjAp|"-NSEZ0+w`:z;?^֪| N#X?% i\ӏ\(Z<{;.Ls iN\'x7\7mX4 ^n*6AgDZXsӚʣڭ?Z-љ +;AM`o=nD̵+±LTU{}ђidUcGN.Ed|Y BeɵS(bfZN6S1>`"uǍn,kc'lB +_4*x8E>-M8ג{o~GtScHi}ޝ!mf+؜\V  +p6/Xv|ze3Q \ܹ2<WpUot7%^޹rbd.|C`]>NlV}Y6uZ5f;O.ε-h*-֨*^yA(>Whuwxgcv㰹lY&94}KUr'erؘ5bmHd*dz>5[P[&RKGuZ+ZQ'0GjԦ5}6\;(γɤNI ǚw;^k~IN+f߯C}A: 3YFPFGP dU)|ornXΫe8rZ/ +ff +QP'ܾQJBv" 4+]fyuœTww-D{3/kn^a9e%ue:%D4k@*Z_\\’Ji#}}:6w:ENIž)(e{̔DB:g`{w=W0H^ JՄE xN>kja fKv<ۊw3@/ڥߞ15>*lV$# ,X7h1 4M6_~WS:Zx&xuy P'M3(al9 +HX,>32Ϊ +]y-wl6 iLhTc>8GmfTU3x_f)<9$G$_-3mƾ]΄-q"n;ωZV'Jm|P$<ʼn_ M8CX3a)Ԁ؊l[Wטota=!='y籴RiglzgoLB9.<릏n$ CRjTe7V陟YwGݰ2뼮±n*slH4t={LPN"}`-J /Y-s# Bpިfg{xhp`%"ɸ= =hgxi\}m(_{0hshnQ#-К#ߪӻpe:Zc%$x K+aaҹ77Eq '| ](^OdM܏f!WL6,:rG }@] syױɫ& ^=w\qOHS.AeE6,0N԰oPWaC2^ ;3n5xnm$hmYzc_a$9<p}Tu xF,vL4ɽEաξ u9yeX@7ꆑ}PAčiœYvNȓȆ&(ble o6bu8ci  `fdV g>l0XN\zWh 5[5GFk`;ZmvvIGXD F\ +csLn/õdPB܆bF]$˘)$^+}:-#1AܒuwyQFplWˎ-GAYYKtBȺ!>}F환p y;>&!-WlӑKKJ&Z@pSPlerY(e}]+mt%(?k_h`3,&H%ms_bц%I0VΖV5POgZa ,5( ޜXCAG.chnpw:cQDYdw`YlZ,' F.)aژ9wp0F"B1{ Y sI6ɘ:n,K6#Ls,;A"4WWR7Yj8s'0!h/5>4T痿/ۉma{%'\tn?rdH ֦K-Zhc+{iR;Laһ`(IJ4)Ce b\.Ռ@DGM6琨@o~A|la^/EbbE Ϥ`v: +AWV +s0no8kcsL] ]:Y/weay]Y  X)ZQyf**ɧK\]:=? +Rf7p/q3[8Eg-BdO6Y}#1{Vc))T`T-wT1 0 xfP<1}W/?߼}?~{s޾}}/n"(}u?El,MȮ?짞/xɷ_x(7?̊^g8d?Ōvغii~e[:h]L8W;HĆQ)Qf7qZcl{U8VU\LuΥ3jeJtwDc܋W0)D@fL] Yg +  3m{MrOXBn]Rbt2pSԼÖ7}vkĘ +P[ Ib{u\w@X?ckԮ16͵f(z`T5G5zXE>NGo̼v|I\_5lm&ذ֤kkYčÈiB33 *X(]SgE^Ɉ8C)1;2doP%4's,FRk၊ǏZig㶂%α;O Al"SPlxDX,K+'8+1.W+04CT0djR̛!lB,\b9iAk>F *L0 +xwK^b82^l-4)M^sN[X|n)E;9#U-@5%7W&.5:{%ѹZʺs +BaE-X&Bߗ?\C h yxrgZ]wK#k7cw<`m'2.‹ݣZGST~Mv@0 V#܏PfHcLO7,jZ#CcQ]q -(PZ,TԆ )zZސE Y9]nߕQɃ `* ? JjaVM%^$%E,(Z:q)\k6" -xa'vjNh> JZ뛸:ƻ]H}ɋL%w~ѨsU j}X ͯ{$ЮAeTeg& *| +*a}~ULH 0Ied -cK>ٙB $N4pѮ~X' ė#`ٴ[ "H9vYoC5?kWCYpIfW}qq7 &+C(]2R?醓GȔ6&-市%Mޤ|3(5clxR%p`sɎ7_n7HlXηw0>1[m endstream endobj 13 0 obj <>stream +H|ݪ^ `wRh-˖gSJ)NKr0a%yyoSB'ZՃCC5/{퐖d/AIUVA!|Ҫpޏ h \Vm dKV'Dפ\D)JI ͩ8d{kzpvttK֬갫 +ͿxNvQp}\ +JǛ>1ji?$ |~½:S1œ *lf,ZJ"(8N8`OCS%<`Z4ߤH8Bd" B]ܟ`nU7,.2JS)%,%reewêtzln# ^As&LvkEjpNWzE=ب7pXE0Uw]_}x#|~_ ~r?w?}|L~_|#ʏ4}__@?=w?%=>?kE-CwP?l߽YԷ zt//_ 5~P -(AOip0W 0y;h3BO #0wϲ Dyja72/̃WEmoY^ BYV]zWs%1sS, ! &u@`kYL]U!o$(qCg-N`5|Ø=ƠJr\S$(3q*:Np7]#VX.}xĮjl6.;gqjh|zϲB>*1jFpzbO8t]Z78`d1]e%/:+r6уCI62rBZɎ)l琙 -`ڞG@7vбud:ϲ\ca)&Y'ʜbA)X's H|2|,O6Ξg3Wof+#iU5ǎ9!ދBkG;IskdXg#pj10*8`*/ h ٮ+kotdmm6|d0js(ܼ<ո*Q8jHl#[mcgkɂ"AXM ձ@[&f(s*rTMVWV v): 0Lцɳ ]2*:z]0KLKav&^8eu hV-,L݉DW +wiB͇yTgUf۸E|kpGrK],^PF״ɮݕk {_0/,;|][oC}S iĻ|SZD@#IԌ5ծc&]1<0[X*=ױkop/< gLΦ-9oW%t[{^P1Z\VVVA\5A2*#&]ax|:4f"rEbKH zm95kZF +:4Ǜ3d[A}NKe9JZMI*Ҁ9BGԞ{RtrQ Mv6b݀ . +sPL3o@PFz\tQ<&CEW5b $QDWzYQ{.1LU ED0?c˒ra`Q?5WmMbK ֤%op(Jtԩi0`æff 퀶:E@~:5 b\;v5\K) '\|=wxs..H}-5b\S7׹tȥGj.AAxv 6hꨕ-ff(hvׄP?{iVc9H\,kL'$7QH_!J Pe}x~ac&[ZÔ{\eb}w8KIpbn!ѽV`QsjĶK::1Zja³cHd̥m!1K6,!+#ͥP5tv'@CUjaLHcBZ{E#0)4]vWOEwDSӜ gp> xBnl -˓,/Á|1\r,Mq(Cl$ 8jqI=i Y _~r+{ɳϺ"2)팲+hm|a +۪@!%UмLЦtyc.D^{¢8]5c'9,] Z;uIgs̤) zmۊpx֒`L"ln>>ʭ֪77Do  G4 Jh0i!] +PAJqV؃\\![!a1ʪ<ٕ>bb8#cp'}9 +aX֚RW*{[]Ǯ qd53W[fqisRbޱ$fUun +HYw,:@0X;?$ͧ v8/^+_/fͶe#㘜p8eս ϩ`*3HaVuvt"{W~i}-~U#'YaQ̲.j}H_2Ε Wf)0^l,oFJ* +;^e\ܡWPՌ69"lev!q K-v: f41ہB[qdyc ! ` +w@x/v8G@8*VM\2p _VӁR-L:;8d?&B.wlfQSYx7;u3FB%-\nѨO, +AOfX e'{hEEK@/qou|+9ff/f8n1g/"ns|}Xyyvն.<;!=/=qa>4bڸprӋK*T{@9^~]Lm|+FJX_U$yWCۅ~5\lM״#MgP Z[.}wuSOF'SqFOsmOpeo:!t% 14ԃsK.eL}"\ 8uyhm5 Gē>adGv;pot.l{Ziy%Aّ[_?@Wd11 /_̆aN' YGz}>AuBG螅yc U(>bWwi/+\kO`0G^~Ma0s-uxn x͑o,7yۮm IH͘g]g:|rIB ӲZ+:4Zk-^S Y|* +X;b'jb/?:tYv}cQ5/ + w7p- zČ0t+69ժG&a̳j콯7u͌;6-7ɡGl. +7|׶ NtWl5f?W{280ҲƄc1E-3]?bNb#h>vNrX. +<- Olۮ?X |aKU\_:FEHcO{`>c0!-@4tJ +5 %;B7#<=#CTŠs_"_I*H ⑨0S"m̻O)MmMjħnx$"OpZ3h5T. +h`QI+sb6ȭݱe1ʆ.@p*:tA'wKPDվ;nVV?2#+ws&B{04bۯ1b/ڽ,pAʾ@k6S8,p Ͱ XnRP(lsUelyp)oW2ە6변P@Y-iC.32AclU;V{8rdҽU cO;c;RXkXo366F3Jc"r%D凢t\K[#ysW;ZkH +1jWO?Y+N}}ܱHNkX5ONw";kPe{J:Mfk܏XtOT{%~ 7FZ u z)Ӥzh4Qؗx$rD dXvBT FgǷ퀋4:=)K숇{ꞽDb8a,uE"&w6 FBš^ꮫ=9b''ZݒΡ "@40$ƒ]5.(}˾D+ߍ[u#}"}F\}bQ;<)5@\ ײp;f#z +XebNxXhmſ&3يV@߆G!O;]"_fvǣ$ѷe+'=y>XK]Ͼ.Mg^v">1`O8hV\o޲ǦJ4L3V^m=\W7d.KwYw?nx%hzVtD к$QW5c9չ0}4>1k6vX|$U;YfOo&evpᩝZ#/hoE6i.YWwٟ]xxzC )/A"̅B> oHa߽@)g8#m2{@Iޝ9ܓ?6DA_ +~X7_y&\PmI~[,9ѨmK0x83g C+i@bb^- +%'d?P{xbb=)QT'T^P-بDFɰiUJAKwFIGڛS H9>aB^S|/$%Q(l4Vy\Ff[gͩafx?L@t0eeq%t2愠2PY`4=e9 6M-"< 1x`D]4Sq̄|)eBqiT8&Ց6?@q(UT&O5sRlX 5xUƁaƛkȰ1(i9N${@$wHN0Fmkh1h; +ύ@qt;Oˇ}5'0h:\^fs _ +:HO:V!,$3`z%v9_#4qP~xv۫v۫.vNH {zÉװ}lvk׽vk׽vk׽{^S"H*4n+.+ڃxA6|&nC`QI{7*:A~n 3^Mᔤ .yihi\zXNXkE2Lt6\,e%8Vw%m?5jx֭ӿm4T]S.~tMtv49#qBoQS$>l]0W@(<̓'@Ab,ex"It䶄=1Hv`$2T GHlȍʁK];&-9><;ICJ2s:eb\U!KP$7],9+u)F&uЪsTM|8$-MQ.\Sh5i.-Fk-U֒kflZ=="w`$.~\zX֨ϗYNboUv|;f*]gK3 Aa:[ 0A~,灥͡\,L^#FhOmu?gf:| ]?C4n9hsNN.|vgxCPp /A6U'pn^}KB)*"gj#G,@q)Y;pIW *}jǂ#ځ#(4"a7! ƃ(H)au,`xg,_Kh&KgDMpO%$ueUK{AlJC +Bd'~mv 1xCWx*~\zX֨ϗYN< 8 Y9ҿP|_ߪ$opNK HˁB{B '^Ӧk*x.C * &bG߀g9(JYaҌd;wc1Ŗ yv ? y`ishF<" 1ȭ0Z#SFGO_P\'qUE{suBu;=5x&H G:sSDĦǯ +yaj,)Xpp5E +d$J8 /;gG/aDyr0#I\ [k;w@ p 9/rb5EkU \)6ծe_fah-Մ2[dB޶D|V>kZg|ZguV|]"bSꋇLk`gxCPp pdJkt۶Mm3zm߶o-9>)B+'®\%\db.wdEӃf830e.O&R)ȍllfW[;+j h#'u.ןy3:@kt3?S/:Pqw` 񒨩 ]@y"3 endstream endobj 14 0 obj <>stream +HQo8?A_ڵ&mq~QI׻&G[Q?Z#9{Jr$Lğf8sjDo1z "h ^kMDД An? ?JSa69l5€OJNw j?~M}I2x#-a3Xs= Im)i(0h]xt¹SE񮗉3< km$0'(8ӊ/*5AZz8p8d0y1\|(3:ԛj*+#[lV.Fˉ>R?^ > =d1qF3Ъ՗2"BވhCL{jV,a5kj<xZF!lyX\KWzT`i +wGVds6!7dq$NdIpJM\ 0 l4+U+.<:㄄Kngz/dՌ,)Z`@Ќn NN&D'fV_G7 Hnp7ܷp a'GCω%Ϫj<~G|n.<&Wcޠ@?AS52Zjftkt~[:=j"&' Qwexwz E+n Q,Ji`=Lk0a w=o?&ƧDOa+Y`:˵:q*hD&^'a6OWƭL4\cp!/ +^4KXٻZ |SF!l#lɳ}1whUM~5)v&!ڿzaӿ`ޠ =ҍI`Kf,gXOw+G jL&Ƨ Q'Do W}"H}@ ~¹SG: `:iq>WY +ޫN47iC5gaSIn83).:>kA -FnOBͬ n]A(1no702OƏ KUx/ |zitEt r(0,;n%j +"O CtuMrU֮,$ɋC!5螸mË9<Ҫic<“pfR4]tREvљ;P3UV2w܍ÅAkPDG=x]¼cJgP"{N `Q`I]QCր͔pC=;+\D'^9, +؅&^vOu   >58D×hⴽT80wz؎wt|Gwt|Gwt|GwP<_I])V!Z>s( u3)zzA["~)B +UB$ 2>[i=r=^6./tYJ?6̮vT}cT up?N/e6 DW $ Ezt"XlIB`BgbYr;ov5m-)u6’P^Sn Ԩ,cݳJ˦!b֙PVJ{*psbC2} JڟȨE<.7 +jw 7/ qiK=,`^j +{LS ·)4BI#gTPu E@2T3Gq@S.d|XlD^w/_q'XvG =kI+F\7ӥRމ;;N[.mSrn,uqtu65Z>р#^o B,h_Sb8~1E$chF6hOUC:^f&=~- ,qNf$X>U~f9᳑KËAPBIh/mi{ +WIأ>"W[{V*waTp!v0ŘmwuO_fa;A( +QDZ3w +9fD+?< endstream endobj 15 0 obj <>stream +HWnT}HuwU_`İn &kx}NuU_, 8ލLMwu}So;?,'bzhrv<;ip|yd)~p^,g |wyqWYq׉7|)Ӱra~Ƀ=g·<wxx6_}K#?鯜rWjD\^ꅵ{r||զgG덽ǿoqlv4ԱS 8tN6>5 yrXoQkl}pb&&mw/hHݾ&9 al1[Nգj}[Ϥq[F?m|=ܫwn<:_?EwWecǨ3Gig6(/, Ljɥ'l}ڸ2./:|?=a֟L@O-7͝"Wtz~ZÓu>|iKDi3_7;ýo˿,Q)(OO5loby֝fxXr}e?7T^B\3"hB*bWc,qry+HhbKcGb⛟ϔST)9sI-w#e}MmdY*?|jGӈ@{1V(Ru5(BafƘ4fdR PRk5"cn/efP-)ޒΔ y+TyP'2I|$ CbKm#o^ʟHchZ~ {UM K]V>!ݭ{8дIJ 4K m.+lP*nPH/"\MhZiCJ! + R;x)VzQ2 +F̺ +ܫQYפd.leUTs5K'a3%8`U#Vd]E"`HpRY1[+ufdC\2纰:&y֣!!f{k! 2^$)4Ī>&ou.pOXzQAK,RtAm JmBs:ܹ;0ӫSh{sm\8׆͵As]\ 0׆˵rm\(׆Ӊ )t +S)66w`Ie8:ywuy39s^q3\u-\i]FG9VScPt dJ0UagZcC!id zL1h [sZ,$b֎w1-`BE/LV`_@W! vm[䳊CVdZ-ЁJPZH}T-hZ(r򦴊5fNb U ʥbva?,:[iT^ɶ +bzu鰍R)3ϺAzIN+'}B]p8Rh%͐;m@'OV+]}L5WSV核zR5־xg;aLrv?NB3*m0Q'G~v0f@}%׃D5N>V=8S72 l 9tN++v  ~]fa-B=]N"kcNb=MRDƫnWΣ>yH6\'\D B!eS]=Nlkgj&)p [CrJ#V1(]đD J!D}pb R>C9@h(O ڹ3^_ ]aTx=&oCt#Q`L &t^QDrI`Fx־F̠>\ҏa[dSI`CLEĪL^J?瑙[Kka3Wl*D( EABz<-8rb*!V08 +ɶpOְ8bxy4+j a`5DnV"@H(-6a[']dI ۇަf*v$1~&ȑ" ocQkP +dtI=B MsE&tZL&en.-_MEB<ީ\G7N#Y&~7%aNt<:$A܎JrQk)k: ibJ9%Fu(`X"+FjZP !j΃ɍc^jQz]=Y͋(軤3/}SۢJ1hG޳gmѷD$tU69C&{y|zx3HAJm:\L#"8^r(#$EAlu pj!=Pyfg fG6/,'uoT]`ם#5[4;AAl}|ԡȘqx ԑh0Z=*B! jrn6+A S2$+ +p ~3+es3e|9WI+=QU]ʛܷ .Y.y[Csב=StRlťH+ ⻂XPgAR[8Yn3lU +A ^{ ermS 5VP K"wf:!tvœ zbT#":AR acXkl^aQ&ьMPZSIe^&q5Dfu~QʵSD'(Tn4(.<;W,|Jrs +EX6V ETmm7$Y,ulk{Jm{Zq飖+labB63â.M_2QV0<&Mw+ e^6vZa7߀m(1\1]4N +1#&xJ+~My=zr7 gwݸk:f⋙8,Xc|xFU*j{e'?€|sG22<5Eb~v߰@6Q~8^aA ۜ홀dJ6 h͊skS65n =?&k GE+[)aA97|m3זd!p'}iB\TL!NwXh͑v44njG>zۥe %kr_K7>|3`10X]8n~& +K??×zs?\ryb+{OJI=/|/?zj`*>i   2AcZzf§gtd#l~XL7w#WO"c='CL䚇n1}k K'h/pT/ЛK̬N 3kfXB5ozm"t5?[|d_YN6x]hXA4 qS, !Čǩ9i0G?XuU4c;O1,؉ɹJl0&]P}y ݟV]-FIpI6)E^C$v*?KX`<9<XB"1 NX;rlY}jtĒ靭]]娵4`s01(MYN6b7܆< ;E_<*a!N'a:Yc2צ![BJP:l^/㱯ۇ딽ntGRe}w %Y&rIYx9^V#>1dic/5ߨxwK_]#R> IUq_,zFI_a,{I_P%97ww\{s \[='a6`q #!9j}gz]zi\ )2YX̖eσ +re8<\MyzNvB0!Xԯ5$a3[:'^X2%uq.QLQ6Yf{uoN{}][ԃXmA-T%iXB-r +~Vemй͐j{, K؋+ &w|'A`?)$\y)B"XdDeMjpFgb"_d[@)ks(5mo7%đ"Y2;9q2|s%;ߪC_Ռ9LE"1ɺyIJK>'met3 +^8!oqe#(D PJiL_Sv@5qAı<P>Z{ BTԋ +AaD$ʔ`gєG.S`W/ vw`Okֹ:psFnښ&cpP%rd!˿}wo~:~Wo߼|c%Lg>_o|B^^TOt|9 Kp6--{ +6Ye7Xqx_ ??=Kg%ϬB;t|m:^̓_| {w~_*ſz]p^36=NK.n~p2O k[Lk!Im`+~h}i2E@FkOfOxvN8̬ipmHLY# mM“<89KM0mnCZuo4/nzOIAQf#;y@ a6 zm~/1F E.[MNޠP;Yҡpvh>8[uJP .,ZyEx5xG*.l{ՃAOWue4aOi̙tJ\[͌9 +kfd4=9!L!&%د,0~+f~l(._W8F˥zg]+8R-\8ZQwgPS731J`=+s KaC@cnT5ڈjSP.=xy>źВ#2uyY]ْT.YEZy5N1NsXxe9,:9W[ `lȭǩ[ez0X,ia^^6Cg,0M00xB 4e:ܜQjk qˏVho5)'+̙bW8KfbPy]LmFa*78vK9q`3ߔQ">M(iud@if'egh2t~`H>d era F!0hr^qc涆qI۹~A0ٛˋW/}w}uv'sSx{zbm_芯$OA/TٴTDAq +zI/|)LG?wYݗ-A6oW!av-\}<.=n?vNEZtS` <ϸO,nɍhk^=V};/>{?uGNu_G#'{x^_􅲥쟾?Q?ESlNm{BO|λ T8_0  NvAE5664aqr #ڧOd.14RFYҸ)Sb  +.blc+)C6fRF"i515= ߗa:,xZ*auh砻ձ{2.["Ne,!֌}d"6_ rK: +y5ا~Ղµ * WzܖBw5{ʖjhˆqI9TL,)!mD9 +m9/Xm<=?[ۊS uXg}ɋ>bkȁKQՈ1gAze;"@w(B6"$A߆BJ3ϰkX&iKEf"c-ݵDjU#(KA1g`^ m\᪩aB)A2<BU#ƒ[*N\;ځ?үy_AC@H)b-AE%:cFjuW0a6?k 6c a=x +iQ,#j[n!̸%Nmw[p9P_8 iQ "Y"_Iy1V,g3]MT/ة-ps,;q6s|JCVmpxp%P\qBa5Eq3nqAK\*lEN~IJ\\AL 8H WԂ Ip/35=uCڢ$b F3׸7!v#jta3P9}\L$j=9`:kcSpA yu=nRMQ`|[nfVvF>[3i"^s+A 'EHJ}u^fr=O pN|\16j'U䔪.9w~~j%Vfq.W[Wm{pw^v{|uwW7m_^zF˃CRKLW)~CxFerϔ;[~(< * q-?#:8G+}-ɓu|z Onh{aaU!=d8d6^+dzWB^,ҹyyMmrwozRi^V7w.WG VB!L \k%=7J+/I =f42Pn4ZESbT6[!vMНmq iZny#eRJhղQ_e?җztq&[uH/`Iv]]]2i$dA[|m,M1 +$2/x鏀;/y4A'7bN)s]`yj + uACRЧ~Ͼb_Ѥ=\h@Ӡ +w`9ˉh`d=f1Ƌh9T ȍP|H;HaxsNj@=[?p $X5R@iU;0Boe=ArPe yp?ri\zתvK-dUaAk3)X*ΰVQANױ-:~ s_5ݵ%qާ9T&y]Zs_8 9jrJye߮7W~k-QiK ]c0 QxaЯ7Jn ˒m^CKQP3o5g ƹx$LUNq +!:w2dטgjr5V ̦50v$@RhݐX1O;R;.hR7$v M✞nLp>.N!I@`0P:R uZ.SG+u-<:=Zu5|ʗjaikObmN9Zðp›?98 v~]ӷ%mkJ`$&bDYKT0訽*֡wϴVEfM{i4q?>1Кb*~__]3>~Oo?!r-J?#Ӈ~_06L︆N3rSWܙ]:EԲr$QR +'بl+N(sR,*p/?(Sn`T^r,P}4HK>B]z*4 L\ tW~K S=~|8O.1 ޔAAr +/ڹVsP# +$;pmB6[׉!eT,OE˪$6N=U?`XX&r\Z IOH'.Wo*; yvy28̮ IK2wkwrS)P 5>TsXTVwڄ)\r yi6V)0zGšmZ)Y7XA5oy`v2v#YHrd;Qe"N4 y\Z1xk7{ʘMyWhE>_X\?S@npaa*P<'YQ$ـXX be8K*Loގ")L,+ RAB-ĕg@W5j٥!J\7O +%^?o]M>sybRN,3_XV Ǵz9-sMW8ms,hh7R~ FS9 qU@qH[Q5Mvy!vR1_𿊫flA=)΋vn'ͫW¸ p3T/PDp ZF 7ּVT5f,QPr l_)~Bn +6g5hRC"_QT P{"7TŃY + v9Y(;4M!*͐J{^Xk3\R9rSSg +*)\~OѼ;Ŝt^ tGf^)SaN. _DJubXhij^ TL 7]vחLtiE:2(=MnoS.;qcHªh=E5nYA)m"xf:T7S",)O1<8*5j,nac"IVJusTR&"B{gHv9@SN`9 ١`>C\!blҭ$&5ǵG=+Baű^fQA ho|00Imf<N+0FSb4hVl] 9>=oUVn.=1֋#삥A + :$`IS ]'`: I.WŠf6CC|Fs CyE"?ޚ |.XR[p#mݡ +‡X#Dk W0}OR8NcΝho4!,4Pj;aAmCDK\2^癧a+6siP72w P/`.t. +F +TAreCByn1N;/C_ĂnS1T h kc1~8Us"@A ,Szuп 53ZJRhIPHw6{DckEXGuʅ7#OIPF>j+Լ9l Ćb4Bk1. W/TjU\U%bJdr{8Df=PTI3*|yОرO%ͺcLƑN#bfls9}[NeX|59]tγ䊍OX7jq"qFtC2T,H,] Y5n lbGyXvt:{&zXMV4Rd Yf=;r(mہ(츠> o4#[QU#h}J2qvd;+Dt9Yl~haOG@jqh,Vrt/%b?f4an?zGX/Er'U#ܟ#ܞgg77~իopO>_.PXwl P#Jg}~@VKu=_kn~yB_Z_?Rszp%N?}HAOᗱ͡YE~^ljOL}T +yҒ' 7n IQ1's0QH!D84 Vh7BB/OiܑON W0e>rS7ʍv Jos1?V•HZ.ֺn:r4VF%!N!&hҊf 6vxpJ@ +cYusP(Sv0Ha"Ojƭ5a'Ma9*Ic!On{5 1d9VigY50jԝYl:ĵٽ ~*f嫗7rT%hA2`{\0F9mo۬(hZܐVZ.@e0evhߨCRJ>f*€ɡ-)G۰<#;haR.Q\إ߆Dȩ6d=}nw`|wo[='>Q?grҖIe3CsuzK]>.-|'Td'o*{a9[0rj!G;1c_A%1dYqRv36JFRN&<" |`7T%eYiv’W{Fȶ1Ʋڙ>\j5gy jGh]=N]xUmeH[UMFF\zDi>q"ƙE6"Q9#4j&=D$Q57Z[i=w1xT`w%(« !Ujm!ܳϥ,:"L.wu.Xji2K?}$퉨 (Hu׌h|D>t]G`- 3, +-Z:Ϟ:;*'q(=m]0DYk(,Fe/mB+u6*0"`0@qrp3 9bfCšpcԟ#q!]_3J~}xiKi-ې['ёu/xҒUJ!7{shU>F9G2fq/ӚƆ9;z9~5g1)"n΍3*E\kk(z+Di#>D,n49+\4FLf$qwS` #|n|V37lͱD5W.KD^cޢBkIr2 + f\58ǀc܇-201l0'<;6?6\|DXgE]a`ƲK4E(Y3Ō0ʊ!#W19UDn4M7\0UKNXsǾȵ-6Z}IþsD7OR^N3{֫nWۈʾR8o<ϬY[9k7PIt΀+ws7%ְt0mR,})8KSdDKԄ,LOQv` πѢu 6zkIdikqP|~*"'1r*b2!YCa0^4f?ځeV SA(!`g<pS78C:βSvZ<,E|(z$]Pgbpz99xn:]Yw!wRgC=iTY>};sӦ⻳i|Vb4..0YE=K;hGf,HMJ/jLv"ӣMNR:gl» ]To> (~S8JZR  yDC%"$UcܣGGXy?P١Se-m +Ō :cWn/<ȞL0:iMc^PGgSRP,OÕ~kߠ۝ۓ_w?v<~?ؾyuw-m +Oc'P ?`쟧 '=2=/~S}TհG){8˿{y_ciͻ/};afLwau +]I1 i8$$/ZY5e'X0M{gzPDcqAHN3kq M,BB)Z?@Z,83 *dyVIv Ige/$ˇ炜 oy.G'{zka]FȲSHzM[1"G|>tM{ªZAS ;F%((A˙mɳR4ײnV ]FFIJF**-К4|D)0}rKc bn!]a!~ + 1 p`It}obnǡSq#X2]yC~`u+Yf5hl&wa-]6OJLKڹh7Gŵ@۱eršnrPߤ#۔_X +4}XԽ7ݴWvN[qÐ醬 kuAdXv=b fҵ&VWa<0r?CY^"S%y#줲 VpQ[{w.H<1wY;dEC $9yhM&}cc/ F\`r̵<9r>T9!Vt(. +iR_qt)IYx=>_E <|yk_d玐œĶs*gE o}di]ڀ?M$KU#~bvcJހR4NvFHyX! h CM8I k"7DB&+FyV`,;!;Avf9jQA(}c¾]MD󛮷1.j=_=ϗ+{)=_MpOO?[Dyd]5fJP55EWla{n5ae +6o:PW03˸3H.|wHa$!~8T6'||Yޞƚa,}RJ:Y-F[X;(e6eY N= Sؼx"y9M^BIU Z/ Us)l[!+ !)Kd + ԑL v80^)*VYB=ʗ1yYIis8m>E*'BFm$Z&LHrbʺҖ̓?f=.82[ +/{՛ѹ"!TbR/c(ShЛS91n[-Fu&d֧ IJr0yuN+QTLaA7meR!<up\ho9/[]Kj\S]XVҤj!ٶ +M`QT2a)O\DG[^Ēuzl +lKsWu#b$00 (WĬ= 0h}}85n=->stream +HWn] % 29L+6E\n\aKJIg R ppH55-R>JZ~Tz*sōURu#wja,-Ke.H囃& hV $9dn0fx6f p]63pӤf!u!TF򓆋æKgܮaƎ-UTQXR,H9;pW?/Y6 o7tV B 9xc{ܹw}dtypq7W[B 3<ʔ ?SþCR zȣDq(C}laZx0NO^kLw??A:Y͡hm r̩g`{ɂ)s5>>^2j +)O|`Q)z­H|At^q7m\kkOO^|>{<,w~:]~:^}]ᗖow$!O|M_a֦?_X͘umZNH(LC{ҵ58F$̨7%KSO}vt\d'7kDݹGVF>|p||u|FTdH]UK +/4 LHKhJ?cI.*zZeWmtUoc̻md֌Q5#pWMvF&sH4MViz +DMp)+nlB箸|䊡5d7͛j! ){M2Q)k@^@#zo2z@{(h?@v#.v!gCPB ƜLi"54 郐XKJdB'٨P m&jNx.*Ԉժ.'w6XA;@)sl̔J8K6$HʚV2XeP-RMi KaaYQy\!j-mVT.udtV0R>h쮓Q2 7ˆsujQv22:jTT⃫q$c}a 5ehL+dڳBC9pNsێ[|Zbrr &yv"痘xdݛOĮvw?~x<_.P8%m^[iD3jN˩w1'!v,)wV;IGѸ9`.J?Тhꉴs44jqMXe3VŒ:Y$o^ݥLvFQPds*B̎ 4[uZo. }Vu_B'@x7f_#ꗝ$caCJ`\9b>؀YCe%kxcJpa;8Ez hQ^Ȭw]AZdhPBB=rF2x?B4֤ `$וc5YUk¿Ug2,Ӫ8gQаDJ+M[~vFеYV`77F923Sb$J2\|xsӁ S+@'/{C): e%<10`Jxc}J@,7,GunCd?hīM2 :BnLF硆I{3ZOAAZ)lPǻ? ʼr}&}ќz6-!t1$*ݥw1S쵕o<9jjdjDptα:<ڽrjM*PA;+Ľ)uv%f`TW'wd1C"mH&~átMtUKH@ГM +PbQ');jXxbP3%%] AHzp|)jTec~ݒ:{39jJEU cPLj* BU5U:Yp+0@Ø&!OiIH\Hz= +Y&z/qmrAT:dƔ` {e{&Fj*1*Ucn5ED t "_l4.%v6Ȧ_lUOw87Fy"8!881lI N4 vYî>22c!议jA5o-~>?`Z|j0Y,&s6з0dw7a ĪK cav{cB;xvnF/6^jf6Z-b4<"41Π-T$V /(IpPpae^&Th8dpv8]PzvLhA%Bu`yXQ}eK`ib #&{i]C< Y n:13q3`(0x!:M^Ƭt@`tj5PG?/ X},֦7Udp'}ND$(>8~.R-a7k"ő-NČMpz@0s_Dx`Ub {:6-84[rFqr& FYIq;NV4rЇNrzo~,w}a9r8??TܕBp{6P#qӑÑ8l[4Pԡs Lt +W Ҷ?F!&άVRϑ-yu ʑZ`"1+l/\SِyX> +U2zhqcۼvJDm>q,ieLҙ+O4'/?{ Py)%cSw xS; CcRꪑIOACQ0P;"P&`?\i|<~@Fr|,#)RjW`\Nٿ%oj k0DS Ht&;SڪbG$ Ճn5mlp,[qc}a$hjٯY^է=DTm[zanױ7&y 's J +\>&qBHc8:vyyk`sSK^ŽJDb{7!Wtm^>5߂Ip=Q5c{'7_?qonE͛^|߽~JOq/ɇ/߾~||՛ts/o_>^} `3}K@!Ͽw_XK7OW aITXIRgiCrZEj:#q(=)C^`lKEʒЁj`"2 Mœ: +EFT∘$>kc)ZO$h@ya1F '!RL {HD;u尯[b5`74N) ЮP=7[ĠcF/O̍ԲFib(d8%j 19˯g2wkEQWr +Z)gX4tsfRD@)J]Bޑ!kcx`¹y%|\50D(Wܯ5؇Ujc}N0廙uqu^:k=G`d|ed>dnfSŨv踑Lr$Fe(7p%Qai,T)̬E.Xe3fTggYJ.]Za;sw,+i`&'.:hޠ2s8)@%?qR'Hܼ(de >l t")38 v0huiUؿ}ŮËVR:^gqtA#3] E;3HDhirm@jۙ>Qk]C|CM5 7`cwHYďZG{LW瞪l8#.ټx[魢h&@/pQ}U+r&/([=2ի8wo@{H !( +Ȁ9yU+Y>>kXrOtcvvɈ8z @DgXLCaqN=qvur5UťCǎ[)Ѐ_|/T>)de"ّ^_8]&5-@KiG<5 +D%i ΄]@ 0 16~Ĝ$j@$:݆<%K$at=w2.~-"F!Rڣ& 4ʹ[MarZx^jy *513DQ)Lh=[,FrLI\1[ytMU4$2h9H[9Wr/.W!Y{xfc`H(Xi"ʼTl!״@p&Y,ݧ@3Mg {pT`qd\Vѐ4:P3H)Xmec8)̬^kdԦŝ~NpZ<==}C Ef׫fۇ?TnxyCpMj)!>oA5WP@-cc!S][mj$h{;~>jFK }/:)"bU݆f!*?@5RU~'nGQU!8Par2 NnL}& @4D6bL>2Q&ȫMexSR- {}@ ra\O1Zjd])CYjqywX2>49 E +YCW@¤rr݌.TeN<(CZgER,C;;pC<;g5'WZN(9jf/h!>b5py r0D BOiA*˸&;!QnYn4lgB[J;urYr!5Gq$ 43Ԥ(,{T(7,jxYXаʫD}B8jVoIϜyJ#=4?[C .gϪs9nw"BDG:J3SYQ"*0 CϧA-C=}዇wWO?{,>{u{OkWxɿ|ⱶ͏O޻Gϟ;?~|wO^Ʈ>/^=w}طw߻}wXwDrO^<~=<ه?W 5~0@kO,t;n}eƯUϜ*vmvQHI`ޗ1] 0H503#O])\kOzr"HiEWMh(YbLB@ `S&QfIn.1卷`D ++ZU}VI +p|1Ԑy?Pkk#TW:t{6ၥz^p/vKv\l"JҕFEVOTMgEʙ}ȠH*%@n +\U3 eRq[e&isuDUe9$^*Ym^ԯwTu| +pb(T2G3)T:2!:[ȑW|^HrgD>4 ʼkX`6:"C-Nl-Mjat*1/RMڠ*Jח7i~_QcʕR + +3,z Y̹ƒV2ϕL&iw+ cT{(۫uC4k1{$Z4PZO˹1CIZz R|sgцI2ft|ސ\&81 =d/ymzT!|hh M|"4gۡˡtg`u{-%H/˒"?Aٸ*;iEq#P0ooDFֹrzɮ>2#~ MC8M)i`pqH'm7 !~06.x($D.(OU!^>'ELɃ䰧` UFJԴ4~I!5 IaygEq;hb8A\05 &[jh%pg]f !jJDA (+78W-BPшq N9b}Rk@CROAUS,?j5<tjʘ CEM@4Ӛb-Ƽ0G0N;fI%`ߵ轢XL V9@l>^$8χD j1>G>+D@EkLxvEػޔIXr'ù rYւĄ6jK|3Q~G+qdɼc[tSԂట?޹sVg u +=R$u׳B3]!GtЉ{H%`Iv^4s@̳mwvAّ>2 `|R*X9^wГasf`W.Jt4Y]9܄_&`\1]3H'MAO1gdrm)u}s_?}P krg.]~hżADF突ҟ3ʝJ\ZZAEUz %@Y1@^KuهrytMk7U0裞b9:y'溾mUaxY-L ƨpR4#[s}3o~& bjˠɰxŃHa2*Q +! 2{-u88@٦Sw]tC5Xktr}C*,^L$D+r0iP0Mp96H]\Tmװ>D0hd-V#;j~{(3Ż ӵm!4q)f;ԞwEO.>dbFGgkǤk+v +"z2Wܚ-=BXMm{Z=!_HTwj, 3PbЄ$4wLś%E F1 SA7-V<1dT譂[WP*v +M*pR3c+(5ucC蘓>@ Y%[T /hMxVWU[u0ټ&)p? P'w9(nRS;.&cs1D05Khz>5"[&y`== %Sy Gxj +o*()otI4,sl3Q,~᳑n-(AMM5& b[[@`2梭}h[r/&~ +J\:83L*:ah /K 4CΰR~'am02bj}Y3ߝWUv^Z{h$}2 8[&mg}dqzKhXV2<_F2&!)~µݸ- nD:J@o&C9N zz1N[OhQ9!]C0U ^,ѧQwm#vVuiek&TƈhnD]U5җ=VkOEǖ=!b“uwM)Uvk:W3̢y[&1aqŀqy2YtOGCqtl /MiǯmbZh>@&ĔɆLGat*pѼXoYV] />8=Ԁ/,oވhWN  xP2` W\^N夣Ǔ&K=-R] 褫$y~_(_M7k6^)@Kp_bBfQq }՛ ']ev80GAkl-o5?PЃ띨ӆ̌y ܱ0(e`^#^]&Þlm>ƒ\?%%qʧ1ɾ [MZ>-*<;|ZT¸;Ul0̣;%dJLqlg{i[ Nt7hGW1c]N ysoA +M8ŕC^[ѫg:mP85hT p+ hHٟ.yEN+O1{4i(>q$"yk-vW-_ӿySa04kfX禔4gxʍk)}0tvQZr.cKBrxi3gZ{T-j Kkk*U +iߺ>x}CUfH^W=JY¬Ak`wt*yA+? +$[k$ l NNOE=!gj/\|:Ud>lW#a^TH1Ripwi\Z ֊{ԻCW|%cHU +'؄{HA_gBntl b6+Jmtjڽ1Y%ֽ6u=h2P' ڀEg3鷂]*k=O],+ưW^[9Sa[]ׄxKfl4Jz1^-*mR{18c;xE,ޛ -F15!U*,Jk L&t jDf#F+*twjZZ1mm3 @FvY'`0" +dyFH7uqV@9pNٵA p_ B $Pg5XC__ ! +sxOO;+2t˓?GvܱEײith{vp'p}y7+%Al޳p};sT# nq۠Q:}TujuM9ڥ%'`Ɨ0y:p[m^ ݐ~-HjE MQػ\Op;ԝcaDbm0HFޗB_Kk + ٬, ^h]{&QHeNjZ׀ta$KX M1$BjEz{w%QqѶ%<àlti͈(-pT+i/1Vz_(J͖o Vq=N|kÿkIs3Xa%ffrX5!:i2Q]q0C<DW7`܀Jj{8/[pUDkA !!^!. &FUl͵L'^{H.\Kx)o&&? \Q9)Gw43v֊u፭6sI`1  Ac6>6kr8JUR$s>>`kq?eAO=zMPlUlfGkHڕ GTU=*Q{(AdUy\*D.oJ?Z> +CE٢,kVXEJ4hf<)cWxæu] Ti8&hY%^ug4DJx0Ym>,T7pH@|6QT F +=VC~*{*j[쨢__PRYU0bޚD0ih[w\fB}YVլ9e%V 3R3yd|I,Z3!d̻$t >mj'P+ip3c]ұsԴnGEIⰶ n N^ƺOC/v9iC^GB;k-ƪiRE^S30|y\ݫ.x-2OZحb9`.^C p6+h0E{ ?L-WZ3$ߡ7X%;u^3[XЉUd4+- +㧟"9 (jOϡh,;'S3]ִa,Y!:B;A@BVOf"Iݹ `EO @mF ދ M7ͣ7zLp# Ι`SlCIp\qk~ya3Nvs> 7Vj%L i%Odzg2#a`ڷ/^$\'C) -J[EM+9 lb%q) O M鑷 G!NnLۊLu9G1Ӣ$HNhFFT~)5"zzZ孙ȶSizl1 B~0Skm8"9W 7p9+GҀq'lsp q+p<h ׂpg>r;DcSHX,Pv ](^zj@$E%M&4̳T +/\p%j LkA2vy4j VJL2t +0Șė1`iͷoTYpy +$sGcel# r~f +5mdCgbpi6aCWgDmsAm"h rI{N{r/Nѝ0:#wޘ~3Q.7& +M%+l:TA+b|#3KK'4By,ڣf8I| ,tbt&Fgq&wsMX%c_ח4ٺw&+#ܵh'^/966|#bɳg>q.qβkYb3SԁT}Ц @SNk<\NU5iv>y9):IIםۍ\{8Kq!Xyns976Ojqx8x[dpE ="r@3->` а CXS(bdTvxA.lX~ Z"ўnRuAZ#-'e-N`zs +F9Hut3 H֫ҋػzhiLh;ZC{Y3E;6gKR!Ik~O/;tk2c\8M}y$z_nVh)l! LJͻ]1ݟ Յ`(7޶8"70yڬC{(Q0Xy _p i r$4PմMNU:mW9i:0 4 fK;l1^GmkӒlw9WB{k8 tC~ 'G ~j _}7;;`Krk ShhDe,$ *,^Jdtp ”d]Ay9`Tp喵RBǾP\yڕZSӷSm7~k^^6i4p'6np"7ǹ_CQҐE5wTrنuxnz :3s0jʽ/a>eta?WhO[zIؠԬ̺n\/d嶺9 +;I> j#@n>;a5",T0Fp Ek{ +dں r0`+nS3S|t 1`D]Lp_%c(Ĕh0¤s mF>-\pk~u}/}ۿ~?7xݿOoG}.^.+GRN0_)>k}M^+(7Sͻ<~߿?7߽~>~ +O?l`__*xeIPJx2?iP}ྕ^ hiAǿ4.4THxK[f=A[ +Z ]dc~bH2O*CKLs;ڍC/oaؠӊ .`=z86#pDwrJAh}bEօ^o~6bK:VAfXHXtw937n_$%:Vź",T$Qw +"ߥ-[ RX[;>}cAH^ih%(1p>ڶ-@"z耀cPf7ݨuQ0, m2SP;uC*TԉT56hSb HO۴Q4dH=!d_a8[6Ke{\D/^!CFQѠW4YʵkЀfʧKR 3 +nmR'^qMJ::&~ԓ5~J@Hlu EcorLOrdLR zZׇJG*LéOcM !7р[=.=(RveU4 wёqF'Z1HN9/j34pJH m-ط |#‡ÜAMUvrձO|tv +{1ݣL&( j>צ8![>2Ŏ $qEֈynH 0O#h=lo|S&Mt=9#3;ޞ#ϵiFI+#A KV-2qP $"J,oQ {02(Jb kQ.x2#{: {!X3=C/"1C0`sɡ7OwE/e?PDon}+|Ta'_>C/szw~ûon| +o~~M#@ lx-^DQ"VG'RA) 8uJCJ)39,RxWO{Ż6rt^煋#c׳]9Frcx1͙~.]0Uu5cYjmVapd_S&D'4rdB HlkhzEZڤ!e w1nbn!_ J_|c4M@p} WG@;j$͹6,9 WB:@b5J hXQt\urI1#y̳SOk22Aiz'ق}m,Һ24YW 7 ji31CSc2KU{ l}1$7]Admt80Y=١)]Оݔ:K w}~4h2Ok +^qaWf03K> T@CVhПQݠaɿmDTbAy -ݐs^(a6W͇* +fgD򀈔-Fq(Nfs +4>S Qw>@Nm##бf+RC!C>9rk,«2QH"bRk%,}"-l5Sۥ=j˧m"nc{>8 )(C_2pynKO8#H~e3:B,UIޜz+(+ƬbuEu绔爑,k m(AgʡG!P,R\W&TރS +,)םn jKXvyBhB8DnEMXMYzG7`OFpߊ^`]%P aQVH.sԧ5QTDX_uWgE*8 +lJH(AgȿG3gYIJn1`@-X9 O WU;<Ұcro Nc +֡+75[Uhxp]Emǂe7 Ovm[cv;Uݬ駡'J6cx ʨ͓*GjqG%W VT`1+VߕahtaMjB/-윥ɝi#;ȴ)1ͪ~utYTB5GŦS!pw9shPG([ =/eGh9Rc0,abs06gX}zym. +l=sw9{mq~\Tu9S.6 !WO@wg ^vuI5Eo.q^#]4_(&>(ʒTJgl"v;2,DT.GS ߃q[#.+h?06+r,ⶵ EƼGE`JBy2o2 Fy}t,>.d|h3lVwoǏO79%淗o>|\|w|AϾ'?N]Hd &H2Jp]hx:9YO[,htu/SnVvBY7mJj[qk_T_# J+rJb*7SicjEJC^J)@4ڣo +Kh?F5hVkǞ*8)2%ZcS3O=p Wӗp$Ah]SgXr/BfIy,1Jt'q@>ƺ^^G0j Y+.b쾋cν=SȜ-Z*_,`Sڬ2$-Wcĝ2<ӿP\Bvw H\t婄un0ߕӁ!oH+QeKʁ6n%<l\ڝ;q>l)eF^^-kmA~0߄tUun]AXv +R;DzwVlYAA$8˵)vv9{kg}QiMAmIko>mJ6%ulTϧ`iXY5<LpyJcyM/ޯIČ4t۽hӡK<|YePht^nI`O|wv6SmAH B,Z`eDL2aݸAQz'p&P.40WuP &ޜPF ;jSPf@5>|=Nm0y)丫`08Hܢ{=@R\hjYO;N$Iva+ЉR*RQJKj߾kX(dXcԒJoaJyd#1zXх3'S{,BEԐPS]9Ě lGw6],wذBq?U[!ϛq~.@ѽ iqwֺ.L}6ҎJp{?KV^ &Tg\8LfިF^e4&TmMlky}ħ 8n&Kzv qD^yDED57T:%P`Ji06aﭭYh1nߦ?bX D()Vtigt7/Ǒz ?4Uv|3`.ۢ"W O\q5 X&J쓜_=nCku.idCl/á(@&` É͏O*Fxߞ> _|Q󛯾_}է˓/ÇϨ}'Wj  TMA^?_[pr{+^1fɕyL7 ӎ%u(evv fڷŎ>T[6` 9VɲbkHDZVx'@kOL$גɃE,- z RyAI SRF32+[{ҕBE3o{Lҧ8$H… Ê &:.6u*'I)*aL yӰHF 3f=R( ͛^ĸ1b*r5>:F!h1k4J\N X`|}8е=ZЙm5d3 ;@J Uӛq0l2No; +SX[*|I%-;GsөL0V|SS3 +έoA1C.guV.2@c5ȽGjӎDiHbAaNTKu &)߂pq3@V.2|:H@h!/ g;wuQ isC,TXtAQ(S[K=gS F{Vտv?_G'&zUfw}(`ǨYcRar_krUu|TlCgc>?ۆaQ-"Gc 73ƍ}~nĀ?&Lؚ4eh]c)lmȖpT#jM\|6jlh{wHW1Eڏ1XxfJ{n% VELLF3K"#:s&p,Jq-h V4L1wh 77vȭ ++c< avǖX_bp[Ƽ}*ou;ҫVOXz)w#H|L!FЦ9/("[x  9NMȬ !qM@0MpE&pKb> ZbJ޻[%Im'dpjl}t;KRE; da4ֹ C<ˁ{ +6yP!/ChhP`n6eR3ö! k)z AFH-+0NW/\V{ɂ]'nA?dؚ}M]VJS`KhFQ + ܋ +ȝ׉1 "Gb:Sл\b3t"Cn +y,cn}6#\6R`#U,)B]fC׀d ,%L%tAy+~Q -|Wvg: NOwX1h'cm7dc= O:]23nm"FȖ(U–ދ\#H~F%]"v'X9Q!0O`tb% U8fK]8 b}I㬸hN+|ee^0[LX z;*l+[_r3.O.&2o +1<13~/LS,SՃH9wzpiq٬G|;' endstream endobj 17 0 obj <>stream +HlW˵e|$@q&Ը@;3, B5~6xDpJc  f#n6T__ahB_["K +"JlFBBZ(ًѝFmVHУh9CDΜQvϽ)̨4wY\ꙙms̈́4ފ.%9\ h1־d(nH:SLmYk14peC(Dߤג<ŵiu DR~ES<O~ ,b$Œ7aB&:O| *o[7)ȩmAf,nڬ|y.#4߰/CyN*oGJs +yR9#VBidkI,3n>+%#/O}w0N 2S/,AM +q7e<>g^kpHʽUA /x0Ou1DZ3qاG$)GTݹ9 Ƨ('yN*`eIN!¤Fevj;;5(?m} KstyJ@);e&ǨL' 22άRlWO!4(=ma)+/8\V;7F3ߺTuwb쏒}rwE#ȖWȌ 4RSSZ} 6 Y $:|459 WopH#sc7o9Wv 82&B&R^KVEswP"߉J^8!:a2S\Ml$5'V 8Dki ?+ {KDd2| {䲧bRMhmB1@bI};i2Ϻ<)6+=+ QTb܅YC M^ e . 3qSbot_顤z:BO_9wv{PzUO. +9w|ԩ>ő&X +x(pȲbqKf-w!䉢ɜvN Z﹁W}U<*kA"N!APz.4vt%EgCSR wS#P+W.Q(u ;V 6jJѼ To rh;W=oc=yy5k} [&fsa+bheMK,ߢg.?ԯ\7z +EϬbz: Lg= ϼf>;4s !H,J_h ,O\dkcCQJ18&ck#&1\DmDeJ^H5$X,$/1CB:JƯ&V=KdW Y܆OkFk;ᦔ'Gɜi&|+e<,@A=jiwQ]Xft C,72;iuiQOj-J"!l{"Ai=v1_W"|xQi-,cPuHm=oʓհ,_@ttC6JYg͈tue^~8*FqSq_Q&%'[</Tdb"CZoQWom1Hks(aZ$*jgΙNzs 2HtVf(3u>N1y&wN5B??d?"CI sV> ee =D or ?X]q$֑F[`sEɂkVq*:7HP;]n< Ҡ9mj9n= +͝ʹ{g8Glo饋Z±Coܕw]tY߂C7⃍#tb⟗`T#`3IR,݉{(:e ߍi&,}捫i B5Ft56F鵕F;_O3zTL XG&]6|5IvtWw@u5,+gȢ4돋ܙXDEnyG+X~N "gpF7BWƴhI^t{<v,c.D[&} 2B)ٿ Rx}fW\xJvUIΜϹ620ϮԴ ז'][~QY/'M 7Tڤsr7gJݿE i-梗qOiȺ2cĻOPՋX6 +ϓ'b QR0Qk{-tm'ñ75x>n9qH Q~|IboDb[tqyQ\㦈RHs'|Xm,@II8bɷ*WDˮn*9v|*T:Tzh +g|;\fG>6[ S1,em +h> ޡIm4yviy:rЁH>n=\Z7<^G O#%oQ1 LuQ{D)wA]ʣF=VoV!iHb[ͿL?ȺŹ;&'jJ)q +7c"W$[hF6aZJJLº p[O^AZD7QAx6q0k3BCMUXrTqa%Y} e;ϟgKhY=߹B-sZ=^]&Zت2Ɗ@?FML St$ߟW$ +Mxΐt &OՒuH ڹٺ9𰋢:j ICW4UZp +PROlј'`?J2_yoN,-_EQ9w)~.qZgĥpvה mkO~~8K|*˜`NNZ$ X;k\iL \=HE!Ve2"gvc||s[hw11Cl&E%XшΏ.-t OASg:j뤉F;mZ\`qyҹ7l p_Tk Xx1Kja&/{Xo&ΨF-Ai\C)fN> hJ&U8Ԭyi2c:o|o쑘rͶ3|"ݬOD׺EN;saC%cqnj,4P`N-sR~; zJk`5rMq8 +$&>w6hF&?qцkɉ/B@-nT:vPvNn_fjUbVֽqӌi1M}_%FFs$`(H. (! +ʯeI oC|^b{zK' +>t)CdTޙsB̌l, +(Vۃ + Ӄ,hI/4tu0FdZ^ 2 %N(ڈLbiYٍͨģ)5GPK18U44sFw[K +HI).H +м6,( +zZ<hQ#r4ٺ =Af\-Mzf+ޫMGIIʳ1iҔfռVH-9]0ap Pʔp᧻_:S˺,bI3%Vp/ N!T1n ]䬜"{83C>pYR79O/ }LMɧ=h@w +RFg᪬G> +}`{q[hJ ˶&/T>ivV 2\9x0UҋPSw[Ni`7 K)57ax<1p}'56D+^x_>|`»ޖBd<{觧fh>5b0 +uӈYzWTQs6cǗy^.fa%.Al 9^\ɾ*eL&M:]yxvo{l#,>K#xie+¤{G30M +פfT\*c@(<1` ?$6K{$K[v(- ʼY=S1Kn@"XnrB8HUxXNaTA-cm w$̌ ԵAB5Pu{LP T(6٦o\KDZ8!wϡ.;P(`ڰr*9֝$WeYKplJ>  F~WyD 4:&,3 + ;P=J]Еqϳd,Ԇ96HRH1u 1yny?E. %,|vi] kS'eDYɘJX[(d!Sĥ@rl``~@ Yup4hyr\&Sۚ7Pfrh-;_29g\楠nyM$!;mxጂx %4ohӀB @af}q +λĺ:iDbUg S$p/IŸ@K& o#:)z >\ЬgQ̪@'-2:TܔPaxtIsiuӒ[^c5zJlmN6'pΟ?9I~`>|u}G莧adrph;PtGtIa &7ON]k4{xa~Q^,5ͮ$빵1/_W>巛vοߴG~֦x8V9 $/eM 912]˭[痢GݪC,M8* ZI <2Ϭqn$4tEN($kn`ԟ%n;ILUy;\;o p]0毽7ޗ4K:#_fIhxa]q'QYFdɘvOf2gŇ3?9 tO2j b)BdA_Al91JK^QV3ZcfWzkb̢đ/:i$> #GOe-\.! wk5ri6W#OmbLp .EǀB+ht$u^71|9i)*iT12^= +:[j4רU\Mn5x鼪]`g(rUtV~wyH8Qn" ^7kK1(q0L̇E$wV+F֪J6i5W_jqu6˔~FyB֌!W@[X!hM`jkIp\V&8͐V&8aYUh檥&ؑNeT.#T8:w 7N#?҅'P%{Lkklx0v^NpLY{Ej[hmn7鴣h$;d[I։uVi& +XT. +RMꤹD3=h42{k}rڪF Y"+$+60H-i~'UkhؔwQ>ꛬ?eX8sl!_/+o~ QqZx՜{bQxAՂ7<|3rvIs|.I`|4gE(\l&3,^XU>Lzۡ?̏S-ztM S/ 5 I~AH!\jk>őzt>UFg#׏.|^!.tv͹Š%W0ܑ1l.{ \EOK= +[pqz1a%̩p 2,3Qyb)VTmje +Do|DX*"/Xz p1kaTiCGÆT%ǣq=} XD( f2p*(r-Hqp\S +9*[!F H"v'z +/c}B$`ge`Mt:k<)C7-tkWşXB)%5d"\%U&ѤLGQ+XnZʄsXɖ%WU\ݢT*A^_'`)unGr!ZU1#H*7%'e  녓u~L3#;BKK)#u5";*H%ٺ8g3$&ln B$1"FXaAg"ng&<~w7jyly IJe՗4""ѡWc0es+b"|h)Fpf34OAnѤF(eYT5[W͜FG߽y%w3j$:q'b_g*9;+{ij~֞+)ϒDxa*alTNiQڮ8c^E`FV`Y{41ah߯'aaK S.dl<(olv( +o mbY/0"jã( HJYE p'^wat?8xAu^"N1cY7T [8?y } KhdEh51૘F3Kk[j:i]9\ڑT8ة Iq\A=r9ܷQf:½!I?G8 fE:#I-˖:.w d*Vz5ڽ{=[,5P+LOrRBZ[?_zWx*QljvZ;Mu&břpl !CTdE\q v!rc}`lily(V)3=bG}5G=4 o=mJlWhlHE?#x}NGRg+9eV7cIJ3Y܎4 +s:>VYrzGDkf68,y",6߹>BW%),xna@ aNhΓ5gW/nj9"@W>U EFݨR!Zi#˰ W$"9 Letm) 8щYC,,B#5]-tf\*>P:9f*LAυw15܇ |]=N6GhYLp(,x EH׎ZG'[# +X6Z#២ƨPx|KZbR-(5Rϝ;e9uJV<ښ4$ur-vХr[xT%NuR/~;v* +N>hiߧM֦°#+Lz77;;o= y=8P&WyǬ! 9jRM[8rwqpc O1qiр&[6]M{:TΤ~(\y2_R=ՃK}6.{jY-l>{S.fxP@$i +v93> +%,*i T,lDF &\~L1BݙJtkZ_,A0"/6inGAȏUL- iUjKUŸ@P(&ivP1Cӿ3uygYXsCO7%x|Nb@wZNA'7 \{=-ۿuNⴔ?Cs3*ꋨYp;ݓן5&M=.TU# xc@0_G_KL`b>WPP_#{j뚱^z˷8 XGsi.kqyj7=gic`(`*2guձllSG]/>5栿;PZW6~Z-lo|DQ>% bd*|S01y +)9S\0 d3Ml)&jSV5Y#~0R vB&ﮱ?[1\6`vMČG 8 c̱ U nVLY.rnU0͸LDg} +[&PXL5f-eޓ9 ɶb̃ߏ;=Wxa @K];Vt #ؼ,B7`%92}6m~?b9)F.\bsc yE"8l$Z92@;n}dӦ7~ !kV OPH@^~W? +> +mzAY* GI]p7W ӝ[*>P?Y+-jiw U+CB&DTt'MgbZ'M$ +Zw52\bLP-m^T˵ldVL$2lyrŮrΖ['vS'[)=UL\%D=q naV\L %p:g LsSbL;OϜ,_; +w-R BLk)V!%/Ҙ:”zl냭Ty% 'C Ć8S n b'&L +#x#X!z' :iYUwdqkzf +eo%c9^&my͵z[nN!,_XT n6(*9[mdRֳUFEujh%CH+l%:?nn-𠖸(/'2P5$*}_YAB CJ1!N ~F&7kL%J%"6B…!W Fl*ڕ]H\c\H>C0L-zcg@Wzn 7^1 7x‹jp1T #9S9Y8}QS)O:Kpjsy>G<vV]{ݑ7v>Oٗ[,qܫiO`V9#py2-kPjaxୋ>q M2Ac8f~õÐ#-/Nn:뿡c2{AH>N."3aJvkL%.궝t ąU^YJMz3Jqk%n|[LjIx@miяb +F:pթόKh)ԗFDpHƂ2H-OpϪ" P3(6A7[g`oΫzoM1m150^ܶ&l_n(Rs+D%!KEM}l+ncΕc.:ܹzv2Eikꠤ p} KnJ;PCy)FG-xuu*ZL⦻]n8?ofR'>/QlwrgC![Z,|bijtFDLz{݉AJܐѓjSa! eKe_n._EA97W!܃KäkE{N7uPZ"}R*e/[y"usPb89`f>k(54&CDXsbٳeJ4Y~s]p&OXy?ն*D a &DE@@5^4Jzy7ݫv+2LwH )MM2OOHN|T꾖O,>Upɖ1p `g͒@< tS$3Pd>e<>Bm#XTOg&lwVBF(d2gCEaiW]m/.o$3x%|E gʣgkNvOAҡ}O!ow~w8y+IvFZEԛvbl̢Sw3CfcuocGoR\ F"Kз$ ;%bg.#Wƶ7#/0 rDS2.8 )v +BX\bbiw">.,\F~~H +趔yd*Bs 8 O_A)CB TNXJ,I]0t:l3~2)ǤSiS;h ,TT@9k ?)G,}8Bm.Cb%ʋlpp?Ykwe18-xd߅08RKtQR\h-M&'f`"U}CjoT x NU^l75Djk3K:jǛhn$T`Wu]Z|./(,pAT!BiO8%u3{F?6>xއ߸;wgW06N1NT`zS|9 >$Zwzd90(En#jud ڹf2eXW2Y Q1}gW!&_lobN%Vxj=\I̻Xcu؅𨥅15n$4B辉΢43׌^-9Ur:6 :O{b}lnw'$3soھr*+jKOv7 + Mϑ~H]g8~(\r*5D h=yHԺpVR瞸uĈ^: ދ.>y㌣|D@u J\'Ys /YkނՑx܍bcvTj)9Sq/yZGńӁ/ƀ`zcTj> Ifȏ-b13o b"F F*'M6zsc07, +\h*R9Cq8ZFމZ*֊Y|!sP|B?`x`s endstream endobj 38 0 obj <> endobj xref 0 48 0000000003 65535 f +0000000016 00000 n +0000047811 00000 n +0000000004 00001 f +0000000020 00000 f +0000048074 00000 n +0000129852 00000 n +0000129924 00000 n +0000130307 00000 n +0000154490 00000 n +0000160188 00000 n +0000181053 00000 n +0000202901 00000 n +0000225104 00000 n +0000233450 00000 n +0000237159 00000 n +0000257233 00000 n +0000282454 00000 n +0000131230 00000 n +0000144831 00000 n +0000000000 00001 f +0000048049 00000 n +0000047862 00000 n +0000047933 00000 n +0000047964 00000 n +0000049662 00000 n +0000050302 00000 n +0000057587 00000 n +0000049775 00000 n +0000100574 00000 n +0000049969 00000 n +0000059456 00000 n +0000050678 00000 n +0000050806 00000 n +0000051129 00000 n +0000051287 00000 n +0000051532 00000 n +0000048483 00000 n +0000300546 00000 n +0000057847 00000 n +0000100850 00000 n +0000059723 00000 n +0000051601 00000 n +0000050831 00000 n +0000129291 00000 n +0000129339 00000 n +0000049242 00000 n +0000000148 00000 n +trailer <<252CDCE32BEB48F1A4AC1838A61EFD15>]>> startxref 300713 %%EOF \ No newline at end of file diff --git a/doc/src/week47/figures/RBM.png b/doc/src/week47/figures/RBM.png new file mode 100644 index 000000000..c8dbcdc3b Binary files /dev/null and b/doc/src/week47/figures/RBM.png differ diff --git a/doc/src/week47/figures/exam1.jpeg b/doc/src/week47/figures/exam1.jpeg new file mode 100644 index 000000000..ba4c2edba Binary files /dev/null and b/doc/src/week47/figures/exam1.jpeg differ diff --git a/doc/src/week47/figures/exam2.jpg b/doc/src/week47/figures/exam2.jpg new file mode 100644 index 000000000..e48127f98 Binary files /dev/null and b/doc/src/week47/figures/exam2.jpg differ diff --git a/doc/src/week47/figures/whatmeworry.jpeg b/doc/src/week47/figures/whatmeworry.jpeg new file mode 100644 index 000000000..429988ffa Binary files /dev/null and b/doc/src/week47/figures/whatmeworry.jpeg differ diff --git a/doc/src/week47/week47.do.txt b/doc/src/week47/week47.do.txt index da515f66b..c2e39ae1a 100644 --- a/doc/src/week47/week47.do.txt +++ b/doc/src/week47/week47.do.txt @@ -1,12 +1,12 @@ -TITLE: Week 47: Support Vector Machines +TITLE: Week 47: Support Vector Machines and Summary of Course AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University DATE: today !split ===== Overview of week 47 ===== -* _Thursday_: Support Vector Machines, classification and regression. "Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember19.mp4?vrtx=view-as-webpage" -* _Friday_: Workshop on project 3. "Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember20.mp4?vrtx=view-as-webpage" +* _Thursday_: Support Vector Machines, classification and regression. +* _Friday_: Support Vector Machines and Summary of Course Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) of Hastie et al contains also a good discussion. @@ -14,29 +14,6 @@ Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) o "See overview video on Support Vector Machines":"https://www.youtube.com/watch?v=efR1C6CvhmE&ab_channel=StatQuestwithJoshStarmer". See also "this video":"https://www.youtube.com/watch?v=N1vOgolbjSc&ab_channel=AliceZhao". -!split -===== Thursday ===== - -We start with our final topic this semester, Support Vector Machines - -!split -===== Friday ===== - -Friday's lecture is split in two parts. The first lecture is deveoted to a presentation of possible data sets and projects for project 3. The second lecture continues with support vector machines. It may happen that we will need parts of the second lecture as well for the project ideas. - -!split -===== Workshop plan Friday (first lecture) ===== - -Here are the various projects that will be presented during the first lecture (and possibly parts of the second lecture as well). *Titles are tentative*. Talks are approx 5-10 mins with roughly 5 minutes for discussions. - -* Maria Emine Nylund: _Lego Bricks Classifier_ -* Fabio Rodrigues Pereira: _Financial Machine Learning_ -* Markus Borud Pettersen: _Machine Learning and Brain Grid Cells_ -* Jing Sun and Endrias Getachew Asgedom: _Machine learning-based approaches to denoising microseismic data_ -* Felicia Jacobsen: _Analysis of Breast Cancer Data_ -* Simon Elias Schrader: _Predicting atomization energies of molecules_ -* Varvara Bazilova and Sergio Andres Diaz Mesa: _Glacier Mapping and Machine Learning_ -* Gert Werner Kluge, Hanna Alida Fossen Hardersen and Sushma Sharma Adhikari: _Gamma ray signals stemming from dark matter in the galactic center_ !split ===== Support Vector Machines, overarching aims ===== @@ -51,7 +28,7 @@ medium-sized datasets. The case with two well-separated classes only can be understood in an intuitive way in terms of lines in a two-dimensional space separating -the two classes. +the two classes (see figure below). The basic mathematics behind the SVM is however less familiar to most of us. It relies on the definition of hyperplanes and the @@ -186,7 +163,7 @@ We can then rewrite the above equation as ===== A $p$-dimensional space of features ===== We limit ourselves to two classes of outputs $y_i$ and assign these classes the values $y_i = \pm 1$. -In a $p$-dimensional space of say $p$ features we have a hyperplane defined as +In a $p$-dimensional space of say $p$ features we have a hyperplane defines as !bt \[ b+wx_1+w_2x_2+\dots +w_px_p=0. @@ -235,7 +212,7 @@ that could be chosen. Our objective is to find a plane that has the maximum margin, i.e the maximum distance between data points of both classes. Maximizing the margin distance provides some reinforcement so that future data points can be classified with -more confidence. Figure 12.1 of Hastie et al is a good illustration. +more confidence. What a linear classifier attempts to accomplish is to split the feature space into two half spaces by placing a hyperplane between the @@ -253,13 +230,13 @@ for our data sample. Let us define the function !bt \[ -f(x) = \bm{x}^T\bm{w}+b = 0, +f(x) = \bm{w}^T\bm{x}+b = 0, \] !et -as the function that determines the line $L$ that separates two classes (our two features), see Figure 12.1 of "Hastie et al":"https://www.springer.com/gp/book/9780387848570". +as the function that determines the line $L$ that separates two classes (our two features), see the figure here. -Any point defined by $\bm{x}_i$ and $\bm{x}_2$ on the line $L$ will satisfy $\bm{x}^T(\bm{w}_1-\bm{x}_2)=0$. +Any point defined by $\bm{x}_i$ and $\bm{x}_2$ on the line $L$ will satisfy $\bm{w}^T(\bm{x}_1-\bm{x}_2)=0$. The signed distance $\delta$ from any point defined by a vector $\bm{x}$ and a point $\bm{x}_0$ on the line $L$ is then !bt @@ -277,7 +254,7 @@ misclassified points $M$ and attempt to minimize this function !bt \[ -C(\bm{w},b) = -\sum_{i\in M} y_i(\bm{x}^T\bm{w}_i+b). +C(\bm{w},b) = -\sum_{i\in M} y_i(\bm{w}^T\bm{x}_i+b). \] !et @@ -313,18 +290,21 @@ where $\eta$ is our by now well-known learning rate. !split -===== Can we code this? ===== +===== Code Example ===== The equations we discussed above can be coded rather easily (the -framework is similar to what we developed for logistic regression). We -can set up a simple case with two classes only and we want to find a -line which separates them the best possible way. +framework is similar to what we developed for logistic +regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way. +!bc pycod + +!ec + +!split +===== Problems with the Simpler Approach ===== There are however problems with this approach, although it looks -pretty straightforward to implement. When running a code for such a -case we can easily end up with many diffeent lines which separate the -two classes. +pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes. For small @@ -344,7 +324,7 @@ $\vert\vert \bm{w}\vert\vert =1$ subject to the condition !bt \[ -y_i(\bm{x}^T\bm{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. \] !et All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $b$ and $w_1$ and $w_2$ define this line. @@ -352,13 +332,13 @@ All points are thus at a signed distance from the decision boundary defined by t We seek thus the largest value $M$ defined by !bt \[ -\frac{1}{\vert \vert \bm{w}\vert\vert}y_i(\bm{x}^T\bm{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, +\frac{1}{\vert \vert \bm{w}\vert\vert}y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, \] !et or just !bt \[ -y_i(\bm{x}^T\bm{w}_i+b) \geq M\vert \vert \bm{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\bm{w}^T\bm{x}_i+b) \geq M\vert \vert \bm{w}\vert\vert \hspace{0.1cm}\forall i. \] !et If we scale the equation so that $\vert \vert \bm{w}\vert\vert = 1/M$, we have to find the minimum of @@ -369,10 +349,10 @@ y_i(\bm{w}^T\bm{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. \] !et -We have thus defined our margin as the inverse of the norm of +We have thus defined our margin as the invers of the norm of $\bm{w}$. We want to minimize the norm in order to have a as large as possible margin $M$. Before we proceed, we need to remind ourselves -about Lagrangian multipliers and optimzation problems. +about Lagrangian multipliers. !split ===== A quick Reminder on Lagrangian Multipliers ===== @@ -505,14 +485,12 @@ subject to the constraints $\lambda_i\geq 0$ and $\sum_i\lambda_iy_i=0$. We must in addition satisfy the "Karush-Kuhn-Tucker":"https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" (KKT) condition !bt \[ -\lambda_i\left[y_i(\bm{x}^T\bm{w}_i+b) -1\right] \hspace{0.1cm}\forall i. +\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b) -1\right] \hspace{0.1cm}\forall i. \] !et -o If $\lambda_i > 0$, then $y_i(\bm{x}^T\bm{w}_i+b)=1$ and we say that $x_i$ is on the boundary. -o If $y_i(\bm{x}^T\bm{w}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$. -When $\lambda_i > 0$, the vectors $\bm{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. - -The support vectors (the points that define the margin $M$) are the quantities we keep in order to make predictions. +o If $\lambda_i > 0$, then $y_i(\bm{w}^T\bm{x}_i+b)=1$ and we say that $x_i$ is on the boundary. +o If $y_i(\bm{w}^T\bm{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$. +When $\lambda_i > 0$, the vectors $\bm{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. !split ===== The problem to solve ===== @@ -551,13 +529,13 @@ To find the coefficients of your hyperplane we need simply to compute With our vector $\bm{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via !bt \[ -y_i(\bm{x}^T\bm{w}_i+b)=1, +y_i(\bm{w}^T\bm{x}_i+b)=1, \] !et resulting in !bt \[ -b = \frac{1}{y_i}-\bm{x}_1^T\bm{w}, +b = \frac{1}{y_i}-\bm{w}^T\bm{x}_i, \] !et or if we write it out in terms of the support vectors only, with $N_s$ being their number, we have @@ -569,7 +547,7 @@ b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\bm{x}_i^T\bm{ With our hyperplane coefficients we can use our classifier to assign any observation by simply using !bt \[ -y_i = \mathrm{sign}(\bm{x}_i^T\bm{w}+b). +y_i = \mathrm{sign}(\bm{w}^T\bm{x}_i+b). \] !et Below we discuss how to find the optimal values of $\lambda_i$. Before we proceed however, we discuss now the so-called soft classifier. @@ -579,10 +557,8 @@ Below we discuss how to find the optimal values of $\lambda_i$. Before we procee Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined. -Suppose now that the two classes overlap in feature space, as shown in Figure 12.1 of -"Hastie et al":"https://www.springer.com/gp/book/9780387848570". - -One way to deal with this problem before we define the +Suppose now that classes overlap in feature space, as shown in the +figure here. One way to deal with this problem before we define the so-called _kernel approach_, is to allow a kind of slack in the sense that we allow some points to be on the wrong side of the margin. @@ -590,18 +566,18 @@ We introduce thus the so-called _slack_ variables $\bm{\xi} =[\xi_1,x_2,\dots,x_ modify our previous equation !bt \[ -y_i(\bm{x}_i^T\bm{w}+b)=1, +y_i(\bm{w}^T\bm{x}_i+b)=1, \] !et to !bt \[ -y_i(\bm{x}_1^T\bm{w}+b)=1-\xi_i, +y_i(\bm{w}^T\bm{x}_i+b)=1-\xi_i, \] !et with the requirement $\xi_i\geq 0$. The total violation is now $\sum_i\xi$. The value $\xi_i$ in the constraint the last constraint corresponds to the amount by which the prediction -$y_i(\bm{x}_i^T\bm{w}+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$, +$y_i(\bm{w}^T\bm{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$, we bound the total amount by which predictions fall on the wrong side of their margins. Misclassifications occur when $\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of @@ -620,7 +596,7 @@ This has in turn the consequences that we change our optmization problem to find subject to !bt \[ -y_i(\bm{x}_1^T\bm{w}+b)=1-\xi_i \hspace{0.1cm}\forall i, +y_i(\bm{w}^T\bm{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, \] !et with the requirement $\xi_i\geq 0$. @@ -653,7 +629,7 @@ but now subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$ and We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads !bt \[ -\lambda_i\left[y_i(\bm{x}_1^T\bm{w}+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, +\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, \] !et !bt @@ -664,7 +640,1232 @@ We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads and !bt \[ -y_i(\bm{x}_i^T\bm{w}+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. +y_i(\bm{w}^T\bm{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. \] !et +!split +===== Kernels and non-linearity ===== + +The cases we have studied till now, were all characterized by two classes +with a close to linear separability. The classifiers we have described +so far find linear boundaries in our input feature space. It is +possible to make our procedure more flexible by exploring the feature +space using other basis expansions such as higher-order polynomials, +wavelets, splines etc. + +If our feature space is not easy to separate, as shown in the figure +here, we can achieve a better separation by introducing more complex +basis functions. The ideal would be, as shown in the next figure, to, via a specific transformation to +obtain a separation between the classes which is almost linear. + +The change of basis, from $x\rightarrow z=\phi(x)$ leads to the same type of equations to be solved, except that +we need to introduce for example a polynomial transformation to a two-dimensional training set. + +!bc pycod +import numpy as np +import os + +np.random.seed(42) + +# To plot pretty figures +import matplotlib +import matplotlib.pyplot as plt +plt.rcParams['axes.labelsize'] = 14 +plt.rcParams['xtick.labelsize'] = 12 +plt.rcParams['ytick.labelsize'] = 12 + + +from sklearn.svm import SVC +from sklearn import datasets + + + +X1D = np.linspace(-4, 4, 9).reshape(-1, 1) +X2D = np.c_[X1D, X1D**2] +y = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0]) + +plt.figure(figsize=(11, 4)) + +plt.subplot(121) +plt.grid(True, which='both') +plt.axhline(y=0, color='k') +plt.plot(X1D[:, 0][y==0], np.zeros(4), "bs") +plt.plot(X1D[:, 0][y==1], np.zeros(5), "g^") +plt.gca().get_yaxis().set_ticks([]) +plt.xlabel(r"$x_1$", fontsize=20) +plt.axis([-4.5, 4.5, -0.2, 0.2]) + +plt.subplot(122) +plt.grid(True, which='both') +plt.axhline(y=0, color='k') +plt.axvline(x=0, color='k') +plt.plot(X2D[:, 0][y==0], X2D[:, 1][y==0], "bs") +plt.plot(X2D[:, 0][y==1], X2D[:, 1][y==1], "g^") +plt.xlabel(r"$x_1$", fontsize=20) +plt.ylabel(r"$x_2$", fontsize=20, rotation=0) +plt.gca().get_yaxis().set_ticks([0, 4, 8, 12, 16]) +plt.plot([-4.5, 4.5], [6.5, 6.5], "r--", linewidth=3) +plt.axis([-4.5, 4.5, -1, 17]) +plt.subplots_adjust(right=1) +plt.show() + +!ec + + + +!split +===== The equations ===== + +Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with $x_i$ and $y_i$ as variables) +!bt +\[ +z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right). +\] +!et + +With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity) +!bt +\[ +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{z}_i^T\bm{z}_j, +\] +!et +subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$, and for the support vectors +!bt +\[ +y_i(\bm{w}^T\bm{z}_i+b)= 1 \hspace{0.1cm}\forall i, +\] +!et +from which we also find $b$. +To compute $\bm{z}_i^T\bm{z}_j$ we define the kernel $K(\bm{x}_i,\bm{x}_j)$ as +!bt +\[ +K(\bm{x}_i,\bm{x}_j)=\bm{z}_i^T\bm{z}_j= \phi(\bm{x}_i)^T\phi(\bm{x}_j). +\] +!et +For the above example, the kernel reads +!bt +\[ +K(\bm{x}_i,\bm{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2. +\] +!et + +We note that this is nothing but the dot product of the two original +vectors $(\bm{x}_i^T\bm{x}_j)^2$. Instead of thus computing the +product in the Lagrangian of $\bm{z}_i^T\bm{z}_j$ we simply compute +the dot product $(\bm{x}_i^T\bm{x}_j)^2$. + + +This leads to the so-called +kernel trick and the result leads to the same as if we went through +the trouble of performing the transformation +$\phi(\bm{x}_i)^T\phi(\bm{x}_j)$ during the SVM calculations. + + +!split +===== The problem to solve ===== +Using our definition of the kernel We can rewrite again the Lagrangian +!bt +\[ +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{z}_j, +\] +!et +subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$ in terms of a convex optimization problem +!bt +\[ +\frac{1}{2} \bm{\lambda}^T\begin{bmatrix} y_1y_1K(\bm{x}_1,\bm{x}_1) & y_1y_2K(\bm{x}_1,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_1,\bm{x}_n) \\ +y_2y_1K(\bm{x}_2,\bm{x}_1) & y_2y_2(\bm{x}_2,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_2,\bm{x}_n) \\ +\dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots \\ +y_ny_1K(\bm{x}_n,\bm{x}_1) & y_ny_2K(\bm{x}_n\bm{x}_2) & \dots & \dots & y_ny_nK(\bm{x}_n,\bm{x}_n) \\ +\end{bmatrix}\bm{\lambda}-\mathbb{1}\bm{\lambda}, +\] +!et +subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and +$\bm{y}=[y_1,y_2,\dots,y_n]$. +If we add the slack constants this leads to the additional constraint $0\leq \lambda_i \leq C$. + +We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type +!bt +\begin{align*} + &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\bm{\lambda}^T\bm{P}\bm{\lambda}+\bm{q}^T\bm{\lambda},\\ \nonumber + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \hspace{0.2cm} \wedge \bm{A}\bm{\lambda}=f. +\end{align*} +!et +Below we discuss how to solve these equations. Here we note that the matrix $\bm{P}$ has matrix elements $p_{ij}=y_iy_jK(\bm{x}_i,\bm{x}_j)$. +Given a kernel $K$ and the targets $y_i$ this matrix is easy to set up. The constraint $\bm{y}^T\bm{\lambda}=0$ leads to $f=0$ and $\bm{A}=\bm{y}$. How to set up the matrix $\bm{G}$ is discussed later. Here note that the inequalities $0\leq \lambda_i \leq C$ can be split up into +$0\leq \lambda_i$ and $\lambda_i \leq C$. These two inequalities define then the matrix $\bm{G}$ and the vector $\bm{h}$. + + +!split +===== Different kernels and Mercer's theorem ===== + +There are several popular kernels being used. These are +o Linear: $K(\bm{x},\bm{y})=\bm{x}^T\bm{y}$, +o Polynomial: $K(\bm{x},\bm{y})=(\bm{x}^T\bm{y}+\gamma)^d$, +o Gaussian Radial Basis Function: $K(\bm{x},\bm{y})=\exp{\left(-\gamma\vert\vert\bm{x}-\bm{y}\vert\vert^2\right)}$, +o Tanh: $K(\bm{x},\bm{y})=\tanh{(\bm{x}^T\bm{y}+\gamma)}$, +and many other ones. + +An important theorem for us is "Mercer's +theorem":"https://en.wikipedia.org/wiki/Mercer%27s_theorem". The +theorem states that if a kernel function $K$ is symmetric, continuous +and leads to a positive semi-definite matrix $\bm{P}$ then there +exists a function $\phi$ that maps $\bm{x}_i$ and $\bm{x}_j$ into +another space (possibly with much higher dimensions) such that + +!bt +\[ +K(\bm{x}_i,\bm{x}_j)=\phi(\bm{x}_i)^T\phi(\bm{x}_j). +\] +!et + +So you can use $K$ as a kernel since you know $\phi$ exists, even if +you don’t know what $\phi$ is. + +Note that some frequently used kernels (such as the Sigmoid kernel) +don’t respect all of Mercer’s conditions, yet they generally work well +in practice. + + +!split +===== The moons example ===== +!bc pycod +from __future__ import division, print_function, unicode_literals + +import numpy as np +np.random.seed(42) + +import matplotlib +import matplotlib.pyplot as plt +plt.rcParams['axes.labelsize'] = 14 +plt.rcParams['xtick.labelsize'] = 12 +plt.rcParams['ytick.labelsize'] = 12 + + +from sklearn.svm import SVC +from sklearn import datasets + + + +from sklearn.pipeline import Pipeline +from sklearn.preprocessing import StandardScaler +from sklearn.svm import LinearSVC + + +from sklearn.datasets import make_moons +X, y = make_moons(n_samples=100, noise=0.15, random_state=42) + +def plot_dataset(X, y, axes): + plt.plot(X[:, 0][y==0], X[:, 1][y==0], "bs") + plt.plot(X[:, 0][y==1], X[:, 1][y==1], "g^") + plt.axis(axes) + plt.grid(True, which='both') + plt.xlabel(r"$x_1$", fontsize=20) + plt.ylabel(r"$x_2$", fontsize=20, rotation=0) + +plot_dataset(X, y, [-1.5, 2.5, -1, 1.5]) +plt.show() + +from sklearn.datasets import make_moons +from sklearn.pipeline import Pipeline +from sklearn.preprocessing import PolynomialFeatures + +polynomial_svm_clf = Pipeline([ + ("poly_features", PolynomialFeatures(degree=3)), + ("scaler", StandardScaler()), + ("svm_clf", LinearSVC(C=10, loss="hinge", random_state=42)) + ]) + +polynomial_svm_clf.fit(X, y) + +def plot_predictions(clf, axes): + x0s = np.linspace(axes[0], axes[1], 100) + x1s = np.linspace(axes[2], axes[3], 100) + x0, x1 = np.meshgrid(x0s, x1s) + X = np.c_[x0.ravel(), x1.ravel()] + y_pred = clf.predict(X).reshape(x0.shape) + y_decision = clf.decision_function(X).reshape(x0.shape) + plt.contourf(x0, x1, y_pred, cmap=plt.cm.brg, alpha=0.2) + plt.contourf(x0, x1, y_decision, cmap=plt.cm.brg, alpha=0.1) + +plot_predictions(polynomial_svm_clf, [-1.5, 2.5, -1, 1.5]) +plot_dataset(X, y, [-1.5, 2.5, -1, 1.5]) + +plt.show() + + +from sklearn.svm import SVC + +poly_kernel_svm_clf = Pipeline([ + ("scaler", StandardScaler()), + ("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5)) + ]) +poly_kernel_svm_clf.fit(X, y) + +poly100_kernel_svm_clf = Pipeline([ + ("scaler", StandardScaler()), + ("svm_clf", SVC(kernel="poly", degree=10, coef0=100, C=5)) + ]) +poly100_kernel_svm_clf.fit(X, y) + +plt.figure(figsize=(11, 4)) + +plt.subplot(121) +plot_predictions(poly_kernel_svm_clf, [-1.5, 2.5, -1, 1.5]) +plot_dataset(X, y, [-1.5, 2.5, -1, 1.5]) +plt.title(r"$d=3, r=1, C=5$", fontsize=18) + +plt.subplot(122) +plot_predictions(poly100_kernel_svm_clf, [-1.5, 2.5, -1, 1.5]) +plot_dataset(X, y, [-1.5, 2.5, -1, 1.5]) +plt.title(r"$d=10, r=100, C=5$", fontsize=18) + +plt.show() + +def gaussian_rbf(x, landmark, gamma): + return np.exp(-gamma * np.linalg.norm(x - landmark, axis=1)**2) + +gamma = 0.3 + +x1s = np.linspace(-4.5, 4.5, 200).reshape(-1, 1) +x2s = gaussian_rbf(x1s, -2, gamma) +x3s = gaussian_rbf(x1s, 1, gamma) + +XK = np.c_[gaussian_rbf(X1D, -2, gamma), gaussian_rbf(X1D, 1, gamma)] +yk = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0]) + +plt.figure(figsize=(11, 4)) + +plt.subplot(121) +plt.grid(True, which='both') +plt.axhline(y=0, color='k') +plt.scatter(x=[-2, 1], y=[0, 0], s=150, alpha=0.5, c="red") +plt.plot(X1D[:, 0][yk==0], np.zeros(4), "bs") +plt.plot(X1D[:, 0][yk==1], np.zeros(5), "g^") +plt.plot(x1s, x2s, "g--") +plt.plot(x1s, x3s, "b:") +plt.gca().get_yaxis().set_ticks([0, 0.25, 0.5, 0.75, 1]) +plt.xlabel(r"$x_1$", fontsize=20) +plt.ylabel(r"Similarity", fontsize=14) +plt.annotate(r'$\mathbf{x}$', + xy=(X1D[3, 0], 0), + xytext=(-0.5, 0.20), + ha="center", + arrowprops=dict(facecolor='black', shrink=0.1), + fontsize=18, + ) +plt.text(-2, 0.9, "$x_2$", ha="center", fontsize=20) +plt.text(1, 0.9, "$x_3$", ha="center", fontsize=20) +plt.axis([-4.5, 4.5, -0.1, 1.1]) + +plt.subplot(122) +plt.grid(True, which='both') +plt.axhline(y=0, color='k') +plt.axvline(x=0, color='k') +plt.plot(XK[:, 0][yk==0], XK[:, 1][yk==0], "bs") +plt.plot(XK[:, 0][yk==1], XK[:, 1][yk==1], "g^") +plt.xlabel(r"$x_2$", fontsize=20) +plt.ylabel(r"$x_3$ ", fontsize=20, rotation=0) +plt.annotate(r'$\phi\left(\mathbf{x}\right)$', + xy=(XK[3, 0], XK[3, 1]), + xytext=(0.65, 0.50), + ha="center", + arrowprops=dict(facecolor='black', shrink=0.1), + fontsize=18, + ) +plt.plot([-0.1, 1.1], [0.57, -0.1], "r--", linewidth=3) +plt.axis([-0.1, 1.1, -0.1, 1.1]) + +plt.subplots_adjust(right=1) + +plt.show() + + +x1_example = X1D[3, 0] +for landmark in (-2, 1): + k = gaussian_rbf(np.array([[x1_example]]), np.array([[landmark]]), gamma) + print("Phi({}, {}) = {}".format(x1_example, landmark, k)) + +rbf_kernel_svm_clf = Pipeline([ + ("scaler", StandardScaler()), + ("svm_clf", SVC(kernel="rbf", gamma=5, C=0.001)) + ]) +rbf_kernel_svm_clf.fit(X, y) + + +from sklearn.svm import SVC + +gamma1, gamma2 = 0.1, 5 +C1, C2 = 0.001, 1000 +hyperparams = (gamma1, C1), (gamma1, C2), (gamma2, C1), (gamma2, C2) + +svm_clfs = [] +for gamma, C in hyperparams: + rbf_kernel_svm_clf = Pipeline([ + ("scaler", StandardScaler()), + ("svm_clf", SVC(kernel="rbf", gamma=gamma, C=C)) + ]) + rbf_kernel_svm_clf.fit(X, y) + svm_clfs.append(rbf_kernel_svm_clf) + +plt.figure(figsize=(11, 7)) + +for i, svm_clf in enumerate(svm_clfs): + plt.subplot(221 + i) + plot_predictions(svm_clf, [-1.5, 2.5, -1, 1.5]) + plot_dataset(X, y, [-1.5, 2.5, -1, 1.5]) + gamma, C = hyperparams[i] + plt.title(r"$\gamma = {}, C = {}$".format(gamma, C), fontsize=16) + +plt.show() + +!ec + + + +!split +===== Mathematical optimization of convex functions ===== + +A mathematical (quadratic) optimization problem, or just optimization problem, has the form +!bt +\begin{align*} + &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\bm{\lambda}^T\bm{P}\bm{\lambda}+\bm{q}^T\bm{\lambda},\\ \nonumber + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \wedge \bm{A}\bm{\lambda}=f. +\end{align*} +!et +subject to some constraints for say a selected set $i=1,2,\dots, n$. +In our case we are optimizing with respect to the Lagrangian multipliers $\lambda_i$, and the +vector $\bm{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n]$ is the optimization variable we are dealing with. + +In our case we are particularly interested in a class of optimization problems called convex optmization problems. +In our discussion on gradient descent methods we discussed at length the definition of a convex function. + +Convex optimization problems play a central role in applied mathematics and we recommend strongly "Boyd and Vandenberghe's text on the topics":"http://web.stanford.edu/~boyd/cvxbook/". + + + +!split +===== How do we solve these problems? ===== + +If we use Python as programming language and wish to venture beyond +_scikit-learn_, _tensorflow_ and similar software which makes our +lives so much easier, we need to dive into the wonderful world of +quadratic programming. We can, if we wish, solve the minimization +problem using say standard gradient methods or conjugate gradient +methods. However, these methods tend to exhibit a rather slow +converge. So, welcome to the promised land of quadratic programming. + +The functions we need are contained in the quadratic programming package _CVXOPT_ and we need to import it together with _numpy_ as + +!bc pycod +import numpy +import cvxopt +!ec + +This will make our life much easier. You don't need t write your own optimizer. + + +!split +===== A simple example ===== + +We remind ourselves about the general problem we want to solve +!bt +\begin{align*} + &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\bm{x}^T\bm{P}\bm{x}+\bm{q}^T\bm{x},\\ \nonumber + &\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \bm{G}\bm{x} \preceq \bm{h} \wedge \bm{A}\bm{x}=f. +\end{align*} +!et + +Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem +!bt +\begin{align*} + &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber + &\mathrm{subject to} \\ \nonumber + &x, y \geq 0 \\ \nonumber + &x+3y \geq 15 \\ \nonumber + &2x+5y \leq 100 \\ \nonumber + &3x+4y \leq 80. \\ \nonumber +\end{align*} +!et +The minimization problem can be rewritten in terms of vectors and matrices as (with $x$ and $y$ being the unknowns) +!bt +\[ +\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}. +\] +!et +Similarly, we can now set up the inequalities (we need to change $\geq$ to $\leq$ by multiplying with $-1$ on bot sides) as the following matrix-vector equation +!bt +\[ +\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +\] +!et +We have collapsed all the inequalities into a single matrix $\bm{G}$. We see also that our matrix +!bt +\[ +\bm{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} +\] +!et +is clearly positive semi-definite (all eigenvalues larger or equal zero). +Finally, the vector $\bm{h}$ is defined as +!bt +\[ +\bm{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +\] +!et + + +Since we don't have any equalities the matrix $\bm{A}$ is set to zero +The following code solves the equations for us +!bc pycod +# Import the necessary packages +import numpy +from cvxopt import matrix +from cvxopt import solvers +P = matrix(numpy.diag([1,0]), tc=’d’) +q = matrix(numpy.array([3,4]), tc=’d’) +G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’) +h = matrix(numpy.array([0,0,-15,100,80]), tc=’d’) +# Construct the QP, invoke solver +sol = solvers.qp(P,q,G,h) +# Extract optimal value and solution +sol[’x’] +sol[’primal objective’] +!ec + +!split +===== Back to the more realistic cases ===== + +We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the _slack_ parameter $C$ we have +!bt +\[ +\frac{1}{2} \bm{\lambda}^T\begin{bmatrix} y_1y_1K(\bm{x}_1,\bm{x}_1) & y_1y_2K(\bm{x}_1,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_1,\bm{x}_n) \\ +y_2y_1K(\bm{x}_2,\bm{x}_1) & y_2y_2K(\bm{x}_2,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_2,\bm{x}_n) \\ +\dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots \\ +y_ny_1K(\bm{x}_n,\bm{x}_1) & y_ny_2K(\bm{x}_n\bm{x}_2) & \dots & \dots & y_ny_nK(\bm{x}_n,\bm{x}_n) \\ +\end{bmatrix}\bm{\lambda}-\mathbb{I}\bm{\lambda}, +\] +!et +subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and +$\bm{y}=[y_1,y_2,\dots,y_n]$. +With the slack constants this leads to the additional constraint $0\leq \lambda_i \leq C$. + +_code will be added_ + +!split +===== Summary of course ===== + +!split +===== What? Me worry? No final exam in this course! ===== +FIGURE: [figures/exam1.jpeg, width=500 frac=0.6] + + + +!split +===== Topics we have covered this year ===== + +The course has two central parts + +o Statistical analysis and optimization of data +o Machine learning + +!split +===== Statistical analysis and optimization of data ===== + +The following topics have been discussed: +o Basic concepts, expectation values, variance, covariance, correlation functions and errors; +o Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions; +o Central elements from linear algebra, matrix inversion and SVD +o Gradient methods for data optimization +o Estimation of errors using cross-validation, bootstrapping and jackknife methods; +o Practical optimization using Singular-value decomposition and least squares for parameterizing data. +o Principal Component Analysis to reduce the number of features. + +!split +===== Machine learning ===== + +The following topics will be covered +o Linear methods for regression and classification: + o Ordinary Least Squares + o Ridge regression + o Lasso regression + o Logistic regression +o Neural networks and deep learning: + o Feed Forward Neural Networks + o Convolutional Neural Networks + o Recurrent Neural Networks +o Decisions trees and ensemble methods: + o Decision trees + o Bagging and voting + o Random forests + o Boosting and gradient boosting +o Support vector machines + o Binary classification and multiclass classification + o Kernel methods + o Regression + + +!split +===== Learning outcomes and overarching aims of this course ===== + +The course introduces a variety of central algorithms and methods +essential for studies of data analysis and machine learning. The +course is project based and through the various projects, normally +three, you will be exposed to fundamental research problems +in these fields, with the aim to reproduce state of the art scientific +results. The students will learn to develop and structure large codes +for studying these systems, get acquainted with computing facilities +and learn to handle large scientific projects. A good scientific and +ethical conduct is emphasized throughout the course. + +* Understand linear methods for regression and classification; +* Learn about neural network; +* Learn about bagging, boosting and trees +* Support vector machines +* Learn about basic data analysis; +* Be capable of extending the acquired knowledge to other systems and cases; +* Have an understanding of central algorithms used in data analysis and machine learning; +* Work on numerical projects to illustrate the theory. The projects play a central role and you are expected to know modern programming languages like Python or C++. + + + +!split +===== Perspective on Machine Learning ===== + +o Rapidly emerging application area +o Experiment AND theory are evolving in many many fields. Still many low-hanging fruits. +o Requires education/retraining for more widespread adoption +o A lot of “word-of-mouth” development methods + +Huge amounts of data sets require automation, classical analysis tools often inadequate. +High energy physics hit this wall in the 90’s. +In 2009 single top quark production was determined via "Boosted decision trees, Bayesian +Neural Networks, etc.":"https://arxiv.org/pdf/0903.0850.pdf" + + +!split +===== Machine Learning Research ===== + +Where to find recent results: +o Conference proceedings, arXiv and blog posts! +o _NIPS_: "Neural Information Processing Systems":"https://papers.nips.cc" +o _ICLR_: "International Conference on Learning Representations":"https://openreview.net/group?id=ICLR.cc/2018/Conference#accepted-oral-papers" +o _ICML_: International Conference on Machine Learning +o "Journal of Machine Learning Research":"http://www.jmlr.org/papers/v19/" +o "Follow ML on ArXiv":"https://arxiv.org/list/cs.LG/recent" + + +!split +===== Starting your Machine Learning Project ===== + +o Identify problem type: classification, regression +o Consider your data carefully +o Choose a simple model that fits 1. and 2. +o Consider your data carefully again! Think of data representation more carefully. +o Based on your results, feedback loop to earliest possible point + + + + +!split +===== Choose a Model and Algorithm ===== + +o Supervised? +o Start with the simplest model that fits your problem +o Start with minimal processing of data + +!split +===== Preparing Your Data ===== + +o Shuffle your data +o Mean center your data + * Why? +o Normalize the variance + * Why? +o _Whitening_ + * Decorrelates data + * Can be hit or miss +o When to do train/test split? + + +!split +===== Which Activation and Weights to Choose in Neural Networks ===== + +o RELU? ELU? +o Sigmoid or Tanh? +o Set all weights to 0? + * Terrible idea +o Set all weights to random values? + * Small random values + + +!split +===== Optimization Methods and Hyperparameters ===== +o Stochastic gradient descent +o Stochastic gradient descent + momentum +o State-of-the-art approaches: + * RMSProp + * Adam + * and more + +Which regularization and hyperparameters? $L_1$ or $L_2$, soft +classifiers, depths of trees and many other. Need to explore a large +set of hyperparameters and regularization methods. + + +!split +===== Resampling ===== + +When do we resample? + +o "Bootstrap":"https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A" +o "Cross-validation":"https://www.youtube.com/watch?v=fSytzGwwBVw&ab_channel=StatQuestwithJoshStarmer" +o Jackknife and many other + + +!split +===== Other courses on Data science and Machine Learning at UiO ===== + +The link here URL:"https://www.mn.uio.no/english/research/about/centre-focus/innovation/data-science/studies/" gives an excellent overview of courses on Machine learning at UiO. + +o "STK2100 Machine learning and statistical methods for prediction and classification":"http://www.uio.no/studier/emner/matnat/math/STK2100/index-eng.html". +o "IN3050/IN4050 Introduction to Artificial Intelligence and Machine Learning":"https://www.uio.no/studier/emner/matnat/ifi/IN3050/index-eng.html". Introductory course in machine learning and AI with an algorithmic approach. +o "STK-INF3000/4000 Selected Topics in Data Science":"http://www.uio.no/studier/emner/matnat/math/STK-INF3000/index-eng.html". The course provides insight into selected contemporary relevant topics within Data Science. +o "IN4080 Natural Language Processing":"https://www.uio.no/studier/emner/matnat/ifi/IN4080/index.html". Probabilistic and machine learning techniques applied to natural language processing. +o "STK-IN4300 – Statistical learning methods in Data Science":"https://www.uio.no/studier/emner/matnat/math/STK-IN4300/index-eng.html". An advanced introduction to statistical and machine learning. For students with a good mathematics and statistics background. +o "IN-STK5000 Adaptive Methods for Data-Based Decision Making":"https://www.uio.no/studier/emner/matnat/ifi/IN-STK5000/index-eng.html". Methods for adaptive collection and processing of data based on machine learning techniques. +o "IN5400/INF5860 – Machine Learning for Image Analysis":"https://www.uio.no/studier/emner/matnat/ifi/IN5400/". An introduction to deep learning with particular emphasis on applications within Image analysis, but useful for other application areas too. +o "TEK5040 – Dyp læring for autonome systemer":"https://www.uio.no/studier/emner/matnat/its/TEK5040/". The course addresses advanced algorithms and architectures for deep learning with neural networks. The course provides an introduction to how deep-learning techniques can be used in the construction of key parts of advanced autonomous systems that exist in physical environments and cyber environments. + +!split +===== Additional courses of interest ===== + +o "STK4051 Computational Statistics":"https://www.uio.no/studier/emner/matnat/math/STK4051/index-eng.html" +o "STK4021 Applied Bayesian Analysis and Numerical Methods":"https://www.uio.no/studier/emner/matnat/math/STK4021/index-eng.html" + +!split +===== What's the future like? ===== + +Based on multi-layer nonlinear neural networks, deep learning can +learn directly from raw data, automatically extract and abstract +features from layer to layer, and then achieve the goal of regression, +classification, or ranking. Deep learning has made breakthroughs in +computer vision, speech processing and natural language, and reached +or even surpassed human level. The success of deep learning is mainly +due to the three factors: big data, big model, and big computing. + +In the past few decades, many different architectures of deep neural +networks have been proposed, such as +o Convolutional neural networks, which are mostly used in image and video data processing, and have also been applied to sequential data such as text processing; +o Recurrent neural networks, which can process sequential data of variable length and have been widely used in natural language understanding and speech processing; +o Encoder-decoder framework, which is mostly used for image or sequence generation, such as machine translation, text summarization, and image captioning. + +!split +===== Types of Machine Learning, a repetition ===== + +!bblock +The approaches to machine learning are many, but are often split into two main categories. +In *supervised learning* we know the answer to a problem, +and let the computer deduce the logic behind it. On the other hand, *unsupervised learning* +is a method for finding patterns and relationship in data sets without any prior knowledge of the system. +Some authours also operate with a third category, namely *reinforcement learning*. This is a paradigm +of learning inspired by behavioural psychology, where learning is achieved by trial-and-error, +solely from rewards and punishment. + +Another way to categorize machine learning tasks is to consider the desired output of a system. +Some of the most common tasks are: + + * Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning. + + * Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values. + + * Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning. + + * Other unsupervised learning algortihms like _Boltzmann machines_ +!eblock + + +!split +===== Why Boltzmann machines? ===== + +What is known as restricted Boltzmann Machines (RMB) have received a lot of attention lately. +One of the major reasons is that they can be stacked layer-wise to build deep neural networks that capture complicated statistics. + +The original RBMs had just one visible layer and a hidden layer, but recently so-called Gaussian-binary RBMs have gained quite some popularity in imaging since they are capable of modeling continuous data that are common to natural images. + +Furthermore, they have been used to solve complicated "quantum mechanical many-particle problems or classical statistical physics problems like the Ising and Potts classes of models":"https://journals.aps.org/rmp/abstract/10.1103/RevModPhys.91.045002". + + +!split +===== Boltzmann Machines ===== + +Why use a generative model rather than the more well known discriminative deep neural networks (DNN)? + +* Discriminitave methods have several limitations: They are mainly supervised learning methods, thus requiring labeled data. And there are tasks they cannot accomplish, like drawing new examples from an unknown probability distribution. + +* A generative model can learn to represent and sample from a probability distribution. The core idea is to learn a parametric model of the probability distribution from which the training data was drawn. As an example + + o A model for images could learn to draw new examples of cats and dogs, given a training dataset of images of cats and dogs. + o Generate a sample of an ordered or disordered phase, having been given samples of such phases. + o Model the trial function for "Monte Carlo calculations":"https://journals.aps.org/rmp/abstract/10.1103/RevModPhys.91.045002". + + +!split +===== Some similarities and differences from DNNs ===== + + o Both use gradient-descent based learning procedures for minimizing cost functions + o Energy based models don't use backpropagation and automatic differentiation for computing gradients, instead turning to Markov Chain Monte Carlo methods. + o DNNs often have several hidden layers. A restricted Boltzmann machine has only one hidden layer, however several RBMs can be stacked to make up Deep Belief Networks, of which they constitute the building blocks. + +History: The RBM was developed by amongst others "Geoffrey Hinton":"https://en.wikipedia.org/wiki/Geoffrey_Hinton", called by some the "Godfather of Deep Learning", working with the University of Toronto and Google. + + +!split +===== Boltzmann machines (BM) ===== + +!bblock +A BM is what we would call an undirected probabilistic graphical model +with stochastic continuous or discrete units. +!eblock +!bblock +It is interpreted as a stochastic recurrent neural network where the +state of each unit(neurons/nodes) depends on the units it is connected +to. The weights in the network represent thus the strength of the +interaction between various units/nodes. +!eblock +!bblock +It turns into a Hopfield network if we choose deterministic rather +than stochastic units. In contrast to a Hopfield network, a BM is a +so-called generative model. It allows us to generate new samples from +the learned distribution. +!eblock + +!split +===== A standard BM setup ===== + +!bblock +A standard BM network is divided into a set of observable and visible units $\hat{x}$ and a set of unknown hidden units/nodes $\hat{h}$. +!eblock + +!bblock +Additionally there can be bias nodes for the hidden and visible layers. These biases are normally set to $1$. +!eblock + +!bblock +BMs are stackable, meaning they cwe can train a BM which serves as input to another BM. We can construct deep networks for learning complex PDFs. The layers can be trained one after another, a feature which makes them popular in deep learning +!eblock + +However, they are often hard to train. This leads to the introduction of so-called restricted BMs, or RBMS. +Here we take away all lateral connections between nodes in the visible layer as well as connections between nodes in the hidden layer. The network is illustrated in the figure below. + + + +!split +===== The structure of the RBM network ===== + +FIGURE: [figures/RBM.pdf, width=800 frac=1.0] + + + +!split +===== The network ===== + +_The network layers_: + o A function $\mathbf{x}$ that represents the visible layer, a vector of $M$ elements (nodes). This layer represents both what the RBM might be given as training input, and what we want it to be able to reconstruct. This might for example be given by the pixels of an image or coefficients representing speech, or the coordinates of a quantum mechanical state function. + o The function $\mathbf{h}$ represents the hidden, or latent, layer. A vector of $N$ elements (nodes). Also called "feature detectors". + +!split +===== Goals ===== + +The goal of the hidden layer is to increase the model's expressive +power. We encode complex interactions between visible variables by +introducing additional, hidden variables that interact with visible +degrees of freedom in a simple manner, yet still reproduce the complex +correlations between visible degrees in the data once marginalized +over (integrated out). + +_The network parameters, to be optimized/learned_: + o $\mathbf{a}$ represents the visible bias, a vector of same length as $\mathbf{x}$. + o $\mathbf{b}$ represents the hidden bias, a vector of same lenght as $\mathbf{h}$. + o $W$ represents the interaction weights, a matrix of size $M\times N$. + + + + +!split +===== Joint distribution ===== + +The restricted Boltzmann machine is described by a Boltzmann distribution +!bt +\begin{align} + P_{rbm}(\mathbf{x},\mathbf{h}) = \frac{1}{Z} e^{-\frac{1}{T_0}E(\mathbf{x},\mathbf{h})}, +\end{align} +!et +where $Z$ is the normalization constant or partition function, defined as +!bt +\begin{align} + Z = \int \int e^{-\frac{1}{T_0}E(\mathbf{x},\mathbf{h})} d\mathbf{x} d\mathbf{h}. +\end{align} +!et +It is common to ignore $T_0$ by setting it to one. + + + + +!split +===== Network Elements, the energy function ===== + +The function $E(\mathbf{x},\mathbf{h})$ gives the _energy_ of a +configuration (pair of vectors) $(\mathbf{x}, \mathbf{h})$. The lower +the energy of a configuration, the higher the probability of it. This +function also depends on the parameters $\mathbf{a}$, $\mathbf{b}$ and +$W$. Thus, when we adjust them during the learning procedure, we are +adjusting the energy function to best fit our problem. + +An expression for the energy function is +!bt +\[ +E(\hat{x},\hat{h}) = -\sum_{ia}^{NA}b_i^a \alpha_i^a(x_i)-\sum_{jd}^{MD}c_j^d \beta_j^d(h_j)-\sum_{ijad}^{NAMD}b_i^a \alpha_i^a(x_i)c_j^d \beta_j^d(h_j)w_{ij}^{ad}. +\] +!et + +Here $\beta_j^d(h_j)$ and $\alpha_i^a(x_j)$ are so-called transfer functions that map a given input value to a desired feature value. The labels $a$ and $d$ denote that there can be multiple transfer functions per variable. The first sum depends only on the visible units. The second on the hidden ones. _Note_ that there is no connection between nodes in a layer. + +The quantities $b$ and $c$ can be interpreted as the visible and hidden biases, respectively. + +The connection between the nodes in the two layers is given by the weights $w_{ij}$. + +!split +===== Defining different types of RBMs ===== +There are different variants of RBMs, and the differences lie in the types of visible and hidden units we choose as well as in the implementation of the energy function $E(\mathbf{x},\mathbf{h})$. + +!bblock Binary-Binary RBM: + +RBMs were first developed using binary units in both the visible and hidden layer. The corresponding energy function is defined as follows: +!bt +\begin{align} + E(\mathbf{x}, \mathbf{h}) = - \sum_i^M x_i a_i- \sum_j^N b_j h_j - \sum_{i,j}^{M,N} x_i w_{ij} h_j, +\end{align} +!et +where the binary values taken on by the nodes are most commonly 0 and 1. +!eblock +!bblock Gaussian-Binary RBM: + +Another varient is the RBM where the visible units are Gaussian while the hidden units remain binary: +!bt +\begin{align} + E(\mathbf{x}, \mathbf{h}) = \sum_i^M \frac{(x_i - a_i)^2}{2\sigma_i^2} - \sum_j^N b_j h_j - \sum_{i,j}^{M,N} \frac{x_i w_{ij} h_j}{\sigma_i^2}. +\end{align} +!et +!eblock + +!split +===== More about RBMs ===== +o Useful when we model continuous data (i.e., we wish $\mathbf{x}$ to be continuous) +o Requires a smaller learning rate, since there's no upper bound to the value a component might take in the reconstruction + +Other types of units include: + o Softmax and multinomial units + o Gaussian visible and hidden units + o Binomial units + o Rectified linear units + +To read more, see "Lectures on Boltzmann machines in Physics":"https://github.com/CompPhysics/ComputationalPhysics2/blob/gh-pages/doc/pub/notebook2/ipynb/notebook2.ipynb". + + +!split +===== Autoencoders: Overarching view ===== + +Autoencoders are artificial neural networks capable of learning +efficient representations of the input data (these representations are called codings) without +any supervision (i.e., the training set is unlabeled). These codings +typically have a much lower dimensionality than the input data, making +autoencoders useful for dimensionality reduction. + +More importantly, autoencoders act as powerful feature detectors, and +they can be used for unsupervised pretraining of deep neural networks. + +Lastly, they are capable of randomly generating new data that looks +very similar to the training data; this is called a generative +model. For example, you could train an autoencoder on pictures of +faces, and it would then be able to generate new faces. Surprisingly, +autoencoders work by simply learning to copy their inputs to their +outputs. This may sound like a trivial task, but we will see that +constraining the network in various ways can make it rather +difficult. For example, you can limit the size of the internal +representation, or you can add noise to the inputs and train the +network to recover the original inputs. These constraints prevent the +autoencoder from trivially copying the inputs directly to the outputs, +which forces it to learn efficient ways of representing the data. In +short, the codings are byproducts of the autoencoder’s attempt to +learn the identity function under some constraints. + +"Video on autoencoders":"https://www.coursera.org/lecture/building-deep-learning-models-with-tensorflow/autoencoders-1U4L3" + +See also A. Geron's textbook, chapter 15. + +!split +===== Bayesian Machine Learning ===== + +This is an important topic if we aim at extracting a probability +distribution. This gives us also a confidence interval and error +estimates. + +Bayesian machine learning allows us to encode our prior beliefs about +what those models should look like, independent of what the data tells +us. This is especially useful when we don’t have a ton of data to +confidently learn our model. + +"Video on Bayesian deep learning":"https://www.youtube.com/watch?v=E1qhGw8QxqY&ab_channel=AndrewGordonWilson" + +See also the "slides here":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Articles/lec03.pdf". + +!split +===== Reinforcement Learning ===== + +Reinforcement Learning (RL) is one of the most exciting fields of +Machine Learning today, and also one of the oldest. It has been around +since the 1950s, producing many interesting applications over the +years. + +It studies +how agents take actions based on trial and error, so as to maximize +some notion of cumulative reward in a dynamic system or +environment. Due to its generality, the problem has also been studied +in many other disciplines, such as game theory, control theory, +operations research, information theory, multi-agent systems, swarm +intelligence, statistics, and genetic algorithms. + +In March 2016, AlphaGo, a computer program that plays the board game +Go, beat Lee Sedol in a five-game match. This was the first time a +computer Go program had beaten a 9-dan (highest rank) professional +without handicaps. AlphaGo is based on deep convolutional neural +networks and reinforcement learning. AlphaGo’s victory was a major +milestone in artificial intelligence and it has also made +reinforcement learning a hot research area in the field of machine +learning. + +"Lecture on Reinforcement Learning":"https://www.youtube.com/watch?v=FgzM3zpZ55o&ab_channel=stanfordonline". + +See also A. Geron's textbook, chapter 16. + +!split +===== Transfer learning ===== + +The goal of transfer learning is to transfer the model or knowledge +obtained from a source task to the target task, in order to resolve +the issues of insufficient training data in the target task. The +rationality of doing so lies in that usually the source and target +tasks have inter-correlations, and therefore either the features, +samples, or models in the source task might provide useful information +for us to better solve the target task. Transfer learning is a hot +research topic in recent years, with many problems still waiting to be studied. + +"Lecture on transfer learning":"https://www.ias.edu/video/machinelearning/2020/0331-SamoryKpotufe". + +!split +===== Adversarial learning ===== + +The conventional deep generative model has a potential problem: the +model tends to generate extreme instances to maximize the +probabilistic likelihood, which will hurt its performance. Adversarial +learning utilizes the adversarial behaviors (e.g., generating +adversarial instances or training an adversarial model) to enhance the +robustness of the model and improve the quality of the generated +data. In recent years, one of the most promising unsupervised learning +technologies, generative adversarial networks (GAN), has already been +successfully applied to image, speech, and text. + +"Lecture on adversial learning":"https://www.youtube.com/watch?v=CIfsB_EYsVI&ab_channel=StanfordUniversitySchoolofEngineering". + +!split +===== Dual learning ===== + +Dual learning is a new learning paradigm, the basic idea of which is +to use the primal-dual structure between machine learning tasks to +obtain effective feedback/regularization, and guide and strengthen the +learning process, thus reducing the requirement of large-scale labeled +data for deep learning. The idea of dual learning has been applied to +many problems in machine learning, including machine translation, +image style conversion, question answering and generation, image +classification and generation, text classification and generation, +image-to-text, and text-to-image. + +!split +===== Distributed machine learning ===== + +Distributed computation will speed up machine learning algorithms, +significantly improve their efficiency, and thus enlarge their +application. When distributed meets machine learning, more than just +implementing the machine learning algorithms in parallel is required. + + +!split +===== Meta learning ===== + +Meta learning is an emerging research direction in machine +learning. Roughly speaking, meta learning concerns learning how to +learn, and focuses on the understanding and adaptation of the learning +itself, instead of just completing a specific learning task. That is, +a meta learner needs to be able to evaluate its own learning methods +and adjust its own learning methods according to specific learning +tasks. + +!split +===== The Challenges Facing Machine Learning ===== + +While there has been much progress in machine learning, there are also challenges. + +For example, the mainstream machine learning technologies are +black-box approaches, making us concerned about their potential +risks. To tackle this challenge, we may want to make machine learning +more explainable and controllable. As another example, the +computational complexity of machine learning algorithms is usually +very high and we may want to invent lightweight algorithms or +implementations. Furthermore, in many domains such as physics, +chemistry, biology, and social sciences, people usually seek elegantly +simple equations (e.g., the Schrödinger equation) to uncover the +underlying laws behind various phenomena. In the field of machine +learning, can we reveal simple laws instead of designing more complex +models for data fitting? Although there are many challenges, we are +still very optimistic about the future of machine learning. As we look +forward to the future, here are what we think the research hotspots in +the next ten years will be. + +See the article on "Discovery of Physics From Data: Universal Laws and Discrepancies":"https://www.frontiersin.org/articles/10.3389/frai.2020.00025/full" + +!split +===== Explainable machine learning ===== + +Machine learning, especially deep learning, evolves rapidly. The +ability gap between machine and human on many complex cognitive tasks +becomes narrower and narrower. However, we are still in the very early +stage in terms of explaining why those effective models work and how +they work. + +_What is missing: the gap between correlation and causation_. Standard Machine Learning is based on what e have called a frequentist approach. + +Most +machine learning techniques, especially the statistical ones, depend +highly on correlations in data sets to make predictions and analyses. In +contrast, rational humans tend to reply on clear and trustworthy +causality relations obtained via logical reasoning on real and clear +facts. It is one of the core goals of explainable machine learning to +transition from solving problems by data correlation to solving +problems by logical reasoning. + +_Bayesian Machine Learning is one of the exciting research directions in this field_. + +!split +===== Quantum machine learning ===== + +Quantum machine learning is an emerging interdisciplinary research +area at the intersection of quantum computing and machine learning. + +Quantum computers use effects such as quantum coherence and quantum +entanglement to process information, which is fundamentally different +from classical computers. Quantum algorithms have surpassed the best +classical algorithms in several problems (e.g., searching for an +unsorted database, inverting a sparse matrix), which we call quantum +acceleration. + +When quantum computing meets machine learning, it can be a mutually +beneficial and reinforcing process, as it allows us to take advantage +of quantum computing to improve the performance of classical machine +learning algorithms. In addition, we can also use the machine learning +algorithms (on classic computers) to analyze and improve quantum +computing systems. + +"Lecture on Quantum ML":"https://www.youtube.com/watch?v=Xh9pUu3-WxM&ab_channel=InstituteforPure%26AppliedMathematics%28IPAM%29". + + +"Read interview with Maria Schuld on her work on Quantum Machine Learning":"https://physics.aps.org/articles/v13/179?utm_campaign=weekly&utm_medium=email&utm_source=emailalert". See also "her recent textbook":"https://www.springer.com/gp/book/9783319964232". + + +!split +===== Quantum machine learning algorithms based on linear algebra ===== + +Many quantum machine learning algorithms are based on variants of +quantum algorithms for solving linear equations, which can efficiently +solve N-variable linear equations with complexity of O(log2 N) under +certain conditions. The quantum matrix inversion algorithm can +accelerate many machine learning methods, such as least square linear +regression, least square version of support vector machine, Gaussian +process, and more. The training of these algorithms can be simplified +to solve linear equations. The key bottleneck of this type of quantum +machine learning algorithms is data input—that is, how to initialize +the quantum system with the entire data set. Although efficient +data-input algorithms exist for certain situations, how to efficiently +input data into a quantum system is as yet unknown for most cases. + +!split +===== Quantum reinforcement learning ===== + +In quantum reinforcement learning, a quantum agent interacts with the +classical environment to obtain rewards from the environment, so as to +adjust and improve its behavioral strategies. In some cases, it +achieves quantum acceleration by the quantum processing capabilities +of the agent or the possibility of exploring the environment through +quantum superposition. Such algorithms have been proposed in +superconducting circuits and systems of trapped ions. + +!split +===== Quantum deep learning ===== + +Dedicated quantum information processors, such as quantum annealers +and programmable photonic circuits, are well suited for building deep +quantum networks. The simplest deep quantum network is the Boltzmann +machine. The classical Boltzmann machine consists of bits with tunable +interactions and is trained by adjusting the interaction of these bits +so that the distribution of its expression conforms to the statistics +of the data. To quantize the Boltzmann machine, the neural network can +simply be represented as a set of interacting quantum spins that +correspond to an adjustable Ising model. Then, by initializing the +input neurons in the Boltzmann machine to a fixed state and allowing +the system to heat up, we can read out the output qubits to get the +result. + + +!split +===== Social machine learning ===== + +Machine learning aims to imitate how humans +learn. While we have developed successful machine learning algorithms, +until now we have ignored one important fact: humans are social. Each +of us is one part of the total society and it is difficult for us to +live, learn, and improve ourselves, alone and isolated. Therefore, we +should design machines with social properties. Can we let machines +evolve by imitating human society so as to achieve more effective, +intelligent, interpretable “social machine learning”? + +And much more. + + + +!split +===== The last words? ===== + +Early computer scientist Alan Kay said, _The best way to predict the +future is to create it_. Therefore, all machine learning +practitioners, whether scholars or engineers, professors or students, +need to work together to advance these important research +topics. Together, we will not just predict the future, but create it. + + + + +!split +===== Best wishes to you all and thanks so much for your heroic efforts this semester ===== + +FIGURE: [figures/Nebbdyr2.png, width=500 frac=0.6] + + + + + + + + + +