From 8ac7d9e4af1a6ad4ae75d667c4098f5d9165c387 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Sat, 12 Nov 2022 10:25:11 +0100 Subject: [PATCH] preparing week 46 --- doc/pub/week46/html/._week46-bs000.html | 65 ++- doc/pub/week46/html/._week46-bs001.html | 63 ++- doc/pub/week46/html/._week46-bs002.html | 105 ++--- doc/pub/week46/html/._week46-bs003.html | 172 ++++++-- doc/pub/week46/html/._week46-bs004.html | 105 +++-- doc/pub/week46/html/._week46-bs005.html | 196 +++------ doc/pub/week46/html/._week46-bs006.html | 103 +++-- doc/pub/week46/html/._week46-bs007.html | 104 ++--- doc/pub/week46/html/._week46-bs008.html | 101 +++-- doc/pub/week46/html/._week46-bs009.html | 78 ++-- doc/pub/week46/html/._week46-bs010.html | 105 ++--- doc/pub/week46/html/._week46-bs011.html | 84 ++-- doc/pub/week46/html/._week46-bs012.html | 122 +++--- doc/pub/week46/html/._week46-bs013.html | 115 ++++-- doc/pub/week46/html/._week46-bs014.html | 112 +++--- doc/pub/week46/html/._week46-bs015.html | 136 +++---- doc/pub/week46/html/._week46-bs016.html | 106 ++--- doc/pub/week46/html/._week46-bs017.html | 118 +++--- doc/pub/week46/html/._week46-bs018.html | 98 ++--- doc/pub/week46/html/._week46-bs019.html | 121 +++--- doc/pub/week46/html/._week46-bs020.html | 171 +++++--- doc/pub/week46/html/._week46-bs021.html | 117 +++--- doc/pub/week46/html/._week46-bs022.html | 176 +++------ doc/pub/week46/html/._week46-bs023.html | 124 +++--- doc/pub/week46/html/._week46-bs024.html | 299 +++++++++++--- doc/pub/week46/html/._week46-bs025.html | 100 ++--- doc/pub/week46/html/._week46-bs026.html | 268 ++----------- doc/pub/week46/html/._week46-bs027.html | 147 ++++--- doc/pub/week46/html/._week46-bs028.html | 114 ++---- doc/pub/week46/html/week46-bs.html | 65 ++- doc/pub/week46/html/week46-reveal.html | 38 +- doc/pub/week46/html/week46-solarized.html | 39 +- doc/pub/week46/html/week46.html | 39 +- doc/pub/week46/ipynb/ipynb-week46-src.tar.gz | Bin 190 -> 191 bytes doc/pub/week46/ipynb/week46.ipynb | 395 ++++++++----------- doc/src/week46/week46.do.txt | 31 -- 36 files changed, 1989 insertions(+), 2343 deletions(-) diff --git a/doc/pub/week46/html/._week46-bs000.html b/doc/pub/week46/html/._week46-bs000.html index 91ebb8d77..64fb461b5 100644 --- a/doc/pub/week46/html/._week46-bs000.html +++ b/doc/pub/week46/html/._week46-bs000.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -192,7 +185,7 @@ MathJax.Hub.Config({
    -

    Aug 23, 2022

    +

    Nov 12, 2022


    @@ -217,7 +210,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs001.html b/doc/pub/week46/html/._week46-bs001.html index c93f12cd0..60ed29a13 100644 --- a/doc/pub/week46/html/._week46-bs001.html +++ b/doc/pub/week46/html/._week46-bs001.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -221,7 +214,7 @@ MathJax.Hub.Config({
  • 10
  • 11
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs002.html b/doc/pub/week46/html/._week46-bs002.html index 0a3a14dc1..a84277617 100644 --- a/doc/pub/week46/html/._week46-bs002.html +++ b/doc/pub/week46/html/._week46-bs002.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,25 +167,35 @@ MathJax.Hub.Config({

     

     

     

    -

    Friday

    +

    Support Vector Machines, overarching aims

    -

    Last year we had a very interesting workshop with many presentations. These were (program 2020)

    - - -

    We wish to organize something similar this coming Friday. The presentation last typically 5-10 minutes (some 3-5 slides) with time for questions afterwards. -Feel free to suggest topics. +

    A Support Vector Machine (SVM) is a very powerful and versatile +Machine Learning method, capable of performing linear or nonlinear +classification, regression, and even outlier detection. It is one of +the most popular models in Machine Learning, and anyone interested in +Machine Learning should have it in their toolbox. SVMs are +particularly well suited for classification of complex but small-sized or +medium-sized datasets.

    -

    The program will be available asap. It depends on input from you!

    +

    The case with two well-separated classes only can be understood in an +intuitive way in terms of lines in a two-dimensional space separating +the two classes (see figure below). +

    + +

    The basic mathematics behind the SVM is however less familiar to most of us. +It relies on the definition of hyperplanes and the +definition of a margin which separates classes (in case of +classification problems) of variables. It is also used for regression +problems. +

    + +

    With SVMs we distinguish between hard margin and soft margins. The +latter introduces a so-called softening parameter to be discussed +below. We distinguish also between linear and non-linear +approaches. The latter are the most frequent ones since it is rather +unlikely that we can separate classes easily by say straight lines. +

    @@ -211,7 +214,7 @@ Feel free to suggest topics.

  • 11
  • 12
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs003.html b/doc/pub/week46/html/._week46-bs003.html index 48375dca6..5e5dc7b82 100644 --- a/doc/pub/week46/html/._week46-bs003.html +++ b/doc/pub/week46/html/._week46-bs003.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,15 +167,108 @@ MathJax.Hub.Config({

     

     

     

    -

    Workshop plan Friday November 19 and the rest of the lecture

    +

    Hyperplanes and all that

    + +

    The theory behind support vector machines (SVM hereafter) is based on +the mathematical description of so-called hyperplanes. Let us start +with a two-dimensional case. This will also allow us to introduce our +first SVM examples. These will be tailored to the case of two specific +classes, as displayed in the figure here based on the usage of the petal data. +

    + +

    We assume here that our data set can be well separated into two +domains, where a straight line does the job in the separating the two +classes. Here the two classes are represented by either squares or +circles. +

    + + +
    +
    +
    +
    +
    + 
    from sklearn import datasets
+from sklearn.svm import SVC, LinearSVC
+from sklearn.linear_model import SGDClassifier
+from sklearn.preprocessing import StandardScaler
+import matplotlib
+import matplotlib.pyplot as plt
+plt.rcParams['axes.labelsize'] = 14
+plt.rcParams['xtick.labelsize'] = 12
+plt.rcParams['ytick.labelsize'] = 12
+
+
+iris = datasets.load_iris()
+X = iris["data"][:, (2, 3)]  # petal length, petal width
+y = iris["target"]
+
+setosa_or_versicolor = (y == 0) | (y == 1)
+X = X[setosa_or_versicolor]
+y = y[setosa_or_versicolor]
+
+
+
+C = 5
+alpha = 1 / (C * len(X))
+
+lin_clf = LinearSVC(loss="hinge", C=C, random_state=42)
+svm_clf = SVC(kernel="linear", C=C)
+sgd_clf = SGDClassifier(loss="hinge", learning_rate="constant", eta0=0.001, alpha=alpha,
+                        max_iter=100000, random_state=42)
+
+scaler = StandardScaler()
+X_scaled = scaler.fit_transform(X)
+
+lin_clf.fit(X_scaled, y)
+svm_clf.fit(X_scaled, y)
+sgd_clf.fit(X_scaled, y)
+
+print("LinearSVC:                   ", lin_clf.intercept_, lin_clf.coef_)
+print("SVC:                         ", svm_clf.intercept_, svm_clf.coef_)
+print("SGDClassifier(alpha={:.5f}):".format(sgd_clf.alpha), sgd_clf.intercept_, sgd_clf.coef_)
+
+# Compute the slope and bias of each decision boundary
+w1 = -lin_clf.coef_[0, 0]/lin_clf.coef_[0, 1]
+b1 = -lin_clf.intercept_[0]/lin_clf.coef_[0, 1]
+w2 = -svm_clf.coef_[0, 0]/svm_clf.coef_[0, 1]
+b2 = -svm_clf.intercept_[0]/svm_clf.coef_[0, 1]
+w3 = -sgd_clf.coef_[0, 0]/sgd_clf.coef_[0, 1]
+b3 = -sgd_clf.intercept_[0]/sgd_clf.coef_[0, 1]
+
+# Transform the decision boundary lines back to the original scale
+line1 = scaler.inverse_transform([[-10, -10 * w1 + b1], [10, 10 * w1 + b1]])
+line2 = scaler.inverse_transform([[-10, -10 * w2 + b2], [10, 10 * w2 + b2]])
+line3 = scaler.inverse_transform([[-10, -10 * w3 + b3], [10, 10 * w3 + b3]])
+
+# Plot all three decision boundaries
+plt.figure(figsize=(11, 4))
+plt.plot(line1[:, 0], line1[:, 1], "k:", label="LinearSVC")
+plt.plot(line2[:, 0], line2[:, 1], "b--", linewidth=2, label="SVC")
+plt.plot(line3[:, 0], line3[:, 1], "r-", label="SGDClassifier")
+plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs") # label="Iris-Versicolor"
+plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo") # label="Iris-Setosa"
+plt.xlabel("Petal length", fontsize=14)
+plt.ylabel("Petal width", fontsize=14)
+plt.legend(loc="upper center", fontsize=14)
+plt.axis([0, 5.5, 0, 2])
+
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    -
      -
    1. 1215-1225pm: Are Frode Helvig Kvanum, Gard Høivang, and David Andreas Bordvik, Next-day forecasts on spot prices for electricity
      2. -
    2. 1225-1235pm: Lidia Luque, Voxel-wise multi-label brain tumor classification
      3. -
    3. 1235-1245pm: Marcus Berget et al, Locating suspicious brain activity using neural networks
      4. -
    4. 1245-1255pm: William Ho and Tom-Ruben Traavik Kvalvaag, Comparing semi-supervised learning and supervised learning for image classification
      5. -
    -

    We will use the second part of the lecture for further discussions of projects 2 and 3 and a summary on boosting methods from last week. Feel free to bring your laptops.

    @@ -202,7 +288,7 @@ MathJax.Hub.Config({

  • 12
  • 13
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs004.html b/doc/pub/week46/html/._week46-bs004.html index aa405f9dd..bd5fa5c15 100644 --- a/doc/pub/week46/html/._week46-bs004.html +++ b/doc/pub/week46/html/._week46-bs004.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,35 +167,33 @@ MathJax.Hub.Config({

     

     

     

    -

    Support Vector Machines, overarching aims

    +

    What is a hyperplane?

    -

    A Support Vector Machine (SVM) is a very powerful and versatile -Machine Learning method, capable of performing linear or nonlinear -classification, regression, and even outlier detection. It is one of -the most popular models in Machine Learning, and anyone interested in -Machine Learning should have it in their toolbox. SVMs are -particularly well suited for classification of complex but small-sized or -medium-sized datasets. +

    The aim of the SVM algorithm is to find a hyperplane in a +\( p \)-dimensional space, where \( p \) is the number of features that +distinctly classifies the data points.

    -

    The case with two well-separated classes only can be understood in an -intuitive way in terms of lines in a two-dimensional space separating -the two classes (see figure below). +

    In a \( p \)-dimensional space, a hyperplane is what we call an affine subspace of dimension of \( p-1 \). +As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is +a two-dimensional subspace, or stated simply, a plane.

    -

    The basic mathematics behind the SVM is however less familiar to most of us. -It relies on the definition of hyperplanes and the -definition of a margin which separates classes (in case of -classification problems) of variables. It is also used for regression -problems. +

    In two dimensions, with the variables \( x_1 \) and \( x_2 \), the hyperplane is defined as

    +$$ +b+w_1x_1+w_2x_2=0, +$$ + +

    where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line +\( b+w_1x_1+w_2x_2=0 \). +In two dimensions we define the vectors \( \boldsymbol{x} =[x1,x2] \) and \( \boldsymbol{w}=[w1,w2] \). +We can then rewrite the above equation as

    -

    With SVMs we distinguish between hard margin and soft margins. The -latter introduces a so-called softening parameter to be discussed -below. We distinguish also between linear and non-linear -approaches. The latter are the most frequent ones since it is rather -unlikely that we can separate classes easily by say straight lines. -

    +$$ +\boldsymbol{x}^T\boldsymbol{w}+b=0. +$$ +

    @@ -223,7 +214,7 @@ unlikely that we can separate classes easily by say straight lines.

  • 13
  • 14
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs005.html b/doc/pub/week46/html/._week46-bs005.html index 64917d238..2ba60a3a0 100644 --- a/doc/pub/week46/html/._week46-bs005.html +++ b/doc/pub/week46/html/._week46-bs005.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,108 +167,45 @@ MathJax.Hub.Config({

     

     

     

    -

    Hyperplanes and all that

    +

    A \( p \)-dimensional space of features

    -

    The theory behind support vector machines (SVM hereafter) is based on -the mathematical description of so-called hyperplanes. Let us start -with a two-dimensional case. This will also allow us to introduce our -first SVM examples. These will be tailored to the case of two specific -classes, as displayed in the figure here based on the usage of the petal data. +

    We limit ourselves to two classes of outputs \( y_i \) and assign these classes the values \( y_i = \pm 1 \). +In a \( p \)-dimensional space of say \( p \) features we have a hyperplane defines as +

    +$$ +b+wx_1+w_2x_2+\dots +w_px_p=0. +$$ + +

    If we define a +matrix \( \boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right] \) +of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldsymbol{X} \), +

    +$$ +\boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. +$$ + +

    If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have

    +$$ +b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0, +$$ + +

    if our output \( y_i=1 \). +In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if +

    +$$ +b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0, +$$ + +

    for the class of observations \( y_i=-1 \), +then \( \boldsymbol{x}_i \) lies on the other side.

    -

    We assume here that our data set can be well separated into two -domains, where a straight line does the job in the separating the two -classes. Here the two classes are represented by either squares or -circles. -

    - - -
    -
    -
    -
    -
    - 
    from sklearn import datasets
-from sklearn.svm import SVC, LinearSVC
-from sklearn.linear_model import SGDClassifier
-from sklearn.preprocessing import StandardScaler
-import matplotlib
-import matplotlib.pyplot as plt
-plt.rcParams['axes.labelsize'] = 14
-plt.rcParams['xtick.labelsize'] = 12
-plt.rcParams['ytick.labelsize'] = 12
-
-
-iris = datasets.load_iris()
-X = iris["data"][:, (2, 3)]  # petal length, petal width
-y = iris["target"]
-
-setosa_or_versicolor = (y == 0) | (y == 1)
-X = X[setosa_or_versicolor]
-y = y[setosa_or_versicolor]
-
-
-
-C = 5
-alpha = 1 / (C * len(X))
-
-lin_clf = LinearSVC(loss="hinge", C=C, random_state=42)
-svm_clf = SVC(kernel="linear", C=C)
-sgd_clf = SGDClassifier(loss="hinge", learning_rate="constant", eta0=0.001, alpha=alpha,
-                        max_iter=100000, random_state=42)
-
-scaler = StandardScaler()
-X_scaled = scaler.fit_transform(X)
-
-lin_clf.fit(X_scaled, y)
-svm_clf.fit(X_scaled, y)
-sgd_clf.fit(X_scaled, y)
-
-print("LinearSVC:                   ", lin_clf.intercept_, lin_clf.coef_)
-print("SVC:                         ", svm_clf.intercept_, svm_clf.coef_)
-print("SGDClassifier(alpha={:.5f}):".format(sgd_clf.alpha), sgd_clf.intercept_, sgd_clf.coef_)
-
-# Compute the slope and bias of each decision boundary
-w1 = -lin_clf.coef_[0, 0]/lin_clf.coef_[0, 1]
-b1 = -lin_clf.intercept_[0]/lin_clf.coef_[0, 1]
-w2 = -svm_clf.coef_[0, 0]/svm_clf.coef_[0, 1]
-b2 = -svm_clf.intercept_[0]/svm_clf.coef_[0, 1]
-w3 = -sgd_clf.coef_[0, 0]/sgd_clf.coef_[0, 1]
-b3 = -sgd_clf.intercept_[0]/sgd_clf.coef_[0, 1]
-
-# Transform the decision boundary lines back to the original scale
-line1 = scaler.inverse_transform([[-10, -10 * w1 + b1], [10, 10 * w1 + b1]])
-line2 = scaler.inverse_transform([[-10, -10 * w2 + b2], [10, 10 * w2 + b2]])
-line3 = scaler.inverse_transform([[-10, -10 * w3 + b3], [10, 10 * w3 + b3]])
-
-# Plot all three decision boundaries
-plt.figure(figsize=(11, 4))
-plt.plot(line1[:, 0], line1[:, 1], "k:", label="LinearSVC")
-plt.plot(line2[:, 0], line2[:, 1], "b--", linewidth=2, label="SVC")
-plt.plot(line3[:, 0], line3[:, 1], "r-", label="SGDClassifier")
-plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs") # label="Iris-Versicolor"
-plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo") # label="Iris-Setosa"
-plt.xlabel("Petal length", fontsize=14)
-plt.ylabel("Petal width", fontsize=14)
-plt.legend(loc="upper center", fontsize=14)
-plt.axis([0, 5.5, 0, 2])
-
-plt.show()
-
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    +

    Equivalently, for the two classes of observations we have

    +$$ +y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0. +$$ +

    When we try to separate hyperplanes, if it exists, we can use it to construct a natural classifier: a test observation is assigned a given class depending on which side of the hyperplane it is located.

    @@ -297,7 +227,7 @@ plt.show()

  • 14
  • 15
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs006.html b/doc/pub/week46/html/._week46-bs006.html index c185e3b0a..d6e66576c 100644 --- a/doc/pub/week46/html/._week46-bs006.html +++ b/doc/pub/week46/html/._week46-bs006.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -173,35 +166,31 @@ MathJax.Hub.Config({

     

     

     

    - -

    What is a hyperplane?

    + +

    The two-dimensional case

    -

    The aim of the SVM algorithm is to find a hyperplane in a -\( p \)-dimensional space, where \( p \) is the number of features that -distinctly classifies the data points. +

    Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional +plane. To separate the two classes of data points, there are many +possible lines (hyperplanes if you prefer a more strict naming) +that could be chosen. Our objective is to find a +plane that has the maximum margin, i.e the maximum distance between +data points of both classes. Maximizing the margin distance provides +some reinforcement so that future data points can be classified with +more confidence.

    -

    In a \( p \)-dimensional space, a hyperplane is what we call an affine subspace of dimension of \( p-1 \). -As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is -a two-dimensional subspace, or stated simply, a plane. +

    What a linear classifier attempts to accomplish is to split the +feature space into two half spaces by placing a hyperplane between the +data points. This hyperplane will be our decision boundary. All +points on one side of the plane will belong to class one and all points +on the other side of the plane will belong to the second class two.

    -

    In two dimensions, with the variables \( x_1 \) and \( x_2 \), the hyperplane is defined as

    -$$ -b+w_1x_1+w_2x_2=0, -$$ - -

    where \( b \) is the intercept and \( w_1 \) and \( w_2 \) define the elements of a vector orthogonal to the line -\( b+w_1x_1+w_2x_2=0 \). -In two dimensions we define the vectors \( \boldsymbol{x} =[x1,x2] \) and \( \boldsymbol{w}=[w1,w2] \). -We can then rewrite the above equation as +

    Unfortunately there are many ways in which we can place a hyperplane +to divide the data. Below is an example of two candidate hyperplanes +for our data sample.

    -$$ -\boldsymbol{x}^T\boldsymbol{w}+b=0. -$$ - -

    diff --git a/doc/pub/week46/html/._week46-bs007.html b/doc/pub/week46/html/._week46-bs007.html index 3db00d5c6..7eb15446e 100644 --- a/doc/pub/week46/html/._week46-bs007.html +++ b/doc/pub/week46/html/._week46-bs007.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,45 +167,22 @@ MathJax.Hub.Config({

     

     

     

    -

    A \( p \)-dimensional space of features

    +

    Getting into the details

    -

    We limit ourselves to two classes of outputs \( y_i \) and assign these classes the values \( y_i = \pm 1 \). -In a \( p \)-dimensional space of say \( p \) features we have a hyperplane defines as -

    +

    Let us define the function

    $$ -b+wx_1+w_2x_2+\dots +w_px_p=0. +f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0, $$ -

    If we define a -matrix \( \boldsymbol{X}=\left[\boldsymbol{x}_1,\boldsymbol{x}_2,\dots, \boldsymbol{x}_p\right] \) -of dimension \( n\times p \), where \( n \) represents the observations for each feature and each vector \( x_i \) is a column vector of the matrix \( \boldsymbol{X} \), -

    +

    as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.

    + +

    Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).

    + +

    The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then

    $$ -\boldsymbol{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \dots \\ \dots \\ x_{ip} \end{bmatrix}. +\delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b). $$ -

    If the above condition is not met for a given vector \( \boldsymbol{x}_i \) we have

    -$$ -b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} >0, -$$ - -

    if our output \( y_i=1 \). -In this case we say that \( \boldsymbol{x}_i \) lies on one of the sides of the hyperplane and if -

    -$$ -b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip} < 0, -$$ - -

    for the class of observations \( y_i=-1 \), -then \( \boldsymbol{x}_i \) lies on the other side. -

    - -

    Equivalently, for the two classes of observations we have

    -$$ -y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) > 0. -$$ - -

    When we try to separate hyperplanes, if it exists, we can use it to construct a natural classifier: a test observation is assigned a given class depending on which side of the hyperplane it is located.

    @@ -236,7 +206,7 @@ $$

  • 16
  • 17
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs008.html b/doc/pub/week46/html/._week46-bs008.html index 3a07d05ef..686322f59 100644 --- a/doc/pub/week46/html/._week46-bs008.html +++ b/doc/pub/week46/html/._week46-bs008.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -173,30 +166,28 @@ MathJax.Hub.Config({

     

     

     

    - -

    The two-dimensional case

    + +

    First attempt at a minimization approach

    -

    Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional -plane. To separate the two classes of data points, there are many -possible lines (hyperplanes if you prefer a more strict naming) -that could be chosen. Our objective is to find a -plane that has the maximum margin, i.e the maximum distance between -data points of both classes. Maximizing the margin distance provides -some reinforcement so that future data points can be classified with -more confidence. +

    How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could +do is to define a cost function which now contains the set of all +misclassified points \( M \) and attempt to minimize this function

    -

    What a linear classifier attempts to accomplish is to split the -feature space into two half spaces by placing a hyperplane between the -data points. This hyperplane will be our decision boundary. All -points on one side of the plane will belong to class one and all points -on the other side of the plane will belong to the second class two. -

    +$$ +C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b). +$$ + +

    We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us

    +$$ +\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i, +$$ + +

    and

    +$$ +\frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i. +$$ -

    Unfortunately there are many ways in which we can place a hyperplane -to divide the data. Below is an example of two candidate hyperplanes -for our data sample. -

    @@ -221,7 +212,7 @@ for our data sample.

  • 17
  • 18
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs009.html b/doc/pub/week46/html/._week46-bs009.html index a58888d25..652bbfd87 100644 --- a/doc/pub/week46/html/._week46-bs009.html +++ b/doc/pub/week46/html/._week46-bs009.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,22 +167,19 @@ MathJax.Hub.Config({

     

     

     

    -

    Getting into the details

    +

    Solving the equations

    -

    Let us define the function

    +

    We can now use the Newton-Raphson method or different variants of the gradient descent family (from plain gradient descent to various stochastic gradient descent approaches) to solve the equations

    $$ -f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0, +b \leftarrow b +\eta \frac{\partial C}{\partial b}, $$ -

    as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.

    - -

    Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).

    - -

    The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then

    +

    and

    $$ -\delta = \frac{1}{\vert\vert \boldsymbol{w}\vert\vert}(\boldsymbol{w}^T\boldsymbol{x}+b). +\boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}}, $$ +

    where \( \eta \) is our by now well-known learning rate.

    @@ -215,7 +205,7 @@ $$

  • 18
  • 19
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs010.html b/doc/pub/week46/html/._week46-bs010.html index 41f0e1101..a08f4f449 100644 --- a/doc/pub/week46/html/._week46-bs010.html +++ b/doc/pub/week46/html/._week46-bs010.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,26 +167,34 @@ MathJax.Hub.Config({

     

     

     

    -

    First attempt at a minimization approach

    +

    Code Example

    -

    How do we find the parameter \( b \) and the vector \( \boldsymbol{w} \)? What we could -do is to define a cost function which now contains the set of all -misclassified points \( M \) and attempt to minimize this function +

    The equations we discussed above can be coded rather easily (the +framework is similar to what we developed for logistic +regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way.

    -$$ -C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b). -$$ - -

    We could now for example define all values \( y_i =1 \) as misclassified in case we have \( \boldsymbol{w}^T\boldsymbol{x}_i+b < 0 \) and the opposite if we have \( y_i=-1 \). Taking the derivatives gives us

    -$$ -\frac{\partial C}{\partial b} = -\sum_{i\in M} y_i, -$$ - -

    and

    -$$ -\frac{\partial C}{\partial \boldsymbol{w}} = -\sum_{i\in M} y_ix_i. -$$ + +
    +
    +
    +
    +
    + 
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +

    @@ -221,7 +222,7 @@ $$

  • 19
  • 20
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs011.html b/doc/pub/week46/html/._week46-bs011.html index 4329e6107..29aacb6ac 100644 --- a/doc/pub/week46/html/._week46-bs011.html +++ b/doc/pub/week46/html/._week46-bs011.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,19 +167,18 @@ MathJax.Hub.Config({

     

     

     

    -

    Solving the equations

    +

    Problems with the Simpler Approach

    -

    We can now use the Newton-Raphson method or different variants of the gradient descent family (from plain gradient descent to various stochastic gradient descent approaches) to solve the equations

    -$$ -b \leftarrow b +\eta \frac{\partial C}{\partial b}, -$$ +

    There are however problems with this approach, although it looks +pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes. +

    -

    and

    -$$ -\boldsymbol{w} \leftarrow \boldsymbol{w} +\eta \frac{\partial C}{\partial \boldsymbol{w}}, -$$ - -

    where \( \eta \) is our by now well-known learning rate.

    +

    For small +gaps between the entries, we may also end up needing many iterations +before the solutions converge and if the data cannot be separated +properly into two distinct classes, we may not experience a converge +at all. +

    @@ -213,7 +205,7 @@ $$

  • 20
  • 21
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs012.html b/doc/pub/week46/html/._week46-bs012.html index 2013063b5..3337c7693 100644 --- a/doc/pub/week46/html/._week46-bs012.html +++ b/doc/pub/week46/html/._week46-bs012.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,35 +167,44 @@ MathJax.Hub.Config({

     

     

     

    -

    Code Example

    +

    A better approach

    -

    The equations we discussed above can be coded rather easily (the -framework is similar to what we developed for logistic -regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way. +

    A better approach is rather to try to define a large margin between +the two classes (if they are well separated from the beginning).

    - -
    -
    -
    -
    -
    - 
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    +

    Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to +\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition +

    +$$ +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +$$ + +

    All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.

    + +

    We seek thus the largest value \( M \) defined by

    +$$ +\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, +$$ + +

    or just

    +$$ +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i. +$$ + +

    If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of +\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition +

    +$$ +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +$$ + +

    We have thus defined our margin as the invers of the norm of +\( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as +possible margin \( M \). Before we proceed, we need to remind ourselves +about Lagrangian multipliers. +

    @@ -229,7 +231,7 @@ regression). We are going to set up a simple case with two classes only and we w

  • 21
  • 22
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs013.html b/doc/pub/week46/html/._week46-bs013.html index 2e4504c1c..0973bab77 100644 --- a/doc/pub/week46/html/._week46-bs013.html +++ b/doc/pub/week46/html/._week46-bs013.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,17 +167,53 @@ MathJax.Hub.Config({

     

     

     

    -

    Problems with the Simpler Approach

    +

    A quick Reminder on Lagrangian Multipliers

    -

    There are however problems with this approach, although it looks -pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes. +

    Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an +extreme we have +

    +$$ +df=0. +$$ + +

    A necessary and sufficient condition is

    +$$ +\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0, +$$ + +

    due to

    +$$ +df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz. +$$ + +

    In many problems the variables \( x,y,z \) are often subject to constraints (such as those above for the margin) +so that they are no longer all independent. It is possible at least in principle to use each +constraint to eliminate one variable +and to proceed with a new and smaller set of independent varables.

    -

    For small -gaps between the entries, we may also end up needing many iterations -before the solutions converge and if the data cannot be separated -properly into two distinct classes, we may not experience a converge -at all. +

    The use of so-called Lagrangian multipliers is an alternative technique when the elimination +of variables is incovenient or undesirable. Assume that we have an equation of constraint on +the variables \( x,y,z \) +

    +$$ +\phi(x,y,z) = 0, +$$ + +

    resulting in

    +$$ +d\phi = \frac{\partial \phi}{\partial x}dx+\frac{\partial \phi}{\partial y}dy+\frac{\partial \phi}{\partial z}dz =0. +$$ + +

    Now we cannot set anymore

    +$$ +\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0, +$$ + +

    if \( df=0 \) is wanted +because there are now only two independent variables! Assume \( x \) and \( y \) are the independent +variables. +Then \( dz \) is no longer arbitrary.

    @@ -212,7 +241,7 @@ at all.

  • 22
  • 23
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs014.html b/doc/pub/week46/html/._week46-bs014.html index a11d76e9f..2386f232d 100644 --- a/doc/pub/week46/html/._week46-bs014.html +++ b/doc/pub/week46/html/._week46-bs014.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,44 +167,45 @@ MathJax.Hub.Config({

     

     

     

    -

    A better approach

    - -

    A better approach is rather to try to define a large margin between -the two classes (if they are well separated from the beginning). -

    - -

    Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to -\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition -

    +

    Adding the Multiplier

    +

    However, we can add to

    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz, $$ -

    All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.

    - -

    We seek thus the largest value \( M \) defined by

    +

    a multiplum of \( d\phi \), viz. \( \lambda d\phi \), resulting in

    $$ -\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, +df+\lambda d\phi = (\frac{\partial f}{\partial z}+\lambda +\frac{\partial \phi}{\partial x})dx+(\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y})dy+ +(\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z})dz =0. $$ -

    or just

    +

    Our multiplier is chosen so that

    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i. +\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z} =0. $$ -

    If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of -\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition +

    We need to remember that we took \( dx \) and \( dy \) to be arbitrary and thus we must have

    +$$ +\frac{\partial f}{\partial x}+\lambda\frac{\partial \phi}{\partial x} =0, +$$ + +

    and

    +$$ +\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y} =0. +$$ + +

    When all these equations are satisfied, \( df=0 \). We have four unknowns, \( x,y,z \) and +\( \lambda \). Actually we want only \( x,y,z \), \( \lambda \) needs not to be determined, +it is therefore often called +Lagrange's undetermined multiplier. +If we have a set of constraints \( \phi_k \) we have the equations

    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +\frac{\partial f}{\partial x_i}+\sum_k\lambda_k\frac{\partial \phi_k}{\partial x_i} =0. $$ -

    We have thus defined our margin as the invers of the norm of -\( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as -possible margin \( M \). Before we proceed, we need to remind ourselves -about Lagrangian multipliers. -

    @@ -238,7 +232,7 @@ about Lagrangian multipliers.

  • 23
  • 24
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs015.html b/doc/pub/week46/html/._week46-bs015.html index 029f3f444..2d8716d5c 100644 --- a/doc/pub/week46/html/._week46-bs015.html +++ b/doc/pub/week46/html/._week46-bs015.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,54 +167,41 @@ MathJax.Hub.Config({

     

     

     

    -

    A quick Reminder on Lagrangian Multipliers

    +

    Setting up the Problem

    +

    In order to solve the above problem, we define the following Lagrangian function to be minimized

    +$$ +{\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right], +$$ -

    Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an -extreme we have +

    where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).

    + +

    Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain

    +$$ +\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, +$$ + +

    and

    +$$ +\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i. +$$ + +

    Inserting these constraints into the equation for \( {\cal L} \) we obtain

    +$$ +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, +$$ + +

    subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). +We must in addition satisfy the Karush-Kuhn-Tucker (KKT) condition

    $$ -df=0. +\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i. $$ -

    A necessary and sufficient condition is

    -$$ -\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0, -$$ - -

    due to

    -$$ -df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz. -$$ - -

    In many problems the variables \( x,y,z \) are often subject to constraints (such as those above for the margin) -so that they are no longer all independent. It is possible at least in principle to use each -constraint to eliminate one variable -and to proceed with a new and smaller set of independent varables. -

    - -

    The use of so-called Lagrangian multipliers is an alternative technique when the elimination -of variables is incovenient or undesirable. Assume that we have an equation of constraint on -the variables \( x,y,z \) -

    -$$ -\phi(x,y,z) = 0, -$$ - -

    resulting in

    -$$ -d\phi = \frac{\partial \phi}{\partial x}dx+\frac{\partial \phi}{\partial y}dy+\frac{\partial \phi}{\partial z}dz =0. -$$ - -

    Now we cannot set anymore

    -$$ -\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0, -$$ - -

    if \( df=0 \) is wanted -because there are now only two independent variables! Assume \( x \) and \( y \) are the independent -variables. -Then \( dz \) is no longer arbitrary. -

    +
      +
    1. If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.
      2. +
    2. If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).
      3. +
    +

    When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).

    @@ -248,7 +228,7 @@ Then \( dz \) is no longer arbitrary.

  • 24
  • 25
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs016.html b/doc/pub/week46/html/._week46-bs016.html index 7c8741cab..8a10a3464 100644 --- a/doc/pub/week46/html/._week46-bs016.html +++ b/doc/pub/week46/html/._week46-bs016.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,45 +167,26 @@ MathJax.Hub.Config({

     

     

     

    -

    Adding the Multiplier

    +

    The problem to solve

    -

    However, we can add to

    +

    We can rewrite

    $$ -df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, $$ -

    a multiplum of \( d\phi \), viz. \( \lambda d\phi \), resulting in

    +

    and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem

    $$ -df+\lambda d\phi = (\frac{\partial f}{\partial z}+\lambda -\frac{\partial \phi}{\partial x})dx+(\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y})dy+ -(\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z})dz =0. +\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\ +y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\ +\dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots \\ +y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 & \dots & \dots & y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\ +\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, $$ -

    Our multiplier is chosen so that

    -$$ -\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z} =0. -$$ - -

    We need to remember that we took \( dx \) and \( dy \) to be arbitrary and thus we must have

    -$$ -\frac{\partial f}{\partial x}+\lambda\frac{\partial \phi}{\partial x} =0, -$$ - -

    and

    -$$ -\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y} =0. -$$ - -

    When all these equations are satisfied, \( df=0 \). We have four unknowns, \( x,y,z \) and -\( \lambda \). Actually we want only \( x,y,z \), \( \lambda \) needs not to be determined, -it is therefore often called -Lagrange's undetermined multiplier. -If we have a set of constraints \( \phi_k \) we have the equations +

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).

    -$$ -\frac{\partial f}{\partial x_i}+\sum_k\lambda_k\frac{\partial \phi_k}{\partial x_i} =0. -$$ -

    @@ -239,7 +213,7 @@ $$

  • 25
  • 26
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs017.html b/doc/pub/week46/html/._week46-bs017.html index 5931e53e6..00e874794 100644 --- a/doc/pub/week46/html/._week46-bs017.html +++ b/doc/pub/week46/html/._week46-bs017.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,41 +167,36 @@ MathJax.Hub.Config({

     

     

     

    -

    Setting up the Problem

    -

    In order to solve the above problem, we define the following Lagrangian function to be minimized

    -$$ -{\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right], -$$ +

    The last steps

    -

    where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).

    - -

    Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain

    -$$ -\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, -$$ - -

    and

    -$$ -\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i. -$$ - -

    Inserting these constraints into the equation for \( {\cal L} \) we obtain

    -$$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, -$$ - -

    subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \). -We must in addition satisfy the Karush-Kuhn-Tucker (KKT) condition +

    Solving the above problem, yields the values of \( \lambda_i \). +To find the coefficients of your hyperplane we need simply to compute

    $$ -\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i. +\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i. $$ -
      -
    1. If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.
      2. -
    2. If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).
      3. -
    -

    When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).

    +

    With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via

    +$$ +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1, +$$ + +

    resulting in

    +$$ +b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i, +$$ + +

    or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have

    +$$ +b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right). +$$ + +

    With our hyperplane coefficients we can use our classifier to assign any observation by simply using

    +$$ +y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b). +$$ + +

    Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.

    @@ -235,7 +223,7 @@ $$

  • 26
  • 27
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs018.html b/doc/pub/week46/html/._week46-bs018.html index 220c3f4f1..c13a0d6c5 100644 --- a/doc/pub/week46/html/._week46-bs018.html +++ b/doc/pub/week46/html/._week46-bs018.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,25 +167,36 @@ MathJax.Hub.Config({

     

     

     

    -

    The problem to solve

    +

    A soft classifier

    -

    We can rewrite

    +

    Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.

    + +

    Suppose now that classes overlap in feature space, as shown in the +figure here. One way to deal with this problem before we define the +so-called kernel approach, is to allow a kind of slack in the sense +that we allow some points to be on the wrong side of the margin. +

    + +

    We introduce thus the so-called slack variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and +modify our previous equation +

    $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1, $$ -

    and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem

    +

    to

    $$ -\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\ -y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\ -\dots & \dots & \dots & \dots & \dots \\ -\dots & \dots & \dots & \dots & \dots \\ -y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 & \dots & \dots & y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\ -\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i, $$ -

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +

    with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \). +The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction +\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), +we bound the total amount by which predictions fall on the wrong side of their margins. +

    + +

    Misclassifications occur when \( \xi_i > 1 \). Thus bounding the total sum by some value \( C \) bounds in turn the total number of +misclassifications.

    @@ -220,7 +224,7 @@ $$

  • 27
  • 28
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs019.html b/doc/pub/week46/html/._week46-bs019.html index 9b4bcb338..c8db5b730 100644 --- a/doc/pub/week46/html/._week46-bs019.html +++ b/doc/pub/week46/html/._week46-bs019.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,36 +167,56 @@ MathJax.Hub.Config({

     

     

     

    -

    The last steps

    +

    Soft optmization problem

    -

    Solving the above problem, yields the values of \( \lambda_i \). -To find the coefficients of your hyperplane we need simply to compute +

    This has in turn the consequences that we change our optmization problem to finding the minimum of

    +$$ +{\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, +$$ + +

    subject to

    +$$ +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, +$$ + +

    with the requirement \( \xi_i\geq 0 \).

    + +

    Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain

    +$$ +\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, +$$ + +

    and

    +$$ +\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i, +$$ + +

    and

    +$$ +\lambda_i = C-\gamma_i \hspace{0.1cm}\forall i. +$$ + +

    Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before

    +$$ +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, +$$ + +

    but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \). +We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads

    $$ -\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i. +\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, $$ -

    With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via

    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1, +\gamma_i\xi_i = 0, $$ -

    resulting in

    +

    and

    $$ -b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i, +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. $$ -

    or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have

    -$$ -b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right). -$$ - -

    With our hyperplane coefficients we can use our classifier to assign any observation by simply using

    -$$ -y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b). -$$ - -

    Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.

    @@ -229,8 +242,6 @@ $$

  • 27
  • 28
  • 29
    • -
  • ...
    • -
  • 31
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs020.html b/doc/pub/week46/html/._week46-bs020.html index 540f2f916..0299de352 100644 --- a/doc/pub/week46/html/._week46-bs020.html +++ b/doc/pub/week46/html/._week46-bs020.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,38 +167,95 @@ MathJax.Hub.Config({

     

     

     

    -

    A soft classifier

    +

    Kernels and non-linearity

    -

    Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.

    - -

    Suppose now that classes overlap in feature space, as shown in the -figure here. One way to deal with this problem before we define the -so-called kernel approach, is to allow a kind of slack in the sense -that we allow some points to be on the wrong side of the margin. +

    The cases we have studied till now, were all characterized by two classes +with a close to linear separability. The classifiers we have described +so far find linear boundaries in our input feature space. It is +possible to make our procedure more flexible by exploring the feature +space using other basis expansions such as higher-order polynomials, +wavelets, splines etc.

    -

    We introduce thus the so-called slack variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and -modify our previous equation -

    -$$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1, -$$ - -

    to

    -$$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i, -$$ - -

    with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \). -The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction -\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \), -we bound the total amount by which predictions fall on the wrong side of their margins. +

    If our feature space is not easy to separate, as shown in the figure +here, we can achieve a better separation by introducing more complex +basis functions. The ideal would be, as shown in the next figure, to, via a specific transformation to +obtain a separation between the classes which is almost linear.

    -

    Misclassifications occur when \( \xi_i > 1 \). Thus bounding the total sum by some value \( C \) bounds in turn the total number of -misclassifications. +

    The change of basis, from \( x\rightarrow z=\phi(x) \) leads to the same type of equations to be solved, except that +we need to introduce for example a polynomial transformation to a two-dimensional training set.

    + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import os
+
+np.random.seed(42)
+
+# To plot pretty figures
+import matplotlib
+import matplotlib.pyplot as plt
+plt.rcParams['axes.labelsize'] = 14
+plt.rcParams['xtick.labelsize'] = 12
+plt.rcParams['ytick.labelsize'] = 12
+
+
+from sklearn.svm import SVC
+from sklearn import datasets
+
+
+
+X1D = np.linspace(-4, 4, 9).reshape(-1, 1)
+X2D = np.c_[X1D, X1D**2]
+y = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])
+
+plt.figure(figsize=(11, 4))
+
+plt.subplot(121)
+plt.grid(True, which='both')
+plt.axhline(y=0, color='k')
+plt.plot(X1D[:, 0][y==0], np.zeros(4), "bs")
+plt.plot(X1D[:, 0][y==1], np.zeros(5), "g^")
+plt.gca().get_yaxis().set_ticks([])
+plt.xlabel(r"$x_1$", fontsize=20)
+plt.axis([-4.5, 4.5, -0.2, 0.2])
+
+plt.subplot(122)
+plt.grid(True, which='both')
+plt.axhline(y=0, color='k')
+plt.axvline(x=0, color='k')
+plt.plot(X2D[:, 0][y==0], X2D[:, 1][y==0], "bs")
+plt.plot(X2D[:, 0][y==1], X2D[:, 1][y==1], "g^")
+plt.xlabel(r"$x_1$", fontsize=20)
+plt.ylabel(r"$x_2$", fontsize=20, rotation=0)
+plt.gca().get_yaxis().set_ticks([0, 4, 8, 12, 16])
+plt.plot([-4.5, 4.5], [6.5, 6.5], "r--", linewidth=3)
+plt.axis([-4.5, 4.5, -1, 17])
+plt.subplots_adjust(right=1)
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    diff --git a/doc/pub/week46/html/._week46-bs021.html b/doc/pub/week46/html/._week46-bs021.html index f76611f08..823878b84 100644 --- a/doc/pub/week46/html/._week46-bs021.html +++ b/doc/pub/week46/html/._week46-bs021.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,56 +167,46 @@ MathJax.Hub.Config({

     

     

     

    -

    Soft optmization problem

    +

    The equations

    -

    This has in turn the consequences that we change our optmization problem to finding the minimum of

    +

    Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)

    $$ -{\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i, +z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right). $$ -

    subject to

    +

    With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)

    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{z}_j, $$ -

    with the requirement \( \xi_i\geq 0 \).

    - -

    Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain

    +

    subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors

    $$ -\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0, +y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i, $$ -

    and

    -$$ -\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i, -$$ - -

    and

    -$$ -\lambda_i = C-\gamma_i \hspace{0.1cm}\forall i. -$$ - -

    Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before

    -$$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j, -$$ - -

    but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \). -We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads +

    from which we also find \( b \). +To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kernel \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as

    $$ -\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i, +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). $$ +

    For the above example, the kernel reads

    $$ -\gamma_i\xi_i = 0, +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2. $$ -

    and

    -$$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i. -$$ +

    We note that this is nothing but the dot product of the two original +vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the +product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute +the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). +

    +

    This leads to the so-called +kernel trick and the result leads to the same as if we went through +the trouble of performing the transformation +\( \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations. +

    @@ -247,8 +230,6 @@ $$

  • 27
  • 28
  • 29
    • -
  • 30
    • -
  • 31
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs022.html b/doc/pub/week46/html/._week46-bs022.html index c2a80a9bb..217c7f12f 100644 --- a/doc/pub/week46/html/._week46-bs022.html +++ b/doc/pub/week46/html/._week46-bs022.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,95 +167,40 @@ MathJax.Hub.Config({

     

     

     

    -

    Kernels and non-linearity

    +

    The problem to solve

    +

    Using our definition of the kernel We can rewrite again the Lagrangian

    +$$ +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j, +$$ -

    The cases we have studied till now, were all characterized by two classes -with a close to linear separability. The classifiers we have described -so far find linear boundaries in our input feature space. It is -possible to make our procedure more flexible by exploring the feature -space using other basis expansions such as higher-order polynomials, -wavelets, splines etc. +

    subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem

    +$$ +\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ +y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ +\dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots \\ +y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ +\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, +$$ + +

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +If we add the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).

    -

    If our feature space is not easy to separate, as shown in the figure -here, we can achieve a better separation by introducing more complex -basis functions. The ideal would be, as shown in the next figure, to, via a specific transformation to -obtain a separation between the classes which is almost linear. +

    We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type

    +$$ +\begin{align*} + &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f. +\end{align*} +$$ + +

    Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \). +Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into +\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).

    -

    The change of basis, from \( x\rightarrow z=\phi(x) \) leads to the same type of equations to be solved, except that -we need to introduce for example a polynomial transformation to a two-dimensional training set. -

    - - - -
    -
    -
    -
    -
    - 
    import numpy as np
-import os
-
-np.random.seed(42)
-
-# To plot pretty figures
-import matplotlib
-import matplotlib.pyplot as plt
-plt.rcParams['axes.labelsize'] = 14
-plt.rcParams['xtick.labelsize'] = 12
-plt.rcParams['ytick.labelsize'] = 12
-
-
-from sklearn.svm import SVC
-from sklearn import datasets
-
-
-
-X1D = np.linspace(-4, 4, 9).reshape(-1, 1)
-X2D = np.c_[X1D, X1D**2]
-y = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])
-
-plt.figure(figsize=(11, 4))
-
-plt.subplot(121)
-plt.grid(True, which='both')
-plt.axhline(y=0, color='k')
-plt.plot(X1D[:, 0][y==0], np.zeros(4), "bs")
-plt.plot(X1D[:, 0][y==1], np.zeros(5), "g^")
-plt.gca().get_yaxis().set_ticks([])
-plt.xlabel(r"$x_1$", fontsize=20)
-plt.axis([-4.5, 4.5, -0.2, 0.2])
-
-plt.subplot(122)
-plt.grid(True, which='both')
-plt.axhline(y=0, color='k')
-plt.axvline(x=0, color='k')
-plt.plot(X2D[:, 0][y==0], X2D[:, 1][y==0], "bs")
-plt.plot(X2D[:, 0][y==1], X2D[:, 1][y==1], "g^")
-plt.xlabel(r"$x_1$", fontsize=20)
-plt.ylabel(r"$x_2$", fontsize=20, rotation=0)
-plt.gca().get_yaxis().set_ticks([0, 4, 8, 12, 16])
-plt.plot([-4.5, 4.5], [6.5, 6.5], "r--", linewidth=3)
-plt.axis([-4.5, 4.5, -1, 17])
-plt.subplots_adjust(right=1)
-plt.show()
-
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    - -

    diff --git a/doc/pub/week46/html/._week46-bs023.html b/doc/pub/week46/html/._week46-bs023.html index 3a9334e73..69f46ba79 100644 --- a/doc/pub/week46/html/._week46-bs023.html +++ b/doc/pub/week46/html/._week46-bs023.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,45 +167,36 @@ MathJax.Hub.Config({

     

     

     

    -

    The equations

    +

    Different kernels and Mercer's theorem

    -

    Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)

    -$$ -z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right). -$$ +

    There are several popular kernels being used. These are

    +
      +
    1. Linear: \( K(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{x}^T\boldsymbol{y} \),
      2. +
    2. Polynomial: \( K(\boldsymbol{x},\boldsymbol{y})=(\boldsymbol{x}^T\boldsymbol{y}+\gamma)^d \),
      3. +
    3. Gaussian Radial Basis Function: \( K(\boldsymbol{x},\boldsymbol{y})=\exp{\left(-\gamma\vert\vert\boldsymbol{x}-\boldsymbol{y}\vert\vert^2\right)} \),
      4. +
    4. Tanh: \( K(\boldsymbol{x},\boldsymbol{y})=\tanh{(\boldsymbol{x}^T\boldsymbol{y}+\gamma)} \),
      5. +
    +

    and many other ones.

    -

    With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)

    -$$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{z}_j, -$$ - -

    subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors

    -$$ -y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i, -$$ - -

    from which we also find \( b \). -To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kernel \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as -

    -$$ -K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). -$$ - -

    For the above example, the kernel reads

    -$$ -K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2. -$$ - -

    We note that this is nothing but the dot product of the two original -vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the -product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute -the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). +

    An important theorem for us is Mercer's +theorem. The +theorem states that if a kernel function \( K \) is symmetric, continuous +and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there +exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into +another space (possibly with much higher dimensions) such that

    -

    This leads to the so-called -kernel trick and the result leads to the same as if we went through -the trouble of performing the transformation -\( \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations. +$$ +K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). +$$ + +

    So you can use \( K \) as a kernel since you know \( \phi \) exists, even if +you don’t know what \( \phi \) is. +

    + +

    Note that some frequently used kernels (such as the Sigmoid kernel) +don’t respect all of Mercer’s conditions, yet they generally work well +in practice.

    @@ -235,8 +219,6 @@ the trouble of performing the transformation

  • 27
  • 28
  • 29
    • -
  • 30
    • -
  • 31
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs024.html b/doc/pub/week46/html/._week46-bs024.html index 92e0ef84a..f8fdb893d 100644 --- a/doc/pub/week46/html/._week46-bs024.html +++ b/doc/pub/week46/html/._week46-bs024.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,39 +167,217 @@ MathJax.Hub.Config({

     

     

     

    -

    The problem to solve

    -

    Using our definition of the kernel We can rewrite again the Lagrangian

    -$$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j, -$$ +

    The moons example

    -

    subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem

    -$$ -\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ -y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ -\dots & \dots & \dots & \dots & \dots \\ -\dots & \dots & \dots & \dots & \dots \\ -y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ -\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, -$$ + +
    +
    +
    +
    +
    + 
    from __future__ import division, print_function, unicode_literals
 
-
    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and 
-\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). 
-If we add the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
-
    
+import numpy as np
+np.random.seed(42)
 
-
    We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type
    
-$$
-\begin{align*}
-    &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
-    &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
-\end{align*}
-$$
+import matplotlib
+import matplotlib.pyplot as plt
+plt.rcParams['axes.labelsize'] = 14
+plt.rcParams['xtick.labelsize'] = 12
+plt.rcParams['ytick.labelsize'] = 12
+
+
+from sklearn.svm import SVC
+from sklearn import datasets
+
+
+
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler
+from sklearn.svm import LinearSVC
+
+
+from sklearn.datasets import make_moons
+X, y = make_moons(n_samples=100, noise=0.15, random_state=42)
+
+def plot_dataset(X, y, axes):
+    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "bs")
+    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "g^")
+    plt.axis(axes)
+    plt.grid(True, which='both')
+    plt.xlabel(r"$x_1$", fontsize=20)
+    plt.ylabel(r"$x_2$", fontsize=20, rotation=0)
+
+plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+plt.show()
+
+from sklearn.datasets import make_moons
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import PolynomialFeatures
+
+polynomial_svm_clf = Pipeline([
+        ("poly_features", PolynomialFeatures(degree=3)),
+        ("scaler", StandardScaler()),
+        ("svm_clf", LinearSVC(C=10, loss="hinge", random_state=42))
+    ])
+
+polynomial_svm_clf.fit(X, y)
+
+def plot_predictions(clf, axes):
+    x0s = np.linspace(axes[0], axes[1], 100)
+    x1s = np.linspace(axes[2], axes[3], 100)
+    x0, x1 = np.meshgrid(x0s, x1s)
+    X = np.c_[x0.ravel(), x1.ravel()]
+    y_pred = clf.predict(X).reshape(x0.shape)
+    y_decision = clf.decision_function(X).reshape(x0.shape)
+    plt.contourf(x0, x1, y_pred, cmap=plt.cm.brg, alpha=0.2)
+    plt.contourf(x0, x1, y_decision, cmap=plt.cm.brg, alpha=0.1)
+
+plot_predictions(polynomial_svm_clf, [-1.5, 2.5, -1, 1.5])
+plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+
+plt.show()
+
+
+from sklearn.svm import SVC
+
+poly_kernel_svm_clf = Pipeline([
+        ("scaler", StandardScaler()),
+        ("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5))
+    ])
+poly_kernel_svm_clf.fit(X, y)
+
+poly100_kernel_svm_clf = Pipeline([
+        ("scaler", StandardScaler()),
+        ("svm_clf", SVC(kernel="poly", degree=10, coef0=100, C=5))
+    ])
+poly100_kernel_svm_clf.fit(X, y)
+
+plt.figure(figsize=(11, 4))
+
+plt.subplot(121)
+plot_predictions(poly_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])
+plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+plt.title(r"$d=3, r=1, C=5$", fontsize=18)
+
+plt.subplot(122)
+plot_predictions(poly100_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])
+plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+plt.title(r"$d=10, r=100, C=5$", fontsize=18)
+
+plt.show()
+
+def gaussian_rbf(x, landmark, gamma):
+    return np.exp(-gamma * np.linalg.norm(x - landmark, axis=1)**2)
+
+gamma = 0.3
+
+x1s = np.linspace(-4.5, 4.5, 200).reshape(-1, 1)
+x2s = gaussian_rbf(x1s, -2, gamma)
+x3s = gaussian_rbf(x1s, 1, gamma)
+
+XK = np.c_[gaussian_rbf(X1D, -2, gamma), gaussian_rbf(X1D, 1, gamma)]
+yk = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])
+
+plt.figure(figsize=(11, 4))
+
+plt.subplot(121)
+plt.grid(True, which='both')
+plt.axhline(y=0, color='k')
+plt.scatter(x=[-2, 1], y=[0, 0], s=150, alpha=0.5, c="red")
+plt.plot(X1D[:, 0][yk==0], np.zeros(4), "bs")
+plt.plot(X1D[:, 0][yk==1], np.zeros(5), "g^")
+plt.plot(x1s, x2s, "g--")
+plt.plot(x1s, x3s, "b:")
+plt.gca().get_yaxis().set_ticks([0, 0.25, 0.5, 0.75, 1])
+plt.xlabel(r"$x_1$", fontsize=20)
+plt.ylabel(r"Similarity", fontsize=14)
+plt.annotate(r'$\mathbf{x}$',
+             xy=(X1D[3, 0], 0),
+             xytext=(-0.5, 0.20),
+             ha="center",
+             arrowprops=dict(facecolor='black', shrink=0.1),
+             fontsize=18,
+            )
+plt.text(-2, 0.9, "$x_2$", ha="center", fontsize=20)
+plt.text(1, 0.9, "$x_3$", ha="center", fontsize=20)
+plt.axis([-4.5, 4.5, -0.1, 1.1])
+
+plt.subplot(122)
+plt.grid(True, which='both')
+plt.axhline(y=0, color='k')
+plt.axvline(x=0, color='k')
+plt.plot(XK[:, 0][yk==0], XK[:, 1][yk==0], "bs")
+plt.plot(XK[:, 0][yk==1], XK[:, 1][yk==1], "g^")
+plt.xlabel(r"$x_2$", fontsize=20)
+plt.ylabel(r"$x_3$  ", fontsize=20, rotation=0)
+plt.annotate(r'$\phi\left(\mathbf{x}\right)$',
+             xy=(XK[3, 0], XK[3, 1]),
+             xytext=(0.65, 0.50),
+             ha="center",
+             arrowprops=dict(facecolor='black', shrink=0.1),
+             fontsize=18,
+            )
+plt.plot([-0.1, 1.1], [0.57, -0.1], "r--", linewidth=3)
+plt.axis([-0.1, 1.1, -0.1, 1.1])
+    
+plt.subplots_adjust(right=1)
+
+plt.show()
+
+
+x1_example = X1D[3, 0]
+for landmark in (-2, 1):
+    k = gaussian_rbf(np.array([[x1_example]]), np.array([[landmark]]), gamma)
+    print("Phi({}, {}) = {}".format(x1_example, landmark, k))
+
+rbf_kernel_svm_clf = Pipeline([
+        ("scaler", StandardScaler()),
+        ("svm_clf", SVC(kernel="rbf", gamma=5, C=0.001))
+    ])
+rbf_kernel_svm_clf.fit(X, y)
+
+
+from sklearn.svm import SVC
+
+gamma1, gamma2 = 0.1, 5
+C1, C2 = 0.001, 1000
+hyperparams = (gamma1, C1), (gamma1, C2), (gamma2, C1), (gamma2, C2)
+
+svm_clfs = []
+for gamma, C in hyperparams:
+    rbf_kernel_svm_clf = Pipeline([
+            ("scaler", StandardScaler()),
+            ("svm_clf", SVC(kernel="rbf", gamma=gamma, C=C))
+        ])
+    rbf_kernel_svm_clf.fit(X, y)
+    svm_clfs.append(rbf_kernel_svm_clf)
+
+plt.figure(figsize=(11, 7))
+
+for i, svm_clf in enumerate(svm_clfs):
+    plt.subplot(221 + i)
+    plot_predictions(svm_clf, [-1.5, 2.5, -1, 1.5])
+    plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
+    gamma, C = hyperparams[i]
+    plt.title(r"$\gamma = {}, C = {}$".format(gamma, C), fontsize=16)
+
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    -

    Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \). -Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into -\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \). -

    @@ -227,8 +398,6 @@ Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up.

  • 27
  • 28
  • 29
    • -
  • 30
    • -
  • 31
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs025.html b/doc/pub/week46/html/._week46-bs025.html index ec82f978d..f37c64b2f 100644 --- a/doc/pub/week46/html/._week46-bs025.html +++ b/doc/pub/week46/html/._week46-bs025.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,38 +167,27 @@ MathJax.Hub.Config({

     

     

     

    -

    Different kernels and Mercer's theorem

    - -

    There are several popular kernels being used. These are

    -
      -
    1. Linear: \( K(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{x}^T\boldsymbol{y} \),
      2. -
    2. Polynomial: \( K(\boldsymbol{x},\boldsymbol{y})=(\boldsymbol{x}^T\boldsymbol{y}+\gamma)^d \),
      3. -
    3. Gaussian Radial Basis Function: \( K(\boldsymbol{x},\boldsymbol{y})=\exp{\left(-\gamma\vert\vert\boldsymbol{x}-\boldsymbol{y}\vert\vert^2\right)} \),
      4. -
    4. Tanh: \( K(\boldsymbol{x},\boldsymbol{y})=\tanh{(\boldsymbol{x}^T\boldsymbol{y}+\gamma)} \),
      5. -
    -

    and many other ones.

    - -

    An important theorem for us is Mercer's -theorem. The -theorem states that if a kernel function \( K \) is symmetric, continuous -and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there -exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into -another space (possibly with much higher dimensions) such that -

    +

    Mathematical optimization of convex functions

    +

    A mathematical (quadratic) optimization problem, or just optimization problem, has the form

    $$ -K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). +\begin{align*} + &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber + &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f. +\end{align*} $$ -

    So you can use \( K \) as a kernel since you know \( \phi \) exists, even if -you don’t know what \( \phi \) is. +

    subject to some constraints for say a selected set \( i=1,2,\dots, n \). +In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the +vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with.

    -

    Note that some frequently used kernels (such as the Sigmoid kernel) -don’t respect all of Mercer’s conditions, yet they generally work well -in practice. +

    In our case we are particularly interested in a class of optimization problems called convex optmization problems. +In our discussion on gradient descent methods we discussed at length the definition of a convex function.

    +

    Convex optimization problems play a central role in applied mathematics and we recommend strongly Boyd and Vandenberghe's text on the topics.

    +

    diff --git a/doc/pub/week46/html/._week46-bs026.html b/doc/pub/week46/html/._week46-bs026.html index 6a8d47061..8dce25949 100644 --- a/doc/pub/week46/html/._week46-bs026.html +++ b/doc/pub/week46/html/._week46-bs026.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,7 +167,19 @@ MathJax.Hub.Config({

     

     

     

    -

    The moons example

    +

    How do we solve these problems?

    + +

    If we use Python as programming language and wish to venture beyond +scikit-learn, tensorflow and similar software which makes our +lives so much easier, we need to dive into the wonderful world of +quadratic programming. We can, if we wish, solve the minimization +problem using say standard gradient methods or conjugate gradient +methods. However, these methods tend to exhibit a rather slow +converge. So, welcome to the promised land of quadratic programming. +

    + +

    The functions we need are contained in the quadratic programming package CVXOPT and we need to import it together with numpy as

    +
    @@ -182,194 +187,8 @@ MathJax.Hub.Config({
    - 
    from __future__ import division, print_function, unicode_literals
-
-import numpy as np
-np.random.seed(42)
-
-import matplotlib
-import matplotlib.pyplot as plt
-plt.rcParams['axes.labelsize'] = 14
-plt.rcParams['xtick.labelsize'] = 12
-plt.rcParams['ytick.labelsize'] = 12
-
-
-from sklearn.svm import SVC
-from sklearn import datasets
-
-
-
-from sklearn.pipeline import Pipeline
-from sklearn.preprocessing import StandardScaler
-from sklearn.svm import LinearSVC
-
-
-from sklearn.datasets import make_moons
-X, y = make_moons(n_samples=100, noise=0.15, random_state=42)
-
-def plot_dataset(X, y, axes):
-    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "bs")
-    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "g^")
-    plt.axis(axes)
-    plt.grid(True, which='both')
-    plt.xlabel(r"$x_1$", fontsize=20)
-    plt.ylabel(r"$x_2$", fontsize=20, rotation=0)
-
-plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
-plt.show()
-
-from sklearn.datasets import make_moons
-from sklearn.pipeline import Pipeline
-from sklearn.preprocessing import PolynomialFeatures
-
-polynomial_svm_clf = Pipeline([
-        ("poly_features", PolynomialFeatures(degree=3)),
-        ("scaler", StandardScaler()),
-        ("svm_clf", LinearSVC(C=10, loss="hinge", random_state=42))
-    ])
-
-polynomial_svm_clf.fit(X, y)
-
-def plot_predictions(clf, axes):
-    x0s = np.linspace(axes[0], axes[1], 100)
-    x1s = np.linspace(axes[2], axes[3], 100)
-    x0, x1 = np.meshgrid(x0s, x1s)
-    X = np.c_[x0.ravel(), x1.ravel()]
-    y_pred = clf.predict(X).reshape(x0.shape)
-    y_decision = clf.decision_function(X).reshape(x0.shape)
-    plt.contourf(x0, x1, y_pred, cmap=plt.cm.brg, alpha=0.2)
-    plt.contourf(x0, x1, y_decision, cmap=plt.cm.brg, alpha=0.1)
-
-plot_predictions(polynomial_svm_clf, [-1.5, 2.5, -1, 1.5])
-plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
-
-plt.show()
-
-
-from sklearn.svm import SVC
-
-poly_kernel_svm_clf = Pipeline([
-        ("scaler", StandardScaler()),
-        ("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5))
-    ])
-poly_kernel_svm_clf.fit(X, y)
-
-poly100_kernel_svm_clf = Pipeline([
-        ("scaler", StandardScaler()),
-        ("svm_clf", SVC(kernel="poly", degree=10, coef0=100, C=5))
-    ])
-poly100_kernel_svm_clf.fit(X, y)
-
-plt.figure(figsize=(11, 4))
-
-plt.subplot(121)
-plot_predictions(poly_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])
-plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
-plt.title(r"$d=3, r=1, C=5$", fontsize=18)
-
-plt.subplot(122)
-plot_predictions(poly100_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])
-plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
-plt.title(r"$d=10, r=100, C=5$", fontsize=18)
-
-plt.show()
-
-def gaussian_rbf(x, landmark, gamma):
-    return np.exp(-gamma * np.linalg.norm(x - landmark, axis=1)**2)
-
-gamma = 0.3
-
-x1s = np.linspace(-4.5, 4.5, 200).reshape(-1, 1)
-x2s = gaussian_rbf(x1s, -2, gamma)
-x3s = gaussian_rbf(x1s, 1, gamma)
-
-XK = np.c_[gaussian_rbf(X1D, -2, gamma), gaussian_rbf(X1D, 1, gamma)]
-yk = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])
-
-plt.figure(figsize=(11, 4))
-
-plt.subplot(121)
-plt.grid(True, which='both')
-plt.axhline(y=0, color='k')
-plt.scatter(x=[-2, 1], y=[0, 0], s=150, alpha=0.5, c="red")
-plt.plot(X1D[:, 0][yk==0], np.zeros(4), "bs")
-plt.plot(X1D[:, 0][yk==1], np.zeros(5), "g^")
-plt.plot(x1s, x2s, "g--")
-plt.plot(x1s, x3s, "b:")
-plt.gca().get_yaxis().set_ticks([0, 0.25, 0.5, 0.75, 1])
-plt.xlabel(r"$x_1$", fontsize=20)
-plt.ylabel(r"Similarity", fontsize=14)
-plt.annotate(r'$\mathbf{x}$',
-             xy=(X1D[3, 0], 0),
-             xytext=(-0.5, 0.20),
-             ha="center",
-             arrowprops=dict(facecolor='black', shrink=0.1),
-             fontsize=18,
-            )
-plt.text(-2, 0.9, "$x_2$", ha="center", fontsize=20)
-plt.text(1, 0.9, "$x_3$", ha="center", fontsize=20)
-plt.axis([-4.5, 4.5, -0.1, 1.1])
-
-plt.subplot(122)
-plt.grid(True, which='both')
-plt.axhline(y=0, color='k')
-plt.axvline(x=0, color='k')
-plt.plot(XK[:, 0][yk==0], XK[:, 1][yk==0], "bs")
-plt.plot(XK[:, 0][yk==1], XK[:, 1][yk==1], "g^")
-plt.xlabel(r"$x_2$", fontsize=20)
-plt.ylabel(r"$x_3$  ", fontsize=20, rotation=0)
-plt.annotate(r'$\phi\left(\mathbf{x}\right)$',
-             xy=(XK[3, 0], XK[3, 1]),
-             xytext=(0.65, 0.50),
-             ha="center",
-             arrowprops=dict(facecolor='black', shrink=0.1),
-             fontsize=18,
-            )
-plt.plot([-0.1, 1.1], [0.57, -0.1], "r--", linewidth=3)
-plt.axis([-0.1, 1.1, -0.1, 1.1])
-    
-plt.subplots_adjust(right=1)
-
-plt.show()
-
-
-x1_example = X1D[3, 0]
-for landmark in (-2, 1):
-    k = gaussian_rbf(np.array([[x1_example]]), np.array([[landmark]]), gamma)
-    print("Phi({}, {}) = {}".format(x1_example, landmark, k))
-
-rbf_kernel_svm_clf = Pipeline([
-        ("scaler", StandardScaler()),
-        ("svm_clf", SVC(kernel="rbf", gamma=5, C=0.001))
-    ])
-rbf_kernel_svm_clf.fit(X, y)
-
-
-from sklearn.svm import SVC
-
-gamma1, gamma2 = 0.1, 5
-C1, C2 = 0.001, 1000
-hyperparams = (gamma1, C1), (gamma1, C2), (gamma2, C1), (gamma2, C2)
-
-svm_clfs = []
-for gamma, C in hyperparams:
-    rbf_kernel_svm_clf = Pipeline([
-            ("scaler", StandardScaler()),
-            ("svm_clf", SVC(kernel="rbf", gamma=gamma, C=C))
-        ])
-    rbf_kernel_svm_clf.fit(X, y)
-    svm_clfs.append(rbf_kernel_svm_clf)
-
-plt.figure(figsize=(11, 7))
-
-for i, svm_clf in enumerate(svm_clfs):
-    plt.subplot(221 + i)
-    plot_predictions(svm_clf, [-1.5, 2.5, -1, 1.5])
-    plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
-    gamma, C = hyperparams[i]
-    plt.title(r"$\gamma = {}, C = {}$".format(gamma, C), fontsize=16)
-
-plt.show()
+  
    import numpy
+import cvxopt
    @@ -385,6 +204,7 @@ plt.show()
    +

    This will make our life much easier. You don't need t write your own optimizer.

    @@ -403,8 +223,6 @@ plt.show()

  • 27
  • 28
  • 29
    • -
  • 30
    • -
  • 31
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs027.html b/doc/pub/week46/html/._week46-bs027.html index 11d270a20..177dfc6a4 100644 --- a/doc/pub/week46/html/._week46-bs027.html +++ b/doc/pub/week46/html/._week46-bs027.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,26 +167,88 @@ MathJax.Hub.Config({

     

     

     

    -

    Mathematical optimization of convex functions

    +

    A simple example

    -

    A mathematical (quadratic) optimization problem, or just optimization problem, has the form

    +

    We remind ourselves about the general problem we want to solve

    $$ \begin{align*} - &\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber - &\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f. + &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber + &\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f. \end{align*} $$ -

    subject to some constraints for say a selected set \( i=1,2,\dots, n \). -In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the -vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with. +

    Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem

    +$$ +\begin{align*} + &\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber + &\mathrm{subject to} \\ \nonumber + &x, y \geq 0 \\ \nonumber + &x+3y \geq 15 \\ \nonumber + &2x+5y \leq 100 \\ \nonumber + &3x+4y \leq 80. \\ \nonumber +\end{align*} +$$ + +

    The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)

    +$$ +\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}. +$$ + +

    Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation

    +$$ +\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +$$ + +

    We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix

    +$$ +\boldsymbol{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} +$$ + +

    is clearly positive semi-definite (all eigenvalues larger or equal zero). +Finally, the vector \( \boldsymbol{h} \) is defined as +

    +$$ +\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}. +$$ + +

    Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero +The following code solves the equations for us

    -

    In our case we are particularly interested in a class of optimization problems called convex optmization problems. -In our discussion on gradient descent methods we discussed at length the definition of a convex function. -

    + +
    +
    +
    +
    +
    + 
    # Import the necessary packages
+import numpy
+from cvxopt import matrix
+from cvxopt import solvers
+P = matrix(numpy.diag([1,0]), tc=’d’)
+q = matrix(numpy.array([3,4]), tc=’d’)
+G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’)
+h = matrix(numpy.array([0,0,-15,100,80]), tc=’d’)
+# Construct the QP, invoke solver
+sol = solvers.qp(P,q,G,h)
+# Extract optimal value and solution
+sol[’x’] 
+sol[’primal objective’]
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    -

    Convex optimization problems play a central role in applied mathematics and we recommend strongly Boyd and Vandenberghe's text on the topics.

    @@ -211,8 +266,6 @@ In our discussion on gradient descent methods we discussed at length the definit

  • 27
  • 28
  • 29
    • -
  • 30
    • -
  • 31
  • »
    • diff --git a/doc/pub/week46/html/._week46-bs028.html b/doc/pub/week46/html/._week46-bs028.html index 4bab3a0cd..6913d0660 100644 --- a/doc/pub/week46/html/._week46-bs028.html +++ b/doc/pub/week46/html/._week46-bs028.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -174,44 +167,24 @@ MathJax.Hub.Config({

     

     

     

    -

    How do we solve these problems?

    +

    Back to the more realistic cases

    -

    If we use Python as programming language and wish to venture beyond -scikit-learn, tensorflow and similar software which makes our -lives so much easier, we need to dive into the wonderful world of -quadratic programming. We can, if we wish, solve the minimization -problem using say standard gradient methods or conjugate gradient -methods. However, these methods tend to exhibit a rather slow -converge. So, welcome to the promised land of quadratic programming. +

    We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the slack parameter \( C \) we have

    +$$ +\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ +y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ +\dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots \\ +y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ +\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda}, +$$ + +

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).

    -

    The functions we need are contained in the quadratic programming package CVXOPT and we need to import it together with numpy as

    - - - -
    -
    -
    -
    -
    - 
    import numpy
-import cvxopt
-
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    -
    - -

    This will make our life much easier. You don't need t write your own optimizer.

    +code will be added

    @@ -228,9 +201,6 @@ converge. So, welcome to the promised land of quadratic programming.

  • 27
  • 28
  • 29
    • -
  • 30
    • -
  • 31
    • -
  • »
    • diff --git a/doc/pub/week46/html/week46-bs.html b/doc/pub/week46/html/week46-bs.html index 91ebb8d77..64fb461b5 100644 --- a/doc/pub/week46/html/week46-bs.html +++ b/doc/pub/week46/html/week46-bs.html @@ -37,11 +37,6 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
  • Overview of week 46
    • -
  • Friday
    • -
  • Workshop plan Friday November 19 and the rest of the lecture
    • -
  • Support Vector Machines, overarching aims
    • -
  • Hyperplanes and all that
    • -
  • What is a hyperplane?
    • -
  • A \( p \)-dimensional space of features
    • -
  • The two-dimensional case
    • -
  • Getting into the details
    • -
  • First attempt at a minimization approach
    • -
  • Solving the equations
    • -
  • Code Example
    • -
  • Problems with the Simpler Approach
    • -
  • A better approach
    • -
  • A quick Reminder on Lagrangian Multipliers
    • -
  • Adding the Multiplier
    • -
  • Setting up the Problem
    • -
  • The problem to solve
    • -
  • The last steps
    • -
  • A soft classifier
    • -
  • Soft optmization problem
    • -
  • Kernels and non-linearity
    • -
  • The equations
    • -
  • The problem to solve
    • -
  • Different kernels and Mercer's theorem
    • -
  • The moons example
    • -
  • Mathematical optimization of convex functions
    • -
  • How do we solve these problems?
    • -
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines, overarching aims
    • +
  • Hyperplanes and all that
    • +
  • What is a hyperplane?
    • +
  • A \( p \)-dimensional space of features
    • +
  • The two-dimensional case
    • +
  • Getting into the details
    • +
  • First attempt at a minimization approach
    • +
  • Solving the equations
    • +
  • Code Example
    • +
  • Problems with the Simpler Approach
    • +
  • A better approach
    • +
  • A quick Reminder on Lagrangian Multipliers
    • +
  • Adding the Multiplier
    • +
  • Setting up the Problem
    • +
  • The problem to solve
    • +
  • The last steps
    • +
  • A soft classifier
    • +
  • Soft optmization problem
    • +
  • Kernels and non-linearity
    • +
  • The equations
    • +
  • The problem to solve
    • +
  • Different kernels and Mercer's theorem
    • +
  • The moons example
    • +
  • Mathematical optimization of convex functions
    • +
  • How do we solve these problems?
    • +
  • A simple example
    • +
  • Back to the more realistic cases
    • @@ -192,7 +185,7 @@ MathJax.Hub.Config({
    -

    Aug 23, 2022

    +

    Nov 12, 2022


    @@ -217,7 +210,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 31
    • +
  • 29
  • »
    • diff --git a/doc/pub/week46/html/week46-reveal.html b/doc/pub/week46/html/week46-reveal.html index 687abee57..8085e4565 100644 --- a/doc/pub/week46/html/week46-reveal.html +++ b/doc/pub/week46/html/week46-reveal.html @@ -184,7 +184,7 @@ MathJax.Hub.Config({
    -

    Aug 23, 2022

    +

    Nov 12, 2022


    @@ -225,42 +225,6 @@ MathJax.Hub.Config({
    -
    -

    Friday

    - -

    Last year we had a very interesting workshop with many presentations. These were (program 2020)

    - - -

    -

    We wish to organize something similar this coming Friday. The presentation last typically 5-10 minutes (some 3-5 slides) with time for questions afterwards. -Feel free to suggest topics. -

    - -

    The program will be available asap. It depends on input from you!

    -
    - -
    -

    Workshop plan Friday November 19 and the rest of the lecture

    - -
      -

    1. 1215-1225pm: Are Frode Helvig Kvanum, Gard Høivang, and David Andreas Bordvik, Next-day forecasts on spot prices for electricity
      2. -

    2. 1225-1235pm: Lidia Luque, Voxel-wise multi-label brain tumor classification
      3. -

    3. 1235-1245pm: Marcus Berget et al, Locating suspicious brain activity using neural networks
      4. -

    4. 1245-1255pm: William Ho and Tom-Ruben Traavik Kvalvaag, Comparing semi-supervised learning and supervised learning for image classification
      5. -
    -

    -

    We will use the second part of the lecture for further discussions of projects 2 and 3 and a summary on boosting methods from last week. Feel free to bring your laptops.

    -
    -

    Support Vector Machines, overarching aims

    diff --git a/doc/pub/week46/html/week46-solarized.html b/doc/pub/week46/html/week46-solarized.html index ce5c0ba54..3fadaf2cf 100644 --- a/doc/pub/week46/html/week46-solarized.html +++ b/doc/pub/week46/html/week46-solarized.html @@ -64,11 +64,6 @@ div.toc p,a {
    @@ -197,38 +192,6 @@ MathJax.Hub.Config({ -









    -

    Friday

    - -

    Last year we had a very interesting workshop with many presentations. These were (program 2020)

    - - -

    We wish to organize something similar this coming Friday. The presentation last typically 5-10 minutes (some 3-5 slides) with time for questions afterwards. -Feel free to suggest topics. -

    - -

    The program will be available asap. It depends on input from you!

    - -









    -

    Workshop plan Friday November 19 and the rest of the lecture

    - -
      -
    1. 1215-1225pm: Are Frode Helvig Kvanum, Gard Høivang, and David Andreas Bordvik, Next-day forecasts on spot prices for electricity
      2. -
    2. 1225-1235pm: Lidia Luque, Voxel-wise multi-label brain tumor classification
      3. -
    3. 1235-1245pm: Marcus Berget et al, Locating suspicious brain activity using neural networks
      4. -
    4. 1245-1255pm: William Ho and Tom-Ruben Traavik Kvalvaag, Comparing semi-supervised learning and supervised learning for image classification
      5. -
    -

    We will use the second part of the lecture for further discussions of projects 2 and 3 and a summary on boosting methods from last week. Feel free to bring your laptops.

    -









    Support Vector Machines, overarching aims

    diff --git a/doc/pub/week46/html/week46.html b/doc/pub/week46/html/week46.html index 48a22cdc9..260dc6c14 100644 --- a/doc/pub/week46/html/week46.html +++ b/doc/pub/week46/html/week46.html @@ -141,11 +141,6 @@ div.toc p,a {
    @@ -274,38 +269,6 @@ MathJax.Hub.Config({ -









    -

    Friday

    - -

    Last year we had a very interesting workshop with many presentations. These were (program 2020)

    - - -

    We wish to organize something similar this coming Friday. The presentation last typically 5-10 minutes (some 3-5 slides) with time for questions afterwards. -Feel free to suggest topics. -

    - -

    The program will be available asap. It depends on input from you!

    - -









    -

    Workshop plan Friday November 19 and the rest of the lecture

    - -
      -
    1. 1215-1225pm: Are Frode Helvig Kvanum, Gard Høivang, and David Andreas Bordvik, Next-day forecasts on spot prices for electricity
      2. -
    2. 1225-1235pm: Lidia Luque, Voxel-wise multi-label brain tumor classification
      3. -
    3. 1235-1245pm: Marcus Berget et al, Locating suspicious brain activity using neural networks
      4. -
    4. 1245-1255pm: William Ho and Tom-Ruben Traavik Kvalvaag, Comparing semi-supervised learning and supervised learning for image classification
      5. -
    -

    We will use the second part of the lecture for further discussions of projects 2 and 3 and a summary on boosting methods from last week. Feel free to bring your laptops.

    -









    Support Vector Machines, overarching aims

    diff --git a/doc/pub/week46/ipynb/ipynb-week46-src.tar.gz b/doc/pub/week46/ipynb/ipynb-week46-src.tar.gz index 9e3d53f3876fd3ce56a82f7dc2d9bc01083a77ce..bbc076ceb11b1f55b1ee0d738064a40fc938f37f 100644 GIT binary patch literal 191 zcmV;w06_mAiwFQ-W^ZEv1MSbv3c@f92k@Qu6nTQtny#J&J$MjAe1VR2uFkcy?a;lw z`v5&DUWN$$UH*iGkQ_Fv^=^~+yN_l=2;q#ukV~G9NfuK*qBI4{5Rr4F$bnEy5hEb; zo%GT=FKmBGU7n?OQooz)$BN>^p6L~M=ASs0%E5Nu`O;|6#>>o@3T{$Xj3=t;bOyPR tZfJp(*H&2xLU$nYTzO@*xWtV$Co5`~#J_$f2!bH^+5_xV4h{ea007A7Tmt|A literal 190 zcmV;v073sBiwFRFmjq(~1MSbj4uUWY24K!SMW29dGT>|?2M;F3_yQvonaGCQi1PLh zNc5!fLX642%b%uA(;o8mW|v3yZiB@TLQ=+H%q&f&B&)feP?`W`u}c~!Sq#L*iUvTI zJL#o2mTPxPTc1%Hl<(%;SXF)4Grt1Q{1eAo7})LyTPq1tSuB)l;D*?+G!oq)3n&$u su>~5hz0e3;4#3M&SfM1pgq>C=&07g0OgBR#{d8T diff --git a/doc/pub/week46/ipynb/week46.ipynb b/doc/pub/week46/ipynb/week46.ipynb index effc6fa31..19524721a 100644 --- a/doc/pub/week46/ipynb/week46.ipynb +++ b/doc/pub/week46/ipynb/week46.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "240f6227", + "id": "fdaa20ed", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "54eb4357", + "id": "5e770c15", "metadata": { "editable": true }, @@ -22,14 +22,14 @@ "# Week 46: Support Vector Machines and Project 3\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Aug 23, 2022**\n", + "Date: **Nov 12, 2022**\n", "\n", "Copyright 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license" ] }, { "cell_type": "markdown", - "id": "9f25e1c2", + "id": "3fc1fc7b", "metadata": { "editable": true }, @@ -60,60 +60,7 @@ }, { "cell_type": "markdown", - "id": "1342f2c8", - "metadata": { - "editable": true - }, - "source": [ - "## Friday\n", - "\n", - "Last year we had a very interesting workshop with many presentations. These were (program 2020)\n", - "\n", - "* Maria Emine Nylund: **Lego Bricks Classifier**\n", - "\n", - "* Fabio Rodrigues Pereira: **Financial Machine Learning**\n", - "\n", - "* Markus Borud Pettersen: **Machine Learning and Brain Grid Cells**\n", - "\n", - "* Jing Sun and Endrias Getachew Asgedom: **Machine learning-based approaches to denoising microseismic data**\n", - "\n", - "* Felicia Jacobsen: **Analysis of Breast Cancer Data**\n", - "\n", - "* Simon Elias Schrader: **Predicting atomization energies of molecules**\n", - "\n", - "* Varvara Bazilova and Sergio Andres Diaz Mesa: **Glacier Mapping and Machine Learning**\n", - "\n", - "* Gert Werner Kluge, Hanna Alida Fossen Hardersen and Sushma Sharma Adhikari: **Gamma ray signals stemming from dark matter in the galactic center**\n", - "\n", - "We wish to organize something similar this coming Friday. The presentation last typically 5-10 minutes (some 3-5 slides) with time for questions afterwards.\n", - "Feel free to suggest topics.\n", - "\n", - "The program will be available asap. It depends on input from you!" - ] - }, - { - "cell_type": "markdown", - "id": "f63624d1", - "metadata": { - "editable": true - }, - "source": [ - "## Workshop plan Friday November 19 and the rest of the lecture\n", - "\n", - "1. **1215-1225pm**: Are Frode Helvig Kvanum, Gard Høivang, and David Andreas Bordvik, *Next-day forecasts on spot prices for electricity*\n", - "\n", - "2. **1225-1235pm**: Lidia Luque, *Voxel-wise multi-label brain tumor classification*\n", - "\n", - "3. **1235-1245pm**: Marcus Berget et al, *Locating suspicious brain activity using neural networks*\n", - "\n", - "4. **1245-1255pm**: William Ho and Tom-Ruben Traavik Kvalvaag, *Comparing semi-supervised learning and supervised learning for image classification*\n", - "\n", - "We will use the second part of the lecture for further discussions of projects 2 and 3 and a summary on boosting methods from last week. Feel free to bring your laptops." - ] - }, - { - "cell_type": "markdown", - "id": "786b642f", + "id": "9a070f4b", "metadata": { "editable": true }, @@ -147,7 +94,7 @@ }, { "cell_type": "markdown", - "id": "54560851", + "id": "00af38aa", "metadata": { "editable": true }, @@ -169,7 +116,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "0906910f", + "id": "b3081144", "metadata": { "collapsed": false, "editable": true @@ -248,7 +195,7 @@ }, { "cell_type": "markdown", - "id": "55a352a0", + "id": "eee872f9", "metadata": { "editable": true }, @@ -268,7 +215,7 @@ }, { "cell_type": "markdown", - "id": "681f91b7", + "id": "2cc70bd4", "metadata": { "editable": true }, @@ -280,7 +227,7 @@ }, { "cell_type": "markdown", - "id": "2cd7db57", + "id": "8acca7a0", "metadata": { "editable": true }, @@ -293,7 +240,7 @@ }, { "cell_type": "markdown", - "id": "9d5f7d2d", + "id": "6407e226", "metadata": { "editable": true }, @@ -305,7 +252,7 @@ }, { "cell_type": "markdown", - "id": "772eb7df", + "id": "ed47c9e4", "metadata": { "editable": true }, @@ -318,7 +265,7 @@ }, { "cell_type": "markdown", - "id": "77cdcd8f", + "id": "1c5b3515", "metadata": { "editable": true }, @@ -330,7 +277,7 @@ }, { "cell_type": "markdown", - "id": "2f386664", + "id": "32d671fe", "metadata": { "editable": true }, @@ -342,7 +289,7 @@ }, { "cell_type": "markdown", - "id": "99f58335", + "id": "4a5d2eec", "metadata": { "editable": true }, @@ -354,7 +301,7 @@ }, { "cell_type": "markdown", - "id": "500e01a9", + "id": "f95d52d7", "metadata": { "editable": true }, @@ -364,7 +311,7 @@ }, { "cell_type": "markdown", - "id": "1f9e2655", + "id": "0a981fb4", "metadata": { "editable": true }, @@ -376,7 +323,7 @@ }, { "cell_type": "markdown", - "id": "3ee0f172", + "id": "7a3bc218", "metadata": { "editable": true }, @@ -387,7 +334,7 @@ }, { "cell_type": "markdown", - "id": "f3aa668f", + "id": "ef75d803", "metadata": { "editable": true }, @@ -399,7 +346,7 @@ }, { "cell_type": "markdown", - "id": "3213eada", + "id": "fefaa223", "metadata": { "editable": true }, @@ -412,7 +359,7 @@ }, { "cell_type": "markdown", - "id": "ed909168", + "id": "c18c2546", "metadata": { "editable": true }, @@ -424,7 +371,7 @@ }, { "cell_type": "markdown", - "id": "a8223cfd", + "id": "80c09fa0", "metadata": { "editable": true }, @@ -434,7 +381,7 @@ }, { "cell_type": "markdown", - "id": "1c385aad", + "id": "577f153f", "metadata": { "editable": true }, @@ -463,7 +410,7 @@ }, { "cell_type": "markdown", - "id": "c6ec4792", + "id": "bfc71c7a", "metadata": { "editable": true }, @@ -475,7 +422,7 @@ }, { "cell_type": "markdown", - "id": "9df22c65", + "id": "dc05e54b", "metadata": { "editable": true }, @@ -487,7 +434,7 @@ }, { "cell_type": "markdown", - "id": "b92b64a2", + "id": "e350ea4c", "metadata": { "editable": true }, @@ -501,7 +448,7 @@ }, { "cell_type": "markdown", - "id": "7359143e", + "id": "4c1af672", "metadata": { "editable": true }, @@ -513,7 +460,7 @@ }, { "cell_type": "markdown", - "id": "9b84c025", + "id": "2d68a9e1", "metadata": { "editable": true }, @@ -527,7 +474,7 @@ }, { "cell_type": "markdown", - "id": "78ff21b0", + "id": "d3cd0ded", "metadata": { "editable": true }, @@ -539,7 +486,7 @@ }, { "cell_type": "markdown", - "id": "4c50d390", + "id": "1b27ce07", "metadata": { "editable": true }, @@ -549,7 +496,7 @@ }, { "cell_type": "markdown", - "id": "e81532c9", + "id": "37b01022", "metadata": { "editable": true }, @@ -561,7 +508,7 @@ }, { "cell_type": "markdown", - "id": "5332b995", + "id": "564d1cf7", "metadata": { "editable": true }, @@ -571,7 +518,7 @@ }, { "cell_type": "markdown", - "id": "4954552a", + "id": "a86038a0", "metadata": { "editable": true }, @@ -583,7 +530,7 @@ }, { "cell_type": "markdown", - "id": "65cf5b6d", + "id": "7a63d750", "metadata": { "editable": true }, @@ -595,7 +542,7 @@ }, { "cell_type": "markdown", - "id": "c1149e3f", + "id": "be570dfb", "metadata": { "editable": true }, @@ -607,7 +554,7 @@ }, { "cell_type": "markdown", - "id": "1c4c11d9", + "id": "a2922b7f", "metadata": { "editable": true }, @@ -617,7 +564,7 @@ }, { "cell_type": "markdown", - "id": "79c86c29", + "id": "d6f23855", "metadata": { "editable": true }, @@ -629,7 +576,7 @@ }, { "cell_type": "markdown", - "id": "5dab4b73", + "id": "2d4ad3ae", "metadata": { "editable": true }, @@ -639,7 +586,7 @@ }, { "cell_type": "markdown", - "id": "4493acab", + "id": "2cc23968", "metadata": { "editable": true }, @@ -653,7 +600,7 @@ }, { "cell_type": "markdown", - "id": "45fe79aa", + "id": "dcee508e", "metadata": { "editable": true }, @@ -672,7 +619,7 @@ }, { "cell_type": "markdown", - "id": "53e286dc", + "id": "539441f5", "metadata": { "editable": true }, @@ -688,7 +635,7 @@ }, { "cell_type": "markdown", - "id": "0f867670", + "id": "a6fe9031", "metadata": { "editable": true }, @@ -700,7 +647,7 @@ }, { "cell_type": "markdown", - "id": "2787e7c8", + "id": "9f442d27", "metadata": { "editable": true }, @@ -712,7 +659,7 @@ }, { "cell_type": "markdown", - "id": "0b8b6e75", + "id": "dbc5a6b3", "metadata": { "editable": true }, @@ -724,7 +671,7 @@ }, { "cell_type": "markdown", - "id": "e6037c14", + "id": "773002da", "metadata": { "editable": true }, @@ -734,7 +681,7 @@ }, { "cell_type": "markdown", - "id": "57363d8a", + "id": "8916c343", "metadata": { "editable": true }, @@ -746,7 +693,7 @@ }, { "cell_type": "markdown", - "id": "47656e56", + "id": "dbb1089b", "metadata": { "editable": true }, @@ -757,7 +704,7 @@ }, { "cell_type": "markdown", - "id": "3c577d1e", + "id": "77f73368", "metadata": { "editable": true }, @@ -769,7 +716,7 @@ }, { "cell_type": "markdown", - "id": "9c315a46", + "id": "26cb9da7", "metadata": { "editable": true }, @@ -782,7 +729,7 @@ }, { "cell_type": "markdown", - "id": "f46843ee", + "id": "d41c488a", "metadata": { "editable": true }, @@ -795,7 +742,7 @@ }, { "cell_type": "markdown", - "id": "a75e5d39", + "id": "beddc517", "metadata": { "editable": true }, @@ -807,7 +754,7 @@ }, { "cell_type": "markdown", - "id": "4af032f3", + "id": "b5846b0b", "metadata": { "editable": true }, @@ -817,7 +764,7 @@ }, { "cell_type": "markdown", - "id": "3b8d27b1", + "id": "3703263d", "metadata": { "editable": true }, @@ -829,7 +776,7 @@ }, { "cell_type": "markdown", - "id": "296d745b", + "id": "4d6f538d", "metadata": { "editable": true }, @@ -839,7 +786,7 @@ }, { "cell_type": "markdown", - "id": "581c7a19", + "id": "4528ddb8", "metadata": { "editable": true }, @@ -851,7 +798,7 @@ }, { "cell_type": "markdown", - "id": "629772bc", + "id": "94aa4b98", "metadata": { "editable": true }, @@ -868,7 +815,7 @@ }, { "cell_type": "markdown", - "id": "5dc9fe11", + "id": "707ba0e7", "metadata": { "editable": true }, @@ -880,7 +827,7 @@ }, { "cell_type": "markdown", - "id": "687f51a2", + "id": "51539d68", "metadata": { "editable": true }, @@ -890,7 +837,7 @@ }, { "cell_type": "markdown", - "id": "455afe66", + "id": "2b4c5395", "metadata": { "editable": true }, @@ -902,7 +849,7 @@ }, { "cell_type": "markdown", - "id": "f0508267", + "id": "93ef8c1a", "metadata": { "editable": true }, @@ -912,7 +859,7 @@ }, { "cell_type": "markdown", - "id": "b6423aea", + "id": "c3c720ba", "metadata": { "editable": true }, @@ -924,7 +871,7 @@ }, { "cell_type": "markdown", - "id": "c2e634d7", + "id": "9fc6c658", "metadata": { "editable": true }, @@ -937,7 +884,7 @@ }, { "cell_type": "markdown", - "id": "f47cec44", + "id": "dbb40739", "metadata": { "editable": true }, @@ -949,7 +896,7 @@ }, { "cell_type": "markdown", - "id": "8a244b8e", + "id": "299d9c43", "metadata": { "editable": true }, @@ -961,7 +908,7 @@ }, { "cell_type": "markdown", - "id": "94172f2a", + "id": "fbdf5d22", "metadata": { "editable": true }, @@ -971,7 +918,7 @@ }, { "cell_type": "markdown", - "id": "f82a3645", + "id": "3ecd29fa", "metadata": { "editable": true }, @@ -985,7 +932,7 @@ }, { "cell_type": "markdown", - "id": "22ce4bcd", + "id": "c912e415", "metadata": { "editable": true }, @@ -995,7 +942,7 @@ }, { "cell_type": "markdown", - "id": "51b9b901", + "id": "44cc2f6d", "metadata": { "editable": true }, @@ -1007,7 +954,7 @@ }, { "cell_type": "markdown", - "id": "3be34ce9", + "id": "e3563988", "metadata": { "editable": true }, @@ -1017,7 +964,7 @@ }, { "cell_type": "markdown", - "id": "c5cfa871", + "id": "91f05ec9", "metadata": { "editable": true }, @@ -1029,7 +976,7 @@ }, { "cell_type": "markdown", - "id": "2f0ad2c1", + "id": "2d866501", "metadata": { "editable": true }, @@ -1039,7 +986,7 @@ }, { "cell_type": "markdown", - "id": "1d58dd03", + "id": "557f0f5a", "metadata": { "editable": true }, @@ -1051,7 +998,7 @@ }, { "cell_type": "markdown", - "id": "ddfcd3a6", + "id": "e75cd54d", "metadata": { "editable": true }, @@ -1065,7 +1012,7 @@ }, { "cell_type": "markdown", - "id": "2903c58f", + "id": "0540b257", "metadata": { "editable": true }, @@ -1077,7 +1024,7 @@ }, { "cell_type": "markdown", - "id": "2dd976bb", + "id": "039fbfd4", "metadata": { "editable": true }, @@ -1088,7 +1035,7 @@ }, { "cell_type": "markdown", - "id": "2a18fd34", + "id": "c674c921", "metadata": { "editable": true }, @@ -1100,7 +1047,7 @@ }, { "cell_type": "markdown", - "id": "79d81b17", + "id": "27fc7fdf", "metadata": { "editable": true }, @@ -1112,7 +1059,7 @@ }, { "cell_type": "markdown", - "id": "49d9d101", + "id": "a29297a5", "metadata": { "editable": true }, @@ -1124,7 +1071,7 @@ }, { "cell_type": "markdown", - "id": "b924d126", + "id": "6067646e", "metadata": { "editable": true }, @@ -1134,7 +1081,7 @@ }, { "cell_type": "markdown", - "id": "c9b10e83", + "id": "8f11d134", "metadata": { "editable": true }, @@ -1146,7 +1093,7 @@ }, { "cell_type": "markdown", - "id": "ff1f98ae", + "id": "f80a1e21", "metadata": { "editable": true }, @@ -1156,7 +1103,7 @@ }, { "cell_type": "markdown", - "id": "d4bfc8f1", + "id": "3b8c8bbb", "metadata": { "editable": true }, @@ -1168,7 +1115,7 @@ }, { "cell_type": "markdown", - "id": "184ae6bc", + "id": "399662d1", "metadata": { "editable": true }, @@ -1179,7 +1126,7 @@ }, { "cell_type": "markdown", - "id": "ebbb8445", + "id": "be96c8df", "metadata": { "editable": true }, @@ -1191,7 +1138,7 @@ }, { "cell_type": "markdown", - "id": "be65faf0", + "id": "94d38036", "metadata": { "editable": true }, @@ -1205,7 +1152,7 @@ }, { "cell_type": "markdown", - "id": "1763e2b7", + "id": "204326ba", "metadata": { "editable": true }, @@ -1217,7 +1164,7 @@ }, { "cell_type": "markdown", - "id": "5702336e", + "id": "930989cb", "metadata": { "editable": true }, @@ -1229,7 +1176,7 @@ }, { "cell_type": "markdown", - "id": "bb22294a", + "id": "dee7875c", "metadata": { "editable": true }, @@ -1239,7 +1186,7 @@ }, { "cell_type": "markdown", - "id": "efa70e73", + "id": "70b43c94", "metadata": { "editable": true }, @@ -1256,7 +1203,7 @@ }, { "cell_type": "markdown", - "id": "6b639a2c", + "id": "1b462027", "metadata": { "editable": true }, @@ -1267,7 +1214,7 @@ }, { "cell_type": "markdown", - "id": "0402efea", + "id": "1b31f910", "metadata": { "editable": true }, @@ -1280,7 +1227,7 @@ }, { "cell_type": "markdown", - "id": "60ff5d90", + "id": "c7b97ef1", "metadata": { "editable": true }, @@ -1292,7 +1239,7 @@ }, { "cell_type": "markdown", - "id": "37188a4a", + "id": "2d37439b", "metadata": { "editable": true }, @@ -1302,7 +1249,7 @@ }, { "cell_type": "markdown", - "id": "18cc7035", + "id": "bc7856b0", "metadata": { "editable": true }, @@ -1314,7 +1261,7 @@ }, { "cell_type": "markdown", - "id": "c59aaa46", + "id": "718a26f2", "metadata": { "editable": true }, @@ -1324,7 +1271,7 @@ }, { "cell_type": "markdown", - "id": "fdc24a81", + "id": "5cabcadb", "metadata": { "editable": true }, @@ -1336,7 +1283,7 @@ }, { "cell_type": "markdown", - "id": "788f4619", + "id": "0242a045", "metadata": { "editable": true }, @@ -1346,7 +1293,7 @@ }, { "cell_type": "markdown", - "id": "bbe8acaa", + "id": "0ddc6cf4", "metadata": { "editable": true }, @@ -1358,7 +1305,7 @@ }, { "cell_type": "markdown", - "id": "8920b5c1", + "id": "87aee740", "metadata": { "editable": true }, @@ -1368,7 +1315,7 @@ }, { "cell_type": "markdown", - "id": "6fe75e96", + "id": "cde4a279", "metadata": { "editable": true }, @@ -1380,7 +1327,7 @@ }, { "cell_type": "markdown", - "id": "9ba2b73f", + "id": "42158438", "metadata": { "editable": true }, @@ -1390,7 +1337,7 @@ }, { "cell_type": "markdown", - "id": "ed011496", + "id": "af22d92e", "metadata": { "editable": true }, @@ -1410,7 +1357,7 @@ }, { "cell_type": "markdown", - "id": "fc7485b8", + "id": "57050b81", "metadata": { "editable": true }, @@ -1422,7 +1369,7 @@ }, { "cell_type": "markdown", - "id": "ed40b0bc", + "id": "49274350", "metadata": { "editable": true }, @@ -1432,7 +1379,7 @@ }, { "cell_type": "markdown", - "id": "8176efb2", + "id": "c7897f75", "metadata": { "editable": true }, @@ -1444,7 +1391,7 @@ }, { "cell_type": "markdown", - "id": "56e47f4f", + "id": "7827f433", "metadata": { "editable": true }, @@ -1460,7 +1407,7 @@ }, { "cell_type": "markdown", - "id": "222d0b1a", + "id": "818ba3db", "metadata": { "editable": true }, @@ -1472,7 +1419,7 @@ }, { "cell_type": "markdown", - "id": "e9738c37", + "id": "7b6e3768", "metadata": { "editable": true }, @@ -1484,7 +1431,7 @@ }, { "cell_type": "markdown", - "id": "d6bbe683", + "id": "b201effc", "metadata": { "editable": true }, @@ -1494,7 +1441,7 @@ }, { "cell_type": "markdown", - "id": "4d9faefd", + "id": "1fcee60a", "metadata": { "editable": true }, @@ -1506,7 +1453,7 @@ }, { "cell_type": "markdown", - "id": "5fae8474", + "id": "5a72560e", "metadata": { "editable": true }, @@ -1518,7 +1465,7 @@ }, { "cell_type": "markdown", - "id": "5813fb6c", + "id": "184e84d0", "metadata": { "editable": true }, @@ -1530,7 +1477,7 @@ }, { "cell_type": "markdown", - "id": "8e4530ea", + "id": "894cb0ce", "metadata": { "editable": true }, @@ -1540,7 +1487,7 @@ }, { "cell_type": "markdown", - "id": "012fe8c9", + "id": "2b69737b", "metadata": { "editable": true }, @@ -1552,7 +1499,7 @@ }, { "cell_type": "markdown", - "id": "2dafc729", + "id": "152d0174", "metadata": { "editable": true }, @@ -1562,7 +1509,7 @@ }, { "cell_type": "markdown", - "id": "84b3a9af", + "id": "db35a010", "metadata": { "editable": true }, @@ -1574,7 +1521,7 @@ }, { "cell_type": "markdown", - "id": "a435659c", + "id": "774b38be", "metadata": { "editable": true }, @@ -1584,7 +1531,7 @@ }, { "cell_type": "markdown", - "id": "34865a12", + "id": "393526ac", "metadata": { "editable": true }, @@ -1596,7 +1543,7 @@ }, { "cell_type": "markdown", - "id": "190a14ee", + "id": "461142ba", "metadata": { "editable": true }, @@ -1607,7 +1554,7 @@ }, { "cell_type": "markdown", - "id": "b88156b5", + "id": "46c9f47a", "metadata": { "editable": true }, @@ -1619,7 +1566,7 @@ }, { "cell_type": "markdown", - "id": "cc1c7ba4", + "id": "49a4d011", "metadata": { "editable": true }, @@ -1631,7 +1578,7 @@ }, { "cell_type": "markdown", - "id": "8e1134ef", + "id": "1fec7c9e", "metadata": { "editable": true }, @@ -1641,7 +1588,7 @@ }, { "cell_type": "markdown", - "id": "8f6d760b", + "id": "cd8bc345", "metadata": { "editable": true }, @@ -1653,7 +1600,7 @@ }, { "cell_type": "markdown", - "id": "1931a40a", + "id": "d9cbf55e", "metadata": { "editable": true }, @@ -1679,7 +1626,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "1cb9eb2d", + "id": "0db994f1", "metadata": { "collapsed": false, "editable": true @@ -1736,7 +1683,7 @@ }, { "cell_type": "markdown", - "id": "e47f4bff", + "id": "8a0057dc", "metadata": { "editable": true }, @@ -1748,7 +1695,7 @@ }, { "cell_type": "markdown", - "id": "7f7cddbd", + "id": "abc0c595", "metadata": { "editable": true }, @@ -1760,7 +1707,7 @@ }, { "cell_type": "markdown", - "id": "f2f988b9", + "id": "d71d8791", "metadata": { "editable": true }, @@ -1770,7 +1717,7 @@ }, { "cell_type": "markdown", - "id": "24dd09ce", + "id": "4c3c1113", "metadata": { "editable": true }, @@ -1782,7 +1729,7 @@ }, { "cell_type": "markdown", - "id": "e7deea0a", + "id": "c66cc7d2", "metadata": { "editable": true }, @@ -1792,7 +1739,7 @@ }, { "cell_type": "markdown", - "id": "d1dcd6ee", + "id": "e184524a", "metadata": { "editable": true }, @@ -1804,7 +1751,7 @@ }, { "cell_type": "markdown", - "id": "999171e6", + "id": "142f8d50", "metadata": { "editable": true }, @@ -1815,7 +1762,7 @@ }, { "cell_type": "markdown", - "id": "01f3e6e0", + "id": "31fd828f", "metadata": { "editable": true }, @@ -1827,7 +1774,7 @@ }, { "cell_type": "markdown", - "id": "546d147b", + "id": "3b0d653a", "metadata": { "editable": true }, @@ -1837,7 +1784,7 @@ }, { "cell_type": "markdown", - "id": "8c4c8657", + "id": "9a727bf4", "metadata": { "editable": true }, @@ -1849,7 +1796,7 @@ }, { "cell_type": "markdown", - "id": "6d5e637a", + "id": "2b663b7e", "metadata": { "editable": true }, @@ -1867,7 +1814,7 @@ }, { "cell_type": "markdown", - "id": "6b4b3d67", + "id": "c3e77b31", "metadata": { "editable": true }, @@ -1878,7 +1825,7 @@ }, { "cell_type": "markdown", - "id": "4a48735a", + "id": "2f17dcaf", "metadata": { "editable": true }, @@ -1890,7 +1837,7 @@ }, { "cell_type": "markdown", - "id": "c65483ef", + "id": "be5e27f0", "metadata": { "editable": true }, @@ -1900,7 +1847,7 @@ }, { "cell_type": "markdown", - "id": "6b0de07c", + "id": "112c23b0", "metadata": { "editable": true }, @@ -1917,7 +1864,7 @@ }, { "cell_type": "markdown", - "id": "eb84729e", + "id": "87b1f8b0", "metadata": { "editable": true }, @@ -1931,7 +1878,7 @@ }, { "cell_type": "markdown", - "id": "1b900aac", + "id": "af672dec", "metadata": { "editable": true }, @@ -1946,7 +1893,7 @@ }, { "cell_type": "markdown", - "id": "56fe34be", + "id": "d402eb2a", "metadata": { "editable": true }, @@ -1958,7 +1905,7 @@ }, { "cell_type": "markdown", - "id": "924b16c7", + "id": "38931e4b", "metadata": { "editable": true }, @@ -1986,7 +1933,7 @@ }, { "cell_type": "markdown", - "id": "9f09f168", + "id": "a1773101", "metadata": { "editable": true }, @@ -1998,7 +1945,7 @@ }, { "cell_type": "markdown", - "id": "9df561ad", + "id": "5167cd90", "metadata": { "editable": true }, @@ -2013,7 +1960,7 @@ }, { "cell_type": "markdown", - "id": "020d1ced", + "id": "c9dcb083", "metadata": { "editable": true }, @@ -2024,7 +1971,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "6bcf82ab", + "id": "49af7f92", "metadata": { "collapsed": false, "editable": true @@ -2223,7 +2170,7 @@ }, { "cell_type": "markdown", - "id": "b1068413", + "id": "99b01441", "metadata": { "editable": true }, @@ -2235,7 +2182,7 @@ }, { "cell_type": "markdown", - "id": "58112473", + "id": "7ce54aca", "metadata": { "editable": true }, @@ -2250,7 +2197,7 @@ }, { "cell_type": "markdown", - "id": "829af050", + "id": "cac73c40", "metadata": { "editable": true }, @@ -2267,7 +2214,7 @@ }, { "cell_type": "markdown", - "id": "95f27079", + "id": "aece7452", "metadata": { "editable": true }, @@ -2288,7 +2235,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "9105f1ac", + "id": "cbdb6008", "metadata": { "collapsed": false, "editable": true @@ -2301,7 +2248,7 @@ }, { "cell_type": "markdown", - "id": "aac44b98", + "id": "26bb80f7", "metadata": { "editable": true }, @@ -2311,7 +2258,7 @@ }, { "cell_type": "markdown", - "id": "ec372b08", + "id": "fbd38783", "metadata": { "editable": true }, @@ -2323,7 +2270,7 @@ }, { "cell_type": "markdown", - "id": "76ce0a8a", + "id": "1fa4fdbf", "metadata": { "editable": true }, @@ -2338,7 +2285,7 @@ }, { "cell_type": "markdown", - "id": "27a668c6", + "id": "0d6c84fb", "metadata": { "editable": true }, @@ -2348,7 +2295,7 @@ }, { "cell_type": "markdown", - "id": "80f2825e", + "id": "a069c872", "metadata": { "editable": true }, @@ -2367,7 +2314,7 @@ }, { "cell_type": "markdown", - "id": "39d5e323", + "id": "9a4c0894", "metadata": { "editable": true }, @@ -2377,7 +2324,7 @@ }, { "cell_type": "markdown", - "id": "bc1cc81e", + "id": "8612c078", "metadata": { "editable": true }, @@ -2389,7 +2336,7 @@ }, { "cell_type": "markdown", - "id": "b07c7927", + "id": "85ec4a5c", "metadata": { "editable": true }, @@ -2399,7 +2346,7 @@ }, { "cell_type": "markdown", - "id": "e6d88a08", + "id": "994b8395", "metadata": { "editable": true }, @@ -2411,7 +2358,7 @@ }, { "cell_type": "markdown", - "id": "da3f535b", + "id": "1fbb16a6", "metadata": { "editable": true }, @@ -2421,7 +2368,7 @@ }, { "cell_type": "markdown", - "id": "1a247ef0", + "id": "af89e786", "metadata": { "editable": true }, @@ -2433,7 +2380,7 @@ }, { "cell_type": "markdown", - "id": "d59f0c1b", + "id": "016c22ed", "metadata": { "editable": true }, @@ -2444,7 +2391,7 @@ }, { "cell_type": "markdown", - "id": "e43e4616", + "id": "ecdcfe97", "metadata": { "editable": true }, @@ -2456,7 +2403,7 @@ }, { "cell_type": "markdown", - "id": "3792f1a3", + "id": "16e0c215", "metadata": { "editable": true }, @@ -2468,7 +2415,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "809f0c13", + "id": "1781bfa9", "metadata": { "collapsed": false, "editable": true @@ -2492,7 +2439,7 @@ }, { "cell_type": "markdown", - "id": "ba0b4ca8", + "id": "9013d663", "metadata": { "editable": true }, @@ -2504,7 +2451,7 @@ }, { "cell_type": "markdown", - "id": "75ef674a", + "id": "67964577", "metadata": { "editable": true }, @@ -2521,7 +2468,7 @@ }, { "cell_type": "markdown", - "id": "6f9af4d5", + "id": "13711084", "metadata": { "editable": true }, diff --git a/doc/src/week46/week46.do.txt b/doc/src/week46/week46.do.txt index 24b41aa0b..83c685f9f 100644 --- a/doc/src/week46/week46.do.txt +++ b/doc/src/week46/week46.do.txt @@ -24,37 +24,6 @@ o "Excellent videos on Gradient Boosting":"https://www.youtube.com/watch?v=3CC4N !eblock -!split -===== Friday ===== - -Last year we had a very interesting workshop with many presentations. These were (program 2020) - -* Maria Emine Nylund: _Lego Bricks Classifier_ -* Fabio Rodrigues Pereira: _Financial Machine Learning_ -* Markus Borud Pettersen: _Machine Learning and Brain Grid Cells_ -* Jing Sun and Endrias Getachew Asgedom: _Machine learning-based approaches to denoising microseismic data_ -* Felicia Jacobsen: _Analysis of Breast Cancer Data_ -* Simon Elias Schrader: _Predicting atomization energies of molecules_ -* Varvara Bazilova and Sergio Andres Diaz Mesa: _Glacier Mapping and Machine Learning_ -* Gert Werner Kluge, Hanna Alida Fossen Hardersen and Sushma Sharma Adhikari: _Gamma ray signals stemming from dark matter in the galactic center_ - -We wish to organize something similar this coming Friday. The presentation last typically 5-10 minutes (some 3-5 slides) with time for questions afterwards. -Feel free to suggest topics. - -The program will be available asap. It depends on input from you! - -!split -===== Workshop plan Friday November 19 and the rest of the lecture ===== - -o _1215-1225pm_: Are Frode Helvig Kvanum, Gard Høivang, and David Andreas Bordvik, *Next-day forecasts on spot prices for electricity* -o _1225-1235pm_: Lidia Luque, *Voxel-wise multi-label brain tumor classification* -o _1235-1245pm_: Marcus Berget et al, *Locating suspicious brain activity using neural networks* -o _1245-1255pm_: William Ho and Tom-Ruben Traavik Kvalvaag, *Comparing semi-supervised learning and supervised learning for image classification* - -We will use the second part of the lecture for further discussions of projects 2 and 3 and a summary on boosting methods from last week. Feel free to bring your laptops. - - - !split ===== Support Vector Machines, overarching aims =====