From afab825af671729c7ead1f38fecd48c01d79a7aa Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Thu, 25 Nov 2021 08:28:57 +0100 Subject: [PATCH] smaller update --- doc/pub/week47/html/week47-bs.html | 6 +- doc/pub/week47/html/week47-reveal.html | 33 +- doc/pub/week47/html/week47-solarized.html | 35 +- doc/pub/week47/html/week47.html | 35 +- doc/pub/week47/ipynb/ipynb-week47-src.tar.gz | Bin 823750 -> 823750 bytes doc/pub/week47/ipynb/week47.ipynb | 501 +++++++++---------- doc/src/week47/week47.do.txt | 32 +- 7 files changed, 275 insertions(+), 367 deletions(-) diff --git a/doc/pub/week47/html/week47-bs.html b/doc/pub/week47/html/week47-bs.html index 527cec9d1..211f2600c 100644 --- a/doc/pub/week47/html/week47-bs.html +++ b/doc/pub/week47/html/week47-bs.html @@ -89,10 +89,10 @@ doconce format html week47.do.txt --html_style=bootstrap --pygments_html_style=d None, 'how-do-we-solve-these-problems'), ('A simple example', 2, None, 'a-simple-example'), - ('Back to the more realistic cases', + ('Support Vector Machines and Regression', 2, None, - 'back-to-the-more-realistic-cases'), + 'support-vector-machines-and-regression'), ('Summary of course', 2, None, 'summary-of-course'), ('What? Me worry? No final exam in this course!', 2, @@ -289,7 +289,7 @@ MathJax.Hub.Config({
  • Mathematical optimization of convex functions
  • How do we solve these problems?
  • A simple example
    • -
  • Back to the more realistic cases
    • +
  • Support Vector Machines and Regression
  • Summary of course
  • What? Me worry? No final exam in this course!
  • What is the link between Artificial Intelligence and Machine Learning and some general Remarks
    • diff --git a/doc/pub/week47/html/week47-reveal.html b/doc/pub/week47/html/week47-reveal.html index 7ac2ab4cb..1e3141286 100644 --- a/doc/pub/week47/html/week47-reveal.html +++ b/doc/pub/week47/html/week47-reveal.html @@ -567,7 +567,7 @@ the two classes (if they are well separated from the beginning).

     
    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n. $$

     
    @@ -583,16 +583,16 @@ $$

    or just

     
    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i=1,2,\dots,n. $$

     

    If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of -\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition +\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert_2^2 \) (the norm) subject to the condition

     
    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i=1,2,\dots,n. $$

     
    @@ -791,8 +791,8 @@ y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x $$

     
    -

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda}^T =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}^T=[y_1,y_2,\dots,y_n] \).

    @@ -1545,26 +1545,9 @@ sol[’primal obj
    -

    Back to the more realistic cases

    +

    Support Vector Machines and Regression

    -

    We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the slack parameter \( C \) we have

    -

     
    -$$ -\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ -y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ -\dots & \dots & \dots & \dots & \dots \\ -\dots & \dots & \dots & \dots & \dots \\ -y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ -\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda}, -$$ -

     
    - -

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). -With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \). -

    - -code will be added +

    Material may be added if of interest. See Bishop chapter 7.1 for a discussion.

    diff --git a/doc/pub/week47/html/week47-solarized.html b/doc/pub/week47/html/week47-solarized.html index f13d51e70..408c50854 100644 --- a/doc/pub/week47/html/week47-solarized.html +++ b/doc/pub/week47/html/week47-solarized.html @@ -116,10 +116,10 @@ div.toc p,a { None, 'how-do-we-solve-these-problems'), ('A simple example', 2, None, 'a-simple-example'), - ('Back to the more realistic cases', + ('Support Vector Machines and Regression', 2, None, - 'back-to-the-more-realistic-cases'), + 'support-vector-machines-and-regression'), ('Summary of course', 2, None, 'summary-of-course'), ('What? Me worry? No final exam in this course!', 2, @@ -632,7 +632,7 @@ the two classes (if they are well separated from the beginning).

    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n. $$

    All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.

    @@ -644,14 +644,14 @@ $$

    or just

    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i=1,2,\dots,n. $$

    If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of -\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition +\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert_2^2 \) (the norm) subject to the condition

    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i=1,2,\dots,n. $$

    We have thus defined our margin as the invers of the norm of @@ -807,8 +807,8 @@ y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x \end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, $$ -

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda}^T =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}^T=[y_1,y_2,\dots,y_n] \).











    @@ -1488,24 +1488,9 @@ sol[’primal obj









    -

    Back to the more realistic cases

    +

    Support Vector Machines and Regression

    -

    We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the slack parameter \( C \) we have

    -$$ -\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ -y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ -\dots & \dots & \dots & \dots & \dots \\ -\dots & \dots & \dots & \dots & \dots \\ -y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ -\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda}, -$$ - -

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). -With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \). -

    - -code will be added +

    Material may be added if of interest. See Bishop chapter 7.1 for a discussion.











    Summary of course

    diff --git a/doc/pub/week47/html/week47.html b/doc/pub/week47/html/week47.html index d39d65e2c..408daae09 100644 --- a/doc/pub/week47/html/week47.html +++ b/doc/pub/week47/html/week47.html @@ -193,10 +193,10 @@ div.toc p,a { None, 'how-do-we-solve-these-problems'), ('A simple example', 2, None, 'a-simple-example'), - ('Back to the more realistic cases', + ('Support Vector Machines and Regression', 2, None, - 'back-to-the-more-realistic-cases'), + 'support-vector-machines-and-regression'), ('Summary of course', 2, None, 'summary-of-course'), ('What? Me worry? No final exam in this course!', 2, @@ -709,7 +709,7 @@ the two classes (if they are well separated from the beginning).

    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n. $$

    All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.

    @@ -721,14 +721,14 @@ $$

    or just

    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i=1,2,\dots,n. $$

    If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of -\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition +\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert_2^2 \) (the norm) subject to the condition

    $$ -y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i=1,2,\dots,n. $$

    We have thus defined our margin as the invers of the norm of @@ -884,8 +884,8 @@ y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x \end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda}, $$ -

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). +

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda}^T =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and +\( \boldsymbol{y}^T=[y_1,y_2,\dots,y_n] \).











    @@ -1565,24 +1565,9 @@ sol[’primal objective’]









    -

    Back to the more realistic cases

    +

    Support Vector Machines and Regression

    -

    We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the slack parameter \( C \) we have

    -$$ -\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\ -y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\ -\dots & \dots & \dots & \dots & \dots \\ -\dots & \dots & \dots & \dots & \dots \\ -y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\ -\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda}, -$$ - -

    subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and -\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \). -With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \). -

    - -code will be added +

    Material may be added if of interest. See Bishop chapter 7.1 for a discussion.











    Summary of course

    diff --git a/doc/pub/week47/ipynb/ipynb-week47-src.tar.gz b/doc/pub/week47/ipynb/ipynb-week47-src.tar.gz index 6640589211639f0798c2198855830184af5e898d..ed7421f30231f479ad5bc45d1f814691fae49024 100644 GIT binary patch delta 53 zcmX@M*yz||BR2VN4h9|TMz&Tq##T0_RyO8VHkMX4)>by42zx6VM=Kj=D;rlU8~0W= Ho&ysBeBKQ( delta 53 zcmX@M*yz||BR2VN4u(f6jcl!KjIC@;t!&J#Y%Hy8tgUQ75%yL#j#f6#RyM9yHtwx# IJO?HM0GA65&j0`b diff --git a/doc/pub/week47/ipynb/week47.ipynb b/doc/pub/week47/ipynb/week47.ipynb index 0969fa0b1..dba1bdab4 100644 --- a/doc/pub/week47/ipynb/week47.ipynb +++ b/doc/pub/week47/ipynb/week47.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "4cc34bad", + "id": "d297b5a6", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "8579d326", + "id": "c3e26ae4", "metadata": { "editable": true }, @@ -29,7 +29,7 @@ }, { "cell_type": "markdown", - "id": "a1de5e67", + "id": "3696edde", "metadata": { "editable": true }, @@ -52,7 +52,7 @@ }, { "cell_type": "markdown", - "id": "9a1ee036", + "id": "3bce99d8", "metadata": { "editable": true }, @@ -86,7 +86,7 @@ }, { "cell_type": "markdown", - "id": "706de241", + "id": "775d98d8", "metadata": { "editable": true }, @@ -108,7 +108,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "a244fa48", + "id": "7d9df056", "metadata": { "collapsed": false, "editable": true @@ -187,7 +187,7 @@ }, { "cell_type": "markdown", - "id": "007f9d3f", + "id": "c64f1a9a", "metadata": { "editable": true }, @@ -207,7 +207,7 @@ }, { "cell_type": "markdown", - "id": "59d6aee7", + "id": "674b12a3", "metadata": { "editable": true }, @@ -219,7 +219,7 @@ }, { "cell_type": "markdown", - "id": "73aaa39a", + "id": "43f0a20b", "metadata": { "editable": true }, @@ -232,7 +232,7 @@ }, { "cell_type": "markdown", - "id": "90533217", + "id": "ce09877b", "metadata": { "editable": true }, @@ -244,7 +244,7 @@ }, { "cell_type": "markdown", - "id": "c811e175", + "id": "d9a14fd7", "metadata": { "editable": true }, @@ -254,7 +254,7 @@ }, { "cell_type": "markdown", - "id": "6084370d", + "id": "ce869655", "metadata": { "editable": true }, @@ -267,7 +267,7 @@ }, { "cell_type": "markdown", - "id": "4127686f", + "id": "c703f2d0", "metadata": { "editable": true }, @@ -279,7 +279,7 @@ }, { "cell_type": "markdown", - "id": "ee2e985c", + "id": "9ce7dd30", "metadata": { "editable": true }, @@ -291,7 +291,7 @@ }, { "cell_type": "markdown", - "id": "d6e38fb6", + "id": "b455795b", "metadata": { "editable": true }, @@ -303,7 +303,7 @@ }, { "cell_type": "markdown", - "id": "d2ca014b", + "id": "010b9238", "metadata": { "editable": true }, @@ -313,7 +313,7 @@ }, { "cell_type": "markdown", - "id": "8abf8383", + "id": "b999e39f", "metadata": { "editable": true }, @@ -325,7 +325,7 @@ }, { "cell_type": "markdown", - "id": "48755c0e", + "id": "42e6b71e", "metadata": { "editable": true }, @@ -336,7 +336,7 @@ }, { "cell_type": "markdown", - "id": "96b14dd6", + "id": "f8c7506b", "metadata": { "editable": true }, @@ -348,7 +348,7 @@ }, { "cell_type": "markdown", - "id": "bf75f7e1", + "id": "902b6664", "metadata": { "editable": true }, @@ -361,7 +361,7 @@ }, { "cell_type": "markdown", - "id": "b9a76b13", + "id": "54f2aab2", "metadata": { "editable": true }, @@ -373,7 +373,7 @@ }, { "cell_type": "markdown", - "id": "c1c92394", + "id": "d2b39bc3", "metadata": { "editable": true }, @@ -383,7 +383,7 @@ }, { "cell_type": "markdown", - "id": "ece075ba", + "id": "d8e438d6", "metadata": { "editable": true }, @@ -412,7 +412,7 @@ }, { "cell_type": "markdown", - "id": "a36c4ce3", + "id": "284ea218", "metadata": { "editable": true }, @@ -424,7 +424,7 @@ }, { "cell_type": "markdown", - "id": "9d9fdae9", + "id": "4d4e8fdc", "metadata": { "editable": true }, @@ -436,7 +436,7 @@ }, { "cell_type": "markdown", - "id": "cb45022e", + "id": "5cfd3056", "metadata": { "editable": true }, @@ -451,7 +451,7 @@ }, { "cell_type": "markdown", - "id": "3b678990", + "id": "6f935fa6", "metadata": { "editable": true }, @@ -463,7 +463,7 @@ }, { "cell_type": "markdown", - "id": "55567107", + "id": "206d9255", "metadata": { "editable": true }, @@ -477,7 +477,7 @@ }, { "cell_type": "markdown", - "id": "2688d372", + "id": "5f7c7666", "metadata": { "editable": true }, @@ -489,7 +489,7 @@ }, { "cell_type": "markdown", - "id": "1c53912a", + "id": "2d213860", "metadata": { "editable": true }, @@ -499,7 +499,7 @@ }, { "cell_type": "markdown", - "id": "3f4bf847", + "id": "276c7e93", "metadata": { "editable": true }, @@ -511,7 +511,7 @@ }, { "cell_type": "markdown", - "id": "3b3876db", + "id": "264b761f", "metadata": { "editable": true }, @@ -521,7 +521,7 @@ }, { "cell_type": "markdown", - "id": "1e239842", + "id": "10dd0353", "metadata": { "editable": true }, @@ -533,7 +533,7 @@ }, { "cell_type": "markdown", - "id": "011ed1d2", + "id": "69606ee8", "metadata": { "editable": true }, @@ -545,7 +545,7 @@ }, { "cell_type": "markdown", - "id": "4618893d", + "id": "3c6f4034", "metadata": { "editable": true }, @@ -557,7 +557,7 @@ }, { "cell_type": "markdown", - "id": "9d515930", + "id": "45ebb87b", "metadata": { "editable": true }, @@ -567,7 +567,7 @@ }, { "cell_type": "markdown", - "id": "d963d9ae", + "id": "dcddfbd3", "metadata": { "editable": true }, @@ -579,7 +579,7 @@ }, { "cell_type": "markdown", - "id": "d92faaa6", + "id": "52da5b57", "metadata": { "editable": true }, @@ -589,7 +589,7 @@ }, { "cell_type": "markdown", - "id": "8d521484", + "id": "4281f447", "metadata": { "editable": true }, @@ -612,7 +612,7 @@ }, { "cell_type": "markdown", - "id": "5d5f78cb", + "id": "dd89f6d6", "metadata": { "editable": true }, @@ -628,19 +628,19 @@ }, { "cell_type": "markdown", - "id": "2b51b0c3", + "id": "894e43d0", "metadata": { "editable": true }, "source": [ "$$\n", - "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, p.\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, n.\n", "$$" ] }, { "cell_type": "markdown", - "id": "dbdab28c", + "id": "ab462cfa", "metadata": { "editable": true }, @@ -652,7 +652,7 @@ }, { "cell_type": "markdown", - "id": "af1c3ae8", + "id": "f66b13c2", "metadata": { "editable": true }, @@ -664,7 +664,7 @@ }, { "cell_type": "markdown", - "id": "751ee766", + "id": "b09478e1", "metadata": { "editable": true }, @@ -674,42 +674,42 @@ }, { "cell_type": "markdown", - "id": "15798c2e", + "id": "e63e1ed8", "metadata": { "editable": true }, "source": [ "$$\n", - "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M\\vert \\vert \\boldsymbol{w}\\vert\\vert \\hspace{0.1cm}\\forall i.\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M\\vert \\vert \\boldsymbol{w}\\vert\\vert \\hspace{0.1cm}\\forall i=1,2,\\dots,n.\n", "$$" ] }, { "cell_type": "markdown", - "id": "ab9bbfa4", + "id": "09d0ab60", "metadata": { "editable": true }, "source": [ "If we scale the equation so that $\\vert \\vert \\boldsymbol{w}\\vert\\vert = 1/M$, we have to find the minimum of \n", - "$\\boldsymbol{w}^T\\boldsymbol{w}=\\vert \\vert \\boldsymbol{w}\\vert\\vert$ (the norm) subject to the condition" + "$\\boldsymbol{w}^T\\boldsymbol{w}=\\vert \\vert \\boldsymbol{w}\\vert\\vert_2^2$ (the norm) subject to the condition" ] }, { "cell_type": "markdown", - "id": "cc5d8b13", + "id": "815f9576", "metadata": { "editable": true }, "source": [ "$$\n", - "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq 1 \\hspace{0.1cm}\\forall i.\n", + "y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq 1 \\hspace{0.1cm}\\forall i=1,2,\\dots,n.\n", "$$" ] }, { "cell_type": "markdown", - "id": "7f69fb14", + "id": "81dff301", "metadata": { "editable": true }, @@ -722,7 +722,7 @@ }, { "cell_type": "markdown", - "id": "dd1321d9", + "id": "6a5bcc89", "metadata": { "editable": true }, @@ -735,7 +735,7 @@ }, { "cell_type": "markdown", - "id": "689ed44a", + "id": "2545b335", "metadata": { "editable": true }, @@ -747,7 +747,7 @@ }, { "cell_type": "markdown", - "id": "35df1f77", + "id": "7d5bf787", "metadata": { "editable": true }, @@ -757,7 +757,7 @@ }, { "cell_type": "markdown", - "id": "4a7bbc5f", + "id": "761895bf", "metadata": { "editable": true }, @@ -769,7 +769,7 @@ }, { "cell_type": "markdown", - "id": "85d7a8b6", + "id": "15ff9911", "metadata": { "editable": true }, @@ -779,7 +779,7 @@ }, { "cell_type": "markdown", - "id": "4fe49eac", + "id": "e5666256", "metadata": { "editable": true }, @@ -791,7 +791,7 @@ }, { "cell_type": "markdown", - "id": "67c642cb", + "id": "9a3d325a", "metadata": { "editable": true }, @@ -808,7 +808,7 @@ }, { "cell_type": "markdown", - "id": "48d25961", + "id": "2cda88e0", "metadata": { "editable": true }, @@ -820,7 +820,7 @@ }, { "cell_type": "markdown", - "id": "7bd7b578", + "id": "70285535", "metadata": { "editable": true }, @@ -830,7 +830,7 @@ }, { "cell_type": "markdown", - "id": "afc29bad", + "id": "3d155c0c", "metadata": { "editable": true }, @@ -842,7 +842,7 @@ }, { "cell_type": "markdown", - "id": "0543d207", + "id": "65cfa308", "metadata": { "editable": true }, @@ -852,7 +852,7 @@ }, { "cell_type": "markdown", - "id": "8ac4e5fa", + "id": "204f80ec", "metadata": { "editable": true }, @@ -864,7 +864,7 @@ }, { "cell_type": "markdown", - "id": "7dac42c7", + "id": "89675f53", "metadata": { "editable": true }, @@ -877,7 +877,7 @@ }, { "cell_type": "markdown", - "id": "abb322a1", + "id": "6ebe98f7", "metadata": { "editable": true }, @@ -889,7 +889,7 @@ }, { "cell_type": "markdown", - "id": "7f0f11ba", + "id": "cb58d54b", "metadata": { "editable": true }, @@ -901,7 +901,7 @@ }, { "cell_type": "markdown", - "id": "0ac75fc9", + "id": "ac15bc71", "metadata": { "editable": true }, @@ -911,7 +911,7 @@ }, { "cell_type": "markdown", - "id": "92428367", + "id": "c0f9ca25", "metadata": { "editable": true }, @@ -925,7 +925,7 @@ }, { "cell_type": "markdown", - "id": "78ae892d", + "id": "5734050e", "metadata": { "editable": true }, @@ -935,7 +935,7 @@ }, { "cell_type": "markdown", - "id": "bd74c1ff", + "id": "3deaaab1", "metadata": { "editable": true }, @@ -947,7 +947,7 @@ }, { "cell_type": "markdown", - "id": "ad60aeb2", + "id": "c1905f47", "metadata": { "editable": true }, @@ -957,7 +957,7 @@ }, { "cell_type": "markdown", - "id": "38368495", + "id": "4e0423a8", "metadata": { "editable": true }, @@ -969,7 +969,7 @@ }, { "cell_type": "markdown", - "id": "c6c520b7", + "id": "e0dec665", "metadata": { "editable": true }, @@ -979,7 +979,7 @@ }, { "cell_type": "markdown", - "id": "9749dfca", + "id": "e2d57eef", "metadata": { "editable": true }, @@ -991,7 +991,7 @@ }, { "cell_type": "markdown", - "id": "7985f058", + "id": "77bc96c6", "metadata": { "editable": true }, @@ -1005,7 +1005,7 @@ }, { "cell_type": "markdown", - "id": "55fa560e", + "id": "38775bb9", "metadata": { "editable": true }, @@ -1017,7 +1017,7 @@ }, { "cell_type": "markdown", - "id": "35b9dc6e", + "id": "2862aca0", "metadata": { "editable": true }, @@ -1029,7 +1029,7 @@ }, { "cell_type": "markdown", - "id": "96ca3b5d", + "id": "b356a9e2", "metadata": { "editable": true }, @@ -1041,7 +1041,7 @@ }, { "cell_type": "markdown", - "id": "dcf1e5fe", + "id": "673fbcec", "metadata": { "editable": true }, @@ -1053,7 +1053,7 @@ }, { "cell_type": "markdown", - "id": "197180b0", + "id": "fce13dac", "metadata": { "editable": true }, @@ -1065,7 +1065,7 @@ }, { "cell_type": "markdown", - "id": "e5601abd", + "id": "5cef7ece", "metadata": { "editable": true }, @@ -1075,7 +1075,7 @@ }, { "cell_type": "markdown", - "id": "0ded669a", + "id": "116189a3", "metadata": { "editable": true }, @@ -1087,7 +1087,7 @@ }, { "cell_type": "markdown", - "id": "df3ef2de", + "id": "192e3d83", "metadata": { "editable": true }, @@ -1097,7 +1097,7 @@ }, { "cell_type": "markdown", - "id": "0a18588a", + "id": "0367b41a", "metadata": { "editable": true }, @@ -1109,7 +1109,7 @@ }, { "cell_type": "markdown", - "id": "fd303bd1", + "id": "73055eee", "metadata": { "editable": true }, @@ -1120,7 +1120,7 @@ }, { "cell_type": "markdown", - "id": "a55ffcfc", + "id": "1e9a8e7d", "metadata": { "editable": true }, @@ -1132,7 +1132,7 @@ }, { "cell_type": "markdown", - "id": "0214d5d7", + "id": "fec4bd3f", "metadata": { "editable": true }, @@ -1146,7 +1146,7 @@ }, { "cell_type": "markdown", - "id": "b74e82e3", + "id": "83d140dc", "metadata": { "editable": true }, @@ -1158,7 +1158,7 @@ }, { "cell_type": "markdown", - "id": "aabee1ef", + "id": "1e8bd1a9", "metadata": { "editable": true }, @@ -1170,7 +1170,7 @@ }, { "cell_type": "markdown", - "id": "c34316ad", + "id": "f9020497", "metadata": { "editable": true }, @@ -1180,7 +1180,7 @@ }, { "cell_type": "markdown", - "id": "093dce59", + "id": "a3777e9e", "metadata": { "editable": true }, @@ -1197,18 +1197,18 @@ }, { "cell_type": "markdown", - "id": "aa1a49f2", + "id": "f4b5422e", "metadata": { "editable": true }, "source": [ - "subject to $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$. Here we defined the vectors $\\boldsymbol{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n", - "$\\boldsymbol{y}=[y_1,y_2,\\dots,y_n]$." + "subject to $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$. Here we defined the vectors $\\boldsymbol{\\lambda}^T =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n", + "$\\boldsymbol{y}^T=[y_1,y_2,\\dots,y_n]$." ] }, { "cell_type": "markdown", - "id": "74f7ad97", + "id": "5434dedf", "metadata": { "editable": true }, @@ -1221,7 +1221,7 @@ }, { "cell_type": "markdown", - "id": "29ab6c37", + "id": "59d9227f", "metadata": { "editable": true }, @@ -1233,7 +1233,7 @@ }, { "cell_type": "markdown", - "id": "422b1e2c", + "id": "e3205aa2", "metadata": { "editable": true }, @@ -1243,7 +1243,7 @@ }, { "cell_type": "markdown", - "id": "bb88d9b6", + "id": "b9773b95", "metadata": { "editable": true }, @@ -1255,7 +1255,7 @@ }, { "cell_type": "markdown", - "id": "b8db4fcb", + "id": "472ad856", "metadata": { "editable": true }, @@ -1265,7 +1265,7 @@ }, { "cell_type": "markdown", - "id": "eacb5e84", + "id": "fbce138f", "metadata": { "editable": true }, @@ -1277,7 +1277,7 @@ }, { "cell_type": "markdown", - "id": "6293d8f3", + "id": "cee42ab7", "metadata": { "editable": true }, @@ -1287,7 +1287,7 @@ }, { "cell_type": "markdown", - "id": "6d20462b", + "id": "5ae1628c", "metadata": { "editable": true }, @@ -1299,7 +1299,7 @@ }, { "cell_type": "markdown", - "id": "26e9a13f", + "id": "dbc046c4", "metadata": { "editable": true }, @@ -1309,7 +1309,7 @@ }, { "cell_type": "markdown", - "id": "90aef52e", + "id": "62b7934c", "metadata": { "editable": true }, @@ -1321,7 +1321,7 @@ }, { "cell_type": "markdown", - "id": "c53e0879", + "id": "cd3f6f0a", "metadata": { "editable": true }, @@ -1331,7 +1331,7 @@ }, { "cell_type": "markdown", - "id": "1c962cdd", + "id": "372433e3", "metadata": { "editable": true }, @@ -1351,7 +1351,7 @@ }, { "cell_type": "markdown", - "id": "37fb436c", + "id": "3b0cb06e", "metadata": { "editable": true }, @@ -1363,7 +1363,7 @@ }, { "cell_type": "markdown", - "id": "d54252a2", + "id": "fd1d9050", "metadata": { "editable": true }, @@ -1373,7 +1373,7 @@ }, { "cell_type": "markdown", - "id": "84beaae2", + "id": "0b22eabb", "metadata": { "editable": true }, @@ -1385,7 +1385,7 @@ }, { "cell_type": "markdown", - "id": "fa603cae", + "id": "f38b37d2", "metadata": { "editable": true }, @@ -1401,7 +1401,7 @@ }, { "cell_type": "markdown", - "id": "66a6c715", + "id": "4560c89b", "metadata": { "editable": true }, @@ -1413,7 +1413,7 @@ }, { "cell_type": "markdown", - "id": "6d3f3f1e", + "id": "8255a80f", "metadata": { "editable": true }, @@ -1425,7 +1425,7 @@ }, { "cell_type": "markdown", - "id": "82c185cb", + "id": "8dbf6a88", "metadata": { "editable": true }, @@ -1435,7 +1435,7 @@ }, { "cell_type": "markdown", - "id": "bce2b968", + "id": "dc353570", "metadata": { "editable": true }, @@ -1447,7 +1447,7 @@ }, { "cell_type": "markdown", - "id": "a970e099", + "id": "6565e7ac", "metadata": { "editable": true }, @@ -1459,7 +1459,7 @@ }, { "cell_type": "markdown", - "id": "237be57f", + "id": "05e47e0a", "metadata": { "editable": true }, @@ -1471,7 +1471,7 @@ }, { "cell_type": "markdown", - "id": "a18e6161", + "id": "d55883da", "metadata": { "editable": true }, @@ -1481,7 +1481,7 @@ }, { "cell_type": "markdown", - "id": "bfc79b5d", + "id": "39f7b542", "metadata": { "editable": true }, @@ -1493,7 +1493,7 @@ }, { "cell_type": "markdown", - "id": "ff9be50a", + "id": "141d2a34", "metadata": { "editable": true }, @@ -1503,7 +1503,7 @@ }, { "cell_type": "markdown", - "id": "c36b4d3b", + "id": "787755c1", "metadata": { "editable": true }, @@ -1515,7 +1515,7 @@ }, { "cell_type": "markdown", - "id": "1f47d2fe", + "id": "ddd7223d", "metadata": { "editable": true }, @@ -1525,7 +1525,7 @@ }, { "cell_type": "markdown", - "id": "de02dfeb", + "id": "de1e5717", "metadata": { "editable": true }, @@ -1537,7 +1537,7 @@ }, { "cell_type": "markdown", - "id": "f2f7f11f", + "id": "0d5ad78d", "metadata": { "editable": true }, @@ -1548,7 +1548,7 @@ }, { "cell_type": "markdown", - "id": "51c9b081", + "id": "c112457b", "metadata": { "editable": true }, @@ -1560,7 +1560,7 @@ }, { "cell_type": "markdown", - "id": "21385c25", + "id": "363c1a47", "metadata": { "editable": true }, @@ -1572,7 +1572,7 @@ }, { "cell_type": "markdown", - "id": "128fe7b5", + "id": "06a0a3ed", "metadata": { "editable": true }, @@ -1582,7 +1582,7 @@ }, { "cell_type": "markdown", - "id": "0e6bd855", + "id": "98083d0f", "metadata": { "editable": true }, @@ -1594,7 +1594,7 @@ }, { "cell_type": "markdown", - "id": "42648dca", + "id": "6e494443", "metadata": { "editable": true }, @@ -1620,7 +1620,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "d489c6c3", + "id": "42524afe", "metadata": { "collapsed": false, "editable": true @@ -1677,7 +1677,7 @@ }, { "cell_type": "markdown", - "id": "5ca6d314", + "id": "8a04c098", "metadata": { "editable": true }, @@ -1689,7 +1689,7 @@ }, { "cell_type": "markdown", - "id": "617b0921", + "id": "8a837902", "metadata": { "editable": true }, @@ -1701,7 +1701,7 @@ }, { "cell_type": "markdown", - "id": "1c83283f", + "id": "5ff37c0d", "metadata": { "editable": true }, @@ -1711,7 +1711,7 @@ }, { "cell_type": "markdown", - "id": "947e5e41", + "id": "59243c73", "metadata": { "editable": true }, @@ -1723,7 +1723,7 @@ }, { "cell_type": "markdown", - "id": "d6a9f842", + "id": "aa20b129", "metadata": { "editable": true }, @@ -1733,7 +1733,7 @@ }, { "cell_type": "markdown", - "id": "872e689a", + "id": "3b67b193", "metadata": { "editable": true }, @@ -1745,7 +1745,7 @@ }, { "cell_type": "markdown", - "id": "cebc00e8", + "id": "28823267", "metadata": { "editable": true }, @@ -1756,7 +1756,7 @@ }, { "cell_type": "markdown", - "id": "7eec7f44", + "id": "c90b67ca", "metadata": { "editable": true }, @@ -1768,7 +1768,7 @@ }, { "cell_type": "markdown", - "id": "672c83dc", + "id": "e4fcdba6", "metadata": { "editable": true }, @@ -1778,7 +1778,7 @@ }, { "cell_type": "markdown", - "id": "91c46fce", + "id": "715664f9", "metadata": { "editable": true }, @@ -1790,7 +1790,7 @@ }, { "cell_type": "markdown", - "id": "b55056e0", + "id": "7b853f11", "metadata": { "editable": true }, @@ -1808,7 +1808,7 @@ }, { "cell_type": "markdown", - "id": "5ddac4ae", + "id": "803a45ce", "metadata": { "editable": true }, @@ -1819,7 +1819,7 @@ }, { "cell_type": "markdown", - "id": "22fa1dc5", + "id": "61310ff9", "metadata": { "editable": true }, @@ -1831,7 +1831,7 @@ }, { "cell_type": "markdown", - "id": "8a644118", + "id": "66491438", "metadata": { "editable": true }, @@ -1841,7 +1841,7 @@ }, { "cell_type": "markdown", - "id": "0dd071ac", + "id": "e8d2d136", "metadata": { "editable": true }, @@ -1858,7 +1858,7 @@ }, { "cell_type": "markdown", - "id": "91e247a6", + "id": "f7952451", "metadata": { "editable": true }, @@ -1872,7 +1872,7 @@ }, { "cell_type": "markdown", - "id": "3d1a70c1", + "id": "b19b5113", "metadata": { "editable": true }, @@ -1887,7 +1887,7 @@ }, { "cell_type": "markdown", - "id": "80dba55b", + "id": "4caf5a80", "metadata": { "editable": true }, @@ -1899,7 +1899,7 @@ }, { "cell_type": "markdown", - "id": "54a00058", + "id": "633e3356", "metadata": { "editable": true }, @@ -1927,7 +1927,7 @@ }, { "cell_type": "markdown", - "id": "19ee429a", + "id": "76ac96ea", "metadata": { "editable": true }, @@ -1939,7 +1939,7 @@ }, { "cell_type": "markdown", - "id": "672c4ae8", + "id": "74510d78", "metadata": { "editable": true }, @@ -1954,7 +1954,7 @@ }, { "cell_type": "markdown", - "id": "65e42723", + "id": "faac8dc8", "metadata": { "editable": true }, @@ -1965,7 +1965,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "ac13df75", + "id": "91ea69e9", "metadata": { "collapsed": false, "editable": true @@ -2164,7 +2164,7 @@ }, { "cell_type": "markdown", - "id": "a9dfad7f", + "id": "d0b7f9cf", "metadata": { "editable": true }, @@ -2176,7 +2176,7 @@ }, { "cell_type": "markdown", - "id": "d5f21bf1", + "id": "9463bbe9", "metadata": { "editable": true }, @@ -2191,7 +2191,7 @@ }, { "cell_type": "markdown", - "id": "dec92019", + "id": "7b64fd8d", "metadata": { "editable": true }, @@ -2208,7 +2208,7 @@ }, { "cell_type": "markdown", - "id": "9fae1ff2", + "id": "8b168bf3", "metadata": { "editable": true }, @@ -2229,7 +2229,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "407db5be", + "id": "fe97004b", "metadata": { "collapsed": false, "editable": true @@ -2242,7 +2242,7 @@ }, { "cell_type": "markdown", - "id": "0559db68", + "id": "9ebe52b7", "metadata": { "editable": true }, @@ -2252,7 +2252,7 @@ }, { "cell_type": "markdown", - "id": "31fb5879", + "id": "ea4caa80", "metadata": { "editable": true }, @@ -2264,7 +2264,7 @@ }, { "cell_type": "markdown", - "id": "70875994", + "id": "447ad01e", "metadata": { "editable": true }, @@ -2279,7 +2279,7 @@ }, { "cell_type": "markdown", - "id": "6e356c92", + "id": "9917fc9c", "metadata": { "editable": true }, @@ -2289,7 +2289,7 @@ }, { "cell_type": "markdown", - "id": "e3665fc9", + "id": "4ec0b02c", "metadata": { "editable": true }, @@ -2308,7 +2308,7 @@ }, { "cell_type": "markdown", - "id": "ee5d3e53", + "id": "23f12ebb", "metadata": { "editable": true }, @@ -2318,7 +2318,7 @@ }, { "cell_type": "markdown", - "id": "d4fc6762", + "id": "e53e01bb", "metadata": { "editable": true }, @@ -2330,7 +2330,7 @@ }, { "cell_type": "markdown", - "id": "5aed4a56", + "id": "953de6d5", "metadata": { "editable": true }, @@ -2340,7 +2340,7 @@ }, { "cell_type": "markdown", - "id": "d3097c98", + "id": "1addd2bf", "metadata": { "editable": true }, @@ -2352,7 +2352,7 @@ }, { "cell_type": "markdown", - "id": "5e9ac7a0", + "id": "818d1345", "metadata": { "editable": true }, @@ -2362,7 +2362,7 @@ }, { "cell_type": "markdown", - "id": "0b7eb4cd", + "id": "7da978de", "metadata": { "editable": true }, @@ -2374,7 +2374,7 @@ }, { "cell_type": "markdown", - "id": "b0c0def1", + "id": "d4d1d665", "metadata": { "editable": true }, @@ -2385,7 +2385,7 @@ }, { "cell_type": "markdown", - "id": "b59f20fc", + "id": "ebf7b0d8", "metadata": { "editable": true }, @@ -2397,7 +2397,7 @@ }, { "cell_type": "markdown", - "id": "d5eed580", + "id": "aa66eb96", "metadata": { "editable": true }, @@ -2409,7 +2409,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "af36849d", + "id": "bcb69ce9", "metadata": { "collapsed": false, "editable": true @@ -2433,50 +2433,19 @@ }, { "cell_type": "markdown", - "id": "ae234a3d", + "id": "60a3d131", "metadata": { "editable": true }, "source": [ - "## Back to the more realistic cases\n", + "## Support Vector Machines and Regression\n", "\n", - "We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the **slack** parameter $C$ we have" + "Material may be added if of interest. See Bishop chapter 7.1 for a discussion." ] }, { "cell_type": "markdown", - "id": "41639135", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1K(\\boldsymbol{x}_1,\\boldsymbol{x}_1) & y_1y_2K(\\boldsymbol{x}_1,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_1,\\boldsymbol{x}_n) \\\\\n", - "y_2y_1K(\\boldsymbol{x}_2,\\boldsymbol{x}_1) & y_2y_2K(\\boldsymbol{x}_2,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_2,\\boldsymbol{x}_n) \\\\\n", - "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", - "\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", - "y_ny_1K(\\boldsymbol{x}_n,\\boldsymbol{x}_1) & y_ny_2K(\\boldsymbol{x}_n\\boldsymbol{x}_2) & \\dots & \\dots & y_ny_nK(\\boldsymbol{x}_n,\\boldsymbol{x}_n) \\\\\n", - "\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{I}\\boldsymbol{\\lambda},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "32fd6400", - "metadata": { - "editable": true - }, - "source": [ - "subject to $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$. Here we defined the vectors $\\boldsymbol{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n", - "$\\boldsymbol{y}=[y_1,y_2,\\dots,y_n]$. \n", - "With the slack constants this leads to the additional constraint $0\\leq \\lambda_i \\leq C$.\n", - "\n", - "**code will be added**" - ] - }, - { - "cell_type": "markdown", - "id": "74d5d1dd", + "id": "09a979c4", "metadata": { "editable": true }, @@ -2486,7 +2455,7 @@ }, { "cell_type": "markdown", - "id": "25c2688b", + "id": "e93c0928", "metadata": { "editable": true }, @@ -2501,7 +2470,7 @@ }, { "cell_type": "markdown", - "id": "9477fb34", + "id": "906a462a", "metadata": { "editable": true }, @@ -2517,7 +2486,7 @@ }, { "cell_type": "markdown", - "id": "20d9a817", + "id": "ce4e7093", "metadata": { "editable": true }, @@ -2542,7 +2511,7 @@ }, { "cell_type": "markdown", - "id": "47038b09", + "id": "030684fc", "metadata": { "editable": true }, @@ -2571,7 +2540,7 @@ }, { "cell_type": "markdown", - "id": "47679fe4", + "id": "8e1d2370", "metadata": { "editable": true }, @@ -2587,7 +2556,7 @@ }, { "cell_type": "markdown", - "id": "006416fc", + "id": "5ceec6b5", "metadata": { "editable": true }, @@ -2612,7 +2581,7 @@ }, { "cell_type": "markdown", - "id": "9b7e17cf", + "id": "5336ae51", "metadata": { "editable": true }, @@ -2659,7 +2628,7 @@ }, { "cell_type": "markdown", - "id": "019587a2", + "id": "69d7807b", "metadata": { "editable": true }, @@ -2695,7 +2664,7 @@ }, { "cell_type": "markdown", - "id": "3746e15c", + "id": "a5788fcf", "metadata": { "editable": true }, @@ -2718,7 +2687,7 @@ }, { "cell_type": "markdown", - "id": "bdf84776", + "id": "fda2577a", "metadata": { "editable": true }, @@ -2741,7 +2710,7 @@ }, { "cell_type": "markdown", - "id": "38966515", + "id": "aa0f65b6", "metadata": { "editable": true }, @@ -2761,7 +2730,7 @@ }, { "cell_type": "markdown", - "id": "d6397d83", + "id": "a387c41a", "metadata": { "editable": true }, @@ -2777,7 +2746,7 @@ }, { "cell_type": "markdown", - "id": "9b9e5831", + "id": "b4412a32", "metadata": { "editable": true }, @@ -2805,7 +2774,7 @@ }, { "cell_type": "markdown", - "id": "9e1c05ba", + "id": "09fe89d1", "metadata": { "editable": true }, @@ -2827,7 +2796,7 @@ }, { "cell_type": "markdown", - "id": "d5a44aa7", + "id": "a1625399", "metadata": { "editable": true }, @@ -2852,7 +2821,7 @@ }, { "cell_type": "markdown", - "id": "b50a77ca", + "id": "a60af950", "metadata": { "editable": true }, @@ -2870,7 +2839,7 @@ }, { "cell_type": "markdown", - "id": "ac817b17", + "id": "717ff832", "metadata": { "editable": true }, @@ -2898,7 +2867,7 @@ }, { "cell_type": "markdown", - "id": "ff3859f2", + "id": "69120211", "metadata": { "editable": true }, @@ -2912,7 +2881,7 @@ }, { "cell_type": "markdown", - "id": "54becfd9", + "id": "cf743a91", "metadata": { "editable": true }, @@ -2938,7 +2907,7 @@ }, { "cell_type": "markdown", - "id": "d716cbf5", + "id": "2b1c821d", "metadata": { "editable": true }, @@ -2967,7 +2936,7 @@ }, { "cell_type": "markdown", - "id": "c3fdf6bc", + "id": "d07a08c1", "metadata": { "editable": true }, @@ -2984,7 +2953,7 @@ }, { "cell_type": "markdown", - "id": "fcfe2c42", + "id": "21760d88", "metadata": { "editable": true }, @@ -3006,7 +2975,7 @@ }, { "cell_type": "markdown", - "id": "08107735", + "id": "a215b485", "metadata": { "editable": true }, @@ -3024,7 +2993,7 @@ }, { "cell_type": "markdown", - "id": "cb0979f6", + "id": "e0e5b2f2", "metadata": { "editable": true }, @@ -3047,7 +3016,7 @@ }, { "cell_type": "markdown", - "id": "fde0921c", + "id": "883ff967", "metadata": { "editable": true }, @@ -3066,7 +3035,7 @@ }, { "cell_type": "markdown", - "id": "321a4de5", + "id": "a8c66672", "metadata": { "editable": true }, @@ -3082,7 +3051,7 @@ }, { "cell_type": "markdown", - "id": "e5213ac1", + "id": "38867651", "metadata": { "editable": true }, @@ -3097,7 +3066,7 @@ }, { "cell_type": "markdown", - "id": "83919fb4", + "id": "31d4a91e", "metadata": { "editable": true }, @@ -3121,7 +3090,7 @@ }, { "cell_type": "markdown", - "id": "7f6ebb56", + "id": "d8eb36e7", "metadata": { "editable": true }, @@ -3133,7 +3102,7 @@ }, { "cell_type": "markdown", - "id": "bc772bc0", + "id": "03f66dfa", "metadata": { "editable": true }, @@ -3151,7 +3120,7 @@ }, { "cell_type": "markdown", - "id": "941c527b", + "id": "15a58ad9", "metadata": { "editable": true }, @@ -3161,7 +3130,7 @@ }, { "cell_type": "markdown", - "id": "1ee5cfa8", + "id": "8fee60a6", "metadata": { "editable": true }, @@ -3179,7 +3148,7 @@ }, { "cell_type": "markdown", - "id": "d8d11e38", + "id": "bd6b9ab1", "metadata": { "editable": true }, @@ -3189,7 +3158,7 @@ }, { "cell_type": "markdown", - "id": "1f4310f2", + "id": "2f1d5ba5", "metadata": { "editable": true }, @@ -3208,7 +3177,7 @@ }, { "cell_type": "markdown", - "id": "81055485", + "id": "cda96c2d", "metadata": { "editable": true }, @@ -3220,7 +3189,7 @@ }, { "cell_type": "markdown", - "id": "47cf1bb7", + "id": "17566561", "metadata": { "editable": true }, @@ -3234,7 +3203,7 @@ }, { "cell_type": "markdown", - "id": "fafde12c", + "id": "161d5743", "metadata": { "editable": true }, @@ -3249,7 +3218,7 @@ }, { "cell_type": "markdown", - "id": "ebc67a04", + "id": "707e8fcb", "metadata": { "editable": true }, @@ -3267,7 +3236,7 @@ }, { "cell_type": "markdown", - "id": "a85c0132", + "id": "d1fff888", "metadata": { "editable": true }, @@ -3281,7 +3250,7 @@ }, { "cell_type": "markdown", - "id": "122db74b", + "id": "ae35a3c3", "metadata": { "editable": true }, @@ -3299,7 +3268,7 @@ }, { "cell_type": "markdown", - "id": "17fbd221", + "id": "de8a68fe", "metadata": { "editable": true }, @@ -3323,7 +3292,7 @@ }, { "cell_type": "markdown", - "id": "1f27d3da", + "id": "6fce707b", "metadata": { "editable": true }, @@ -3361,7 +3330,7 @@ }, { "cell_type": "markdown", - "id": "f58c00db", + "id": "1b2d107e", "metadata": { "editable": true }, @@ -3384,7 +3353,7 @@ }, { "cell_type": "markdown", - "id": "21ac0978", + "id": "649355d5", "metadata": { "editable": true }, @@ -3420,7 +3389,7 @@ }, { "cell_type": "markdown", - "id": "1cb06f52", + "id": "1bd0b6d2", "metadata": { "editable": true }, @@ -3441,7 +3410,7 @@ }, { "cell_type": "markdown", - "id": "29cb1748", + "id": "d3d56d06", "metadata": { "editable": true }, @@ -3463,7 +3432,7 @@ }, { "cell_type": "markdown", - "id": "d4cf1abd", + "id": "281ec76c", "metadata": { "editable": true }, @@ -3483,7 +3452,7 @@ }, { "cell_type": "markdown", - "id": "60a028e5", + "id": "16eb46ae", "metadata": { "editable": true }, @@ -3498,7 +3467,7 @@ }, { "cell_type": "markdown", - "id": "0b3b3776", + "id": "d6880678", "metadata": { "editable": true }, @@ -3516,7 +3485,7 @@ }, { "cell_type": "markdown", - "id": "f933b11b", + "id": "f85f760a", "metadata": { "editable": true }, @@ -3546,7 +3515,7 @@ }, { "cell_type": "markdown", - "id": "49558fe5", + "id": "469842c0", "metadata": { "editable": true }, @@ -3575,7 +3544,7 @@ }, { "cell_type": "markdown", - "id": "68bc325e", + "id": "66388087", "metadata": { "editable": true }, @@ -3606,7 +3575,7 @@ }, { "cell_type": "markdown", - "id": "42068b4c", + "id": "e26957e8", "metadata": { "editable": true }, @@ -3629,7 +3598,7 @@ }, { "cell_type": "markdown", - "id": "0adb028d", + "id": "31886d45", "metadata": { "editable": true }, @@ -3647,7 +3616,7 @@ }, { "cell_type": "markdown", - "id": "ad0660f2", + "id": "36603525", "metadata": { "editable": true }, @@ -3670,7 +3639,7 @@ }, { "cell_type": "markdown", - "id": "90602dfa", + "id": "63bf1e9a", "metadata": { "editable": true }, @@ -3691,7 +3660,7 @@ }, { "cell_type": "markdown", - "id": "3028d0ca", + "id": "c5b98e9b", "metadata": { "editable": true }, @@ -3707,7 +3676,7 @@ }, { "cell_type": "markdown", - "id": "0f5f1146", + "id": "518e3444", "metadata": { "editable": true }, diff --git a/doc/src/week47/week47.do.txt b/doc/src/week47/week47.do.txt index 39d85acbf..2137f86c7 100644 --- a/doc/src/week47/week47.do.txt +++ b/doc/src/week47/week47.do.txt @@ -326,7 +326,7 @@ $\vert\vert \bm{w}\vert\vert =1$ subject to the condition !bt \[ -y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. +y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n. \] !et All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $b$ and $w_1$ and $w_2$ define this line. @@ -340,14 +340,14 @@ We seek thus the largest value $M$ defined by or just !bt \[ -y_i(\bm{w}^T\bm{x}_i+b) \geq M\vert \vert \bm{w}\vert\vert \hspace{0.1cm}\forall i. +y_i(\bm{w}^T\bm{x}_i+b) \geq M\vert \vert \bm{w}\vert\vert \hspace{0.1cm}\forall i=1,2,\dots,n. \] !et If we scale the equation so that $\vert \vert \bm{w}\vert\vert = 1/M$, we have to find the minimum of -$\bm{w}^T\bm{w}=\vert \vert \bm{w}\vert\vert$ (the norm) subject to the condition +$\bm{w}^T\bm{w}=\vert \vert \bm{w}\vert\vert_2^2$ (the norm) subject to the condition !bt \[ -y_i(\bm{w}^T\bm{x}_i+b) \geq 1 \hspace{0.1cm}\forall i. +y_i(\bm{w}^T\bm{x}_i+b) \geq 1 \hspace{0.1cm}\forall i=1,2,\dots,n. \] !et @@ -515,8 +515,8 @@ y_ny_1\bm{x}_n^T\bm{x}_1 & y_ny_2\bm{x}_n^T\bm{x}_2 & \dots & \dots & y_ny_n\bm{ \end{bmatrix}\bm{\lambda}-\mathbb{1}\bm{\lambda}, \] !et -subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and -$\bm{y}=[y_1,y_2,\dots,y_n]$. +subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda}^T =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and +$\bm{y}^T=[y_1,y_2,\dots,y_n]$. !split @@ -1136,25 +1136,11 @@ sol[’x’] sol[’primal objective’] !ec + !split -===== Back to the more realistic cases ===== +===== Support Vector Machines and Regression ===== -We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the _slack_ parameter $C$ we have -!bt -\[ -\frac{1}{2} \bm{\lambda}^T\begin{bmatrix} y_1y_1K(\bm{x}_1,\bm{x}_1) & y_1y_2K(\bm{x}_1,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_1,\bm{x}_n) \\ -y_2y_1K(\bm{x}_2,\bm{x}_1) & y_2y_2K(\bm{x}_2,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_2,\bm{x}_n) \\ -\dots & \dots & \dots & \dots & \dots \\ -\dots & \dots & \dots & \dots & \dots \\ -y_ny_1K(\bm{x}_n,\bm{x}_1) & y_ny_2K(\bm{x}_n\bm{x}_2) & \dots & \dots & y_ny_nK(\bm{x}_n,\bm{x}_n) \\ -\end{bmatrix}\bm{\lambda}-\mathbb{I}\bm{\lambda}, -\] -!et -subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and -$\bm{y}=[y_1,y_2,\dots,y_n]$. -With the slack constants this leads to the additional constraint $0\leq \lambda_i \leq C$. - -_code will be added_ +Material may be added if of interest. See Bishop chapter 7.1 for a discussion. !split ===== Summary of course =====