almost there, svm misses code for real data
This commit is contained in:
mhjensen
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -187,7 +194,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._svm-bs008.html">9</a></li>
|
||||
<li><a href="._svm-bs009.html">10</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs001.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -166,8 +173,14 @@ classification problems) of variables. It is also used for regression
|
||||
problems.
|
||||
|
||||
<p>
|
||||
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
|
||||
We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines.
|
||||
With SVMs we distinguish between hard margin and soft margins. The
|
||||
latter introduces a so-called softening parameter to be discussed
|
||||
below. We distinguish also between linear and non-linear
|
||||
approaches. The latter are the most frequent ones since it is rather
|
||||
unlikely that we can separate classes easily by say straight lines.
|
||||
|
||||
<p>
|
||||
<b>Note: several figures are missing. They will be added shortly. To run the codes, use the jupyter notebook</b>
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -186,7 +199,7 @@ We distinguish also between linear and non-linear approaches. The latter are the
|
||||
<li><a href="._svm-bs009.html">10</a></li>
|
||||
<li><a href="._svm-bs010.html">11</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs002.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -177,7 +184,7 @@ circles.
|
||||
<li><a href="._svm-bs010.html">11</a></li>
|
||||
<li><a href="._svm-bs011.html">12</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs003.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -187,7 +194,7 @@ $$
|
||||
<li><a href="._svm-bs011.html">12</a></li>
|
||||
<li><a href="._svm-bs012.html">13</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs004.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -203,7 +210,7 @@ When we try to separate hyperplanes, if it exists, we can use it to construct a
|
||||
<li><a href="._svm-bs012.html">13</a></li>
|
||||
<li><a href="._svm-bs013.html">14</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs005.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -189,7 +196,7 @@ for our data sample.
|
||||
<li><a href="._svm-bs013.html">14</a></li>
|
||||
<li><a href="._svm-bs014.html">15</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs006.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -185,7 +192,7 @@ $$
|
||||
<li><a href="._svm-bs014.html">15</a></li>
|
||||
<li><a href="._svm-bs015.html">16</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs007.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -189,7 +196,7 @@ $$
|
||||
<li><a href="._svm-bs015.html">16</a></li>
|
||||
<li><a href="._svm-bs016.html">17</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs008.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -193,7 +200,7 @@ at all.
|
||||
<li><a href="._svm-bs016.html">17</a></li>
|
||||
<li><a href="._svm-bs017.html">18</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs009.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -205,7 +212,7 @@ We have thus defined our margin as the invers of the norm of \( \boldsymbol{w} \
|
||||
<li><a href="._svm-bs017.html">18</a></li>
|
||||
<li><a href="._svm-bs018.html">19</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs010.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -217,7 +224,7 @@ Then \( dz \) is no longer arbitrary.
|
||||
<li><a href="._svm-bs018.html">19</a></li>
|
||||
<li><a href="._svm-bs019.html">20</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs011.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -210,7 +217,7 @@ $$
|
||||
<li><a href="._svm-bs019.html">20</a></li>
|
||||
<li><a href="._svm-bs020.html">21</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs012.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -208,7 +215,7 @@ When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support
|
||||
<li><a href="._svm-bs020.html">21</a></li>
|
||||
<li><a href="._svm-bs021.html">22</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs013.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -191,7 +198,7 @@ subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vec
|
||||
<li><a href="._svm-bs021.html">22</a></li>
|
||||
<li><a href="._svm-bs022.html">23</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs014.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -200,6 +207,8 @@ Below we discuss how to find the optimal values of \( \lambda_i \). Before we pr
|
||||
<li><a href="._svm-bs021.html">22</a></li>
|
||||
<li><a href="._svm-bs022.html">23</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs015.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -200,6 +207,9 @@ misclassifications.
|
||||
<li><a href="._svm-bs021.html">22</a></li>
|
||||
<li><a href="._svm-bs022.html">23</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs024.html">25</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs016.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -218,6 +225,8 @@ $$
|
||||
<li><a href="._svm-bs021.html">22</a></li>
|
||||
<li><a href="._svm-bs022.html">23</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs024.html">25</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs017.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -186,6 +193,8 @@ we need to introduce for example a polynomial transformation to a two-dimensiona
|
||||
<li><a href="._svm-bs021.html">22</a></li>
|
||||
<li><a href="._svm-bs022.html">23</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs024.html">25</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs018.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -147,15 +154,15 @@ MathJax.Hub.Config({
|
||||
<h2 id="___sec17" class="anchor">The equations </h2>
|
||||
|
||||
<p>
|
||||
Suppose we define a polynomial transformation of degree two (we continue to live in a plane with \( x_1 \) and \( x_2 \) as variables)
|
||||
Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
|
||||
$$
|
||||
z = \phi(x) =\left(1, x_1, x_2, x_1^2, x_2^2, x_1x_2).
|
||||
z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
|
||||
$$
|
||||
|
||||
<p>
|
||||
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{Z}_j,
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
|
||||
$$
|
||||
|
||||
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors
|
||||
@@ -164,6 +171,17 @@ y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i,
|
||||
$$
|
||||
|
||||
from which we also find \( b \).
|
||||
To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
|
||||
$$
|
||||
|
||||
For the above example, the kernel reads
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
|
||||
$$
|
||||
|
||||
We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -186,6 +204,8 @@ from which we also find \( b \).
|
||||
<li><a href="._svm-bs021.html">22</a></li>
|
||||
<li><a href="._svm-bs022.html">23</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs024.html">25</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs019.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -144,7 +151,38 @@ MathJax.Hub.Config({
|
||||
<a name="part0019"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec18" class="anchor">Different kernels </h2>
|
||||
<h2 id="___sec18" class="anchor">The problem to solve </h2>
|
||||
Using our definition of the kernel We can rewrite again the Lagrangian
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
|
||||
$$
|
||||
|
||||
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
|
||||
$$
|
||||
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
|
||||
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
|
||||
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
|
||||
$$
|
||||
|
||||
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
|
||||
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
|
||||
If we add the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
|
||||
|
||||
<p>
|
||||
We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
|
||||
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \).
|
||||
Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into
|
||||
\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -166,6 +204,8 @@ MathJax.Hub.Config({
|
||||
<li><a href="._svm-bs021.html">22</a></li>
|
||||
<li><a href="._svm-bs022.html">23</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs024.html">25</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs020.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -144,7 +151,33 @@ MathJax.Hub.Config({
|
||||
<a name="part0020"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec19" class="anchor">Quadratic coefficient matrix </h2>
|
||||
<h2 id="___sec19" class="anchor">Different kernels and Mercer's theorem </h2>
|
||||
|
||||
<p>
|
||||
There are several popular kernels being used. These are
|
||||
|
||||
<ol>
|
||||
<li> Linear: \( K(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{x}^T\boldsymbol{y} \),</li>
|
||||
<li> Polynomial: \( K(\boldsymbol{x},\boldsymbol{y})=(\boldsymbol{x}^T\boldsymbol{y}+\gamma)^d \),</li>
|
||||
<li> Gaussian Radial Basis Function: \( K(\boldsymbol{x},\boldsymbol{y})=\exp{\left(-\gamma\vert\vert\boldsymbol{x}-\boldsymbol{y}\vert\vert^2\right)} \),</li>
|
||||
<li> Tanh: \( K(\boldsymbol{x},\boldsymbol{y})=\tanh{(\boldsymbol{x}^T\boldsymbol{y}+\gamma)} \),</li>
|
||||
</ol>
|
||||
|
||||
and many other ones.
|
||||
|
||||
<p>
|
||||
An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
|
||||
The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then
|
||||
there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space
|
||||
(possibly with much higher dimensions) such that
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
|
||||
$$
|
||||
|
||||
So you can use \( K \) as a kernel since you know \( \phi \) exists, even if
|
||||
you don’t know what \( \phi \) is.
|
||||
Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work
|
||||
well in practice.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -165,6 +198,8 @@ MathJax.Hub.Config({
|
||||
<li><a href="._svm-bs021.html">22</a></li>
|
||||
<li><a href="._svm-bs022.html">23</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs024.html">25</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs021.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -142,9 +149,29 @@ MathJax.Hub.Config({
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0021"></a>
|
||||
<!-- !split -->
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec20" class="anchor">Mercer's theorem </h2>
|
||||
<h2 id="___sec20" class="anchor">Mathematical optimization of convex functions </h2>
|
||||
|
||||
<p>
|
||||
A mathematical (quadratic) optimization problem, or just optimization problem, has the form
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
|
||||
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
subject to some constraints for say a selected set \( i=1,2,\dots, n \).
|
||||
In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the
|
||||
vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with.
|
||||
|
||||
<p>
|
||||
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
|
||||
In our disussion on gradient descent methods we discussed at length the definition of a convex function.
|
||||
|
||||
<p>
|
||||
Convex optimization problems play a central role in applied mathematics and we recommend strongly <a href="http://web.stanford.edu/~boyd/cvxbook/" target="_self">Boyd and Vandenberghe's text on the topics</a>.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -164,6 +191,8 @@ MathJax.Hub.Config({
|
||||
<li class="active"><a href="._svm-bs021.html">22</a></li>
|
||||
<li><a href="._svm-bs022.html">23</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs024.html">25</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs022.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -144,25 +151,28 @@ MathJax.Hub.Config({
|
||||
<a name="part0022"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec21" class="anchor">Mathematical optimization of convex functions </h2>
|
||||
<h2 id="___sec21" class="anchor">How do we solve these problems? </h2>
|
||||
|
||||
<p>
|
||||
A mathematical optimization problem, or just optimization problem, has the form
|
||||
$$
|
||||
\mathrm{minimize}\hspace{0.1cm} f(x),
|
||||
$$
|
||||
|
||||
subject to some constraints \( g(\lambda_i) \leq b_i \) for say a selected set \( i=1,2,\dots, n \).
|
||||
In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the
|
||||
vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with.
|
||||
and \( f(x) \) is our objective function while \( g(\lambda_i) \leq b_i \) represents our constraint function.
|
||||
If we use Python as programming language and wish to venture beyond
|
||||
<b>scikit-learn</b>, <b>tensorflow</b> and similar software which makes our
|
||||
lives so much easier, we need to dive into the wonderful world of
|
||||
quadratic programming. We can, if we wish, solve the minimization
|
||||
problem using say standard gradient methods or conjugate gradient
|
||||
methods. However, these methods tend to exhibit a rather slow
|
||||
converge. So, welcome to the promised land of quadratic programming.
|
||||
|
||||
<p>
|
||||
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
|
||||
In our disussion on gradient descent methods we discussed at length the definition of a convex function.
|
||||
The functions we need are contained in the quadratic programming package <b>CVXOPT</b> and we need to import it together with <b>numpy</b> as
|
||||
|
||||
<p>
|
||||
Convex optimization problems play a central role in applied mathematics and we recommend strongly <a href="http://web.stanford.edu/~boyd/cvxbook/" target="_self">Boyd and Vandenberghe's text on the topics</a>.
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
This will make our life much easier. You don't need t write your own optimizer.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -181,6 +191,8 @@ Convex optimization problems play a central role in applied mathematics and we r
|
||||
<li><a href="._svm-bs021.html">22</a></li>
|
||||
<li class="active"><a href="._svm-bs022.html">23</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs024.html">25</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs023.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -0,0 +1,267 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Support Vector Machines">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Support Vector Machines</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
|
||||
('Hyperplanes and all that', 2, None, '___sec1'),
|
||||
('What is a hyperplane?', 2, None, '___sec2'),
|
||||
('A $p$-dimensional space of features', 2, None, '___sec3'),
|
||||
('The two-dimensional case', 2, None, '___sec4'),
|
||||
('Getting into the details', 2, None, '___sec5'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec6'),
|
||||
('Solving the equations', 2, None, '___sec7'),
|
||||
('A better approach', 2, None, '___sec8'),
|
||||
('A quick reminder on Lagrangian multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Adding the muliplier', 2, None, '___sec10'),
|
||||
('Setting up the problem', 2, None, '___sec11'),
|
||||
('The problem to solve', 2, None, '___sec12'),
|
||||
('The last steps', 2, None, '___sec13'),
|
||||
('A soft classifier', 2, None, '___sec14'),
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="svm-bs.html">Data Analysis and Machine Learning: Support Vector Machines</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs001.html#___sec0" style="font-size: 80%;">Support Vector Machines, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs002.html#___sec1" style="font-size: 80%;">Hyperplanes and all that</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs003.html#___sec2" style="font-size: 80%;">What is a hyperplane?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs004.html#___sec3" style="font-size: 80%;">A \( p \)-dimensional space of features</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs005.html#___sec4" style="font-size: 80%;">The two-dimensional case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs006.html#___sec5" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs007.html#___sec6" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs008.html#___sec7" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs009.html#___sec8" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs010.html#___sec9" style="font-size: 80%;">A quick reminder on Lagrangian multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs011.html#___sec10" style="font-size: 80%;">Adding the muliplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs012.html#___sec11" style="font-size: 80%;">Setting up the problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs013.html#___sec12" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs014.html#___sec13" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs015.html#___sec14" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0023"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec22" class="anchor">A simplex example </h2>
|
||||
|
||||
<p>
|
||||
We remind ourselves about the general problem we want to solve
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
|
||||
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
<p>
|
||||
Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
|
||||
&\mathrm{subject to} \\ \nonumber
|
||||
&x, y \geq 0 \\ \nonumber
|
||||
&x+3y \geq 15 \\ \nonumber
|
||||
&2x+5y \leq 100 \\ \nonumber
|
||||
&3x+4y \leq 80. \\ \nonumber
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
|
||||
$$
|
||||
frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
|
||||
$$
|
||||
|
||||
Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
|
||||
$$
|
||||
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
$$
|
||||
|
||||
We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix
|
||||
$$
|
||||
\boldsymbol{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix}
|
||||
$$
|
||||
|
||||
is clearly positive semi-definite (all eigenvalues larger or equal zero).
|
||||
Finally, the vector \( \boldsymbol{h} \) is defined as
|
||||
$$
|
||||
\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero
|
||||
The following code does this for us
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Import the necessary packages</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span> <span style="color: #008000; font-weight: bold">import</span> matrix
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span> <span style="color: #008000; font-weight: bold">import</span> solvers
|
||||
<span style="color: #408080; font-style: italic"># Define QP parameters (directly)</span>
|
||||
P <span style="color: #666666">=</span> matrix([[<span style="color: #666666">1.0</span>,<span style="color: #666666">0.0</span>],[<span style="color: #666666">0.0</span>,<span style="color: #666666">0.0</span>]])
|
||||
q <span style="color: #666666">=</span> matrix([<span style="color: #666666">3.0</span>,<span style="color: #666666">4.0</span>])
|
||||
G <span style="color: #666666">=</span> matrix([[<span style="color: #666666">-1.0</span>,<span style="color: #666666">0.0</span>,<span style="color: #666666">-1.0</span>,<span style="color: #666666">2.0</span>,<span style="color: #666666">3.0</span>],[<span style="color: #666666">0.0</span>,<span style="color: #666666">-1.0</span>,<span style="color: #666666">-3.0</span>,<span style="color: #666666">5.0</span>,<span style="color: #666666">4.0</span>]])
|
||||
h <span style="color: #666666">=</span> matrix([<span style="color: #666666">0.0</span>,<span style="color: #666666">0.0</span>,<span style="color: #666666">-15.0</span>,<span style="color: #666666">100.0</span>,<span style="color: #666666">80.0</span>])
|
||||
<span style="color: #408080; font-style: italic"># Define QP parameters (with NumPy)</span>
|
||||
P <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>diag([<span style="color: #666666">1</span>,<span style="color: #666666">0</span>]), tc<span style="color: #666666">=</span>’d’)
|
||||
q <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>array([<span style="color: #666666">3</span>,<span style="color: #666666">4</span>]), tc<span style="color: #666666">=</span>’d’)
|
||||
G <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>array([[<span style="color: #666666">-1</span>,<span style="color: #666666">0</span>],[<span style="color: #666666">0</span>,<span style="color: #666666">-1</span>],[<span style="color: #666666">-1</span>,<span style="color: #666666">-3</span>],[<span style="color: #666666">2</span>,<span style="color: #666666">5</span>],[<span style="color: #666666">3</span>,<span style="color: #666666">4</span>]]), tc<span style="color: #666666">=</span>’d’)
|
||||
h <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>array([<span style="color: #666666">0</span>,<span style="color: #666666">0</span>,<span style="color: #666666">-15</span>,<span style="color: #666666">100</span>,<span style="color: #666666">80</span>]), tc<span style="color: #666666">=</span>’d’)
|
||||
<span style="color: #408080; font-style: italic"># Construct the QP, invoke solver</span>
|
||||
sol <span style="color: #666666">=</span> solvers<span style="color: #666666">.</span>qp(P,q,G,h)
|
||||
<span style="color: #408080; font-style: italic"># Extract optimal value and solution</span>
|
||||
sol[’x’]
|
||||
sol[’primal objective’]
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._svm-bs022.html">«</a></li>
|
||||
<li><a href="._svm-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs015.html">16</a></li>
|
||||
<li><a href="._svm-bs016.html">17</a></li>
|
||||
<li><a href="._svm-bs017.html">18</a></li>
|
||||
<li><a href="._svm-bs018.html">19</a></li>
|
||||
<li><a href="._svm-bs019.html">20</a></li>
|
||||
<li><a href="._svm-bs020.html">21</a></li>
|
||||
<li><a href="._svm-bs021.html">22</a></li>
|
||||
<li><a href="._svm-bs022.html">23</a></li>
|
||||
<li class="active"><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs024.html">25</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs024.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Support Vector Machines">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Support Vector Machines</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
|
||||
('Hyperplanes and all that', 2, None, '___sec1'),
|
||||
('What is a hyperplane?', 2, None, '___sec2'),
|
||||
('A $p$-dimensional space of features', 2, None, '___sec3'),
|
||||
('The two-dimensional case', 2, None, '___sec4'),
|
||||
('Getting into the details', 2, None, '___sec5'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec6'),
|
||||
('Solving the equations', 2, None, '___sec7'),
|
||||
('A better approach', 2, None, '___sec8'),
|
||||
('A quick reminder on Lagrangian multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Adding the muliplier', 2, None, '___sec10'),
|
||||
('Setting up the problem', 2, None, '___sec11'),
|
||||
('The problem to solve', 2, None, '___sec12'),
|
||||
('The last steps', 2, None, '___sec13'),
|
||||
('A soft classifier', 2, None, '___sec14'),
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="svm-bs.html">Data Analysis and Machine Learning: Support Vector Machines</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs001.html#___sec0" style="font-size: 80%;">Support Vector Machines, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs002.html#___sec1" style="font-size: 80%;">Hyperplanes and all that</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs003.html#___sec2" style="font-size: 80%;">What is a hyperplane?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs004.html#___sec3" style="font-size: 80%;">A \( p \)-dimensional space of features</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs005.html#___sec4" style="font-size: 80%;">The two-dimensional case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs006.html#___sec5" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs007.html#___sec6" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs008.html#___sec7" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs009.html#___sec8" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs010.html#___sec9" style="font-size: 80%;">A quick reminder on Lagrangian multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs011.html#___sec10" style="font-size: 80%;">Adding the muliplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs012.html#___sec11" style="font-size: 80%;">Setting up the problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs013.html#___sec12" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs014.html#___sec13" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs015.html#___sec14" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0024"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec23" class="anchor">Back to the more realistic cases </h2>
|
||||
|
||||
<p>
|
||||
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the <b>slack</b> parameter \( C \) we have
|
||||
$$
|
||||
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
|
||||
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
|
||||
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
|
||||
$$
|
||||
|
||||
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
|
||||
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
|
||||
With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
|
||||
|
||||
<p>
|
||||
<b>code will be added</b>
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._svm-bs023.html">«</a></li>
|
||||
<li><a href="._svm-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs016.html">17</a></li>
|
||||
<li><a href="._svm-bs017.html">18</a></li>
|
||||
<li><a href="._svm-bs018.html">19</a></li>
|
||||
<li><a href="._svm-bs019.html">20</a></li>
|
||||
<li><a href="._svm-bs020.html">21</a></li>
|
||||
<li><a href="._svm-bs021.html">22</a></li>
|
||||
<li><a href="._svm-bs022.html">23</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li class="active"><a href="._svm-bs024.html">25</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs025.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Support Vector Machines">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Support Vector Machines</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Support Vector Machines, overarching aims', 2, None, '___sec0'),
|
||||
('Hyperplanes and all that', 2, None, '___sec1'),
|
||||
('What is a hyperplane?', 2, None, '___sec2'),
|
||||
('A $p$-dimensional space of features', 2, None, '___sec3'),
|
||||
('The two-dimensional case', 2, None, '___sec4'),
|
||||
('Getting into the details', 2, None, '___sec5'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec6'),
|
||||
('Solving the equations', 2, None, '___sec7'),
|
||||
('A better approach', 2, None, '___sec8'),
|
||||
('A quick reminder on Lagrangian multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Adding the muliplier', 2, None, '___sec10'),
|
||||
('Setting up the problem', 2, None, '___sec11'),
|
||||
('The problem to solve', 2, None, '___sec12'),
|
||||
('The last steps', 2, None, '___sec13'),
|
||||
('A soft classifier', 2, None, '___sec14'),
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="svm-bs.html">Data Analysis and Machine Learning: Support Vector Machines</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs001.html#___sec0" style="font-size: 80%;">Support Vector Machines, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs002.html#___sec1" style="font-size: 80%;">Hyperplanes and all that</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs003.html#___sec2" style="font-size: 80%;">What is a hyperplane?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs004.html#___sec3" style="font-size: 80%;">A \( p \)-dimensional space of features</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs005.html#___sec4" style="font-size: 80%;">The two-dimensional case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs006.html#___sec5" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs007.html#___sec6" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs008.html#___sec7" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs009.html#___sec8" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs010.html#___sec9" style="font-size: 80%;">A quick reminder on Lagrangian multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs011.html#___sec10" style="font-size: 80%;">Adding the muliplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs012.html#___sec11" style="font-size: 80%;">Setting up the problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs013.html#___sec12" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs014.html#___sec13" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs015.html#___sec14" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0025"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec24" class="anchor">Multiclass problems and regression with SVMs </h2>
|
||||
This material will be added later.
|
||||
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._svm-bs024.html">«</a></li>
|
||||
<li><a href="._svm-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs017.html">18</a></li>
|
||||
<li><a href="._svm-bs018.html">19</a></li>
|
||||
<li><a href="._svm-bs019.html">20</a></li>
|
||||
<li><a href="._svm-bs020.html">21</a></li>
|
||||
<li><a href="._svm-bs021.html">22</a></li>
|
||||
<li><a href="._svm-bs022.html">23</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs024.html">25</a></li>
|
||||
<li class="active"><a href="._svm-bs025.html">26</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -61,14 +61,19 @@ Automatically generated HTML file from DocOnce source
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -124,11 +129,13 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs016.html#___sec15" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs017.html#___sec16" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs018.html#___sec17" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">Different kernels</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Quadratic coefficient matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">How do we solve these problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs020.html#___sec19" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs021.html#___sec20" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs022.html#___sec21" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs023.html#___sec22" style="font-size: 80%;">A simplex example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs024.html#___sec23" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._svm-bs025.html#___sec24" style="font-size: 80%;">Multiclass problems and regression with SVMs</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -187,7 +194,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._svm-bs008.html">9</a></li>
|
||||
<li><a href="._svm-bs009.html">10</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._svm-bs023.html">24</a></li>
|
||||
<li><a href="._svm-bs025.html">26</a></li>
|
||||
<li><a href="._svm-bs001.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -181,8 +181,14 @@ classification problems) of variables. It is also used for regression
|
||||
problems.
|
||||
|
||||
<p>
|
||||
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
|
||||
We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines.
|
||||
With SVMs we distinguish between hard margin and soft margins. The
|
||||
latter introduces a so-called softening parameter to be discussed
|
||||
below. We distinguish also between linear and non-linear
|
||||
approaches. The latter are the most frequent ones since it is rather
|
||||
unlikely that we can separate classes easily by say straight lines.
|
||||
|
||||
<p>
|
||||
<b>Note: several figures are missing. They will be added shortly. To run the codes, use the jupyter notebook</b>
|
||||
</section>
|
||||
|
||||
|
||||
@@ -829,10 +835,10 @@ we need to introduce for example a polynomial transformation to a two-dimensiona
|
||||
<h2 id="___sec17">The equations </h2>
|
||||
|
||||
<p>
|
||||
Suppose we define a polynomial transformation of degree two (we continue to live in a plane with \( x_1 \) and \( x_2 \) as variables)
|
||||
Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
|
||||
<p> <br>
|
||||
$$
|
||||
z = \phi(x) =\left(1, x_1, x_2, x_1^2, x_2^2, x_1x_2).
|
||||
z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -840,7 +846,7 @@ $$
|
||||
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)
|
||||
<p> <br>
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{Z}_j,
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -852,39 +858,117 @@ $$
|
||||
<p> <br>
|
||||
|
||||
from which we also find \( b \).
|
||||
To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
|
||||
<p> <br>
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
For the above example, the kernel reads
|
||||
<p> <br>
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec18">Different kernels </h2>
|
||||
</section>
|
||||
<h2 id="___sec18">The problem to solve </h2>
|
||||
Using our definition of the kernel We can rewrite again the Lagrangian
|
||||
<p> <br>
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
|
||||
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
|
||||
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<section>
|
||||
<h2 id="___sec19">Quadratic coefficient matrix </h2>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec20">Mercer's theorem </h2>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec21">Mathematical optimization of convex functions </h2>
|
||||
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
|
||||
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
|
||||
If we add the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
|
||||
|
||||
<p>
|
||||
A mathematical optimization problem, or just optimization problem, has the form
|
||||
We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type
|
||||
<p> <br>
|
||||
$$
|
||||
\mathrm{minimize}\hspace{0.1cm} f(x),
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
|
||||
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
subject to some constraints \( g(\lambda_i) \leq b_i \) for say a selected set \( i=1,2,\dots, n \).
|
||||
Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \).
|
||||
Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into
|
||||
\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec19">Different kernels and Mercer's theorem </h2>
|
||||
|
||||
<p>
|
||||
There are several popular kernels being used. These are
|
||||
|
||||
<ol>
|
||||
<p><li> Linear: \( K(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{x}^T\boldsymbol{y} \),</li>
|
||||
<p><li> Polynomial: \( K(\boldsymbol{x},\boldsymbol{y})=(\boldsymbol{x}^T\boldsymbol{y}+\gamma)^d \),</li>
|
||||
<p><li> Gaussian Radial Basis Function: \( K(\boldsymbol{x},\boldsymbol{y})=\exp{\left(-\gamma\vert\vert\boldsymbol{x}-\boldsymbol{y}\vert\vert^2\right)} \),</li>
|
||||
<p><li> Tanh: \( K(\boldsymbol{x},\boldsymbol{y})=\tanh{(\boldsymbol{x}^T\boldsymbol{y}+\gamma)} \),</li>
|
||||
</ol>
|
||||
<p>
|
||||
|
||||
and many other ones.
|
||||
|
||||
<p>
|
||||
An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
|
||||
The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then
|
||||
there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space
|
||||
(possibly with much higher dimensions) such that
|
||||
<p> <br>
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
So you can use \( K \) as a kernel since you know \( \phi \) exists, even if
|
||||
you don’t know what \( \phi \) is.
|
||||
Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work
|
||||
well in practice.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec20">Mathematical optimization of convex functions </h2>
|
||||
|
||||
<p>
|
||||
A mathematical (quadratic) optimization problem, or just optimization problem, has the form
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
|
||||
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
subject to some constraints for say a selected set \( i=1,2,\dots, n \).
|
||||
In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the
|
||||
vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with.
|
||||
and \( f(x) \) is our objective function while \( g(\lambda_i) \leq b_i \) represents our constraint function.
|
||||
|
||||
<p>
|
||||
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
|
||||
@@ -896,7 +980,7 @@ Convex optimization problems play a central role in applied mathematics and we r
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec22">How do we solve these problems </h2>
|
||||
<h2 id="___sec21">How do we solve these problems? </h2>
|
||||
|
||||
<p>
|
||||
If we use Python as programming language and wish to venture beyond
|
||||
@@ -908,7 +992,8 @@ methods. However, these methods tend to exhibit a rather slow
|
||||
converge. So, welcome to the promised land of quadratic programming.
|
||||
|
||||
<p>
|
||||
The functions we need are contained in the quadratic programming package <b>CVXOPT</b> and we need to import it
|
||||
The functions we need are contained in the quadratic programming package <b>CVXOPT</b> and we need to import it together with <b>numpy</b> as
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
@@ -916,32 +1001,71 @@ The functions we need are contained in the quadratic programming package <b>CVXO
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">cvxopt</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
Let us first set up the standard form the of quadratic programming (QP) equations by defining the problem as
|
||||
This will make our life much easier. You don't need t write your own optimizer.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec22">A simplex example </h2>
|
||||
|
||||
<p>
|
||||
We remind ourselves about the general problem we want to solve
|
||||
<p> <br>
|
||||
$$
|
||||
\mathrm{min}
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
|
||||
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
subject to Gx u. Note that x itself is not provided to the solver, since it is an internal
|
||||
variable being optimized over. In particular, this means that the solver has no explicit knowledge
|
||||
of x itself; everything is implicity defined by the supplied parameters. It is essential
|
||||
that the same variable order is maintained for the relevant parameters (e.g., qi
|
||||
Non-convexity implies the existence of local optima, making it difficult to find global optima.
|
||||
Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
|
||||
&\mathrm{subject to} \\ \nonumber
|
||||
&x, y \geq 0 \\ \nonumber
|
||||
&x+3y \geq 15 \\ \nonumber
|
||||
&2x+5y \leq 100 \\ \nonumber
|
||||
&3x+4y \leq 80. \\ \nonumber
|
||||
\end{align*}
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
|
||||
<p> <br>
|
||||
$$
|
||||
frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix}
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
is clearly positive semi-definite (all eigenvalues larger or equal zero).
|
||||
Finally, the vector \( \boldsymbol{h} \) is defined as
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
collapsed all inequality constraints into a single G matrix of the standard form.
|
||||
Since there are no equality constraints, we do not need to provide the empty A, b. Note
|
||||
that even though y
|
||||
2 did not appear in the original objective, we had to include it with zero
|
||||
coefficients in P because the solver parameters must be defined using the full set of variables.
|
||||
Even if certain variables only appear in constraints, they will still need to be expressed with
|
||||
zero coefficients in the objective parameters, and vice versa.
|
||||
Let us first define the above parameters in Python. CVXOPT supplies its own matrix
|
||||
object; all arguments given to its solvers must be in this matrix type. There are two ways
|
||||
to do this. The first is to define the matrix directly with (potentially nested) lists:
|
||||
from cvxopt import matrix
|
||||
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero
|
||||
The following code does this for us
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
@@ -962,12 +1086,43 @@ h = matrix(numpy.array([<span style="color: #B452CD">0</span>,<span style="color
|
||||
<span style="color: #228B22"># Construct the QP, invoke solver</span>
|
||||
sol = solvers.qp(P,q,G,h)
|
||||
<span style="color: #228B22"># Extract optimal value and solution</span>
|
||||
sol[<span style="color: #a61717; background-color: #e3d2d2">’</span>x<span style="color: #a61717; background-color: #e3d2d2">’</span>] <span style="color: #228B22"># [7.13e-07, 5.00e+00]</span>
|
||||
sol[<span style="color: #a61717; background-color: #e3d2d2">’</span>x<span style="color: #a61717; background-color: #e3d2d2">’</span>]
|
||||
sol[<span style="color: #a61717; background-color: #e3d2d2">’</span>primal objective<span style="color: #a61717; background-color: #e3d2d2">’</span>]
|
||||
</pre></div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec23">Back to the more realistic cases </h2>
|
||||
|
||||
<p>
|
||||
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the <b>slack</b> parameter \( C \) we have
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
|
||||
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
|
||||
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
|
||||
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
|
||||
With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
|
||||
|
||||
<p>
|
||||
<b>code will be added</b>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec24">Multiclass problems and regression with SVMs </h2>
|
||||
This material will be added later.
|
||||
</section>
|
||||
|
||||
|
||||
|
||||
</div> <!-- class="slides" -->
|
||||
</div> <!-- class="reveal" -->
|
||||
|
||||
@@ -55,14 +55,19 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -131,8 +136,14 @@ classification problems) of variables. It is also used for regression
|
||||
problems.
|
||||
|
||||
<p>
|
||||
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
|
||||
We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines.
|
||||
With SVMs we distinguish between hard margin and soft margins. The
|
||||
latter introduces a so-called softening parameter to be discussed
|
||||
below. We distinguish also between linear and non-linear
|
||||
approaches. The latter are the most frequent ones since it is rather
|
||||
unlikely that we can separate classes easily by say straight lines.
|
||||
|
||||
<p>
|
||||
<b>Note: several figures are missing. They will be added shortly. To run the codes, use the jupyter notebook</b>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -672,15 +683,15 @@ we need to introduce for example a polynomial transformation to a two-dimensiona
|
||||
<h2 id="___sec17">The equations </h2>
|
||||
|
||||
<p>
|
||||
Suppose we define a polynomial transformation of degree two (we continue to live in a plane with \( x_1 \) and \( x_2 \) as variables)
|
||||
Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
|
||||
$$
|
||||
z = \phi(x) =\left(1, x_1, x_2, x_1^2, x_2^2, x_1x_2).
|
||||
z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
|
||||
$$
|
||||
|
||||
<p>
|
||||
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{Z}_j,
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
|
||||
$$
|
||||
|
||||
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors
|
||||
@@ -689,37 +700,102 @@ y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i,
|
||||
$$
|
||||
|
||||
from which we also find \( b \).
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec18">Different kernels </h2>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec19">Quadratic coefficient matrix </h2>
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec20">Mercer's theorem </h2>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec21">Mathematical optimization of convex functions </h2>
|
||||
|
||||
<p>
|
||||
A mathematical optimization problem, or just optimization problem, has the form
|
||||
To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
|
||||
$$
|
||||
\mathrm{minimize}\hspace{0.1cm} f(x),
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
|
||||
$$
|
||||
|
||||
subject to some constraints \( g(\lambda_i) \leq b_i \) for say a selected set \( i=1,2,\dots, n \).
|
||||
For the above example, the kernel reads
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
|
||||
$$
|
||||
|
||||
We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec18">The problem to solve </h2>
|
||||
Using our definition of the kernel We can rewrite again the Lagrangian
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
|
||||
$$
|
||||
|
||||
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
|
||||
$$
|
||||
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
|
||||
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
|
||||
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
|
||||
$$
|
||||
|
||||
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
|
||||
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
|
||||
If we add the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
|
||||
|
||||
<p>
|
||||
We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
|
||||
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \).
|
||||
Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into
|
||||
\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec19">Different kernels and Mercer's theorem </h2>
|
||||
|
||||
<p>
|
||||
There are several popular kernels being used. These are
|
||||
|
||||
<ol>
|
||||
<li> Linear: \( K(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{x}^T\boldsymbol{y} \),</li>
|
||||
<li> Polynomial: \( K(\boldsymbol{x},\boldsymbol{y})=(\boldsymbol{x}^T\boldsymbol{y}+\gamma)^d \),</li>
|
||||
<li> Gaussian Radial Basis Function: \( K(\boldsymbol{x},\boldsymbol{y})=\exp{\left(-\gamma\vert\vert\boldsymbol{x}-\boldsymbol{y}\vert\vert^2\right)} \),</li>
|
||||
<li> Tanh: \( K(\boldsymbol{x},\boldsymbol{y})=\tanh{(\boldsymbol{x}^T\boldsymbol{y}+\gamma)} \),</li>
|
||||
</ol>
|
||||
|
||||
and many other ones.
|
||||
|
||||
<p>
|
||||
An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
|
||||
The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then
|
||||
there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space
|
||||
(possibly with much higher dimensions) such that
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
|
||||
$$
|
||||
|
||||
So you can use \( K \) as a kernel since you know \( \phi \) exists, even if
|
||||
you don’t know what \( \phi \) is.
|
||||
Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work
|
||||
well in practice.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec20">Mathematical optimization of convex functions </h2>
|
||||
|
||||
<p>
|
||||
A mathematical (quadratic) optimization problem, or just optimization problem, has the form
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
|
||||
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
subject to some constraints for say a selected set \( i=1,2,\dots, n \).
|
||||
In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the
|
||||
vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with.
|
||||
and \( f(x) \) is our objective function while \( g(\lambda_i) \leq b_i \) represents our constraint function.
|
||||
|
||||
<p>
|
||||
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
|
||||
@@ -731,7 +807,7 @@ Convex optimization problems play a central role in applied mathematics and we r
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec22">How do we solve these problems </h2>
|
||||
<h2 id="___sec21">How do we solve these problems? </h2>
|
||||
|
||||
<p>
|
||||
If we use Python as programming language and wish to venture beyond
|
||||
@@ -743,7 +819,8 @@ methods. However, these methods tend to exhibit a rather slow
|
||||
converge. So, welcome to the promised land of quadratic programming.
|
||||
|
||||
<p>
|
||||
The functions we need are contained in the quadratic programming package <b>CVXOPT</b> and we need to import it
|
||||
The functions we need are contained in the quadratic programming package <b>CVXOPT</b> and we need to import it together with <b>numpy</b> as
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
@@ -751,30 +828,59 @@ The functions we need are contained in the quadratic programming package <b>CVXO
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">cvxopt</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
Let us first set up the standard form the of quadratic programming (QP) equations by defining the problem as
|
||||
This will make our life much easier. You don't need t write your own optimizer.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec22">A simplex example </h2>
|
||||
|
||||
<p>
|
||||
We remind ourselves about the general problem we want to solve
|
||||
$$
|
||||
\mathrm{min}
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
|
||||
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
<p>
|
||||
subject to Gx u. Note that x itself is not provided to the solver, since it is an internal
|
||||
variable being optimized over. In particular, this means that the solver has no explicit knowledge
|
||||
of x itself; everything is implicity defined by the supplied parameters. It is essential
|
||||
that the same variable order is maintained for the relevant parameters (e.g., qi
|
||||
Non-convexity implies the existence of local optima, making it difficult to find global optima.
|
||||
Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
|
||||
&\mathrm{subject to} \\ \nonumber
|
||||
&x, y \geq 0 \\ \nonumber
|
||||
&x+3y \geq 15 \\ \nonumber
|
||||
&2x+5y \leq 100 \\ \nonumber
|
||||
&3x+4y \leq 80. \\ \nonumber
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
|
||||
$$
|
||||
frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
|
||||
$$
|
||||
|
||||
Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
|
||||
$$
|
||||
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
$$
|
||||
|
||||
We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix
|
||||
$$
|
||||
\boldsymbol{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix}
|
||||
$$
|
||||
|
||||
is clearly positive semi-definite (all eigenvalues larger or equal zero).
|
||||
Finally, the vector \( \boldsymbol{h} \) is defined as
|
||||
$$
|
||||
\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
collapsed all inequality constraints into a single G matrix of the standard form.
|
||||
Since there are no equality constraints, we do not need to provide the empty A, b. Note
|
||||
that even though y
|
||||
2 did not appear in the original objective, we had to include it with zero
|
||||
coefficients in P because the solver parameters must be defined using the full set of variables.
|
||||
Even if certain variables only appear in constraints, they will still need to be expressed with
|
||||
zero coefficients in the objective parameters, and vice versa.
|
||||
Let us first define the above parameters in Python. CVXOPT supplies its own matrix
|
||||
object; all arguments given to its solvers must be in this matrix type. There are two ways
|
||||
to do this. The first is to define the matrix directly with (potentially nested) lists:
|
||||
from cvxopt import matrix
|
||||
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero
|
||||
The following code does this for us
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
@@ -795,10 +901,37 @@ h = matrix(numpy.array([<span style="color: #B452CD">0</span>,<span style="color
|
||||
<span style="color: #228B22"># Construct the QP, invoke solver</span>
|
||||
sol = solvers.qp(P,q,G,h)
|
||||
<span style="color: #228B22"># Extract optimal value and solution</span>
|
||||
sol[<span style="color: #a61717; background-color: #e3d2d2">’</span>x<span style="color: #a61717; background-color: #e3d2d2">’</span>] <span style="color: #228B22"># [7.13e-07, 5.00e+00]</span>
|
||||
sol[<span style="color: #a61717; background-color: #e3d2d2">’</span>x<span style="color: #a61717; background-color: #e3d2d2">’</span>]
|
||||
sol[<span style="color: #a61717; background-color: #e3d2d2">’</span>primal objective<span style="color: #a61717; background-color: #e3d2d2">’</span>]
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec23">Back to the more realistic cases </h2>
|
||||
|
||||
<p>
|
||||
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the <b>slack</b> parameter \( C \) we have
|
||||
$$
|
||||
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
|
||||
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
|
||||
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
|
||||
$$
|
||||
|
||||
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
|
||||
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
|
||||
With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
|
||||
|
||||
<p>
|
||||
<b>code will be added</b>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec24">Multiclass problems and regression with SVMs </h2>
|
||||
This material will be added later.
|
||||
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
|
||||
+190
-57
@@ -60,14 +60,19 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
('Soft optmization problem', 2, None, '___sec15'),
|
||||
('Kernels and non-linearity', 2, None, '___sec16'),
|
||||
('The equations', 2, None, '___sec17'),
|
||||
('Different kernels', 2, None, '___sec18'),
|
||||
('Quadratic coefficient matrix', 2, None, '___sec19'),
|
||||
("Mercer's theorem", 2, None, '___sec20'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec19'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec21'),
|
||||
('How do we solve these problems', 2, None, '___sec22')]}
|
||||
'___sec20'),
|
||||
('How do we solve these problems?', 2, None, '___sec21'),
|
||||
('A simplex example', 2, None, '___sec22'),
|
||||
('Back to the more realistic cases', 2, None, '___sec23'),
|
||||
('Multiclass problems and regression with SVMs',
|
||||
2,
|
||||
None,
|
||||
'___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -136,8 +141,14 @@ classification problems) of variables. It is also used for regression
|
||||
problems.
|
||||
|
||||
<p>
|
||||
With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
|
||||
We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines.
|
||||
With SVMs we distinguish between hard margin and soft margins. The
|
||||
latter introduces a so-called softening parameter to be discussed
|
||||
below. We distinguish also between linear and non-linear
|
||||
approaches. The latter are the most frequent ones since it is rather
|
||||
unlikely that we can separate classes easily by say straight lines.
|
||||
|
||||
<p>
|
||||
<b>Note: several figures are missing. They will be added shortly. To run the codes, use the jupyter notebook</b>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -677,15 +688,15 @@ we need to introduce for example a polynomial transformation to a two-dimensiona
|
||||
<h2 id="___sec17">The equations </h2>
|
||||
|
||||
<p>
|
||||
Suppose we define a polynomial transformation of degree two (we continue to live in a plane with \( x_1 \) and \( x_2 \) as variables)
|
||||
Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
|
||||
$$
|
||||
z = \phi(x) =\left(1, x_1, x_2, x_1^2, x_2^2, x_1x_2).
|
||||
z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
|
||||
$$
|
||||
|
||||
<p>
|
||||
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{Z}_j,
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
|
||||
$$
|
||||
|
||||
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors
|
||||
@@ -694,37 +705,102 @@ y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i,
|
||||
$$
|
||||
|
||||
from which we also find \( b \).
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec18">Different kernels </h2>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec19">Quadratic coefficient matrix </h2>
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec20">Mercer's theorem </h2>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec21">Mathematical optimization of convex functions </h2>
|
||||
|
||||
<p>
|
||||
A mathematical optimization problem, or just optimization problem, has the form
|
||||
To compute \( \boldsymbol{z}_i^T\boldsymbol{Z}_j \) we define the kerne \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
|
||||
$$
|
||||
\mathrm{minimize}\hspace{0.1cm} f(x),
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{Z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
|
||||
$$
|
||||
|
||||
subject to some constraints \( g(\lambda_i) \leq b_i \) for say a selected set \( i=1,2,\dots, n \).
|
||||
For the above example, the kernel reads
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.
|
||||
$$
|
||||
|
||||
We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{x}_i^T\boldsymbol{z}_j \) we simply compute the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec18">The problem to solve </h2>
|
||||
Using our definition of the kernel We can rewrite again the Lagrangian
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
|
||||
$$
|
||||
|
||||
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
|
||||
$$
|
||||
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
|
||||
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
|
||||
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
|
||||
$$
|
||||
|
||||
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
|
||||
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
|
||||
If we add the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
|
||||
|
||||
<p>
|
||||
We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
|
||||
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \).
|
||||
Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into
|
||||
\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec19">Different kernels and Mercer's theorem </h2>
|
||||
|
||||
<p>
|
||||
There are several popular kernels being used. These are
|
||||
|
||||
<ol>
|
||||
<li> Linear: \( K(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{x}^T\boldsymbol{y} \),</li>
|
||||
<li> Polynomial: \( K(\boldsymbol{x},\boldsymbol{y})=(\boldsymbol{x}^T\boldsymbol{y}+\gamma)^d \),</li>
|
||||
<li> Gaussian Radial Basis Function: \( K(\boldsymbol{x},\boldsymbol{y})=\exp{\left(-\gamma\vert\vert\boldsymbol{x}-\boldsymbol{y}\vert\vert^2\right)} \),</li>
|
||||
<li> Tanh: \( K(\boldsymbol{x},\boldsymbol{y})=\tanh{(\boldsymbol{x}^T\boldsymbol{y}+\gamma)} \),</li>
|
||||
</ol>
|
||||
|
||||
and many other ones.
|
||||
|
||||
<p>
|
||||
An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
|
||||
The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then
|
||||
there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space
|
||||
(possibly with much higher dimensions) such that
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
|
||||
$$
|
||||
|
||||
So you can use \( K \) as a kernel since you know \( \phi \) exists, even if
|
||||
you don’t know what \( \phi \) is.
|
||||
Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work
|
||||
well in practice.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec20">Mathematical optimization of convex functions </h2>
|
||||
|
||||
<p>
|
||||
A mathematical (quadratic) optimization problem, or just optimization problem, has the form
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
|
||||
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{\lambda}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
subject to some constraints for say a selected set \( i=1,2,\dots, n \).
|
||||
In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the
|
||||
vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with.
|
||||
and \( f(x) \) is our objective function while \( g(\lambda_i) \leq b_i \) represents our constraint function.
|
||||
|
||||
<p>
|
||||
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
|
||||
@@ -736,7 +812,7 @@ Convex optimization problems play a central role in applied mathematics and we r
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec22">How do we solve these problems </h2>
|
||||
<h2 id="___sec21">How do we solve these problems? </h2>
|
||||
|
||||
<p>
|
||||
If we use Python as programming language and wish to venture beyond
|
||||
@@ -748,7 +824,8 @@ methods. However, these methods tend to exhibit a rather slow
|
||||
converge. So, welcome to the promised land of quadratic programming.
|
||||
|
||||
<p>
|
||||
The functions we need are contained in the quadratic programming package <b>CVXOPT</b> and we need to import it
|
||||
The functions we need are contained in the quadratic programming package <b>CVXOPT</b> and we need to import it together with <b>numpy</b> as
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
@@ -756,30 +833,59 @@ The functions we need are contained in the quadratic programming package <b>CVXO
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
Let us first set up the standard form the of quadratic programming (QP) equations by defining the problem as
|
||||
This will make our life much easier. You don't need t write your own optimizer.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec22">A simplex example </h2>
|
||||
|
||||
<p>
|
||||
We remind ourselves about the general problem we want to solve
|
||||
$$
|
||||
\mathrm{min}
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
|
||||
&\mathrm{subject to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \hspace{0.2cm} \boldsymbol{A}\boldsymbol{x}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
<p>
|
||||
subject to Gx u. Note that x itself is not provided to the solver, since it is an internal
|
||||
variable being optimized over. In particular, this means that the solver has no explicit knowledge
|
||||
of x itself; everything is implicity defined by the supplied parameters. It is essential
|
||||
that the same variable order is maintained for the relevant parameters (e.g., qi
|
||||
Non-convexity implies the existence of local optima, making it difficult to find global optima.
|
||||
Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
|
||||
&\mathrm{subject to} \\ \nonumber
|
||||
&x, y \geq 0 \\ \nonumber
|
||||
&x+3y \geq 15 \\ \nonumber
|
||||
&2x+5y \leq 100 \\ \nonumber
|
||||
&3x+4y \leq 80. \\ \nonumber
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
|
||||
$$
|
||||
frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
|
||||
$$
|
||||
|
||||
Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
|
||||
$$
|
||||
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
$$
|
||||
|
||||
We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix
|
||||
$$
|
||||
\boldsymbol{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix}
|
||||
$$
|
||||
|
||||
is clearly positive semi-definite (all eigenvalues larger or equal zero).
|
||||
Finally, the vector \( \boldsymbol{h} \) is defined as
|
||||
$$
|
||||
\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
collapsed all inequality constraints into a single G matrix of the standard form.
|
||||
Since there are no equality constraints, we do not need to provide the empty A, b. Note
|
||||
that even though y
|
||||
2 did not appear in the original objective, we had to include it with zero
|
||||
coefficients in P because the solver parameters must be defined using the full set of variables.
|
||||
Even if certain variables only appear in constraints, they will still need to be expressed with
|
||||
zero coefficients in the objective parameters, and vice versa.
|
||||
Let us first define the above parameters in Python. CVXOPT supplies its own matrix
|
||||
object; all arguments given to its solvers must be in this matrix type. There are two ways
|
||||
to do this. The first is to define the matrix directly with (potentially nested) lists:
|
||||
from cvxopt import matrix
|
||||
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero
|
||||
The following code does this for us
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
@@ -800,10 +906,37 @@ h <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666"
|
||||
<span style="color: #408080; font-style: italic"># Construct the QP, invoke solver</span>
|
||||
sol <span style="color: #666666">=</span> solvers<span style="color: #666666">.</span>qp(P,q,G,h)
|
||||
<span style="color: #408080; font-style: italic"># Extract optimal value and solution</span>
|
||||
sol[’x’] <span style="color: #408080; font-style: italic"># [7.13e-07, 5.00e+00]</span>
|
||||
sol[’x’]
|
||||
sol[’primal objective’]
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec23">Back to the more realistic cases </h2>
|
||||
|
||||
<p>
|
||||
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the <b>slack</b> parameter \( C \) we have
|
||||
$$
|
||||
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
|
||||
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
|
||||
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
|
||||
$$
|
||||
|
||||
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
|
||||
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
|
||||
With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
|
||||
|
||||
<p>
|
||||
<b>code will be added</b>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec24">Multiclass problems and regression with SVMs </h2>
|
||||
This material will be added later.
|
||||
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
|
||||
Binary file not shown.
+298
-43
@@ -34,10 +34,13 @@
|
||||
"classification problems) of variables. It is also used for regression\n",
|
||||
"problems.\n",
|
||||
"\n",
|
||||
"With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.\n",
|
||||
"We distinguish also between linear and non-linear approaches. The latter are the most frequent ones since it is rather unlikely that we can separate classes easily by say straight lines. \n",
|
||||
"\n",
|
||||
"With SVMs we distinguish between hard margin and soft margins. The\n",
|
||||
"latter introduces a so-called softening parameter to be discussed\n",
|
||||
"below. We distinguish also between linear and non-linear\n",
|
||||
"approaches. The latter are the most frequent ones since it is rather\n",
|
||||
"unlikely that we can separate classes easily by say straight lines.\n",
|
||||
"\n",
|
||||
"**Note: several figures are missing. They will be added shortly. To run the codes, use the jupyter notebook**\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Hyperplanes and all that\n",
|
||||
@@ -1080,7 +1083,7 @@
|
||||
"\n",
|
||||
"## The equations\n",
|
||||
"\n",
|
||||
"Suppose we define a polynomial transformation of degree two (we continue to live in a plane with $x_1$ and $x_2$ as variables)"
|
||||
"Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with $x_i$ and $y_i$ as variables)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1088,7 +1091,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"z = \\phi(x) =\\left(1, x_1, x_2, x_1^2, x_2^2, x_1x_2).\n",
|
||||
"z = \\phi(x_i) =\\left(x_i^2, y_i^2, \\sqrt{2}x_iy_i\\right).\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -1104,7 +1107,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{z}_i^T\\boldsymbol{Z}_j,\n",
|
||||
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{z}_j,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -1128,18 +1131,8 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"from which we also find $b$. \n",
|
||||
"\n",
|
||||
"## Different kernels\n",
|
||||
"\n",
|
||||
"## Quadratic coefficient matrix\n",
|
||||
"\n",
|
||||
"<!-- !split -->\n",
|
||||
"## Mercer's theorem\n",
|
||||
"\n",
|
||||
"## Mathematical optimization of convex functions\n",
|
||||
"\n",
|
||||
"A mathematical optimization problem, or just optimization problem, has the form"
|
||||
"from which we also find $b$.\n",
|
||||
"To compute $\\boldsymbol{z}_i^T\\boldsymbol{Z}_j$ we define the kerne $K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)$ as"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1147,7 +1140,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\mathrm{minimize}\\hspace{0.1cm} f(x),\n",
|
||||
"K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=\\boldsymbol{z}_i^T\\boldsymbol{Z}_j= \\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j).\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -1155,10 +1148,153 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"subject to some constraints $g(\\lambda_i) \\leq b_i$ for say a selected set $i=1,2,\\dots, n$.\n",
|
||||
"For the above example, the kernel reads"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=[x_i^2, y_i^2, \\sqrt{2}x_iy_i]^T\\begin{bmatrix} x_j^2 \\\\ y_j^2 \\\\ \\sqrt{2}x_jy_j \\end{bmatrix}=x_i^2x_j^2+2x_ixJy_iy_j+y_i^2y_j^2.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We note that this is nothing but the dot product of the two original vectors $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. Instead of thus computing the product in the Lagrangian of $\\boldsymbol{x}_i^T\\boldsymbol{z}_j$ we simply compute the dot product $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation $(\\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j)$ during the SVM calculations. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## The problem to solve\n",
|
||||
"Using our definition of the kernel We can rewrite again the Lagrangian"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{z}_j,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"subject to the constraints $\\lambda_i\\geq 0$, $\\sum_i\\lambda_iy_i=0$ in terms of a convex optimization problem"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1K(\\boldsymbol{x}_1,\\boldsymbol{x}_1) & y_1y_2K(\\boldsymbol{x}_1,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_1,\\boldsymbol{x}_n) \\\\\n",
|
||||
"y_2y_1\\boldsymbol{x}_2^T\\boldsymbol{x}_1 & y_2y_2\\boldsymbol{x}_2^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_2^T\\boldsymbol{x}_n \\\\\n",
|
||||
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
|
||||
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
|
||||
"y_ny_1K(\\boldsymbol{x}_n,\\boldsymbol{x}_1) & y_ny_2K(\\boldsymbol{x}_n\\boldsymbol{x}_2) & \\dots & \\dots & y_ny_nK(\\boldsymbol{x}_n,\\boldsymbol{x}_n) \\\\\n",
|
||||
"\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{I}\\boldsymbol{\\lambda},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"subject to $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$. Here we defined the vectors $\\boldsymbol{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n",
|
||||
"$\\boldsymbol{y}=[y_1,y_2,\\dots,y_n]$. \n",
|
||||
"If we add the slack constants this leads to the additional constraint $0\\leq \\lambda_i \\leq C$.\n",
|
||||
"\n",
|
||||
"We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\begin{align*}\n",
|
||||
" &\\mathrm{min}_{\\lambda}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{\\lambda}^T\\boldsymbol{P}\\boldsymbol{\\lambda}+\\boldsymbol{q}^T\\boldsymbol{\\lambda},\\\\ \\nonumber\n",
|
||||
" &\\mathrm{subject to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n",
|
||||
"\\end{align*}\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Below we discuss how to solve these equations. Here we note that the matrix $\\boldsymbol{P}$ has matrix elements $p_{ij}=y_iy_jK(\\boldsymbol{x}_i,\\boldsymbol{x}_j)$.\n",
|
||||
"Given a kernel $K$ and the targets $y_i$ this matrix is easy to set up. The constraint $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$ leads to $f=0$ and $\\boldsymbol{A}=\\boldsymbol{y}$. How to set up the matrix $\\boldsymbol{G}$ is discussed later. Here note that the inequalities $0\\leq \\lambda_i \\leq C$ can be split up into\n",
|
||||
"$0\\leq \\lambda_i$ and $\\lambda_i \\leq C$. These two inequalities define then the matrix $\\boldsymbol{G}$ and the vector $\\boldsymbol{h}$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Different kernels and Mercer's theorem\n",
|
||||
"\n",
|
||||
"There are several popular kernels being used. These are\n",
|
||||
"1. Linear: $K(\\boldsymbol{x},\\boldsymbol{y})=\\boldsymbol{x}^T\\boldsymbol{y}$,\n",
|
||||
"\n",
|
||||
"2. Polynomial: $K(\\boldsymbol{x},\\boldsymbol{y})=(\\boldsymbol{x}^T\\boldsymbol{y}+\\gamma)^d$,\n",
|
||||
"\n",
|
||||
"3. Gaussian Radial Basis Function: $K(\\boldsymbol{x},\\boldsymbol{y})=\\exp{\\left(-\\gamma\\vert\\vert\\boldsymbol{x}-\\boldsymbol{y}\\vert\\vert^2\\right)}$,\n",
|
||||
"\n",
|
||||
"4. Tanh: $K(\\boldsymbol{x},\\boldsymbol{y})=\\tanh{(\\boldsymbol{x}^T\\boldsymbol{y}+\\gamma)}$,\n",
|
||||
"\n",
|
||||
"and many other ones.\n",
|
||||
"\n",
|
||||
"An important theorem for us is \"Mercer's theorem\":https://en.wikipedia.org/wiki/Mercer%27s_theorem\". \n",
|
||||
"The theorem states that if a kernel function $K$ is symmetric, continuous and leads to a positive semi-definite matrix $\\boldsymbol{P}$ then\n",
|
||||
"there exists a function $\\phi$ that maps $\\boldsymbol{x}_i$ and $\\boldsymbol{x}_j$ into another space\n",
|
||||
"(possibly with much higher dimensions) such that"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"K(\\boldsymbol{x}_i,\\boldsymbol{x}_j)=\\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j).\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"So you can use $K$ as a kernel since you know $\\phi$ exists, even if\n",
|
||||
"you don’t know what $\\phi$ is. \n",
|
||||
"Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work\n",
|
||||
"well in practice.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Mathematical optimization of convex functions\n",
|
||||
"\n",
|
||||
"A mathematical (quadratic) optimization problem, or just optimization problem, has the form"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\begin{align*}\n",
|
||||
" &\\mathrm{min}_{\\lambda}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{\\lambda}^T\\boldsymbol{P}\\boldsymbol{\\lambda}+\\boldsymbol{q}^T\\boldsymbol{\\lambda},\\\\ \\nonumber\n",
|
||||
" &\\mathrm{subject to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{\\lambda} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\boldsymbol{A}\\boldsymbol{\\lambda}=f.\n",
|
||||
"\\end{align*}\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"subject to some constraints for say a selected set $i=1,2,\\dots, n$.\n",
|
||||
"In our case we are optimizing with respect to the Lagrangian multipliers $\\lambda_i$, and the\n",
|
||||
"vector $\\boldsymbol{\\lambda}=[\\lambda_1, \\lambda_2,\\dots, \\lambda_n]$ is the optimization variable we are dealing with.\n",
|
||||
"and $f(x)$ is our objective function while $g(\\lambda_i) \\leq b_i$ represents our constraint function.\n",
|
||||
"\n",
|
||||
"In our case we are particularly interested in a class of optimization problems called convex optmization problems. \n",
|
||||
"In our disussion on gradient descent methods we discussed at length the definition of a convex function. \n",
|
||||
@@ -1167,7 +1303,7 @@
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## How do we solve these problems\n",
|
||||
"## How do we solve these problems?\n",
|
||||
"\n",
|
||||
"If we use Python as programming language and wish to venture beyond\n",
|
||||
"**scikit-learn**, **tensorflow** and similar software which makes our\n",
|
||||
@@ -1177,7 +1313,7 @@
|
||||
"methods. However, these methods tend to exhibit a rather slow\n",
|
||||
"converge. So, welcome to the promised land of quadratic programming.\n",
|
||||
"\n",
|
||||
"The functions we need are contained in the quadratic programming package **CVXOPT** and we need to import it"
|
||||
"The functions we need are contained in the quadratic programming package **CVXOPT** and we need to import it together with **numpy** as"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1196,7 +1332,12 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let us first set up the standard form the of quadratic programming (QP) equations by defining the problem as"
|
||||
"This will make our life much easier. You don't need t write your own optimizer.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## A simplex example\n",
|
||||
"\n",
|
||||
"We remind ourselves about the general problem we want to solve"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1204,7 +1345,10 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\mathrm{min}\n",
|
||||
"\\begin{align*}\n",
|
||||
" &\\mathrm{min}_{x}\\hspace{0.2cm} \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{P}\\boldsymbol{x}+\\boldsymbol{q}^T\\boldsymbol{x},\\\\ \\nonumber\n",
|
||||
" &\\mathrm{subject to} \\hspace{0.2cm} \\boldsymbol{G}\\boldsymbol{x} \\preceq \\boldsymbol{h} \\hspace{0.2cm} \\boldsymbol{A}\\boldsymbol{x}=f.\n",
|
||||
"\\end{align*}\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -1212,23 +1356,96 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"subject to Gx u. Note that x itself is not provided to the solver, since it is an internal\n",
|
||||
"variable being optimized over. In particular, this means that the solver has no explicit knowledge\n",
|
||||
"of x itself; everything is implicity defined by the supplied parameters. It is essential\n",
|
||||
"that the same variable order is maintained for the relevant parameters (e.g., qi\n",
|
||||
"Non-convexity implies the existence of local optima, making it difficult to find global optima.\n",
|
||||
"\n",
|
||||
"collapsed all inequality constraints into a single G matrix of the standard form.\n",
|
||||
"Since there are no equality constraints, we do not need to provide the empty A, b. Note\n",
|
||||
"that even though y\n",
|
||||
"2 did not appear in the original objective, we had to include it with zero\n",
|
||||
"coefficients in P because the solver parameters must be defined using the full set of variables.\n",
|
||||
"Even if certain variables only appear in constraints, they will still need to be expressed with\n",
|
||||
"zero coefficients in the objective parameters, and vice versa.\n",
|
||||
"Let us first define the above parameters in Python. CVXOPT supplies its own matrix\n",
|
||||
"object; all arguments given to its solvers must be in this matrix type. There are two ways\n",
|
||||
"to do this. The first is to define the matrix directly with (potentially nested) lists:\n",
|
||||
"from cvxopt import matrix"
|
||||
"Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\begin{align*}\n",
|
||||
" &\\mathrm{min}_{x}\\hspace{0.2cm} \\frac{1}{2}x^2+5x+3y \\\\ \\nonumber\n",
|
||||
" &\\mathrm{subject to} \\\\ \\nonumber\n",
|
||||
" &x, y \\geq 0 \\\\ \\nonumber\n",
|
||||
" &x+3y \\geq 15 \\\\ \\nonumber\n",
|
||||
" &2x+5y \\leq 100 \\\\ \\nonumber\n",
|
||||
" &3x+4y \\leq 80. \\\\ \\nonumber\n",
|
||||
"\\end{align*}\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The minimization problem can be rewritten in terms of vectors and matrices as (with $x$ and $y$ being the unknowns)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"frac{1}{2}\\begin{bmatrix} x\\\\ y \\end{bmatrix}^T \\begin{bmatrix} 1 & 0\\\\ 0 & 0 \\end{bmatrix} \\begin{bmatrix} x \\\\ y \\end{bmatrix} + \\begin{bmatrix}3\\\\ 4 \\end{bmatrix}^T \\begin{bmatrix}x \\\\ y \\end{bmatrix}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Similarly, we can now set up the inequalities (we need to change $\\geq$ to $\\leq$ by multiplying with $-1$ on bot sides) as the following matrix-vector equation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\begin{bmatrix} -1 & 0 \\\\ 0 & -1 \\\\ -1 & -3 \\\\ 2 & 5 \\\\ 3 & 4\\end{matrix}\\begin{bmatrix} x \\\\ y\\end{bmatrix} \\preceq \\begin{bmatrix}0 \\\\ 0\\\\ -15 \\\\ 100 \\\\ 80\\end{bmatrix}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We have collapsed all the inequalities into a single matrix $\\boldsymbol{G}$. We see also that our matrix"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{P} =\\begin{bmatrix} 1 & 0\\\\ 0 & 0 \\end{bmatrix}\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"is clearly positive semi-definite (all eigenvalues larger or equal zero). \n",
|
||||
"Finally, the vector $\\boldsymbol{h}$ is defined as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{h} = \\begin{bmatrix}0 \\\\ 0\\\\ -15 \\\\ 100 \\\\ 80\\end{bmatrix}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Since we don't have any equalities the matrix $\\boldsymbol{A}$ is set to zero\n",
|
||||
"The following code does this for us"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1256,9 +1473,47 @@
|
||||
"# Construct the QP, invoke solver\n",
|
||||
"sol = solvers.qp(P,q,G,h)\n",
|
||||
"# Extract optimal value and solution\n",
|
||||
"sol[’x’] # [7.13e-07, 5.00e+00]\n",
|
||||
"sol[’x’] \n",
|
||||
"sol[’primal objective’]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Back to the more realistic cases\n",
|
||||
"\n",
|
||||
"We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the **slack** parameter $C$ we have"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1K(\\boldsymbol{x}_1,\\boldsymbol{x}_1) & y_1y_2K(\\boldsymbol{x}_1,\\boldsymbol{x}_2) & \\dots & \\dots & y_1y_nK(\\boldsymbol{x}_1,\\boldsymbol{x}_n) \\\\\n",
|
||||
"y_2y_1\\boldsymbol{x}_2^T\\boldsymbol{x}_1 & y_2y_2\\boldsymbol{x}_2^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_2^T\\boldsymbol{x}_n \\\\\n",
|
||||
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
|
||||
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
|
||||
"y_ny_1K(\\boldsymbol{x}_n,\\boldsymbol{x}_1) & y_ny_2K(\\boldsymbol{x}_n\\boldsymbol{x}_2) & \\dots & \\dots & y_ny_nK(\\boldsymbol{x}_n,\\boldsymbol{x}_n) \\\\\n",
|
||||
"\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{I}\\boldsymbol{\\lambda},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"subject to $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$. Here we defined the vectors $\\boldsymbol{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n",
|
||||
"$\\boldsymbol{y}=[y_1,y_2,\\dots,y_n]$. \n",
|
||||
"With the slack constants this leads to the additional constraint $0\\leq \\lambda_i \\leq C$.\n",
|
||||
"\n",
|
||||
"**code will be added**\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Multiclass problems and regression with SVMs\n",
|
||||
"This material will be added later."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {},
|
||||
|
||||
Binary file not shown.
@@ -621,7 +621,7 @@ We can rewrite this (see the solutions below) in terms of a convex optimization
|
||||
&\mathrm{subject to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \hspace{0.2cm} \bm{A}\bm{\lambda}=f.
|
||||
\end{align*}
|
||||
!et
|
||||
Below we discuss how to solve these equations. Here we note that the matrix $\bm{P} has matrix elements $p_{ij}=y_iy_jK(\bm{x}_i,\bm{x}_j)$.
|
||||
Below we discuss how to solve these equations. Here we note that the matrix $\bm{P}$ has matrix elements $p_{ij}=y_iy_jK(\bm{x}_i,\bm{x}_j)$.
|
||||
Given a kernel $K$ and the targets $y_i$ this matrix is easy to set up. The constraint $\bm{y}^T\bm{\lambda}=0$ leads to $f=0$ and $\bm{A}=\bm{y}$. How to set up the matrix $\bm{G}$ is discussed later. Here note that the inequalities $0\leq \lambda_i \leq C$ can be split up into
|
||||
$0\leq \lambda_i$ and $\lambda_i \leq C$. These two inequalities define then the matrix $\bm{G}$ and the vector $\bm{h}$.
|
||||
|
||||
@@ -637,7 +637,7 @@ o Tanh: $K(\bm{x},\bm{y})=\tanh{(\bm{x}^T\bm{y}+\gamma)}$,
|
||||
and many other ones.
|
||||
|
||||
An important theorem for us is "Mercer's theorem":https://en.wikipedia.org/wiki/Mercer%27s_theorem".
|
||||
The theorem states that if a kernel function $K$ is symmetric, continuous and leads to a positive definite matrix $\bm{P}$ then
|
||||
The theorem states that if a kernel function $K$ is symmetric, continuous and leads to a positive semi-definite matrix $\bm{P}$ then
|
||||
there exists a function $\phi$ that maps $\bm{x}_i$ and $\bm{x}_j$ into another space
|
||||
(possibly with much higher dimensions) such that
|
||||
!bt
|
||||
@@ -673,7 +673,7 @@ Convex optimization problems play a central role in applied mathematics and we r
|
||||
|
||||
|
||||
!split
|
||||
===== How do we solve these problems =====
|
||||
===== How do we solve these problems? =====
|
||||
|
||||
If we use Python as programming language and wish to venture beyond
|
||||
_scikit-learn_, _tensorflow_ and similar software which makes our
|
||||
@@ -696,10 +696,18 @@ This will make our life much easier. You don't need t write your own optimizer.
|
||||
!split
|
||||
===== A simplex example =====
|
||||
|
||||
We remind ourselves about the general problem we want to solve
|
||||
!bt
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\bm{x}^T\bm{P}\bm{x}+\bm{q}^T\bm{x},\\ \nonumber
|
||||
&\mathrm{subject to} \hspace{0.2cm} \bm{G}\bm{x} \preceq \bm{h} \hspace{0.2cm} \bm{A}\bm{x}=f.
|
||||
\end{align*}
|
||||
!et
|
||||
|
||||
Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem
|
||||
!bt
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
|
||||
&\mathrm{subject to} \\ \nonumber
|
||||
&x, y \geq 0 \\ \nonumber
|
||||
&x+3y \geq 15 \\ \nonumber
|
||||
@@ -713,24 +721,29 @@ The minimization problem can be rewritten in terms of vectors and matrices as (w
|
||||
frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
|
||||
\]
|
||||
!et
|
||||
Similarly, we an now
|
||||
subject to Gx u. Note that x itself is not provided to the solver, since it is an internal
|
||||
variable being optimized over. In particular, this means that the solver has no explicit knowledge
|
||||
of x itself; everything is implicity defined by the supplied parameters. It is essential
|
||||
that the same variable order is maintained for the relevant parameters (e.g., qi
|
||||
Non-convexity implies the existence of local optima, making it difficult to find global optima.
|
||||
Similarly, we can now set up the inequalities (we need to change $\geq$ to $\leq$ by multiplying with $-1$ on bot sides) as the following matrix-vector equation
|
||||
!bt
|
||||
\[
|
||||
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{matrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
\]
|
||||
!et
|
||||
We have collapsed all the inequalities into a single matrix $\bm{G}$. We see also that our matrix
|
||||
!bt
|
||||
\[
|
||||
\bm{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix}
|
||||
\]
|
||||
!et
|
||||
is clearly positive semi-definite (all eigenvalues larger or equal zero).
|
||||
Finally, the vector $\bm{h}$ is defined as
|
||||
!bt
|
||||
\[
|
||||
\bm{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
\]
|
||||
!et
|
||||
|
||||
collapsed all inequality constraints into a single G matrix of the standard form.
|
||||
Since there are no equality constraints, we do not need to provide the empty A, b. Note
|
||||
that even though y
|
||||
2 did not appear in the original objective, we had to include it with zero
|
||||
coefficients in P because the solver parameters must be defined using the full set of variables.
|
||||
Even if certain variables only appear in constraints, they will still need to be expressed with
|
||||
zero coefficients in the objective parameters, and vice versa.
|
||||
Let us first define the above parameters in Python. CVXOPT supplies its own matrix
|
||||
object; all arguments given to its solvers must be in this matrix type. There are two ways
|
||||
to do this. The first is to define the matrix directly with (potentially nested) lists:
|
||||
from cvxopt import matrix
|
||||
|
||||
Since we don't have any equalities the matrix $\bm{A}$ is set to zero
|
||||
The following code does this for us
|
||||
!bc pycod
|
||||
# Import the necessary packages
|
||||
import numpy
|
||||
@@ -749,10 +762,30 @@ h = matrix(numpy.array([0,0,-15,100,80]), tc=’d’)
|
||||
# Construct the QP, invoke solver
|
||||
sol = solvers.qp(P,q,G,h)
|
||||
# Extract optimal value and solution
|
||||
sol[’x’] # [7.13e-07, 5.00e+00]
|
||||
sol[’x’]
|
||||
sol[’primal objective’]
|
||||
!ec
|
||||
|
||||
!split
|
||||
===== Back to the more realistic cases =====
|
||||
|
||||
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducint the _slack_ parameter $C$ we have
|
||||
!bt
|
||||
\[
|
||||
\frac{1}{2} \bm{\lambda}^T\begin{bmatrix} y_1y_1K(\bm{x}_1,\bm{x}_1) & y_1y_2K(\bm{x}_1,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_1,\bm{x}_n) \\
|
||||
y_2y_1\bm{x}_2^T\bm{x}_1 & y_2y_2\bm{x}_2^T\bm{x}_2 & \dots & \dots & y_1y_n\bm{x}_2^T\bm{x}_n \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1K(\bm{x}_n,\bm{x}_1) & y_ny_2K(\bm{x}_n\bm{x}_2) & \dots & \dots & y_ny_nK(\bm{x}_n,\bm{x}_n) \\
|
||||
\end{bmatrix}\bm{\lambda}-\mathbb{I}\bm{\lambda},
|
||||
\]
|
||||
!et
|
||||
subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and
|
||||
$\bm{y}=[y_1,y_2,\dots,y_n]$.
|
||||
With the slack constants this leads to the additional constraint $0\leq \lambda_i \leq C$.
|
||||
|
||||
_code will be added_
|
||||
|
||||
|
||||
!split
|
||||
===== Multiclass problems and regression with SVMs =====
|
||||
|
||||
Reference in New Issue
Block a user