svm update
This commit is contained in:
mhjensen
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -204,7 +202,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._week47-bs008.html">9</a></li>
|
||||
<li><a href="._week47-bs009.html">10</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs001.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -190,7 +188,7 @@ Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) o
|
||||
<li><a href="._week47-bs009.html">10</a></li>
|
||||
<li><a href="._week47-bs010.html">11</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs002.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -184,7 +182,7 @@ We start with our final topic this semester, Support Vector Machines
|
||||
<li><a href="._week47-bs010.html">11</a></li>
|
||||
<li><a href="._week47-bs011.html">12</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs003.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -185,7 +183,7 @@ Friday's lecture is split in two parts. The first lecture is deveoted to a prese
|
||||
<li><a href="._week47-bs011.html">12</a></li>
|
||||
<li><a href="._week47-bs012.html">13</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs004.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -196,7 +194,7 @@ Here are the various projects that will be presented during the first lecture (a
|
||||
<li><a href="._week47-bs012.html">13</a></li>
|
||||
<li><a href="._week47-bs013.html">14</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs005.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -212,7 +210,7 @@ unlikely that we can separate classes easily by say straight lines.
|
||||
<li><a href="._week47-bs013.html">14</a></li>
|
||||
<li><a href="._week47-bs014.html">15</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs006.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -267,7 +265,7 @@ plt<span style="color: #666666">.</span>show()
|
||||
<li><a href="._week47-bs014.html">15</a></li>
|
||||
<li><a href="._week47-bs015.html">16</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs007.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -212,7 +210,7 @@ $$
|
||||
<li><a href="._week47-bs015.html">16</a></li>
|
||||
<li><a href="._week47-bs016.html">17</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs008.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -224,7 +222,7 @@ When we try to separate hyperplanes, if it exists, we can use it to construct a
|
||||
<li><a href="._week47-bs016.html">17</a></li>
|
||||
<li><a href="._week47-bs017.html">18</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs009.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -210,7 +208,7 @@ for our data sample.
|
||||
<li><a href="._week47-bs017.html">18</a></li>
|
||||
<li><a href="._week47-bs018.html">19</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs010.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -206,7 +204,7 @@ $$
|
||||
<li><a href="._week47-bs018.html">19</a></li>
|
||||
<li><a href="._week47-bs019.html">20</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs011.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -209,7 +207,7 @@ $$
|
||||
<li><a href="._week47-bs019.html">20</a></li>
|
||||
<li><a href="._week47-bs020.html">21</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs012.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -202,7 +200,7 @@ where \( \eta \) is our by now well-known learning rate.
|
||||
<li><a href="._week47-bs020.html">21</a></li>
|
||||
<li><a href="._week47-bs021.html">22</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs013.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,17 +159,27 @@ MathJax.Hub.Config({
|
||||
<a name="part0013"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec12" class="anchor">Code Example </h2>
|
||||
<h2 id="___sec12" class="anchor">Can we code this? </h2>
|
||||
|
||||
<p>
|
||||
The equations we discussed above can be coded rather easily (the
|
||||
framework is similar to what we developed for logistic
|
||||
regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way.
|
||||
<p>
|
||||
framework is similar to what we developed for logistic regression). We
|
||||
can set up a simple case with two classes only and we want to find a
|
||||
line which separates them the best possible way.
|
||||
|
||||
<p>
|
||||
There are however problems with this approach, although it looks
|
||||
pretty straightforward to implement. When running a code for such a
|
||||
case we can easily end up with many diffeent lines which separate the
|
||||
two classes.
|
||||
|
||||
<p>
|
||||
For small
|
||||
gaps between the entries, we may also end up needing many iterations
|
||||
before the solutions converge and if the data cannot be separated
|
||||
properly into two distinct classes, we may not experience a converge
|
||||
at all.
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -198,7 +206,7 @@ regression). We are going to set up a simple case with two classes only and we w
|
||||
<li><a href="._week47-bs021.html">22</a></li>
|
||||
<li><a href="._week47-bs022.html">23</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs014.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,18 +159,44 @@ MathJax.Hub.Config({
|
||||
<a name="part0014"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec13" class="anchor">Problems with the Simpler Approach </h2>
|
||||
<h2 id="___sec13" class="anchor">A better approach </h2>
|
||||
|
||||
<p>
|
||||
There are however problems with this approach, although it looks
|
||||
pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes.
|
||||
A better approach is rather to try to define a large margin between
|
||||
the two classes (if they are well separated from the beginning).
|
||||
|
||||
<p>
|
||||
For small
|
||||
gaps between the entries, we may also end up needing many iterations
|
||||
before the solutions converge and if the data cannot be separated
|
||||
properly into two distinct classes, we may not experience a converge
|
||||
at all.
|
||||
Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to
|
||||
\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition
|
||||
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
|
||||
$$
|
||||
|
||||
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.
|
||||
|
||||
<p>
|
||||
We seek thus the largest value \( M \) defined by
|
||||
$$
|
||||
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
|
||||
$$
|
||||
|
||||
or just
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
|
||||
$$
|
||||
|
||||
If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of
|
||||
\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
|
||||
$$
|
||||
|
||||
<p>
|
||||
We have thus defined our margin as the invers of the norm of
|
||||
\( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as
|
||||
possible margin \( M \). Before we proceed, we need to remind ourselves
|
||||
about Lagrangian multipliers.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -200,7 +224,7 @@ at all.
|
||||
<li><a href="._week47-bs022.html">23</a></li>
|
||||
<li><a href="._week47-bs023.html">24</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs015.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,44 +159,52 @@ MathJax.Hub.Config({
|
||||
<a name="part0015"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec14" class="anchor">A better approach </h2>
|
||||
<h2 id="___sec14" class="anchor">A quick Reminder on Lagrangian Multipliers </h2>
|
||||
|
||||
<p>
|
||||
A better approach is rather to try to define a large margin between
|
||||
the two classes (if they are well separated from the beginning).
|
||||
Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an
|
||||
extreme we have
|
||||
$$
|
||||
df=0.
|
||||
$$
|
||||
|
||||
A necessary and sufficient condition is
|
||||
$$
|
||||
\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0,
|
||||
$$
|
||||
|
||||
due to
|
||||
$$
|
||||
df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz.
|
||||
$$
|
||||
|
||||
In many problems the variables \( x,y,z \) are often subject to constraints (such as those above for the margin)
|
||||
so that they are no longer all independent. It is possible at least in principle to use each
|
||||
constraint to eliminate one variable
|
||||
and to proceed with a new and smaller set of independent varables.
|
||||
|
||||
<p>
|
||||
Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to
|
||||
\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition
|
||||
|
||||
The use of so-called Lagrangian multipliers is an alternative technique when the elimination
|
||||
of variables is incovenient or undesirable. Assume that we have an equation of constraint on
|
||||
the variables \( x,y,z \)
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
|
||||
\phi(x,y,z) = 0,
|
||||
$$
|
||||
|
||||
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.
|
||||
|
||||
<p>
|
||||
We seek thus the largest value \( M \) defined by
|
||||
resulting in
|
||||
$$
|
||||
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
|
||||
d\phi = \frac{\partial \phi}{\partial x}dx+\frac{\partial \phi}{\partial y}dy+\frac{\partial \phi}{\partial z}dz =0.
|
||||
$$
|
||||
|
||||
or just
|
||||
Now we cannot set anymore
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
|
||||
\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0,
|
||||
$$
|
||||
|
||||
If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of
|
||||
\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
|
||||
$$
|
||||
|
||||
<p>
|
||||
We have thus defined our margin as the invers of the norm of
|
||||
\( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as
|
||||
possible margin \( M \). Before we proceed, we need to remind ourselves
|
||||
about Lagrangian multipliers.
|
||||
if \( df=0 \) is wanted
|
||||
because there are now only two independent variables! Assume \( x \) and \( y \) are the independent
|
||||
variables.
|
||||
Then \( dz \) is no longer arbitrary.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -226,7 +232,7 @@ about Lagrangian multipliers.
|
||||
<li><a href="._week47-bs023.html">24</a></li>
|
||||
<li><a href="._week47-bs024.html">25</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs016.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,53 +159,46 @@ MathJax.Hub.Config({
|
||||
<a name="part0016"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec15" class="anchor">A quick Reminder on Lagrangian Multipliers </h2>
|
||||
<h2 id="___sec15" class="anchor">Adding the Multiplier </h2>
|
||||
|
||||
<p>
|
||||
Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an
|
||||
extreme we have
|
||||
However, we can add to
|
||||
$$
|
||||
df=0.
|
||||
df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz,
|
||||
$$
|
||||
|
||||
A necessary and sufficient condition is
|
||||
a multiplum of \( d\phi \), viz. \( \lambda d\phi \), resulting in
|
||||
$$
|
||||
\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0,
|
||||
df+\lambda d\phi = (\frac{\partial f}{\partial z}+\lambda
|
||||
\frac{\partial \phi}{\partial x})dx+(\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y})dy+
|
||||
(\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z})dz =0.
|
||||
$$
|
||||
|
||||
due to
|
||||
Our multiplier is chosen so that
|
||||
$$
|
||||
df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz.
|
||||
\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z} =0.
|
||||
$$
|
||||
|
||||
In many problems the variables \( x,y,z \) are often subject to constraints (such as those above for the margin)
|
||||
so that they are no longer all independent. It is possible at least in principle to use each
|
||||
constraint to eliminate one variable
|
||||
and to proceed with a new and smaller set of independent varables.
|
||||
|
||||
<p>
|
||||
The use of so-called Lagrangian multipliers is an alternative technique when the elimination
|
||||
of variables is incovenient or undesirable. Assume that we have an equation of constraint on
|
||||
the variables \( x,y,z \)
|
||||
We need to remember that we took \( dx \) and \( dy \) to be arbitrary and thus we must have
|
||||
$$
|
||||
\phi(x,y,z) = 0,
|
||||
\frac{\partial f}{\partial x}+\lambda\frac{\partial \phi}{\partial x} =0,
|
||||
$$
|
||||
|
||||
resulting in
|
||||
and
|
||||
$$
|
||||
d\phi = \frac{\partial \phi}{\partial x}dx+\frac{\partial \phi}{\partial y}dy+\frac{\partial \phi}{\partial z}dz =0.
|
||||
\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y} =0.
|
||||
$$
|
||||
|
||||
Now we cannot set anymore
|
||||
When all these equations are satisfied, \( df=0 \). We have four unknowns, \( x,y,z \) and
|
||||
\( \lambda \). Actually we want only \( x,y,z \), \( \lambda \) needs not to be determined,
|
||||
it is therefore often called
|
||||
Lagrange's undetermined multiplier.
|
||||
If we have a set of constraints \( \phi_k \) we have the equations
|
||||
$$
|
||||
\frac{\partial f}{\partial x} =\frac{\partial f}{\partial y}=\frac{\partial f}{\partial z}=0,
|
||||
\frac{\partial f}{\partial x_i}+\sum_k\lambda_k\frac{\partial \phi_k}{\partial x_i} =0.
|
||||
$$
|
||||
|
||||
if \( df=0 \) is wanted
|
||||
because there are now only two independent variables! Assume \( x \) and \( y \) are the independent
|
||||
variables.
|
||||
Then \( dz \) is no longer arbitrary.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -234,7 +225,7 @@ Then \( dz \) is no longer arbitrary.
|
||||
<li><a href="._week47-bs024.html">25</a></li>
|
||||
<li><a href="._week47-bs025.html">26</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs017.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,45 +159,43 @@ MathJax.Hub.Config({
|
||||
<a name="part0017"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec16" class="anchor">Adding the Multiplier </h2>
|
||||
<h2 id="___sec16" class="anchor">Setting up the Problem </h2>
|
||||
In order to solve the above problem, we define the following Lagrangian function to be minimized
|
||||
$$
|
||||
{\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right],
|
||||
$$
|
||||
|
||||
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
|
||||
|
||||
<p>
|
||||
However, we can add to
|
||||
Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
|
||||
$$
|
||||
df = \frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz,
|
||||
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
|
||||
$$
|
||||
|
||||
a multiplum of \( d\phi \), viz. \( \lambda d\phi \), resulting in
|
||||
and
|
||||
$$
|
||||
df+\lambda d\phi = (\frac{\partial f}{\partial z}+\lambda
|
||||
\frac{\partial \phi}{\partial x})dx+(\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y})dy+
|
||||
(\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z})dz =0.
|
||||
\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
|
||||
$$
|
||||
|
||||
Our multiplier is chosen so that
|
||||
Inserting these constraints into the equation for \( {\cal L} \) we obtain
|
||||
$$
|
||||
\frac{\partial f}{\partial z}+\lambda\frac{\partial \phi}{\partial z} =0.
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
|
||||
$$
|
||||
|
||||
<p>
|
||||
We need to remember that we took \( dx \) and \( dy \) to be arbitrary and thus we must have
|
||||
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
|
||||
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_self">Karush-Kuhn-Tucker</a> (KKT) condition
|
||||
$$
|
||||
\frac{\partial f}{\partial x}+\lambda\frac{\partial \phi}{\partial x} =0,
|
||||
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
|
||||
$$
|
||||
|
||||
and
|
||||
$$
|
||||
\frac{\partial f}{\partial y}+\lambda\frac{\partial \phi}{\partial y} =0.
|
||||
$$
|
||||
|
||||
When all these equations are satisfied, \( df=0 \). We have four unknowns, \( x,y,z \) and
|
||||
\( \lambda \). Actually we want only \( x,y,z \), \( \lambda \) needs not to be determined,
|
||||
it is therefore often called
|
||||
Lagrange's undetermined multiplier.
|
||||
If we have a set of constraints \( \phi_k \) we have the equations
|
||||
$$
|
||||
\frac{\partial f}{\partial x_i}+\sum_k\lambda_k\frac{\partial \phi_k}{\partial x_i} =0.
|
||||
$$
|
||||
<ol>
|
||||
<li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
|
||||
<li> If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
|
||||
</ol>
|
||||
|
||||
When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -227,7 +223,7 @@ $$
|
||||
<li><a href="._week47-bs025.html">26</a></li>
|
||||
<li><a href="._week47-bs026.html">27</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs018.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,43 +159,26 @@ MathJax.Hub.Config({
|
||||
<a name="part0018"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec17" class="anchor">Setting up the Problem </h2>
|
||||
In order to solve the above problem, we define the following Lagrangian function to be minimized
|
||||
$$
|
||||
{\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right],
|
||||
$$
|
||||
|
||||
where \( \lambda_i \) is a so-called Lagrange multiplier subject to the condition \( \lambda_i \geq 0 \).
|
||||
<h2 id="___sec17" class="anchor">The problem to solve </h2>
|
||||
|
||||
<p>
|
||||
Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
|
||||
$$
|
||||
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
|
||||
$$
|
||||
|
||||
and
|
||||
$$
|
||||
\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
|
||||
$$
|
||||
|
||||
Inserting these constraints into the equation for \( {\cal L} \) we obtain
|
||||
We can rewrite
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
|
||||
$$
|
||||
|
||||
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
|
||||
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_self">Karush-Kuhn-Tucker</a> (KKT) condition
|
||||
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
|
||||
$$
|
||||
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
|
||||
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\
|
||||
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 & \dots & \dots & y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\
|
||||
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
|
||||
$$
|
||||
|
||||
|
||||
<ol>
|
||||
<li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
|
||||
<li> If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
|
||||
</ol>
|
||||
|
||||
When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
|
||||
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
|
||||
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -225,7 +206,7 @@ When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support
|
||||
<li><a href="._week47-bs026.html">27</a></li>
|
||||
<li><a href="._week47-bs027.html">28</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs019.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,26 +159,36 @@ MathJax.Hub.Config({
|
||||
<a name="part0019"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec18" class="anchor">The problem to solve </h2>
|
||||
<h2 id="___sec18" class="anchor">The last steps </h2>
|
||||
|
||||
<p>
|
||||
We can rewrite
|
||||
Solving the above problem, yields the values of \( \lambda_i \).
|
||||
To find the coefficients of your hyperplane we need simply to compute
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
|
||||
\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
|
||||
$$
|
||||
|
||||
and its constraints in terms of a matrix-vector problem where we minimize w.r.t. \( \lambda \) the following problem
|
||||
With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
|
||||
$$
|
||||
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1\boldsymbol{x}_1^T\boldsymbol{x}_1 & y_1y_2\boldsymbol{x}_1^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_1^T\boldsymbol{x}_n \\
|
||||
y_2y_1\boldsymbol{x}_2^T\boldsymbol{x}_1 & y_2y_2\boldsymbol{x}_2^T\boldsymbol{x}_2 & \dots & \dots & y_1y_n\boldsymbol{x}_2^T\boldsymbol{x}_n \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x}_2 & \dots & \dots & y_ny_n\boldsymbol{x}_n^T\boldsymbol{x}_n \\
|
||||
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
|
||||
$$
|
||||
|
||||
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
|
||||
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
|
||||
resulting in
|
||||
$$
|
||||
b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
|
||||
$$
|
||||
|
||||
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
|
||||
$$
|
||||
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right).
|
||||
$$
|
||||
|
||||
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
|
||||
$$
|
||||
y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
|
||||
$$
|
||||
|
||||
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -208,7 +216,7 @@ subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vec
|
||||
<li><a href="._week47-bs027.html">28</a></li>
|
||||
<li><a href="._week47-bs028.html">29</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs020.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,36 +159,37 @@ MathJax.Hub.Config({
|
||||
<a name="part0020"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec19" class="anchor">The last steps </h2>
|
||||
<h2 id="___sec19" class="anchor">A soft classifier </h2>
|
||||
|
||||
<p>
|
||||
Solving the above problem, yields the values of \( \lambda_i \).
|
||||
To find the coefficients of your hyperplane we need simply to compute
|
||||
$$
|
||||
\boldsymbol{w}=\sum_{i} \lambda_iy_i\boldsymbol{x}_i.
|
||||
$$
|
||||
Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
|
||||
|
||||
With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
|
||||
<p>
|
||||
Suppose now that classes overlap in feature space, as shown in the
|
||||
figure here. One way to deal with this problem before we define the
|
||||
so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
|
||||
that we allow some points to be on the wrong side of the margin.
|
||||
|
||||
<p>
|
||||
We introduce thus the so-called <b>slack</b> variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and
|
||||
modify our previous equation
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
|
||||
$$
|
||||
|
||||
resulting in
|
||||
to
|
||||
$$
|
||||
b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i,
|
||||
$$
|
||||
|
||||
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
|
||||
$$
|
||||
b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\boldsymbol{x}_i^T\boldsymbol{x}_j\right).
|
||||
$$
|
||||
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
|
||||
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
|
||||
\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
|
||||
we bound the total amount by which predictions fall on the wrong side of their margins.
|
||||
|
||||
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
|
||||
$$
|
||||
y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
|
||||
$$
|
||||
|
||||
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
|
||||
<p>
|
||||
Misclassifications occur when \( \xi_i > 1 \). Thus bounding the total sum by some value \( C \) bounds in turn the total number of
|
||||
misclassifications.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -218,7 +217,7 @@ Below we discuss how to find the optimal values of \( \lambda_i \). Before we pr
|
||||
<li><a href="._week47-bs028.html">29</a></li>
|
||||
<li><a href="._week47-bs029.html">30</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs021.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,37 +159,56 @@ MathJax.Hub.Config({
|
||||
<a name="part0021"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec20" class="anchor">A soft classifier </h2>
|
||||
<h2 id="___sec20" class="anchor">Soft optmization problem </h2>
|
||||
|
||||
<p>
|
||||
Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
|
||||
This has in turn the consequences that we change our optmization problem to finding the minimum of
|
||||
$$
|
||||
{\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
|
||||
$$
|
||||
|
||||
subject to
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
|
||||
$$
|
||||
|
||||
with the requirement \( \xi_i\geq 0 \).
|
||||
|
||||
<p>
|
||||
Suppose now that classes overlap in feature space, as shown in the
|
||||
figure here. One way to deal with this problem before we define the
|
||||
so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
|
||||
that we allow some points to be on the wrong side of the margin.
|
||||
|
||||
<p>
|
||||
We introduce thus the so-called <b>slack</b> variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and
|
||||
modify our previous equation
|
||||
Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
|
||||
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
|
||||
$$
|
||||
|
||||
to
|
||||
and
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i,
|
||||
\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i,
|
||||
$$
|
||||
|
||||
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
|
||||
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
|
||||
\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
|
||||
we bound the total amount by which predictions fall on the wrong side of their margins.
|
||||
and
|
||||
$$
|
||||
\lambda_i = C-\gamma_i \hspace{0.1cm}\forall i.
|
||||
$$
|
||||
|
||||
<p>
|
||||
Misclassifications occur when \( \xi_i > 1 \). Thus bounding the total sum by some value \( C \) bounds in turn the total number of
|
||||
misclassifications.
|
||||
Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
|
||||
$$
|
||||
|
||||
but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \).
|
||||
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
|
||||
$$
|
||||
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
|
||||
$$
|
||||
|
||||
$$
|
||||
\gamma_i\xi_i = 0,
|
||||
$$
|
||||
|
||||
and
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
|
||||
$$
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -218,8 +235,6 @@ misclassifications.
|
||||
<li><a href="._week47-bs028.html">29</a></li>
|
||||
<li><a href="._week47-bs029.html">30</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs022.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,57 +159,76 @@ MathJax.Hub.Config({
|
||||
<a name="part0022"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec21" class="anchor">Soft optmization problem </h2>
|
||||
<h2 id="___sec21" class="anchor">Kernels and non-linearity </h2>
|
||||
|
||||
<p>
|
||||
This has in turn the consequences that we change our optmization problem to finding the minimum of
|
||||
$$
|
||||
{\cal L}=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-(1-\xi_)\right]+C\sum_{i=1}^n\xi_i-\sum_{i=1}^n\gamma_i\xi_i,
|
||||
$$
|
||||
|
||||
subject to
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
|
||||
$$
|
||||
|
||||
with the requirement \( \xi_i\geq 0 \).
|
||||
The cases we have studied till now, were all characterized by two classes
|
||||
with a close to linear separability. The classifiers we have described
|
||||
so far find linear boundaries in our input feature space. It is
|
||||
possible to make our procedure more flexible by exploring the feature
|
||||
space using other basis expansions such as higher-order polynomials,
|
||||
wavelets, splines etc.
|
||||
|
||||
<p>
|
||||
Taking the derivatives with respect to \( b \) and \( \boldsymbol{w} \) we obtain
|
||||
$$
|
||||
\frac{\partial {\cal L}}{\partial b} = -\sum_{i} \lambda_iy_i=0,
|
||||
$$
|
||||
If our feature space is not easy to separate, as shown in the figure
|
||||
here, we can achieve a better separation by introducing more complex
|
||||
basis functions. The ideal would be, as shown in the next figure, to, via a specific transformation to
|
||||
obtain a separation between the classes which is almost linear.
|
||||
|
||||
and
|
||||
$$
|
||||
\frac{\partial {\cal L}}{\partial \boldsymbol{w}} = 0 = \boldsymbol{w}-\sum_{i} \lambda_iy_i\boldsymbol{x}_i,
|
||||
$$
|
||||
<p>
|
||||
The change of basis, from \( x\rightarrow z=\phi(x) \) leads to the same type of equations to be solved, except that
|
||||
we need to introduce for example a polynomial transformation to a two-dimensional training set.
|
||||
|
||||
and
|
||||
$$
|
||||
\lambda_i = C-\gamma_i \hspace{0.1cm}\forall i.
|
||||
$$
|
||||
<p>
|
||||
|
||||
Inserting these constraints into the equation for \( {\cal L} \) we obtain the same equation as before
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j,
|
||||
$$
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">os</span>
|
||||
|
||||
but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \).
|
||||
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
|
||||
$$
|
||||
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
|
||||
$$
|
||||
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">42</span>)
|
||||
|
||||
$$
|
||||
\gamma_i\xi_i = 0,
|
||||
$$
|
||||
<span style="color: #408080; font-style: italic"># To plot pretty figures</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">'axes.labelsize'</span>] <span style="color: #666666">=</span> <span style="color: #666666">14</span>
|
||||
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">'xtick.labelsize'</span>] <span style="color: #666666">=</span> <span style="color: #666666">12</span>
|
||||
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">'ytick.labelsize'</span>] <span style="color: #666666">=</span> <span style="color: #666666">12</span>
|
||||
|
||||
and
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
|
||||
$$
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
|
||||
|
||||
|
||||
|
||||
X1D <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">-4</span>, <span style="color: #666666">4</span>, <span style="color: #666666">9</span>)<span style="color: #666666">.</span>reshape(<span style="color: #666666">-1</span>, <span style="color: #666666">1</span>)
|
||||
X2D <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[X1D, X1D<span style="color: #666666">**2</span>]
|
||||
y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">0</span>, <span style="color: #666666">0</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">0</span>])
|
||||
|
||||
plt<span style="color: #666666">.</span>figure(figsize<span style="color: #666666">=</span>(<span style="color: #666666">11</span>, <span style="color: #666666">4</span>))
|
||||
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">121</span>)
|
||||
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">'both'</span>)
|
||||
plt<span style="color: #666666">.</span>axhline(y<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X1D[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==0</span>], np<span style="color: #666666">.</span>zeros(<span style="color: #666666">4</span>), <span style="color: #BA2121">"bs"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X1D[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==1</span>], np<span style="color: #666666">.</span>zeros(<span style="color: #666666">5</span>), <span style="color: #BA2121">"g^"</span>)
|
||||
plt<span style="color: #666666">.</span>gca()<span style="color: #666666">.</span>get_yaxis()<span style="color: #666666">.</span>set_ticks([])
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r"$x_1$"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">-4.5</span>, <span style="color: #666666">4.5</span>, <span style="color: #666666">-0.2</span>, <span style="color: #666666">0.2</span>])
|
||||
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">122</span>)
|
||||
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">'both'</span>)
|
||||
plt<span style="color: #666666">.</span>axhline(y<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
||||
plt<span style="color: #666666">.</span>axvline(x<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X2D[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==0</span>], X2D[:, <span style="color: #666666">1</span>][y<span style="color: #666666">==0</span>], <span style="color: #BA2121">"bs"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X2D[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==1</span>], X2D[:, <span style="color: #666666">1</span>][y<span style="color: #666666">==1</span>], <span style="color: #BA2121">"g^"</span>)
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r"$x_1$"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">r"$x_2$"</span>, fontsize<span style="color: #666666">=20</span>, rotation<span style="color: #666666">=0</span>)
|
||||
plt<span style="color: #666666">.</span>gca()<span style="color: #666666">.</span>get_yaxis()<span style="color: #666666">.</span>set_ticks([<span style="color: #666666">0</span>, <span style="color: #666666">4</span>, <span style="color: #666666">8</span>, <span style="color: #666666">12</span>, <span style="color: #666666">16</span>])
|
||||
plt<span style="color: #666666">.</span>plot([<span style="color: #666666">-4.5</span>, <span style="color: #666666">4.5</span>], [<span style="color: #666666">6.5</span>, <span style="color: #666666">6.5</span>], <span style="color: #BA2121">"r--"</span>, linewidth<span style="color: #666666">=3</span>)
|
||||
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">-4.5</span>, <span style="color: #666666">4.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">17</span>])
|
||||
plt<span style="color: #666666">.</span>subplots_adjust(right<span style="color: #666666">=1</span>)
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -236,7 +253,6 @@ $$
|
||||
<li><a href="._week47-bs028.html">29</a></li>
|
||||
<li><a href="._week47-bs029.html">30</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs023.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,76 +159,48 @@ MathJax.Hub.Config({
|
||||
<a name="part0023"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec22" class="anchor">Kernels and non-linearity </h2>
|
||||
<h2 id="___sec22" class="anchor">The equations </h2>
|
||||
|
||||
<p>
|
||||
The cases we have studied till now, were all characterized by two classes
|
||||
with a close to linear separability. The classifiers we have described
|
||||
so far find linear boundaries in our input feature space. It is
|
||||
possible to make our procedure more flexible by exploring the feature
|
||||
space using other basis expansions such as higher-order polynomials,
|
||||
wavelets, splines etc.
|
||||
Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
|
||||
$$
|
||||
z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
|
||||
$$
|
||||
|
||||
<p>
|
||||
If our feature space is not easy to separate, as shown in the figure
|
||||
here, we can achieve a better separation by introducing more complex
|
||||
basis functions. The ideal would be, as shown in the next figure, to, via a specific transformation to
|
||||
obtain a separation between the classes which is almost linear.
|
||||
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{z}_j,
|
||||
$$
|
||||
|
||||
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i,
|
||||
$$
|
||||
|
||||
from which we also find \( b \).
|
||||
To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kernel \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
|
||||
$$
|
||||
|
||||
For the above example, the kernel reads
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2.
|
||||
$$
|
||||
|
||||
<p>
|
||||
The change of basis, from \( x\rightarrow z=\phi(x) \) leads to the same type of equations to be solved, except that
|
||||
we need to introduce for example a polynomial transformation to a two-dimensional training set.
|
||||
We note that this is nothing but the dot product of the two original
|
||||
vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the
|
||||
product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute
|
||||
the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \).
|
||||
|
||||
<p>
|
||||
This leads to the so-called
|
||||
kernel trick and the result leads to the same as if we went through
|
||||
the trouble of performing the transformation
|
||||
\( \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">os</span>
|
||||
|
||||
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">42</span>)
|
||||
|
||||
<span style="color: #408080; font-style: italic"># To plot pretty figures</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">'axes.labelsize'</span>] <span style="color: #666666">=</span> <span style="color: #666666">14</span>
|
||||
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">'xtick.labelsize'</span>] <span style="color: #666666">=</span> <span style="color: #666666">12</span>
|
||||
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">'ytick.labelsize'</span>] <span style="color: #666666">=</span> <span style="color: #666666">12</span>
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
|
||||
|
||||
|
||||
|
||||
X1D <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">-4</span>, <span style="color: #666666">4</span>, <span style="color: #666666">9</span>)<span style="color: #666666">.</span>reshape(<span style="color: #666666">-1</span>, <span style="color: #666666">1</span>)
|
||||
X2D <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[X1D, X1D<span style="color: #666666">**2</span>]
|
||||
y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">0</span>, <span style="color: #666666">0</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">0</span>])
|
||||
|
||||
plt<span style="color: #666666">.</span>figure(figsize<span style="color: #666666">=</span>(<span style="color: #666666">11</span>, <span style="color: #666666">4</span>))
|
||||
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">121</span>)
|
||||
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">'both'</span>)
|
||||
plt<span style="color: #666666">.</span>axhline(y<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X1D[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==0</span>], np<span style="color: #666666">.</span>zeros(<span style="color: #666666">4</span>), <span style="color: #BA2121">"bs"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X1D[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==1</span>], np<span style="color: #666666">.</span>zeros(<span style="color: #666666">5</span>), <span style="color: #BA2121">"g^"</span>)
|
||||
plt<span style="color: #666666">.</span>gca()<span style="color: #666666">.</span>get_yaxis()<span style="color: #666666">.</span>set_ticks([])
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r"$x_1$"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">-4.5</span>, <span style="color: #666666">4.5</span>, <span style="color: #666666">-0.2</span>, <span style="color: #666666">0.2</span>])
|
||||
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">122</span>)
|
||||
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">'both'</span>)
|
||||
plt<span style="color: #666666">.</span>axhline(y<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
||||
plt<span style="color: #666666">.</span>axvline(x<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X2D[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==0</span>], X2D[:, <span style="color: #666666">1</span>][y<span style="color: #666666">==0</span>], <span style="color: #BA2121">"bs"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X2D[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==1</span>], X2D[:, <span style="color: #666666">1</span>][y<span style="color: #666666">==1</span>], <span style="color: #BA2121">"g^"</span>)
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r"$x_1$"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">r"$x_2$"</span>, fontsize<span style="color: #666666">=20</span>, rotation<span style="color: #666666">=0</span>)
|
||||
plt<span style="color: #666666">.</span>gca()<span style="color: #666666">.</span>get_yaxis()<span style="color: #666666">.</span>set_ticks([<span style="color: #666666">0</span>, <span style="color: #666666">4</span>, <span style="color: #666666">8</span>, <span style="color: #666666">12</span>, <span style="color: #666666">16</span>])
|
||||
plt<span style="color: #666666">.</span>plot([<span style="color: #666666">-4.5</span>, <span style="color: #666666">4.5</span>], [<span style="color: #666666">6.5</span>, <span style="color: #666666">6.5</span>], <span style="color: #BA2121">"r--"</span>, linewidth<span style="color: #666666">=3</span>)
|
||||
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">-4.5</span>, <span style="color: #666666">4.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">17</span>])
|
||||
plt<span style="color: #666666">.</span>subplots_adjust(right<span style="color: #666666">=1</span>)
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -254,7 +224,6 @@ plt<span style="color: #666666">.</span>show()
|
||||
<li><a href="._week47-bs028.html">29</a></li>
|
||||
<li><a href="._week47-bs029.html">30</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs024.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,47 +159,38 @@ MathJax.Hub.Config({
|
||||
<a name="part0024"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec23" class="anchor">The equations </h2>
|
||||
<h2 id="___sec23" class="anchor">The problem to solve </h2>
|
||||
Using our definition of the kernel We can rewrite again the Lagrangian
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
|
||||
$$
|
||||
|
||||
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
|
||||
$$
|
||||
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
|
||||
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
|
||||
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
|
||||
$$
|
||||
|
||||
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
|
||||
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
|
||||
If we add the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
|
||||
|
||||
<p>
|
||||
Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
|
||||
We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type
|
||||
$$
|
||||
z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
|
||||
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
<p>
|
||||
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{z}_j,
|
||||
$$
|
||||
|
||||
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors
|
||||
$$
|
||||
y_i(\boldsymbol{w}^T\boldsymbol{z}_i+b)= 1 \hspace{0.1cm}\forall i,
|
||||
$$
|
||||
|
||||
from which we also find \( b \).
|
||||
To compute \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we define the kernel \( K(\boldsymbol{x}_i,\boldsymbol{x}_j) \) as
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\boldsymbol{z}_i^T\boldsymbol{z}_j= \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
|
||||
$$
|
||||
|
||||
For the above example, the kernel reads
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2.
|
||||
$$
|
||||
|
||||
<p>
|
||||
We note that this is nothing but the dot product of the two original
|
||||
vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the
|
||||
product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute
|
||||
the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \).
|
||||
|
||||
<p>
|
||||
This leads to the so-called
|
||||
kernel trick and the result leads to the same as if we went through
|
||||
the trouble of performing the transformation
|
||||
\( \phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations.
|
||||
Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \).
|
||||
Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into
|
||||
\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -225,7 +214,6 @@ the trouble of performing the transformation
|
||||
<li><a href="._week47-bs028.html">29</a></li>
|
||||
<li><a href="._week47-bs029.html">30</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs025.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,38 +159,40 @@ MathJax.Hub.Config({
|
||||
<a name="part0025"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec24" class="anchor">The problem to solve </h2>
|
||||
Using our definition of the kernel We can rewrite again the Lagrangian
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
|
||||
$$
|
||||
|
||||
subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) in terms of a convex optimization problem
|
||||
$$
|
||||
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
|
||||
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
|
||||
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
|
||||
$$
|
||||
|
||||
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
|
||||
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
|
||||
If we add the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
|
||||
<h2 id="___sec24" class="anchor">Different kernels and Mercer's theorem </h2>
|
||||
|
||||
<p>
|
||||
We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type
|
||||
There are several popular kernels being used. These are
|
||||
|
||||
<ol>
|
||||
<li> Linear: \( K(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{x}^T\boldsymbol{y} \),</li>
|
||||
<li> Polynomial: \( K(\boldsymbol{x},\boldsymbol{y})=(\boldsymbol{x}^T\boldsymbol{y}+\gamma)^d \),</li>
|
||||
<li> Gaussian Radial Basis Function: \( K(\boldsymbol{x},\boldsymbol{y})=\exp{\left(-\gamma\vert\vert\boldsymbol{x}-\boldsymbol{y}\vert\vert^2\right)} \),</li>
|
||||
<li> Tanh: \( K(\boldsymbol{x},\boldsymbol{y})=\tanh{(\boldsymbol{x}^T\boldsymbol{y}+\gamma)} \),</li>
|
||||
</ol>
|
||||
|
||||
and many other ones.
|
||||
|
||||
<p>
|
||||
An important theorem for us is <a href="https://en.wikipedia.org/wiki/Mercer%27s_theorem" target="_self">Mercer's
|
||||
theorem</a>. The
|
||||
theorem states that if a kernel function \( K \) is symmetric, continuous
|
||||
and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there
|
||||
exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into
|
||||
another space (possibly with much higher dimensions) such that
|
||||
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
|
||||
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \hspace{0.2cm} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
|
||||
\end{align*}
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
|
||||
$$
|
||||
|
||||
Below we discuss how to solve these equations. Here we note that the matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \).
|
||||
Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up. The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \). How to set up the matrix \( \boldsymbol{G} \) is discussed later. Here note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into
|
||||
\( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).
|
||||
<p>
|
||||
So you can use \( K \) as a kernel since you know \( \phi \) exists, even if
|
||||
you don’t know what \( \phi \) is.
|
||||
|
||||
<p>
|
||||
Note that some frequently used kernels (such as the Sigmoid kernel)
|
||||
don’t respect all of Mercer’s conditions, yet they generally work well
|
||||
in practice.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -215,7 +215,6 @@ Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up.
|
||||
<li><a href="._week47-bs028.html">29</a></li>
|
||||
<li><a href="._week47-bs029.html">30</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs026.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,41 +159,199 @@ MathJax.Hub.Config({
|
||||
<a name="part0026"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec25" class="anchor">Different kernels and Mercer's theorem </h2>
|
||||
|
||||
<h2 id="___sec25" class="anchor">The moons example </h2>
|
||||
<p>
|
||||
There are several popular kernels being used. These are
|
||||
|
||||
<ol>
|
||||
<li> Linear: \( K(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{x}^T\boldsymbol{y} \),</li>
|
||||
<li> Polynomial: \( K(\boldsymbol{x},\boldsymbol{y})=(\boldsymbol{x}^T\boldsymbol{y}+\gamma)^d \),</li>
|
||||
<li> Gaussian Radial Basis Function: \( K(\boldsymbol{x},\boldsymbol{y})=\exp{\left(-\gamma\vert\vert\boldsymbol{x}-\boldsymbol{y}\vert\vert^2\right)} \),</li>
|
||||
<li> Tanh: \( K(\boldsymbol{x},\boldsymbol{y})=\tanh{(\boldsymbol{x}^T\boldsymbol{y}+\gamma)} \),</li>
|
||||
</ol>
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">__future__</span> <span style="color: #008000; font-weight: bold">import</span> division, print_function, unicode_literals
|
||||
|
||||
and many other ones.
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">42</span>)
|
||||
|
||||
<p>
|
||||
An important theorem for us is <a href="https://en.wikipedia.org/wiki/Mercer%27s_theorem" target="_self">Mercer's
|
||||
theorem</a>. The
|
||||
theorem states that if a kernel function \( K \) is symmetric, continuous
|
||||
and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there
|
||||
exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into
|
||||
another space (possibly with much higher dimensions) such that
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">'axes.labelsize'</span>] <span style="color: #666666">=</span> <span style="color: #666666">14</span>
|
||||
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">'xtick.labelsize'</span>] <span style="color: #666666">=</span> <span style="color: #666666">12</span>
|
||||
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">'ytick.labelsize'</span>] <span style="color: #666666">=</span> <span style="color: #666666">12</span>
|
||||
|
||||
$$
|
||||
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
|
||||
$$
|
||||
|
||||
<p>
|
||||
So you can use \( K \) as a kernel since you know \( \phi \) exists, even if
|
||||
you don’t know what \( \phi \) is.
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
|
||||
|
||||
<p>
|
||||
Note that some frequently used kernels (such as the Sigmoid kernel)
|
||||
don’t respect all of Mercer’s conditions, yet they generally work well
|
||||
in practice.
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVC
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> make_moons
|
||||
X, y <span style="color: #666666">=</span> make_moons(n_samples<span style="color: #666666">=100</span>, noise<span style="color: #666666">=0.15</span>, random_state<span style="color: #666666">=42</span>)
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">plot_dataset</span>(X, y, axes):
|
||||
plt<span style="color: #666666">.</span>plot(X[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==0</span>], X[:, <span style="color: #666666">1</span>][y<span style="color: #666666">==0</span>], <span style="color: #BA2121">"bs"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==1</span>], X[:, <span style="color: #666666">1</span>][y<span style="color: #666666">==1</span>], <span style="color: #BA2121">"g^"</span>)
|
||||
plt<span style="color: #666666">.</span>axis(axes)
|
||||
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">'both'</span>)
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r"$x_1$"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">r"$x_2$"</span>, fontsize<span style="color: #666666">=20</span>, rotation<span style="color: #666666">=0</span>)
|
||||
|
||||
plot_dataset(X, y, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> make_moons
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> PolynomialFeatures
|
||||
|
||||
polynomial_svm_clf <span style="color: #666666">=</span> Pipeline([
|
||||
(<span style="color: #BA2121">"poly_features"</span>, PolynomialFeatures(degree<span style="color: #666666">=3</span>)),
|
||||
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
||||
(<span style="color: #BA2121">"svm_clf"</span>, LinearSVC(C<span style="color: #666666">=10</span>, loss<span style="color: #666666">=</span><span style="color: #BA2121">"hinge"</span>, random_state<span style="color: #666666">=42</span>))
|
||||
])
|
||||
|
||||
polynomial_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">plot_predictions</span>(clf, axes):
|
||||
x0s <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(axes[<span style="color: #666666">0</span>], axes[<span style="color: #666666">1</span>], <span style="color: #666666">100</span>)
|
||||
x1s <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(axes[<span style="color: #666666">2</span>], axes[<span style="color: #666666">3</span>], <span style="color: #666666">100</span>)
|
||||
x0, x1 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>meshgrid(x0s, x1s)
|
||||
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[x0<span style="color: #666666">.</span>ravel(), x1<span style="color: #666666">.</span>ravel()]
|
||||
y_pred <span style="color: #666666">=</span> clf<span style="color: #666666">.</span>predict(X)<span style="color: #666666">.</span>reshape(x0<span style="color: #666666">.</span>shape)
|
||||
y_decision <span style="color: #666666">=</span> clf<span style="color: #666666">.</span>decision_function(X)<span style="color: #666666">.</span>reshape(x0<span style="color: #666666">.</span>shape)
|
||||
plt<span style="color: #666666">.</span>contourf(x0, x1, y_pred, cmap<span style="color: #666666">=</span>plt<span style="color: #666666">.</span>cm<span style="color: #666666">.</span>brg, alpha<span style="color: #666666">=0.2</span>)
|
||||
plt<span style="color: #666666">.</span>contourf(x0, x1, y_decision, cmap<span style="color: #666666">=</span>plt<span style="color: #666666">.</span>cm<span style="color: #666666">.</span>brg, alpha<span style="color: #666666">=0.1</span>)
|
||||
|
||||
plot_predictions(polynomial_svm_clf, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plot_dataset(X, y, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
|
||||
|
||||
poly_kernel_svm_clf <span style="color: #666666">=</span> Pipeline([
|
||||
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
||||
(<span style="color: #BA2121">"svm_clf"</span>, SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">"poly"</span>, degree<span style="color: #666666">=3</span>, coef0<span style="color: #666666">=1</span>, C<span style="color: #666666">=5</span>))
|
||||
])
|
||||
poly_kernel_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
||||
|
||||
poly100_kernel_svm_clf <span style="color: #666666">=</span> Pipeline([
|
||||
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
||||
(<span style="color: #BA2121">"svm_clf"</span>, SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">"poly"</span>, degree<span style="color: #666666">=10</span>, coef0<span style="color: #666666">=100</span>, C<span style="color: #666666">=5</span>))
|
||||
])
|
||||
poly100_kernel_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
||||
|
||||
plt<span style="color: #666666">.</span>figure(figsize<span style="color: #666666">=</span>(<span style="color: #666666">11</span>, <span style="color: #666666">4</span>))
|
||||
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">121</span>)
|
||||
plot_predictions(poly_kernel_svm_clf, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plot_dataset(X, y, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">r"$d=3, r=1, C=5$"</span>, fontsize<span style="color: #666666">=18</span>)
|
||||
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">122</span>)
|
||||
plot_predictions(poly100_kernel_svm_clf, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plot_dataset(X, y, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">r"$d=10, r=100, C=5$"</span>, fontsize<span style="color: #666666">=18</span>)
|
||||
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">gaussian_rbf</span>(x, landmark, gamma):
|
||||
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>gamma <span style="color: #666666">*</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>norm(x <span style="color: #666666">-</span> landmark, axis<span style="color: #666666">=1</span>)<span style="color: #666666">**2</span>)
|
||||
|
||||
gamma <span style="color: #666666">=</span> <span style="color: #666666">0.3</span>
|
||||
|
||||
x1s <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">-4.5</span>, <span style="color: #666666">4.5</span>, <span style="color: #666666">200</span>)<span style="color: #666666">.</span>reshape(<span style="color: #666666">-1</span>, <span style="color: #666666">1</span>)
|
||||
x2s <span style="color: #666666">=</span> gaussian_rbf(x1s, <span style="color: #666666">-2</span>, gamma)
|
||||
x3s <span style="color: #666666">=</span> gaussian_rbf(x1s, <span style="color: #666666">1</span>, gamma)
|
||||
|
||||
XK <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[gaussian_rbf(X1D, <span style="color: #666666">-2</span>, gamma), gaussian_rbf(X1D, <span style="color: #666666">1</span>, gamma)]
|
||||
yk <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">0</span>, <span style="color: #666666">0</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">0</span>])
|
||||
|
||||
plt<span style="color: #666666">.</span>figure(figsize<span style="color: #666666">=</span>(<span style="color: #666666">11</span>, <span style="color: #666666">4</span>))
|
||||
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">121</span>)
|
||||
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">'both'</span>)
|
||||
plt<span style="color: #666666">.</span>axhline(y<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
||||
plt<span style="color: #666666">.</span>scatter(x<span style="color: #666666">=</span>[<span style="color: #666666">-2</span>, <span style="color: #666666">1</span>], y<span style="color: #666666">=</span>[<span style="color: #666666">0</span>, <span style="color: #666666">0</span>], s<span style="color: #666666">=150</span>, alpha<span style="color: #666666">=0.5</span>, c<span style="color: #666666">=</span><span style="color: #BA2121">"red"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X1D[:, <span style="color: #666666">0</span>][yk<span style="color: #666666">==0</span>], np<span style="color: #666666">.</span>zeros(<span style="color: #666666">4</span>), <span style="color: #BA2121">"bs"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X1D[:, <span style="color: #666666">0</span>][yk<span style="color: #666666">==1</span>], np<span style="color: #666666">.</span>zeros(<span style="color: #666666">5</span>), <span style="color: #BA2121">"g^"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(x1s, x2s, <span style="color: #BA2121">"g--"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(x1s, x3s, <span style="color: #BA2121">"b:"</span>)
|
||||
plt<span style="color: #666666">.</span>gca()<span style="color: #666666">.</span>get_yaxis()<span style="color: #666666">.</span>set_ticks([<span style="color: #666666">0</span>, <span style="color: #666666">0.25</span>, <span style="color: #666666">0.5</span>, <span style="color: #666666">0.75</span>, <span style="color: #666666">1</span>])
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r"$x_1$"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">r"Similarity"</span>, fontsize<span style="color: #666666">=14</span>)
|
||||
plt<span style="color: #666666">.</span>annotate(<span style="color: #BA2121">r'$\mathbf</span><span style="color: #BB6688; font-weight: bold">{x}</span><span style="color: #BA2121">$'</span>,
|
||||
xy<span style="color: #666666">=</span>(X1D[<span style="color: #666666">3</span>, <span style="color: #666666">0</span>], <span style="color: #666666">0</span>),
|
||||
xytext<span style="color: #666666">=</span>(<span style="color: #666666">-0.5</span>, <span style="color: #666666">0.20</span>),
|
||||
ha<span style="color: #666666">=</span><span style="color: #BA2121">"center"</span>,
|
||||
arrowprops<span style="color: #666666">=</span><span style="color: #008000">dict</span>(facecolor<span style="color: #666666">=</span><span style="color: #BA2121">'black'</span>, shrink<span style="color: #666666">=0.1</span>),
|
||||
fontsize<span style="color: #666666">=18</span>,
|
||||
)
|
||||
plt<span style="color: #666666">.</span>text(<span style="color: #666666">-2</span>, <span style="color: #666666">0.9</span>, <span style="color: #BA2121">"$x_2$"</span>, ha<span style="color: #666666">=</span><span style="color: #BA2121">"center"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>text(<span style="color: #666666">1</span>, <span style="color: #666666">0.9</span>, <span style="color: #BA2121">"$x_3$"</span>, ha<span style="color: #666666">=</span><span style="color: #BA2121">"center"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">-4.5</span>, <span style="color: #666666">4.5</span>, <span style="color: #666666">-0.1</span>, <span style="color: #666666">1.1</span>])
|
||||
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">122</span>)
|
||||
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">'both'</span>)
|
||||
plt<span style="color: #666666">.</span>axhline(y<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
||||
plt<span style="color: #666666">.</span>axvline(x<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
||||
plt<span style="color: #666666">.</span>plot(XK[:, <span style="color: #666666">0</span>][yk<span style="color: #666666">==0</span>], XK[:, <span style="color: #666666">1</span>][yk<span style="color: #666666">==0</span>], <span style="color: #BA2121">"bs"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(XK[:, <span style="color: #666666">0</span>][yk<span style="color: #666666">==1</span>], XK[:, <span style="color: #666666">1</span>][yk<span style="color: #666666">==1</span>], <span style="color: #BA2121">"g^"</span>)
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r"$x_2$"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">r"$x_3$ "</span>, fontsize<span style="color: #666666">=20</span>, rotation<span style="color: #666666">=0</span>)
|
||||
plt<span style="color: #666666">.</span>annotate(<span style="color: #BA2121">r'$\phi\left(\mathbf</span><span style="color: #BB6688; font-weight: bold">{x}</span><span style="color: #BA2121">\right)$'</span>,
|
||||
xy<span style="color: #666666">=</span>(XK[<span style="color: #666666">3</span>, <span style="color: #666666">0</span>], XK[<span style="color: #666666">3</span>, <span style="color: #666666">1</span>]),
|
||||
xytext<span style="color: #666666">=</span>(<span style="color: #666666">0.65</span>, <span style="color: #666666">0.50</span>),
|
||||
ha<span style="color: #666666">=</span><span style="color: #BA2121">"center"</span>,
|
||||
arrowprops<span style="color: #666666">=</span><span style="color: #008000">dict</span>(facecolor<span style="color: #666666">=</span><span style="color: #BA2121">'black'</span>, shrink<span style="color: #666666">=0.1</span>),
|
||||
fontsize<span style="color: #666666">=18</span>,
|
||||
)
|
||||
plt<span style="color: #666666">.</span>plot([<span style="color: #666666">-0.1</span>, <span style="color: #666666">1.1</span>], [<span style="color: #666666">0.57</span>, <span style="color: #666666">-0.1</span>], <span style="color: #BA2121">"r--"</span>, linewidth<span style="color: #666666">=3</span>)
|
||||
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">-0.1</span>, <span style="color: #666666">1.1</span>, <span style="color: #666666">-0.1</span>, <span style="color: #666666">1.1</span>])
|
||||
|
||||
plt<span style="color: #666666">.</span>subplots_adjust(right<span style="color: #666666">=1</span>)
|
||||
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
|
||||
|
||||
x1_example <span style="color: #666666">=</span> X1D[<span style="color: #666666">3</span>, <span style="color: #666666">0</span>]
|
||||
<span style="color: #008000; font-weight: bold">for</span> landmark <span style="color: #AA22FF; font-weight: bold">in</span> (<span style="color: #666666">-2</span>, <span style="color: #666666">1</span>):
|
||||
k <span style="color: #666666">=</span> gaussian_rbf(np<span style="color: #666666">.</span>array([[x1_example]]), np<span style="color: #666666">.</span>array([[landmark]]), gamma)
|
||||
<span style="color: #008000">print</span>(<span style="color: #BA2121">"Phi(</span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">, </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">) = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">"</span><span style="color: #666666">.</span>format(x1_example, landmark, k))
|
||||
|
||||
rbf_kernel_svm_clf <span style="color: #666666">=</span> Pipeline([
|
||||
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
||||
(<span style="color: #BA2121">"svm_clf"</span>, SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">"rbf"</span>, gamma<span style="color: #666666">=5</span>, C<span style="color: #666666">=0.001</span>))
|
||||
])
|
||||
rbf_kernel_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
|
||||
|
||||
gamma1, gamma2 <span style="color: #666666">=</span> <span style="color: #666666">0.1</span>, <span style="color: #666666">5</span>
|
||||
C1, C2 <span style="color: #666666">=</span> <span style="color: #666666">0.001</span>, <span style="color: #666666">1000</span>
|
||||
hyperparams <span style="color: #666666">=</span> (gamma1, C1), (gamma1, C2), (gamma2, C1), (gamma2, C2)
|
||||
|
||||
svm_clfs <span style="color: #666666">=</span> []
|
||||
<span style="color: #008000; font-weight: bold">for</span> gamma, C <span style="color: #AA22FF; font-weight: bold">in</span> hyperparams:
|
||||
rbf_kernel_svm_clf <span style="color: #666666">=</span> Pipeline([
|
||||
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
||||
(<span style="color: #BA2121">"svm_clf"</span>, SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">"rbf"</span>, gamma<span style="color: #666666">=</span>gamma, C<span style="color: #666666">=</span>C))
|
||||
])
|
||||
rbf_kernel_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
||||
svm_clfs<span style="color: #666666">.</span>append(rbf_kernel_svm_clf)
|
||||
|
||||
plt<span style="color: #666666">.</span>figure(figsize<span style="color: #666666">=</span>(<span style="color: #666666">11</span>, <span style="color: #666666">7</span>))
|
||||
|
||||
<span style="color: #008000; font-weight: bold">for</span> i, svm_clf <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">enumerate</span>(svm_clfs):
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">221</span> <span style="color: #666666">+</span> i)
|
||||
plot_predictions(svm_clf, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plot_dataset(X, y, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
gamma, C <span style="color: #666666">=</span> hyperparams[i]
|
||||
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">r"$\gamma = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">, C = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">$"</span><span style="color: #666666">.</span>format(gamma, C), fontsize<span style="color: #666666">=16</span>)
|
||||
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -216,7 +372,6 @@ in practice.
|
||||
<li><a href="._week47-bs028.html">29</a></li>
|
||||
<li><a href="._week47-bs029.html">30</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs027.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,199 +159,28 @@ MathJax.Hub.Config({
|
||||
<a name="part0027"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec26" class="anchor">The moons example </h2>
|
||||
<h2 id="___sec26" class="anchor">Mathematical optimization of convex functions </h2>
|
||||
|
||||
<p>
|
||||
A mathematical (quadratic) optimization problem, or just optimization problem, has the form
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
|
||||
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">__future__</span> <span style="color: #008000; font-weight: bold">import</span> division, print_function, unicode_literals
|
||||
subject to some constraints for say a selected set \( i=1,2,\dots, n \).
|
||||
In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the
|
||||
vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with.
|
||||
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">42</span>)
|
||||
<p>
|
||||
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
|
||||
In our discussion on gradient descent methods we discussed at length the definition of a convex function.
|
||||
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">'axes.labelsize'</span>] <span style="color: #666666">=</span> <span style="color: #666666">14</span>
|
||||
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">'xtick.labelsize'</span>] <span style="color: #666666">=</span> <span style="color: #666666">12</span>
|
||||
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">'ytick.labelsize'</span>] <span style="color: #666666">=</span> <span style="color: #666666">12</span>
|
||||
<p>
|
||||
Convex optimization problems play a central role in applied mathematics and we recommend strongly <a href="http://web.stanford.edu/~boyd/cvxbook/" target="_self">Boyd and Vandenberghe's text on the topics</a>.
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
|
||||
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> LinearSVC
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> make_moons
|
||||
X, y <span style="color: #666666">=</span> make_moons(n_samples<span style="color: #666666">=100</span>, noise<span style="color: #666666">=0.15</span>, random_state<span style="color: #666666">=42</span>)
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">plot_dataset</span>(X, y, axes):
|
||||
plt<span style="color: #666666">.</span>plot(X[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==0</span>], X[:, <span style="color: #666666">1</span>][y<span style="color: #666666">==0</span>], <span style="color: #BA2121">"bs"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X[:, <span style="color: #666666">0</span>][y<span style="color: #666666">==1</span>], X[:, <span style="color: #666666">1</span>][y<span style="color: #666666">==1</span>], <span style="color: #BA2121">"g^"</span>)
|
||||
plt<span style="color: #666666">.</span>axis(axes)
|
||||
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">'both'</span>)
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r"$x_1$"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">r"$x_2$"</span>, fontsize<span style="color: #666666">=20</span>, rotation<span style="color: #666666">=0</span>)
|
||||
|
||||
plot_dataset(X, y, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> make_moons
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.pipeline</span> <span style="color: #008000; font-weight: bold">import</span> Pipeline
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> PolynomialFeatures
|
||||
|
||||
polynomial_svm_clf <span style="color: #666666">=</span> Pipeline([
|
||||
(<span style="color: #BA2121">"poly_features"</span>, PolynomialFeatures(degree<span style="color: #666666">=3</span>)),
|
||||
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
||||
(<span style="color: #BA2121">"svm_clf"</span>, LinearSVC(C<span style="color: #666666">=10</span>, loss<span style="color: #666666">=</span><span style="color: #BA2121">"hinge"</span>, random_state<span style="color: #666666">=42</span>))
|
||||
])
|
||||
|
||||
polynomial_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">plot_predictions</span>(clf, axes):
|
||||
x0s <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(axes[<span style="color: #666666">0</span>], axes[<span style="color: #666666">1</span>], <span style="color: #666666">100</span>)
|
||||
x1s <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(axes[<span style="color: #666666">2</span>], axes[<span style="color: #666666">3</span>], <span style="color: #666666">100</span>)
|
||||
x0, x1 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>meshgrid(x0s, x1s)
|
||||
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[x0<span style="color: #666666">.</span>ravel(), x1<span style="color: #666666">.</span>ravel()]
|
||||
y_pred <span style="color: #666666">=</span> clf<span style="color: #666666">.</span>predict(X)<span style="color: #666666">.</span>reshape(x0<span style="color: #666666">.</span>shape)
|
||||
y_decision <span style="color: #666666">=</span> clf<span style="color: #666666">.</span>decision_function(X)<span style="color: #666666">.</span>reshape(x0<span style="color: #666666">.</span>shape)
|
||||
plt<span style="color: #666666">.</span>contourf(x0, x1, y_pred, cmap<span style="color: #666666">=</span>plt<span style="color: #666666">.</span>cm<span style="color: #666666">.</span>brg, alpha<span style="color: #666666">=0.2</span>)
|
||||
plt<span style="color: #666666">.</span>contourf(x0, x1, y_decision, cmap<span style="color: #666666">=</span>plt<span style="color: #666666">.</span>cm<span style="color: #666666">.</span>brg, alpha<span style="color: #666666">=0.1</span>)
|
||||
|
||||
plot_predictions(polynomial_svm_clf, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plot_dataset(X, y, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
|
||||
|
||||
poly_kernel_svm_clf <span style="color: #666666">=</span> Pipeline([
|
||||
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
||||
(<span style="color: #BA2121">"svm_clf"</span>, SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">"poly"</span>, degree<span style="color: #666666">=3</span>, coef0<span style="color: #666666">=1</span>, C<span style="color: #666666">=5</span>))
|
||||
])
|
||||
poly_kernel_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
||||
|
||||
poly100_kernel_svm_clf <span style="color: #666666">=</span> Pipeline([
|
||||
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
||||
(<span style="color: #BA2121">"svm_clf"</span>, SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">"poly"</span>, degree<span style="color: #666666">=10</span>, coef0<span style="color: #666666">=100</span>, C<span style="color: #666666">=5</span>))
|
||||
])
|
||||
poly100_kernel_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
||||
|
||||
plt<span style="color: #666666">.</span>figure(figsize<span style="color: #666666">=</span>(<span style="color: #666666">11</span>, <span style="color: #666666">4</span>))
|
||||
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">121</span>)
|
||||
plot_predictions(poly_kernel_svm_clf, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plot_dataset(X, y, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">r"$d=3, r=1, C=5$"</span>, fontsize<span style="color: #666666">=18</span>)
|
||||
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">122</span>)
|
||||
plot_predictions(poly100_kernel_svm_clf, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plot_dataset(X, y, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">r"$d=10, r=100, C=5$"</span>, fontsize<span style="color: #666666">=18</span>)
|
||||
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">gaussian_rbf</span>(x, landmark, gamma):
|
||||
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>gamma <span style="color: #666666">*</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>norm(x <span style="color: #666666">-</span> landmark, axis<span style="color: #666666">=1</span>)<span style="color: #666666">**2</span>)
|
||||
|
||||
gamma <span style="color: #666666">=</span> <span style="color: #666666">0.3</span>
|
||||
|
||||
x1s <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">-4.5</span>, <span style="color: #666666">4.5</span>, <span style="color: #666666">200</span>)<span style="color: #666666">.</span>reshape(<span style="color: #666666">-1</span>, <span style="color: #666666">1</span>)
|
||||
x2s <span style="color: #666666">=</span> gaussian_rbf(x1s, <span style="color: #666666">-2</span>, gamma)
|
||||
x3s <span style="color: #666666">=</span> gaussian_rbf(x1s, <span style="color: #666666">1</span>, gamma)
|
||||
|
||||
XK <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[gaussian_rbf(X1D, <span style="color: #666666">-2</span>, gamma), gaussian_rbf(X1D, <span style="color: #666666">1</span>, gamma)]
|
||||
yk <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">0</span>, <span style="color: #666666">0</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">0</span>])
|
||||
|
||||
plt<span style="color: #666666">.</span>figure(figsize<span style="color: #666666">=</span>(<span style="color: #666666">11</span>, <span style="color: #666666">4</span>))
|
||||
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">121</span>)
|
||||
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">'both'</span>)
|
||||
plt<span style="color: #666666">.</span>axhline(y<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
||||
plt<span style="color: #666666">.</span>scatter(x<span style="color: #666666">=</span>[<span style="color: #666666">-2</span>, <span style="color: #666666">1</span>], y<span style="color: #666666">=</span>[<span style="color: #666666">0</span>, <span style="color: #666666">0</span>], s<span style="color: #666666">=150</span>, alpha<span style="color: #666666">=0.5</span>, c<span style="color: #666666">=</span><span style="color: #BA2121">"red"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X1D[:, <span style="color: #666666">0</span>][yk<span style="color: #666666">==0</span>], np<span style="color: #666666">.</span>zeros(<span style="color: #666666">4</span>), <span style="color: #BA2121">"bs"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(X1D[:, <span style="color: #666666">0</span>][yk<span style="color: #666666">==1</span>], np<span style="color: #666666">.</span>zeros(<span style="color: #666666">5</span>), <span style="color: #BA2121">"g^"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(x1s, x2s, <span style="color: #BA2121">"g--"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(x1s, x3s, <span style="color: #BA2121">"b:"</span>)
|
||||
plt<span style="color: #666666">.</span>gca()<span style="color: #666666">.</span>get_yaxis()<span style="color: #666666">.</span>set_ticks([<span style="color: #666666">0</span>, <span style="color: #666666">0.25</span>, <span style="color: #666666">0.5</span>, <span style="color: #666666">0.75</span>, <span style="color: #666666">1</span>])
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r"$x_1$"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">r"Similarity"</span>, fontsize<span style="color: #666666">=14</span>)
|
||||
plt<span style="color: #666666">.</span>annotate(<span style="color: #BA2121">r'$\mathbf</span><span style="color: #BB6688; font-weight: bold">{x}</span><span style="color: #BA2121">$'</span>,
|
||||
xy<span style="color: #666666">=</span>(X1D[<span style="color: #666666">3</span>, <span style="color: #666666">0</span>], <span style="color: #666666">0</span>),
|
||||
xytext<span style="color: #666666">=</span>(<span style="color: #666666">-0.5</span>, <span style="color: #666666">0.20</span>),
|
||||
ha<span style="color: #666666">=</span><span style="color: #BA2121">"center"</span>,
|
||||
arrowprops<span style="color: #666666">=</span><span style="color: #008000">dict</span>(facecolor<span style="color: #666666">=</span><span style="color: #BA2121">'black'</span>, shrink<span style="color: #666666">=0.1</span>),
|
||||
fontsize<span style="color: #666666">=18</span>,
|
||||
)
|
||||
plt<span style="color: #666666">.</span>text(<span style="color: #666666">-2</span>, <span style="color: #666666">0.9</span>, <span style="color: #BA2121">"$x_2$"</span>, ha<span style="color: #666666">=</span><span style="color: #BA2121">"center"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>text(<span style="color: #666666">1</span>, <span style="color: #666666">0.9</span>, <span style="color: #BA2121">"$x_3$"</span>, ha<span style="color: #666666">=</span><span style="color: #BA2121">"center"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">-4.5</span>, <span style="color: #666666">4.5</span>, <span style="color: #666666">-0.1</span>, <span style="color: #666666">1.1</span>])
|
||||
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">122</span>)
|
||||
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>, which<span style="color: #666666">=</span><span style="color: #BA2121">'both'</span>)
|
||||
plt<span style="color: #666666">.</span>axhline(y<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
||||
plt<span style="color: #666666">.</span>axvline(x<span style="color: #666666">=0</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">'k'</span>)
|
||||
plt<span style="color: #666666">.</span>plot(XK[:, <span style="color: #666666">0</span>][yk<span style="color: #666666">==0</span>], XK[:, <span style="color: #666666">1</span>][yk<span style="color: #666666">==0</span>], <span style="color: #BA2121">"bs"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(XK[:, <span style="color: #666666">0</span>][yk<span style="color: #666666">==1</span>], XK[:, <span style="color: #666666">1</span>][yk<span style="color: #666666">==1</span>], <span style="color: #BA2121">"g^"</span>)
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r"$x_2$"</span>, fontsize<span style="color: #666666">=20</span>)
|
||||
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">r"$x_3$ "</span>, fontsize<span style="color: #666666">=20</span>, rotation<span style="color: #666666">=0</span>)
|
||||
plt<span style="color: #666666">.</span>annotate(<span style="color: #BA2121">r'$\phi\left(\mathbf</span><span style="color: #BB6688; font-weight: bold">{x}</span><span style="color: #BA2121">\right)$'</span>,
|
||||
xy<span style="color: #666666">=</span>(XK[<span style="color: #666666">3</span>, <span style="color: #666666">0</span>], XK[<span style="color: #666666">3</span>, <span style="color: #666666">1</span>]),
|
||||
xytext<span style="color: #666666">=</span>(<span style="color: #666666">0.65</span>, <span style="color: #666666">0.50</span>),
|
||||
ha<span style="color: #666666">=</span><span style="color: #BA2121">"center"</span>,
|
||||
arrowprops<span style="color: #666666">=</span><span style="color: #008000">dict</span>(facecolor<span style="color: #666666">=</span><span style="color: #BA2121">'black'</span>, shrink<span style="color: #666666">=0.1</span>),
|
||||
fontsize<span style="color: #666666">=18</span>,
|
||||
)
|
||||
plt<span style="color: #666666">.</span>plot([<span style="color: #666666">-0.1</span>, <span style="color: #666666">1.1</span>], [<span style="color: #666666">0.57</span>, <span style="color: #666666">-0.1</span>], <span style="color: #BA2121">"r--"</span>, linewidth<span style="color: #666666">=3</span>)
|
||||
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">-0.1</span>, <span style="color: #666666">1.1</span>, <span style="color: #666666">-0.1</span>, <span style="color: #666666">1.1</span>])
|
||||
|
||||
plt<span style="color: #666666">.</span>subplots_adjust(right<span style="color: #666666">=1</span>)
|
||||
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
|
||||
|
||||
x1_example <span style="color: #666666">=</span> X1D[<span style="color: #666666">3</span>, <span style="color: #666666">0</span>]
|
||||
<span style="color: #008000; font-weight: bold">for</span> landmark <span style="color: #AA22FF; font-weight: bold">in</span> (<span style="color: #666666">-2</span>, <span style="color: #666666">1</span>):
|
||||
k <span style="color: #666666">=</span> gaussian_rbf(np<span style="color: #666666">.</span>array([[x1_example]]), np<span style="color: #666666">.</span>array([[landmark]]), gamma)
|
||||
<span style="color: #008000">print</span>(<span style="color: #BA2121">"Phi(</span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">, </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">) = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">"</span><span style="color: #666666">.</span>format(x1_example, landmark, k))
|
||||
|
||||
rbf_kernel_svm_clf <span style="color: #666666">=</span> Pipeline([
|
||||
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
||||
(<span style="color: #BA2121">"svm_clf"</span>, SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">"rbf"</span>, gamma<span style="color: #666666">=5</span>, C<span style="color: #666666">=0.001</span>))
|
||||
])
|
||||
rbf_kernel_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVC
|
||||
|
||||
gamma1, gamma2 <span style="color: #666666">=</span> <span style="color: #666666">0.1</span>, <span style="color: #666666">5</span>
|
||||
C1, C2 <span style="color: #666666">=</span> <span style="color: #666666">0.001</span>, <span style="color: #666666">1000</span>
|
||||
hyperparams <span style="color: #666666">=</span> (gamma1, C1), (gamma1, C2), (gamma2, C1), (gamma2, C2)
|
||||
|
||||
svm_clfs <span style="color: #666666">=</span> []
|
||||
<span style="color: #008000; font-weight: bold">for</span> gamma, C <span style="color: #AA22FF; font-weight: bold">in</span> hyperparams:
|
||||
rbf_kernel_svm_clf <span style="color: #666666">=</span> Pipeline([
|
||||
(<span style="color: #BA2121">"scaler"</span>, StandardScaler()),
|
||||
(<span style="color: #BA2121">"svm_clf"</span>, SVC(kernel<span style="color: #666666">=</span><span style="color: #BA2121">"rbf"</span>, gamma<span style="color: #666666">=</span>gamma, C<span style="color: #666666">=</span>C))
|
||||
])
|
||||
rbf_kernel_svm_clf<span style="color: #666666">.</span>fit(X, y)
|
||||
svm_clfs<span style="color: #666666">.</span>append(rbf_kernel_svm_clf)
|
||||
|
||||
plt<span style="color: #666666">.</span>figure(figsize<span style="color: #666666">=</span>(<span style="color: #666666">11</span>, <span style="color: #666666">7</span>))
|
||||
|
||||
<span style="color: #008000; font-weight: bold">for</span> i, svm_clf <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">enumerate</span>(svm_clfs):
|
||||
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">221</span> <span style="color: #666666">+</span> i)
|
||||
plot_predictions(svm_clf, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
plot_dataset(X, y, [<span style="color: #666666">-1.5</span>, <span style="color: #666666">2.5</span>, <span style="color: #666666">-1</span>, <span style="color: #666666">1.5</span>])
|
||||
gamma, C <span style="color: #666666">=</span> hyperparams[i]
|
||||
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">r"$\gamma = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">, C = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">$"</span><span style="color: #666666">.</span>format(gamma, C), fontsize<span style="color: #666666">=16</span>)
|
||||
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -373,7 +200,6 @@ plt<span style="color: #666666">.</span>show()
|
||||
<li><a href="._week47-bs028.html">29</a></li>
|
||||
<li><a href="._week47-bs029.html">30</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs028.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,27 +159,28 @@ MathJax.Hub.Config({
|
||||
<a name="part0028"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec27" class="anchor">Mathematical optimization of convex functions </h2>
|
||||
<h2 id="___sec27" class="anchor">How do we solve these problems? </h2>
|
||||
|
||||
<p>
|
||||
A mathematical (quadratic) optimization problem, or just optimization problem, has the form
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
|
||||
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
subject to some constraints for say a selected set \( i=1,2,\dots, n \).
|
||||
In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the
|
||||
vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with.
|
||||
If we use Python as programming language and wish to venture beyond
|
||||
<b>scikit-learn</b>, <b>tensorflow</b> and similar software which makes our
|
||||
lives so much easier, we need to dive into the wonderful world of
|
||||
quadratic programming. We can, if we wish, solve the minimization
|
||||
problem using say standard gradient methods or conjugate gradient
|
||||
methods. However, these methods tend to exhibit a rather slow
|
||||
converge. So, welcome to the promised land of quadratic programming.
|
||||
|
||||
<p>
|
||||
In our case we are particularly interested in a class of optimization problems called convex optmization problems.
|
||||
In our discussion on gradient descent methods we discussed at length the definition of a convex function.
|
||||
The functions we need are contained in the quadratic programming package <b>CVXOPT</b> and we need to import it together with <b>numpy</b> as
|
||||
|
||||
<p>
|
||||
Convex optimization problems play a central role in applied mathematics and we recommend strongly <a href="http://web.stanford.edu/~boyd/cvxbook/" target="_self">Boyd and Vandenberghe's text on the topics</a>.
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
This will make our life much easier. You don't need t write your own optimizer.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -201,7 +200,6 @@ Convex optimization problems play a central role in applied mathematics and we r
|
||||
<li class="active"><a href="._week47-bs028.html">29</a></li>
|
||||
<li><a href="._week47-bs029.html">30</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs029.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,29 +159,71 @@ MathJax.Hub.Config({
|
||||
<a name="part0029"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec28" class="anchor">How do we solve these problems? </h2>
|
||||
<h2 id="___sec28" class="anchor">A simple example </h2>
|
||||
|
||||
<p>
|
||||
If we use Python as programming language and wish to venture beyond
|
||||
<b>scikit-learn</b>, <b>tensorflow</b> and similar software which makes our
|
||||
lives so much easier, we need to dive into the wonderful world of
|
||||
quadratic programming. We can, if we wish, solve the minimization
|
||||
problem using say standard gradient methods or conjugate gradient
|
||||
methods. However, these methods tend to exhibit a rather slow
|
||||
converge. So, welcome to the promised land of quadratic programming.
|
||||
We remind ourselves about the general problem we want to solve
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
|
||||
&\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
<p>
|
||||
The functions we need are contained in the quadratic programming package <b>CVXOPT</b> and we need to import it together with <b>numpy</b> as
|
||||
Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
|
||||
&\mathrm{subject to} \\ \nonumber
|
||||
&x, y \geq 0 \\ \nonumber
|
||||
&x+3y \geq 15 \\ \nonumber
|
||||
&2x+5y \leq 100 \\ \nonumber
|
||||
&3x+4y \leq 80. \\ \nonumber
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
|
||||
$$
|
||||
\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
|
||||
$$
|
||||
|
||||
Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
|
||||
$$
|
||||
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
$$
|
||||
|
||||
We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix
|
||||
$$
|
||||
\boldsymbol{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix}
|
||||
$$
|
||||
|
||||
is clearly positive semi-definite (all eigenvalues larger or equal zero).
|
||||
Finally, the vector \( \boldsymbol{h} \) is defined as
|
||||
$$
|
||||
\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero
|
||||
The following code solves the equations for us
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span>
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Import the necessary packages</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span> <span style="color: #008000; font-weight: bold">import</span> matrix
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span> <span style="color: #008000; font-weight: bold">import</span> solvers
|
||||
P <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>diag([<span style="color: #666666">1</span>,<span style="color: #666666">0</span>]), tc<span style="color: #666666">=</span>’d’)
|
||||
q <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>array([<span style="color: #666666">3</span>,<span style="color: #666666">4</span>]), tc<span style="color: #666666">=</span>’d’)
|
||||
G <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>array([[<span style="color: #666666">-1</span>,<span style="color: #666666">0</span>],[<span style="color: #666666">0</span>,<span style="color: #666666">-1</span>],[<span style="color: #666666">-1</span>,<span style="color: #666666">-3</span>],[<span style="color: #666666">2</span>,<span style="color: #666666">5</span>],[<span style="color: #666666">3</span>,<span style="color: #666666">4</span>]]), tc<span style="color: #666666">=</span>’d’)
|
||||
h <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>array([<span style="color: #666666">0</span>,<span style="color: #666666">0</span>,<span style="color: #666666">-15</span>,<span style="color: #666666">100</span>,<span style="color: #666666">80</span>]), tc<span style="color: #666666">=</span>’d’)
|
||||
<span style="color: #408080; font-style: italic"># Construct the QP, invoke solver</span>
|
||||
sol <span style="color: #666666">=</span> solvers<span style="color: #666666">.</span>qp(P,q,G,h)
|
||||
<span style="color: #408080; font-style: italic"># Extract optimal value and solution</span>
|
||||
sol[’x’]
|
||||
sol[’primal objective’]
|
||||
</pre></div>
|
||||
<p>
|
||||
This will make our life much easier. You don't need t write your own optimizer.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -201,7 +241,6 @@ This will make our life much easier. You don't need t write your own optimizer.
|
||||
<li><a href="._week47-bs028.html">29</a></li>
|
||||
<li class="active"><a href="._week47-bs029.html">30</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -161,72 +159,25 @@ MathJax.Hub.Config({
|
||||
<a name="part0030"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec29" class="anchor">A simple example </h2>
|
||||
<h2 id="___sec29" class="anchor">Back to the more realistic cases </h2>
|
||||
|
||||
<p>
|
||||
We remind ourselves about the general problem we want to solve
|
||||
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the <b>slack</b> parameter \( C \) we have
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}\boldsymbol{x}^T\boldsymbol{P}\boldsymbol{x}+\boldsymbol{q}^T\boldsymbol{x},\\ \nonumber
|
||||
&\mathrm{subject\hspace{0.1cm} to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{x} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{x}=f.
|
||||
\end{align*}
|
||||
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
|
||||
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots \\
|
||||
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
|
||||
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
|
||||
$$
|
||||
|
||||
subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
|
||||
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
|
||||
With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
|
||||
|
||||
<p>
|
||||
Let us show how to perform the optmization using a simple case. Assume we want to optimize the following problem
|
||||
$$
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{x}\hspace{0.2cm} \frac{1}{2}x^2+5x+3y \\ \nonumber
|
||||
&\mathrm{subject to} \\ \nonumber
|
||||
&x, y \geq 0 \\ \nonumber
|
||||
&x+3y \geq 15 \\ \nonumber
|
||||
&2x+5y \leq 100 \\ \nonumber
|
||||
&3x+4y \leq 80. \\ \nonumber
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
The minimization problem can be rewritten in terms of vectors and matrices as (with \( x \) and \( y \) being the unknowns)
|
||||
$$
|
||||
\frac{1}{2}\begin{bmatrix} x\\ y \end{bmatrix}^T \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} + \begin{bmatrix}3\\ 4 \end{bmatrix}^T \begin{bmatrix}x \\ y \end{bmatrix}.
|
||||
$$
|
||||
|
||||
Similarly, we can now set up the inequalities (we need to change \( \geq \) to \( \leq \) by multiplying with \( -1 \) on bot sides) as the following matrix-vector equation
|
||||
$$
|
||||
\begin{bmatrix} -1 & 0 \\ 0 & -1 \\ -1 & -3 \\ 2 & 5 \\ 3 & 4\end{bmatrix}\begin{bmatrix} x \\ y\end{bmatrix} \preceq \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
$$
|
||||
|
||||
We have collapsed all the inequalities into a single matrix \( \boldsymbol{G} \). We see also that our matrix
|
||||
$$
|
||||
\boldsymbol{P} =\begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix}
|
||||
$$
|
||||
|
||||
is clearly positive semi-definite (all eigenvalues larger or equal zero).
|
||||
Finally, the vector \( \boldsymbol{h} \) is defined as
|
||||
$$
|
||||
\boldsymbol{h} = \begin{bmatrix}0 \\ 0\\ -15 \\ 100 \\ 80\end{bmatrix}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
Since we don't have any equalities the matrix \( \boldsymbol{A} \) is set to zero
|
||||
The following code solves the equations for us
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Import the necessary packages</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span> <span style="color: #008000; font-weight: bold">import</span> matrix
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span> <span style="color: #008000; font-weight: bold">import</span> solvers
|
||||
P <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>diag([<span style="color: #666666">1</span>,<span style="color: #666666">0</span>]), tc<span style="color: #666666">=</span>’d’)
|
||||
q <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>array([<span style="color: #666666">3</span>,<span style="color: #666666">4</span>]), tc<span style="color: #666666">=</span>’d’)
|
||||
G <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>array([[<span style="color: #666666">-1</span>,<span style="color: #666666">0</span>],[<span style="color: #666666">0</span>,<span style="color: #666666">-1</span>],[<span style="color: #666666">-1</span>,<span style="color: #666666">-3</span>],[<span style="color: #666666">2</span>,<span style="color: #666666">5</span>],[<span style="color: #666666">3</span>,<span style="color: #666666">4</span>]]), tc<span style="color: #666666">=</span>’d’)
|
||||
h <span style="color: #666666">=</span> matrix(numpy<span style="color: #666666">.</span>array([<span style="color: #666666">0</span>,<span style="color: #666666">0</span>,<span style="color: #666666">-15</span>,<span style="color: #666666">100</span>,<span style="color: #666666">80</span>]), tc<span style="color: #666666">=</span>’d’)
|
||||
<span style="color: #408080; font-style: italic"># Construct the QP, invoke solver</span>
|
||||
sol <span style="color: #666666">=</span> solvers<span style="color: #666666">.</span>qp(P,q,G,h)
|
||||
<span style="color: #408080; font-style: italic"># Extract optimal value and solution</span>
|
||||
sol[’x’]
|
||||
sol[’primal objective’]
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
@@ -242,8 +193,6 @@ sol[’primal objective’]
|
||||
<li><a href="._week47-bs028.html">29</a></li>
|
||||
<li><a href="._week47-bs029.html">30</a></li>
|
||||
<li class="active"><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs031.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
|
||||
@@ -53,31 +53,30 @@ Automatically generated HTML file from DocOnce source
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -127,25 +126,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs010.html#___sec9" style="font-size: 80%;">Getting into the details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs011.html#___sec10" style="font-size: 80%;">First attempt at a minimization approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs012.html#___sec11" style="font-size: 80%;">Solving the equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Code Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs031.html#___sec30" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs013.html#___sec12" style="font-size: 80%;">Can we code this?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs014.html#___sec13" style="font-size: 80%;">A better approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs015.html#___sec14" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs016.html#___sec15" style="font-size: 80%;">Adding the Multiplier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs017.html#___sec16" style="font-size: 80%;">Setting up the Problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs018.html#___sec17" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs019.html#___sec18" style="font-size: 80%;">The last steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs020.html#___sec19" style="font-size: 80%;">A soft classifier</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs021.html#___sec20" style="font-size: 80%;">Soft optmization problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs022.html#___sec21" style="font-size: 80%;">Kernels and non-linearity</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs023.html#___sec22" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs024.html#___sec23" style="font-size: 80%;">The problem to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs025.html#___sec24" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs026.html#___sec25" style="font-size: 80%;">The moons example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs027.html#___sec26" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs028.html#___sec27" style="font-size: 80%;">How do we solve these problems?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs029.html#___sec28" style="font-size: 80%;">A simple example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week47-bs030.html#___sec29" style="font-size: 80%;">Back to the more realistic cases</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -204,7 +202,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._week47-bs008.html">9</a></li>
|
||||
<li><a href="._week47-bs009.html">10</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week47-bs031.html">32</a></li>
|
||||
<li><a href="._week47-bs030.html">31</a></li>
|
||||
<li><a href="._week47-bs001.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -524,26 +524,19 @@ where \( \eta \) is our by now well-known learning rate.
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec12">Code Example </h2>
|
||||
<h2 id="___sec12">Can we code this? </h2>
|
||||
|
||||
<p>
|
||||
The equations we discussed above can be coded rather easily (the
|
||||
framework is similar to what we developed for logistic
|
||||
regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way.
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>
|
||||
</pre></div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec13">Problems with the Simpler Approach </h2>
|
||||
framework is similar to what we developed for logistic regression). We
|
||||
can set up a simple case with two classes only and we want to find a
|
||||
line which separates them the best possible way.
|
||||
|
||||
<p>
|
||||
There are however problems with this approach, although it looks
|
||||
pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes.
|
||||
pretty straightforward to implement. When running a code for such a
|
||||
case we can easily end up with many diffeent lines which separate the
|
||||
two classes.
|
||||
|
||||
<p>
|
||||
For small
|
||||
@@ -555,7 +548,7 @@ at all.
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec14">A better approach </h2>
|
||||
<h2 id="___sec13">A better approach </h2>
|
||||
|
||||
<p>
|
||||
A better approach is rather to try to define a large margin between
|
||||
@@ -605,7 +598,7 @@ about Lagrangian multipliers.
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec15">A quick Reminder on Lagrangian Multipliers </h2>
|
||||
<h2 id="___sec14">A quick Reminder on Lagrangian Multipliers </h2>
|
||||
|
||||
<p>
|
||||
Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an
|
||||
@@ -667,7 +660,7 @@ Then \( dz \) is no longer arbitrary.
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec16">Adding the Multiplier </h2>
|
||||
<h2 id="___sec15">Adding the Multiplier </h2>
|
||||
|
||||
<p>
|
||||
However, we can add to
|
||||
@@ -722,7 +715,7 @@ $$
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec17">Setting up the Problem </h2>
|
||||
<h2 id="___sec16">Setting up the Problem </h2>
|
||||
In order to solve the above problem, we define the following Lagrangian function to be minimized
|
||||
<p> <br>
|
||||
$$
|
||||
@@ -774,7 +767,7 @@ When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec18">The problem to solve </h2>
|
||||
<h2 id="___sec17">The problem to solve </h2>
|
||||
|
||||
<p>
|
||||
We can rewrite
|
||||
@@ -802,7 +795,7 @@ subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vec
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec19">The last steps </h2>
|
||||
<h2 id="___sec18">The last steps </h2>
|
||||
|
||||
<p>
|
||||
Solving the above problem, yields the values of \( \lambda_i \).
|
||||
@@ -846,7 +839,7 @@ Below we discuss how to find the optimal values of \( \lambda_i \). Before we pr
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec20">A soft classifier </h2>
|
||||
<h2 id="___sec19">A soft classifier </h2>
|
||||
|
||||
<p>
|
||||
Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
|
||||
@@ -885,7 +878,7 @@ misclassifications.
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec21">Soft optmization problem </h2>
|
||||
<h2 id="___sec20">Soft optmization problem </h2>
|
||||
|
||||
<p>
|
||||
This has in turn the consequences that we change our optmization problem to finding the minimum of
|
||||
@@ -957,7 +950,7 @@ $$
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec22">Kernels and non-linearity </h2>
|
||||
<h2 id="___sec21">Kernels and non-linearity </h2>
|
||||
|
||||
<p>
|
||||
The cases we have studied till now, were all characterized by two classes
|
||||
@@ -1031,7 +1024,7 @@ plt.show()
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec23">The equations </h2>
|
||||
<h2 id="___sec22">The equations </h2>
|
||||
|
||||
<p>
|
||||
Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
|
||||
@@ -1086,7 +1079,7 @@ the trouble of performing the transformation
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec24">The problem to solve </h2>
|
||||
<h2 id="___sec23">The problem to solve </h2>
|
||||
Using our definition of the kernel We can rewrite again the Lagrangian
|
||||
<p> <br>
|
||||
$$
|
||||
@@ -1128,7 +1121,7 @@ Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up.
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec25">Different kernels and Mercer's theorem </h2>
|
||||
<h2 id="___sec24">Different kernels and Mercer's theorem </h2>
|
||||
|
||||
<p>
|
||||
There are several popular kernels being used. These are
|
||||
@@ -1169,7 +1162,7 @@ in practice.
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec26">The moons example </h2>
|
||||
<h2 id="___sec25">The moons example </h2>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
@@ -1366,7 +1359,7 @@ plt.show()
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec27">Mathematical optimization of convex functions </h2>
|
||||
<h2 id="___sec26">Mathematical optimization of convex functions </h2>
|
||||
|
||||
<p>
|
||||
A mathematical (quadratic) optimization problem, or just optimization problem, has the form
|
||||
@@ -1393,7 +1386,7 @@ Convex optimization problems play a central role in applied mathematics and we r
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec28">How do we solve these problems? </h2>
|
||||
<h2 id="___sec27">How do we solve these problems? </h2>
|
||||
|
||||
<p>
|
||||
If we use Python as programming language and wish to venture beyond
|
||||
@@ -1419,7 +1412,7 @@ This will make our life much easier. You don't need t write your own optimizer.
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec29">A simple example </h2>
|
||||
<h2 id="___sec28">A simple example </h2>
|
||||
|
||||
<p>
|
||||
We remind ourselves about the general problem we want to solve
|
||||
@@ -1500,7 +1493,7 @@ sol[<span style="color: #a61717; background-color: #e3d2d2">’</span>primal obj
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec30">Back to the more realistic cases </h2>
|
||||
<h2 id="___sec29">Back to the more realistic cases </h2>
|
||||
|
||||
<p>
|
||||
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the <b>slack</b> parameter \( C \) we have
|
||||
|
||||
@@ -47,31 +47,30 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -452,25 +451,19 @@ where \( \eta \) is our by now well-known learning rate.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec12">Code Example </h2>
|
||||
<h2 id="___sec12">Can we code this? </h2>
|
||||
|
||||
<p>
|
||||
The equations we discussed above can be coded rather easily (the
|
||||
framework is similar to what we developed for logistic
|
||||
regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way.
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec13">Problems with the Simpler Approach </h2>
|
||||
framework is similar to what we developed for logistic regression). We
|
||||
can set up a simple case with two classes only and we want to find a
|
||||
line which separates them the best possible way.
|
||||
|
||||
<p>
|
||||
There are however problems with this approach, although it looks
|
||||
pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes.
|
||||
pretty straightforward to implement. When running a code for such a
|
||||
case we can easily end up with many diffeent lines which separate the
|
||||
two classes.
|
||||
|
||||
<p>
|
||||
For small
|
||||
@@ -482,7 +475,7 @@ at all.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec14">A better approach </h2>
|
||||
<h2 id="___sec13">A better approach </h2>
|
||||
|
||||
<p>
|
||||
A better approach is rather to try to define a large margin between
|
||||
@@ -524,7 +517,7 @@ about Lagrangian multipliers.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec15">A quick Reminder on Lagrangian Multipliers </h2>
|
||||
<h2 id="___sec14">A quick Reminder on Lagrangian Multipliers </h2>
|
||||
|
||||
<p>
|
||||
Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an
|
||||
@@ -574,7 +567,7 @@ Then \( dz \) is no longer arbitrary.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec16">Adding the Multiplier </h2>
|
||||
<h2 id="___sec15">Adding the Multiplier </h2>
|
||||
|
||||
<p>
|
||||
However, we can add to
|
||||
@@ -617,7 +610,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec17">Setting up the Problem </h2>
|
||||
<h2 id="___sec16">Setting up the Problem </h2>
|
||||
In order to solve the above problem, we define the following Lagrangian function to be minimized
|
||||
$$
|
||||
{\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right],
|
||||
@@ -658,7 +651,7 @@ When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec18">The problem to solve </h2>
|
||||
<h2 id="___sec17">The problem to solve </h2>
|
||||
|
||||
<p>
|
||||
We can rewrite
|
||||
@@ -682,7 +675,7 @@ subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vec
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec19">The last steps </h2>
|
||||
<h2 id="___sec18">The last steps </h2>
|
||||
|
||||
<p>
|
||||
Solving the above problem, yields the values of \( \lambda_i \).
|
||||
@@ -716,7 +709,7 @@ Below we discuss how to find the optimal values of \( \lambda_i \). Before we pr
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec20">A soft classifier </h2>
|
||||
<h2 id="___sec19">A soft classifier </h2>
|
||||
|
||||
<p>
|
||||
Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
|
||||
@@ -751,7 +744,7 @@ misclassifications.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec21">Soft optmization problem </h2>
|
||||
<h2 id="___sec20">Soft optmization problem </h2>
|
||||
|
||||
<p>
|
||||
This has in turn the consequences that we change our optmization problem to finding the minimum of
|
||||
@@ -805,7 +798,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec22">Kernels and non-linearity </h2>
|
||||
<h2 id="___sec21">Kernels and non-linearity </h2>
|
||||
|
||||
<p>
|
||||
The cases we have studied till now, were all characterized by two classes
|
||||
@@ -878,7 +871,7 @@ plt.show()
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec23">The equations </h2>
|
||||
<h2 id="___sec22">The equations </h2>
|
||||
|
||||
<p>
|
||||
Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
|
||||
@@ -923,7 +916,7 @@ the trouble of performing the transformation
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec24">The problem to solve </h2>
|
||||
<h2 id="___sec23">The problem to solve </h2>
|
||||
Using our definition of the kernel We can rewrite again the Lagrangian
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
|
||||
@@ -959,7 +952,7 @@ Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec25">Different kernels and Mercer's theorem </h2>
|
||||
<h2 id="___sec24">Different kernels and Mercer's theorem </h2>
|
||||
|
||||
<p>
|
||||
There are several popular kernels being used. These are
|
||||
@@ -997,7 +990,7 @@ in practice.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec26">The moons example </h2>
|
||||
<h2 id="___sec25">The moons example </h2>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
@@ -1193,7 +1186,7 @@ plt.show()
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec27">Mathematical optimization of convex functions </h2>
|
||||
<h2 id="___sec26">Mathematical optimization of convex functions </h2>
|
||||
|
||||
<p>
|
||||
A mathematical (quadratic) optimization problem, or just optimization problem, has the form
|
||||
@@ -1218,7 +1211,7 @@ Convex optimization problems play a central role in applied mathematics and we r
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec28">How do we solve these problems? </h2>
|
||||
<h2 id="___sec27">How do we solve these problems? </h2>
|
||||
|
||||
<p>
|
||||
If we use Python as programming language and wish to venture beyond
|
||||
@@ -1244,7 +1237,7 @@ This will make our life much easier. You don't need t write your own optimizer.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec29">A simple example </h2>
|
||||
<h2 id="___sec28">A simple example </h2>
|
||||
|
||||
<p>
|
||||
We remind ourselves about the general problem we want to solve
|
||||
@@ -1312,7 +1305,7 @@ sol[<span style="color: #a61717; background-color: #e3d2d2">’</span>primal obj
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec30">Back to the more realistic cases </h2>
|
||||
<h2 id="___sec29">Back to the more realistic cases </h2>
|
||||
|
||||
<p>
|
||||
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the <b>slack</b> parameter \( C \) we have
|
||||
|
||||
@@ -52,31 +52,30 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
('Getting into the details', 2, None, '___sec9'),
|
||||
('First attempt at a minimization approach', 2, None, '___sec10'),
|
||||
('Solving the equations', 2, None, '___sec11'),
|
||||
('Code Example', 2, None, '___sec12'),
|
||||
('Problems with the Simpler Approach', 2, None, '___sec13'),
|
||||
('A better approach', 2, None, '___sec14'),
|
||||
('Can we code this?', 2, None, '___sec12'),
|
||||
('A better approach', 2, None, '___sec13'),
|
||||
('A quick Reminder on Lagrangian Multipliers',
|
||||
2,
|
||||
None,
|
||||
'___sec15'),
|
||||
('Adding the Multiplier', 2, None, '___sec16'),
|
||||
('Setting up the Problem', 2, None, '___sec17'),
|
||||
('The problem to solve', 2, None, '___sec18'),
|
||||
('The last steps', 2, None, '___sec19'),
|
||||
('A soft classifier', 2, None, '___sec20'),
|
||||
('Soft optmization problem', 2, None, '___sec21'),
|
||||
('Kernels and non-linearity', 2, None, '___sec22'),
|
||||
('The equations', 2, None, '___sec23'),
|
||||
('The problem to solve', 2, None, '___sec24'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec25'),
|
||||
('The moons example', 2, None, '___sec26'),
|
||||
'___sec14'),
|
||||
('Adding the Multiplier', 2, None, '___sec15'),
|
||||
('Setting up the Problem', 2, None, '___sec16'),
|
||||
('The problem to solve', 2, None, '___sec17'),
|
||||
('The last steps', 2, None, '___sec18'),
|
||||
('A soft classifier', 2, None, '___sec19'),
|
||||
('Soft optmization problem', 2, None, '___sec20'),
|
||||
('Kernels and non-linearity', 2, None, '___sec21'),
|
||||
('The equations', 2, None, '___sec22'),
|
||||
('The problem to solve', 2, None, '___sec23'),
|
||||
("Different kernels and Mercer's theorem", 2, None, '___sec24'),
|
||||
('The moons example', 2, None, '___sec25'),
|
||||
('Mathematical optimization of convex functions',
|
||||
2,
|
||||
None,
|
||||
'___sec27'),
|
||||
('How do we solve these problems?', 2, None, '___sec28'),
|
||||
('A simple example', 2, None, '___sec29'),
|
||||
('Back to the more realistic cases', 2, None, '___sec30')]}
|
||||
'___sec26'),
|
||||
('How do we solve these problems?', 2, None, '___sec27'),
|
||||
('A simple example', 2, None, '___sec28'),
|
||||
('Back to the more realistic cases', 2, None, '___sec29')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -457,25 +456,19 @@ where \( \eta \) is our by now well-known learning rate.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec12">Code Example </h2>
|
||||
<h2 id="___sec12">Can we code this? </h2>
|
||||
|
||||
<p>
|
||||
The equations we discussed above can be coded rather easily (the
|
||||
framework is similar to what we developed for logistic
|
||||
regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way.
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec13">Problems with the Simpler Approach </h2>
|
||||
framework is similar to what we developed for logistic regression). We
|
||||
can set up a simple case with two classes only and we want to find a
|
||||
line which separates them the best possible way.
|
||||
|
||||
<p>
|
||||
There are however problems with this approach, although it looks
|
||||
pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes.
|
||||
pretty straightforward to implement. When running a code for such a
|
||||
case we can easily end up with many diffeent lines which separate the
|
||||
two classes.
|
||||
|
||||
<p>
|
||||
For small
|
||||
@@ -487,7 +480,7 @@ at all.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec14">A better approach </h2>
|
||||
<h2 id="___sec13">A better approach </h2>
|
||||
|
||||
<p>
|
||||
A better approach is rather to try to define a large margin between
|
||||
@@ -529,7 +522,7 @@ about Lagrangian multipliers.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec15">A quick Reminder on Lagrangian Multipliers </h2>
|
||||
<h2 id="___sec14">A quick Reminder on Lagrangian Multipliers </h2>
|
||||
|
||||
<p>
|
||||
Consider a function of three independent variables \( f(x,y,z) \) . For the function \( f \) to be an
|
||||
@@ -579,7 +572,7 @@ Then \( dz \) is no longer arbitrary.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec16">Adding the Multiplier </h2>
|
||||
<h2 id="___sec15">Adding the Multiplier </h2>
|
||||
|
||||
<p>
|
||||
However, we can add to
|
||||
@@ -622,7 +615,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec17">Setting up the Problem </h2>
|
||||
<h2 id="___sec16">Setting up the Problem </h2>
|
||||
In order to solve the above problem, we define the following Lagrangian function to be minimized
|
||||
$$
|
||||
{\cal L}(\lambda,b,\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}-\sum_{i=1}^n\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)-1\right],
|
||||
@@ -663,7 +656,7 @@ When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec18">The problem to solve </h2>
|
||||
<h2 id="___sec17">The problem to solve </h2>
|
||||
|
||||
<p>
|
||||
We can rewrite
|
||||
@@ -687,7 +680,7 @@ subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vec
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec19">The last steps </h2>
|
||||
<h2 id="___sec18">The last steps </h2>
|
||||
|
||||
<p>
|
||||
Solving the above problem, yields the values of \( \lambda_i \).
|
||||
@@ -721,7 +714,7 @@ Below we discuss how to find the optimal values of \( \lambda_i \). Before we pr
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec20">A soft classifier </h2>
|
||||
<h2 id="___sec19">A soft classifier </h2>
|
||||
|
||||
<p>
|
||||
Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
|
||||
@@ -756,7 +749,7 @@ misclassifications.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec21">Soft optmization problem </h2>
|
||||
<h2 id="___sec20">Soft optmization problem </h2>
|
||||
|
||||
<p>
|
||||
This has in turn the consequences that we change our optmization problem to finding the minimum of
|
||||
@@ -810,7 +803,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec22">Kernels and non-linearity </h2>
|
||||
<h2 id="___sec21">Kernels and non-linearity </h2>
|
||||
|
||||
<p>
|
||||
The cases we have studied till now, were all characterized by two classes
|
||||
@@ -883,7 +876,7 @@ plt<span style="color: #666666">.</span>show()
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec23">The equations </h2>
|
||||
<h2 id="___sec22">The equations </h2>
|
||||
|
||||
<p>
|
||||
Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with \( x_i \) and \( y_i \) as variables)
|
||||
@@ -928,7 +921,7 @@ the trouble of performing the transformation
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec24">The problem to solve </h2>
|
||||
<h2 id="___sec23">The problem to solve </h2>
|
||||
Using our definition of the kernel We can rewrite again the Lagrangian
|
||||
$$
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
|
||||
@@ -964,7 +957,7 @@ Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec25">Different kernels and Mercer's theorem </h2>
|
||||
<h2 id="___sec24">Different kernels and Mercer's theorem </h2>
|
||||
|
||||
<p>
|
||||
There are several popular kernels being used. These are
|
||||
@@ -1002,7 +995,7 @@ in practice.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec26">The moons example </h2>
|
||||
<h2 id="___sec25">The moons example </h2>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
@@ -1198,7 +1191,7 @@ plt<span style="color: #666666">.</span>show()
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec27">Mathematical optimization of convex functions </h2>
|
||||
<h2 id="___sec26">Mathematical optimization of convex functions </h2>
|
||||
|
||||
<p>
|
||||
A mathematical (quadratic) optimization problem, or just optimization problem, has the form
|
||||
@@ -1223,7 +1216,7 @@ Convex optimization problems play a central role in applied mathematics and we r
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec28">How do we solve these problems? </h2>
|
||||
<h2 id="___sec27">How do we solve these problems? </h2>
|
||||
|
||||
<p>
|
||||
If we use Python as programming language and wish to venture beyond
|
||||
@@ -1249,7 +1242,7 @@ This will make our life much easier. You don't need t write your own optimizer.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec29">A simple example </h2>
|
||||
<h2 id="___sec28">A simple example </h2>
|
||||
|
||||
<p>
|
||||
We remind ourselves about the general problem we want to solve
|
||||
@@ -1317,7 +1310,7 @@ sol[’primal objective’]
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec30">Back to the more realistic cases </h2>
|
||||
<h2 id="___sec29">Back to the more realistic cases </h2>
|
||||
|
||||
<p>
|
||||
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the <b>slack</b> parameter \( C \) we have
|
||||
|
||||
Binary file not shown.
@@ -463,22 +463,18 @@
|
||||
"where $\\eta$ is our by now well-known learning rate. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Code Example\n",
|
||||
"## Can we code this?\n",
|
||||
"\n",
|
||||
"The equations we discussed above can be coded rather easily (the\n",
|
||||
"framework is similar to what we developed for logistic\n",
|
||||
"regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Problems with the Simpler Approach\n",
|
||||
"framework is similar to what we developed for logistic regression). We\n",
|
||||
"can set up a simple case with two classes only and we want to find a\n",
|
||||
"line which separates them the best possible way.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"There are however problems with this approach, although it looks\n",
|
||||
"pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes.\n",
|
||||
"pretty straightforward to implement. When running a code for such a\n",
|
||||
"case we can easily end up with many diffeent lines which separate the\n",
|
||||
"two classes.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"For small\n",
|
||||
|
||||
@@ -313,21 +313,18 @@ where $\eta$ is our by now well-known learning rate.
|
||||
|
||||
|
||||
!split
|
||||
===== Code Example =====
|
||||
===== Can we code this? =====
|
||||
|
||||
The equations we discussed above can be coded rather easily (the
|
||||
framework is similar to what we developed for logistic
|
||||
regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way.
|
||||
!bc pycod
|
||||
|
||||
!ec
|
||||
|
||||
!split
|
||||
===== Problems with the Simpler Approach =====
|
||||
framework is similar to what we developed for logistic regression). We
|
||||
can set up a simple case with two classes only and we want to find a
|
||||
line which separates them the best possible way.
|
||||
|
||||
|
||||
There are however problems with this approach, although it looks
|
||||
pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes.
|
||||
pretty straightforward to implement. When running a code for such a
|
||||
case we can easily end up with many diffeent lines which separate the
|
||||
two classes.
|
||||
|
||||
|
||||
For small
|
||||
|
||||
Reference in New Issue
Block a user