Files
FYS-STK4155/doc/pub/week47/html/._week47-bs067.html
T
Morten Hjorth-Jensen dd5df5a0e1 added figures
2021-11-25 08:30:39 +01:00

406 lines
25 KiB
HTML

<!--
HTML file automatically generated from DocOnce source
(https://github.com/doconce/doconce/)
doconce format html week47.do.txt --html_style=bootstrap --pygments_html_style=default --html_admon=bootstrap_panel --html_output=week47-bs --no_mako
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 47: Support Vector Machines and Summary of Course">
<title>Week 47: Support Vector Machines and Summary of Course</title>
<!-- Bootstrap style: bootstrap -->
<!-- doconce format html week47.do.txt --html_style=bootstrap --pygments_html_style=default --html_admon=bootstrap_panel --html_output=week47-bs --no_mako -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Overview of week 47', 2, None, 'overview-of-week-47'),
('Support Vector Machines, overarching aims',
2,
None,
'support-vector-machines-overarching-aims'),
('Hyperplanes and all that', 2, None, 'hyperplanes-and-all-that'),
('What is a hyperplane?', 2, None, 'what-is-a-hyperplane'),
('A $p$-dimensional space of features',
2,
None,
'a-p-dimensional-space-of-features'),
('The two-dimensional case', 2, None, 'the-two-dimensional-case'),
('Getting into the details', 2, None, 'getting-into-the-details'),
('First attempt at a minimization approach',
2,
None,
'first-attempt-at-a-minimization-approach'),
('Solving the equations', 2, None, 'solving-the-equations'),
('Problems with the Simpler Approach',
2,
None,
'problems-with-the-simpler-approach'),
('A better approach', 2, None, 'a-better-approach'),
('A quick Reminder on Lagrangian Multipliers',
2,
None,
'a-quick-reminder-on-lagrangian-multipliers'),
('Adding the Multiplier', 2, None, 'adding-the-multiplier'),
('Setting up the Problem', 2, None, 'setting-up-the-problem'),
('The problem to solve', 2, None, 'the-problem-to-solve'),
('The last steps', 2, None, 'the-last-steps'),
('A soft classifier', 2, None, 'a-soft-classifier'),
('Soft optmization problem', 2, None, 'soft-optmization-problem'),
('Kernels and non-linearity',
2,
None,
'kernels-and-non-linearity'),
('The equations', 2, None, 'the-equations'),
('The problem to solve', 2, None, 'the-problem-to-solve'),
("Different kernels and Mercer's theorem",
2,
None,
'different-kernels-and-mercer-s-theorem'),
('The moons example', 2, None, 'the-moons-example'),
('Mathematical optimization of convex functions',
2,
None,
'mathematical-optimization-of-convex-functions'),
('How do we solve these problems?',
2,
None,
'how-do-we-solve-these-problems'),
('A simple example', 2, None, 'a-simple-example'),
('Support Vector Machines and Regression',
2,
None,
'support-vector-machines-and-regression'),
('Summary of course', 2, None, 'summary-of-course'),
('What? Me worry? No final exam in this course!',
2,
None,
'what-me-worry-no-final-exam-in-this-course'),
('What is the link between Artificial Intelligence and Machine '
'Learning and some general Remarks',
2,
None,
'what-is-the-link-between-artificial-intelligence-and-machine-learning-and-some-general-remarks'),
('Going back to the beginning of the semester',
2,
None,
'going-back-to-the-beginning-of-the-semester'),
('Not so sharp distinctions',
2,
None,
'not-so-sharp-distinctions'),
('Topics we have covered this year',
2,
None,
'topics-we-have-covered-this-year'),
('Statistical analysis and optimization of data',
2,
None,
'statistical-analysis-and-optimization-of-data'),
('Machine learning', 2, None, 'machine-learning'),
('Learning outcomes and overarching aims of this course',
2,
None,
'learning-outcomes-and-overarching-aims-of-this-course'),
('Perspective on Machine Learning',
2,
None,
'perspective-on-machine-learning'),
('Machine Learning Research',
2,
None,
'machine-learning-research'),
('Starting your Machine Learning Project',
2,
None,
'starting-your-machine-learning-project'),
('Choose a Model and Algorithm',
2,
None,
'choose-a-model-and-algorithm'),
('Preparing Your Data', 2, None, 'preparing-your-data'),
('Which Activation and Weights to Choose in Neural Networks',
2,
None,
'which-activation-and-weights-to-choose-in-neural-networks'),
('Optimization Methods and Hyperparameters',
2,
None,
'optimization-methods-and-hyperparameters'),
('Resampling', 2, None, 'resampling'),
('Other courses on Data science and Machine Learning at UiO',
2,
None,
'other-courses-on-data-science-and-machine-learning-at-uio'),
('Additional courses of interest',
2,
None,
'additional-courses-of-interest'),
("What's the future like?", 2, None, 'what-s-the-future-like'),
('Types of Machine Learning, a repetition',
2,
None,
'types-of-machine-learning-a-repetition'),
('Why Boltzmann machines?', 2, None, 'why-boltzmann-machines'),
('Boltzmann Machines', 2, None, 'boltzmann-machines'),
('Some similarities and differences from DNNs',
2,
None,
'some-similarities-and-differences-from-dnns'),
('Boltzmann machines (BM)', 2, None, 'boltzmann-machines-bm'),
('A standard BM setup', 2, None, 'a-standard-bm-setup'),
('The structure of the RBM network',
2,
None,
'the-structure-of-the-rbm-network'),
('The network', 2, None, 'the-network'),
('Goals', 2, None, 'goals'),
('Joint distribution', 2, None, 'joint-distribution'),
('Network Elements, the energy function',
2,
None,
'network-elements-the-energy-function'),
('Defining different types of RBMs',
2,
None,
'defining-different-types-of-rbms'),
('More about RBMs', 2, None, 'more-about-rbms'),
('Autoencoders: Overarching view',
2,
None,
'autoencoders-overarching-view'),
('Bayesian Machine Learning',
2,
None,
'bayesian-machine-learning'),
('Reinforcement Learning', 2, None, 'reinforcement-learning'),
('Transfer learning', 2, None, 'transfer-learning'),
('Adversarial learning', 2, None, 'adversarial-learning'),
('Dual learning', 2, None, 'dual-learning'),
('Distributed machine learning',
2,
None,
'distributed-machine-learning'),
('Meta learning', 2, None, 'meta-learning'),
('The Challenges Facing Machine Learning',
2,
None,
'the-challenges-facing-machine-learning'),
('Explainable machine learning',
2,
None,
'explainable-machine-learning'),
('Quantum machine learning', 2, None, 'quantum-machine-learning'),
('Quantum machine learning algorithms based on linear algebra',
2,
None,
'quantum-machine-learning-algorithms-based-on-linear-algebra'),
('Quantum reinforcement learning',
2,
None,
'quantum-reinforcement-learning'),
('Quantum deep learning', 2, None, 'quantum-deep-learning'),
('Social machine learning', 2, None, 'social-machine-learning'),
('The last words?', 2, None, 'the-last-words'),
('Best wishes to you all and thanks so much for your heroic '
'efforts this semester',
2,
None,
'best-wishes-to-you-all-and-thanks-so-much-for-your-heroic-efforts-this-semester')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="week47-bs.html">Week 47: Support Vector Machines and Summary of Course</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week47-bs001.html#overview-of-week-47" style="font-size: 80%;">Overview of week 47</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs002.html#support-vector-machines-overarching-aims" style="font-size: 80%;">Support Vector Machines, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs003.html#hyperplanes-and-all-that" style="font-size: 80%;">Hyperplanes and all that</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs004.html#what-is-a-hyperplane" style="font-size: 80%;">What is a hyperplane?</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs005.html#a-p-dimensional-space-of-features" style="font-size: 80%;">A \( p \)-dimensional space of features</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs006.html#the-two-dimensional-case" style="font-size: 80%;">The two-dimensional case</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs007.html#getting-into-the-details" style="font-size: 80%;">Getting into the details</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs008.html#first-attempt-at-a-minimization-approach" style="font-size: 80%;">First attempt at a minimization approach</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs009.html#solving-the-equations" style="font-size: 80%;">Solving the equations</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs010.html#problems-with-the-simpler-approach" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs011.html#a-better-approach" style="font-size: 80%;">A better approach</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs012.html#a-quick-reminder-on-lagrangian-multipliers" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs013.html#adding-the-multiplier" style="font-size: 80%;">Adding the Multiplier</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs014.html#setting-up-the-problem" style="font-size: 80%;">Setting up the Problem</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs021.html#the-problem-to-solve" style="font-size: 80%;">The problem to solve</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs016.html#the-last-steps" style="font-size: 80%;">The last steps</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs017.html#a-soft-classifier" style="font-size: 80%;">A soft classifier</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs018.html#soft-optmization-problem" style="font-size: 80%;">Soft optmization problem</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs019.html#kernels-and-non-linearity" style="font-size: 80%;">Kernels and non-linearity</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs020.html#the-equations" style="font-size: 80%;">The equations</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs021.html#the-problem-to-solve" style="font-size: 80%;">The problem to solve</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs022.html#different-kernels-and-mercer-s-theorem" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs023.html#the-moons-example" style="font-size: 80%;">The moons example</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs024.html#mathematical-optimization-of-convex-functions" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs025.html#how-do-we-solve-these-problems" style="font-size: 80%;">How do we solve these problems?</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs026.html#a-simple-example" style="font-size: 80%;">A simple example</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs027.html#support-vector-machines-and-regression" style="font-size: 80%;">Support Vector Machines and Regression</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs028.html#summary-of-course" style="font-size: 80%;">Summary of course</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs029.html#what-me-worry-no-final-exam-in-this-course" style="font-size: 80%;">What? Me worry? No final exam in this course!</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs030.html#what-is-the-link-between-artificial-intelligence-and-machine-learning-and-some-general-remarks" style="font-size: 80%;">What is the link between Artificial Intelligence and Machine Learning and some general Remarks</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs031.html#going-back-to-the-beginning-of-the-semester" style="font-size: 80%;">Going back to the beginning of the semester</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs032.html#not-so-sharp-distinctions" style="font-size: 80%;">Not so sharp distinctions</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs033.html#topics-we-have-covered-this-year" style="font-size: 80%;">Topics we have covered this year</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs034.html#statistical-analysis-and-optimization-of-data" style="font-size: 80%;">Statistical analysis and optimization of data</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs035.html#machine-learning" style="font-size: 80%;">Machine learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs036.html#learning-outcomes-and-overarching-aims-of-this-course" style="font-size: 80%;">Learning outcomes and overarching aims of this course</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs037.html#perspective-on-machine-learning" style="font-size: 80%;">Perspective on Machine Learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs038.html#machine-learning-research" style="font-size: 80%;">Machine Learning Research</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs039.html#starting-your-machine-learning-project" style="font-size: 80%;">Starting your Machine Learning Project</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs040.html#choose-a-model-and-algorithm" style="font-size: 80%;">Choose a Model and Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs041.html#preparing-your-data" style="font-size: 80%;">Preparing Your Data</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs042.html#which-activation-and-weights-to-choose-in-neural-networks" style="font-size: 80%;">Which Activation and Weights to Choose in Neural Networks</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs043.html#optimization-methods-and-hyperparameters" style="font-size: 80%;">Optimization Methods and Hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs044.html#resampling" style="font-size: 80%;">Resampling</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs045.html#other-courses-on-data-science-and-machine-learning-at-uio" style="font-size: 80%;">Other courses on Data science and Machine Learning at UiO</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs046.html#additional-courses-of-interest" style="font-size: 80%;">Additional courses of interest</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs047.html#what-s-the-future-like" style="font-size: 80%;">What's the future like?</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs048.html#types-of-machine-learning-a-repetition" style="font-size: 80%;">Types of Machine Learning, a repetition</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs049.html#why-boltzmann-machines" style="font-size: 80%;">Why Boltzmann machines?</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs050.html#boltzmann-machines" style="font-size: 80%;">Boltzmann Machines</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs051.html#some-similarities-and-differences-from-dnns" style="font-size: 80%;">Some similarities and differences from DNNs</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs052.html#boltzmann-machines-bm" style="font-size: 80%;">Boltzmann machines (BM)</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs053.html#a-standard-bm-setup" style="font-size: 80%;">A standard BM setup</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs054.html#the-structure-of-the-rbm-network" style="font-size: 80%;">The structure of the RBM network</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs055.html#the-network" style="font-size: 80%;">The network</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs056.html#goals" style="font-size: 80%;">Goals</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs057.html#joint-distribution" style="font-size: 80%;">Joint distribution</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs058.html#network-elements-the-energy-function" style="font-size: 80%;">Network Elements, the energy function</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs059.html#defining-different-types-of-rbms" style="font-size: 80%;">Defining different types of RBMs</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs060.html#more-about-rbms" style="font-size: 80%;">More about RBMs</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs061.html#autoencoders-overarching-view" style="font-size: 80%;">Autoencoders: Overarching view</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs062.html#bayesian-machine-learning" style="font-size: 80%;">Bayesian Machine Learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs063.html#reinforcement-learning" style="font-size: 80%;">Reinforcement Learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs064.html#transfer-learning" style="font-size: 80%;">Transfer learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs065.html#adversarial-learning" style="font-size: 80%;">Adversarial learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs066.html#dual-learning" style="font-size: 80%;">Dual learning</a></li>
<!-- navigation toc: --> <li><a href="#distributed-machine-learning" style="font-size: 80%;">Distributed machine learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs068.html#meta-learning" style="font-size: 80%;">Meta learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs069.html#the-challenges-facing-machine-learning" style="font-size: 80%;">The Challenges Facing Machine Learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs070.html#explainable-machine-learning" style="font-size: 80%;">Explainable machine learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs071.html#quantum-machine-learning" style="font-size: 80%;">Quantum machine learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs072.html#quantum-machine-learning-algorithms-based-on-linear-algebra" style="font-size: 80%;">Quantum machine learning algorithms based on linear algebra</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs073.html#quantum-reinforcement-learning" style="font-size: 80%;">Quantum reinforcement learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs074.html#quantum-deep-learning" style="font-size: 80%;">Quantum deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs075.html#social-machine-learning" style="font-size: 80%;">Social machine learning</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs076.html#the-last-words" style="font-size: 80%;">The last words?</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs077.html#best-wishes-to-you-all-and-thanks-so-much-for-your-heroic-efforts-this-semester" style="font-size: 80%;">Best wishes to you all and thanks so much for your heroic efforts this semester</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0067"></a>
<!-- !split -->
<h2 id="distributed-machine-learning" class="anchor">Distributed machine learning </h2>
<p>Distributed computation will speed up machine learning algorithms,
significantly improve their efficiency, and thus enlarge their
application. When distributed meets machine learning, more than just
implementing the machine learning algorithms in parallel is required.
</p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._week47-bs066.html">&laquo;</a></li>
<li><a href="._week47-bs000.html">1</a></li>
<li><a href="">...</a></li>
<li><a href="._week47-bs059.html">60</a></li>
<li><a href="._week47-bs060.html">61</a></li>
<li><a href="._week47-bs061.html">62</a></li>
<li><a href="._week47-bs062.html">63</a></li>
<li><a href="._week47-bs063.html">64</a></li>
<li><a href="._week47-bs064.html">65</a></li>
<li><a href="._week47-bs065.html">66</a></li>
<li><a href="._week47-bs066.html">67</a></li>
<li class="active"><a href="._week47-bs067.html">68</a></li>
<li><a href="._week47-bs068.html">69</a></li>
<li><a href="._week47-bs069.html">70</a></li>
<li><a href="._week47-bs070.html">71</a></li>
<li><a href="._week47-bs071.html">72</a></li>
<li><a href="._week47-bs072.html">73</a></li>
<li><a href="._week47-bs073.html">74</a></li>
<li><a href="._week47-bs074.html">75</a></li>
<li><a href="._week47-bs075.html">76</a></li>
<li><a href="._week47-bs076.html">77</a></li>
<li><a href="">...</a></li>
<li><a href="._week47-bs077.html">78</a></li>
<li><a href="._week47-bs068.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="https://..."><img width="250" align=right src="https://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>