lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added program

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-11-19 05:47:05 +01:00
parent 9ab3ced48f
commit e524f10d01
37 changed files with 2246 additions and 2018 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week46/html/._week46-bs027.html
+49 -61
View File
@@ -38,6 +38,10 @@ doconce format html week46.do.txt --html_style=bootstrap --pygments_html_style=d
{'highest level': 2,
'sections': [('Overview of week 46', 2, None, 'overview-of-week-46'),
('Friday', 2, None, 'friday'),
('Workshop plan Friday November 19 and the rest of the lecture',
2,
None,
'workshop-plan-friday-november-19-and-the-rest-of-the-lecture'),
('Support Vector Machines, overarching aims',
2,
None,
@@ -131,33 +135,34 @@ MathJax.Hub.Config({
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week46-bs001.html#overview-of-week-46" style="font-size: 80%;">Overview of week 46</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs002.html#friday" style="font-size: 80%;">Friday</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs003.html#support-vector-machines-overarching-aims" style="font-size: 80%;">Support Vector Machines, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs004.html#hyperplanes-and-all-that" style="font-size: 80%;">Hyperplanes and all that</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs005.html#what-is-a-hyperplane" style="font-size: 80%;">What is a hyperplane?</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs006.html#a-p-dimensional-space-of-features" style="font-size: 80%;">A \( p \)-dimensional space of features</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs007.html#the-two-dimensional-case" style="font-size: 80%;">The two-dimensional case</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs008.html#getting-into-the-details" style="font-size: 80%;">Getting into the details</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs009.html#first-attempt-at-a-minimization-approach" style="font-size: 80%;">First attempt at a minimization approach</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs010.html#solving-the-equations" style="font-size: 80%;">Solving the equations</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs011.html#code-example" style="font-size: 80%;">Code Example</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs012.html#problems-with-the-simpler-approach" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs013.html#a-better-approach" style="font-size: 80%;">A better approach</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs014.html#a-quick-reminder-on-lagrangian-multipliers" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs015.html#adding-the-multiplier" style="font-size: 80%;">Adding the Multiplier</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs016.html#setting-up-the-problem" style="font-size: 80%;">Setting up the Problem</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs023.html#the-problem-to-solve" style="font-size: 80%;">The problem to solve</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs018.html#the-last-steps" style="font-size: 80%;">The last steps</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs019.html#a-soft-classifier" style="font-size: 80%;">A soft classifier</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs020.html#soft-optmization-problem" style="font-size: 80%;">Soft optmization problem</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs021.html#kernels-and-non-linearity" style="font-size: 80%;">Kernels and non-linearity</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs022.html#the-equations" style="font-size: 80%;">The equations</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs023.html#the-problem-to-solve" style="font-size: 80%;">The problem to solve</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs024.html#different-kernels-and-mercer-s-theorem" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs025.html#the-moons-example" style="font-size: 80%;">The moons example</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs026.html#mathematical-optimization-of-convex-functions" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
<!-- navigation toc: --> <li><a href="#how-do-we-solve-these-problems" style="font-size: 80%;">How do we solve these problems?</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs028.html#a-simple-example" style="font-size: 80%;">A simple example</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs029.html#back-to-the-more-realistic-cases" style="font-size: 80%;">Back to the more realistic cases</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs003.html#workshop-plan-friday-november-19-and-the-rest-of-the-lecture" style="font-size: 80%;">Workshop plan Friday November 19 and the rest of the lecture</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs004.html#support-vector-machines-overarching-aims" style="font-size: 80%;">Support Vector Machines, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs005.html#hyperplanes-and-all-that" style="font-size: 80%;">Hyperplanes and all that</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs006.html#what-is-a-hyperplane" style="font-size: 80%;">What is a hyperplane?</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs007.html#a-p-dimensional-space-of-features" style="font-size: 80%;">A \( p \)-dimensional space of features</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs008.html#the-two-dimensional-case" style="font-size: 80%;">The two-dimensional case</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs009.html#getting-into-the-details" style="font-size: 80%;">Getting into the details</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs010.html#first-attempt-at-a-minimization-approach" style="font-size: 80%;">First attempt at a minimization approach</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs011.html#solving-the-equations" style="font-size: 80%;">Solving the equations</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs012.html#code-example" style="font-size: 80%;">Code Example</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs013.html#problems-with-the-simpler-approach" style="font-size: 80%;">Problems with the Simpler Approach</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs014.html#a-better-approach" style="font-size: 80%;">A better approach</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs015.html#a-quick-reminder-on-lagrangian-multipliers" style="font-size: 80%;">A quick Reminder on Lagrangian Multipliers</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs016.html#adding-the-multiplier" style="font-size: 80%;">Adding the Multiplier</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs017.html#setting-up-the-problem" style="font-size: 80%;">Setting up the Problem</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs024.html#the-problem-to-solve" style="font-size: 80%;">The problem to solve</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs019.html#the-last-steps" style="font-size: 80%;">The last steps</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs020.html#a-soft-classifier" style="font-size: 80%;">A soft classifier</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs021.html#soft-optmization-problem" style="font-size: 80%;">Soft optmization problem</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs022.html#kernels-and-non-linearity" style="font-size: 80%;">Kernels and non-linearity</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs023.html#the-equations" style="font-size: 80%;">The equations</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs024.html#the-problem-to-solve" style="font-size: 80%;">The problem to solve</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs025.html#different-kernels-and-mercer-s-theorem" style="font-size: 80%;">Different kernels and Mercer's theorem</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs026.html#the-moons-example" style="font-size: 80%;">The moons example</a></li>
<!-- navigation toc: --> <li><a href="#mathematical-optimization-of-convex-functions" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs028.html#how-do-we-solve-these-problems" style="font-size: 80%;">How do we solve these problems?</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs029.html#a-simple-example" style="font-size: 80%;">A simple example</a></li>
<!-- navigation toc: --> <li><a href="._week46-bs030.html#back-to-the-more-realistic-cases" style="font-size: 80%;">Back to the more realistic cases</a></li>
</ul>
</li>
@@ -169,44 +174,26 @@ MathJax.Hub.Config({
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0027"></a>
<!-- !split -->
<h2 id="how-do-we-solve-these-problems" class="anchor">How do we solve these problems? </h2>
<h2 id="mathematical-optimization-of-convex-functions" class="anchor">Mathematical optimization of convex functions </h2>
<p>If we use Python as programming language and wish to venture beyond
<b>scikit-learn</b>, <b>tensorflow</b> and similar software which makes our
lives so much easier, we need to dive into the wonderful world of
quadratic programming. We can, if we wish, solve the minimization
problem using say standard gradient methods or conjugate gradient
methods. However, these methods tend to exhibit a rather slow
converge. So, welcome to the promised land of quadratic programming.
<p>A mathematical (quadratic) optimization problem, or just optimization problem, has the form</p>
$$
\begin{align*}
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\boldsymbol{\lambda}^T\boldsymbol{P}\boldsymbol{\lambda}+\boldsymbol{q}^T\boldsymbol{\lambda},\\ \nonumber
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \boldsymbol{G}\boldsymbol{\lambda} \preceq \boldsymbol{h} \wedge \boldsymbol{A}\boldsymbol{\lambda}=f.
\end{align*}
$$
<p>subject to some constraints for say a selected set \( i=1,2,\dots, n \).
In our case we are optimizing with respect to the Lagrangian multipliers \( \lambda_i \), and the
vector \( \boldsymbol{\lambda}=[\lambda_1, \lambda_2,\dots, \lambda_n] \) is the optimization variable we are dealing with.
</p>
<p>The functions we need are contained in the quadratic programming package <b>CVXOPT</b> and we need to import it together with <b>numpy</b> as</p>
<p>In our case we are particularly interested in a class of optimization problems called convex optmization problems.
In our discussion on gradient descent methods we discussed at length the definition of a convex function.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">cvxopt</span>
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>This will make our life much easier. You don't need t write your own optimizer.</p>
<p>Convex optimization problems play a central role in applied mathematics and we recommend strongly <a href="http://web.stanford.edu/~boyd/cvxbook/" target="_self">Boyd and Vandenberghe's text on the topics</a>.</p>
<p>
<!-- navigation buttons at the bottom of the page -->
@@ -225,6 +212,7 @@ converge. So, welcome to the promised land of quadratic programming.
<li class="active"><a href="._week46-bs027.html">28</a></li>
<li><a href="._week46-bs028.html">29</a></li>
<li><a href="._week46-bs029.html">30</a></li>
<li><a href="._week46-bs030.html">31</a></li>
<li><a href="._week46-bs028.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->