lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

smaller update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-11-25 08:28:57 +01:00
parent 44cdffd893
commit afab825af6
7 changed files with 275 additions and 367 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week47/html/week47-bs.html
+3 -3
View File
@@ -89,10 +89,10 @@ doconce format html week47.do.txt --html_style=bootstrap --pygments_html_style=d
None,
'how-do-we-solve-these-problems'),
('A simple example', 2, None, 'a-simple-example'),
('Back to the more realistic cases',
('Support Vector Machines and Regression',
2,
None,
'back-to-the-more-realistic-cases'),
'support-vector-machines-and-regression'),
('Summary of course', 2, None, 'summary-of-course'),
('What? Me worry? No final exam in this course!',
2,
@@ -289,7 +289,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week47-bs024.html#mathematical-optimization-of-convex-functions" style="font-size: 80%;">Mathematical optimization of convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs025.html#how-do-we-solve-these-problems" style="font-size: 80%;">How do we solve these problems?</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs026.html#a-simple-example" style="font-size: 80%;">A simple example</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs027.html#back-to-the-more-realistic-cases" style="font-size: 80%;">Back to the more realistic cases</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs027.html#support-vector-machines-and-regression" style="font-size: 80%;">Support Vector Machines and Regression</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs028.html#summary-of-course" style="font-size: 80%;">Summary of course</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs029.html#what-me-worry-no-final-exam-in-this-course" style="font-size: 80%;">What? Me worry? No final exam in this course!</a></li>
<!-- navigation toc: --> <li><a href="._week47-bs030.html#what-is-the-link-between-artificial-intelligence-and-machine-learning-and-some-general-remarks" style="font-size: 80%;">What is the link between Artificial Intelligence and Machine Learning and some general Remarks</a></li>
doc/pub/week47/html/week47-reveal.html
+8 -25
View File
@@ -567,7 +567,7 @@ the two classes (if they are well separated from the beginning).
<p>&nbsp;<br>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n.
$$
<p>&nbsp;<br>
@@ -583,16 +583,16 @@ $$
<p>or just </p>
<p>&nbsp;<br>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i=1,2,\dots,n.
$$
<p>&nbsp;<br>
<p>If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of
\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition
\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert_2^2 \) (the norm) subject to the condition
</p>
<p>&nbsp;<br>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i=1,2,\dots,n.
$$
<p>&nbsp;<br>
@@ -791,8 +791,8 @@ y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x
$$
<p>&nbsp;<br>
<p>subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
<p>subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda}^T =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldsymbol{y}^T=[y_1,y_2,\dots,y_n] \).
</p>
</section>
@@ -1545,26 +1545,9 @@ sol[<span style="color: #a61717; background-color: #e3d2d2">’</span>primal obj
</section>
<section>
<h2 id="back-to-the-more-realistic-cases">Back to the more realistic cases </h2>
<h2 id="support-vector-machines-and-regression">Support Vector Machines and Regression </h2>
<p>We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the <b>slack</b> parameter \( C \) we have</p>
<p>&nbsp;<br>
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
$$
<p>&nbsp;<br>
<p>subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
</p>
<b>code will be added</b>
<p>Material may be added if of interest. See Bishop chapter 7.1 for a discussion.</p>
</section>
<section>
doc/pub/week47/html/week47-solarized.html
+10 -25
View File
@@ -116,10 +116,10 @@ div.toc p,a {
None,
'how-do-we-solve-these-problems'),
('A simple example', 2, None, 'a-simple-example'),
('Back to the more realistic cases',
('Support Vector Machines and Regression',
2,
None,
'back-to-the-more-realistic-cases'),
'support-vector-machines-and-regression'),
('Summary of course', 2, None, 'summary-of-course'),
('What? Me worry? No final exam in this course!',
2,
@@ -632,7 +632,7 @@ the two classes (if they are well separated from the beginning).
</p>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n.
$$
<p>All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line. </p>
@@ -644,14 +644,14 @@ $$
<p>or just </p>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i=1,2,\dots,n.
$$
<p>If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of
\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition
\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert_2^2 \) (the norm) subject to the condition
</p>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i=1,2,\dots,n.
$$
<p>We have thus defined our margin as the invers of the norm of
@@ -807,8 +807,8 @@ y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
<p>subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
<p>subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda}^T =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldsymbol{y}^T=[y_1,y_2,\dots,y_n] \).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1488,24 +1488,9 @@ sol[<span style="color: #a61717; background-color: #e3d2d2">’</span>primal obj
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="back-to-the-more-realistic-cases">Back to the more realistic cases </h2>
<h2 id="support-vector-machines-and-regression">Support Vector Machines and Regression </h2>
<p>We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the <b>slack</b> parameter \( C \) we have</p>
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
$$
<p>subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
</p>
<b>code will be added</b>
<p>Material may be added if of interest. See Bishop chapter 7.1 for a discussion.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="summary-of-course">Summary of course </h2>
doc/pub/week47/html/week47.html
+10 -25
View File
@@ -193,10 +193,10 @@ div.toc p,a {
None,
'how-do-we-solve-these-problems'),
('A simple example', 2, None, 'a-simple-example'),
('Back to the more realistic cases',
('Support Vector Machines and Regression',
2,
None,
'back-to-the-more-realistic-cases'),
'support-vector-machines-and-regression'),
('Summary of course', 2, None, 'summary-of-course'),
('What? Me worry? No final exam in this course!',
2,
@@ -709,7 +709,7 @@ the two classes (if they are well separated from the beginning).
</p>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n.
$$
<p>All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line. </p>
@@ -721,14 +721,14 @@ $$
<p>or just </p>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i=1,2,\dots,n.
$$
<p>If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of
\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert \) (the norm) subject to the condition
\( \boldsymbol{w}^T\boldsymbol{w}=\vert \vert \boldsymbol{w}\vert\vert_2^2 \) (the norm) subject to the condition
</p>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i=1,2,\dots,n.
$$
<p>We have thus defined our margin as the invers of the norm of
@@ -884,8 +884,8 @@ y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 & y_ny_2\boldsymbol{x}_n^T\boldsymbol{x
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{1}\boldsymbol{\lambda},
$$
<p>subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
<p>subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda}^T =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldsymbol{y}^T=[y_1,y_2,\dots,y_n] \).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1565,24 +1565,9 @@ sol[’primal objective’]
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="back-to-the-more-realistic-cases">Back to the more realistic cases </h2>
<h2 id="support-vector-machines-and-regression">Support Vector Machines and Regression </h2>
<p>We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the <b>slack</b> parameter \( C \) we have</p>
$$
\frac{1}{2} \boldsymbol{\lambda}^T\begin{bmatrix} y_1y_1K(\boldsymbol{x}_1,\boldsymbol{x}_1) & y_1y_2K(\boldsymbol{x}_1,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_1,\boldsymbol{x}_n) \\
y_2y_1K(\boldsymbol{x}_2,\boldsymbol{x}_1) & y_2y_2K(\boldsymbol{x}_2,\boldsymbol{x}_2) & \dots & \dots & y_1y_nK(\boldsymbol{x}_2,\boldsymbol{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\boldsymbol{x}_n,\boldsymbol{x}_1) & y_ny_2K(\boldsymbol{x}_n\boldsymbol{x}_2) & \dots & \dots & y_ny_nK(\boldsymbol{x}_n,\boldsymbol{x}_n) \\
\end{bmatrix}\boldsymbol{\lambda}-\mathbb{I}\boldsymbol{\lambda},
$$
<p>subject to \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \). Here we defined the vectors \( \boldsymbol{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n] \) and
\( \boldsymbol{y}=[y_1,y_2,\dots,y_n] \).
With the slack constants this leads to the additional constraint \( 0\leq \lambda_i \leq C \).
</p>
<b>code will be added</b>
<p>Material may be added if of interest. See Bishop chapter 7.1 for a discussion.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="summary-of-course">Summary of course </h2>
doc/pub/week47/ipynb/ipynb-week47-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week47/ipynb/week47.ipynb
+235 -266
View File
File diff suppressed because it is too large Load Diff
doc/src/week47/week47.do.txt
+9 -23
View File
@@ -326,7 +326,7 @@ $\vert\vert \bm{w}\vert\vert =1$ subject to the condition
!bt
\[
y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n.
\]
!et
All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $b$ and $w_1$ and $w_2$ define this line.
@@ -340,14 +340,14 @@ We seek thus the largest value $M$ defined by
or just
!bt
\[
y_i(\bm{w}^T\bm{x}_i+b) \geq M\vert \vert \bm{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\bm{w}^T\bm{x}_i+b) \geq M\vert \vert \bm{w}\vert\vert \hspace{0.1cm}\forall i=1,2,\dots,n.
\]
!et
If we scale the equation so that $\vert \vert \bm{w}\vert\vert = 1/M$, we have to find the minimum of
$\bm{w}^T\bm{w}=\vert \vert \bm{w}\vert\vert$ (the norm) subject to the condition
$\bm{w}^T\bm{w}=\vert \vert \bm{w}\vert\vert_2^2$ (the norm) subject to the condition
!bt
\[
y_i(\bm{w}^T\bm{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
y_i(\bm{w}^T\bm{x}_i+b) \geq 1 \hspace{0.1cm}\forall i=1,2,\dots,n.
\]
!et
@@ -515,8 +515,8 @@ y_ny_1\bm{x}_n^T\bm{x}_1 & y_ny_2\bm{x}_n^T\bm{x}_2 & \dots & \dots & y_ny_n\bm{
\end{bmatrix}\bm{\lambda}-\mathbb{1}\bm{\lambda},
\]
!et
subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and
$\bm{y}=[y_1,y_2,\dots,y_n]$.
subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda}^T =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and
$\bm{y}^T=[y_1,y_2,\dots,y_n]$.
!split
@@ -1136,25 +1136,11 @@ sol[’x’]
sol[’primal objective’]
!ec
!split
===== Back to the more realistic cases =====
===== Support Vector Machines and Regression =====
We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the _slack_ parameter $C$ we have
!bt
\[
\frac{1}{2} \bm{\lambda}^T\begin{bmatrix} y_1y_1K(\bm{x}_1,\bm{x}_1) & y_1y_2K(\bm{x}_1,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_1,\bm{x}_n) \\
y_2y_1K(\bm{x}_2,\bm{x}_1) & y_2y_2K(\bm{x}_2,\bm{x}_2) & \dots & \dots & y_1y_nK(\bm{x}_2,\bm{x}_n) \\
\dots & \dots & \dots & \dots & \dots \\
\dots & \dots & \dots & \dots & \dots \\
y_ny_1K(\bm{x}_n,\bm{x}_1) & y_ny_2K(\bm{x}_n\bm{x}_2) & \dots & \dots & y_ny_nK(\bm{x}_n,\bm{x}_n) \\
\end{bmatrix}\bm{\lambda}-\mathbb{I}\bm{\lambda},
\]
!et
subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =[\lambda_1,\lambda_2,\dots,\lambda_n]$ and
$\bm{y}=[y_1,y_2,\dots,y_n]$.
With the slack constants this leads to the additional constraint $0\leq \lambda_i \leq C$.
_code will be added_
Material may be added if of interest. See Bishop chapter 7.1 for a discussion.
!split
===== Summary of course =====