lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week 47

Browse Source
This commit is contained in:
mhjensen
2020-11-26 05:33:45 +01:00
parent 04dff80908
commit 7f86040fb6
16 changed files with 188 additions and 924 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week47/html/._week47-bs000.html
+1 -1
View File
@@ -157,7 +157,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 22, 2020</h4></center> <!-- date -->
<center><h4>Nov 26, 2020</h4></center> <!-- date -->
<br>
<p>
doc/pub/week47/html/._week47-bs009.html
+1 -1
View File
@@ -148,7 +148,7 @@ that could be chosen. Our objective is to find a
plane that has the maximum margin, i.e the maximum distance between
data points of both classes. Maximizing the margin distance provides
some reinforcement so that future data points can be classified with
more confidence.
more confidence. Figure 12.1 of Hastie et al is a good illustration.
<p>
What a linear classifier attempts to accomplish is to split the
doc/pub/week47/html/._week47-bs010.html
+3 -3
View File
@@ -143,13 +143,13 @@ MathJax.Hub.Config({
<p>
Let us define the function
$$
f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0,
f(x) = \boldsymbol{x}^T\boldsymbol{w}+b = 0,
$$
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
as the function that determines the line \( L \) that separates two classes (our two features), see Figure 12.1 of <a href="https://www.springer.com/gp/book/9780387848570" target="_self">Hastie et al</a>.
<p>
Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).
Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{x}^T(\boldsymbol{w}_1-\boldsymbol{x}_2)=0 \).
<p>
The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then
doc/pub/week47/html/._week47-bs011.html
+1 -1
View File
@@ -146,7 +146,7 @@ do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
$$
C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b).
C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b).
$$
<p>
doc/pub/week47/html/._week47-bs014.html
+5 -5
View File
@@ -149,7 +149,7 @@ Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to
\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.
@@ -157,12 +157,12 @@ All points are thus at a signed distance from the decision boundary defined by t
<p>
We seek thus the largest value \( M \) defined by
$$
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
or just
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
$$
If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of
@@ -172,10 +172,10 @@ y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
$$
<p>
We have thus defined our margin as the invers of the norm of
We have thus defined our margin as the inverse of the norm of
\( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as
possible margin \( M \). Before we proceed, we need to remind ourselves
about Lagrangian multipliers.
about Lagrangian multipliers and optimzation problems.
<p>
<p>
doc/pub/week47/html/._week47-bs017.html
+6 -3
View File
@@ -165,17 +165,20 @@ $$
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_self">Karush-Kuhn-Tucker</a> (KKT) condition
$$
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
<ol>
<li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
<li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
<p>
The support vectors (the points that define the margin \( M \)) are the quantities we keep in order to make predictions.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
doc/pub/week47/html/._week47-bs019.html
+3 -3
View File
@@ -149,12 +149,12 @@ $$
With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)=1,
$$
resulting in
$$
b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
b = \frac{1}{y_i}-\boldsymbol{x}_1^T\boldsymbol{w},
$$
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
@@ -164,7 +164,7 @@ $$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
y_i = \mathrm{sign}(\boldsymbol{x}_i^T\boldsymbol{w}+b).
$$
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
doc/pub/week47/html/._week47-bs020.html
+8 -5
View File
@@ -144,8 +144,11 @@ MathJax.Hub.Config({
Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
<p>
Suppose now that classes overlap in feature space, as shown in the
figure here. One way to deal with this problem before we define the
Suppose now that the two classes overlap in feature space, as shown in Figure 12.1 of
<a href="https://www.springer.com/gp/book/9780387848570" target="_self">Hastie et al</a>.
<p>
One way to deal with this problem before we define the
so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
@@ -153,17 +156,17 @@ that we allow some points to be on the wrong side of the margin.
We introduce thus the so-called <b>slack</b> variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and
modify our previous equation
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b)=1,
$$
to
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i,
y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b)=1-\xi_i,
$$
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
\( y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
<p>
doc/pub/week47/html/._week47-bs021.html
+3 -3
View File
@@ -148,7 +148,7 @@ $$
subject to
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
with the requirement \( \xi_i\geq 0 \).
@@ -177,7 +177,7 @@ $$
but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \).
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
$$
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
$$
@@ -186,7 +186,7 @@ $$
and
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
doc/pub/week47/html/week47-bs.html
+1 -1
View File
@@ -157,7 +157,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 22, 2020</h4></center> <!-- date -->
<center><h4>Nov 26, 2020</h4></center> <!-- date -->
<br>
<p>
doc/pub/week47/html/week47-reveal.html
+31 -25
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Nov 22, 2020</h4></center> <!-- date -->
<center><h4>Nov 26, 2020</h4></center> <!-- date -->
<br>
<p>
@@ -428,7 +428,7 @@ that could be chosen. Our objective is to find a
plane that has the maximum margin, i.e the maximum distance between
data points of both classes. Maximizing the margin distance provides
some reinforcement so that future data points can be classified with
more confidence.
more confidence. Figure 12.1 of Hastie et al is a good illustration.
<p>
What a linear classifier attempts to accomplish is to split the
@@ -451,14 +451,14 @@ for our data sample.
Let us define the function
<p>&nbsp;<br>
$$
f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0,
f(x) = \boldsymbol{x}^T\boldsymbol{w}+b = 0,
$$
<p>&nbsp;<br>
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
as the function that determines the line \( L \) that separates two classes (our two features), see Figure 12.1 of <a href="https://www.springer.com/gp/book/9780387848570" target="_blank">Hastie et al</a>.
<p>
Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).
Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{x}^T(\boldsymbol{w}_1-\boldsymbol{x}_2)=0 \).
<p>
The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then
@@ -480,7 +480,7 @@ misclassified points \( M \) and attempt to minimize this function
<p>&nbsp;<br>
$$
C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b).
C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b).
$$
<p>&nbsp;<br>
@@ -560,7 +560,7 @@ Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to
<p>&nbsp;<br>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
<p>&nbsp;<br>
@@ -570,14 +570,14 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value \( M \) defined by
<p>&nbsp;<br>
$$
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
<p>&nbsp;<br>
or just
<p>&nbsp;<br>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
$$
<p>&nbsp;<br>
@@ -590,10 +590,10 @@ $$
<p>&nbsp;<br>
<p>
We have thus defined our margin as the invers of the norm of
We have thus defined our margin as the inverse of the norm of
\( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as
possible margin \( M \). Before we proceed, we need to remind ourselves
about Lagrangian multipliers.
about Lagrangian multipliers and optimzation problems.
</section>
@@ -751,18 +751,21 @@ subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_blank">Karush-Kuhn-Tucker</a> (KKT) condition
<p>&nbsp;<br>
$$
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
<p>&nbsp;<br>
<ol>
<p><li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<p><li> If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
<p><li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<p><li> If \( y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
<p>
When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
<p>
The support vectors (the points that define the margin \( M \)) are the quantities we keep in order to make predictions.
</section>
@@ -809,14 +812,14 @@ $$
With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
<p>&nbsp;<br>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)=1,
$$
<p>&nbsp;<br>
resulting in
<p>&nbsp;<br>
$$
b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
b = \frac{1}{y_i}-\boldsymbol{x}_1^T\boldsymbol{w},
$$
<p>&nbsp;<br>
@@ -830,7 +833,7 @@ $$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
<p>&nbsp;<br>
$$
y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
y_i = \mathrm{sign}(\boldsymbol{x}_i^T\boldsymbol{w}+b).
$$
<p>&nbsp;<br>
@@ -845,8 +848,11 @@ Below we discuss how to find the optimal values of \( \lambda_i \). Before we pr
Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
<p>
Suppose now that classes overlap in feature space, as shown in the
figure here. One way to deal with this problem before we define the
Suppose now that the two classes overlap in feature space, as shown in Figure 12.1 of
<a href="https://www.springer.com/gp/book/9780387848570" target="_blank">Hastie et al</a>.
<p>
One way to deal with this problem before we define the
so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
@@ -855,20 +861,20 @@ We introduce thus the so-called <b>slack</b> variables \( \boldsymbol{\xi} =[\xi
modify our previous equation
<p>&nbsp;<br>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b)=1,
$$
<p>&nbsp;<br>
to
<p>&nbsp;<br>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i,
y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b)=1-\xi_i,
$$
<p>&nbsp;<br>
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
\( y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
<p>
@@ -891,7 +897,7 @@ $$
subject to
<p>&nbsp;<br>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
<p>&nbsp;<br>
@@ -930,7 +936,7 @@ but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
<p>&nbsp;<br>
$$
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
<p>&nbsp;<br>
@@ -943,7 +949,7 @@ $$
and
<p>&nbsp;<br>
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
<p>&nbsp;<br>
</section>
doc/pub/week47/html/week47-solarized.html
+31 -25
View File
@@ -100,7 +100,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 22, 2020</h4></center> <!-- date -->
<center><h4>Nov 26, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -357,7 +357,7 @@ that could be chosen. Our objective is to find a
plane that has the maximum margin, i.e the maximum distance between
data points of both classes. Maximizing the margin distance provides
some reinforcement so that future data points can be classified with
more confidence.
more confidence. Figure 12.1 of Hastie et al is a good illustration.
<p>
What a linear classifier attempts to accomplish is to split the
@@ -379,13 +379,13 @@ for our data sample.
<p>
Let us define the function
$$
f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0,
f(x) = \boldsymbol{x}^T\boldsymbol{w}+b = 0,
$$
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
as the function that determines the line \( L \) that separates two classes (our two features), see Figure 12.1 of <a href="https://www.springer.com/gp/book/9780387848570" target="_blank">Hastie et al</a>.
<p>
Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).
Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{x}^T(\boldsymbol{w}_1-\boldsymbol{x}_2)=0 \).
<p>
The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then
@@ -404,7 +404,7 @@ do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
$$
C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b).
C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b).
$$
<p>
@@ -474,7 +474,7 @@ Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to
\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.
@@ -482,12 +482,12 @@ All points are thus at a signed distance from the decision boundary defined by t
<p>
We seek thus the largest value \( M \) defined by
$$
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
or just
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
$$
If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of
@@ -497,10 +497,10 @@ y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
$$
<p>
We have thus defined our margin as the invers of the norm of
We have thus defined our margin as the inverse of the norm of
\( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as
possible margin \( M \). Before we proceed, we need to remind ourselves
about Lagrangian multipliers.
about Lagrangian multipliers and optimzation problems.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -625,17 +625,20 @@ $$
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_blank">Karush-Kuhn-Tucker</a> (KKT) condition
$$
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
<ol>
<li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
<li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
<p>
The support vectors (the points that define the margin \( M \)) are the quantities we keep in order to make predictions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -674,12 +677,12 @@ $$
With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)=1,
$$
resulting in
$$
b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
b = \frac{1}{y_i}-\boldsymbol{x}_1^T\boldsymbol{w},
$$
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
@@ -689,7 +692,7 @@ $$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
y_i = \mathrm{sign}(\boldsymbol{x}_i^T\boldsymbol{w}+b).
$$
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
@@ -703,8 +706,11 @@ Below we discuss how to find the optimal values of \( \lambda_i \). Before we pr
Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
<p>
Suppose now that classes overlap in feature space, as shown in the
figure here. One way to deal with this problem before we define the
Suppose now that the two classes overlap in feature space, as shown in Figure 12.1 of
<a href="https://www.springer.com/gp/book/9780387848570" target="_blank">Hastie et al</a>.
<p>
One way to deal with this problem before we define the
so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
@@ -712,17 +718,17 @@ that we allow some points to be on the wrong side of the margin.
We introduce thus the so-called <b>slack</b> variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and
modify our previous equation
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b)=1,
$$
to
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i,
y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b)=1-\xi_i,
$$
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
\( y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
<p>
@@ -742,7 +748,7 @@ $$
subject to
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
with the requirement \( \xi_i\geq 0 \).
@@ -771,7 +777,7 @@ $$
but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \).
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
$$
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
$$
@@ -780,7 +786,7 @@ $$
and
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
doc/pub/week47/html/week47.html
+31 -25
View File
@@ -105,7 +105,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 22, 2020</h4></center> <!-- date -->
<center><h4>Nov 26, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -362,7 +362,7 @@ that could be chosen. Our objective is to find a
plane that has the maximum margin, i.e the maximum distance between
data points of both classes. Maximizing the margin distance provides
some reinforcement so that future data points can be classified with
more confidence.
more confidence. Figure 12.1 of Hastie et al is a good illustration.
<p>
What a linear classifier attempts to accomplish is to split the
@@ -384,13 +384,13 @@ for our data sample.
<p>
Let us define the function
$$
f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0,
f(x) = \boldsymbol{x}^T\boldsymbol{w}+b = 0,
$$
as the function that determines the line \( L \) that separates two classes (our two features), see the figure here.
as the function that determines the line \( L \) that separates two classes (our two features), see Figure 12.1 of <a href="https://www.springer.com/gp/book/9780387848570" target="_blank">Hastie et al</a>.
<p>
Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{w}^T(\boldsymbol{x}_1-\boldsymbol{x}_2)=0 \).
Any point defined by \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_2 \) on the line \( L \) will satisfy \( \boldsymbol{x}^T(\boldsymbol{w}_1-\boldsymbol{x}_2)=0 \).
<p>
The signed distance \( \delta \) from any point defined by a vector \( \boldsymbol{x} \) and a point \( \boldsymbol{x}_0 \) on the line \( L \) is then
@@ -409,7 +409,7 @@ do is to define a cost function which now contains the set of all
misclassified points \( M \) and attempt to minimize this function
$$
C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b).
C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b).
$$
<p>
@@ -479,7 +479,7 @@ Thus, we wish to find a margin \( M \) with \( \boldsymbol{w} \) normalized to
\( \vert\vert \boldsymbol{w}\vert\vert =1 \) subject to the condition
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
$$
All points are thus at a signed distance from the decision boundary defined by the line \( L \). The parameters \( b \) and \( w_1 \) and \( w_2 \) define this line.
@@ -487,12 +487,12 @@ All points are thus at a signed distance from the decision boundary defined by t
<p>
We seek thus the largest value \( M \) defined by
$$
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \boldsymbol{w}\vert\vert}y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
$$
or just
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) \geq M\vert \vert \boldsymbol{w}\vert\vert \hspace{0.1cm}\forall i.
$$
If we scale the equation so that \( \vert \vert \boldsymbol{w}\vert\vert = 1/M \), we have to find the minimum of
@@ -502,10 +502,10 @@ y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
$$
<p>
We have thus defined our margin as the invers of the norm of
We have thus defined our margin as the inverse of the norm of
\( \boldsymbol{w} \). We want to minimize the norm in order to have a as large as
possible margin \( M \). Before we proceed, we need to remind ourselves
about Lagrangian multipliers.
about Lagrangian multipliers and optimzation problems.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -630,17 +630,20 @@ $$
subject to the constraints \( \lambda_i\geq 0 \) and \( \sum_i\lambda_iy_i=0 \).
We must in addition satisfy the <a href="https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" target="_blank">Karush-Kuhn-Tucker</a> (KKT) condition
$$
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b) -1\right] \hspace{0.1cm}\forall i.
$$
<ol>
<li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
<li> If \( \lambda_i > 0 \), then \( y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)=1 \) and we say that \( x_i \) is on the boundary.</li>
<li> If \( y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)> 1 \), we say \( x_i \) is not on the boundary and we set \( \lambda_i=0 \).</li>
</ol>
When \( \lambda_i > 0 \), the vectors \( \boldsymbol{x}_i \) are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin \( M \).
<p>
The support vectors (the points that define the margin \( M \)) are the quantities we keep in order to make predictions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -679,12 +682,12 @@ $$
With our vector \( \boldsymbol{w} \) we can in turn find the value of the intercept \( b \) (here in two dimensions) via
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
y_i(\boldsymbol{x}^T\boldsymbol{w}_i+b)=1,
$$
resulting in
$$
b = \frac{1}{y_i}-\boldsymbol{w}^T\boldsymbol{x}_i,
b = \frac{1}{y_i}-\boldsymbol{x}_1^T\boldsymbol{w},
$$
or if we write it out in terms of the support vectors only, with \( N_s \) being their number, we have
@@ -694,7 +697,7 @@ $$
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
$$
y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
y_i = \mathrm{sign}(\boldsymbol{x}_i^T\boldsymbol{w}+b).
$$
Below we discuss how to find the optimal values of \( \lambda_i \). Before we proceed however, we discuss now the so-called soft classifier.
@@ -708,8 +711,11 @@ Below we discuss how to find the optimal values of \( \lambda_i \). Before we pr
Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
<p>
Suppose now that classes overlap in feature space, as shown in the
figure here. One way to deal with this problem before we define the
Suppose now that the two classes overlap in feature space, as shown in Figure 12.1 of
<a href="https://www.springer.com/gp/book/9780387848570" target="_blank">Hastie et al</a>.
<p>
One way to deal with this problem before we define the
so-called <b>kernel approach</b>, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
@@ -717,17 +723,17 @@ that we allow some points to be on the wrong side of the margin.
We introduce thus the so-called <b>slack</b> variables \( \boldsymbol{\xi} =[\xi_1,x_2,\dots,x_n] \) and
modify our previous equation
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1,
y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b)=1,
$$
to
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i,
y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b)=1-\xi_i,
$$
with the requirement \( \xi_i\geq 0 \). The total violation is now \( \sum_i\xi \).
The value \( \xi_i \) in the constraint the last constraint corresponds to the amount by which the prediction
\( y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
\( y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b)=1 \) is on the wrong side of its margin. Hence by bounding the sum \( \sum_i \xi_i \),
we bound the total amount by which predictions fall on the wrong side of their margins.
<p>
@@ -747,7 +753,7 @@ $$
subject to
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b)=1-\xi_i \hspace{0.1cm}\forall i,
$$
with the requirement \( \xi_i\geq 0 \).
@@ -776,7 +782,7 @@ $$
but now subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \) and \( 0\leq\lambda_i \leq C \).
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
$$
\lambda_i\left[y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\boldsymbol{x}_1^T\boldsymbol{w}+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
$$
$$
@@ -785,7 +791,7 @@ $$
and
$$
y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\boldsymbol{x}_i^T\boldsymbol{w}+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
$$
doc/pub/week47/ipynb/ipynb-week47-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week47/ipynb/week47.ipynb
+34 -798
View File
File diff suppressed because one or more lines are too long
doc/src/week47/week47.do.txt
+29 -25
View File
@@ -235,7 +235,7 @@ that could be chosen. Our objective is to find a
plane that has the maximum margin, i.e the maximum distance between
data points of both classes. Maximizing the margin distance provides
some reinforcement so that future data points can be classified with
more confidence.
more confidence. Figure 12.1 of Hastie et al is a good illustration.
What a linear classifier attempts to accomplish is to split the
feature space into two half spaces by placing a hyperplane between the
@@ -253,13 +253,13 @@ for our data sample.
Let us define the function
!bt
\[
f(x) = \bm{w}^T\bm{x}+b = 0,
f(x) = \bm{x}^T\bm{w}+b = 0,
\]
!et
as the function that determines the line $L$ that separates two classes (our two features), see the figure here.
as the function that determines the line $L$ that separates two classes (our two features), see Figure 12.1 of "Hastie et al":"https://www.springer.com/gp/book/9780387848570".
Any point defined by $\bm{x}_i$ and $\bm{x}_2$ on the line $L$ will satisfy $\bm{w}^T(\bm{x}_1-\bm{x}_2)=0$.
Any point defined by $\bm{x}_i$ and $\bm{x}_2$ on the line $L$ will satisfy $\bm{x}^T(\bm{w}_1-\bm{x}_2)=0$.
The signed distance $\delta$ from any point defined by a vector $\bm{x}$ and a point $\bm{x}_0$ on the line $L$ is then
!bt
@@ -277,7 +277,7 @@ misclassified points $M$ and attempt to minimize this function
!bt
\[
C(\bm{w},b) = -\sum_{i\in M} y_i(\bm{w}^T\bm{x}_i+b).
C(\bm{w},b) = -\sum_{i\in M} y_i(\bm{x}^T\bm{w}_i+b).
\]
!et
@@ -344,7 +344,7 @@ $\vert\vert \bm{w}\vert\vert =1$ subject to the condition
!bt
\[
y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
y_i(\bm{x}^T\bm{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p.
\]
!et
All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $b$ and $w_1$ and $w_2$ define this line.
@@ -352,13 +352,13 @@ All points are thus at a signed distance from the decision boundary defined by t
We seek thus the largest value $M$ defined by
!bt
\[
\frac{1}{\vert \vert \bm{w}\vert\vert}y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\frac{1}{\vert \vert \bm{w}\vert\vert}y_i(\bm{x}^T\bm{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n,
\]
!et
or just
!bt
\[
y_i(\bm{w}^T\bm{x}_i+b) \geq M\vert \vert \bm{w}\vert\vert \hspace{0.1cm}\forall i.
y_i(\bm{x}^T\bm{w}_i+b) \geq M\vert \vert \bm{w}\vert\vert \hspace{0.1cm}\forall i.
\]
!et
If we scale the equation so that $\vert \vert \bm{w}\vert\vert = 1/M$, we have to find the minimum of
@@ -369,10 +369,10 @@ y_i(\bm{w}^T\bm{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
\]
!et
We have thus defined our margin as the invers of the norm of
We have thus defined our margin as the inverse of the norm of
$\bm{w}$. We want to minimize the norm in order to have a as large as
possible margin $M$. Before we proceed, we need to remind ourselves
about Lagrangian multipliers.
about Lagrangian multipliers and optimzation problems.
!split
===== A quick Reminder on Lagrangian Multipliers =====
@@ -505,12 +505,14 @@ subject to the constraints $\lambda_i\geq 0$ and $\sum_i\lambda_iy_i=0$.
We must in addition satisfy the "Karush-Kuhn-Tucker":"https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" (KKT) condition
!bt
\[
\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
\lambda_i\left[y_i(\bm{x}^T\bm{w}_i+b) -1\right] \hspace{0.1cm}\forall i.
\]
!et
o If $\lambda_i > 0$, then $y_i(\bm{w}^T\bm{x}_i+b)=1$ and we say that $x_i$ is on the boundary.
o If $y_i(\bm{w}^T\bm{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$.
When $\lambda_i > 0$, the vectors $\bm{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$.
o If $\lambda_i > 0$, then $y_i(\bm{x}^T\bm{w}_i+b)=1$ and we say that $x_i$ is on the boundary.
o If $y_i(\bm{x}^T\bm{w}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$.
When $\lambda_i > 0$, the vectors $\bm{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$.
The support vectors (the points that define the margin $M$) are the quantities we keep in order to make predictions.
!split
===== The problem to solve =====
@@ -549,13 +551,13 @@ To find the coefficients of your hyperplane we need simply to compute
With our vector $\bm{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via
!bt
\[
y_i(\bm{w}^T\bm{x}_i+b)=1,
y_i(\bm{x}^T\bm{w}_i+b)=1,
\]
!et
resulting in
!bt
\[
b = \frac{1}{y_i}-\bm{w}^T\bm{x}_i,
b = \frac{1}{y_i}-\bm{x}_1^T\bm{w},
\]
!et
or if we write it out in terms of the support vectors only, with $N_s$ being their number, we have
@@ -567,7 +569,7 @@ b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\bm{x}_i^T\bm{
With our hyperplane coefficients we can use our classifier to assign any observation by simply using
!bt
\[
y_i = \mathrm{sign}(\bm{w}^T\bm{x}_i+b).
y_i = \mathrm{sign}(\bm{x}_i^T\bm{w}+b).
\]
!et
Below we discuss how to find the optimal values of $\lambda_i$. Before we proceed however, we discuss now the so-called soft classifier.
@@ -577,8 +579,10 @@ Below we discuss how to find the optimal values of $\lambda_i$. Before we procee
Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
Suppose now that classes overlap in feature space, as shown in the
figure here. One way to deal with this problem before we define the
Suppose now that the two classes overlap in feature space, as shown in Figure 12.1 of
"Hastie et al":"https://www.springer.com/gp/book/9780387848570".
One way to deal with this problem before we define the
so-called _kernel approach_, is to allow a kind of slack in the sense
that we allow some points to be on the wrong side of the margin.
@@ -586,18 +590,18 @@ We introduce thus the so-called _slack_ variables $\bm{\xi} =[\xi_1,x_2,\dots,x_
modify our previous equation
!bt
\[
y_i(\bm{w}^T\bm{x}_i+b)=1,
y_i(\bm{x}_i^T\bm{w}+b)=1,
\]
!et
to
!bt
\[
y_i(\bm{w}^T\bm{x}_i+b)=1-\xi_i,
y_i(\bm{x}_1^T\bm{w}+b)=1-\xi_i,
\]
!et
with the requirement $\xi_i\geq 0$. The total violation is now $\sum_i\xi$.
The value $\xi_i$ in the constraint the last constraint corresponds to the amount by which the prediction
$y_i(\bm{w}^T\bm{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$,
$y_i(\bm{x}_i^T\bm{w}+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$,
we bound the total amount by which predictions fall on the wrong side of their margins.
Misclassifications occur when $\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of
@@ -616,7 +620,7 @@ This has in turn the consequences that we change our optmization problem to find
subject to
!bt
\[
y_i(\bm{w}^T\bm{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
y_i(\bm{x}_1^T\bm{w}+b)=1-\xi_i \hspace{0.1cm}\forall i,
\]
!et
with the requirement $\xi_i\geq 0$.
@@ -649,7 +653,7 @@ but now subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$ and
We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
!bt
\[
\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\lambda_i\left[y_i(\bm{x}_1^T\bm{w}+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
\]
!et
!bt
@@ -660,7 +664,7 @@ We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
and
!bt
\[
y_i(\bm{w}^T\bm{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
y_i(\bm{x}_i^T\bm{w}+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
\]
!et