update week 47

doc/src/week47/week47.do.txt

@@ -235,7 +235,7 @@ that could be chosen. Our objective is to find a
 plane that has the maximum margin, i.e the maximum distance between
 data points of both classes. Maximizing the margin distance provides
 some reinforcement so that future data points can be classified with
-more confidence.
+more confidence. Figure 12.1 of Hastie et al is a good illustration.
 
 What a linear classifier attempts to accomplish is to split the
 feature space into two half spaces by placing a hyperplane between the
@@ -253,13 +253,13 @@ for our data sample.
 Let us define the function
 !bt
 \[
-f(x) = \bm{w}^T\bm{x}+b = 0,
+f(x) = \bm{x}^T\bm{w}+b = 0,
 \]
 !et
-as the function that determines the line $L$ that separates two classes (our two features), see the figure here. 
+as the function that determines the line $L$ that separates two classes (our two features), see Figure 12.1 of "Hastie et al":"https://www.springer.com/gp/book/9780387848570".
 
 
-Any point defined by $\bm{x}_i$ and $\bm{x}_2$ on the line $L$ will satisfy $\bm{w}^T(\bm{x}_1-\bm{x}_2)=0$. 
+Any point defined by $\bm{x}_i$ and $\bm{x}_2$ on the line $L$ will satisfy $\bm{x}^T(\bm{w}_1-\bm{x}_2)=0$. 
 
 The signed distance $\delta$ from any point defined by a vector $\bm{x}$ and a point $\bm{x}_0$ on the line $L$ is then
 !bt
@@ -277,7 +277,7 @@ misclassified points $M$ and attempt to minimize this function
 
 !bt
 \[
-C(\bm{w},b) = -\sum_{i\in M} y_i(\bm{w}^T\bm{x}_i+b).
+C(\bm{w},b) = -\sum_{i\in M} y_i(\bm{x}^T\bm{w}_i+b).
 \]
 !et
 
@@ -344,7 +344,7 @@ $\vert\vert \bm{w}\vert\vert =1$ subject to the condition
 
 !bt
 \[
-y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. 
+y_i(\bm{x}^T\bm{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, p. 
 \]
 !et
 All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $b$ and $w_1$ and $w_2$ define this line. 
@@ -352,13 +352,13 @@ All points are thus at a signed distance from the decision boundary defined by t
 We seek thus the largest value $M$ defined by
 !bt
 \[
-\frac{1}{\vert \vert \bm{w}\vert\vert}y_i(\bm{w}^T\bm{x}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, 
+\frac{1}{\vert \vert \bm{w}\vert\vert}y_i(\bm{x}^T\bm{w}_i+b) \geq M \hspace{0.1cm}\forall i=1,2,\dots, n, 
 \]
 !et
 or just 
 !bt
 \[
-y_i(\bm{w}^T\bm{x}_i+b) \geq M\vert \vert \bm{w}\vert\vert \hspace{0.1cm}\forall i. 
+y_i(\bm{x}^T\bm{w}_i+b) \geq M\vert \vert \bm{w}\vert\vert \hspace{0.1cm}\forall i. 
 \]
 !et
 If we scale the equation so that $\vert \vert \bm{w}\vert\vert = 1/M$, we have to find the minimum of 
@@ -369,10 +369,10 @@ y_i(\bm{w}^T\bm{x}_i+b) \geq 1 \hspace{0.1cm}\forall i.
 \]
 !et
 
-We have thus defined our margin as the invers of the norm of
+We have thus defined our margin as the inverse of the norm of
 $\bm{w}$. We want to minimize the norm in order to have a as large as
 possible margin $M$. Before we proceed, we need to remind ourselves
-about Lagrangian multipliers.
+about Lagrangian multipliers and optimzation problems.
 
 !split
 ===== A quick Reminder on Lagrangian Multipliers =====
@@ -505,12 +505,14 @@ subject to the constraints $\lambda_i\geq 0$ and $\sum_i\lambda_iy_i=0$.
 We must in addition satisfy the "Karush-Kuhn-Tucker":"https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions" (KKT) condition
 !bt
 \[
-\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b) -1\right] \hspace{0.1cm}\forall i.
+\lambda_i\left[y_i(\bm{x}^T\bm{w}_i+b) -1\right] \hspace{0.1cm}\forall i.
 \]
 !et
-o If $\lambda_i > 0$, then $y_i(\bm{w}^T\bm{x}_i+b)=1$ and we say that $x_i$ is on the boundary.
-o If $y_i(\bm{w}^T\bm{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$. 
-When $\lambda_i > 0$, the vectors $\bm{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$. 
+o If $\lambda_i > 0$, then $y_i(\bm{x}^T\bm{w}_i+b)=1$ and we say that $x_i$ is on the boundary.
+o If $y_i(\bm{x}^T\bm{w}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\lambda_i=0$. 
+When $\lambda_i > 0$, the vectors $\bm{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$.  
+
+The support vectors (the points that define the margin $M$) are the quantities we keep in order to make predictions.
 
 !split
 ===== The problem to solve =====
@@ -549,13 +551,13 @@ To find the coefficients of your hyperplane we need simply to compute
 With our vector $\bm{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via 
 !bt
 \[
-y_i(\bm{w}^T\bm{x}_i+b)=1,
+y_i(\bm{x}^T\bm{w}_i+b)=1,
 \]
 !et
 resulting in
 !bt
 \[
-b = \frac{1}{y_i}-\bm{w}^T\bm{x}_i,
+b = \frac{1}{y_i}-\bm{x}_1^T\bm{w},
 \]
 !et
 or if we write it out in terms of the support vectors only, with $N_s$ being their number,  we have
@@ -567,7 +569,7 @@ b = \frac{1}{N_s}\sum_{j\in N_s}\left(y_j-\sum_{i=1}^n\lambda_iy_i\bm{x}_i^T\bm{
 With our hyperplane coefficients we can use our classifier to assign any observation by simply using 
 !bt
 \[
-y_i = \mathrm{sign}(\bm{w}^T\bm{x}_i+b).
+y_i = \mathrm{sign}(\bm{x}_i^T\bm{w}+b).
 \]
 !et
 Below we discuss how to find the optimal values of $\lambda_i$. Before we proceed however, we discuss now the so-called soft classifier. 
@@ -577,8 +579,10 @@ Below we discuss how to find the optimal values of $\lambda_i$. Before we procee
 
 Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.
 
-Suppose now that classes overlap in feature space, as shown in the
-figure here. One way to deal with this problem before we define the
+Suppose now that the two classes overlap in feature space, as shown in Figure 12.1 of
+"Hastie et al":"https://www.springer.com/gp/book/9780387848570".
+
+One way to deal with this problem before we define the
 so-called _kernel approach_, is to allow a kind of slack in the sense
 that we allow some points to be on the wrong side of the margin.
 
@@ -586,18 +590,18 @@ We introduce thus the so-called _slack_ variables $\bm{\xi} =[\xi_1,x_2,\dots,x_
 modify our previous equation
 !bt
 \[
-y_i(\bm{w}^T\bm{x}_i+b)=1,
+y_i(\bm{x}_i^T\bm{w}+b)=1,
 \]
 !et
 to 
 !bt
 \[
-y_i(\bm{w}^T\bm{x}_i+b)=1-\xi_i,
+y_i(\bm{x}_1^T\bm{w}+b)=1-\xi_i,
 \]
 !et
 with the requirement $\xi_i\geq 0$. The total violation is now $\sum_i\xi$. 
 The value $\xi_i$ in the constraint the last constraint corresponds to the  amount by which the prediction
-$y_i(\bm{w}^T\bm{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$,
+$y_i(\bm{x}_i^T\bm{w}+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\sum_i \xi_i$,
 we bound the total amount by which predictions fall on the wrong side of their margins.
 
 Misclassifications occur when $\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of
@@ -616,7 +620,7 @@ This has in turn the consequences that we change our optmization problem to find
 subject to 
 !bt
 \[
-y_i(\bm{w}^T\bm{x}_i+b)=1-\xi_i \hspace{0.1cm}\forall i,
+y_i(\bm{x}_1^T\bm{w}+b)=1-\xi_i \hspace{0.1cm}\forall i,
 \]
 !et
 with the requirement $\xi_i\geq 0$.
@@ -649,7 +653,7 @@ but now subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$ and
 We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
 !bt
 \[
-\lambda_i\left[y_i(\bm{w}^T\bm{x}_i+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
+\lambda_i\left[y_i(\bm{x}_1^T\bm{w}+b) -(1-\xi_)\right]=0 \hspace{0.1cm}\forall i,
 \]
 !et
 !bt
@@ -660,7 +664,7 @@ We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads
 and 
 !bt
 \[
-y_i(\bm{w}^T\bm{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
+y_i(\bm{x}_i^T\bm{w}+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i.
 \]
 !et