update week 48
This commit is contained in:
mhjensen
@@ -57,12 +57,12 @@ space using other basis expansions such as higher-order polynomials,
|
||||
wavelets, splines etc.
|
||||
|
||||
If our feature space is not easy to separate, as shown in the figure
|
||||
here, we can achieve a better separation by introducing more complex
|
||||
basis functions. The ideal would be, as shown in the next figure, to, via a specific transformation to
|
||||
here generated by the code below (see also Figures 12.2 and 12.3 of "Hastie et al.":"https://www.springer.com/gp/book/9780387848570"), we can achieve a better separation by introducing more complex
|
||||
basis functions. The ideal would be (see Figures 12.2 and 12.3) to, via a specific transformation to
|
||||
obtain a separation between the classes which is almost linear.
|
||||
|
||||
The change of basis, from $x\rightarrow z=\phi(x)$ leads to the same type of equations to be solved, except that
|
||||
we need to introduce for example a polynomial transformation to a two-dimensional training set.
|
||||
we need to introduce, for example, a polynomial transformation to a two-dimensional training set.
|
||||
|
||||
!bc pycod
|
||||
import numpy as np
|
||||
@@ -122,7 +122,7 @@ plt.show()
|
||||
Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with $x_i$ and $y_i$ as variables)
|
||||
!bt
|
||||
\[
|
||||
z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
|
||||
z = \phi(x_i) =\left(1, \sqrt{2}x_i, \sqrt{2}y_i, x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -135,7 +135,7 @@ With our new basis, the equations we solved earlier are basically the same, that
|
||||
subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$, and for the support vectors
|
||||
!bt
|
||||
\[
|
||||
y_i(\bm{w}^T\bm{z}_i+b)= 1 \hspace{0.1cm}\forall i,
|
||||
y_i(\bm{z}_i^T\bm{w}+b)= 1 \hspace{0.1cm}\forall i,
|
||||
\]
|
||||
!et
|
||||
from which we also find $b$.
|
||||
@@ -148,25 +148,27 @@ K(\bm{x}_i,\bm{x}_j)=\bm{z}_i^T\bm{z}_j= \phi(\bm{x}_i)^T\phi(\bm{x}_j).
|
||||
For the above example, the kernel reads
|
||||
!bt
|
||||
\[
|
||||
K(\bm{x}_i,\bm{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_j^2 \\ \sqrt{2}x_jy_j \end{bmatrix}=x_i^2x_j^2+2x_ix_jy_iy_j+y_i^2y_j^2.
|
||||
K(\bm{x}_i,\bm{x}_j)=[1, \sqrt{2}x_i, \sqrt{2}y_i, x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} 1\\ \sqrt{2}x_j \\ \sqrt{2}y_j \\ x_j^2\\ y_i^2\\ \sqrt{2}x_jy_j \end{bmatrix}=1+2x_ix_j+2y_iy_j+(x_ix_j)^2+(y_iy_j)^2+2x_ix_jy_iy_j.
|
||||
\]
|
||||
!et
|
||||
|
||||
We note that this is nothing but the dot product of the two original
|
||||
vectors $(\bm{x}_i^T\bm{x}_j)^2$. Instead of thus computing the
|
||||
product in the Lagrangian of $\bm{z}_i^T\bm{z}_j$ we simply compute
|
||||
We note that this dot product can be rewritten as
|
||||
!bt
|
||||
\[
|
||||
K(\bm{x}_i,\bm{x}_j)=[1+\bm{x}^T\bm{x}']^d,
|
||||
\]
|
||||
!et
|
||||
where $d=2$ in our case and $\bm{x}=[x_i,y_i]$ and $\bm{x}=[x_j,y_j]$.
|
||||
To compute the last equation is however inefficient from a computational stand.
|
||||
Instead of computing the last equation for the kernel, we simply compute
|
||||
the dot product $(\bm{x}_i^T\bm{x}_j)^2$.
|
||||
|
||||
|
||||
This leads to the so-called
|
||||
kernel trick and the result leads to the same as if we went through
|
||||
the trouble of performing the transformation
|
||||
$\phi(\bm{x}_i)^T\phi(\bm{x}_j)$ during the SVM calculations.
|
||||
|
||||
kernel trick.
|
||||
|
||||
!split
|
||||
===== The problem to solve =====
|
||||
Using our definition of the kernel We can rewrite again the Lagrangian
|
||||
|
||||
Using our definition of the kernel, we can rewrite again the Lagrangian
|
||||
!bt
|
||||
\[
|
||||
{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{z}_j,
|
||||
@@ -187,6 +189,9 @@ subject to $\bm{y}^T\bm{\lambda}=0$. Here we defined the vectors $\bm{\lambda} =
|
||||
$\bm{y}=[y_1,y_2,\dots,y_n]$.
|
||||
If we add the slack constants this leads to the additional constraint $0\leq \lambda_i \leq C$.
|
||||
|
||||
!split
|
||||
===== Tailoring the equations to the usage of CVXOPT =====
|
||||
|
||||
We can rewrite this (see the solutions below) in terms of a convex optimization problem of the type
|
||||
!bt
|
||||
\begin{align*}
|
||||
@@ -231,7 +236,7 @@ in practice.
|
||||
|
||||
|
||||
!split
|
||||
===== The moons example =====
|
||||
===== The moons example ("Adapted from Geron, chapter 5":"https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/") =====
|
||||
!bc pycod
|
||||
from __future__ import division, print_function, unicode_literals
|
||||
|
||||
@@ -465,7 +470,7 @@ import numpy
|
||||
import cvxopt
|
||||
!ec
|
||||
|
||||
This will make our life much easier. You don't need t write your own optimizer.
|
||||
This will make our life much easier. You don't need to write your own optimizer.
|
||||
|
||||
|
||||
!split
|
||||
@@ -554,6 +559,68 @@ $\bm{y}=[y_1,y_2,\dots,y_n]$.
|
||||
With the slack constants this leads to the additional constraint $0\leq \lambda_i \leq C$.
|
||||
|
||||
|
||||
!split
|
||||
===== Setting up the matrices and the problem =====
|
||||
|
||||
We have the general problem
|
||||
!bt
|
||||
\begin{align*}
|
||||
&\mathrm{min}_{\lambda}\hspace{0.2cm} \frac{1}{2}\bm{\lambda}^T\bm{P}\bm{\lambda}+\bm{q}^T\bm{\lambda},\\ \nonumber
|
||||
&\mathrm{subject\hspace{0.1cm}to} \hspace{0.2cm} \bm{G}\bm{\lambda} \preceq \bm{h} \wedge \bm{A}\bm{\lambda}=f.
|
||||
\end{align*}
|
||||
!et
|
||||
|
||||
|
||||
|
||||
o With a given kernel we can thus define the matrix $\bm{P}$.
|
||||
o The matrix $\bm{P}$ has matrix elements $p_{ij}=y_iy_jK(\bm{x}_i,\bm{x}_j)$. Given a kernel $K$ and the targets $y_i$ this matrix is easy to set up.
|
||||
o The $\bm{q}$ is zero.
|
||||
o The constraint $\bm{y}^T\bm{\lambda}=0$ leads to $f=0$ and $\bm{A}=\bm{y}$.
|
||||
o To set up the matrix $\bm{G}$ we note that the inequalities $0\leq \lambda_i \leq C$ can be split up into
|
||||
$0\leq \lambda_i$ and $\lambda_i \leq C$. These two inequalities define then the matrix $\bm{G}$ and the vector $\bm{h}$.
|
||||
|
||||
!split
|
||||
===== Setting up $\bm{G}\bm{\lambda} \preceq \bm{h}$ =====
|
||||
|
||||
We have two constraints, $0\le \lambda_i$ and $\lambda_i \le C$. To do this we multiply the ones with the contraint
|
||||
$\ge$ with $-1$ in order to get $\le$. It means that the problem $\bm{G}\bm{\lambda} \preceq \bm{h}$
|
||||
can be written as
|
||||
!bt
|
||||
\[
|
||||
\begin{bmatrix} -1& 0 & 0 & \dots & 0 \\
|
||||
0& -1 & 0 & \dots & 0 \\
|
||||
0& 0 & -1 & \dots & 0 \\
|
||||
\dots& \dots & \dots & \dots & \dots \\
|
||||
0& 0 & 0 & \dots & -1 \\
|
||||
-& 0 & 0 & \dots & 0 \\
|
||||
0& 1 & 0 & \dots & 0 \\
|
||||
0& 0 & 1 & \dots & 0 \\
|
||||
\dots& \dots & \dots & \dots & \dots \\
|
||||
0& 0 & 0 & \dots & 1 \\
|
||||
\end{bmatrix}\bm{\lambda}
|
||||
\begin{bmatrix} \lambda_1 \\
|
||||
\lambda_2 \\
|
||||
\lambda_3 \\
|
||||
\dots \\
|
||||
\lambda_n \\
|
||||
\end{bmatrix}\bm{\lambda}=
|
||||
\begin{bmatrix} 0 \\
|
||||
0 \\
|
||||
0 \\
|
||||
\dots \\
|
||||
0 \\
|
||||
C \\
|
||||
C \\
|
||||
C \\
|
||||
\dots \\
|
||||
C \\
|
||||
\end{bmatrix}\bm{\lambda}
|
||||
\]
|
||||
!et
|
||||
|
||||
And then we are ready to go.
|
||||
|
||||
|
||||
|
||||
!split
|
||||
===== Summary of course =====
|
||||
|
||||
Reference in New Issue
Block a user