-
diff --git a/doc/pub/svm/html/._svm-bs020.html b/doc/pub/svm/html/._svm-bs020.html index 9e90df4cc..372609da7 100644 --- a/doc/pub/svm/html/._svm-bs020.html +++ b/doc/pub/svm/html/._svm-bs020.html @@ -166,7 +166,7 @@ $$
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity) $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{z}_j, $$ subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors @@ -189,7 +189,10 @@ $$ We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute -the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called +the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). + +
+This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations. diff --git a/doc/pub/svm/html/._svm-bs022.html b/doc/pub/svm/html/._svm-bs022.html index a529ff2e2..425544721 100644 --- a/doc/pub/svm/html/._svm-bs022.html +++ b/doc/pub/svm/html/._svm-bs022.html @@ -170,18 +170,25 @@ There are several popular kernels being used. These are and many other ones.
-An important theorem for us is Mercer's theorem. -The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then -there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space -(possibly with much higher dimensions) such that +An important theorem for us is Mercer's +theorem. The +theorem states that if a kernel function \( K \) is symmetric, continuous +and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there +exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into +another space (possibly with much higher dimensions) such that + $$ K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). $$ +
So you can use \( K \) as a kernel since you know \( \phi \) exists, even if -you don’t know what \( \phi \) is. -Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work -well in practice. +you don’t know what \( \phi \) is. + +
+Note that some frequently used kernels (such as the Sigmoid kernel) +don’t respect all of Mercer’s conditions, yet they generally work well +in practice.
diff --git a/doc/pub/svm/html/svm-bs.html b/doc/pub/svm/html/svm-bs.html index 52d9034be..0ad4b19d0 100644 --- a/doc/pub/svm/html/svm-bs.html +++ b/doc/pub/svm/html/svm-bs.html @@ -174,7 +174,7 @@ MathJax.Hub.Config({
-
diff --git a/doc/pub/svm/html/svm-reveal.html b/doc/pub/svm/html/svm-reveal.html index 2b298cddc..eaa2e0cef 100644 --- a/doc/pub/svm/html/svm-reveal.html +++ b/doc/pub/svm/html/svm-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({
-
@@ -988,7 +988,7 @@ $$ With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)
$$
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{z}_j,
$$
@@ -1018,7 +1018,10 @@ $$
We note that this is nothing but the dot product of the two original
vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the
product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute
-the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called
+the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \).
+
+
+This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations. @@ -1084,20 +1087,27 @@ There are several popular kernels being used. These are and many other ones.
-An important theorem for us is Mercer's theorem. -The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then -there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space -(possibly with much higher dimensions) such that +An important theorem for us is Mercer's +theorem. The +theorem states that if a kernel function \( K \) is symmetric, continuous +and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there +exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into +another space (possibly with much higher dimensions) such that +
$$
K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j).
$$
+
So you can use \( K \) as a kernel since you know \( \phi \) exists, even if -you don’t know what \( \phi \) is. -Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work -well in practice. +you don’t know what \( \phi \) is. + +
+Note that some frequently used kernels (such as the Sigmoid kernel) +don’t respect all of Mercer’s conditions, yet they generally work well +in practice. diff --git a/doc/pub/svm/html/svm-solarized.html b/doc/pub/svm/html/svm-solarized.html index e8d78b39d..f9d5ef494 100644 --- a/doc/pub/svm/html/svm-solarized.html +++ b/doc/pub/svm/html/svm-solarized.html @@ -110,7 +110,7 @@ MathJax.Hub.Config({
-
@@ -831,7 +831,7 @@ $$
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity) $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{z}_j, $$ subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors @@ -854,7 +854,10 @@ $$ We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute -the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called +the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). + +
+This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations. @@ -913,18 +916,25 @@ There are several popular kernels being used. These are and many other ones.
-An important theorem for us is Mercer's theorem. -The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then -there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space -(possibly with much higher dimensions) such that +An important theorem for us is Mercer's +theorem. The +theorem states that if a kernel function \( K \) is symmetric, continuous +and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there +exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into +another space (possibly with much higher dimensions) such that + $$ K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). $$ +
So you can use \( K \) as a kernel since you know \( \phi \) exists, even if -you don’t know what \( \phi \) is. -Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work -well in practice. +you don’t know what \( \phi \) is. + +
+Note that some frequently used kernels (such as the Sigmoid kernel) +don’t respect all of Mercer’s conditions, yet they generally work well +in practice.
diff --git a/doc/pub/svm/html/svm.html b/doc/pub/svm/html/svm.html
index af6d4c338..804b3f318 100644
--- a/doc/pub/svm/html/svm.html
+++ b/doc/pub/svm/html/svm.html
@@ -115,7 +115,7 @@ MathJax.Hub.Config({
-
@@ -836,7 +836,7 @@ $$
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity) $$ -{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{z}_j, +{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\boldsymbol{z}_i^T\boldsymbol{z}_j, $$ subject to the constraints \( \lambda_i\geq 0 \), \( \sum_i\lambda_iy_i=0 \), and for the support vectors @@ -859,7 +859,10 @@ $$ We note that this is nothing but the dot product of the two original vectors \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). Instead of thus computing the product in the Lagrangian of \( \boldsymbol{z}_i^T\boldsymbol{z}_j \) we simply compute -the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). This leads to the so-called +the dot product \( (\boldsymbol{x}_i^T\boldsymbol{x}_j)^2 \). + +
+This leads to the so-called kernel trick and the result leads to the same as if we went through the trouble of performing the transformation \( (\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j) \) during the SVM calculations. @@ -918,18 +921,25 @@ There are several popular kernels being used. These are and many other ones.
-An important theorem for us is Mercer's theorem. -The theorem states that if a kernel function \( K \) is symmetric, continuous and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then -there exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into another space -(possibly with much higher dimensions) such that +An important theorem for us is Mercer's +theorem. The +theorem states that if a kernel function \( K \) is symmetric, continuous +and leads to a positive semi-definite matrix \( \boldsymbol{P} \) then there +exists a function \( \phi \) that maps \( \boldsymbol{x}_i \) and \( \boldsymbol{x}_j \) into +another space (possibly with much higher dimensions) such that + $$ K(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j). $$ +
So you can use \( K \) as a kernel since you know \( \phi \) exists, even if -you don’t know what \( \phi \) is. -Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work -well in practice. +you don’t know what \( \phi \) is. + +
+Note that some frequently used kernels (such as the Sigmoid kernel) +don’t respect all of Mercer’s conditions, yet they generally work well +in practice.
diff --git a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz
index 87cddf0b1..5ee1bf504 100644
Binary files a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz and b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz differ
diff --git a/doc/pub/svm/ipynb/svm.ipynb b/doc/pub/svm/ipynb/svm.ipynb
index fe2c161a8..e5c37dc32 100644
--- a/doc/pub/svm/ipynb/svm.ipynb
+++ b/doc/pub/svm/ipynb/svm.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Nov 18, 2019**\n",
+ "Date: **Nov 22, 2019**\n",
"\n",
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -1267,7 +1267,7 @@
"metadata": {},
"source": [
"$$\n",
- "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{z}_j,\n",
+ "{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{z}_i^T\\boldsymbol{z}_j,\n",
"$$"
]
},
@@ -1327,7 +1327,10 @@
"We note that this is nothing but the dot product of the two original\n",
"vectors $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. Instead of thus computing the\n",
"product in the Lagrangian of $\\boldsymbol{z}_i^T\\boldsymbol{z}_j$ we simply compute\n",
- "the dot product $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$. This leads to the so-called\n",
+ "the dot product $(\\boldsymbol{x}_i^T\\boldsymbol{x}_j)^2$.\n",
+ "\n",
+ "\n",
+ "This leads to the so-called\n",
"kernel trick and the result leads to the same as if we went through\n",
"the trouble of performing the transformation\n",
"$(\\phi(\\boldsymbol{x}_i)^T\\phi(\\boldsymbol{x}_j)$ during the SVM calculations.\n",
@@ -1412,10 +1415,12 @@
"\n",
"and many other ones.\n",
"\n",
- "An important theorem for us is [Mercer's theorem](https://en.wikipedia.org/wiki/Mercer%27s_theorem). \n",
- "The theorem states that if a kernel function $K$ is symmetric, continuous and leads to a positive semi-definite matrix $\\boldsymbol{P}$ then\n",
- "there exists a function $\\phi$ that maps $\\boldsymbol{x}_i$ and $\\boldsymbol{x}_j$ into another space\n",
- "(possibly with much higher dimensions) such that"
+ "An important theorem for us is [Mercer's\n",
+ "theorem](https://en.wikipedia.org/wiki/Mercer%27s_theorem). The\n",
+ "theorem states that if a kernel function $K$ is symmetric, continuous\n",
+ "and leads to a positive semi-definite matrix $\\boldsymbol{P}$ then there\n",
+ "exists a function $\\phi$ that maps $\\boldsymbol{x}_i$ and $\\boldsymbol{x}_j$ into\n",
+ "another space (possibly with much higher dimensions) such that"
]
},
{
@@ -1433,8 +1438,10 @@
"source": [
"So you can use $K$ as a kernel since you know $\\phi$ exists, even if\n",
"you don’t know what $\\phi$ is. \n",
- "Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work\n",
- "well in practice.\n",
+ "\n",
+ "Note that some frequently used kernels (such as the Sigmoid kernel)\n",
+ "don’t respect all of Mercer’s conditions, yet they generally work well\n",
+ "in practice.\n",
"\n",
"\n",
"## The moons example"
diff --git a/doc/pub/svm/pdf/svm-minted.pdf b/doc/pub/svm/pdf/svm-minted.pdf
index bb217b4de..3c8036925 100644
Binary files a/doc/pub/svm/pdf/svm-minted.pdf and b/doc/pub/svm/pdf/svm-minted.pdf differ
diff --git a/doc/src/SupportVMachines/svm.do.txt b/doc/src/SupportVMachines/svm.do.txt
index 5bbb29bd3..f1417f311 100644
--- a/doc/src/SupportVMachines/svm.do.txt
+++ b/doc/src/SupportVMachines/svm.do.txt
@@ -708,7 +708,7 @@ z = \phi(x_i) =\left(x_i^2, y_i^2, \sqrt{2}x_iy_i\right).
With our new basis, the equations we solved earlier are basically the same, that is we have now (without the slack option for simplicity)
!bt
\[
-{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{x}_i^T\bm{z}_j,
+{\cal L}=\sum_i\lambda_i-\frac{1}{2}\sum_{ij}^n\lambda_i\lambda_jy_iy_j\bm{z}_i^T\bm{z}_j,
\]
!et
subject to the constraints $\lambda_i\geq 0$, $\sum_i\lambda_iy_i=0$, and for the support vectors
@@ -734,7 +734,10 @@ K(\bm{x}_i,\bm{x}_j)=[x_i^2, y_i^2, \sqrt{2}x_iy_i]^T\begin{bmatrix} x_j^2 \\ y_
We note that this is nothing but the dot product of the two original
vectors $(\bm{x}_i^T\bm{x}_j)^2$. Instead of thus computing the
product in the Lagrangian of $\bm{z}_i^T\bm{z}_j$ we simply compute
-the dot product $(\bm{x}_i^T\bm{x}_j)^2$. This leads to the so-called
+the dot product $(\bm{x}_i^T\bm{x}_j)^2$.
+
+
+This leads to the so-called
kernel trick and the result leads to the same as if we went through
the trouble of performing the transformation
$(\phi(\bm{x}_i)^T\phi(\bm{x}_j)$ during the SVM calculations.
@@ -785,19 +788,25 @@ o Gaussian Radial Basis Function: $K(\bm{x},\bm{y})=\exp{\left(-\gamma\vert\vert
o Tanh: $K(\bm{x},\bm{y})=\tanh{(\bm{x}^T\bm{y}+\gamma)}$,
and many other ones.
-An important theorem for us is "Mercer's theorem":"https://en.wikipedia.org/wiki/Mercer%27s_theorem".
-The theorem states that if a kernel function $K$ is symmetric, continuous and leads to a positive semi-definite matrix $\bm{P}$ then
-there exists a function $\phi$ that maps $\bm{x}_i$ and $\bm{x}_j$ into another space
-(possibly with much higher dimensions) such that
+An important theorem for us is "Mercer's
+theorem":"https://en.wikipedia.org/wiki/Mercer%27s_theorem". The
+theorem states that if a kernel function $K$ is symmetric, continuous
+and leads to a positive semi-definite matrix $\bm{P}$ then there
+exists a function $\phi$ that maps $\bm{x}_i$ and $\bm{x}_j$ into
+another space (possibly with much higher dimensions) such that
+
!bt
\[
K(\bm{x}_i,\bm{x}_j)=\phi(\bm{x}_i)^T\phi(\bm{x}_j).
\]
!et
+
So you can use $K$ as a kernel since you know $\phi$ exists, even if
you don’t know what $\phi$ is.
-Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all of Mercer’s conditions, yet they generally work
-well in practice.
+
+Note that some frequently used kernels (such as the Sigmoid kernel)
+don’t respect all of Mercer’s conditions, yet they generally work well
+in practice.
!split