-
@@ -109,44 +114,129 @@ end of tocinfo -->
+A Support Vector Machine (SVM) is a very powerful and versatile +Machine Learning model, capable of performing linear or nonlinear +classification, regression, and even outlier detection. It is one of +the most popular models in Machine Learning, and anyone interested in +Machine Learning should have it in their toolbox. SVMs are +particularly well suited for classification of complex but small-sized or +medium-sized datasets. + +
+The basic mathematics relies on the definition of hyperplanes and the definition of a margin which separates +classes (in case of classification problems) of variables. It is also used for regression problems. + +
+With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below. +We distringuish also between linearn and non-linear approaches. + +
+These notes will be updated shortly with more material + +
+ + +
+The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM +model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect +Iris-Virginica flowers.
import numpy as np
-from sklearn.svm import SVR
-import matplotlib.pyplot as plt
-
-# Generate sample data
-X = np.sort(5*np.random.rand(40,1), axis=0)
-y = X**3
-y=y.ravel()
-
-# Add noise to targets
-X[::4] +=3*(0.5 - np.random.rand(1))
-y[::5] += 50 * (0.5 - np.random.rand(8))
-
-plt.plot(X,y, 'g^')
-
-#SVR Fit
-svr_poly = SVR(kernel='poly', C=1e3, degree=3)
-y_poly = svr_poly.fit(X, y).predict(X)
-
-# Plots
-z = np.arange(0, 5, 0.1)
-t = z**3
-fig = plt.figure()
-ax = fig.add_subplot(111)
-plt.plot(z,z**3, 'r--', label='Cubic Function with No Noise')
-lw = 2
-plt.scatter(X, y, color='darkorange', label='Gaussian Cubic Noise')
-plt.plot(X, y_poly, color='green', lw=lw, label='Polynomial model')
-plt.xlabel('data')
-plt.ylabel('target')
-plt.title('Cubic Gaussian Distribution')
-plt.legend()
-plt.show()
+from sklearn import datasets
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler
+from sklearn.svm import LinearSVC
+iris = datasets.load_iris()
+X = iris["data"][:, (2, 3)] # petal length, petal width
+y = (iris["target"] == 2).astype(np.float64) # Iris-Virginica
+svm_clf = Pipeline((
+("scaler", StandardScaler()),
+("linear_svc", LinearSVC(C=1, loss="hinge")),
+))
+svm_clf.fit(X_scaled, y)
+Alternatively, you could use the SVC class, using SVC(kernel="linear", C=1), but it is much slower, +especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier +class, with SGDClassifier(loss="hinge", alpha=1/(m*C)). This applies regular Stochastic +Gradient Descentto train a linear SVM classifier. It does not converge as fast as the +LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core +training), or to handle online classification tasks. + +
+ + +
+Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets +are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more +features, such as polynomial features. In some cases this can result in a linearly +separable dataset. + +
+ + +
from sklearn.datasets import make_moons
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import PolynomialFeatures
+polynomial_svm_clf = Pipeline((
+("poly_features", PolynomialFeatures(degree=3)),
+("scaler", StandardScaler()),
+("svm_clf", LinearSVC(C=10, loss="hinge"))
+))
+polynomial_svm_clf.fit(X, y)
++ + +
+Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning +algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets, +and with a high polynomial degree it creates a huge number of features, making the model too slow. +Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the +kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many +polynomial features, even with very high-degree polynomials, without actually having to add them. So +there is no combinatorial explosion of the number of features since you don’t actually add any features. +This trick is implemented by the SVC class. + +
+ + +
from sklearn.svm import SVC
+poly_kernel_svm_clf = Pipeline((
+("scaler", StandardScaler()),
+("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5))
+))
+poly_kernel_svm_clf.fit(X, y)
++This code trains an SVM classifier using a 3rd-degree polynomial kernel. + +
+ + +
+You can use Scikit-Learn’s LinearSVR class to perform linear SVM Regression. The following code +shows how to use the regression option (more material to come) +
+ + +
from sklearn.svm import LinearSVR
+svm_reg = LinearSVR(epsilon=1.5)
+svm_reg.fit(X, y)
++To tackle nonlinear regression tasks, you can use a kernelized SVM model diff --git a/doc/pub/svm/html/svm-reveal.html b/doc/pub/svm/html/svm-reveal.html index 17a17f1ce..1782b55ba 100644 --- a/doc/pub/svm/html/svm-reveal.html +++ b/doc/pub/svm/html/svm-reveal.html @@ -132,7 +132,7 @@ td.padding {
-
@@ -145,43 +145,130 @@ td.padding {
+A Support Vector Machine (SVM) is a very powerful and versatile
+Machine Learning model, capable of performing linear or nonlinear
+classification, regression, and even outlier detection. It is one of
+the most popular models in Machine Learning, and anyone interested in
+Machine Learning should have it in their toolbox. SVMs are
+particularly well suited for classification of complex but small-sized or
+medium-sized datasets.
+
+
+The basic mathematics relies on the definition of hyperplanes and the definition of a margin which separates
+classes (in case of classification problems) of variables. It is also used for regression problems.
+
+
+With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.
+We distringuish also between linearn and non-linear approaches.
+
+
+These notes will be updated shortly with more material
+
+The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM
+model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect
+Iris-Virginica flowers.
+Alternatively, you could use the SVC class, using SVC(kernel="linear", C=1), but it is much slower,
+especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier
+class, with SGDClassifier(loss="hinge", alpha=1/(m*C)). This applies regular Stochastic
+Gradient Descentto train a linear SVM classifier. It does not converge as fast as the
+LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core
+training), or to handle online classification tasks.
+
+Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets
+are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more
+features, such as polynomial features. In some cases this can result in a linearly
+separable dataset.
+
+
+
+
+
+Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning
+algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets,
+and with a high polynomial degree it creates a huge number of features, making the model too slow.
+Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the
+kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many
+polynomial features, even with very high-degree polynomials, without actually having to add them. So
+there is no combinatorial explosion of the number of features since you don’t actually add any features.
+This trick is implemented by the SVC class.
+
+
+
+
+
+This code trains an SVM classifier using a 3rd-degree polynomial kernel.
+
+You can use Scikit-Learn’s LinearSVR class to perform linear SVM Regression. The following code
+shows how to use the regression option (more material to come)
+
+
+
+
+To tackle nonlinear regression tasks, you can use a kernelized SVM model
Support Vector Machines, overarching aims
+SVMs, basics with scikit-learn
+
+import numpy as np
-from sklearn.svm import SVR
-import matplotlib.pyplot as plt
-
-# Generate sample data
-X = np.sort(5*np.random.rand(40,1), axis=0)
-y = X**3
-y=y.ravel()
-
-# Add noise to targets
-X[::4] +=3*(0.5 - np.random.rand(1))
-y[::5] += 50 * (0.5 - np.random.rand(8))
-
-plt.plot(X,y, 'g^')
-
-#SVR Fit
-svr_poly = SVR(kernel='poly', C=1e3, degree=3)
-y_poly = svr_poly.fit(X, y).predict(X)
-
-# Plots
-z = np.arange(0, 5, 0.1)
-t = z**3
-fig = plt.figure()
-ax = fig.add_subplot(111)
-plt.plot(z,z**3, 'r--', label='Cubic Function with No Noise')
-lw = 2
-plt.scatter(X, y, color='darkorange', label='Gaussian Cubic Noise')
-plt.plot(X, y_poly, color='green', lw=lw, label='Polynomial model')
-plt.xlabel('data')
-plt.ylabel('target')
-plt.title('Cubic Gaussian Distribution')
-plt.legend()
-plt.show()
+from sklearn import datasets
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler
+from sklearn.svm import LinearSVC
+iris = datasets.load_iris()
+X = iris["data"][:, (2, 3)] # petal length, petal width
+y = (iris["target"] == 2).astype(np.float64) # Iris-Virginica
+svm_clf = Pipeline((
+("scaler", StandardScaler()),
+("linear_svc", LinearSVC(C=1, loss="hinge")),
+))
+svm_clf.fit(X_scaled, y)
SVMs, adding polynomial features
+
+from sklearn.datasets import make_moons
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import PolynomialFeatures
+polynomial_svm_clf = Pipeline((
+("poly_features", PolynomialFeatures(degree=3)),
+("scaler", StandardScaler()),
+("svm_clf", LinearSVC(C=10, loss="hinge"))
+))
+polynomial_svm_clf.fit(X, y)
+
SVMs, polynomials and kernels
+
+from sklearn.svm import SVC
+poly_kernel_svm_clf = Pipeline((
+("scaler", StandardScaler()),
+("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5))
+))
+poly_kernel_svm_clf.fit(X, y)
+
SVMs, regression
+
+from sklearn.svm import LinearSVR
+svm_reg = LinearSVR(epsilon=1.5)
+svm_reg.fit(X, y)
+
-
+A Support Vector Machine (SVM) is a very powerful and versatile +Machine Learning model, capable of performing linear or nonlinear +classification, regression, and even outlier detection. It is one of +the most popular models in Machine Learning, and anyone interested in +Machine Learning should have it in their toolbox. SVMs are +particularly well suited for classification of complex but small-sized or +medium-sized datasets. + +
+The basic mathematics relies on the definition of hyperplanes and the definition of a margin which separates +classes (in case of classification problems) of variables. It is also used for regression problems. + +
+With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below. +We distringuish also between linearn and non-linear approaches. + +
+These notes will be updated shortly with more material + +
+
+
+
+The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM +model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect +Iris-Virginica flowers.
import numpy as np
-from sklearn.svm import SVR
-import matplotlib.pyplot as plt
-
-# Generate sample data
-X = np.sort(5*np.random.rand(40,1), axis=0)
-y = X**3
-y=y.ravel()
-
-# Add noise to targets
-X[::4] +=3*(0.5 - np.random.rand(1))
-y[::5] += 50 * (0.5 - np.random.rand(8))
-
-plt.plot(X,y, 'g^')
-
-#SVR Fit
-svr_poly = SVR(kernel='poly', C=1e3, degree=3)
-y_poly = svr_poly.fit(X, y).predict(X)
-
-# Plots
-z = np.arange(0, 5, 0.1)
-t = z**3
-fig = plt.figure()
-ax = fig.add_subplot(111)
-plt.plot(z,z**3, 'r--', label='Cubic Function with No Noise')
-lw = 2
-plt.scatter(X, y, color='darkorange', label='Gaussian Cubic Noise')
-plt.plot(X, y_poly, color='green', lw=lw, label='Polynomial model')
-plt.xlabel('data')
-plt.ylabel('target')
-plt.title('Cubic Gaussian Distribution')
-plt.legend()
-plt.show()
+from sklearn import datasets
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler
+from sklearn.svm import LinearSVC
+iris = datasets.load_iris()
+X = iris["data"][:, (2, 3)] # petal length, petal width
+y = (iris["target"] == 2).astype(np.float64) # Iris-Virginica
+svm_clf = Pipeline((
+("scaler", StandardScaler()),
+("linear_svc", LinearSVC(C=1, loss="hinge")),
+))
+svm_clf.fit(X_scaled, y)
+Alternatively, you could use the SVC class, using SVC(kernel="linear", C=1), but it is much slower, +especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier +class, with SGDClassifier(loss="hinge", alpha=1/(m*C)). This applies regular Stochastic +Gradient Descentto train a linear SVM classifier. It does not converge as fast as the +LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core +training), or to handle online classification tasks. + +
+
+
+
+Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets +are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more +features, such as polynomial features. In some cases this can result in a linearly +separable dataset. + +
+ + +
from sklearn.datasets import make_moons
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import PolynomialFeatures
+polynomial_svm_clf = Pipeline((
+("poly_features", PolynomialFeatures(degree=3)),
+("scaler", StandardScaler()),
+("svm_clf", LinearSVC(C=10, loss="hinge"))
+))
+polynomial_svm_clf.fit(X, y)
+
+
+
+
+Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning +algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets, +and with a high polynomial degree it creates a huge number of features, making the model too slow. +Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the +kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many +polynomial features, even with very high-degree polynomials, without actually having to add them. So +there is no combinatorial explosion of the number of features since you don’t actually add any features. +This trick is implemented by the SVC class. + +
+ + +
from sklearn.svm import SVC
+poly_kernel_svm_clf = Pipeline((
+("scaler", StandardScaler()),
+("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5))
+))
+poly_kernel_svm_clf.fit(X, y)
++This code trains an SVM classifier using a 3rd-degree polynomial kernel. + +
+
+
+
+You can use Scikit-Learn’s LinearSVR class to perform linear SVM Regression. The following code +shows how to use the regression option (more material to come) +
+ + +
from sklearn.svm import LinearSVR
+svm_reg = LinearSVR(epsilon=1.5)
+svm_reg.fit(X, y)
++To tackle nonlinear regression tasks, you can use a kernelized SVM model diff --git a/doc/pub/svm/html/svm.html b/doc/pub/svm/html/svm.html index 497df0d1e..cf2fe755b 100644 --- a/doc/pub/svm/html/svm.html +++ b/doc/pub/svm/html/svm.html @@ -39,10 +39,11 @@ div { text-align: justify; text-justify: inter-word; }
@@ -68,51 +69,136 @@ end of tocinfo -->-
+A Support Vector Machine (SVM) is a very powerful and versatile +Machine Learning model, capable of performing linear or nonlinear +classification, regression, and even outlier detection. It is one of +the most popular models in Machine Learning, and anyone interested in +Machine Learning should have it in their toolbox. SVMs are +particularly well suited for classification of complex but small-sized or +medium-sized datasets. + +
+The basic mathematics relies on the definition of hyperplanes and the definition of a margin which separates +classes (in case of classification problems) of variables. It is also used for regression problems. + +
+With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below. +We distringuish also between linearn and non-linear approaches. + +
+These notes will be updated shortly with more material + +
+
+
+
+The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM +model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect +Iris-Virginica flowers.
import numpy as np
-from sklearn.svm import SVR
-import matplotlib.pyplot as plt
-
-# Generate sample data
-X = np.sort(5*np.random.rand(40,1), axis=0)
-y = X**3
-y=y.ravel()
-
-# Add noise to targets
-X[::4] +=3*(0.5 - np.random.rand(1))
-y[::5] += 50 * (0.5 - np.random.rand(8))
-
-plt.plot(X,y, 'g^')
-
-#SVR Fit
-svr_poly = SVR(kernel='poly', C=1e3, degree=3)
-y_poly = svr_poly.fit(X, y).predict(X)
-
-# Plots
-z = np.arange(0, 5, 0.1)
-t = z**3
-fig = plt.figure()
-ax = fig.add_subplot(111)
-plt.plot(z,z**3, 'r--', label='Cubic Function with No Noise')
-lw = 2
-plt.scatter(X, y, color='darkorange', label='Gaussian Cubic Noise')
-plt.plot(X, y_poly, color='green', lw=lw, label='Polynomial model')
-plt.xlabel('data')
-plt.ylabel('target')
-plt.title('Cubic Gaussian Distribution')
-plt.legend()
-plt.show()
+from sklearn import datasets
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler
+from sklearn.svm import LinearSVC
+iris = datasets.load_iris()
+X = iris["data"][:, (2, 3)] # petal length, petal width
+y = (iris["target"] == 2).astype(np.float64) # Iris-Virginica
+svm_clf = Pipeline((
+("scaler", StandardScaler()),
+("linear_svc", LinearSVC(C=1, loss="hinge")),
+))
+svm_clf.fit(X_scaled, y)
+Alternatively, you could use the SVC class, using SVC(kernel="linear", C=1), but it is much slower, +especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier +class, with SGDClassifier(loss="hinge", alpha=1/(m*C)). This applies regular Stochastic +Gradient Descentto train a linear SVM classifier. It does not converge as fast as the +LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core +training), or to handle online classification tasks. + +
+
+
+
+Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets +are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more +features, such as polynomial features. In some cases this can result in a linearly +separable dataset. + +
+ + +
from sklearn.datasets import make_moons
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import PolynomialFeatures
+polynomial_svm_clf = Pipeline((
+("poly_features", PolynomialFeatures(degree=3)),
+("scaler", StandardScaler()),
+("svm_clf", LinearSVC(C=10, loss="hinge"))
+))
+polynomial_svm_clf.fit(X, y)
+
+
+
+
+Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning +algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets, +and with a high polynomial degree it creates a huge number of features, making the model too slow. +Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the +kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many +polynomial features, even with very high-degree polynomials, without actually having to add them. So +there is no combinatorial explosion of the number of features since you don’t actually add any features. +This trick is implemented by the SVC class. + +
+ + +
from sklearn.svm import SVC
+poly_kernel_svm_clf = Pipeline((
+("scaler", StandardScaler()),
+("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5))
+))
+poly_kernel_svm_clf.fit(X, y)
++This code trains an SVM classifier using a 3rd-degree polynomial kernel. + +
+
+
+
+You can use Scikit-Learn’s LinearSVR class to perform linear SVM Regression. The following code +shows how to use the regression option (more material to come) +
+ + +
from sklearn.svm import LinearSVR
+svm_reg = LinearSVR(epsilon=1.5)
+svm_reg.fit(X, y)
++To tackle nonlinear regression tasks, you can use a kernelized SVM model diff --git a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz index 0d17f826b..d5c9e5f73 100644 Binary files a/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz and b/doc/pub/svm/ipynb/ipynb-svm-src.tar.gz differ diff --git a/doc/pub/svm/ipynb/svm.ipynb b/doc/pub/svm/ipynb/svm.ipynb index 697c1a740..2876278d5 100644 --- a/doc/pub/svm/ipynb/svm.ipynb +++ b/doc/pub/svm/ipynb/svm.ipynb @@ -10,14 +10,35 @@ " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **May 30, 2018**\n", + "Date: **Nov 1, 2018**\n", "\n", "Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", "\n", "\n", "\n", + "## Support Vector Machines, overarching aims\n", "\n", - "## Support Vector Machines, overarching aims" + "A Support Vector Machine (SVM) is a very powerful and versatile\n", + "Machine Learning model, capable of performing linear or nonlinear\n", + "classification, regression, and even outlier detection. It is one of\n", + "the most popular models in Machine Learning, and anyone interested in\n", + "Machine Learning should have it in their toolbox. SVMs are\n", + "particularly well suited for classification of complex but small-sized or\n", + "medium-sized datasets. \n", + "\n", + "The basic mathematics relies on the definition of hyperplanes and the definition of a **margin** which separates\n", + "classes (in case of classification problems) of variables. It is also used for regression problems.\n", + "\n", + "With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below.\n", + "We distringuish also between linearn and non-linear approaches.\n", + "\n", + "**These notes will be updated shortly with more material**\n", + "\n", + "## SVMs, basics with scikit-learn\n", + "\n", + "The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM\n", + "model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect\n", + "Iris-Virginica flowers." ] }, { @@ -28,41 +49,122 @@ }, "outputs": [], "source": [ - "%matplotlib inline\n", - "\n", "import numpy as np\n", - "from sklearn.svm import SVR\n", - "import matplotlib.pyplot as plt\n", + "from sklearn import datasets\n", + "from sklearn.pipeline import Pipeline\n", + "from sklearn.preprocessing import StandardScaler\n", + "from sklearn.svm import LinearSVC\n", + "iris = datasets.load_iris()\n", + "X = iris[\"data\"][:, (2, 3)] # petal length, petal width\n", + "y = (iris[\"target\"] == 2).astype(np.float64) # Iris-Virginica\n", + "svm_clf = Pipeline((\n", + "(\"scaler\", StandardScaler()),\n", + "(\"linear_svc\", LinearSVC(C=1, loss=\"hinge\")),\n", + "))\n", + "svm_clf.fit(X_scaled, y)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Alternatively, you could use the SVC class, using **SVC(kernel=\"linear\", C=1)**, but it is much slower,\n", + "especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier\n", + "class, with **SGDClassifier(loss=\"hinge\", alpha=1/(m*C))**. This applies regular Stochastic\n", + "Gradient Descentto train a linear SVM classifier. It does not converge as fast as the\n", + "LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core\n", + "training), or to handle online classification tasks.\n", "\n", - "# Generate sample data\n", - "X = np.sort(5*np.random.rand(40,1), axis=0)\n", - "y = X**3\n", - "y=y.ravel()\n", + "## SVMs, adding polynomial features\n", "\n", - "# Add noise to targets\n", - "X[::4] +=3*(0.5 - np.random.rand(1))\n", - "y[::5] += 50 * (0.5 - np.random.rand(8))\n", + "Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets\n", + "are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more\n", + "features, such as polynomial features. In some cases this can result in a linearly\n", + "separable dataset." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "from sklearn.datasets import make_moons\n", + "from sklearn.pipeline import Pipeline\n", + "from sklearn.preprocessing import PolynomialFeatures\n", + "polynomial_svm_clf = Pipeline((\n", + "(\"poly_features\", PolynomialFeatures(degree=3)),\n", + "(\"scaler\", StandardScaler()),\n", + "(\"svm_clf\", LinearSVC(C=10, loss=\"hinge\"))\n", + "))\n", + "polynomial_svm_clf.fit(X, y)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## SVMs, polynomials and kernels\n", "\n", - "plt.plot(X,y, 'g^')\n", + "Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning\n", + "algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets,\n", + "and with a high polynomial degree it creates a huge number of features, making the model too slow.\n", + "Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the\n", + "kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many\n", + "polynomial features, even with very high-degree polynomials, without actually having to add them. So\n", + "there is no combinatorial explosion of the number of features since you don’t actually add any features.\n", + "This trick is implemented by the SVC class." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "from sklearn.svm import SVC\n", + "poly_kernel_svm_clf = Pipeline((\n", + "(\"scaler\", StandardScaler()),\n", + "(\"svm_clf\", SVC(kernel=\"poly\", degree=3, coef0=1, C=5))\n", + "))\n", + "poly_kernel_svm_clf.fit(X, y)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This code trains an SVM classifier using a 3rd-degree polynomial kernel.\n", "\n", - "#SVR Fit\n", - "svr_poly = SVR(kernel='poly', C=1e3, degree=3)\n", - "y_poly = svr_poly.fit(X, y).predict(X)\n", "\n", - "# Plots\n", - "z = np.arange(0, 5, 0.1)\n", - "t = z**3\n", - "fig = plt.figure()\n", - "ax = fig.add_subplot(111)\n", - "plt.plot(z,z**3, 'r--', label='Cubic Function with No Noise')\n", - "lw = 2\n", - "plt.scatter(X, y, color='darkorange', label='Gaussian Cubic Noise')\n", - "plt.plot(X, y_poly, color='green', lw=lw, label='Polynomial model')\n", - "plt.xlabel('data')\n", - "plt.ylabel('target')\n", - "plt.title('Cubic Gaussian Distribution')\n", - "plt.legend()\n", - "plt.show()" + "## SVMs, regression\n", + "\n", + "You can use Scikit-Learn’s LinearSVR class to perform linear SVM Regression. The following code\n", + "shows how to use the regression option (more material to come)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "from sklearn.svm import LinearSVR\n", + "svm_reg = LinearSVR(epsilon=1.5)\n", + "svm_reg.fit(X, y)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To tackle nonlinear regression tasks, you can use a kernelized SVM model" ] } ], diff --git a/doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf b/doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf index 3991ef9c0..9ca94c518 100644 Binary files a/doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf and b/doc/pub/svm/pdf/svm-beamer-handouts2x3.pdf differ diff --git a/doc/pub/svm/pdf/svm-beamer.pdf b/doc/pub/svm/pdf/svm-beamer.pdf index 77324339a..a6d8ec491 100644 Binary files a/doc/pub/svm/pdf/svm-beamer.pdf and b/doc/pub/svm/pdf/svm-beamer.pdf differ diff --git a/doc/pub/svm/pdf/svm-minted.pdf b/doc/pub/svm/pdf/svm-minted.pdf index 64e86e7f4..a773ef0fd 100644 Binary files a/doc/pub/svm/pdf/svm-minted.pdf and b/doc/pub/svm/pdf/svm-minted.pdf differ diff --git a/doc/src/DecisionTrees/DecisionTrees.do.txt b/doc/src/DecisionTrees/DecisionTrees.do.txt index 3ba739bee..7ba5efc40 100644 --- a/doc/src/DecisionTrees/DecisionTrees.do.txt +++ b/doc/src/DecisionTrees/DecisionTrees.do.txt @@ -6,7 +6,15 @@ DATE: today !split ===== Decision trees, overarching aims ===== !bblock -Add text about decision trees and include about random forests (use Ising model classification) + +Decision trees are supervised learning algorithms used for both, +classification and regression tasks where we will concentrate on +classification in this first part of our decision tree tutorial. +Decision trees are assigned to the information based learning +algorithms which use different measures of information gain for +learning. We can use decision trees for issues where we have +continuous but also categorical input and target features. + !eblock !split diff --git a/doc/src/SupportVMachines/svm.do.txt b/doc/src/SupportVMachines/svm.do.txt index 2ca500283..c9777f557 100644 --- a/doc/src/SupportVMachines/svm.do.txt +++ b/doc/src/SupportVMachines/svm.do.txt @@ -2,42 +2,106 @@ TITLE: Data Analysis and Machine Learning: Support Vector Machines AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University DATE: today - !split ===== Support Vector Machines, overarching aims ===== +A Support Vector Machine (SVM) is a very powerful and versatile +Machine Learning model, capable of performing linear or nonlinear +classification, regression, and even outlier detection. It is one of +the most popular models in Machine Learning, and anyone interested in +Machine Learning should have it in their toolbox. SVMs are +particularly well suited for classification of complex but small-sized or +medium-sized datasets. + +The basic mathematics relies on the definition of hyperplanes and the definition of a _margin_ which separates +classes (in case of classification problems) of variables. It is also used for regression problems. + +With SVMs we distinguish between hard margin and soft margins. The latter introduces a so-called softening parameter to be discussed below. +We distringuish also between linearn and non-linear approaches. + +_These notes will be updated shortly with more material_ + +!split +===== SVMs, basics with scikit-learn ===== + +The following Scikit-Learn code loads the iris dataset, scales the features, and then trains a linear SVM +model (using the LinearSVC class with C = 0.1 and the hinge loss function, described shortly) to detect +Iris-Virginica flowers. !bc pycod import numpy as np -from sklearn.svm import SVR -import matplotlib.pyplot as plt - -# Generate sample data -X = np.sort(5*np.random.rand(40,1), axis=0) -y = X**3 -y=y.ravel() - -# Add noise to targets -X[::4] +=3*(0.5 - np.random.rand(1)) -y[::5] += 50 * (0.5 - np.random.rand(8)) - -plt.plot(X,y, 'g^') - -#SVR Fit -svr_poly = SVR(kernel='poly', C=1e3, degree=3) -y_poly = svr_poly.fit(X, y).predict(X) - -# Plots -z = np.arange(0, 5, 0.1) -t = z**3 -fig = plt.figure() -ax = fig.add_subplot(111) -plt.plot(z,z**3, 'r--', label='Cubic Function with No Noise') -lw = 2 -plt.scatter(X, y, color='darkorange', label='Gaussian Cubic Noise') -plt.plot(X, y_poly, color='green', lw=lw, label='Polynomial model') -plt.xlabel('data') -plt.ylabel('target') -plt.title('Cubic Gaussian Distribution') -plt.legend() -plt.show() +from sklearn import datasets +from sklearn.pipeline import Pipeline +from sklearn.preprocessing import StandardScaler +from sklearn.svm import LinearSVC +iris = datasets.load_iris() +X = iris["data"][:, (2, 3)] # petal length, petal width +y = (iris["target"] == 2).astype(np.float64) # Iris-Virginica +svm_clf = Pipeline(( +("scaler", StandardScaler()), +("linear_svc", LinearSVC(C=1, loss="hinge")), +)) +svm_clf.fit(X_scaled, y) !ec + +Alternatively, you could use the SVC class, using _SVC(kernel="linear", C=1)_, but it is much slower, +especially with large training sets, so it is not recommended. Another option is to use the SGDClassifier +class, with _SGDClassifier(loss="hinge", alpha=1/(m*C))_. This applies regular Stochastic +Gradient Descentto train a linear SVM classifier. It does not converge as fast as the +LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core +training), or to handle online classification tasks. + +!split +===== SVMs, adding polynomial features ===== + +Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets +are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more +features, such as polynomial features. In some cases this can result in a linearly +separable dataset. + +!bc pycod +from sklearn.datasets import make_moons +from sklearn.pipeline import Pipeline +from sklearn.preprocessing import PolynomialFeatures +polynomial_svm_clf = Pipeline(( +("poly_features", PolynomialFeatures(degree=3)), +("scaler", StandardScaler()), +("svm_clf", LinearSVC(C=10, loss="hinge")) +)) +polynomial_svm_clf.fit(X, y) +!ec + + +!split +===== SVMs, polynomials and kernels ===== + +Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning +algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets, +and with a high polynomial degree it creates a huge number of features, making the model too slow. +Fortunately, when using SVMs you can apply an almost miraculous mathematical technique called the +kernel trick discussed during the lectures. It makes it possible to get the same result as if you added many +polynomial features, even with very high-degree polynomials, without actually having to add them. So +there is no combinatorial explosion of the number of features since you don’t actually add any features. +This trick is implemented by the SVC class. + +!bc pycod +from sklearn.svm import SVC +poly_kernel_svm_clf = Pipeline(( +("scaler", StandardScaler()), +("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5)) +)) +poly_kernel_svm_clf.fit(X, y) +!ec +This code trains an SVM classifier using a 3rd-degree polynomial kernel. + + +!split +===== SVMs, regression ===== + +You can use Scikit-Learn’s LinearSVR class to perform linear SVM Regression. The following code +shows how to use the regression option (more material to come) +!bc pycod +from sklearn.svm import LinearSVR +svm_reg = LinearSVR(epsilon=1.5) +svm_reg.fit(X, y) +!ec +To tackle nonlinear regression tasks, you can use a kernelized SVM model