Morten Hjorth-Jensen
2966 lines
79 KiB
Plaintext
2966 lines
79 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cd7f4b8e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html week45.do.txt --no_mako -->\n",
|
|
"<!-- dom:TITLE: Week 45: Decisions Trees, Random Forests, Bagging and Boosting -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e69a03fa",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Week 45: Decisions Trees, Random Forests, Bagging and Boosting\n",
|
|
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
|
|
"\n",
|
|
"Date: **Nov 11, 2022**\n",
|
|
"\n",
|
|
"Copyright 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b6699ca4",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Overview of week 45\n",
|
|
"\n",
|
|
"* Thursday: Boosting methods, froma AdaBoost to Gradient boosting\n",
|
|
"\n",
|
|
" * [Video of lecture](https://youtu.be/mK48PfCxgYk)\n",
|
|
"\n",
|
|
"* Friday: Gradient boosting and discussion of Decision trees and ensemble methods. Wrapping up trees and start discussing Support Vector Machines\n",
|
|
"\n",
|
|
" * [Video of lecture](https://youtu.be/v8eJBFeZKuI)\n",
|
|
"\n",
|
|
"**Videos.**\n",
|
|
"\n",
|
|
"1. [Video on Decision trees](https://www.youtube.com/watch?v=RmajweUFKvM&ab_channel=Simplilearn)\n",
|
|
"\n",
|
|
"2. [Video on boosting methods by Hastie](https://www.youtube.com/watch?v=wPqtzj5VZus&ab_channel=H2O.ai).\n",
|
|
"\n",
|
|
"3. [Video on AdaBoost](https://www.youtube.com/watch?v=LsK-xG1cLYA)\n",
|
|
"\n",
|
|
"4. [Video on Gradient boost, part 1, parts 2-4 follows](https://www.youtube.com/watch?v=3CC4N4z3GJc)\n",
|
|
"\n",
|
|
"**Reading.**\n",
|
|
"\n",
|
|
"1. [Hastie et al, chapter 10.1-10.10](https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/elementsstat.pdf). Geron's chapters 6 and 7 are also useful."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ecf8a21b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Brief code reminder from last wekk"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "1b4aadd7",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"%matplotlib inline\n",
|
|
"# Common imports\n",
|
|
"from IPython.display import Image \n",
|
|
"from pydot import graph_from_dot_data\n",
|
|
"import pandas as pd\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from sklearn.model_selection import train_test_split\n",
|
|
"from sklearn.tree import export_graphviz\n",
|
|
"from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
|
|
"from sklearn.compose import ColumnTransformer\n",
|
|
"from pydot import graph_from_dot_data\n",
|
|
"from sklearn.datasets import load_breast_cancer\n",
|
|
"from sklearn.svm import SVC\n",
|
|
"from sklearn.linear_model import LogisticRegression\n",
|
|
"from sklearn.tree import DecisionTreeClassifier\n",
|
|
"from sklearn.ensemble import RandomForestClassifier\n",
|
|
"from sklearn.preprocessing import LabelEncoder\n",
|
|
"from sklearn.model_selection import cross_validate\n",
|
|
"import scikitplot as skplt\n",
|
|
"from sklearn.preprocessing import StandardScaler\n",
|
|
"import os\n",
|
|
"\n",
|
|
"# Where to save the figures and data files\n",
|
|
"PROJECT_ROOT_DIR = \"Results\"\n",
|
|
"FIGURE_ID = \"Results/FigureFiles\"\n",
|
|
"DATA_ID = \"DataFiles/\"\n",
|
|
"\n",
|
|
"if not os.path.exists(PROJECT_ROOT_DIR):\n",
|
|
" os.mkdir(PROJECT_ROOT_DIR)\n",
|
|
"\n",
|
|
"if not os.path.exists(FIGURE_ID):\n",
|
|
" os.makedirs(FIGURE_ID)\n",
|
|
"\n",
|
|
"if not os.path.exists(DATA_ID):\n",
|
|
" os.makedirs(DATA_ID)\n",
|
|
"\n",
|
|
"def image_path(fig_id):\n",
|
|
" return os.path.join(FIGURE_ID, fig_id)\n",
|
|
"\n",
|
|
"def data_path(dat_id):\n",
|
|
" return os.path.join(DATA_ID, dat_id)\n",
|
|
"\n",
|
|
"def save_fig(fig_id):\n",
|
|
" plt.savefig(image_path(fig_id) + \".png\", format='png')\n",
|
|
"\n",
|
|
"# Load the cancer data\n",
|
|
"cancer = load_breast_cancer()\n",
|
|
"\n",
|
|
"X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)\n",
|
|
"print(X_train.shape)\n",
|
|
"print(X_test.shape)\n",
|
|
"#Scale the data\n",
|
|
"scaler = StandardScaler()\n",
|
|
"scaler.fit(X_train)\n",
|
|
"X_train_scaled = scaler.transform(X_train)\n",
|
|
"X_test_scaled = scaler.transform(X_test)\n",
|
|
"#define methods\n",
|
|
"# Logistic Regression\n",
|
|
"logreg = LogisticRegression(solver='lbfgs')\n",
|
|
"logreg.fit(X_train_scaled, y_train)\n",
|
|
"print(\"Test set accuracy Logistic Regression with scaled data: {:.2f}\".format(logreg.score(X_test_scaled,y_test)))\n",
|
|
"# Decision Trees\n",
|
|
"deep_tree_clf = DecisionTreeClassifier(max_depth=None)\n",
|
|
"deep_tree_clf.fit(X_train_scaled, y_train)\n",
|
|
"print(\"Test set accuracy with Decision Trees and scaled data: {:.2f}\".format(deep_tree_clf.score(X_test_scaled,y_test)))\n",
|
|
"# Support Vector Machine\n",
|
|
"svm = SVC(gamma='auto', C=100)\n",
|
|
"svm.fit(X_train_scaled, y_train)\n",
|
|
"print(\"Test set accuracy SVM with scaled data: {:.2f}\".format(logreg.score(X_test_scaled,y_test)))\n",
|
|
"# Random forests\n",
|
|
"#Instantiate the model with 500 trees and entropy as splitting criteria\n",
|
|
"Random_Forest_model = RandomForestClassifier(n_estimators=500,criterion=\"entropy\")\n",
|
|
"Random_Forest_model.fit(X_train_scaled, y_train)\n",
|
|
"print(\"Test set accuracy with Random Forests and scaled data: {:.2f}\".format(Random_Forest_model.score(X_test_scaled,y_test)))\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"y_pred = Random_Forest_model.predict(X_test_scaled)\n",
|
|
"skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)\n",
|
|
"plt.show()\n",
|
|
"y_probas = Random_Forest_model.predict_proba(X_test_scaled)\n",
|
|
"skplt.metrics.plot_roc(y_test, y_probas)\n",
|
|
"plt.show()\n",
|
|
"skplt.metrics.plot_cumulative_gain(y_test, y_probas)\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5538e026",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Boosting, a Bird's Eye View\n",
|
|
"\n",
|
|
"The basic idea is to combine weak classifiers in order to create a good\n",
|
|
"classifier. With a weak classifier we often intend a classifier which\n",
|
|
"produces results which are only slightly better than we would get by\n",
|
|
"random guesses.\n",
|
|
"\n",
|
|
"This is done by applying in an iterative way a weak (or a standard\n",
|
|
"classifier like decision trees) to modify the data. In each iteration\n",
|
|
"we emphasize those observations which are misclassified by weighting\n",
|
|
"them with a factor."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "29717f07",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## What is boosting? Additive Modelling/Iterative Fitting\n",
|
|
"\n",
|
|
"Boosting is a way of fitting an additive expansion in a set of\n",
|
|
"elementary basis functions like for example some simple polynomials.\n",
|
|
"Assume for example that we have a function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "06c553fc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"f_M(x) = \\sum_{i=1}^M \\beta_m b(x;\\gamma_m),\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7fe6be24",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"where $\\beta_m$ are the expansion parameters to be determined in a\n",
|
|
"minimization process and $b(x;\\gamma_m)$ are some simple functions of\n",
|
|
"the multivariable parameter $x$ which is characterized by the\n",
|
|
"parameters $\\gamma_m$.\n",
|
|
"\n",
|
|
"As an example, consider the Sigmoid function we used in logistic\n",
|
|
"regression. In that case, we can translate the function\n",
|
|
"$b(x;\\gamma_m)$ into the Sigmoid function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0a5e2909",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\sigma(t) = \\frac{1}{1+\\exp{(-t)}},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c17e4276",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"where $t=\\gamma_0+\\gamma_1 x$ and the parameters $\\gamma_0$ and\n",
|
|
"$\\gamma_1$ were determined by the Logistic Regression fitting\n",
|
|
"algorithm.\n",
|
|
"\n",
|
|
"As another example, consider the cost function we defined for linear regression"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4aade453",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"C(\\boldsymbol{y},\\boldsymbol{f}) = \\frac{1}{n} \\sum_{i=0}^{n-1}(y_i-f(x_i))^2.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3e0f9b03",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"In this case the function $f(x)$ was replaced by the design matrix\n",
|
|
"$\\boldsymbol{X}$ and the unknown linear regression parameters $\\boldsymbol{\\beta}$,\n",
|
|
"that is $\\boldsymbol{f}=\\boldsymbol{X}\\boldsymbol{\\beta}$. In linear regression we can \n",
|
|
"simply invert a matrix and obtain the parameters $\\beta$ by"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5f490e78",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\boldsymbol{\\beta}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "68381e7a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"In iterative fitting or additive modeling, we minimize the cost function with respect to the parameters $\\beta_m$ and $\\gamma_m$."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "74833197",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Iterative Fitting, Regression and Squared-error Cost Function\n",
|
|
"\n",
|
|
"The way we proceed is as follows (here we specialize to the squared-error cost function)\n",
|
|
"\n",
|
|
"1. Establish a cost function, here ${\\cal C}(\\boldsymbol{y},\\boldsymbol{f}) = \\frac{1}{n} \\sum_{i=0}^{n-1}(y_i-f_M(x_i))^2$ with $f_M(x) = \\sum_{i=1}^M \\beta_m b(x;\\gamma_m)$.\n",
|
|
"\n",
|
|
"2. Initialize with a guess $f_0(x)$. It could be one or even zero or some random numbers.\n",
|
|
"\n",
|
|
"3. For $m=1:M$\n",
|
|
"\n",
|
|
"a. minimize $\\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta b(x;\\gamma))^2$ wrt $\\gamma$ and $\\beta$\n",
|
|
"\n",
|
|
"b. This gives the optimal values $\\beta_m$ and $\\gamma_m$\n",
|
|
"\n",
|
|
"c. Determine then the new values $f_m(x)=f_{m-1}(x) +\\beta_m b(x;\\gamma_m)$\n",
|
|
"\n",
|
|
"We could use any of the algorithms we have discussed till now. If we\n",
|
|
"use trees, $\\gamma$ parameterizes the split variables and split points\n",
|
|
"at the internal nodes, and the predictions at the terminal nodes."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1665b814",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Squared-Error Example and Iterative Fitting\n",
|
|
"\n",
|
|
"To better understand what happens, let us develop the steps for the iterative fitting using the above squared error function.\n",
|
|
"\n",
|
|
"For simplicity we assume also that our functions $b(x;\\gamma)=1+\\gamma x$. \n",
|
|
"\n",
|
|
"This means that for every iteration $m$, we need to optimize"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2a26b317",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"(\\beta_m,\\gamma_m) = \\mathrm{argmin}_{\\beta,\\lambda}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta b(x;\\gamma))^2=\\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta(1+\\gamma x_i))^2.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3e62ea5c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We start our iteration by simply setting $f_0(x)=0$. \n",
|
|
"Taking the derivatives with respect to $\\beta$ and $\\gamma$ we obtain"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bd2a83b1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial {\\cal C}}{\\partial \\beta} = -2\\sum_{i}(1+\\gamma x_i)(y_i-\\beta(1+\\gamma x_i))=0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e8c0b65f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"and"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8e6601d0",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial {\\cal C}}{\\partial \\gamma} =-2\\sum_{i}\\beta x_i(y_i-\\beta(1+\\gamma x_i))=0.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b4b64e05",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We can then rewrite these equations as (defining $\\boldsymbol{w}=\\boldsymbol{e}+\\gamma \\boldsymbol{x})$ with $\\boldsymbol{e}$ being the unit vector)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f22c6be2",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\gamma \\boldsymbol{w}^T(\\boldsymbol{y}-\\beta\\gamma \\boldsymbol{w})=0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "eeffe058",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"which gives us $\\beta = \\boldsymbol{w}^T\\boldsymbol{y}/(\\boldsymbol{w}^T\\boldsymbol{w})$. Similarly we have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1a9e224c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\beta\\gamma \\boldsymbol{x}^T(\\boldsymbol{y}-\\beta(1+\\gamma \\boldsymbol{x}))=0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0f69098e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"which leads to $\\gamma =(\\boldsymbol{x}^T\\boldsymbol{y}-\\beta\\boldsymbol{x}^T\\boldsymbol{e})/(\\beta\\boldsymbol{x}^T\\boldsymbol{x})$. Inserting\n",
|
|
"for $\\beta$ gives us an equation for $\\gamma$. This is a non-linear equation in the unknown $\\gamma$ and has to be solved numerically. \n",
|
|
"\n",
|
|
"The solution to these two equations gives us in turn $\\beta_1$ and $\\gamma_1$ leading to the new expression for $f_1(x)$ as\n",
|
|
"$f_1(x) = \\beta_1(1+\\gamma_1x)$. Doing this $M$ times results in our final estimate for the function $f$."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9c740382",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Iterative Fitting, Classification and AdaBoost\n",
|
|
"\n",
|
|
"Let us consider a binary classification problem with two outcomes $y_i \\in \\{-1,1\\}$ and $i=0,1,2,\\dots,n-1$ as our set of\n",
|
|
"observations. We define a classification function $G(x)$ which produces a prediction taking one or the other of the two values \n",
|
|
"$\\{-1,1\\}$.\n",
|
|
"\n",
|
|
"The error rate of the training sample is then"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e8e50bc7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\mathrm{\\overline{err}}=\\frac{1}{n} \\sum_{i=0}^{n-1} I(y_i\\ne G(x_i)).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1e8ba0b5",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The iterative procedure starts with defining a weak classifier whose\n",
|
|
"error rate is barely better than random guessing. The iterative\n",
|
|
"procedure in boosting is to sequentially apply a weak\n",
|
|
"classification algorithm to repeatedly modified versions of the data\n",
|
|
"producing a sequence of weak classifiers $G_m(x)$.\n",
|
|
"\n",
|
|
"Here we will express our function $f(x)$ in terms of $G(x)$. That is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "989c7213",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"f_M(x) = \\sum_{i=1}^M \\beta_m b(x;\\gamma_m),\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e83bfc29",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"will be a function of"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b639ef4c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"G_M(x) = \\mathrm{sign} \\sum_{i=1}^M \\alpha_m G_m(x).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ed79d30c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Adaptive Boosting, AdaBoost\n",
|
|
"\n",
|
|
"In our iterative procedure we define thus"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "72f95b03",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"f_m(x) = f_{m-1}(x)+\\beta_mG_m(x).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b663c55e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the\n",
|
|
"exponential cost/loss function defined as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ccb209ab",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"C(\\boldsymbol{y},\\boldsymbol{f}) = \\sum_{i=0}^{n-1}\\exp{(-y_i(f_{m-1}(x_i)+\\beta G(x_i))}.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "28c98008",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We optimize $\\beta$ and $G$ for each value of $m=1:M$ as we did in the regression case.\n",
|
|
"This is normally done in two steps. Let us however first rewrite the cost function as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d3049d99",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"C(\\boldsymbol{y},\\boldsymbol{f}) = \\sum_{i=0}^{n-1}w_i^{m}\\exp{(-y_i\\beta G(x_i))},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "16b769d1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"where we have defined $w_i^m= \\exp{(-y_if_{m-1}(x_i))}$."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d4b782f4",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Building up AdaBoost\n",
|
|
"\n",
|
|
"First, for any $\\beta > 0$, we optimize $G$ by setting"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2ca48812",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"G_m(x) = \\mathrm{sign} \\sum_{i=0}^{n-1} w_i^m I(y_i \\ne G_(x_i)),\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f93041cc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"which is the classifier that minimizes the weighted error rate in predicting $y$.\n",
|
|
"\n",
|
|
"We can do this by rewriting"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "54cc66f0",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\exp{-(\\beta)}\\sum_{y_i=G(x_i)}w_i^m+\\exp{(\\beta)}\\sum_{y_i\\ne G(x_i)}w_i^m,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c1295837",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"which can be rewritten as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e852f6f4",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"(\\exp{(\\beta)}-\\exp{-(\\beta)})\\sum_{i=0}^{n-1}w_i^mI(y_i\\ne G(x_i))+\\exp{(-\\beta)}\\sum_{i=0}^{n-1}w_i^m=0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0e97ee90",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"which leads to"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2693f93d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\beta_m = \\frac{1}{2}\\log{\\frac{1-\\mathrm{\\overline{err}}}{\\mathrm{\\overline{err}}}},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "11d0f282",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"where we have redefined the error as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9e95bb3c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\mathrm{\\overline{err}}_m=\\frac{1}{n}\\frac{\\sum_{i=0}^{n-1}w_i^mI(y_i\\ne G(x_i)}{\\sum_{i=0}^{n-1}w_i^m},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a4f400fd",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"which leads to an update of"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ab213472",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"f_m(x) = f_{m-1}(x) +\\beta_m G_m(x).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a226047c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"This leads to the new weights"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "33c07d01",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"w_i^{m+1} = w_i^m \\exp{(-y_i\\beta_m G_m(x_i))}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "95c74250",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Adaptive boosting: AdaBoost, Basic Algorithm\n",
|
|
"\n",
|
|
"The algorithm here is rather straightforward. Assume that our weak\n",
|
|
"classifier is a decision tree and we consider a binary set of outputs\n",
|
|
"with $y_i \\in \\{-1,1\\}$ and $i=0,1,2,\\dots,n-1$ as our set of\n",
|
|
"observations. Our design matrix is given in terms of the\n",
|
|
"feature/predictor vectors\n",
|
|
"$\\boldsymbol{X}=[\\boldsymbol{x}_0\\boldsymbol{x}_1\\dots\\boldsymbol{x}_{p-1}]$. Finally, we define also a\n",
|
|
"classifier determined by our data via a function $G(x)$. This function tells us how well we are able to classify our outputs/targets $\\boldsymbol{y}$. \n",
|
|
"\n",
|
|
"We have already defined the misclassification error $\\mathrm{err}$ as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d0615c8e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\mathrm{err}=\\frac{1}{n}\\sum_{i=0}^{n-1}I(y_i\\ne G(x_i)),\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2344d3a7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"where the function $I()$ is one if we misclassify and zero if we classify correctly."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3a76c5b8",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Basic Steps of AdaBoost\n",
|
|
"\n",
|
|
"With the above definitions we are now ready to set up the algorithm for AdaBoost.\n",
|
|
"The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.\n",
|
|
"1. We start by initializing all weights to $w_i = 1/n$, with $i=0,1,2,\\dots n-1$. It is easy to see that we must have $\\sum_{i=0}^{n-1}w_i = 1$.\n",
|
|
"\n",
|
|
"2. We rewrite the misclassification error as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f0f99425",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\mathrm{\\overline{err}}_m=\\frac{\\sum_{i=0}^{n-1}w_i^m I(y_i\\ne G(x_i))}{\\sum_{i=0}^{n-1}w_i},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "191399f6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"1. Then we start looping over all attempts at classifying, namely we start an iterative process for $m=1:M$, where $M$ is the final number of classifications. Our given classifier could for example be a plain decision tree.\n",
|
|
"\n",
|
|
"a. Fit then a given classifier to the training set using the weights $w_i$.\n",
|
|
"\n",
|
|
"b. Compute then $\\mathrm{err}$ and figure out which events are classified properly and which are classified wrongly.\n",
|
|
"\n",
|
|
"c. Define a quantity $\\alpha_{m} = \\log{(1-\\mathrm{\\overline{err}}_m)/\\mathrm{\\overline{err}}_m}$\n",
|
|
"\n",
|
|
"d. Set the new weights to $w_i = w_i\\times \\exp{(\\alpha_m I(y_i\\ne G(x_i)}$.\n",
|
|
"\n",
|
|
"5. Compute the new classifier $G(x)= \\sum_{i=0}^{n-1}\\alpha_m I(y_i\\ne G(x_i)$.\n",
|
|
"\n",
|
|
"For the iterations with $m \\le 2$ the weights are modified\n",
|
|
"individually at each steps. The observations which were misclassified\n",
|
|
"at iteration $m-1$ have a weight which is larger than those which were\n",
|
|
"classified properly. As this proceeds, the observations which were\n",
|
|
"difficult to classifiy correctly are given a larger influence. Each\n",
|
|
"new classification step $m$ is then forced to concentrate on those\n",
|
|
"observations that are missed in the previous iterations."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ab4b7eda",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## AdaBoost Examples\n",
|
|
"\n",
|
|
"Using **Scikit-Learn** it is easy to apply the adaptive boosting algorithm, as done here."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "6971c306",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn.ensemble import AdaBoostClassifier\n",
|
|
"\n",
|
|
"ada_clf = AdaBoostClassifier(\n",
|
|
" DecisionTreeClassifier(max_depth=2), n_estimators=200,\n",
|
|
" algorithm=\"SAMME.R\", learning_rate=0.01, random_state=42)\n",
|
|
"ada_clf.fit(X_train, y_train)\n",
|
|
"y_pred = ada_clf.predict(X_test)\n",
|
|
"skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)\n",
|
|
"plt.show()\n",
|
|
"y_probas = ada_clf.predict_proba(X_test)\n",
|
|
"skplt.metrics.plot_roc(y_test, y_probas)\n",
|
|
"plt.show()\n",
|
|
"skplt.metrics.plot_cumulative_gain(y_test, y_probas)\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "47fb4471",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent\n",
|
|
"\n",
|
|
"Gradient boosting is again a similar technique to Adaptive boosting,\n",
|
|
"it combines so-called weak classifiers or regressors into a strong\n",
|
|
"method via a series of iterations.\n",
|
|
"\n",
|
|
"In order to understand the method, let us illustrate its basics by\n",
|
|
"bringing back the essential steps in linear regression, where our cost\n",
|
|
"function was the least squares function."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "325ac013",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## The Squared-Error again! Steepest Descent\n",
|
|
"\n",
|
|
"We start again with our cost function ${\\cal C}(\\boldsymbol{y}m\\boldsymbol{f})=\\sum_{i=0}^{n-1}{\\cal L}(y_i, f(x_i))$ where we want to minimize\n",
|
|
"This means that for every iteration, we need to optimize"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "69711cc3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"(\\hat{\\boldsymbol{f}}) = \\mathrm{argmin}_{\\boldsymbol{f}}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i-f(x_i))^2.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "74c7d65e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We define a real function $h_m(x)$ that defines our final function $f_M(x)$ as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0845ab63",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"f_M(x) = \\sum_{m=0}^M h_m(x).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c88fc6c2",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"In the steepest decent approach we approximate $h_m(x) = -\\rho_m g_m(x)$, where $\\rho_m$ is a scalar and $g_m(x)$ the gradient defined as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "52fec20d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"g_m(x_i) = \\left[ \\frac{\\partial {\\cal L}(y_i, f(x_i))}{\\partial f(x_i)}\\right]_{f(x_i)=f_{m-1}(x_i)}.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "616b81c7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"With the new gradient we can update $f_m(x) = f_{m-1}(x) -\\rho_m g_m(x)$. Using the above squared-error function we see that\n",
|
|
"the gradient is $g_m(x_i) = -2(y_i-f(x_i))$.\n",
|
|
"\n",
|
|
"Choosing $f_0(x)=0$ we obtain $g_m(x) = -2y_i$ and inserting this into the minimization problem for the cost function we have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e4666f20",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"(\\rho_1) = \\mathrm{argmin}_{\\rho}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i+2\\rho y_i)^2.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ece3d7b6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Steepest Descent Example\n",
|
|
"\n",
|
|
"Optimizing with respect to $\\rho$ we obtain (taking the derivative) that $\\rho_1 = -1/2$. We have then that"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3b53f90b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"f_1(x) = f_{0}(x) -\\rho_1 g_1(x)=-y_i.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0ca7cbdf",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We can then proceed and compute"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ffaf8839",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"g_2(x_i) = \\left[ \\frac{\\partial {\\cal L}(y_i, f(x_i))}{\\partial f(x_i)}\\right]_{f(x_i)=f_{1}(x_i)=y_i}=-4y_i,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "58a6e958",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"and find a new value for $\\rho_2=-1/2$ and continue till we have reached $m=M$. We can modify the steepest descent method, or steepest boosting, by introducing what is called **gradient boosting**."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "00f5f746",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Gradient Boosting, algorithm\n",
|
|
"\n",
|
|
"Steepest descent is however not much used, since it only optimizes $f$ at a fixed set of $n$ points,\n",
|
|
"so we do not learn a function that can generalize. However, we can modify the algorithm by\n",
|
|
"fitting a weak learner to approximate the negative gradient signal. \n",
|
|
"\n",
|
|
"Suppose we have a cost function $C(f)=\\sum_{i=0}^{n-1}L(y_i, f(x_i))$ where $y_i$ is our target and $f(x_i)$ the function which is meant to model $y_i$. The above cost function could be our standard squared-error function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4ad37156",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"C(\\boldsymbol{y},\\boldsymbol{f})=\\sum_{i=0}^{n-1}(y_i-f(x_i))^2.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c29dcd04",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The way we proceed in an iterative fashion is to\n",
|
|
"1. Initialize our estimate $f_0(x)$.\n",
|
|
"\n",
|
|
"2. For $m=1:M$, we\n",
|
|
"\n",
|
|
"a. compute the negative gradient vector $\\boldsymbol{u}_m = -\\partial C(\\boldsymbol{y},\\boldsymbol{f})/\\partial \\boldsymbol{f}(x)$ at $f(x) = f_{m-1}(x)$;\n",
|
|
"\n",
|
|
"b. fit the so-called base-learner to the negative gradient $h_m(u_m,x)$;\n",
|
|
"\n",
|
|
"c. update the estimate $f_m(x) = f_{m-1}(x)+h_m(u_m,x)$;\n",
|
|
"\n",
|
|
"4. The final estimate is then $f_M(x) = \\sum_{m=1}^M h_m(u_m,x)$."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "22ae18f1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Gradient Boosting, Examples of Regression"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "f8965766",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import numpy as np\n",
|
|
"from sklearn.model_selection import train_test_split\n",
|
|
"from sklearn.ensemble import GradientBoostingRegressor\n",
|
|
"import scikitplot as skplt\n",
|
|
"from sklearn.metrics import mean_squared_error\n",
|
|
"\n",
|
|
"n = 100\n",
|
|
"maxdegree = 6\n",
|
|
"\n",
|
|
"# Make data set.\n",
|
|
"x = np.linspace(-3, 3, n).reshape(-1, 1)\n",
|
|
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)\n",
|
|
"\n",
|
|
"error = np.zeros(maxdegree)\n",
|
|
"bias = np.zeros(maxdegree)\n",
|
|
"variance = np.zeros(maxdegree)\n",
|
|
"polydegree = np.zeros(maxdegree)\n",
|
|
"X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)\n",
|
|
"\n",
|
|
"for degree in range(1,maxdegree):\n",
|
|
" model = GradientBoostingRegressor(max_depth=degree, n_estimators=100, learning_rate=1.0) \n",
|
|
" model.fit(X_train,y_train)\n",
|
|
" y_pred = model.predict(X_test)\n",
|
|
" polydegree[degree] = degree\n",
|
|
" error[degree] = np.mean( np.mean((y_test - y_pred)**2) )\n",
|
|
" bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )\n",
|
|
" variance[degree] = np.mean( np.var(y_pred) )\n",
|
|
" print('Max depth:', degree)\n",
|
|
" print('Error:', error[degree])\n",
|
|
" print('Bias^2:', bias[degree])\n",
|
|
" print('Var:', variance[degree])\n",
|
|
" print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))\n",
|
|
"\n",
|
|
"plt.xlim(1,maxdegree-1)\n",
|
|
"plt.plot(polydegree, error, label='Error')\n",
|
|
"plt.plot(polydegree, bias, label='bias')\n",
|
|
"plt.plot(polydegree, variance, label='Variance')\n",
|
|
"plt.legend()\n",
|
|
"save_fig(\"gdregression\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4c2f9143",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Gradient Boosting, Classification Example"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "8de8fbc9",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import numpy as np\n",
|
|
"from sklearn.model_selection import train_test_split \n",
|
|
"from sklearn.datasets import load_breast_cancer\n",
|
|
"import scikitplot as skplt\n",
|
|
"from sklearn.ensemble import GradientBoostingClassifier\n",
|
|
"from sklearn.model_selection import cross_validate\n",
|
|
"\n",
|
|
"# Load the data\n",
|
|
"cancer = load_breast_cancer()\n",
|
|
"\n",
|
|
"X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)\n",
|
|
"print(X_train.shape)\n",
|
|
"print(X_test.shape)\n",
|
|
"#now scale the data\n",
|
|
"from sklearn.preprocessing import StandardScaler\n",
|
|
"scaler = StandardScaler()\n",
|
|
"scaler.fit(X_train)\n",
|
|
"X_train_scaled = scaler.transform(X_train)\n",
|
|
"X_test_scaled = scaler.transform(X_test)\n",
|
|
"\n",
|
|
"gd_clf = GradientBoostingClassifier(max_depth=3, n_estimators=100, learning_rate=1.0) \n",
|
|
"gd_clf.fit(X_train_scaled, y_train)\n",
|
|
"#Cross validation\n",
|
|
"accuracy = cross_validate(gd_clf,X_test_scaled,y_test,cv=10)['test_score']\n",
|
|
"print(accuracy)\n",
|
|
"print(\"Test set accuracy with Gradient boosting and scaled data: {:.2f}\".format(gd_clf.score(X_test_scaled,y_test)))\n",
|
|
"\n",
|
|
"import scikitplot as skplt\n",
|
|
"y_pred = gd_clf.predict(X_test_scaled)\n",
|
|
"skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)\n",
|
|
"save_fig(\"gdclassiffierconfusion\")\n",
|
|
"plt.show()\n",
|
|
"y_probas = gd_clf.predict_proba(X_test_scaled)\n",
|
|
"skplt.metrics.plot_roc(y_test, y_probas)\n",
|
|
"save_fig(\"gdclassiffierroc\")\n",
|
|
"plt.show()\n",
|
|
"skplt.metrics.plot_cumulative_gain(y_test, y_probas)\n",
|
|
"save_fig(\"gdclassiffiercgain\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "91d0cb80",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## XGBoost: Extreme Gradient Boosting\n",
|
|
"\n",
|
|
"[XGBoost](https://github.com/dmlc/xgboost) or Extreme Gradient\n",
|
|
"Boosting, is an optimized distributed gradient boosting library\n",
|
|
"designed to be highly efficient, flexible and portable. It implements\n",
|
|
"machine learning algorithms under the Gradient Boosting\n",
|
|
"framework. XGBoost provides a parallel tree boosting that solve many\n",
|
|
"data science problems in a fast and accurate way. See the [article by Chen and Guestrin](https://arxiv.org/abs/1603.02754).\n",
|
|
"\n",
|
|
"The authors design and build a highly scalable end-to-end tree\n",
|
|
"boosting system. It has a theoretically justified weighted quantile\n",
|
|
"sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning.\n",
|
|
"\n",
|
|
"It is now the algorithm which wins essentially all ML competitions!!!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f699d80a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Regression Case"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "b52540f8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import numpy as np\n",
|
|
"from sklearn.model_selection import train_test_split\n",
|
|
"import xgboost as xgb\n",
|
|
"import scikitplot as skplt\n",
|
|
"from sklearn.metrics import mean_squared_error\n",
|
|
"\n",
|
|
"n = 100\n",
|
|
"maxdegree = 6\n",
|
|
"\n",
|
|
"# Make data set.\n",
|
|
"x = np.linspace(-3, 3, n).reshape(-1, 1)\n",
|
|
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)\n",
|
|
"\n",
|
|
"error = np.zeros(maxdegree)\n",
|
|
"bias = np.zeros(maxdegree)\n",
|
|
"variance = np.zeros(maxdegree)\n",
|
|
"polydegree = np.zeros(maxdegree)\n",
|
|
"X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)\n",
|
|
"\n",
|
|
"for degree in range(maxdegree):\n",
|
|
" model = xgb.XGBRegressor(objective ='reg:squarederror', colsaobjective ='reg:squarederror', colsample_bytree = 0.3, learning_rate = 0.1,max_depth = degree, alpha = 10, n_estimators = 200)\n",
|
|
"\n",
|
|
" model.fit(X_train,y_train)\n",
|
|
" y_pred = model.predict(X_test)\n",
|
|
" polydegree[degree] = degree\n",
|
|
" error[degree] = np.mean( np.mean((y_test - y_pred)**2) )\n",
|
|
" bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )\n",
|
|
" variance[degree] = np.mean( np.var(y_pred) )\n",
|
|
" print('Max depth:', degree)\n",
|
|
" print('Error:', error[degree])\n",
|
|
" print('Bias^2:', bias[degree])\n",
|
|
" print('Var:', variance[degree])\n",
|
|
" print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))\n",
|
|
"\n",
|
|
"plt.xlim(1,maxdegree-1)\n",
|
|
"plt.plot(polydegree, error, label='Error')\n",
|
|
"plt.plot(polydegree, bias, label='bias')\n",
|
|
"plt.plot(polydegree, variance, label='Variance')\n",
|
|
"plt.legend()\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c8724acc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Xgboost on the Cancer Data\n",
|
|
"\n",
|
|
"As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "418f2c72",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import numpy as np\n",
|
|
"from sklearn.model_selection import train_test_split \n",
|
|
"from sklearn.datasets import load_breast_cancer\n",
|
|
"from sklearn.preprocessing import LabelEncoder\n",
|
|
"from sklearn.model_selection import cross_validate\n",
|
|
"import scikitplot as skplt\n",
|
|
"import xgboost as xgb\n",
|
|
"# Load the data\n",
|
|
"cancer = load_breast_cancer()\n",
|
|
"\n",
|
|
"X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)\n",
|
|
"print(X_train.shape)\n",
|
|
"print(X_test.shape)\n",
|
|
"#now scale the data\n",
|
|
"from sklearn.preprocessing import StandardScaler\n",
|
|
"scaler = StandardScaler()\n",
|
|
"scaler.fit(X_train)\n",
|
|
"X_train_scaled = scaler.transform(X_train)\n",
|
|
"X_test_scaled = scaler.transform(X_test)\n",
|
|
"\n",
|
|
"xg_clf = xgb.XGBClassifier()\n",
|
|
"xg_clf.fit(X_train_scaled,y_train)\n",
|
|
"\n",
|
|
"y_test = xg_clf.predict(X_test_scaled)\n",
|
|
"\n",
|
|
"print(\"Test set accuracy with Gradient Boosting and scaled data: {:.2f}\".format(xg_clf.score(X_test_scaled,y_test)))\n",
|
|
"\n",
|
|
"import scikitplot as skplt\n",
|
|
"y_pred = xg_clf.predict(X_test_scaled)\n",
|
|
"skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)\n",
|
|
"save_fig(\"xdclassiffierconfusion\")\n",
|
|
"plt.show()\n",
|
|
"y_probas = xg_clf.predict_proba(X_test_scaled)\n",
|
|
"skplt.metrics.plot_roc(y_test, y_probas)\n",
|
|
"save_fig(\"xdclassiffierroc\")\n",
|
|
"plt.show()\n",
|
|
"skplt.metrics.plot_cumulative_gain(y_test, y_probas)\n",
|
|
"save_fig(\"gdclassiffiercgain\")\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"\n",
|
|
"xgb.plot_tree(xg_clf,num_trees=0)\n",
|
|
"plt.rcParams['figure.figsize'] = [50, 10]\n",
|
|
"save_fig(\"xgtree\")\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"xgb.plot_importance(xg_clf)\n",
|
|
"plt.rcParams['figure.figsize'] = [5, 5]\n",
|
|
"save_fig(\"xgparams\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "583f6d4e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Support Vector Machines, overarching aims\n",
|
|
"\n",
|
|
"A Support Vector Machine (SVM) is a very powerful and versatile\n",
|
|
"Machine Learning method, capable of performing linear or nonlinear\n",
|
|
"classification, regression, and even outlier detection. It is one of\n",
|
|
"the most popular models in Machine Learning, and anyone interested in\n",
|
|
"Machine Learning should have it in their toolbox. SVMs are\n",
|
|
"particularly well suited for classification of complex but small-sized or\n",
|
|
"medium-sized datasets. \n",
|
|
"\n",
|
|
"The case with two well-separated classes only can be understood in an\n",
|
|
"intuitive way in terms of lines in a two-dimensional space separating\n",
|
|
"the two classes (see figure below).\n",
|
|
"\n",
|
|
"The basic mathematics behind the SVM is however less familiar to most of us. \n",
|
|
"It relies on the definition of hyperplanes and the\n",
|
|
"definition of a **margin** which separates classes (in case of\n",
|
|
"classification problems) of variables. It is also used for regression\n",
|
|
"problems.\n",
|
|
"\n",
|
|
"With SVMs we distinguish between hard margin and soft margins. The\n",
|
|
"latter introduces a so-called softening parameter to be discussed\n",
|
|
"below. We distinguish also between linear and non-linear\n",
|
|
"approaches. The latter are the most frequent ones since it is rather\n",
|
|
"unlikely that we can separate classes easily by say straight lines."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "53c10211",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Hyperplanes and all that\n",
|
|
"\n",
|
|
"The theory behind support vector machines (SVM hereafter) is based on\n",
|
|
"the mathematical description of so-called hyperplanes. Let us start\n",
|
|
"with a two-dimensional case. This will also allow us to introduce our\n",
|
|
"first SVM examples. These will be tailored to the case of two specific\n",
|
|
"classes, as displayed in the figure here based on the usage of the petal data.\n",
|
|
"\n",
|
|
"We assume here that our data set can be well separated into two\n",
|
|
"domains, where a straight line does the job in the separating the two\n",
|
|
"classes. Here the two classes are represented by either squares or\n",
|
|
"circles."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "7f78f3ee",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn import datasets\n",
|
|
"from sklearn.svm import SVC, LinearSVC\n",
|
|
"from sklearn.linear_model import SGDClassifier\n",
|
|
"from sklearn.preprocessing import StandardScaler\n",
|
|
"import matplotlib\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"plt.rcParams['axes.labelsize'] = 14\n",
|
|
"plt.rcParams['xtick.labelsize'] = 12\n",
|
|
"plt.rcParams['ytick.labelsize'] = 12\n",
|
|
"\n",
|
|
"\n",
|
|
"iris = datasets.load_iris()\n",
|
|
"X = iris[\"data\"][:, (2, 3)] # petal length, petal width\n",
|
|
"y = iris[\"target\"]\n",
|
|
"\n",
|
|
"setosa_or_versicolor = (y == 0) | (y == 1)\n",
|
|
"X = X[setosa_or_versicolor]\n",
|
|
"y = y[setosa_or_versicolor]\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"C = 5\n",
|
|
"alpha = 1 / (C * len(X))\n",
|
|
"\n",
|
|
"lin_clf = LinearSVC(loss=\"hinge\", C=C, random_state=42)\n",
|
|
"svm_clf = SVC(kernel=\"linear\", C=C)\n",
|
|
"sgd_clf = SGDClassifier(loss=\"hinge\", learning_rate=\"constant\", eta0=0.001, alpha=alpha,\n",
|
|
" max_iter=100000, random_state=42)\n",
|
|
"\n",
|
|
"scaler = StandardScaler()\n",
|
|
"X_scaled = scaler.fit_transform(X)\n",
|
|
"\n",
|
|
"lin_clf.fit(X_scaled, y)\n",
|
|
"svm_clf.fit(X_scaled, y)\n",
|
|
"sgd_clf.fit(X_scaled, y)\n",
|
|
"\n",
|
|
"print(\"LinearSVC: \", lin_clf.intercept_, lin_clf.coef_)\n",
|
|
"print(\"SVC: \", svm_clf.intercept_, svm_clf.coef_)\n",
|
|
"print(\"SGDClassifier(alpha={:.5f}):\".format(sgd_clf.alpha), sgd_clf.intercept_, sgd_clf.coef_)\n",
|
|
"\n",
|
|
"# Compute the slope and bias of each decision boundary\n",
|
|
"w1 = -lin_clf.coef_[0, 0]/lin_clf.coef_[0, 1]\n",
|
|
"b1 = -lin_clf.intercept_[0]/lin_clf.coef_[0, 1]\n",
|
|
"w2 = -svm_clf.coef_[0, 0]/svm_clf.coef_[0, 1]\n",
|
|
"b2 = -svm_clf.intercept_[0]/svm_clf.coef_[0, 1]\n",
|
|
"w3 = -sgd_clf.coef_[0, 0]/sgd_clf.coef_[0, 1]\n",
|
|
"b3 = -sgd_clf.intercept_[0]/sgd_clf.coef_[0, 1]\n",
|
|
"\n",
|
|
"# Transform the decision boundary lines back to the original scale\n",
|
|
"line1 = scaler.inverse_transform([[-10, -10 * w1 + b1], [10, 10 * w1 + b1]])\n",
|
|
"line2 = scaler.inverse_transform([[-10, -10 * w2 + b2], [10, 10 * w2 + b2]])\n",
|
|
"line3 = scaler.inverse_transform([[-10, -10 * w3 + b3], [10, 10 * w3 + b3]])\n",
|
|
"\n",
|
|
"# Plot all three decision boundaries\n",
|
|
"plt.figure(figsize=(11, 4))\n",
|
|
"plt.plot(line1[:, 0], line1[:, 1], \"k:\", label=\"LinearSVC\")\n",
|
|
"plt.plot(line2[:, 0], line2[:, 1], \"b--\", linewidth=2, label=\"SVC\")\n",
|
|
"plt.plot(line3[:, 0], line3[:, 1], \"r-\", label=\"SGDClassifier\")\n",
|
|
"plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"bs\") # label=\"Iris-Versicolor\"\n",
|
|
"plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"yo\") # label=\"Iris-Setosa\"\n",
|
|
"plt.xlabel(\"Petal length\", fontsize=14)\n",
|
|
"plt.ylabel(\"Petal width\", fontsize=14)\n",
|
|
"plt.legend(loc=\"upper center\", fontsize=14)\n",
|
|
"plt.axis([0, 5.5, 0, 2])\n",
|
|
"\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "da52486f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## What is a hyperplane?\n",
|
|
"\n",
|
|
"The aim of the SVM algorithm is to find a hyperplane in a\n",
|
|
"$p$-dimensional space, where $p$ is the number of features that\n",
|
|
"distinctly classifies the data points.\n",
|
|
"\n",
|
|
"In a $p$-dimensional space, a hyperplane is what we call an affine subspace of dimension of $p-1$.\n",
|
|
"As an example, in two dimension, a hyperplane is simply as straight line while in three dimensions it is \n",
|
|
"a two-dimensional subspace, or stated simply, a plane. \n",
|
|
"\n",
|
|
"In two dimensions, with the variables $x_1$ and $x_2$, the hyperplane is defined as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e5d439a9",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"b+w_1x_1+w_2x_2=0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d9da16f9",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"where $b$ is the intercept and $w_1$ and $w_2$ define the elements of a vector orthogonal to the line \n",
|
|
"$b+w_1x_1+w_2x_2=0$. \n",
|
|
"In two dimensions we define the vectors $\\boldsymbol{x} =[x1,x2]$ and $\\boldsymbol{w}=[w1,w2]$. \n",
|
|
"We can then rewrite the above equation as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "33c2fdbd",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\boldsymbol{x}^T\\boldsymbol{w}+b=0.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e033ff1d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## A $p$-dimensional space of features\n",
|
|
"\n",
|
|
"We limit ourselves to two classes of outputs $y_i$ and assign these classes the values $y_i = \\pm 1$. \n",
|
|
"In a $p$-dimensional space of say $p$ features we have a hyperplane defines as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9b531b3b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"b+wx_1+w_2x_2+\\dots +w_px_p=0.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7c591e42",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"If we define a \n",
|
|
"matrix $\\boldsymbol{X}=\\left[\\boldsymbol{x}_1,\\boldsymbol{x}_2,\\dots, \\boldsymbol{x}_p\\right]$\n",
|
|
"of dimension $n\\times p$, where $n$ represents the observations for each feature and each vector $x_i$ is a column vector of the matrix $\\boldsymbol{X}$,"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7e9a17e4",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\boldsymbol{x}_i = \\begin{bmatrix} x_{i1} \\\\ x_{i2} \\\\ \\dots \\\\ \\dots \\\\ x_{ip} \\end{bmatrix}.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c58c26c5",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"If the above condition is not met for a given vector $\\boldsymbol{x}_i$ we have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9e2066c0",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"b+w_1x_{i1}+w_2x_{i2}+\\dots +w_px_{ip} >0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3faaa864",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"if our output $y_i=1$.\n",
|
|
"In this case we say that $\\boldsymbol{x}_i$ lies on one of the sides of the hyperplane and if"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "395bee48",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"b+w_1x_{i1}+w_2x_{i2}+\\dots +w_px_{ip} < 0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "41aada85",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"for the class of observations $y_i=-1$, \n",
|
|
"then $\\boldsymbol{x}_i$ lies on the other side. \n",
|
|
"\n",
|
|
"Equivalently, for the two classes of observations we have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5ac73dbf",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"y_i\\left(b+w_1x_{i1}+w_2x_{i2}+\\dots +w_px_{ip}\\right) > 0.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "96b37700",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"When we try to separate hyperplanes, if it exists, we can use it to construct a natural classifier: a test observation is assigned a given class depending on which side of the hyperplane it is located."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "10daf062",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## The two-dimensional case\n",
|
|
"\n",
|
|
"Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional\n",
|
|
"plane. To separate the two classes of data points, there are many\n",
|
|
"possible lines (hyperplanes if you prefer a more strict naming) \n",
|
|
"that could be chosen. Our objective is to find a\n",
|
|
"plane that has the maximum margin, i.e the maximum distance between\n",
|
|
"data points of both classes. Maximizing the margin distance provides\n",
|
|
"some reinforcement so that future data points can be classified with\n",
|
|
"more confidence.\n",
|
|
"\n",
|
|
"What a linear classifier attempts to accomplish is to split the\n",
|
|
"feature space into two half spaces by placing a hyperplane between the\n",
|
|
"data points. This hyperplane will be our decision boundary. All\n",
|
|
"points on one side of the plane will belong to class one and all points\n",
|
|
"on the other side of the plane will belong to the second class two.\n",
|
|
"\n",
|
|
"Unfortunately there are many ways in which we can place a hyperplane\n",
|
|
"to divide the data. Below is an example of two candidate hyperplanes\n",
|
|
"for our data sample."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "67ec75d6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Getting into the details\n",
|
|
"\n",
|
|
"Let us define the function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cd25694b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"f(x) = \\boldsymbol{w}^T\\boldsymbol{x}+b = 0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9748a76d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"as the function that determines the line $L$ that separates two classes (our two features), see the figure here. \n",
|
|
"\n",
|
|
"Any point defined by $\\boldsymbol{x}_i$ and $\\boldsymbol{x}_2$ on the line $L$ will satisfy $\\boldsymbol{w}^T(\\boldsymbol{x}_1-\\boldsymbol{x}_2)=0$. \n",
|
|
"\n",
|
|
"The signed distance $\\delta$ from any point defined by a vector $\\boldsymbol{x}$ and a point $\\boldsymbol{x}_0$ on the line $L$ is then"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "843e4e0a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\delta = \\frac{1}{\\vert\\vert \\boldsymbol{w}\\vert\\vert}(\\boldsymbol{w}^T\\boldsymbol{x}+b).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "73f3da0f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## First attempt at a minimization approach\n",
|
|
"\n",
|
|
"How do we find the parameter $b$ and the vector $\\boldsymbol{w}$? What we could\n",
|
|
"do is to define a cost function which now contains the set of all\n",
|
|
"misclassified points $M$ and attempt to minimize this function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "babc1e77",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"C(\\boldsymbol{w},b) = -\\sum_{i\\in M} y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "623e621b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We could now for example define all values $y_i =1$ as misclassified in case we have $\\boldsymbol{w}^T\\boldsymbol{x}_i+b < 0$ and the opposite if we have $y_i=-1$. Taking the derivatives gives us"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8e046a42",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial C}{\\partial b} = -\\sum_{i\\in M} y_i,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d5679c1e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"and"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6d2851cc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial C}{\\partial \\boldsymbol{w}} = -\\sum_{i\\in M} y_ix_i.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "95c2b99e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Solving the equations\n",
|
|
"\n",
|
|
"We can now use the Newton-Raphson method or different variants of the gradient descent family (from plain gradient descent to various stochastic gradient descent approaches) to solve the equations"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "015ae6c3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"b \\leftarrow b +\\eta \\frac{\\partial C}{\\partial b},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "22e98f1d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"and"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6655140e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\boldsymbol{w} \\leftarrow \\boldsymbol{w} +\\eta \\frac{\\partial C}{\\partial \\boldsymbol{w}},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7b117af1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"where $\\eta$ is our by now well-known learning rate."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fd9de362",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Code Example\n",
|
|
"\n",
|
|
"The equations we discussed above can be coded rather easily (the\n",
|
|
"framework is similar to what we developed for logistic\n",
|
|
"regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "58765651",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Problems with the Simpler Approach\n",
|
|
"\n",
|
|
"There are however problems with this approach, although it looks\n",
|
|
"pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes.\n",
|
|
"\n",
|
|
"For small\n",
|
|
"gaps between the entries, we may also end up needing many iterations\n",
|
|
"before the solutions converge and if the data cannot be separated\n",
|
|
"properly into two distinct classes, we may not experience a converge\n",
|
|
"at all."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b09f0526",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## A better approach\n",
|
|
"\n",
|
|
"A better approach is rather to try to define a large margin between\n",
|
|
"the two classes (if they are well separated from the beginning).\n",
|
|
"\n",
|
|
"Thus, we wish to find a margin $M$ with $\\boldsymbol{w}$ normalized to\n",
|
|
"$\\vert\\vert \\boldsymbol{w}\\vert\\vert =1$ subject to the condition"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "de95005e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, p.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "794e96fa",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"All points are thus at a signed distance from the decision boundary defined by the line $L$. The parameters $b$ and $w_1$ and $w_2$ define this line. \n",
|
|
"\n",
|
|
"We seek thus the largest value $M$ defined by"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "12dec4af",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{1}{\\vert \\vert \\boldsymbol{w}\\vert\\vert}y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M \\hspace{0.1cm}\\forall i=1,2,\\dots, n,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2762e903",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"or just"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "511d64f7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq M\\vert \\vert \\boldsymbol{w}\\vert\\vert \\hspace{0.1cm}\\forall i.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "90c5eca5",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"If we scale the equation so that $\\vert \\vert \\boldsymbol{w}\\vert\\vert = 1/M$, we have to find the minimum of \n",
|
|
"$\\boldsymbol{w}^T\\boldsymbol{w}=\\vert \\vert \\boldsymbol{w}\\vert\\vert$ (the norm) subject to the condition"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f900218d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) \\geq 1 \\hspace{0.1cm}\\forall i.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1786094b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We have thus defined our margin as the invers of the norm of\n",
|
|
"$\\boldsymbol{w}$. We want to minimize the norm in order to have a as large as\n",
|
|
"possible margin $M$. Before we proceed, we need to remind ourselves\n",
|
|
"about Lagrangian multipliers."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "69a4e00b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## A quick Reminder on Lagrangian Multipliers\n",
|
|
"\n",
|
|
"Consider a function of three independent variables $f(x,y,z)$ . For the function $f$ to be an\n",
|
|
"extreme we have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cb9e90fd",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"df=0.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1fc347c8",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"A necessary and sufficient condition is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8cac3603",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial f}{\\partial x} =\\frac{\\partial f}{\\partial y}=\\frac{\\partial f}{\\partial z}=0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "487d6f5c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"due to"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "95cdfc00",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"df = \\frac{\\partial f}{\\partial x}dx+\\frac{\\partial f}{\\partial y}dy+\\frac{\\partial f}{\\partial z}dz.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1d74eb3a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"In many problems the variables $x,y,z$ are often subject to constraints (such as those above for the margin)\n",
|
|
"so that they are no longer all independent. It is possible at least in principle to use each \n",
|
|
"constraint to eliminate one variable\n",
|
|
"and to proceed with a new and smaller set of independent varables.\n",
|
|
"\n",
|
|
"The use of so-called Lagrangian multipliers is an alternative technique when the elimination\n",
|
|
"of variables is incovenient or undesirable. Assume that we have an equation of constraint on \n",
|
|
"the variables $x,y,z$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9f659d07",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\phi(x,y,z) = 0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cf8d75cc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"resulting in"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a40803bd",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"d\\phi = \\frac{\\partial \\phi}{\\partial x}dx+\\frac{\\partial \\phi}{\\partial y}dy+\\frac{\\partial \\phi}{\\partial z}dz =0.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3bc45792",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we cannot set anymore"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4102959b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial f}{\\partial x} =\\frac{\\partial f}{\\partial y}=\\frac{\\partial f}{\\partial z}=0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5a568b07",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"if $df=0$ is wanted\n",
|
|
"because there are now only two independent variables! Assume $x$ and $y$ are the independent \n",
|
|
"variables.\n",
|
|
"Then $dz$ is no longer arbitrary."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a09a9765",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Adding the Multiplier\n",
|
|
"\n",
|
|
"However, we can add to"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "04593213",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"df = \\frac{\\partial f}{\\partial x}dx+\\frac{\\partial f}{\\partial y}dy+\\frac{\\partial f}{\\partial z}dz,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ef2a5532",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"a multiplum of $d\\phi$, viz. $\\lambda d\\phi$, resulting in"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ff77b6b3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"df+\\lambda d\\phi = (\\frac{\\partial f}{\\partial z}+\\lambda\n",
|
|
"\\frac{\\partial \\phi}{\\partial x})dx+(\\frac{\\partial f}{\\partial y}+\\lambda\\frac{\\partial \\phi}{\\partial y})dy+\n",
|
|
"(\\frac{\\partial f}{\\partial z}+\\lambda\\frac{\\partial \\phi}{\\partial z})dz =0.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0dc32396",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Our multiplier is chosen so that"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c474965a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial f}{\\partial z}+\\lambda\\frac{\\partial \\phi}{\\partial z} =0.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1a51f6e0",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We need to remember that we took $dx$ and $dy$ to be arbitrary and thus we must have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bba4ef61",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial f}{\\partial x}+\\lambda\\frac{\\partial \\phi}{\\partial x} =0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "983ba7f7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"and"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1e947b54",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial f}{\\partial y}+\\lambda\\frac{\\partial \\phi}{\\partial y} =0.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9107b993",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"When all these equations are satisfied, $df=0$. We have four unknowns, $x,y,z$ and\n",
|
|
"$\\lambda$. Actually we want only $x,y,z$, $\\lambda$ needs not to be determined, \n",
|
|
"it is therefore often called\n",
|
|
"Lagrange's undetermined multiplier.\n",
|
|
"If we have a set of constraints $\\phi_k$ we have the equations"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e809390c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial f}{\\partial x_i}+\\sum_k\\lambda_k\\frac{\\partial \\phi_k}{\\partial x_i} =0.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e24bc5db",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Setting up the Problem\n",
|
|
"In order to solve the above problem, we define the following Lagrangian function to be minimized"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3346e60b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"{\\cal L}(\\lambda,b,\\boldsymbol{w})=\\frac{1}{2}\\boldsymbol{w}^T\\boldsymbol{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)-1\\right],\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "dc6ef22c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"where $\\lambda_i$ is a so-called Lagrange multiplier subject to the condition $\\lambda_i \\geq 0$.\n",
|
|
"\n",
|
|
"Taking the derivatives with respect to $b$ and $\\boldsymbol{w}$ we obtain"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "491f34e3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial {\\cal L}}{\\partial b} = -\\sum_{i} \\lambda_iy_i=0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "923e79ef",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"and"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "dea0dba6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial {\\cal L}}{\\partial \\boldsymbol{w}} = 0 = \\boldsymbol{w}-\\sum_{i} \\lambda_iy_i\\boldsymbol{x}_i.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3931cdc1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Inserting these constraints into the equation for ${\\cal L}$ we obtain"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a14fc496",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{x}_j,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "286f16a3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"subject to the constraints $\\lambda_i\\geq 0$ and $\\sum_i\\lambda_iy_i=0$. \n",
|
|
"We must in addition satisfy the [Karush-Kuhn-Tucker](https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions) (KKT) condition"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1aed6b37",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) -1\\right] \\hspace{0.1cm}\\forall i.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "23ef61de",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"1. If $\\lambda_i > 0$, then $y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1$ and we say that $x_i$ is on the boundary.\n",
|
|
"\n",
|
|
"2. If $y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)> 1$, we say $x_i$ is not on the boundary and we set $\\lambda_i=0$. \n",
|
|
"\n",
|
|
"When $\\lambda_i > 0$, the vectors $\\boldsymbol{x}_i$ are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin $M$."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b29e56f4",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## The problem to solve\n",
|
|
"\n",
|
|
"We can rewrite"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a228b6b0",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{x}_j,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bc00cb70",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"and its constraints in terms of a matrix-vector problem where we minimize w.r.t. $\\lambda$ the following problem"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e4bfe014",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{1}{2} \\boldsymbol{\\lambda}^T\\begin{bmatrix} y_1y_1\\boldsymbol{x}_1^T\\boldsymbol{x}_1 & y_1y_2\\boldsymbol{x}_1^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_1^T\\boldsymbol{x}_n \\\\\n",
|
|
"y_2y_1\\boldsymbol{x}_2^T\\boldsymbol{x}_1 & y_2y_2\\boldsymbol{x}_2^T\\boldsymbol{x}_2 & \\dots & \\dots & y_1y_n\\boldsymbol{x}_2^T\\boldsymbol{x}_n \\\\\n",
|
|
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
|
|
"\\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
|
|
"y_ny_1\\boldsymbol{x}_n^T\\boldsymbol{x}_1 & y_ny_2\\boldsymbol{x}_n^T\\boldsymbol{x}_2 & \\dots & \\dots & y_ny_n\\boldsymbol{x}_n^T\\boldsymbol{x}_n \\\\\n",
|
|
"\\end{bmatrix}\\boldsymbol{\\lambda}-\\mathbb{1}\\boldsymbol{\\lambda},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7541e0ca",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"subject to $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$. Here we defined the vectors $\\boldsymbol{\\lambda} =[\\lambda_1,\\lambda_2,\\dots,\\lambda_n]$ and \n",
|
|
"$\\boldsymbol{y}=[y_1,y_2,\\dots,y_n]$."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "99174d29",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## The last steps\n",
|
|
"\n",
|
|
"Solving the above problem, yields the values of $\\lambda_i$.\n",
|
|
"To find the coefficients of your hyperplane we need simply to compute"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "29032445",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\boldsymbol{w}=\\sum_{i} \\lambda_iy_i\\boldsymbol{x}_i.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c952a920",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"With our vector $\\boldsymbol{w}$ we can in turn find the value of the intercept $b$ (here in two dimensions) via"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e0ebf4c1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9288a562",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"resulting in"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6ffad746",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"b = \\frac{1}{y_i}-\\boldsymbol{w}^T\\boldsymbol{x}_i,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ffdb1d0f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"or if we write it out in terms of the support vectors only, with $N_s$ being their number, we have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d5738333",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"b = \\frac{1}{N_s}\\sum_{j\\in N_s}\\left(y_j-\\sum_{i=1}^n\\lambda_iy_i\\boldsymbol{x}_i^T\\boldsymbol{x}_j\\right).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b198d30b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"With our hyperplane coefficients we can use our classifier to assign any observation by simply using"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "19174eda",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"y_i = \\mathrm{sign}(\\boldsymbol{w}^T\\boldsymbol{x}_i+b).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d00c15c8",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Below we discuss how to find the optimal values of $\\lambda_i$. Before we proceed however, we discuss now the so-called soft classifier."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fc7c4f64",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## A soft classifier\n",
|
|
"\n",
|
|
"Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.\n",
|
|
"\n",
|
|
"Suppose now that classes overlap in feature space, as shown in the\n",
|
|
"figure here. One way to deal with this problem before we define the\n",
|
|
"so-called **kernel approach**, is to allow a kind of slack in the sense\n",
|
|
"that we allow some points to be on the wrong side of the margin.\n",
|
|
"\n",
|
|
"We introduce thus the so-called **slack** variables $\\boldsymbol{\\xi} =[\\xi_1,x_2,\\dots,x_n]$ and \n",
|
|
"modify our previous equation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8b7dd152",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "18859e16",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"to"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ec440a10",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1-\\xi_i,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ff372247",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"with the requirement $\\xi_i\\geq 0$. The total violation is now $\\sum_i\\xi$. \n",
|
|
"The value $\\xi_i$ in the constraint the last constraint corresponds to the amount by which the prediction\n",
|
|
"$y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1$ is on the wrong side of its margin. Hence by bounding the sum $\\sum_i \\xi_i$,\n",
|
|
"we bound the total amount by which predictions fall on the wrong side of their margins.\n",
|
|
"\n",
|
|
"Misclassifications occur when $\\xi_i > 1$. Thus bounding the total sum by some value $C$ bounds in turn the total number of\n",
|
|
"misclassifications."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a35c1138",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Soft optmization problem\n",
|
|
"\n",
|
|
"This has in turn the consequences that we change our optmization problem to finding the minimum of"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c3b4fde1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"{\\cal L}=\\frac{1}{2}\\boldsymbol{w}^T\\boldsymbol{w}-\\sum_{i=1}^n\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)-(1-\\xi_)\\right]+C\\sum_{i=1}^n\\xi_i-\\sum_{i=1}^n\\gamma_i\\xi_i,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c90fe182",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"subject to"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "312adb24",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b)=1-\\xi_i \\hspace{0.1cm}\\forall i,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9346e584",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"with the requirement $\\xi_i\\geq 0$.\n",
|
|
"\n",
|
|
"Taking the derivatives with respect to $b$ and $\\boldsymbol{w}$ we obtain"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "94fc1b4e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial {\\cal L}}{\\partial b} = -\\sum_{i} \\lambda_iy_i=0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b098af10",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"and"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "226974b7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial {\\cal L}}{\\partial \\boldsymbol{w}} = 0 = \\boldsymbol{w}-\\sum_{i} \\lambda_iy_i\\boldsymbol{x}_i,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6f76bc36",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"and"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1ca8ada8",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\lambda_i = C-\\gamma_i \\hspace{0.1cm}\\forall i.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6ac4cc1a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Inserting these constraints into the equation for ${\\cal L}$ we obtain the same equation as before"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "81f53bc7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"{\\cal L}=\\sum_i\\lambda_i-\\frac{1}{2}\\sum_{ij}^n\\lambda_i\\lambda_jy_iy_j\\boldsymbol{x}_i^T\\boldsymbol{x}_j,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "46e66214",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"but now subject to the constraints $\\lambda_i\\geq 0$, $\\sum_i\\lambda_iy_i=0$ and $0\\leq\\lambda_i \\leq C$. \n",
|
|
"We must in addition satisfy the Karush-Kuhn-Tucker condition which now reads"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5749e12c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\lambda_i\\left[y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) -(1-\\xi_)\\right]=0 \\hspace{0.1cm}\\forall i,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "92ea69ff",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\gamma_i\\xi_i = 0,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "914ef9af",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"and"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f2bbc774",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"y_i(\\boldsymbol{w}^T\\boldsymbol{x}_i+b) -(1-\\xi_) \\geq 0 \\hspace{0.1cm}\\forall i.\n",
|
|
"$$"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|