mhjensen
814 lines
27 KiB
Plaintext
814 lines
27 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- dom:TITLE: Data Analysis and Machine Learning: Logistic Regression -->\n",
|
|
"# Data Analysis and Machine Learning: Logistic Regression\n",
|
|
"<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
|
|
"<!-- Author: --> \n",
|
|
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
|
|
"\n",
|
|
"Date: **Oct 17, 2019**\n",
|
|
"\n",
|
|
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"<!-- !split -->\n",
|
|
"## Logistic Regression\n",
|
|
"\n",
|
|
"In linear regression our main interest was centered on learning the\n",
|
|
"coefficients of a functional fit (say a polynomial) in order to be\n",
|
|
"able to predict the response of a continuous variable on some unseen\n",
|
|
"data. The fit to the continuous variable $y_i$ is based on some\n",
|
|
"independent variables $\\hat{x}_i$. Linear regression resulted in\n",
|
|
"analytical expressions for standard ordinary Least Squares or Ridge\n",
|
|
"regression (in terms of matrices to invert) for several quantities,\n",
|
|
"ranging from the variance and thereby the confidence intervals of the\n",
|
|
"parameters $\\hat{\\beta}$ to the mean squared error. If we can invert\n",
|
|
"the product of the design matrices, linear regression gives then a\n",
|
|
"simple recipe for fitting our data.\n",
|
|
"\n",
|
|
"\n",
|
|
"Classification problems, however, are concerned with outcomes taking\n",
|
|
"the form of discrete variables (i.e. categories). We may for example,\n",
|
|
"on the basis of DNA sequencing for a number of patients, like to find\n",
|
|
"out which mutations are important for a certain disease; or based on\n",
|
|
"scans of various patients' brains, figure out if there is a tumor or\n",
|
|
"not; or given a specific physical system, we'd like to identify its\n",
|
|
"state, say whether it is an ordered or disordered system (typical\n",
|
|
"situation in solid state physics); or classify the status of a\n",
|
|
"patient, whether she/he has a stroke or not and many other similar\n",
|
|
"situations.\n",
|
|
"\n",
|
|
"The most common situation we encounter when we apply logistic\n",
|
|
"regression is that of two possible outcomes, normally denoted as a\n",
|
|
"binary outcome, true or false, positive or negative, success or\n",
|
|
"failure etc.\n",
|
|
"\n",
|
|
"## Optimization and Deep learning\n",
|
|
"\n",
|
|
"Logistic regression will also serve as our stepping stone towards\n",
|
|
"neural network algorithms and supervised deep learning. For logistic\n",
|
|
"learning, the minimization of the cost function leads to a non-linear\n",
|
|
"equation in the parameters $\\hat{\\beta}$. The optimization of the\n",
|
|
"problem calls therefore for minimization algorithms. This forms the\n",
|
|
"bottle neck of all machine learning algorithms, namely how to find\n",
|
|
"reliable minima of a multi-variable function. This leads us to the\n",
|
|
"family of gradient descent methods. The latter are the working horses\n",
|
|
"of basically all modern machine learning algorithms.\n",
|
|
"\n",
|
|
"We note also that many of the topics discussed here on logistic \n",
|
|
"regression are also commonly used in modern supervised Deep Learning\n",
|
|
"models, as we will see later.\n",
|
|
"\n",
|
|
"\n",
|
|
"<!-- !split -->\n",
|
|
"## Basics\n",
|
|
"\n",
|
|
"We consider the case where the dependent variables, also called the\n",
|
|
"responses or the outcomes, $y_i$ are discrete and only take values\n",
|
|
"from $k=0,\\dots,K-1$ (i.e. $K$ classes).\n",
|
|
"\n",
|
|
"The goal is to predict the\n",
|
|
"output classes from the design matrix $\\hat{X}\\in\\mathbb{R}^{n\\times p}$\n",
|
|
"made of $n$ samples, each of which carries $p$ features or predictors. The\n",
|
|
"primary goal is to identify the classes to which new unseen samples\n",
|
|
"belong.\n",
|
|
"\n",
|
|
"Let us specialize to the case of two classes only, with outputs\n",
|
|
"$y_i=0$ and $y_i=1$. Our outcomes could represent the status of a\n",
|
|
"credit card user that could default or not on her/his credit card\n",
|
|
"debt. That is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"y_i = \\begin{bmatrix} 0 & \\mathrm{no}\\\\ 1 & \\mathrm{yes} \\end{bmatrix}.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Linear classifier\n",
|
|
"\n",
|
|
"Before moving to the logistic model, let us try to use our linear\n",
|
|
"regression model to classify these two outcomes. We could for example\n",
|
|
"fit a linear model to the default case if $y_i > 0.5$ and the no\n",
|
|
"default case $y_i \\leq 0.5$.\n",
|
|
"\n",
|
|
"We would then have our \n",
|
|
"weighted linear combination, namely"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto1\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
"\\hat{y} = \\hat{X}^T\\hat{\\beta} + \\hat{\\epsilon},\n",
|
|
"\\label{_auto1} \\tag{1}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"where $\\hat{y}$ is a vector representing the possible outcomes, $\\hat{X}$ is our\n",
|
|
"$n\\times p$ design matrix and $\\hat{\\beta}$ represents our estimators/predictors.\n",
|
|
"\n",
|
|
"## Some selected properties\n",
|
|
"\n",
|
|
"The main problem with our function is that it takes values on the\n",
|
|
"entire real axis. In the case of logistic regression, however, the\n",
|
|
"labels $y_i$ are discrete variables. A typical example is the credit\n",
|
|
"card data discussed below here, where we can set the state of\n",
|
|
"defaulting the debt to $y_i=1$ and not to $y_i=0$ for one the persons\n",
|
|
"in the data set (see the full example below).\n",
|
|
"\n",
|
|
"One simple way to get a discrete output is to have sign\n",
|
|
"functions that map the output of a linear regressor to values $\\{0,1\\}$,\n",
|
|
"$f(s_i)=sign(s_i)=1$ if $s_i\\ge 0$ and 0 if otherwise. \n",
|
|
"We will encounter this model in our first demonstration of neural networks. Historically it is called the \"perceptron\" model in the machine learning\n",
|
|
"literature. This model is extremely simple. However, in many cases it is more\n",
|
|
"favorable to use a ``soft\" classifier that outputs\n",
|
|
"the probability of a given category. This leads us to the logistic function.\n",
|
|
"\n",
|
|
"\n",
|
|
"## The logistic function\n",
|
|
"\n",
|
|
"The perceptron is an example of a ``hard classification\" model. We\n",
|
|
"will encounter this model when we discuss neural networks as\n",
|
|
"well. Each datapoint is deterministically assigned to a category (i.e\n",
|
|
"$y_i=0$ or $y_i=1$). In many cases, it is favorable to have a \"soft\"\n",
|
|
"classifier that outputs the probability of a given category rather\n",
|
|
"than a single value. For example, given $x_i$, the classifier\n",
|
|
"outputs the probability of being in a category $k$. Logistic regression\n",
|
|
"is the most common example of a so-called soft classifier. In logistic\n",
|
|
"regression, the probability that a data point $x_i$\n",
|
|
"belongs to a category $y_i=\\{0,1\\}$ is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"p(t) = \\frac{1}{1+\\mathrm \\exp{-t}}=\\frac{\\exp{t}}{1+\\mathrm \\exp{t}}.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Note that $1-p(t)= p(-t)$.\n",
|
|
"\n",
|
|
"## Examples of likelihood functions used in logistic regression and nueral networks\n",
|
|
"\n",
|
|
"\n",
|
|
"The following code plots the logistic function, the step function and other functions we will encounter from here and on."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"\"\"\"The sigmoid function (or the logistic curve) is a\n",
|
|
"function that takes any real number, z, and outputs a number (0,1).\n",
|
|
"It is useful in neural networks for assigning weights on a relative scale.\n",
|
|
"The value z is the weighted sum of parameters involved in the learning algorithm.\"\"\"\n",
|
|
"\n",
|
|
"import numpy\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import math as mt\n",
|
|
"\n",
|
|
"z = numpy.arange(-5, 5, .1)\n",
|
|
"sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))\n",
|
|
"sigma = sigma_fn(z)\n",
|
|
"\n",
|
|
"fig = plt.figure()\n",
|
|
"ax = fig.add_subplot(111)\n",
|
|
"ax.plot(z, sigma)\n",
|
|
"ax.set_ylim([-0.1, 1.1])\n",
|
|
"ax.set_xlim([-5,5])\n",
|
|
"ax.grid(True)\n",
|
|
"ax.set_xlabel('z')\n",
|
|
"ax.set_title('sigmoid function')\n",
|
|
"\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"\"\"\"Step Function\"\"\"\n",
|
|
"z = numpy.arange(-5, 5, .02)\n",
|
|
"step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)\n",
|
|
"step = step_fn(z)\n",
|
|
"\n",
|
|
"fig = plt.figure()\n",
|
|
"ax = fig.add_subplot(111)\n",
|
|
"ax.plot(z, step)\n",
|
|
"ax.set_ylim([-0.5, 1.5])\n",
|
|
"ax.set_xlim([-5,5])\n",
|
|
"ax.grid(True)\n",
|
|
"ax.set_xlabel('z')\n",
|
|
"ax.set_title('step function')\n",
|
|
"\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"\"\"\"tanh Function\"\"\"\n",
|
|
"z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)\n",
|
|
"t = numpy.tanh(z)\n",
|
|
"\n",
|
|
"fig = plt.figure()\n",
|
|
"ax = fig.add_subplot(111)\n",
|
|
"ax.plot(z, t)\n",
|
|
"ax.set_ylim([-1.0, 1.0])\n",
|
|
"ax.set_xlim([-2*mt.pi,2*mt.pi])\n",
|
|
"ax.grid(True)\n",
|
|
"ax.set_xlabel('z')\n",
|
|
"ax.set_title('tanh function')\n",
|
|
"\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Two parameters\n",
|
|
"\n",
|
|
"We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\\beta$ in our fitting of the Sigmoid function, that is we define probabilities"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\begin{align*}\n",
|
|
"p(y_i=1|x_i,\\hat{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n",
|
|
"p(y_i=0|x_i,\\hat{\\beta}) &= 1 - p(y_i=1|x_i,\\hat{\\beta}),\n",
|
|
"\\end{align*}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"where $\\hat{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
|
|
"\n",
|
|
"Note that we used"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"p(y_i=0\\vert x_i, \\hat{\\beta}) = 1-p(y_i=1\\vert x_i, \\hat{\\beta}).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- !split -->\n",
|
|
"## Maximum likelihood\n",
|
|
"\n",
|
|
"In order to define the total likelihood for all possible outcomes from a \n",
|
|
"dataset $\\mathcal{D}=\\{(y_i,x_i)\\}$, with the binary labels\n",
|
|
"$y_i\\in\\{0,1\\}$ and where the data points are drawn independently, we use the so-called [Maximum Likelihood Estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation) (MLE) principle. \n",
|
|
"We aim thus at maximizing \n",
|
|
"the probability of seeing the observed data. We can then approximate the \n",
|
|
"likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\begin{align*}\n",
|
|
"P(\\mathcal{D}|\\hat{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\hat{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n",
|
|
"\\end{align*}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"from which we obtain the log-likelihood and our **cost/loss** function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\hat{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]\\right).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## The cost function rewritten\n",
|
|
"\n",
|
|
"Reordering the logarithms, we can rewrite the **cost/loss** function as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n",
|
|
"Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\mathcal{C}(\\hat{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n",
|
|
"in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.\n",
|
|
"\n",
|
|
"## Minimizing the cross entropy\n",
|
|
"\n",
|
|
"The cross entropy is a convex function of the weights $\\hat{\\beta}$ and,\n",
|
|
"therefore, any local minimizer is a global minimizer. \n",
|
|
"\n",
|
|
"\n",
|
|
"Minimizing this\n",
|
|
"cost function with respect to the two parameters $\\beta_0$ and $\\beta_1$ we obtain"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"and"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## A more compact expression\n",
|
|
"\n",
|
|
"Let us now define a vector $\\hat{y}$ with $n$ elements $y_i$, an\n",
|
|
"$n\\times p$ matrix $\\hat{X}$ which contains the $x_i$ values and a\n",
|
|
"vector $\\hat{p}$ of fitted probabilities $p(y_i\\vert x_i,\\hat{\\beta})$. We can rewrite in a more compact form the first\n",
|
|
"derivative of cost function as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}} = -\\hat{X}^T\\left(\\hat{y}-\\hat{p}\\right).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"If we in addition define a diagonal matrix $\\hat{W}$ with elements \n",
|
|
"$p(y_i\\vert x_i,\\hat{\\beta})(1-p(y_i\\vert x_i,\\hat{\\beta})$, we can obtain a compact expression of the second derivative as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\frac{\\partial^2 \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}\\partial \\hat{\\beta}^T} = \\hat{X}^T\\hat{W}\\hat{X}.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Extending to more predictors\n",
|
|
"\n",
|
|
"Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\log{ \\frac{p(\\hat{\\beta}\\hat{x})}{1-p(\\hat{\\beta}\\hat{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here we defined $\\hat{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\hat{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"p(\\hat{\\beta}\\hat{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Including more classes\n",
|
|
"\n",
|
|
"Till now we have mainly focused on two classes, the so-called binary\n",
|
|
"system. Suppose we wish to extend to $K$ classes. Let us for the sake\n",
|
|
"of simplicity assume we have only two predictors. We have then\n",
|
|
"following model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"1\n",
|
|
"5\n",
|
|
" \n",
|
|
"<\n",
|
|
"<\n",
|
|
"<\n",
|
|
"!\n",
|
|
"!\n",
|
|
"M\n",
|
|
"A\n",
|
|
"T\n",
|
|
"H\n",
|
|
"_\n",
|
|
"B\n",
|
|
"L\n",
|
|
"O\n",
|
|
"C\n",
|
|
"K"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"and so on till the class $C=K-1$ class"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"and the model is specified in term of $K-1$ so-called log-odds or\n",
|
|
"**logit** transformations.\n",
|
|
"\n",
|
|
"\n",
|
|
"## More classes\n",
|
|
"\n",
|
|
"In our discussion of neural networks we will encounter the above again\n",
|
|
"in terms of a slightly modified function, the so-called **Softmax** function.\n",
|
|
"\n",
|
|
"The softmax function is used in various multiclass classification\n",
|
|
"methods, such as multinomial logistic regression (also known as\n",
|
|
"softmax regression), multiclass linear discriminant analysis, naive\n",
|
|
"Bayes classifiers, and artificial neural networks. Specifically, in\n",
|
|
"multinomial logistic regression and linear discriminant analysis, the\n",
|
|
"input to the function is the result of $K$ distinct linear functions,\n",
|
|
"and the predicted probability for the $k$-th class given a sample\n",
|
|
"vector $\\hat{x}$ and a weighting vector $\\hat{\\beta}$ is (with two\n",
|
|
"predictors):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"It is easy to extend to more predictors. The final class is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"and they sum to one. Our earlier discussions were all specialized to\n",
|
|
"the case with two classes only. It is easy to see from the above that\n",
|
|
"what we derived earlier is compatible with these equations.\n",
|
|
"\n",
|
|
"To find the optimal parameters we would typically use a gradient\n",
|
|
"descent method. Newton's method and gradient descent methods are\n",
|
|
"discussed in the material on [optimization\n",
|
|
"methods](https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html).\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"## A simple classification problem"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import numpy as np\n",
|
|
"from sklearn import datasets, linear_model\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"\n",
|
|
"def generate_data():\n",
|
|
" np.random.seed(0)\n",
|
|
" X, y = datasets.make_moons(200, noise=0.20)\n",
|
|
" return X, y\n",
|
|
"\n",
|
|
"\n",
|
|
"def visualize(X, y, clf):\n",
|
|
" plot_decision_boundary(lambda x: clf.predict(x), X, y)\n",
|
|
"\n",
|
|
"def plot_decision_boundary(pred_func, X, y):\n",
|
|
" # Set min and max values and give it some padding\n",
|
|
" x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5\n",
|
|
" y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5\n",
|
|
" h = 0.01\n",
|
|
" # Generate a grid of points with distance h between them\n",
|
|
" xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
|
|
" # Predict the function value for the whole gid\n",
|
|
" Z = pred_func(np.c_[xx.ravel(), yy.ravel()])\n",
|
|
" Z = Z.reshape(xx.shape)\n",
|
|
" # Plot the contour and training examples\n",
|
|
" plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)\n",
|
|
" plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)\n",
|
|
" plt.show()\n",
|
|
"\n",
|
|
"\n",
|
|
"def classify(X, y):\n",
|
|
" clf = linear_model.LogisticRegressionCV()\n",
|
|
" clf.fit(X, y)\n",
|
|
" return clf\n",
|
|
"\n",
|
|
"\n",
|
|
"def main():\n",
|
|
" X, y = generate_data()\n",
|
|
" # visualize(X, y)\n",
|
|
" clf = classify(X, y)\n",
|
|
" visualize(X, y, clf)\n",
|
|
"\n",
|
|
"if __name__ == \"__main__\":\n",
|
|
" main()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## The Credit Card example\n",
|
|
"Here we use the the [credit card data](https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients). \n",
|
|
"The data are from an extensive database from Taiwan and include more than ten predictors.\n",
|
|
"\n",
|
|
"For categorical data -Scikit-Learn- provides a so-called **one-hot encoder**.\n",
|
|
"This is called one-hot\n",
|
|
"encoding, because only one attribute will be equal to 1 (hot), while the others will be 0 (cold).\n",
|
|
"**Scikit-Learn** provides a OneHotEncoder encoder to convert integer categorical values into one-hot"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn.preprocessing import OneHotEncoder\n",
|
|
"encoder = OneHotEncoder()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## How to read the Credit Card data"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import pandas as pd\n",
|
|
"import os\n",
|
|
"import numpy as np\n",
|
|
"\n",
|
|
"\n",
|
|
"from sklearn.model_selection import train_test_split\n",
|
|
"from sklearn.preprocessing import OneHotEncoder\n",
|
|
"from sklearn.compose import ColumnTransformer\n",
|
|
"from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
|
|
"from sklearn.metrics import confusion_matrix, accuracy_score, roc_auc_score\n",
|
|
"\n",
|
|
"# Trying to set the seed\n",
|
|
"np.random.seed(0)\n",
|
|
"import random\n",
|
|
"random.seed(0)\n",
|
|
"\n",
|
|
"# Reading file into data frame\n",
|
|
"cwd = os.getcwd()\n",
|
|
"filename = cwd + '/default of credit card clients.xls'\n",
|
|
"nanDict = {}\n",
|
|
"df = pd.read_excel(filename, header=1, skiprows=0, index_col=0, na_values=nanDict)\n",
|
|
"\n",
|
|
"df.rename(index=str, columns={\"default payment next month\": \"defaultPaymentNextMonth\"}, inplace=True)\n",
|
|
"\n",
|
|
"# Features and targets \n",
|
|
"X = df.loc[:, df.columns != 'defaultPaymentNextMonth'].values\n",
|
|
"y = df.loc[:, df.columns == 'defaultPaymentNextMonth'].values\n",
|
|
"\n",
|
|
"# Categorical variables to one-hot's\n",
|
|
"onehotencoder = OneHotEncoder(categories=\"auto\")\n",
|
|
"\n",
|
|
"X = ColumnTransformer(\n",
|
|
" [(\"\", onehotencoder, [3]),],\n",
|
|
" remainder=\"passthrough\"\n",
|
|
").fit_transform(X)\n",
|
|
"\n",
|
|
"y.shape\n",
|
|
"\n",
|
|
"# Train-test split\n",
|
|
"trainingShare = 0.5 \n",
|
|
"seed = 1\n",
|
|
"XTrain, XTest, yTrain, yTest=train_test_split(X, y, train_size=trainingShare, \\\n",
|
|
" test_size = 1-trainingShare,\n",
|
|
" random_state=seed)\n",
|
|
"\n",
|
|
"# Input Scaling\n",
|
|
"sc = StandardScaler()\n",
|
|
"XTrain = sc.fit_transform(XTrain)\n",
|
|
"XTest = sc.transform(XTest)\n",
|
|
"\n",
|
|
"# One-hot's of the target vector\n",
|
|
"Y_train_onehot, Y_test_onehot = onehotencoder.fit_transform(yTrain), onehotencoder.fit_transform(yTest)\n",
|
|
"\n",
|
|
"# Remove instances with zeros only for past bill statements or paid amounts\n",
|
|
"'''\n",
|
|
"df = df.drop(df[(df.BILL_AMT1 == 0) &\n",
|
|
" (df.BILL_AMT2 == 0) &\n",
|
|
" (df.BILL_AMT3 == 0) &\n",
|
|
" (df.BILL_AMT4 == 0) &\n",
|
|
" (df.BILL_AMT5 == 0) &\n",
|
|
" (df.BILL_AMT6 == 0) &\n",
|
|
" (df.PAY_AMT1 == 0) &\n",
|
|
" (df.PAY_AMT2 == 0) &\n",
|
|
" (df.PAY_AMT3 == 0) &\n",
|
|
" (df.PAY_AMT4 == 0) &\n",
|
|
" (df.PAY_AMT5 == 0) &\n",
|
|
" (df.PAY_AMT6 == 0)].index)\n",
|
|
"'''\n",
|
|
"df = df.drop(df[(df.BILL_AMT1 == 0) &\n",
|
|
" (df.BILL_AMT2 == 0) &\n",
|
|
" (df.BILL_AMT3 == 0) &\n",
|
|
" (df.BILL_AMT4 == 0) &\n",
|
|
" (df.BILL_AMT5 == 0) &\n",
|
|
" (df.BILL_AMT6 == 0)].index)\n",
|
|
"\n",
|
|
"df = df.drop(df[(df.PAY_AMT1 == 0) &\n",
|
|
" (df.PAY_AMT2 == 0) &\n",
|
|
" (df.PAY_AMT3 == 0) &\n",
|
|
" (df.PAY_AMT4 == 0) &\n",
|
|
" (df.PAY_AMT5 == 0) &\n",
|
|
" (df.PAY_AMT6 == 0)].index)\n",
|
|
"\n",
|
|
"from sklearn.linear_model import LogisticRegression\n",
|
|
"from sklearn.model_selection import GridSearchCV\n",
|
|
"\n",
|
|
"lambdas=np.logspace(-5,7,13)\n",
|
|
"parameters = [{'C': 1./lambdas, \"solver\":[\"lbfgs\"]}]#*len(parameters)}]\n",
|
|
"scoring = ['accuracy', 'roc_auc']\n",
|
|
"logReg = LogisticRegression()\n",
|
|
"gridSearch = GridSearchCV(logReg, parameters, cv=5, scoring=scoring, refit='roc_auc')"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.7.4"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|