mhjensen
296 lines
10 KiB
Plaintext
296 lines
10 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- dom:TITLE: Data Analysis and Machine Learning: Logistic Regression -->\n",
|
|
"# Data Analysis and Machine Learning: Logistic Regression\n",
|
|
"<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
|
|
"<!-- Author: --> \n",
|
|
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
|
|
"\n",
|
|
"Date: **Sep 17, 2018**\n",
|
|
"\n",
|
|
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"<!-- !split -->\n",
|
|
"## Logistic Regression\n",
|
|
"\n",
|
|
"So far we have focused on learning from datasets for which there is a\n",
|
|
"**continuous** output. In linear regression we have been \n",
|
|
"concerned with learning the coefficients of a polynomial to predict\n",
|
|
"the response of a continuous variable $y_i$ on unseen data based on\n",
|
|
"its independent variables ${\\bf x}_i$. \n",
|
|
"\n",
|
|
"Classification problems,\n",
|
|
"however, are concerned with outcomes taking the form of discrete\n",
|
|
"variables (i.e. categories). For example, we may want to detect if\n",
|
|
"there's a cat or a dog in an image. Or given a specific system,\n",
|
|
"we'd like to identify its state, say whether it is an ordered or disordered system (typical situation in solid state physics).\n",
|
|
"(e.g. ordered/disordered). \n",
|
|
"\n",
|
|
"**Logistic regression deals with binary, dichotomous outcomes (e.g. True or\n",
|
|
"False, Success or Failure, etc.). It is worth noting that logistic\n",
|
|
"regression is also commonly used in modern supervised Deep Learning\n",
|
|
"models**, as we will see later.\n",
|
|
"\n",
|
|
"\n",
|
|
"<!-- !split -->\n",
|
|
"## Basics\n",
|
|
"\n",
|
|
"We consider the case where the dependent variables $y_i\\in\\mathbb{Z}$\n",
|
|
"are discrete and only take values from $m=0,\\dots,M-1$ (i.e. $M$\n",
|
|
"classes).\n",
|
|
"\n",
|
|
"The goal is to predict the\n",
|
|
"output classes from the design matrix $X\\in\\mathbb{R}^{n\\times p}$\n",
|
|
"made of $n$ samples, each of which bears $p$ features. The\n",
|
|
"primary goal is to identify the classes to which new unseen samples\n",
|
|
"belong.\n",
|
|
"\n",
|
|
"\n",
|
|
"## Linear classifier\n",
|
|
"\n",
|
|
"Let us start by considering a slightly simpler classifier: a linear classifier that categorizes examples using a weighted linear-combination of the features and an additive offset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto1\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
"s_i = \\boldsymbol{x}_i^T\\boldsymbol{w} + b_0 \\equiv \\mathbf{x}_i^T\\mathbf{w},\n",
|
|
"\\label{_auto1} \\tag{1}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"where we use the short-hand notation \n",
|
|
"$\\mathbf{x}_i = (1,\\boldsymbol{x}_i)$ and $\\mathbf{w}_i = (b_0,\\boldsymbol{w}_i)$. \n",
|
|
"\n",
|
|
"## Some selected properties\n",
|
|
"\n",
|
|
"This function takes values on the entire real axis. In the case of\n",
|
|
"logistic regression, however, the labels $y_i$ are discrete\n",
|
|
"variables. One simple way to get a discrete output is to have sign\n",
|
|
"functions that map the output of a linear regressor to $\\{0,1\\}$,\n",
|
|
"$f(s_i)=sign(s_i)=1$ if $s_i\\ge 0$ and 0 if otherwise. Indeed,\n",
|
|
"this is commonly known as the \"perceptron\" in the machine learning\n",
|
|
"literature. This model is extremely simple, and it is favorable in\n",
|
|
"many cases (e.g. noisy data) to have a ``soft\" classifier that outputs\n",
|
|
"the probability of a given category. For example, given\n",
|
|
"$\\mathbf{x}_i$, the classifier outputs the probability of being in\n",
|
|
"category $m$. One such function is the logistic (or sigmoid) function:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"eq:log_fun\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
"f(s) = \\frac{1}{1+\\mathrm e^{-s}}.\n",
|
|
"\\label{eq:log_fun} \\tag{2}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Note that $1-f(s)= f(-s)$, which will be useful shortly. \n",
|
|
"\n",
|
|
"## The cross-entropy as a cost function for logistic regression\n",
|
|
"\n",
|
|
"The perceptron is an example of a ``hard classification\": each datapoint is deterministically assigned to a category (i.e $y_i=0$ or $y_i=1$). In many cases, it is favorable to have a \"soft\" classifier that outputs the probability of a given category rather than a single value. For example, given $\\mathbf{x}_i$, the classifier outputs the probability of being in category $m$. \n",
|
|
"Logistic regression is the most canonical example of a soft classifier. In logistic regression, the probability that a data point $\\boldsymbol{x}_i$ belongs to a category $y_i=\\{0,1\\}$ is is given by"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\begin{eqnarray}\n",
|
|
"P(y_i=1|\\boldsymbol{x}_i,\\boldsymbol{\\theta)} &=& \\frac{1}{1+\\mathrm{e}^{-\\mathbf{x}^T_i\\mathbf{w}}},\\nonumber\\\\\n",
|
|
"P(y_i=0|\\boldsymbol{x}_i,\\boldsymbol{\\theta)} &=& 1 - P(y_i=1|\\boldsymbol{x}_i,\\boldsymbol{\\theta)},\n",
|
|
"\\end{eqnarray}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"where $\\boldsymbol{\\theta}=\\mathbf{w}$ are the weights we wish to learn from the data. \n",
|
|
"\n",
|
|
"\n",
|
|
"Notice that in terms of the logistic function, we can write"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"P(y_i=1) =f(\\mathbf{x}_i^T\\mathbf{w})=1-P(y_i=0).\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- !split -->\n",
|
|
"## Maximum likelihood\n",
|
|
"\n",
|
|
"We now define the cost function for logistic regression using Maximum\n",
|
|
"Likelihood Estimation (MLE). Recall, that in MLE we choose parameters\n",
|
|
"to maximize the probability of seeing the observed data. Consider a\n",
|
|
"dataset $\\mathcal{D}=\\{(y_i,\\boldsymbol{x}_i)\\}$ with binary labels\n",
|
|
"$y_i\\in\\{0,1\\}$ where the data points are drawn independently. The\n",
|
|
"likelihood of the seeing the data under our model is just:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"P(\\mathcal{D}|\\mathbf{w}) = \\prod_{i=1}^n \\left[f(\\mathbf{x}_i^T\\mathbf{w})\\right]^{y_i}\\left[1-f(\\mathbf{x}_i^T\\mathbf{w})\\right]^{1-y_i}\\nonumber\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto2\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation} \n",
|
|
"\\label{_auto2} \\tag{3}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"from which we can readily compute the log-likelihood:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto3\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
"l(\\mathbf{w}) = \\sum_{i=1}^n y_i\\log f(\\mathbf{x}_i^T\\mathbf{w}) + (1-y_i)\\log\\left[1-f(\\mathbf{x}_i^T\\mathbf{w})\\right].\n",
|
|
"\\label{_auto3} \\tag{4}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\theta$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\hat{\\mathbf{w}} = \\sum_{i=1}^n y_i\\log f(\\mathbf{x}_i^T\\mathbf{w}) + (1-y_i)\\log\\left[1-f(\\mathbf{x}_i^T\\mathbf{w})\\right].\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\n",
|
|
"\\begin{eqnarray}\n",
|
|
"\\mathcal{C}(\\mathbf{w}) &=& - l(\\mathbf{w}) \\\\\n",
|
|
"&=& \\sum_{i=1}^n -y_i\\log f(\\mathbf{x}_i^T\\mathbf{w}) - (1-y_i)\\log\\left[1-f(\\mathbf{x}_i^T\\mathbf{w})\\right].\\nonumber\n",
|
|
"\\end{eqnarray}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"This equation is known in statistics as the \\emph{cross entropy}. Finally, we note that just as in linear regression, \n",
|
|
"in practice we usually supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.\n",
|
|
"\n",
|
|
"## Minimizing the cross entropy\n",
|
|
"\n",
|
|
"The cross entropy is a convex function of the weights $\\mathbf{w}$ and,\n",
|
|
"therefore, any local minimizer is a global minimizer. Minimizing this\n",
|
|
"cost function leads to the following equation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto4\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
"\\boldsymbol{0}=\\boldsymbol{\\nabla} \\mathcal{C}(\\mathbf{w}) = \\sum_{i=1}^n\\left[f(\\mathbf{x}_i^T\\mathbf{w})-y_i\\right]\\mathbf{x}_i,\n",
|
|
"\\label{_auto4} \\tag{5}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"where we made use of the logistic function identity $\\partial_z f(z) =\n",
|
|
"f(z)[1-f(z)]$. This equation defines a transcendental equation for\n",
|
|
"$\\mathbf{w}$, the solution of which, unlike linear regression, cannot\n",
|
|
"be written in a closed form. \n",
|
|
"Here we need gradient descent methods!"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|