diff --git a/doc/pub/LogReg/ipynb/LogReg.ipynb b/doc/pub/LogReg/ipynb/LogReg.ipynb new file mode 100644 index 000000000..4c9b78863 --- /dev/null +++ b/doc/pub/LogReg/ipynb/LogReg.ipynb @@ -0,0 +1,295 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "# Data Analysis and Machine Learning: Logistic Regression\n", + "\n", + " \n", + "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", + "\n", + "Date: **Sep 17, 2018**\n", + "\n", + "Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "## Logistic Regression\n", + "\n", + "So far we have focused on learning from datasets for which there is a\n", + "**continuous** output. In linear regression we have been \n", + "concerned with learning the coefficients of a polynomial to predict\n", + "the response of a continuous variable $y_i$ on unseen data based on\n", + "its independent variables ${\\bf x}_i$. \n", + "\n", + "Classification problems,\n", + "however, are concerned with outcomes taking the form of discrete\n", + "variables (i.e. categories). For example, we may want to detect if\n", + "there's a cat or a dog in an image. Or given a specific system,\n", + "we'd like to identify its state, say whether it is an ordered or disordered system (typical situation in solid state physics).\n", + "(e.g. ordered/disordered). \n", + "\n", + "**Logistic regression deals with binary, dichotomous outcomes (e.g. True or\n", + "False, Success or Failure, etc.). It is worth noting that logistic\n", + "regression is also commonly used in modern supervised Deep Learning\n", + "models**, as we will see later.\n", + "\n", + "\n", + "\n", + "## Basics\n", + "\n", + "We consider the case where the dependent variables $y_i\\in\\mathbb{Z}$\n", + "are discrete and only take values from $m=0,\\dots,M-1$ (i.e. $M$\n", + "classes).\n", + "\n", + "The goal is to predict the\n", + "output classes from the design matrix $X\\in\\mathbb{R}^{n\\times p}$\n", + "made of $n$ samples, each of which bears $p$ features. The\n", + "primary goal is to identify the classes to which new unseen samples\n", + "belong.\n", + "\n", + "\n", + "## Linear classifier\n", + "\n", + "Let us start by considering a slightly simpler classifier: a linear classifier that categorizes examples using a weighted linear-combination of the features and an additive offset" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "s_i = \\boldsymbol{x}_i^T\\boldsymbol{w} + b_0 \\equiv \\mathbf{x}_i^T\\mathbf{w},\n", + "\\label{_auto1} \\tag{1}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where we use the short-hand notation \n", + "$\\mathbf{x}_i = (1,\\boldsymbol{x}_i)$ and $\\mathbf{w}_i = (b_0,\\boldsymbol{w}_i)$. \n", + "\n", + "## Some selected properties\n", + "\n", + "This function takes values on the entire real axis. In the case of\n", + "logistic regression, however, the labels $y_i$ are discrete\n", + "variables. One simple way to get a discrete output is to have sign\n", + "functions that map the output of a linear regressor to $\\{0,1\\}$,\n", + "$f(s_i)=sign(s_i)=1$ if $s_i\\ge 0$ and 0 if otherwise. Indeed,\n", + "this is commonly known as the \"perceptron\" in the machine learning\n", + "literature. This model is extremely simple, and it is favorable in\n", + "many cases (e.g. noisy data) to have a ``soft\" classifier that outputs\n", + "the probability of a given category. For example, given\n", + "$\\mathbf{x}_i$, the classifier outputs the probability of being in\n", + "category $m$. One such function is the logistic (or sigmoid) function:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "f(s) = \\frac{1}{1+\\mathrm e^{-s}}.\n", + "\\label{eq:log_fun} \\tag{2}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note that $1-f(s)= f(-s)$, which will be useful shortly. \n", + "\n", + "## The cross-entropy as a cost function for logistic regression\n", + "\n", + "The perceptron is an example of a ``hard classification\": each datapoint is deterministically assigned to a category (i.e $y_i=0$ or $y_i=1$). In many cases, it is favorable to have a \"soft\" classifier that outputs the probability of a given category rather than a single value. For example, given $\\mathbf{x}_i$, the classifier outputs the probability of being in category $m$. \n", + "Logistic regression is the most canonical example of a soft classifier. In logistic regression, the probability that a data point $\\boldsymbol{x}_i$ belongs to a category $y_i=\\{0,1\\}$ is is given by" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{eqnarray}\n", + "P(y_i=1|\\boldsymbol{x}_i,\\boldsymbol{\\theta)} &=& \\frac{1}{1+\\mathrm{e}^{-\\mathbf{x}^T_i\\mathbf{w}}},\\nonumber\\\\\n", + "P(y_i=0|\\boldsymbol{x}_i,\\boldsymbol{\\theta)} &=& 1 - P(y_i=1|\\boldsymbol{x}_i,\\boldsymbol{\\theta)},\n", + "\\end{eqnarray}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where $\\boldsymbol{\\theta}=\\mathbf{w}$ are the weights we wish to learn from the data. \n", + "\n", + "\n", + "Notice that in terms of the logistic function, we can write" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "P(y_i=1) =f(\\mathbf{x}_i^T\\mathbf{w})=1-P(y_i=0).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "## Maximum likelihood\n", + "\n", + "We now define the cost function for logistic regression using Maximum\n", + "Likelihood Estimation (MLE). Recall, that in MLE we choose parameters\n", + "to maximize the probability of seeing the observed data. Consider a\n", + "dataset $\\mathcal{D}=\\{(y_i,\\boldsymbol{x}_i)\\}$ with binary labels\n", + "$y_i\\in\\{0,1\\}$ where the data points are drawn independently. The\n", + "likelihood of the seeing the data under our model is just:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "P(\\mathcal{D}|\\mathbf{w}) = \\prod_{i=1}^n \\left[f(\\mathbf{x}_i^T\\mathbf{w})\\right]^{y_i}\\left[1-f(\\mathbf{x}_i^T\\mathbf{w})\\right]^{1-y_i}\\nonumber\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation} \n", + "\\label{_auto2} \\tag{3}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "from which we can readily compute the log-likelihood:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "l(\\mathbf{w}) = \\sum_{i=1}^n y_i\\log f(\\mathbf{x}_i^T\\mathbf{w}) + (1-y_i)\\log\\left[1-f(\\mathbf{x}_i^T\\mathbf{w})\\right].\n", + "\\label{_auto3} \\tag{4}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\theta$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\hat{\\mathbf{w}} = \\sum_{i=1}^n y_i\\log f(\\mathbf{x}_i^T\\mathbf{w}) + (1-y_i)\\log\\left[1-f(\\mathbf{x}_i^T\\mathbf{w})\\right].\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{eqnarray}\n", + "\\mathcal{C}(\\mathbf{w}) &=& - l(\\mathbf{w}) \\\\\n", + "&=& \\sum_{i=1}^n -y_i\\log f(\\mathbf{x}_i^T\\mathbf{w}) - (1-y_i)\\log\\left[1-f(\\mathbf{x}_i^T\\mathbf{w})\\right].\\nonumber\n", + "\\end{eqnarray}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This equation is known in statistics as the \\emph{cross entropy}. Finally, we note that just as in linear regression, \n", + "in practice we usually supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.\n", + "\n", + "## Minimizing the cross entropy\n", + "\n", + "The cross entropy is a convex function of the weights $\\mathbf{w}$ and,\n", + "therefore, any local minimizer is a global minimizer. Minimizing this\n", + "cost function leads to the following equation" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\boldsymbol{0}=\\boldsymbol{\\nabla} \\mathcal{C}(\\mathbf{w}) = \\sum_{i=1}^n\\left[f(\\mathbf{x}_i^T\\mathbf{w})-y_i\\right]\\mathbf{x}_i,\n", + "\\label{_auto4} \\tag{5}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where we made use of the logistic function identity $\\partial_z f(z) =\n", + "f(z)[1-f(z)]$. This equation defines a transcendental equation for\n", + "$\\mathbf{w}$, the solution of which, unlike linear regression, cannot\n", + "be written in a closed form. \n", + "Here we need gradient descent methods!" + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/doc/pub/LogReg/ipynb/ipynb-LogReg-src.tar.gz b/doc/pub/LogReg/ipynb/ipynb-LogReg-src.tar.gz new file mode 100644 index 000000000..4f1f36dae Binary files /dev/null and b/doc/pub/LogReg/ipynb/ipynb-LogReg-src.tar.gz differ