Created chapter on Log regression
This commit is contained in:
mhjensen
@@ -0,0 +1,153 @@
|
||||
TITLE: Data Analysis and Machine Learning: Logistic Regression
|
||||
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
|
||||
DATE: today
|
||||
|
||||
|
||||
!split
|
||||
===== Logistic Regression =====
|
||||
|
||||
So far we have focused on learning from datasets for which there is a
|
||||
_continuous_ output. In linear regression we have been
|
||||
concerned with learning the coefficients of a polynomial to predict
|
||||
the response of a continuous variable $y_i$ on unseen data based on
|
||||
its independent variables ${\bf x}_i$.
|
||||
|
||||
Classification problems,
|
||||
however, are concerned with outcomes taking the form of discrete
|
||||
variables (i.e. categories). For example, we may want to detect if
|
||||
there's a cat or a dog in an image. Or given a specific system,
|
||||
we'd like to identify its state, say whether it is an ordered or disordered system (typical situation in solid state physics).
|
||||
(e.g. ordered/disordered).
|
||||
|
||||
_Logistic regression deals with binary, dichotomous outcomes (e.g. True or
|
||||
False, Success or Failure, etc.). It is worth noting that logistic
|
||||
regression is also commonly used in modern supervised Deep Learning
|
||||
models_, as we will see later.
|
||||
|
||||
|
||||
!split
|
||||
===== Basics =====
|
||||
|
||||
We consider the case where the dependent variables $y_i\in\mathbb{Z}$
|
||||
are discrete and only take values from $m=0,\dots,M-1$ (i.e. $M$
|
||||
classes).
|
||||
|
||||
The goal is to predict the
|
||||
output classes from the design matrix $X\in\mathbb{R}^{n\times p}$
|
||||
made of $n$ samples, each of which bears $p$ features. The
|
||||
primary goal is to identify the classes to which new unseen samples
|
||||
belong.
|
||||
|
||||
|
||||
!split
|
||||
===== Linear classifier =====
|
||||
|
||||
Let us start by considering a slightly simpler classifier: a linear classifier that categorizes examples using a weighted linear-combination of the features and an additive offset
|
||||
!bt
|
||||
\begin{equation}
|
||||
s_i = \boldsymbol{x}_i^T\boldsymbol{w} + b_0 \equiv \mathbf{x}_i^T\mathbf{w},
|
||||
\end{equation}
|
||||
!et
|
||||
where we use the short-hand notation
|
||||
$\mathbf{x}_i = (1,\boldsymbol{x}_i)$ and $\mathbf{w}_i = (b_0,\boldsymbol{w}_i)$.
|
||||
|
||||
!split
|
||||
===== Some selected properties =====
|
||||
|
||||
This function takes values on the entire real axis. In the case of
|
||||
logistic regression, however, the labels $y_i$ are discrete
|
||||
variables. One simple way to get a discrete output is to have sign
|
||||
functions that map the output of a linear regressor to $\{0,1\}$,
|
||||
$f(s_i)=sign(s_i)=1$ if $s_i\ge 0$ and 0 if otherwise. Indeed,
|
||||
this is commonly known as the ``perceptron" in the machine learning
|
||||
literature. This model is extremely simple, and it is favorable in
|
||||
many cases (e.g. noisy data) to have a ``soft" classifier that outputs
|
||||
the probability of a given category. For example, given
|
||||
$\mathbf{x}_i$, the classifier outputs the probability of being in
|
||||
category $m$. One such function is the logistic (or sigmoid) function:
|
||||
|
||||
!bt
|
||||
\begin{equation}
|
||||
f(s) = \frac{1}{1+\mathrm e^{-s}}.
|
||||
\label{eq:log_fun}
|
||||
\end{equation}
|
||||
!et
|
||||
Note that $1-f(s)= f(-s)$, which will be useful shortly.
|
||||
|
||||
!split
|
||||
===== The cross-entropy as a cost function for logistic regression =====
|
||||
|
||||
The perceptron is an example of a ``hard classification'': each datapoint is deterministically assigned to a category (i.e $y_i=0$ or $y_i=1$). In many cases, it is favorable to have a ``soft'' classifier that outputs the probability of a given category rather than a single value. For example, given $\mathbf{x}_i$, the classifier outputs the probability of being in category $m$.
|
||||
Logistic regression is the most canonical example of a soft classifier. In logistic regression, the probability that a data point $\boldsymbol{x}_i$ belongs to a category $y_i=\{0,1\}$ is is given by
|
||||
!bt
|
||||
\begin{eqnarray}
|
||||
P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& \frac{1}{1+\mathrm{e}^{-\mathbf{x}^T_i\mathbf{w}}},\nonumber\\
|
||||
P(y_i=0|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& 1 - P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)},
|
||||
\end{eqnarray}
|
||||
!et
|
||||
where $\boldsymbol{\theta}=\mathbf{w}$ are the weights we wish to learn from the data.
|
||||
|
||||
|
||||
Notice that in terms of the logistic function, we can write
|
||||
!bt
|
||||
\[
|
||||
P(y_i=1) =f(\mathbf{x}_i^T\mathbf{w})=1-P(y_i=0).
|
||||
\]
|
||||
!et
|
||||
|
||||
!split
|
||||
===== Maximum likelihood =====
|
||||
|
||||
We now define the cost function for logistic regression using Maximum
|
||||
Likelihood Estimation (MLE). Recall, that in MLE we choose parameters
|
||||
to maximize the probability of seeing the observed data. Consider a
|
||||
dataset $\mathcal{D}=\{(y_i,\boldsymbol{x}_i)\}$ with binary labels
|
||||
$y_i\in\{0,1\}$ where the data points are drawn independently. The
|
||||
likelihood of the seeing the data under our model is just:
|
||||
!bt
|
||||
\begin{align}
|
||||
P(\mathcal{D}|\mathbf{w})& = \prod_{i=1}^n \left[f(\mathbf{x}_i^T\mathbf{w})\right]^{y_i}\left[1-f(\mathbf{x}_i^T\mathbf{w})\right]^{1-y_i}\nonumber \\
|
||||
\end{align}
|
||||
!et
|
||||
from which we can readily compute the log-likelihood:
|
||||
!bt
|
||||
\begin{equation}
|
||||
l(\mathbf{w}) = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].
|
||||
\end{equation}
|
||||
!et
|
||||
|
||||
!split
|
||||
The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\theta$
|
||||
!bt
|
||||
\[
|
||||
\hat{\mathbf{w}} = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].
|
||||
\]
|
||||
!et
|
||||
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
|
||||
!bt
|
||||
\begin{eqnarray}
|
||||
\mathcal{C}(\mathbf{w}) &=& - l(\mathbf{w}) \\
|
||||
&=& \sum_{i=1}^n -y_i\log f(\mathbf{x}_i^T\mathbf{w}) - (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].\nonumber
|
||||
\end{eqnarray}
|
||||
!et
|
||||
This equation is known in statistics as the \emph{cross entropy}. Finally, we note that just as in linear regression,
|
||||
in practice we usually supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.
|
||||
|
||||
!split
|
||||
===== Minimizing the cross entropy =====
|
||||
|
||||
The cross entropy is a convex function of the weights $\mathbf{w}$ and,
|
||||
therefore, any local minimizer is a global minimizer. Minimizing this
|
||||
cost function leads to the following equation
|
||||
|
||||
!bt
|
||||
\begin{equation}
|
||||
\boldsymbol{0}=\boldsymbol{\nabla} \mathcal{C}(\mathbf{w}) = \sum_{i=1}^n\left[f(\mathbf{x}_i^T\mathbf{w})-y_i\right]\mathbf{x}_i,
|
||||
\end{equation}
|
||||
!et
|
||||
|
||||
where we made use of the logistic function identity $\partial_z f(z) =
|
||||
f(z)[1-f(z)]$. This equation defines a transcendental equation for
|
||||
$\mathbf{w}$, the solution of which, unlike linear regression, cannot
|
||||
be written in a closed form.
|
||||
Here we need gradient descent methods!
|
||||
@@ -0,0 +1,12 @@
|
||||
\mode<presentation>
|
||||
\usecolortheme[rgb={0.8, 0.2, 0}]{structure}
|
||||
\usefonttheme[onlysmall]{structurebold}
|
||||
|
||||
\setbeamertemplate{navigation symbols}{}
|
||||
%\setbeamertemplate{footline}[frame number]
|
||||
|
||||
\usepackage{tikz}
|
||||
\usetikzlibrary{arrows,shapes,backgrounds,decorations,mindmap}
|
||||
|
||||
\mode
|
||||
<all>
|
||||
@@ -0,0 +1,15 @@
|
||||
\mode<presentation>
|
||||
|
||||
\useoutertheme{smoothbars}
|
||||
\useinnertheme[shadow=true]{rounded}
|
||||
\usecolortheme{orchid}
|
||||
\usecolortheme{whale}
|
||||
\usecolortheme[rgb={0.7, 0.2, 0}]{structure} % (darker red)
|
||||
\useoutertheme{shadow}
|
||||
\usefonttheme[onlysmall]{structurebold}
|
||||
|
||||
\setbeamercolor{title}{use=structure,fg=white,bg=structure.fg}
|
||||
\setbeamerfont{block title}{size={}}
|
||||
|
||||
\mode
|
||||
<all>
|
||||
Executable
+3
@@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
doconce clean
|
||||
rm -rf *.pdf *.tex ipynb*.tar.gz *.html ._*.html *~ reveal.js Trash README.txt
|
||||
Executable
+118
@@ -0,0 +1,118 @@
|
||||
#!/bin/sh
|
||||
set -x
|
||||
|
||||
function system {
|
||||
"$@"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "make.sh: unsuccessful command $@"
|
||||
echo "abort!"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
if [ $# -eq 0 ]; then
|
||||
echo 'bash make.sh slides1|slides2'
|
||||
exit 1
|
||||
fi
|
||||
|
||||
name=$1
|
||||
rm -f *.tar.gz
|
||||
|
||||
opt="--encoding=utf-8"
|
||||
# Note: Makefile examples contain constructions like ${PROG} which
|
||||
# looks like Mako constructions, but they are not. Use --no_mako
|
||||
# to turn off Mako processing.
|
||||
opt="--no_mako"
|
||||
|
||||
rm -f *.aux
|
||||
|
||||
|
||||
html=${name}-reveal
|
||||
system doconce format html $name --pygments_html_style=perldoc --keep_pygments_html_bg --html_links_in_new_window --html_output=$html $opt
|
||||
system doconce slides_html $html reveal --html_slide_theme=beige
|
||||
|
||||
# Plain HTML documents
|
||||
|
||||
html=${name}-solarized
|
||||
system doconce format html $name --pygments_html_style=perldoc --html_style=solarized3 --html_links_in_new_window --html_output=$html $opt
|
||||
system doconce split_html $html.html --method=space10
|
||||
|
||||
html=${name}
|
||||
system doconce format html $name --pygments_html_style=default --html_style=bloodish --html_links_in_new_window --html_output=$html $opt
|
||||
system doconce split_html $html.html --method=space10
|
||||
|
||||
# Bootstrap style
|
||||
html=${name}-bs
|
||||
system doconce format html $name --html_style=bootstrap --pygments_html_style=default --html_admon=bootstrap_panel --html_output=$html $opt
|
||||
system doconce split_html $html.html --method=split --pagination --nav_button=bottom
|
||||
|
||||
# IPython notebook
|
||||
system doconce format ipynb $name $opt
|
||||
|
||||
# LaTeX Beamer slides
|
||||
beamertheme=red_plain
|
||||
system doconce format pdflatex $name --latex_title_layout=beamer --latex_table_format=footnotesize $opt
|
||||
system doconce ptex2tex $name envir=minted
|
||||
# Add special packages
|
||||
doconce subst "% Add user's preamble" "\g<1>\n\\usepackage{simplewick}" $name.tex
|
||||
system doconce slides_beamer $name --beamer_slide_theme=$beamertheme
|
||||
system pdflatex -shell-escape ${name}
|
||||
system pdflatex -shell-escape ${name}
|
||||
cp $name.pdf ${name}-beamer.pdf
|
||||
cp $name.tex ${name}-beamer.tex
|
||||
|
||||
# Handouts
|
||||
system doconce format pdflatex $name --latex_title_layout=beamer --latex_table_format=footnotesize $opt
|
||||
system doconce ptex2tex $name envir=minted
|
||||
# Add special packages
|
||||
doconce subst "% Add user's preamble" "\g<1>\n\\usepackage{simplewick}" $name.tex
|
||||
system doconce slides_beamer $name --beamer_slide_theme=red_shadow --handout
|
||||
system pdflatex -shell-escape $name
|
||||
pdflatex -shell-escape $name
|
||||
pdflatex -shell-escape $name
|
||||
pdfnup --nup 2x3 --frame true --delta "1cm 1cm" --scale 0.9 --outfile ${name}-beamer-handouts2x3.pdf ${name}.pdf
|
||||
rm -f ${name}.pdf
|
||||
|
||||
# Ordinary plain LaTeX document
|
||||
rm -f *.aux # important after beamer
|
||||
system doconce format pdflatex $name --minted_latex_style=trac --latex_admon=paragraph $opt
|
||||
system doconce ptex2tex $name envir=minted
|
||||
# Add special packages
|
||||
doconce subst "% Add user's preamble" "\g<1>\n\\usepackage{simplewick}" $name.tex
|
||||
doconce replace 'section{' 'section*{' $name.tex
|
||||
pdflatex -shell-escape $name
|
||||
pdflatex -shell-escape $name
|
||||
mv -f $name.pdf ${name}-minted.pdf
|
||||
cp $name.tex ${name}-plain-minted.tex
|
||||
|
||||
|
||||
|
||||
# Publish
|
||||
dest=../../pub
|
||||
if [ ! -d $dest/$name ]; then
|
||||
mkdir $dest/$name
|
||||
mkdir $dest/$name/pdf
|
||||
mkdir $dest/$name/html
|
||||
mkdir $dest/$name/ipynb
|
||||
fi
|
||||
cp ${name}*.pdf $dest/$name/pdf
|
||||
cp -r ${name}*.html ._${name}*.html reveal.js $dest/$name/html
|
||||
|
||||
# Figures: cannot just copy link, need to physically copy the files
|
||||
if [ -d fig-${name} ]; then
|
||||
if [ ! -d $dest/$name/html/fig-$name ]; then
|
||||
mkdir $dest/$name/html/fig-$name
|
||||
fi
|
||||
cp -r fig-${name}/* $dest/$name/html/fig-$name
|
||||
fi
|
||||
|
||||
cp ${name}.ipynb $dest/$name/ipynb
|
||||
ipynb_tarfile=ipynb-${name}-src.tar.gz
|
||||
if [ ! -f ${ipynb_tarfile} ]; then
|
||||
cat > README.txt <<EOF
|
||||
This IPython notebook ${name}.ipynb does not require any additional
|
||||
programs.
|
||||
EOF
|
||||
tar czf ${ipynb_tarfile} README.txt
|
||||
fi
|
||||
cp ${ipynb_tarfile} $dest/$name/ipynb
|
||||
Reference in New Issue
Block a user