In our discussion of neural networks we will encounter the above again
-in terms of the so-called Softmax function.
+in terms of a slightly modified function, the so-called Softmax function.
The softmax function is used in various multiclass classification
diff --git a/doc/pub/LogReg/html/._LogReg-bs015.html b/doc/pub/LogReg/html/._LogReg-bs015.html
index 10e41b46f..b63592ebe 100644
--- a/doc/pub/LogReg/html/._LogReg-bs015.html
+++ b/doc/pub/LogReg/html/._LogReg-bs015.html
@@ -54,7 +54,7 @@ Automatically generated HTML file from DocOnce source
('A more compact expression', 2, None, '___sec10'),
('Extending to more predictors', 2, None, '___sec11'),
('Including more classes', 2, None, '___sec12'),
- ('The Softmax function', 2, None, '___sec13'),
+ ('More classes', 2, None, '___sec13'),
('A simple classification problem', 2, None, '___sec14'),
('The Credit Card example', 2, None, '___sec15')]}
end of tocinfo -->
@@ -107,7 +107,7 @@ MathJax.Hub.Config({
diff --git a/doc/pub/LogReg/html/LogReg-reveal.html b/doc/pub/LogReg/html/LogReg-reveal.html
index d8a85ea5f..ec8858f80 100644
--- a/doc/pub/LogReg/html/LogReg-reveal.html
+++ b/doc/pub/LogReg/html/LogReg-reveal.html
@@ -552,11 +552,11 @@ and the model is specified in term of \( K-1 \) so-called log-odds or
-
The Softmax function
+
More classes
In our discussion of neural networks we will encounter the above again
-in terms of the so-called Softmax function.
+in terms of a slightly modified function, the so-called Softmax function.
The softmax function is used in various multiclass classification
diff --git a/doc/pub/LogReg/html/LogReg-solarized.html b/doc/pub/LogReg/html/LogReg-solarized.html
index bf9db1109..0221a032e 100644
--- a/doc/pub/LogReg/html/LogReg-solarized.html
+++ b/doc/pub/LogReg/html/LogReg-solarized.html
@@ -48,7 +48,7 @@ div { text-align: justify; text-justify: inter-word; }
('A more compact expression', 2, None, '___sec10'),
('Extending to more predictors', 2, None, '___sec11'),
('Including more classes', 2, None, '___sec12'),
- ('The Softmax function', 2, None, '___sec13'),
+ ('More classes', 2, None, '___sec13'),
('A simple classification problem', 2, None, '___sec14'),
('The Credit Card example', 2, None, '___sec15')]}
end of tocinfo -->
@@ -453,11 +453,11 @@ and the model is specified in term of \( K-1 \) so-called log-odds or
-
The Softmax function
+
More classes
In our discussion of neural networks we will encounter the above again
-in terms of the so-called Softmax function.
+in terms of a slightly modified function, the so-called Softmax function.
The softmax function is used in various multiclass classification
diff --git a/doc/pub/LogReg/html/LogReg.html b/doc/pub/LogReg/html/LogReg.html
index 7e5157338..f648e4505 100644
--- a/doc/pub/LogReg/html/LogReg.html
+++ b/doc/pub/LogReg/html/LogReg.html
@@ -53,7 +53,7 @@ div { text-align: justify; text-justify: inter-word; }
('A more compact expression', 2, None, '___sec10'),
('Extending to more predictors', 2, None, '___sec11'),
('Including more classes', 2, None, '___sec12'),
- ('The Softmax function', 2, None, '___sec13'),
+ ('More classes', 2, None, '___sec13'),
('A simple classification problem', 2, None, '___sec14'),
('The Credit Card example', 2, None, '___sec15')]}
end of tocinfo -->
@@ -458,11 +458,11 @@ and the model is specified in term of \( K-1 \) so-called log-odds or
-
The Softmax function
+
More classes
In our discussion of neural networks we will encounter the above again
-in terms of the so-called Softmax function.
+in terms of a slightly modified function, the so-called Softmax function.
The softmax function is used in various multiclass classification
diff --git a/doc/pub/LogReg/ipynb/LogReg.ipynb b/doc/pub/LogReg/ipynb/LogReg.ipynb
index f9502ccdb..d2044aef2 100644
--- a/doc/pub/LogReg/ipynb/LogReg.ipynb
+++ b/doc/pub/LogReg/ipynb/LogReg.ipynb
@@ -535,10 +535,10 @@
"**logit** transformations.\n",
"\n",
"\n",
- "## The Softmax function\n",
+ "## More classes\n",
"\n",
"In our discussion of neural networks we will encounter the above again\n",
- "in terms of the so-called **Softmax** function.\n",
+ "in terms of a slightly modified function, the so-called **Softmax** function.\n",
"\n",
"The softmax function is used in various multiclass classification\n",
"methods, such as multinomial logistic regression (also known as\n",
diff --git a/doc/pub/LogReg/ipynb/ipynb-LogReg-src.tar.gz b/doc/pub/LogReg/ipynb/ipynb-LogReg-src.tar.gz
index aa6290087..14b03d78d 100644
Binary files a/doc/pub/LogReg/ipynb/ipynb-LogReg-src.tar.gz and b/doc/pub/LogReg/ipynb/ipynb-LogReg-src.tar.gz differ
diff --git a/doc/pub/LogReg/pdf/LogReg-minted.pdf b/doc/pub/LogReg/pdf/LogReg-minted.pdf
index d2d742491..93a91182f 100644
Binary files a/doc/pub/LogReg/pdf/LogReg-minted.pdf and b/doc/pub/LogReg/pdf/LogReg-minted.pdf differ
diff --git a/doc/src/LogisticRegression/LogReg-bs.html b/doc/src/LogisticRegression/LogReg-bs.html
deleted file mode 100644
index 650c44c89..000000000
--- a/doc/src/LogisticRegression/LogReg-bs.html
+++ /dev/null
@@ -1,189 +0,0 @@
-
-
-
-
-
-
-
-
-
diff --git a/doc/src/LogisticRegression/LogReg-minted.pdf b/doc/src/LogisticRegression/LogReg-minted.pdf
deleted file mode 100644
index d2d742491..000000000
Binary files a/doc/src/LogisticRegression/LogReg-minted.pdf and /dev/null differ
diff --git a/doc/src/LogisticRegression/LogReg-plain-minted.tex b/doc/src/LogisticRegression/LogReg-plain-minted.tex
deleted file mode 100644
index 26660e796..000000000
--- a/doc/src/LogisticRegression/LogReg-plain-minted.tex
+++ /dev/null
@@ -1,738 +0,0 @@
-%%
-%% Automatically generated file from DocOnce source
-%% (https://github.com/hplgit/doconce/)
-%%
-%%
-
-
-%-------------------- begin preamble ----------------------
-
-\documentclass[%
-oneside, % oneside: electronic viewing, twoside: printing
-final, % draft: marks overfull hboxes, figures with paths
-10pt]{article}
-
-\listfiles % print all files needed to compile this document
-
-\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb}
-\usepackage[table]{xcolor}
-\usepackage{bm,ltablex,microtype}
-
-\usepackage[pdftex]{graphicx}
-
-\usepackage{fancyvrb} % packages needed for verbatim environments
-\usepackage{minted}
-\usemintedstyle{default}
-
-\usepackage[T1]{fontenc}
-%\usepackage[latin1]{inputenc}
-\usepackage{ucs}
-\usepackage[utf8x]{inputenc}
-
-\usepackage{lmodern} % Latin Modern fonts derived from Computer Modern
-
-% Hyperlinks in PDF:
-\definecolor{linkcolor}{rgb}{0,0,0.4}
-\usepackage{hyperref}
-\hypersetup{
- breaklinks=true,
- colorlinks=true,
- linkcolor=linkcolor,
- urlcolor=linkcolor,
- citecolor=black,
- filecolor=black,
- %filecolor=blue,
- pdfmenubar=true,
- pdftoolbar=true,
- bookmarksdepth=3 % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC
- }
-%\hyperbaseurl{} % hyperlinks are relative to this root
-
-\setcounter{tocdepth}{2} % levels in table of contents
-
-% --- fancyhdr package for fancy headers ---
-\usepackage{fancyhdr}
-\fancyhf{} % sets both header and footer to nothing
-\renewcommand{\headrulewidth}{0pt}
-\fancyfoot[LE,RO]{\thepage}
-% Ensure copyright on titlepage (article style) and chapter pages (book style)
-\fancypagestyle{plain}{
- \fancyhf{}
- \fancyfoot[C]{{\footnotesize \copyright\ 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license}}
-% \renewcommand{\footrulewidth}{0mm}
- \renewcommand{\headrulewidth}{0mm}
-}
-% Ensure copyright on titlepages with \thispagestyle{empty}
-\fancypagestyle{empty}{
- \fancyhf{}
- \fancyfoot[C]{{\footnotesize \copyright\ 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license}}
- \renewcommand{\footrulewidth}{0mm}
- \renewcommand{\headrulewidth}{0mm}
-}
-
-\pagestyle{fancy}
-
-
-% prevent orhpans and widows
-\clubpenalty = 10000
-\widowpenalty = 10000
-
-% --- end of standard preamble for documents ---
-
-
-% insert custom LaTeX commands...
-
-\raggedbottom
-\makeindex
-\usepackage[totoc]{idxlayout} % for index in the toc
-\usepackage[nottoc]{tocbibind} % for references/bibliography in the toc
-
-%-------------------- end preamble ----------------------
-
-\begin{document}
-
-% matching end for #ifdef PREAMBLE
-
-\newcommand{\exercisesection}[1]{\subsection*{#1}}
-
-
-% ------------------- main content ----------------------
-
-
-
-% ----------------- title -------------------------
-
-\thispagestyle{empty}
-
-\begin{center}
-{\LARGE\bf
-\begin{spacing}{1.25}
-Data Analysis and Machine Learning: Logistic Regression
-\end{spacing}
-}
-\end{center}
-
-% ----------------- author(s) -------------------------
-
-\begin{center}
-{\bf Morten Hjorth-Jensen${}^{1, 2}$} \\ [0mm]
-\end{center}
-
-\begin{center}
-% List of all institutions:
-\centerline{{\small ${}^1$Department of Physics, University of Oslo}}
-\centerline{{\small ${}^2$Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University}}
-\end{center}
-
-% ----------------- end author(s) -------------------------
-
-% --- begin date ---
-\begin{center}
-Sep 19, 2019
-\end{center}
-% --- end date ---
-
-\vspace{1cm}
-
-
-% !split
-\subsection*{Logistic Regression}
-
-In linear regression our main interest was centered on learning the
-coefficients of a functional fit (say a polynomial) in order to be
-able to predict the response of a continuous variable on some unseen
-data. The fit to the continuous variable $y_i$ is based on some
-independent variables $\hat{x}_i$. Linear regression resulted in
-analytical expressions for standard ordinary Least Squares or Ridge
-regression (in terms of matrices to invert) for several quantities,
-ranging from the variance and thereby the confidence intervals of the
-parameters $\hat{\beta}$ to the mean squared error. If we can invert
-the product of the design matrices, linear regression gives then a
-simple recipe for fitting our data.
-
-
-Classification problems, however, are concerned with outcomes taking
-the form of discrete variables (i.e.~categories). We may for example,
-on the basis of DNA sequencing for a number of patients, like to find
-out which mutations are important for a certain disease; or based on
-scans of various patients' brains, figure out if there is a tumor or
-not; or given a specific physical system, we'd like to identify its
-state, say whether it is an ordered or disordered system (typical
-situation in solid state physics); or classify the status of a
-patient, whether she/he has a stroke or not and many other similar
-situations.
-
-The most common situation we encounter when we apply logistic
-regression is that of two possible outcomes, normally denoted as a
-binary outcome, true or false, positive or negative, success or
-failure etc.
-
-% !split
-\subsection*{Optimization and Deep learning}
-
-Logistic regression will also serve as our stepping stone towards neural
-network algorithms and supervised deep learning. For logistic
-learning, the minimization of the cost function leads to a non-linear
-equation in the parameters $\hat{\beta}$. The optimization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the family of gradient descent methods. The latter are the working horses of basically all modern machine learning algorithms.
-
-We note also that many of the topics discussed here
-regression are also commonly used in modern supervised Deep Learning
-models, as we will see later.
-
-
-% !split
-\subsection*{Basics}
-
-We consider the case where the dependent variables, also called the
-responses or the outcomes, $y_i$ are discrete and only take values
-from $k=0,\dots,K-1$ (i.e.~$K$ classes).
-
-The goal is to predict the
-output classes from the design matrix $\hat{X}\in\mathbb{R}^{n\times p}$
-made of $n$ samples, each of which carries $p$ features or predictors. The
-primary goal is to identify the classes to which new unseen samples
-belong.
-
-Let us specialize to the case of two classes only, with outputs
-$y_i=0$ and $y_i=1$. Our outcomes could represent the status of a
-credit card user that could default or not on her/his credit card
-debt. That is
-
-
-\[
-y_i = \begin{bmatrix} 0 & \mathrm{no}\\ 1 & \mathrm{yes} \end{bmatrix}.
-\]
-
-
-
-% !split
-\subsection*{Linear classifier}
-
-Before moving to the logistic model, let us try to use our linear regression model to classify these two outcomes. We could for example fit a linear model to the default case if $y_i > 0.5$ and the no default case $y_i \leq 0.5$.
-
-We would then have our
-weighted linear combination, namely
-\begin{equation}
-\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon},
-\end{equation}
-where $\hat{y}$ is a vector representing the possible outcomes, $\hat{X}$ is our
-$n\times p$ design matrix and $\hat{\beta}$ represents our estimators/predictors.
-
-% !split
-\subsection*{Some selected properties}
-
-The main problem with our function is that it
-takes values on the entire real axis. In the case of
-logistic regression, however, the labels $y_i$ are discrete
-variables. A typical example is the credit card data discussed below here, where we can set the state of defaulting the debt to $y_i=1$ and not to $y_i=0$ for one the persons in the data set (see the full example below).
-
-One simple way to get a discrete output is to have sign
-functions that map the output of a linear regressor to values $\{0,1\}$,
-$f(s_i)=sign(s_i)=1$ if $s_i\ge 0$ and 0 if otherwise.
-We will encounter this model in our first demonstration of neural networks. Historically it is called the ``perceptron" model in the machine learning
-literature. This model is extremely simple. However, in many cases it is more
-favorable to use a ``soft" classifier that outputs
-the probability of a given category. This leads us to the logistic function.
-
-
-% !split
-\subsection*{The logistic function}
-
-The perceptron is an example of a ``hard classification'' model. We
-will encounter this model when we discuss neural networks as
-well. Each datapoint is deterministically assigned to a category (i.e
-$y_i=0$ or $y_i=1$). In many cases, it is favorable to have a ``soft''
-classifier that outputs the probability of a given category rather
-than a single value. For example, given $x_i$, the classifier
-outputs the probability of being in a category $k$. Logistic regression
-is the most common example of a so-called soft classifier. In logistic
-regression, the probability that a data point $x_i$
-belongs to a category $y_i=\{0,1\}$ is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,
-\[
-p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
-\]
-Note that $1-p(t)= p(-t)$.
-The following code plots the logistic function, the step function and other functions we will encounter from here and on.
-
-
-\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python}
-"""The sigmoid function (or the logistic curve) is a
-function that takes any real number, z, and outputs a number (0,1).
-It is useful in neural networks for assigning weights on a relative scale.
-The value z is the weighted sum of parameters involved in the learning algorithm."""
-
-import numpy
-import matplotlib.pyplot as plt
-import math as mt
-
-z = numpy.arange(-5, 5, .1)
-sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))
-sigma = sigma_fn(z)
-
-fig = plt.figure()
-ax = fig.add_subplot(111)
-ax.plot(z, sigma)
-ax.set_ylim([-0.1, 1.1])
-ax.set_xlim([-5,5])
-ax.grid(True)
-ax.set_xlabel('z')
-ax.set_title('sigmoid function')
-
-plt.show()
-
-"""Step Function"""
-z = numpy.arange(-5, 5, .02)
-step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)
-step = step_fn(z)
-
-fig = plt.figure()
-ax = fig.add_subplot(111)
-ax.plot(z, step)
-ax.set_ylim([-0.5, 1.5])
-ax.set_xlim([-5,5])
-ax.grid(True)
-ax.set_xlabel('z')
-ax.set_title('step function')
-
-plt.show()
-
-"""tanh Function"""
-z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)
-t = numpy.tanh(z)
-
-fig = plt.figure()
-ax = fig.add_subplot(111)
-ax.plot(z, t)
-ax.set_ylim([-1.0, 1.0])
-ax.set_xlim([-2*mt.pi,2*mt.pi])
-ax.grid(True)
-ax.set_xlabel('z')
-ax.set_title('tanh function')
-
-plt.show()
-\end{minted}
-
-
-
-
-
-
-
-% !split
-\subsection*{Two parameters}
-
-We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\beta$ in our fitting of the Sigmoid function, that is we define probabilities
-\begin{align*}
-p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
-p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
-\end{align*}
-where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$.
-
-Note that we used
-\[
-p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
-\]
-
-% !split
-\subsection*{Maximum likelihood}
-
-In order to define the total likelihood for all possible outcomes from a
-dataset $\mathcal{D}=\{(y_i,x_i)\}$, with the binary labels
-$y_i\in\{0,1\}$ and where the data points are drawn independently, we use the so-called \href{{https://en.wikipedia.org/wiki/Maximum_likelihood_estimation}}{Maximum Likelihood Estimation} (MLE) principle.
-We aim thus at maximizing
-the probability of seeing the observed data. We can then approximate the
-likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is
-\begin{align*}
-P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
-\end{align*}
-from which we obtain the log-likelihood and our \textbf{cost/loss} function
-\[
-\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
-\]
-
-% !split
-\subsection*{The cost function rewritten}
-
-Reordering the logarithms, we can rewrite the \textbf{cost/loss} function as
-\[
-\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
-\]
-
-The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\beta$.
-Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
-\[
-\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
-\]
-This equation is known in statistics as the \textbf{cross entropy}. Finally, we note that just as in linear regression,
-in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.
-
-% !split
-\subsection*{Minimizing the cross entropy}
-
-The cross entropy is a convex function of the weights $\hat{\beta}$ and,
-therefore, any local minimizer is a global minimizer.
-
-
-Minimizing this
-cost function with respect to the two parameters $\beta_0$ and $\beta_1$ we obtain
-
-\[
-\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
-\]
-and
-\[
-\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
-\]
-
-% !split
-\subsection*{A more compact expression}
-
-Let us now define a vector $\hat{y}$ with $n$ elements $y_i$, an
-$n\times p$ matrix $\hat{X}$ which contains the $x_i$ values and a
-vector $\hat{p}$ of fitted probabilities $p(y_i\vert x_i,\hat{\beta})$. We can rewrite in a more compact form the first
-derivative of cost function as
-
-\[
-\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
-\]
-
-If we in addition define a diagonal matrix $\hat{W}$ with elements
-$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as
-
-\[
-\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
-\]
-
-% !split
-\subsection*{Extending to more predictors}
-
-Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors
-\[
-\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
-\]
-Here we defined $\hat{x}=[1,x_1,x_2,\dots,x_p]$ and $\hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to
-\[
-p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
-\]
-
-% !split
-\subsection*{Including more classes}
-
-Till now we have mainly focused on two classes, the so-called binary
-system. Suppose we wish to extend to $K$ classes. Let us for the sake
-of simplicity assume we have only two predictors. We have then
-following model
-
-\[
-\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1,
-\]
-\[
-\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1,
-\]
-and so on till the class $C=K-1$ class
-\[
-\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1,
-\]
-
-and the model is specified in term of $K-1$ so-called log-odds or
-\textbf{logit} transformations.
-
-
-% !split
-\subsection*{The Softmax function}
-
-In our discussion of neural networks we will encounter the above again
-in terms of the so-called \textbf{Softmax} function.
-
-The softmax function is used in various multiclass classification
-methods, such as multinomial logistic regression (also known as
-softmax regression), multiclass linear discriminant analysis, naive
-Bayes classifiers, and artificial neural networks. Specifically, in
-multinomial logistic regression and linear discriminant analysis, the
-input to the function is the result of $K$ distinct linear functions,
-and the predicted probability for the $k$-th class given a sample
-vector $\hat{x}$ and a weighting vector $\hat{\beta}$ is (with two
-predictors):
-
-\[
-p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}.
-\]
-It is easy to extend to more predictors. The final class is
-\[
-p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}},
-\]
-
-and they sum to one. Our earlier discussions were all specialized to
-the case with two classes only. It is easy to see from the above that
-what we derived earlier is compatible with these equations.
-
-To find the optimal parameters we would typically use a gradient
-descent method. Newton's method and gradient descent methods are
-discussed in the material on \href{{https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html}}{optimization
-methods}.
-
-
-
-
-% !split
-\subsection*{A simple classification problem}
-\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python}
-import numpy as np
-from sklearn import datasets, linear_model
-import matplotlib.pyplot as plt
-
-
-def generate_data():
- np.random.seed(0)
- X, y = datasets.make_moons(200, noise=0.20)
- return X, y
-
-
-def visualize(X, y, clf):
- plot_decision_boundary(lambda x: clf.predict(x), X, y)
-
-def plot_decision_boundary(pred_func, X, y):
- # Set min and max values and give it some padding
- x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
- y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
- h = 0.01
- # Generate a grid of points with distance h between them
- xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
- # Predict the function value for the whole gid
- Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
- Z = Z.reshape(xx.shape)
- # Plot the contour and training examples
- plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
- plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
- plt.show()
-
-
-def classify(X, y):
- clf = linear_model.LogisticRegressionCV()
- clf.fit(X, y)
- return clf
-
-
-def main():
- X, y = generate_data()
- # visualize(X, y)
- clf = classify(X, y)
- visualize(X, y, clf)
-
-if __name__ == "__main__":
- main()
-\end{minted}
-
-
-
-% !split
-\subsection*{The Credit Card example}
-Here we use the the \href{{https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients}}{credit card data}. More text to come.
-
-\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python}
-import pandas as pd
-import os
-import numpy as np
-
-
-from sklearn.model_selection import train_test_split
-from sklearn.preprocessing import OneHotEncoder
-from sklearn.compose import ColumnTransformer
-from sklearn.preprocessing import StandardScaler, OneHotEncoder
-from sklearn.metrics import confusion_matrix, accuracy_score, roc_auc_score
-
-# Trying to set the seed
-np.random.seed(0)
-import random
-random.seed(0)
-
-# Reading file into data frame
-cwd = os.getcwd()
-filename = cwd + '/default of credit card clients.xls'
-nanDict = {}
-df = pd.read_excel(filename, header=1, skiprows=0, index_col=0, na_values=nanDict)
-
-df.rename(index=str, columns={"default payment next month": "defaultPaymentNextMonth"}, inplace=True)
-
-# Features and targets
-X = df.loc[:, df.columns != 'defaultPaymentNextMonth'].values
-y = df.loc[:, df.columns == 'defaultPaymentNextMonth'].values
-
-# Categorical variables to one-hot's
-onehotencoder = OneHotEncoder(categories="auto")
-
-X = ColumnTransformer(
- [("", onehotencoder, [3]),],
- remainder="passthrough"
-).fit_transform(X)
-
-y.shape
-
-# Train-test split
-trainingShare = 0.5
-seed = 1
-XTrain, XTest, yTrain, yTest=train_test_split(X, y, train_size=trainingShare, \
- test_size = 1-trainingShare,
- random_state=seed)
-
-# Input Scaling
-sc = StandardScaler()
-XTrain = sc.fit_transform(XTrain)
-XTest = sc.transform(XTest)
-
-# One-hot's of the target vector
-Y_train_onehot, Y_test_onehot = onehotencoder.fit_transform(yTrain), onehotencoder.fit_transform(yTest)
-
-# Remove instances with zeros only for past bill statements or paid amounts
-'''
-df = df.drop(df[(df.BILL_AMT1 == 0) &
- (df.BILL_AMT2 == 0) &
- (df.BILL_AMT3 == 0) &
- (df.BILL_AMT4 == 0) &
- (df.BILL_AMT5 == 0) &
- (df.BILL_AMT6 == 0) &
- (df.PAY_AMT1 == 0) &
- (df.PAY_AMT2 == 0) &
- (df.PAY_AMT3 == 0) &
- (df.PAY_AMT4 == 0) &
- (df.PAY_AMT5 == 0) &
- (df.PAY_AMT6 == 0)].index)
-'''
-df = df.drop(df[(df.BILL_AMT1 == 0) &
- (df.BILL_AMT2 == 0) &
- (df.BILL_AMT3 == 0) &
- (df.BILL_AMT4 == 0) &
- (df.BILL_AMT5 == 0) &
- (df.BILL_AMT6 == 0)].index)
-
-df = df.drop(df[(df.PAY_AMT1 == 0) &
- (df.PAY_AMT2 == 0) &
- (df.PAY_AMT3 == 0) &
- (df.PAY_AMT4 == 0) &
- (df.PAY_AMT5 == 0) &
- (df.PAY_AMT6 == 0)].index)
-
-from sklearn.linear_model import LogisticRegression
-from sklearn.model_selection import GridSearchCV
-
-lambdas=np.logspace(-5,7,13)
-parameters = [{'C': 1./lambdas, "solver":["lbfgs"]}]#*len(parameters)}]
-scoring = ['accuracy', 'roc_auc']
-logReg = LogisticRegression()
-gridSearch = GridSearchCV(logReg, parameters, cv=5, scoring=scoring, refit='roc_auc')
-\end{minted}
-
-\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python}
-
-# "refit" gives the metric used deciding best model.
-# See more http://scikit-learn.org/stable/auto_examples/model_selection/plot_multi_metric_evaluation.html
-gridSearch.fit(XTrain, yTrain.ravel())
-
-def gridSearchSummary(method, scoring):
- """Prints best parameters from Grid search
- and AUC with standard deviation for all
- parameter combos """
-
- method = eval(method)
- if scoring == 'accuracy':
- mean = 'mean_test_score'
- sd = 'std_test_score'
- elif scoring == 'auc':
- mean = 'mean_test_roc_auc'
- sd = 'std_test_roc_auc'
- print("Best: %f using %s" % (method.best_score_, method.best_params_))
- means = method.cv_results_[mean]
- stds = method.cv_results_[sd]
- params = method.cv_results_['params']
- for mean, stdev, param in zip(means, stds, params):
- print("%f (%f) with: %r" % (mean, stdev, param))
-
-def createConfusionMatrix(method, printOut=True):
- """
- Computes and prints confusion matrices, accuracy scores,
- and AUC for test and training sets
- """
- confusionArray = np.zeros(6, dtype=object)
- method = eval(method)
-
- # Train
- yPredTrain = method.predict(XTrain)
- yPredTrain = (yPredTrain > 0.5)
- cm = confusion_matrix(
- yTrain, yPredTrain)
- cm = np.around(cm/cm.sum(axis=1)[:,None], 2)
- confusionArray[0] = cm
-
- accScore = accuracy_score(yTrain, yPredTrain)
- confusionArray[1] = accScore
-
- AUC = roc_auc_score(yTrain, yPredTrain)
- confusionArray[2] = AUC
-
- if printOut:
- print('\n################### Training ###############')
- print('\nTraining Confusion matrix: \n', cm)
- print('\nTraining Accuracy score: \n', accScore)
- print('\nTrain AUC: \n', AUC)
-
- # Test
- yPred = method.predict(XTest)
- yPred = (yPred > 0.5)
- cm = confusion_matrix(
- yTest, yPred)
- cm = np.around(cm/cm.sum(axis=1)[:,None], 2)
- confusionArray[3] = cm
-
- accScore = accuracy_score(yTest, yPred)
- confusionArray[4] = accScore
-
- AUC = roc_auc_score(yTest, yPred)
- confusionArray[5] = AUC
-
- if printOut:
- print('\n################### Testing ###############')
- print('\nTest Confusion matrix: \n', cm)
- print('\nTest Accuracy score: \n', accScore)
- print('\nTestAUC: \n', AUC)
-
- return confusionArray
-
-
-import matplotlib.pyplot as plt
-import seaborn
-import scikitplot as skplt
-
-seaborn.set(style="white", context="notebook", font_scale=1.5,
- rc={"axes.grid": True, "legend.frameon": False,
-"lines.markeredgewidth": 1.4, "lines.markersize": 10})
-seaborn.set_context("notebook", font_scale=1.5, rc={"lines.linewidth": 4.5})
-
-yPred = gridSearch.predict_proba(XTest)
-print(yTest.ravel().shape, yPred.shape)
-
-#skplt.metrics.plot_cumulative_gain(yTest.ravel(), yPred_onehot)
-skplt.metrics.plot_cumulative_gain(yTest.ravel(), yPred)
-
-defaults = sum(yTest == 1)
-total = len(yTest)
-defaultRate = defaults/total
-def bestCurve(defaults, total, defaultRate):
- x = np.linspace(0, 1, total)
-
- y1 = np.linspace(0, 1, defaults)
- y2 = np.ones(total-defaults)
- y3 = np.concatenate([y1,y2])
- return x, y3
-
-x, best = bestCurve(defaults=defaults, total=total, defaultRate=defaultRate)
-plt.plot(x, best)
-
-
-plt.show()
-
-\end{minted}
-
-% ------------------- end of main content ---------------
-
-\end{document}
-
diff --git a/doc/src/LogisticRegression/LogReg-reveal.html b/doc/src/LogisticRegression/LogReg-reveal.html
deleted file mode 100644
index d8a85ea5f..000000000
--- a/doc/src/LogisticRegression/LogReg-reveal.html
+++ /dev/null
@@ -1,1004 +0,0 @@
-
-
-
-
-
-
-
-Data Analysis and Machine Learning: Logistic Regression
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Data Analysis and Machine Learning: Logistic Regression
-
-
-
-
-
-Morten Hjorth-Jensen [1, 2]
-
-
-
-
-
-
[1] Department of Physics, University of Oslo
-
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-In linear regression our main interest was centered on learning the
-coefficients of a functional fit (say a polynomial) in order to be
-able to predict the response of a continuous variable on some unseen
-data. The fit to the continuous variable \( y_i \) is based on some
-independent variables \( \hat{x}_i \). Linear regression resulted in
-analytical expressions for standard ordinary Least Squares or Ridge
-regression (in terms of matrices to invert) for several quantities,
-ranging from the variance and thereby the confidence intervals of the
-parameters \( \hat{\beta} \) to the mean squared error. If we can invert
-the product of the design matrices, linear regression gives then a
-simple recipe for fitting our data.
-
-
-Classification problems, however, are concerned with outcomes taking
-the form of discrete variables (i.e. categories). We may for example,
-on the basis of DNA sequencing for a number of patients, like to find
-out which mutations are important for a certain disease; or based on
-scans of various patients' brains, figure out if there is a tumor or
-not; or given a specific physical system, we'd like to identify its
-state, say whether it is an ordered or disordered system (typical
-situation in solid state physics); or classify the status of a
-patient, whether she/he has a stroke or not and many other similar
-situations.
-
-
-The most common situation we encounter when we apply logistic
-regression is that of two possible outcomes, normally denoted as a
-binary outcome, true or false, positive or negative, success or
-failure etc.
-
-
-
-
-
Optimization and Deep learning
-
-
-Logistic regression will also serve as our stepping stone towards neural
-network algorithms and supervised deep learning. For logistic
-learning, the minimization of the cost function leads to a non-linear
-equation in the parameters \( \hat{\beta} \). The optimization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the family of gradient descent methods. The latter are the working horses of basically all modern machine learning algorithms.
-
-
-We note also that many of the topics discussed here
-regression are also commonly used in modern supervised Deep Learning
-models, as we will see later.
-
-
-
-
-
Basics
-
-
-We consider the case where the dependent variables, also called the
-responses or the outcomes, \( y_i \) are discrete and only take values
-from \( k=0,\dots,K-1 \) (i.e. \( K \) classes).
-
-
-The goal is to predict the
-output classes from the design matrix \( \hat{X}\in\mathbb{R}^{n\times p} \)
-made of \( n \) samples, each of which carries \( p \) features or predictors. The
-primary goal is to identify the classes to which new unseen samples
-belong.
-
-
-Let us specialize to the case of two classes only, with outputs
-\( y_i=0 \) and \( y_i=1 \). Our outcomes could represent the status of a
-credit card user that could default or not on her/his credit card
-debt. That is
-
-
-Before moving to the logistic model, let us try to use our linear regression model to classify these two outcomes. We could for example fit a linear model to the default case if \( y_i > 0.5 \) and the no default case \( y_i \leq 0.5 \).
-
-
-We would then have our
-weighted linear combination, namely
-
-
-where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our
-\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators/predictors.
-
-
-
-
-
Some selected properties
-
-
-The main problem with our function is that it
-takes values on the entire real axis. In the case of
-logistic regression, however, the labels \( y_i \) are discrete
-variables. A typical example is the credit card data discussed below here, where we can set the state of defaulting the debt to \( y_i=1 \) and not to \( y_i=0 \) for one the persons in the data set (see the full example below).
-
-
-One simple way to get a discrete output is to have sign
-functions that map the output of a linear regressor to values \( \{0,1\} \),
-\( f(s_i)=sign(s_i)=1 \) if \( s_i\ge 0 \) and 0 if otherwise.
-We will encounter this model in our first demonstration of neural networks. Historically it is called the "perceptron" model in the machine learning
-literature. This model is extremely simple. However, in many cases it is more
-favorable to use a ``soft" classifier that outputs
-the probability of a given category. This leads us to the logistic function.
-
-
-
-
-
The logistic function
-
-
-The perceptron is an example of a ``hard classification" model. We
-will encounter this model when we discuss neural networks as
-well. Each datapoint is deterministically assigned to a category (i.e
-\( y_i=0 \) or \( y_i=1 \)). In many cases, it is favorable to have a "soft"
-classifier that outputs the probability of a given category rather
-than a single value. For example, given \( x_i \), the classifier
-outputs the probability of being in a category \( k \). Logistic regression
-is the most common example of a so-called soft classifier. In logistic
-regression, the probability that a data point \( x_i \)
-belongs to a category \( y_i=\{0,1\} \) is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,
-
-
-Note that \( 1-p(t)= p(-t) \).
-The following code plots the logistic function, the step function and other functions we will encounter from here and on.
-
-
-
-
-
"""The sigmoid function (or the logistic curve) is a
-function that takes any real number, z, and outputs a number (0,1).
-It is useful in neural networks for assigning weights on a relative scale.
-The value z is the weighted sum of parameters involved in the learning algorithm."""
-
-importnumpy
-importmatplotlib.pyplotasplt
-importmathasmt
-
-z = numpy.arange(-5, 5, .1)
-sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))
-sigma = sigma_fn(z)
-
-fig = plt.figure()
-ax = fig.add_subplot(111)
-ax.plot(z, sigma)
-ax.set_ylim([-0.1, 1.1])
-ax.set_xlim([-5,5])
-ax.grid(True)
-ax.set_xlabel('z')
-ax.set_title('sigmoid function')
-
-plt.show()
-
-"""Step Function"""
-z = numpy.arange(-5, 5, .02)
-step_fn = numpy.vectorize(lambda z: 1.0if z >= 0.0else0.0)
-step = step_fn(z)
-
-fig = plt.figure()
-ax = fig.add_subplot(111)
-ax.plot(z, step)
-ax.set_ylim([-0.5, 1.5])
-ax.set_xlim([-5,5])
-ax.grid(True)
-ax.set_xlabel('z')
-ax.set_title('step function')
-
-plt.show()
-
-"""tanh Function"""
-z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)
-t = numpy.tanh(z)
-
-fig = plt.figure()
-ax = fig.add_subplot(111)
-ax.plot(z, t)
-ax.set_ylim([-1.0, 1.0])
-ax.set_xlim([-2*mt.pi,2*mt.pi])
-ax.grid(True)
-ax.set_xlabel('z')
-ax.set_title('tanh function')
-
-plt.show()
-
-
-
-
-
-
Two parameters
-
-
-We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities
-
-In order to define the total likelihood for all possible outcomes from a
-dataset \( \mathcal{D}=\{(y_i,x_i)\} \), with the binary labels
-\( y_i\in\{0,1\} \) and where the data points are drawn independently, we use the so-called Maximum Likelihood Estimation (MLE) principle.
-We aim thus at maximizing
-the probability of seeing the observed data. We can then approximate the
-likelihood in terms of the product of the individual probabilities of a specific outcome \( y_i \), that is
-
-The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \beta \).
-Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
-
-
-This equation is known in statistics as the cross entropy. Finally, we note that just as in linear regression,
-in practice we often supplement the cross-entropy with additional regularization terms, usually \( L_1 \) and \( L_2 \) regularization as we did for Ridge and Lasso regression.
-
-
-
-
-
Minimizing the cross entropy
-
-
-The cross entropy is a convex function of the weights \( \hat{\beta} \) and,
-therefore, any local minimizer is a global minimizer.
-
-
-Minimizing this
-cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \) we obtain
-
-
-Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an
-\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a
-vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first
-derivative of cost function as
-
-
-If we in addition define a diagonal matrix \( \hat{W} \) with elements
-\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as
-
-
-Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors
-
-Till now we have mainly focused on two classes, the so-called binary
-system. Suppose we wish to extend to \( K \) classes. Let us for the sake
-of simplicity assume we have only two predictors. We have then
-following model
-
-
-and the model is specified in term of \( K-1 \) so-called log-odds or
-logit transformations.
-
-
-
-
-
The Softmax function
-
-
-In our discussion of neural networks we will encounter the above again
-in terms of the so-called Softmax function.
-
-
-The softmax function is used in various multiclass classification
-methods, such as multinomial logistic regression (also known as
-softmax regression), multiclass linear discriminant analysis, naive
-Bayes classifiers, and artificial neural networks. Specifically, in
-multinomial logistic regression and linear discriminant analysis, the
-input to the function is the result of \( K \) distinct linear functions,
-and the predicted probability for the \( k \)-th class given a sample
-vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two
-predictors):
-
-
-and they sum to one. Our earlier discussions were all specialized to
-the case with two classes only. It is easy to see from the above that
-what we derived earlier is compatible with these equations.
-
-
-To find the optimal parameters we would typically use a gradient
-descent method. Newton's method and gradient descent methods are
-discussed in the material on optimization
-methods.
-
-
-
-
-
A simple classification problem
-
-
-
-
importnumpyasnp
-fromsklearnimport datasets, linear_model
-importmatplotlib.pyplotasplt
-
-
-defgenerate_data():
- np.random.seed(0)
- X, y = datasets.make_moons(200, noise=0.20)
- return X, y
-
-
-defvisualize(X, y, clf):
- plot_decision_boundary(lambda x: clf.predict(x), X, y)
-
-defplot_decision_boundary(pred_func, X, y):
- # Set min and max values and give it some padding
- x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
- y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
- h = 0.01
- # Generate a grid of points with distance h between them
- xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
- # Predict the function value for the whole gid
- Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
- Z = Z.reshape(xx.shape)
- # Plot the contour and training examples
- plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
- plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
- plt.show()
-
-
-defclassify(X, y):
- clf = linear_model.LogisticRegressionCV()
- clf.fit(X, y)
- return clf
-
-
-defmain():
- X, y = generate_data()
- # visualize(X, y)
- clf = classify(X, y)
- visualize(X, y, clf)
-
-if__name__ == "__main__":
- main()
-