new update

doc/LectureNotes/DataFiles/cancer.dot

@@ -6,17 +6,17 @@ edge [fontname="helvetica"] ;
 0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ;
 2 [label="worst concave points <= 0.135\ngini = 0.031\nsamples = 253\nvalue = [[249, 4]\n[4, 249]]", fillcolor="#e78946"] ;
 1 -> 2 ;
-3 [label="area error <= 48.975\ngini = 0.008\nsamples = 242\nvalue = [[241, 1]\n[1, 241]]", fillcolor="#e5833c"] ;
+3 [label="radius error <= 0.643\ngini = 0.008\nsamples = 242\nvalue = [[241, 1]\n[1, 241]]", fillcolor="#e5833c"] ;
 2 -> 3 ;
 4 [label="gini = 0.0\nsamples = 239\nvalue = [[239, 0]\n[0, 239]]", fillcolor="#e58139"] ;
 3 -> 4 ;
-5 [label="area error <= 51.38\ngini = 0.444\nsamples = 3\nvalue = [[2, 1]\n[1, 2]]", fillcolor="#fdf6f0"] ;
+5 [label="mean perimeter <= 78.51\ngini = 0.444\nsamples = 3\nvalue = [[2, 1]\n[1, 2]]", fillcolor="#fdf6f0"] ;
 3 -> 5 ;
 6 [label="gini = 0.0\nsamples = 1\nvalue = [[0, 1]\n[1, 0]]", fillcolor="#e58139"] ;
 5 -> 6 ;
 7 [label="gini = 0.0\nsamples = 2\nvalue = [[2, 0]\n[0, 2]]", fillcolor="#e58139"] ;
 5 -> 7 ;
-8 [label="worst texture <= 29.455\ngini = 0.397\nsamples = 11\nvalue = [[8, 3]\n[3, 8]]", fillcolor="#fae9dd"] ;
+8 [label="mean texture <= 20.84\ngini = 0.397\nsamples = 11\nvalue = [[8, 3]\n[3, 8]]", fillcolor="#fae9dd"] ;
 2 -> 8 ;
 9 [label="gini = 0.0\nsamples = 8\nvalue = [[8, 0]\n[0, 8]]", fillcolor="#e58139"] ;
 8 -> 9 ;
@@ -34,7 +34,7 @@ edge [fontname="helvetica"] ;
 14 -> 15 ;
 16 [label="gini = 0.0\nsamples = 11\nvalue = [[11, 0]\n[0, 11]]", fillcolor="#e58139"] ;
 15 -> 16 ;
-17 [label="mean concavity <= 0.07\ngini = 0.32\nsamples = 5\nvalue = [[1, 4]\n[4, 1]]", fillcolor="#f6d5bd"] ;
+17 [label="worst perimeter <= 115.95\ngini = 0.32\nsamples = 5\nvalue = [[1, 4]\n[4, 1]]", fillcolor="#f6d5bd"] ;
 15 -> 17 ;
 18 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139"] ;
 17 -> 18 ;
@@ -48,10 +48,10 @@ edge [fontname="helvetica"] ;
 21 -> 22 ;
 23 [label="gini = 0.0\nsamples = 6\nvalue = [[6, 0]\n[0, 6]]", fillcolor="#e58139"] ;
 21 -> 23 ;
-24 [label="mean smoothness <= 0.079\ngini = 0.015\nsamples = 136\nvalue = [[1, 135]\n[135, 1]]", fillcolor="#e6853f"] ;
+24 [label="fractal dimension error <= 0.013\ngini = 0.015\nsamples = 136\nvalue = [[1, 135]\n[135, 1]]", fillcolor="#e6853f"] ;
 20 -> 24 ;
-25 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139"] ;
+25 [label="gini = 0.0\nsamples = 135\nvalue = [[0, 135]\n[135, 0]]", fillcolor="#e58139"] ;
 24 -> 25 ;
-26 [label="gini = 0.0\nsamples = 135\nvalue = [[0, 135]\n[135, 0]]", fillcolor="#e58139"] ;
+26 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139"] ;
 24 -> 26 ;
 }

doc/LectureNotes/_build/.doctrees/glue_cache.json

@@ -0,0 +1 @@
+{}

doc/LectureNotes/_build/html/.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 2fd3d019b83b9c58018e90b52a9aa1b0
+tags: 645f666f9bcd5a90fca523b33c5a78b7

doc/LectureNotes/_build/html/_sources/chapter4.ipynb

@@ -0,0 +1,912 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Logistic Regression\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Logistic Regression\n",
+    "\n",
+    "In linear regression our main interest was centered on learning the\n",
+    "coefficients of a functional fit (say a polynomial) in order to be\n",
+    "able to predict the response of a continuous variable on some unseen\n",
+    "data. The fit to the continuous variable $y_i$ is based on some\n",
+    "independent variables $x_i$. Linear regression resulted in\n",
+    "analytical expressions for standard ordinary Least Squares or Ridge\n",
+    "regression (in terms of matrices to invert) for several quantities,\n",
+    "ranging from the variance and thereby the confidence intervals of the\n",
+    "optimal parameters $\\hat{\\beta}$ to the mean squared error. If we can invert\n",
+    "the product of the design matrices, linear regression gives then a\n",
+    "simple recipe for fitting our data.\n",
+    "\n",
+    "\n",
+    "Classification problems, however, are concerned with outcomes taking\n",
+    "the form of discrete variables (i.e. categories). We may for example,\n",
+    "on the basis of DNA sequencing for a number of patients, like to find\n",
+    "out which mutations are important for a certain disease; or based on\n",
+    "scans of various patients' brains, figure out if there is a tumor or\n",
+    "not; or given a specific physical system, we'd like to identify its\n",
+    "state, say whether it is an ordered or disordered system (typical\n",
+    "situation in solid state physics); or classify the status of a\n",
+    "patient, whether she/he has a stroke or not and many other similar\n",
+    "situations.\n",
+    "\n",
+    "The most common situation we encounter when we apply logistic\n",
+    "regression is that of two possible outcomes, normally denoted as a\n",
+    "binary outcome, true or false, positive or negative, success or\n",
+    "failure etc.\n",
+    "\n",
+    "\n",
+    "Logistic regression will also serve as our stepping stone towards\n",
+    "neural network algorithms and supervised deep learning. For logistic\n",
+    "learning, the minimization of the cost function leads to a non-linear\n",
+    "equation in the parameters $\\hat{\\beta}$. The optimization of the\n",
+    "problem calls therefore for minimization algorithms. This forms the\n",
+    "bottle neck of all machine learning algorithms, namely how to find\n",
+    "reliable minima of a multi-variable function. This leads us to the\n",
+    "family of gradient descent methods. The latter are the working horses\n",
+    "of basically all modern machine learning algorithms.\n",
+    "\n",
+    "We note also that many of the topics discussed here on logistic \n",
+    "regression are also commonly used in modern supervised Deep Learning\n",
+    "models, as we will see later.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Basics\n",
+    "\n",
+    "We consider the case where the dependent variables, also called the\n",
+    "responses or the outcomes, $y_i$ are discrete and only take values\n",
+    "from $k=0,\\dots,K-1$ (i.e. $K$ classes).\n",
+    "\n",
+    "The goal is to predict the\n",
+    "output classes from the design matrix $\\boldsymbol{X}\\in\\mathbb{R}^{n\\times p}$\n",
+    "made of $n$ samples, each of which carries $p$ features or predictors. The\n",
+    "primary goal is to identify the classes to which new unseen samples\n",
+    "belong.\n",
+    "\n",
+    "Let us specialize to the case of two classes only, with outputs\n",
+    "$y_i=0$ and $y_i=1$. Our outcomes could represent the status of a\n",
+    "credit card user that could default or not on her/his credit card\n",
+    "debt. That is"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "y_i = \\begin{bmatrix} 0 & \\mathrm{no}\\\\  1 & \\mathrm{yes} \\end{bmatrix}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Before moving to the logistic model, let us try to use our linear\n",
+    "regression model to classify these two outcomes. We could for example\n",
+    "fit a linear model to the default case if $y_i > 0.5$ and the no\n",
+    "default case $y_i \\leq 0.5$.\n",
+    "\n",
+    "We would then have our \n",
+    "weighted linear combination, namely"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<!-- Equation labels as ordinary links -->\n",
+    "<div id=\"_auto1\"></div>\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    "\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{\\beta} +  \\boldsymbol{\\epsilon},\n",
+    "\\label{_auto1} \\tag{1}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n",
+    "$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors.\n",
+    "\n",
+    "\n",
+    "The main problem with our function is that it takes values on the\n",
+    "entire real axis. In the case of logistic regression, however, the\n",
+    "labels $y_i$ are discrete variables. A typical example is the credit\n",
+    "card data discussed below here, where we can set the state of\n",
+    "defaulting the debt to $y_i=1$ and not to $y_i=0$ for one the persons\n",
+    "in the data set (see the full example below).\n",
+    "\n",
+    "One simple way to get a discrete output is to have sign\n",
+    "functions that map the output of a linear regressor to values $\\{0,1\\}$,\n",
+    "$f(s_i)=sign(s_i)=1$ if $s_i\\ge 0$ and 0 if otherwise. \n",
+    "We will encounter this model in our first demonstration of neural networks. Historically it is called the ``perceptron\" model in the machine learning\n",
+    "literature. This model is extremely simple. However, in many cases it is more\n",
+    "favorable to use a ``soft\" classifier that outputs\n",
+    "the probability of a given category. This leads us to the logistic function.\n",
+    "\n",
+    "\n",
+    "The following example on data for coronary heart disease (CHD) as function of age may serve as an illustration. In the code here we read and plot whether a person has had CHD (output = 1) or not (output = 0). This ouput  is plotted the person's against age. Clearly, the figure shows that attempting to make a standard linear regression fit may not be very meaningful."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "%matplotlib inline\n",
+    "\n",
+    "# Common imports\n",
+    "import os\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "import matplotlib.pyplot as plt\n",
+    "from sklearn.linear_model import LinearRegression, Ridge, Lasso\n",
+    "from sklearn.model_selection import train_test_split\n",
+    "from sklearn.utils import resample\n",
+    "from sklearn.metrics import mean_squared_error\n",
+    "from IPython.display import display\n",
+    "from pylab import plt, mpl\n",
+    "plt.style.use('seaborn')\n",
+    "mpl.rcParams['font.family'] = 'serif'\n",
+    "\n",
+    "# Where to save the figures and data files\n",
+    "PROJECT_ROOT_DIR = \"Results\"\n",
+    "FIGURE_ID = \"Results/FigureFiles\"\n",
+    "DATA_ID = \"DataFiles/\"\n",
+    "\n",
+    "if not os.path.exists(PROJECT_ROOT_DIR):\n",
+    "    os.mkdir(PROJECT_ROOT_DIR)\n",
+    "\n",
+    "if not os.path.exists(FIGURE_ID):\n",
+    "    os.makedirs(FIGURE_ID)\n",
+    "\n",
+    "if not os.path.exists(DATA_ID):\n",
+    "    os.makedirs(DATA_ID)\n",
+    "\n",
+    "def image_path(fig_id):\n",
+    "    return os.path.join(FIGURE_ID, fig_id)\n",
+    "\n",
+    "def data_path(dat_id):\n",
+    "    return os.path.join(DATA_ID, dat_id)\n",
+    "\n",
+    "def save_fig(fig_id):\n",
+    "    plt.savefig(image_path(fig_id) + \".png\", format='png')\n",
+    "\n",
+    "infile = open(data_path(\"chddata.csv\"),'r')\n",
+    "\n",
+    "# Read the chd data as  csv file and organize the data into arrays with age group, age, and chd\n",
+    "chd = pd.read_csv(infile, names=('ID', 'Age', 'Agegroup', 'CHD'))\n",
+    "chd.columns = ['ID', 'Age', 'Agegroup', 'CHD']\n",
+    "output = chd['CHD']\n",
+    "age = chd['Age']\n",
+    "agegroup = chd['Agegroup']\n",
+    "numberID  = chd['ID'] \n",
+    "display(chd)\n",
+    "\n",
+    "plt.scatter(age, output, marker='o')\n",
+    "plt.axis([18,70.0,-0.1, 1.2])\n",
+    "plt.xlabel(r'Age')\n",
+    "plt.ylabel(r'CHD')\n",
+    "plt.title(r'Age distribution and Coronary heart disease')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "What we could attempt however is to plot the mean value for each group."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "agegroupmean = np.array([0.1, 0.133, 0.250, 0.333, 0.462, 0.625, 0.765, 0.800])\n",
+    "group = np.array([1, 2, 3, 4, 5, 6, 7, 8])\n",
+    "plt.plot(group, agegroupmean, \"r-\")\n",
+    "plt.axis([0,9,0, 1.0])\n",
+    "plt.xlabel(r'Age group')\n",
+    "plt.ylabel(r'CHD mean values')\n",
+    "plt.title(r'Mean values for each age group')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We are now trying to find a function $f(y\\vert x)$, that is a function which gives us an expected value for the output $y$ with a given input $x$.\n",
+    "In standard linear regression with a linear dependence on $x$, we would write this in terms of our model"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "f(y_i\\vert x_i)=\\beta_0+\\beta_1 x_i.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This expression implies however that $f(y_i\\vert x_i)$ could take any\n",
+    "value from minus infinity to plus infinity. If we however let\n",
+    "$f(y\\vert y)$ be represented by the mean value, the above example\n",
+    "shows us that we can constrain the function to take values between\n",
+    "zero and one, that is we have $0 \\le f(y_i\\vert x_i) \\le 1$. Looking\n",
+    "at our last curve we see also that it has an S-shaped form. This leads\n",
+    "us to a very popular model for the function $f$, namely the so-called\n",
+    "Sigmoid function or logistic model. We will consider this function as\n",
+    "representing the probability for finding a value of $y_i$ with a given\n",
+    "$x_i$.\n",
+    "\n",
+    "\n",
+    "## The logistic function\n",
+    "\n",
+    "Another widely studied model, is the so-called \n",
+    "perceptron model, which is an example of a \"hard classification\" model. We\n",
+    "will encounter this model when we discuss neural networks as\n",
+    "well. Each datapoint is deterministically assigned to a category (i.e\n",
+    "$y_i=0$ or $y_i=1$). In many cases, and the coronary heart disease data forms one of many such examples, it is favorable to have a \"soft\"\n",
+    "classifier that outputs the probability of a given category rather\n",
+    "than a single value. For example, given $x_i$, the classifier\n",
+    "outputs the probability of being in a category $k$.  Logistic regression\n",
+    "is the most common example of a so-called soft classifier. In logistic\n",
+    "regression, the probability that a data point $x_i$\n",
+    "belongs to a category $y_i=\\{0,1\\}$ is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "p(t) = \\frac{1}{1+\\mathrm \\exp{-t}}=\\frac{\\exp{t}}{1+\\mathrm \\exp{t}}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note that $1-p(t)= p(-t)$.\n",
+    "\n",
+    "## Examples of likelihood functions used in logistic regression and nueral networks\n",
+    "\n",
+    "\n",
+    "The following code plots the logistic function, the step function and other functions we will encounter from here and on."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "\"\"\"The sigmoid function (or the logistic curve) is a\n",
+    "function that takes any real number, z, and outputs a number (0,1).\n",
+    "It is useful in neural networks for assigning weights on a relative scale.\n",
+    "The value z is the weighted sum of parameters involved in the learning algorithm.\"\"\"\n",
+    "\n",
+    "import numpy\n",
+    "import matplotlib.pyplot as plt\n",
+    "import math as mt\n",
+    "\n",
+    "z = numpy.arange(-5, 5, .1)\n",
+    "sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))\n",
+    "sigma = sigma_fn(z)\n",
+    "\n",
+    "fig = plt.figure()\n",
+    "ax = fig.add_subplot(111)\n",
+    "ax.plot(z, sigma)\n",
+    "ax.set_ylim([-0.1, 1.1])\n",
+    "ax.set_xlim([-5,5])\n",
+    "ax.grid(True)\n",
+    "ax.set_xlabel('z')\n",
+    "ax.set_title('sigmoid function')\n",
+    "\n",
+    "plt.show()\n",
+    "\n",
+    "\"\"\"Step Function\"\"\"\n",
+    "z = numpy.arange(-5, 5, .02)\n",
+    "step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)\n",
+    "step = step_fn(z)\n",
+    "\n",
+    "fig = plt.figure()\n",
+    "ax = fig.add_subplot(111)\n",
+    "ax.plot(z, step)\n",
+    "ax.set_ylim([-0.5, 1.5])\n",
+    "ax.set_xlim([-5,5])\n",
+    "ax.grid(True)\n",
+    "ax.set_xlabel('z')\n",
+    "ax.set_title('step function')\n",
+    "\n",
+    "plt.show()\n",
+    "\n",
+    "\"\"\"tanh Function\"\"\"\n",
+    "z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)\n",
+    "t = numpy.tanh(z)\n",
+    "\n",
+    "fig = plt.figure()\n",
+    "ax = fig.add_subplot(111)\n",
+    "ax.plot(z, t)\n",
+    "ax.set_ylim([-1.0, 1.0])\n",
+    "ax.set_xlim([-2*mt.pi,2*mt.pi])\n",
+    "ax.grid(True)\n",
+    "ax.set_xlabel('z')\n",
+    "ax.set_title('tanh function')\n",
+    "\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\\beta$ in our fitting of the Sigmoid function, that is we define probabilities"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\begin{align*}\n",
+    "p(y_i=1|x_i,\\boldsymbol{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n",
+    "p(y_i=0|x_i,\\boldsymbol{\\beta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\beta}),\n",
+    "\\end{align*}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
+    "\n",
+    "Note that we used"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In order to define the total likelihood for all possible outcomes from a  \n",
+    "dataset $\\mathcal{D}=\\{(y_i,x_i)\\}$, with the binary labels\n",
+    "$y_i\\in\\{0,1\\}$ and where the data points are drawn independently, we use the so-called [Maximum Likelihood Estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation) (MLE) principle. \n",
+    "We aim thus at maximizing \n",
+    "the probability of seeing the observed data. We can then approximate the \n",
+    "likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\begin{align*}\n",
+    "P(\\mathcal{D}|\\boldsymbol{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\boldsymbol{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n",
+    "\\end{align*}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "from which we obtain the log-likelihood and our **cost/loss** function"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Reordering the logarithms, we can rewrite the **cost/loss** function as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n  \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n",
+    "Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n  \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n",
+    "in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.\n",
+    "\n",
+    "\n",
+    "The cross entropy is a convex function of the weights $\\boldsymbol{\\beta}$ and,\n",
+    "therefore, any local minimizer is a global minimizer. \n",
+    "\n",
+    "\n",
+    "Minimizing this\n",
+    "cost function with respect to the two parameters $\\beta_0$ and $\\beta_1$ we obtain"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n  \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n  \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let us now define a vector $\\boldsymbol{y}$ with $n$ elements $y_i$, an\n",
+    "$n\\times p$ matrix $\\boldsymbol{X}$ which contains the $x_i$ values and a\n",
+    "vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\beta})$. We can rewrite in a more compact form the first\n",
+    "derivative of cost function as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n",
+    "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Till now we have mainly focused on two classes, the so-called binary\n",
+    "system. Suppose we wish to extend to $K$ classes.  Let us for the sake\n",
+    "of simplicity assume we have only two predictors. We have then following model"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\beta_{10}+\\beta_{11}x_1,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and so on till the class $C=K-1$ class"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and the model is specified in term of $K-1$ so-called log-odds or\n",
+    "**logit** transformations.\n",
+    "\n",
+    "\n",
+    "\n",
+    "In our discussion of neural networks we will encounter the above again\n",
+    "in terms of a slightly modified function, the so-called **Softmax** function.\n",
+    "\n",
+    "The softmax function is used in various multiclass classification\n",
+    "methods, such as multinomial logistic regression (also known as\n",
+    "softmax regression), multiclass linear discriminant analysis, naive\n",
+    "Bayes classifiers, and artificial neural networks.  Specifically, in\n",
+    "multinomial logistic regression and linear discriminant analysis, the\n",
+    "input to the function is the result of $K$ distinct linear functions,\n",
+    "and the predicted probability for the $k$-th class given a sample\n",
+    "vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\beta}$ is (with two\n",
+    "predictors):"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It is easy to extend to more predictors. The final class is"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and they sum to one. Our earlier discussions were all specialized to\n",
+    "the case with two classes only. It is easy to see from the above that\n",
+    "what we derived earlier is compatible with these equations.\n",
+    "\n",
+    "To find the optimal parameters we would typically use a gradient\n",
+    "descent method.  Newton's method and gradient descent methods are\n",
+    "discussed in the material on [optimization\n",
+    "methods](https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html).\n",
+    "\n",
+    "## Wisconsin Cancer Data\n",
+    "\n",
+    "We show here how we can use a simple regression case on the breast\n",
+    "cancer data using Logistic regression as our algorithm for\n",
+    "classification."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "from sklearn.model_selection import  train_test_split \n",
+    "from sklearn.datasets import load_breast_cancer\n",
+    "from sklearn.linear_model import LogisticRegression\n",
+    "\n",
+    "# Load the data\n",
+    "cancer = load_breast_cancer()\n",
+    "\n",
+    "X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)\n",
+    "print(X_train.shape)\n",
+    "print(X_test.shape)\n",
+    "# Logistic Regression\n",
+    "logreg = LogisticRegression(solver='lbfgs')\n",
+    "logreg.fit(X_train, y_train)\n",
+    "print(\"Test set accuracy with Logistic Regression: {:.2f}\".format(logreg.score(X_test,y_test)))\n",
+    "#now scale the data\n",
+    "from sklearn.preprocessing import StandardScaler\n",
+    "scaler = StandardScaler()\n",
+    "scaler.fit(X_train)\n",
+    "X_train_scaled = scaler.transform(X_train)\n",
+    "X_test_scaled = scaler.transform(X_test)\n",
+    "# Logistic Regression\n",
+    "logreg.fit(X_train_scaled, y_train)\n",
+    "print(\"Test set accuracy Logistic Regression with scaled data: {:.2f}\".format(logreg.score(X_test_scaled,y_test)))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In addition to the above scores, we could also study the covariance (and the correlation matrix).\n",
+    "We use **Pandas** to compute the correlation matrix."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "from sklearn.model_selection import  train_test_split \n",
+    "from sklearn.datasets import load_breast_cancer\n",
+    "from sklearn.linear_model import LogisticRegression\n",
+    "cancer = load_breast_cancer()\n",
+    "import pandas as pd\n",
+    "# Making a data frame\n",
+    "cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)\n",
+    "\n",
+    "fig, axes = plt.subplots(15,2,figsize=(10,20))\n",
+    "malignant = cancer.data[cancer.target == 0]\n",
+    "benign = cancer.data[cancer.target == 1]\n",
+    "ax = axes.ravel()\n",
+    "\n",
+    "for i in range(30):\n",
+    "    _, bins = np.histogram(cancer.data[:,i], bins =50)\n",
+    "    ax[i].hist(malignant[:,i], bins = bins, alpha = 0.5)\n",
+    "    ax[i].hist(benign[:,i], bins = bins, alpha = 0.5)\n",
+    "    ax[i].set_title(cancer.feature_names[i])\n",
+    "    ax[i].set_yticks(())\n",
+    "ax[0].set_xlabel(\"Feature magnitude\")\n",
+    "ax[0].set_ylabel(\"Frequency\")\n",
+    "ax[0].legend([\"Malignant\", \"Benign\"], loc =\"best\")\n",
+    "fig.tight_layout()\n",
+    "plt.show()\n",
+    "\n",
+    "import seaborn as sns\n",
+    "correlation_matrix = cancerpd.corr().round(1)\n",
+    "# use the heatmap function from seaborn to plot the correlation matrix\n",
+    "# annot = True to print the values inside the square\n",
+    "plt.figure(figsize=(15,8))\n",
+    "sns.heatmap(data=correlation_matrix, annot=True)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In the above example we note two things. In the first plot we display\n",
+    "the overlap of benign and malignant tumors as functions of the various\n",
+    "features in the Wisconsing breast cancer data set. We see that for\n",
+    "some of the features we can distinguish clearly the benign and\n",
+    "malignant cases while for other features we cannot. This can point to\n",
+    "us which features may be of greater interest when we wish to classify\n",
+    "a benign or not benign tumour.\n",
+    "\n",
+    "In the second figure we have computed the so-called correlation\n",
+    "matrix, which in our case with thirty features becomes a $30\\times 30$\n",
+    "matrix.\n",
+    "\n",
+    "We constructed this matrix using **pandas** via the statements"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and then"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "correlation_matrix = cancerpd.corr().round(1)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Diagonalizing this matrix we can in turn say something about which\n",
+    "features are of relevance and which are not. This leads  us to\n",
+    "the classical Principal Component Analysis (PCA) theorem with\n",
+    "applications. This will be discussed later this semester ([week 43](https://compphysics.github.io/MachineLearning/doc/pub/week43/html/week43-bs.html))."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "from sklearn.model_selection import  train_test_split \n",
+    "from sklearn.datasets import load_breast_cancer\n",
+    "from sklearn.linear_model import LogisticRegression\n",
+    "\n",
+    "# Load the data\n",
+    "cancer = load_breast_cancer()\n",
+    "\n",
+    "X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)\n",
+    "print(X_train.shape)\n",
+    "print(X_test.shape)\n",
+    "# Logistic Regression\n",
+    "logreg = LogisticRegression(solver='lbfgs')\n",
+    "logreg.fit(X_train, y_train)\n",
+    "print(\"Test set accuracy with Logistic Regression: {:.2f}\".format(logreg.score(X_test,y_test)))\n",
+    "#now scale the data\n",
+    "from sklearn.preprocessing import StandardScaler\n",
+    "scaler = StandardScaler()\n",
+    "scaler.fit(X_train)\n",
+    "X_train_scaled = scaler.transform(X_train)\n",
+    "X_test_scaled = scaler.transform(X_test)\n",
+    "# Logistic Regression\n",
+    "logreg.fit(X_train_scaled, y_train)\n",
+    "print(\"Test set accuracy Logistic Regression with scaled data: {:.2f}\".format(logreg.score(X_test_scaled,y_test)))\n",
+    "\n",
+    "\n",
+    "from sklearn.preprocessing import LabelEncoder\n",
+    "from sklearn.model_selection import cross_validate\n",
+    "#Cross validation\n",
+    "accuracy = cross_validate(logreg,X_test_scaled,y_test,cv=10)['test_score']\n",
+    "print(accuracy)\n",
+    "print(\"Test set accuracy with Logistic Regression  and scaled data: {:.2f}\".format(logreg.score(X_test_scaled,y_test)))\n",
+    "\n",
+    "\n",
+    "import scikitplot as skplt\n",
+    "y_pred = logreg.predict(X_test_scaled)\n",
+    "skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)\n",
+    "plt.show()\n",
+    "y_probas = logreg.predict_proba(X_test_scaled)\n",
+    "skplt.metrics.plot_roc(y_test, y_probas)\n",
+    "plt.show()\n",
+    "skplt.metrics.plot_cumulative_gain(y_test, y_probas)\n",
+    "plt.show()"
+   ]
+  }
+ ],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 4
+}