-
@@ -146,6 +155,8 @@ MathJax.Hub.Config({
-So far we have focused on learning from datasets for which there is a -continuous output. In linear regression we have been -concerned with learning the coefficients of a polynomial to predict -the response of a continuous variable \( y_i \) on unseen data based on -its independent variables \( {\bf x}_i \). +In linear regression our main interest was centered on learning the +coefficients of a functional fit (say a polynomial) in order to be +able to predict the response of a continuous variable on some unseen +data. The fit to the continuous variable \( y_i \) is based on some +independent variables \( \hat{x}_i \). Linear regression resulted in +analytical expressions (in terms of matrices to invert) for several +quantities, ranging from the variance and thereby the confidence +intervals of the parameters \( \hat{\beta} \) to the mean squared +error. If we can invert the product of the design matrices, linear +regression gives then a simple recipe for fitting our data.
-Classification problems, -however, are concerned with outcomes taking the form of discrete -variables (i.e. categories). For example, we may want to detect if -there's a cat or a dog in an image. Or given a specific system, -we'd like to identify its state, say whether it is an ordered or disordered system (typical situation in solid state physics). -(e.g. ordered/disordered). +Classification problems, however, are concerned with outcomes taking +the form of discrete variables (i.e. categories). We may for example, +on the basis of DNA sequencing for a number of patients, like to find +out which mutations are important for a certain disease; or based on +scans of various patients' brains, figure out if there is a tumor or +not; or given a specific physical system, we'd like to identify its +state, say whether it is an ordered or disordered system (typical +situation in solid state physics); or classify the status of a +patient, whether she/he has a stroke or not and many other similar +situations.
-Logistic regression deals with binary, dichotomous outcomes (e.g. True or -False, Success or Failure, etc.). It is worth noting that logistic -regression is also commonly used in modern supervised Deep Learning -models, as we will see later. +The most common situation we encounter when we apply logistic +regression is that of two possible outcomes, normally denoted as a +binary outcome, true or false, positive or negative, success or +failure etc.
@@ -152,6 +170,9 @@ models, as we will see later.
- + -
-We consider the case where the dependent variables \( y_i\in\mathbb{Z} \) -are discrete and only take values from \( m=0,\dots,M-1 \) (i.e. \( M \) -classes). +Logistic regression will also serve as our stepping towards neural +network algorithms and supervised deep learning. For logistic +learning, the minimization of the cost function leads to a non-linear +equation in the parameters \( \hat{\beta} \). The optmization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the family of gradient descent methods. The latter are the working horses of basically all modern machine learning algorithms.
-The goal is to predict the -output classes from the design matrix \( X\in\mathbb{R}^{n\times p} \) -made of \( n \) samples, each of which bears \( p \) features. The -primary goal is to identify the classes to which new unseen samples -belong. +We note also that many of the topics discussed here +regression are also commonly used in modern supervised Deep Learning +models, as we will see later.
@@ -140,6 +148,10 @@ belong.
- + -
-Let us start by considering a slightly simpler classifier: a linear classifier that categorizes examples using a weighted linear-combination of the features and an additive offset -$$ -\begin{equation} -s_i = \boldsymbol{x}_i^T\boldsymbol{w} + b_0 \equiv \mathbf{x}_i^T\mathbf{w}, -\tag{1} -\end{equation} -$$ +We consider the case where the dependent variables, also called the +responses or the outcomes) \( y_i \) are discrete and only take values +from \( k=0,\dots,K-1 \) (i.e. \( K \) classes). -where we use the short-hand notation -\( \mathbf{x}_i = (1,\boldsymbol{x}_i) \) and \( \mathbf{w}_i = (b_0,\boldsymbol{w}_i) \). +
+The goal is to predict the +output classes from the design matrix \( X\in\mathbb{R}^{n\times p} \) +made of \( n \) samples, each of which carries \( p \) features. The +primary goal is to identify the classes to which new unseen samples +belong. + +
+Let us specialize to the case of two classes only, with outputs \( y_i=0 \) and \( y_i=1 \). Our outcomes could represent the status of a credit card use who could default or not on her/his debt. That is +$$ +y_i = \begin{bmatrix} 0 & \mathrm{no default}\\ 1 & \mathrm{default} \end{bmatrix}. +$$
@@ -140,6 +155,11 @@ where we use the short-hand notation
-This function takes values on the entire real axis. In the case of -logistic regression, however, the labels \( y_i \) are discrete -variables. One simple way to get a discrete output is to have sign -functions that map the output of a linear regressor to \( \{0,1\} \), -\( f(s_i)=sign(s_i)=1 \) if \( s_i\ge 0 \) and 0 if otherwise. Indeed, -this is commonly known as the "perceptron" in the machine learning -literature. This model is extremely simple, and it is favorable in -many cases (e.g. noisy data) to have a ``soft" classifier that outputs -the probability of a given category. For example, given -\( \mathbf{x}_i \), the classifier outputs the probability of being in -category \( m \). One such function is the logistic (or sigmoid) function: +Before moving to the logistic model, let us try to use our linear regression model to classify these two outcomes. We could for example fit a linear model to the default case if \( y_i > 0.5 \) and the no default case \( y_i \leq 0.5 \). +
+We would then have our +weighted linear combination, namely $$ \begin{equation} -f(s) = \frac{1}{1+\mathrm e^{-s}}. -\tag{2} -\end{equation} +\hat{y} = \hat{X}_i^T\hat{\beta} + \hat{\epsilon}, +\tag{1} +\end{equation} $$ -Note that \( 1-f(s)= f(-s) \), which will be useful shortly. +where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our +\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators.
@@ -150,6 +153,12 @@ Note that \( 1-f(s)= f(-s) \), which will be useful shortly.
-The perceptron is an example of a ``hard classification": each datapoint is deterministically assigned to a category (i.e \( y_i=0 \) or \( y_i=1 \)). In many cases, it is favorable to have a "soft" classifier that outputs the probability of a given category rather than a single value. For example, given \( \mathbf{x}_i \), the classifier outputs the probability of being in category \( m \). -Logistic regression is the most canonical example of a soft classifier. In logistic regression, the probability that a data point \( \boldsymbol{x}_i \) belongs to a category \( y_i=\{0,1\} \) is is given by -$$ -\begin{eqnarray} -P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& \frac{1}{1+\mathrm{e}^{-\mathbf{x}^T_i\mathbf{w}}},\nonumber\\ -P(y_i=0|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& 1 - P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)}, -\end{eqnarray} -$$ - -where \( \boldsymbol{\theta}=\mathbf{w} \) are the weights we wish to learn from the data. +The main problem with our function is that it +takes values on the entire real axis. In the case of +logistic regression, however, the labels \( y_i \) are discrete +variables.
-Notice that in terms of the logistic function, we can write -$$ -P(y_i=1) =f(\mathbf{x}_i^T\mathbf{w})=1-P(y_i=0). -$$ +One simple way to get a discrete output is to have sign +functions that map the output of a linear regressor to values \( \{0,1\} \), +\( f(s_i)=sign(s_i)=1 \) if \( s_i\ge 0 \) and 0 if otherwise. +We will encounter this model in our first demonstration of neural networks. Historically it is called the "perceptron" model in the machine learning +literature. This model is extremely simple. However, in many cases it is more +favorable to use a ``soft" classifier that outputs +the probability of a given category. This leads us to the logistic function.
@@ -146,6 +152,13 @@ $$
- + -
-We now define the cost function for logistic regression using Maximum -Likelihood Estimation (MLE). Recall, that in MLE we choose parameters -to maximize the probability of seeing the observed data. Consider a -dataset \( \mathcal{D}=\{(y_i,\boldsymbol{x}_i)\} \) with binary labels -\( y_i\in\{0,1\} \) where the data points are drawn independently. The -likelihood of the seeing the data under our model is just: +A widely used model is the so-called logistic (or sigmoid) function which ranges from \( 0 \) to \( 1 \), $$ -\begin{align} -P(\mathcal{D}|\mathbf{w})& = \prod_{i=1}^n \left[f(\mathbf{x}_i^T\mathbf{w})\right]^{y_i}\left[1-f(\mathbf{x}_i^T\mathbf{w})\right]^{1-y_i}\nonumber \\ -\tag{3} -\end{align} +p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}. $$ -from which we can readily compute the log-likelihood: -$$ -\begin{equation} -l(\mathbf{w}) = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right]. -\tag{4} -\end{equation} -$$ +Note that \( 1-p(t)= p(-t) \).
@@ -150,6 +145,12 @@ $$
+The perceptron is an example of a ``hard classification" model. We +will encounter this model when we discuss neural networks as +well. Each datapoint is deterministically assigned to a category (i.e +\( y_i=0 \) or \( y_i=1 \)). In many cases, it is favorable to have a "soft" +classifier that outputs the probability of a given category rather +than a single value. For example, given \( x_i \), the classifier +outputs the probability of being in a category \( k \). Logistic regression +is the most common example of a so-called soft classifier. In logistic +regression, the probability that a data point \( x_i \) +belongs to a category \( y_i=\{0,1\} \) is given by the so-called logit function (or Sigmoid)
@@ -141,6 +149,12 @@ in practice we usually supplement the cross-entropy with additional regularizati
-The cross entropy is a convex function of the weights \( \mathbf{w} \) and, -therefore, any local minimizer is a global minimizer. Minimizing this -cost function leads to the following equation +We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities +$$ +\begin{align*} +p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}), +\end{align*} +$$ -$$ -\begin{equation} -\boldsymbol{0}=\boldsymbol{\nabla} \mathcal{C}(\mathbf{w}) = \sum_{i=1}^n\left[f(\mathbf{x}_i^T\mathbf{w})-y_i\right]\mathbf{x}_i, -\tag{5} -\end{equation} -$$ +where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
-where we made use of the logistic function identity \( \partial_z f(z) = -f(z)[1-f(z)] \). This equation defines a transcendental equation for -\( \mathbf{w} \), the solution of which, unlike linear regression, cannot -be written in a closed form. -Here we need gradient descent methods! +Note that we used +$$ +p(y_i=1) = 1-p(y_i=0). +$$
@@ -147,6 +154,12 @@ Here we need gradient descent methods!
- + -
+In order to define the total likelihood for all possible outcomes from a +dataset \( \mathcal{D}=\{(y_i,x_i)\} \), with the binary labels +\( y_i\in\{0,1\} \) and where the data points are drawn independently, we use the so-called Maximum Likelihood Estimation (MLE) principle. +We aim thus at maximizing +the probability of seeing the observed data. We can then approximate the +likelihood in terms of the product of the individual probabilities of a specific outcome \( y_i \), that is +$$ +\begin{align*} +P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\ +\end{align*} +$$ - -
import numpy as np
-import matplotlib.pyplot as plt
-from sklearn import datasets
-iris = datasets.load_iris()
-list(iris.keys())
-['data', 'target_names', 'feature_names', 'target', 'DESCR']
-X = iris["data"][:, 3:] # petal width
-y = (iris["target"] == 2).astype(np.int) # 1 if Iris-Virginica, else 0
+from which we obtain the log-likelihood and our cost/loss function
+$$
+\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right].
+$$
-from sklearn.linear_model import LogisticRegression
-log_reg = LogisticRegression()
-log_reg.fit(X, y)
-
-X_new = np.linspace(0, 3, 1000).reshape(-1, 1)
-y_proba = log_reg.predict_proba(X_new)
-plt.plot(X_new, y_proba[:, 1], "g-", label="Iris-Virginica")
-plt.plot(X_new, y_proba[:, 0], "b--", label="Not Iris-Virginica")
-plt.show()
--
diff --git a/doc/pub/LogReg/html/._LogReg-bs010.html b/doc/pub/LogReg/html/._LogReg-bs010.html new file mode 100644 index 000000000..d0ee85632 --- /dev/null +++ b/doc/pub/LogReg/html/._LogReg-bs010.html @@ -0,0 +1,175 @@ + + + + + + + +
+ + + + +
+Reordering the logarithms, we can rewrite the cost/loss function as +$$ +\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{1+\exp{(\beta_0+\beta_1x_i)}}\right). +$$ + +
+ +
+ + +
+ + + +The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \beta \). +Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that +$$ +\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{1+\exp{(\beta_0+\beta_1x_i)}}\right). +$$ + +This equation is known in statistics as the cross entropy. Finally, we note that just as in linear regression, +in practice we often supplement the cross-entropy with additional regularization terms, usually \( L_1 \) and \( L_2 \) regularization as we did for Ridge and Lasso regression. + +
+
+ +
+ + +
+ + + + +
+The cross entropy is a convex function of the weights \( \hat{\beta} \) and, +therefore, any local minimizer is a global minimizer. + +
+Minimizing this +cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \) we obtain + +$$ +\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), +$$ + +and +$$ +\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). +$$ + +
+
+ +
+ + +
+ + + + +
+Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an +\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a +vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first +derivative of cost function as + +$$ +\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +$$ + +
+If we in addition define a diagonal matrix \( \hat{W} \) with elements +\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as + +$$ +\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +$$ + +
+
+ +
+ + +
+ + + + +
+ + +
import numpy as np
+import matplotlib.pyplot as plt
+from sklearn import datasets
+iris = datasets.load_iris()
+list(iris.keys())
+['data', 'target_names', 'feature_names', 'target', 'DESCR']
+X = iris["data"][:, 3:] # petal width
+y = (iris["target"] == 2).astype(np.int) # 1 if Iris-Virginica, else 0
+
+from sklearn.linear_model import LogisticRegression
+log_reg = LogisticRegression()
+log_reg.fit(X, y)
+
+X_new = np.linspace(0, 3, 1000).reshape(-1, 1)
+y_proba = log_reg.predict_proba(X_new)
+plt.plot(X_new, y_proba[:, 1], "g-", label="Iris-Virginica")
+plt.plot(X_new, y_proba[:, 0], "b--", label="Not Iris-Virginica")
+plt.show()
++
+ +
+ + +
+ + + + +
+ + +
import numpy as np
+from sklearn import datasets, linear_model
+import matplotlib.pyplot as plt
+
+
+def generate_data():
+ np.random.seed(0)
+ X, y = datasets.make_moons(200, noise=0.20)
+ return X, y
+
+
+def visualize(X, y, clf):
+ # plt.scatter(X[:, 0], X[:, 1], s=40, c=y, cmap=plt.cm.Spectral)
+ # plt.show()
+ plot_decision_boundary(lambda x: clf.predict(x), X, y)
+ plt.title("Logistic Regression")
+
+
+def plot_decision_boundary(pred_func, X, y):
+ # Set min and max values and give it some padding
+ x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
+ y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
+ h = 0.01
+ # Generate a grid of points with distance h between them
+ xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
+ # Predict the function value for the whole gid
+ Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
+ Z = Z.reshape(xx.shape)
+ # Plot the contour and training examples
+ plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
+ plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
+ plt.show()
+
+
+def classify(X, y):
+ clf = linear_model.LogisticRegressionCV()
+ clf.fit(X, y)
+ return clf
+
+
+def main():
+ X, y = generate_data()
+ # visualize(X, y)
+ clf = classify(X, y)
+ visualize(X, y, clf)
+
+
+if __name__ == "__main__":
+ main()
++ +
+ +
+ + +-
@@ -146,6 +155,8 @@ MathJax.Hub.Config({
-
@@ -162,202 +162,288 @@ MathJax.Hub.Config({
-So far we have focused on learning from datasets for which there is a -continuous output. In linear regression we have been -concerned with learning the coefficients of a polynomial to predict -the response of a continuous variable \( y_i \) on unseen data based on -its independent variables \( {\bf x}_i \). +In linear regression our main interest was centered on learning the +coefficients of a functional fit (say a polynomial) in order to be +able to predict the response of a continuous variable on some unseen +data. The fit to the continuous variable \( y_i \) is based on some +independent variables \( \hat{x}_i \). Linear regression resulted in +analytical expressions (in terms of matrices to invert) for several +quantities, ranging from the variance and thereby the confidence +intervals of the parameters \( \hat{\beta} \) to the mean squared +error. If we can invert the product of the design matrices, linear +regression gives then a simple recipe for fitting our data.
-Classification problems, -however, are concerned with outcomes taking the form of discrete -variables (i.e. categories). For example, we may want to detect if -there's a cat or a dog in an image. Or given a specific system, -we'd like to identify its state, say whether it is an ordered or disordered system (typical situation in solid state physics). -(e.g. ordered/disordered). +Classification problems, however, are concerned with outcomes taking +the form of discrete variables (i.e. categories). We may for example, +on the basis of DNA sequencing for a number of patients, like to find +out which mutations are important for a certain disease; or based on +scans of various patients' brains, figure out if there is a tumor or +not; or given a specific physical system, we'd like to identify its +state, say whether it is an ordered or disordered system (typical +situation in solid state physics); or classify the status of a +patient, whether she/he has a stroke or not and many other similar +situations.
-Logistic regression deals with binary, dichotomous outcomes (e.g. True or
-False, Success or Failure, etc.). It is worth noting that logistic
-regression is also commonly used in modern supervised Deep Learning
-models, as we will see later.
+The most common situation we encounter when we apply logistic
+regression is that of two possible outcomes, normally denoted as a
+binary outcome, true or false, positive or negative, success or
+failure etc.
-We consider the case where the dependent variables \( y_i\in\mathbb{Z} \)
-are discrete and only take values from \( m=0,\dots,M-1 \) (i.e. \( M \)
-classes).
+Logistic regression will also serve as our stepping towards neural
+network algorithms and supervised deep learning. For logistic
+learning, the minimization of the cost function leads to a non-linear
+equation in the parameters \( \hat{\beta} \). The optmization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the family of gradient descent methods. The latter are the working horses of basically all modern machine learning algorithms.
+
+
+We note also that many of the topics discussed here
+regression are also commonly used in modern supervised Deep Learning
+models, as we will see later.
+
+We consider the case where the dependent variables, also called the
+responses or the outcomes) \( y_i \) are discrete and only take values
+from \( k=0,\dots,K-1 \) (i.e. \( K \) classes).
The goal is to predict the
output classes from the design matrix \( X\in\mathbb{R}^{n\times p} \)
-made of \( n \) samples, each of which bears \( p \) features. The
+made of \( n \) samples, each of which carries \( p \) features. The
primary goal is to identify the classes to which new unseen samples
belong.
+
+
+Let us specialize to the case of two classes only, with outputs \( y_i=0 \) and \( y_i=1 \). Our outcomes could represent the status of a credit card use who could default or not on her/his debt. That is
+
-Let us start by considering a slightly simpler classifier: a linear classifier that categorizes examples using a weighted linear-combination of the features and an additive offset
+Before moving to the logistic model, let us try to use our linear regression model to classify these two outcomes. We could for example fit a linear model to the default case if \( y_i > 0.5 \) and the no default case \( y_i \leq 0.5 \).
+
+
+We would then have our
+weighted linear combination, namely
-This function takes values on the entire real axis. In the case of
+The main problem with our function is that it
+takes values on the entire real axis. In the case of
logistic regression, however, the labels \( y_i \) are discrete
-variables. One simple way to get a discrete output is to have sign
-functions that map the output of a linear regressor to \( \{0,1\} \),
-\( f(s_i)=sign(s_i)=1 \) if \( s_i\ge 0 \) and 0 if otherwise. Indeed,
-this is commonly known as the "perceptron" in the machine learning
-literature. This model is extremely simple, and it is favorable in
-many cases (e.g. noisy data) to have a ``soft" classifier that outputs
-the probability of a given category. For example, given
-\( \mathbf{x}_i \), the classifier outputs the probability of being in
-category \( m \). One such function is the logistic (or sigmoid) function:
+variables.
-
+One simple way to get a discrete output is to have sign
+functions that map the output of a linear regressor to values \( \{0,1\} \),
+\( f(s_i)=sign(s_i)=1 \) if \( s_i\ge 0 \) and 0 if otherwise.
+We will encounter this model in our first demonstration of neural networks. Historically it is called the "perceptron" model in the machine learning
+literature. This model is extremely simple. However, in many cases it is more
+favorable to use a ``soft" classifier that outputs
+the probability of a given category. This leads us to the logistic function.
-The perceptron is an example of a ``hard classification": each datapoint is deterministically assigned to a category (i.e \( y_i=0 \) or \( y_i=1 \)). In many cases, it is favorable to have a "soft" classifier that outputs the probability of a given category rather than a single value. For example, given \( \mathbf{x}_i \), the classifier outputs the probability of being in category \( m \).
-Logistic regression is the most canonical example of a soft classifier. In logistic regression, the probability that a data point \( \boldsymbol{x}_i \) belongs to a category \( y_i=\{0,1\} \) is is given by
+A widely used model is the so-called logistic (or sigmoid) function which ranges from \( 0 \) to \( 1 \),
-Notice that in terms of the logistic function, we can write
+The perceptron is an example of a ``hard classification" model. We
+will encounter this model when we discuss neural networks as
+well. Each datapoint is deterministically assigned to a category (i.e
+\( y_i=0 \) or \( y_i=1 \)). In many cases, it is favorable to have a "soft"
+classifier that outputs the probability of a given category rather
+than a single value. For example, given \( x_i \), the classifier
+outputs the probability of being in a category \( k \). Logistic regression
+is the most common example of a so-called soft classifier. In logistic
+regression, the probability that a data point \( x_i \)
+belongs to a category \( y_i=\{0,1\} \) is given by the so-called logit function (or Sigmoid)
+
+We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities
+Note that we used
+
-We now define the cost function for logistic regression using Maximum
-Likelihood Estimation (MLE). Recall, that in MLE we choose parameters
-to maximize the probability of seeing the observed data. Consider a
-dataset \( \mathcal{D}=\{(y_i,\boldsymbol{x}_i)\} \) with binary labels
-\( y_i\in\{0,1\} \) where the data points are drawn independently. The
-likelihood of the seeing the data under our model is just:
+In order to define the total likelihood for all possible outcomes from a
+dataset \( \mathcal{D}=\{(y_i,x_i)\} \), with the binary labels
+\( y_i\in\{0,1\} \) and where the data points are drawn independently, we use the so-called Maximum Likelihood Estimation (MLE) principle.
+We aim thus at maximizing
+the probability of seeing the observed data. We can then approximate the
+likelihood in terms of the product of the individual probabilities of a specific outcome \( y_i \), that is
+Reordering the logarithms, we can rewrite the cost/loss function as
+
-The cross entropy is a convex function of the weights \( \mathbf{w} \) and,
-therefore, any local minimizer is a global minimizer. Minimizing this
-cost function leads to the following equation
+The cross entropy is a convex function of the weights \( \hat{\beta} \) and,
+therefore, any local minimizer is a global minimizer.
+
+
+Minimizing this
+cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \) we obtain
-where we made use of the logistic function identity \( \partial_z f(z) =
-f(z)[1-f(z)] \). This equation defines a transcendental equation for
-\( \mathbf{w} \), the solution of which, unlike linear regression, cannot
-be written in a closed form.
-Here we need gradient descent methods!
+and
+
+Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an
+\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a
+vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first
+derivative of cost function as
+
+
+If we in addition define a diagonal matrix \( \hat{W} \) with elements
+\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as
+
+
@@ -384,6 +470,64 @@ plt.show()
+
+
+Basics
+Optimization and Deep learning
Basics
+
+
+$$
+y_i = \begin{bmatrix} 0 & \mathrm{no default}\\ 1 & \mathrm{default} \end{bmatrix}.
+$$
+
Linear classifier
+Linear classifier
$$
\begin{equation}
-s_i = \boldsymbol{x}_i^T\boldsymbol{w} + b_0 \equiv \mathbf{x}_i^T\mathbf{w},
+\hat{y} = \hat{X}_i^T\hat{\beta} + \hat{\epsilon},
\tag{1}
\end{equation}
$$
-where we use the short-hand notation
-\( \mathbf{x}_i = (1,\boldsymbol{x}_i) \) and \( \mathbf{w}_i = (b_0,\boldsymbol{w}_i) \).
+where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our
+\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators.
Some selected properties
+Some selected properties
-$$
-\begin{equation}
-f(s) = \frac{1}{1+\mathrm e^{-s}}.
-\tag{2}
-\end{equation}
-$$
-
-
-Note that \( 1-f(s)= f(-s) \), which will be useful shortly.
+The cross-entropy as a cost function for logistic regression
+The logistic function
$$
-\begin{eqnarray}
-P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& \frac{1}{1+\mathrm{e}^{-\mathbf{x}^T_i\mathbf{w}}},\nonumber\\
-P(y_i=0|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& 1 - P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)},
-\end{eqnarray}
+p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
$$
-where \( \boldsymbol{\theta}=\mathbf{w} \) are the weights we wish to learn from the data.
+Note that \( 1-p(t)= p(-t) \).
+Probabilistic classifiers
Two parameters
+
+
$$
-P(y_i=1) =f(\mathbf{x}_i^T\mathbf{w})=1-P(y_i=0).
+\begin{align*}
+p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
+p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
+\end{align*}
+$$
+
+
+where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
+
+
+$$
+p(y_i=1) = 1-p(y_i=0).
$$
Maximum likelihood
+Maximum likelihood
$$
-\begin{align}
-P(\mathcal{D}|\mathbf{w})& = \prod_{i=1}^n \left[f(\mathbf{x}_i^T\mathbf{w})\right]^{y_i}\left[1-f(\mathbf{x}_i^T\mathbf{w})\right]^{1-y_i}\nonumber \\
-\tag{3}
-\end{align}
+\begin{align*}
+P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
+\end{align*}
$$
-from which we can readily compute the log-likelihood:
+from which we obtain the log-likelihood and our cost/loss function
$$
-\begin{equation}
-l(\mathbf{w}) = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].
-\tag{4}
-\end{equation}
+\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right].
$$
-$$
-\hat{\mathbf{w}} = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].
-$$
-
+The cost function rewritten
+
+$$
+\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{1+\exp{(\beta_0+\beta_1x_i)}}\right).
+$$
+
+
$$
-\begin{eqnarray}
-\mathcal{C}(\mathbf{w}) &=& - l(\mathbf{w}) \\
-&=& \sum_{i=1}^n -y_i\log f(\mathbf{x}_i^T\mathbf{w}) - (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].\nonumber
-\end{eqnarray}
+\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$
-This equation is known in statistics as the \emph{cross entropy}. Finally, we note that just as in linear regression,
-in practice we usually supplement the cross-entropy with additional regularization terms, usually \( L_1 \) and \( L_2 \) regularization as we did for Ridge and Lasso regression.
+This equation is known in statistics as the cross entropy. Finally, we note that just as in linear regression,
+in practice we often supplement the cross-entropy with additional regularization terms, usually \( L_1 \) and \( L_2 \) regularization as we did for Ridge and Lasso regression.
Minimizing the cross entropy
+Minimizing the cross entropy
$$
-\begin{equation}
-\boldsymbol{0}=\boldsymbol{\nabla} \mathcal{C}(\mathbf{w}) = \sum_{i=1}^n\left[f(\mathbf{x}_i^T\mathbf{w})-y_i\right]\mathbf{x}_i,
-\tag{5}
-\end{equation}
+\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
$$
-
+$$
+\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
+$$
+
A scikit-learn example
+A more compact expression
+
+
+$$
+\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
+$$
+
+
+
+$$
+\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
+$$
+
+A scikit-learn example
A simple classification problem
+import numpy as np
+from sklearn import datasets, linear_model
+import matplotlib.pyplot as plt
+
+
+def generate_data():
+ np.random.seed(0)
+ X, y = datasets.make_moons(200, noise=0.20)
+ return X, y
+
+
+def visualize(X, y, clf):
+ # plt.scatter(X[:, 0], X[:, 1], s=40, c=y, cmap=plt.cm.Spectral)
+ # plt.show()
+ plot_decision_boundary(lambda x: clf.predict(x), X, y)
+ plt.title("Logistic Regression")
+
+
+def plot_decision_boundary(pred_func, X, y):
+ # Set min and max values and give it some padding
+ x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
+ y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
+ h = 0.01
+ # Generate a grid of points with distance h between them
+ xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
+ # Predict the function value for the whole gid
+ Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
+ Z = Z.reshape(xx.shape)
+ # Plot the contour and training examples
+ plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
+ plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
+ plt.show()
+
+
+def classify(X, y):
+ clf = linear_model.LogisticRegressionCV()
+ clf.fit(X, y)
+ return clf
+
+
+def main():
+ X, y = generate_data()
+ # visualize(X, y)
+ clf = classify(X, y)
+ visualize(X, y, clf)
+
+
+if __name__ == "__main__":
+ main()
+
-
@@ -94,183 +97,260 @@ MathJax.Hub.Config({
-So far we have focused on learning from datasets for which there is a -continuous output. In linear regression we have been -concerned with learning the coefficients of a polynomial to predict -the response of a continuous variable \( y_i \) on unseen data based on -its independent variables \( {\bf x}_i \). +In linear regression our main interest was centered on learning the +coefficients of a functional fit (say a polynomial) in order to be +able to predict the response of a continuous variable on some unseen +data. The fit to the continuous variable \( y_i \) is based on some +independent variables \( \hat{x}_i \). Linear regression resulted in +analytical expressions (in terms of matrices to invert) for several +quantities, ranging from the variance and thereby the confidence +intervals of the parameters \( \hat{\beta} \) to the mean squared +error. If we can invert the product of the design matrices, linear +regression gives then a simple recipe for fitting our data.
-Classification problems, -however, are concerned with outcomes taking the form of discrete -variables (i.e. categories). For example, we may want to detect if -there's a cat or a dog in an image. Or given a specific system, -we'd like to identify its state, say whether it is an ordered or disordered system (typical situation in solid state physics). -(e.g. ordered/disordered). +Classification problems, however, are concerned with outcomes taking +the form of discrete variables (i.e. categories). We may for example, +on the basis of DNA sequencing for a number of patients, like to find +out which mutations are important for a certain disease; or based on +scans of various patients' brains, figure out if there is a tumor or +not; or given a specific physical system, we'd like to identify its +state, say whether it is an ordered or disordered system (typical +situation in solid state physics); or classify the status of a +patient, whether she/he has a stroke or not and many other similar +situations.
-Logistic regression deals with binary, dichotomous outcomes (e.g. True or -False, Success or Failure, etc.). It is worth noting that logistic -regression is also commonly used in modern supervised Deep Learning -models, as we will see later. +The most common situation we encounter when we apply logistic +regression is that of two possible outcomes, normally denoted as a +binary outcome, true or false, positive or negative, success or +failure etc. + +
+
+
+
+Logistic regression will also serve as our stepping towards neural +network algorithms and supervised deep learning. For logistic +learning, the minimization of the cost function leads to a non-linear +equation in the parameters \( \hat{\beta} \). The optmization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the family of gradient descent methods. The latter are the working horses of basically all modern machine learning algorithms. + +
+We note also that many of the topics discussed here +regression are also commonly used in modern supervised Deep Learning +models, as we will see later.
-
-We consider the case where the dependent variables \( y_i\in\mathbb{Z} \) -are discrete and only take values from \( m=0,\dots,M-1 \) (i.e. \( M \) -classes). +We consider the case where the dependent variables, also called the +responses or the outcomes) \( y_i \) are discrete and only take values +from \( k=0,\dots,K-1 \) (i.e. \( K \) classes).
The goal is to predict the output classes from the design matrix \( X\in\mathbb{R}^{n\times p} \) -made of \( n \) samples, each of which bears \( p \) features. The +made of \( n \) samples, each of which carries \( p \) features. The primary goal is to identify the classes to which new unseen samples belong.
-
-
-
-Let us start by considering a slightly simpler classifier: a linear classifier that categorizes examples using a weighted linear-combination of the features and an additive offset
+
+
+
+Before moving to the logistic model, let us try to use our linear regression model to classify these two outcomes. We could for example fit a linear model to the default case if \( y_i > 0.5 \) and the no default case \( y_i \leq 0.5 \). + +
+We would then have our +weighted linear combination, namely $$ \begin{equation} -s_i = \boldsymbol{x}_i^T\boldsymbol{w} + b_0 \equiv \mathbf{x}_i^T\mathbf{w}, +\hat{y} = \hat{X}_i^T\hat{\beta} + \hat{\epsilon}, \label{_auto1} \end{equation} $$ -where we use the short-hand notation -\( \mathbf{x}_i = (1,\boldsymbol{x}_i) \) and \( \mathbf{w}_i = (b_0,\boldsymbol{w}_i) \). +where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our +\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators.
-
-This function takes values on the entire real axis. In the case of +The main problem with our function is that it +takes values on the entire real axis. In the case of logistic regression, however, the labels \( y_i \) are discrete -variables. One simple way to get a discrete output is to have sign -functions that map the output of a linear regressor to \( \{0,1\} \), -\( f(s_i)=sign(s_i)=1 \) if \( s_i\ge 0 \) and 0 if otherwise. Indeed, -this is commonly known as the "perceptron" in the machine learning -literature. This model is extremely simple, and it is favorable in -many cases (e.g. noisy data) to have a ``soft" classifier that outputs -the probability of a given category. For example, given -\( \mathbf{x}_i \), the classifier outputs the probability of being in -category \( m \). One such function is the logistic (or sigmoid) function: +variables. -$$ -\begin{equation} -f(s) = \frac{1}{1+\mathrm e^{-s}}. -\label{eq:log_fun} -\end{equation} -$$ - -Note that \( 1-f(s)= f(-s) \), which will be useful shortly. +
+One simple way to get a discrete output is to have sign +functions that map the output of a linear regressor to values \( \{0,1\} \), +\( f(s_i)=sign(s_i)=1 \) if \( s_i\ge 0 \) and 0 if otherwise. +We will encounter this model in our first demonstration of neural networks. Historically it is called the "perceptron" model in the machine learning +literature. This model is extremely simple. However, in many cases it is more +favorable to use a ``soft" classifier that outputs +the probability of a given category. This leads us to the logistic function.
-
-The perceptron is an example of a ``hard classification": each datapoint is deterministically assigned to a category (i.e \( y_i=0 \) or \( y_i=1 \)). In many cases, it is favorable to have a "soft" classifier that outputs the probability of a given category rather than a single value. For example, given \( \mathbf{x}_i \), the classifier outputs the probability of being in category \( m \). -Logistic regression is the most canonical example of a soft classifier. In logistic regression, the probability that a data point \( \boldsymbol{x}_i \) belongs to a category \( y_i=\{0,1\} \) is is given by +A widely used model is the so-called logistic (or sigmoid) function which ranges from \( 0 \) to \( 1 \), $$ -\begin{eqnarray} -P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& \frac{1}{1+\mathrm{e}^{-\mathbf{x}^T_i\mathbf{w}}},\nonumber\\ -P(y_i=0|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& 1 - P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)}, -\end{eqnarray} +p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}. $$ -where \( \boldsymbol{\theta}=\mathbf{w} \) are the weights we wish to learn from the data. +Note that \( 1-p(t)= p(-t) \).
-Notice that in terms of the logistic function, we can write
+
+
+
+The perceptron is an example of a ``hard classification" model. We +will encounter this model when we discuss neural networks as +well. Each datapoint is deterministically assigned to a category (i.e +\( y_i=0 \) or \( y_i=1 \)). In many cases, it is favorable to have a "soft" +classifier that outputs the probability of a given category rather +than a single value. For example, given \( x_i \), the classifier +outputs the probability of being in a category \( k \). Logistic regression +is the most common example of a so-called soft classifier. In logistic +regression, the probability that a data point \( x_i \) +belongs to a category \( y_i=\{0,1\} \) is given by the so-called logit function (or Sigmoid) + +
+
+
+
+We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities $$ -P(y_i=1) =f(\mathbf{x}_i^T\mathbf{w})=1-P(y_i=0). +\begin{align*} +p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}), +\end{align*} +$$ + +where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \). + +
+Note that we used +$$ +p(y_i=1) = 1-p(y_i=0). $$
-
-We now define the cost function for logistic regression using Maximum -Likelihood Estimation (MLE). Recall, that in MLE we choose parameters -to maximize the probability of seeing the observed data. Consider a -dataset \( \mathcal{D}=\{(y_i,\boldsymbol{x}_i)\} \) with binary labels -\( y_i\in\{0,1\} \) where the data points are drawn independently. The -likelihood of the seeing the data under our model is just: +In order to define the total likelihood for all possible outcomes from a +dataset \( \mathcal{D}=\{(y_i,x_i)\} \), with the binary labels +\( y_i\in\{0,1\} \) and where the data points are drawn independently, we use the so-called Maximum Likelihood Estimation (MLE) principle. +We aim thus at maximizing +the probability of seeing the observed data. We can then approximate the +likelihood in terms of the product of the individual probabilities of a specific outcome \( y_i \), that is $$ -\begin{align} -P(\mathcal{D}|\mathbf{w})& = \prod_{i=1}^n \left[f(\mathbf{x}_i^T\mathbf{w})\right]^{y_i}\left[1-f(\mathbf{x}_i^T\mathbf{w})\right]^{1-y_i}\nonumber \\ -\label{_auto2} -\end{align} +\begin{align*} +P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\ +\end{align*} $$ -from which we can readily compute the log-likelihood: +from which we obtain the log-likelihood and our cost/loss function $$ -\begin{equation} -l(\mathbf{w}) = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right]. -\label{_auto3} -\end{equation} +\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]. $$
-The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \theta \)
+
+
+Reordering the logarithms, we can rewrite the cost/loss function as
$$
-\hat{\mathbf{w}} = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].
+\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$
+
+The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \beta \).
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
$$
-\begin{eqnarray}
-\mathcal{C}(\mathbf{w}) &=& - l(\mathbf{w}) \\
-&=& \sum_{i=1}^n -y_i\log f(\mathbf{x}_i^T\mathbf{w}) - (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].\nonumber
-\end{eqnarray}
+\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$
-This equation is known in statistics as the \emph{cross entropy}. Finally, we note that just as in linear regression,
-in practice we usually supplement the cross-entropy with additional regularization terms, usually \( L_1 \) and \( L_2 \) regularization as we did for Ridge and Lasso regression.
+This equation is known in statistics as the cross entropy. Finally, we note that just as in linear regression,
+in practice we often supplement the cross-entropy with additional regularization terms, usually \( L_1 \) and \( L_2 \) regularization as we did for Ridge and Lasso regression.
-
-The cross entropy is a convex function of the weights \( \mathbf{w} \) and, -therefore, any local minimizer is a global minimizer. Minimizing this -cost function leads to the following equation - -$$ -\begin{equation} -\boldsymbol{0}=\boldsymbol{\nabla} \mathcal{C}(\mathbf{w}) = \sum_{i=1}^n\left[f(\mathbf{x}_i^T\mathbf{w})-y_i\right]\mathbf{x}_i, -\label{_auto4} -\end{equation} -$$ +The cross entropy is a convex function of the weights \( \hat{\beta} \) and, +therefore, any local minimizer is a global minimizer.
-where we made use of the logistic function identity \( \partial_z f(z) = -f(z)[1-f(z)] \). This equation defines a transcendental equation for -\( \mathbf{w} \), the solution of which, unlike linear regression, cannot -be written in a closed form. -Here we need gradient descent methods! +Minimizing this +cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \) we obtain + +$$ +\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), +$$ + +and +$$ +\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). +$$
-
+Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an +\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a +vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first +derivative of cost function as + +$$ +\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +$$ + +
+If we in addition define a diagonal matrix \( \hat{W} \) with elements +\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as + +$$ +\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +$$ + +
+
+
+
@@ -295,6 +375,63 @@ plt.plot(X_new, y_proba[:, 0], A simple classification problem
+
+
+
+
diff --git a/doc/pub/LogReg/html/LogReg.html b/doc/pub/LogReg/html/LogReg.html
index 889e00fb5..1dab7dca5 100644
--- a/doc/pub/LogReg/html/LogReg.html
+++ b/doc/pub/LogReg/html/LogReg.html
@@ -40,16 +40,19 @@ div { text-align: justify; text-justify: inter-word; }
-
@@ -99,183 +102,260 @@ MathJax.Hub.Config({
-So far we have focused on learning from datasets for which there is a
-continuous output. In linear regression we have been
-concerned with learning the coefficients of a polynomial to predict
-the response of a continuous variable \( y_i \) on unseen data based on
-its independent variables \( {\bf x}_i \).
+In linear regression our main interest was centered on learning the
+coefficients of a functional fit (say a polynomial) in order to be
+able to predict the response of a continuous variable on some unseen
+data. The fit to the continuous variable \( y_i \) is based on some
+independent variables \( \hat{x}_i \). Linear regression resulted in
+analytical expressions (in terms of matrices to invert) for several
+quantities, ranging from the variance and thereby the confidence
+intervals of the parameters \( \hat{\beta} \) to the mean squared
+error. If we can invert the product of the design matrices, linear
+regression gives then a simple recipe for fitting our data.
-Classification problems,
-however, are concerned with outcomes taking the form of discrete
-variables (i.e. categories). For example, we may want to detect if
-there's a cat or a dog in an image. Or given a specific system,
-we'd like to identify its state, say whether it is an ordered or disordered system (typical situation in solid state physics).
-(e.g. ordered/disordered).
+Classification problems, however, are concerned with outcomes taking
+the form of discrete variables (i.e. categories). We may for example,
+on the basis of DNA sequencing for a number of patients, like to find
+out which mutations are important for a certain disease; or based on
+scans of various patients' brains, figure out if there is a tumor or
+not; or given a specific physical system, we'd like to identify its
+state, say whether it is an ordered or disordered system (typical
+situation in solid state physics); or classify the status of a
+patient, whether she/he has a stroke or not and many other similar
+situations.
-Logistic regression deals with binary, dichotomous outcomes (e.g. True or
-False, Success or Failure, etc.). It is worth noting that logistic
-regression is also commonly used in modern supervised Deep Learning
-models, as we will see later.
+The most common situation we encounter when we apply logistic
+regression is that of two possible outcomes, normally denoted as a
+binary outcome, true or false, positive or negative, success or
+failure etc.
+
+
+
+Logistic regression will also serve as our stepping towards neural
+network algorithms and supervised deep learning. For logistic
+learning, the minimization of the cost function leads to a non-linear
+equation in the parameters \( \hat{\beta} \). The optmization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the family of gradient descent methods. The latter are the working horses of basically all modern machine learning algorithms.
+
+
+We note also that many of the topics discussed here
+regression are also commonly used in modern supervised Deep Learning
+models, as we will see later.
-
-We consider the case where the dependent variables \( y_i\in\mathbb{Z} \)
-are discrete and only take values from \( m=0,\dots,M-1 \) (i.e. \( M \)
-classes).
+We consider the case where the dependent variables, also called the
+responses or the outcomes) \( y_i \) are discrete and only take values
+from \( k=0,\dots,K-1 \) (i.e. \( K \) classes).
The goal is to predict the
output classes from the design matrix \( X\in\mathbb{R}^{n\times p} \)
-made of \( n \) samples, each of which bears \( p \) features. The
+made of \( n \) samples, each of which carries \( p \) features. The
primary goal is to identify the classes to which new unseen samples
belong.
-
-Let us start by considering a slightly simpler classifier: a linear classifier that categorizes examples using a weighted linear-combination of the features and an additive offset
+
+Before moving to the logistic model, let us try to use our linear regression model to classify these two outcomes. We could for example fit a linear model to the default case if \( y_i > 0.5 \) and the no default case \( y_i \leq 0.5 \).
+
+
+We would then have our
+weighted linear combination, namely
$$
\begin{equation}
-s_i = \boldsymbol{x}_i^T\boldsymbol{w} + b_0 \equiv \mathbf{x}_i^T\mathbf{w},
+\hat{y} = \hat{X}_i^T\hat{\beta} + \hat{\epsilon},
\label{_auto1}
\end{equation}
$$
-where we use the short-hand notation
-\( \mathbf{x}_i = (1,\boldsymbol{x}_i) \) and \( \mathbf{w}_i = (b_0,\boldsymbol{w}_i) \).
+where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our
+\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators.
-This function takes values on the entire real axis. In the case of
+The main problem with our function is that it
+takes values on the entire real axis. In the case of
logistic regression, however, the labels \( y_i \) are discrete
-variables. One simple way to get a discrete output is to have sign
-functions that map the output of a linear regressor to \( \{0,1\} \),
-\( f(s_i)=sign(s_i)=1 \) if \( s_i\ge 0 \) and 0 if otherwise. Indeed,
-this is commonly known as the "perceptron" in the machine learning
-literature. This model is extremely simple, and it is favorable in
-many cases (e.g. noisy data) to have a ``soft" classifier that outputs
-the probability of a given category. For example, given
-\( \mathbf{x}_i \), the classifier outputs the probability of being in
-category \( m \). One such function is the logistic (or sigmoid) function:
+variables.
-$$
-\begin{equation}
-f(s) = \frac{1}{1+\mathrm e^{-s}}.
-\label{eq:log_fun}
-\end{equation}
-$$
-
-Note that \( 1-f(s)= f(-s) \), which will be useful shortly.
+
+One simple way to get a discrete output is to have sign
+functions that map the output of a linear regressor to values \( \{0,1\} \),
+\( f(s_i)=sign(s_i)=1 \) if \( s_i\ge 0 \) and 0 if otherwise.
+We will encounter this model in our first demonstration of neural networks. Historically it is called the "perceptron" model in the machine learning
+literature. This model is extremely simple. However, in many cases it is more
+favorable to use a ``soft" classifier that outputs
+the probability of a given category. This leads us to the logistic function.
-The perceptron is an example of a ``hard classification": each datapoint is deterministically assigned to a category (i.e \( y_i=0 \) or \( y_i=1 \)). In many cases, it is favorable to have a "soft" classifier that outputs the probability of a given category rather than a single value. For example, given \( \mathbf{x}_i \), the classifier outputs the probability of being in category \( m \).
-Logistic regression is the most canonical example of a soft classifier. In logistic regression, the probability that a data point \( \boldsymbol{x}_i \) belongs to a category \( y_i=\{0,1\} \) is is given by
+A widely used model is the so-called logistic (or sigmoid) function which ranges from \( 0 \) to \( 1 \),
$$
-\begin{eqnarray}
-P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& \frac{1}{1+\mathrm{e}^{-\mathbf{x}^T_i\mathbf{w}}},\nonumber\\
-P(y_i=0|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& 1 - P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)},
-\end{eqnarray}
+p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
$$
-where \( \boldsymbol{\theta}=\mathbf{w} \) are the weights we wish to learn from the data.
+Note that \( 1-p(t)= p(-t) \).
-Notice that in terms of the logistic function, we can write
+
+The perceptron is an example of a ``hard classification" model. We
+will encounter this model when we discuss neural networks as
+well. Each datapoint is deterministically assigned to a category (i.e
+\( y_i=0 \) or \( y_i=1 \)). In many cases, it is favorable to have a "soft"
+classifier that outputs the probability of a given category rather
+than a single value. For example, given \( x_i \), the classifier
+outputs the probability of being in a category \( k \). Logistic regression
+is the most common example of a so-called soft classifier. In logistic
+regression, the probability that a data point \( x_i \)
+belongs to a category \( y_i=\{0,1\} \) is given by the so-called logit function (or Sigmoid)
+
+
+
+We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities
$$
-P(y_i=1) =f(\mathbf{x}_i^T\mathbf{w})=1-P(y_i=0).
+\begin{align*}
+p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
+p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
+\end{align*}
+$$
+
+where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
+
+
+Note that we used
+$$
+p(y_i=1) = 1-p(y_i=0).
$$
-
-We now define the cost function for logistic regression using Maximum
-Likelihood Estimation (MLE). Recall, that in MLE we choose parameters
-to maximize the probability of seeing the observed data. Consider a
-dataset \( \mathcal{D}=\{(y_i,\boldsymbol{x}_i)\} \) with binary labels
-\( y_i\in\{0,1\} \) where the data points are drawn independently. The
-likelihood of the seeing the data under our model is just:
+In order to define the total likelihood for all possible outcomes from a
+dataset \( \mathcal{D}=\{(y_i,x_i)\} \), with the binary labels
+\( y_i\in\{0,1\} \) and where the data points are drawn independently, we use the so-called Maximum Likelihood Estimation (MLE) principle.
+We aim thus at maximizing
+the probability of seeing the observed data. We can then approximate the
+likelihood in terms of the product of the individual probabilities of a specific outcome \( y_i \), that is
$$
-\begin{align}
-P(\mathcal{D}|\mathbf{w})& = \prod_{i=1}^n \left[f(\mathbf{x}_i^T\mathbf{w})\right]^{y_i}\left[1-f(\mathbf{x}_i^T\mathbf{w})\right]^{1-y_i}\nonumber \\
-\label{_auto2}
-\end{align}
+\begin{align*}
+P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
+\end{align*}
$$
-from which we can readily compute the log-likelihood:
+from which we obtain the log-likelihood and our cost/loss function
$$
-\begin{equation}
-l(\mathbf{w}) = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].
-\label{_auto3}
-\end{equation}
+\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right].
$$
+Reordering the logarithms, we can rewrite the cost/loss function as
$$
-\hat{\mathbf{w}} = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].
+\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$
+
-The cross entropy is a convex function of the weights \( \mathbf{w} \) and,
-therefore, any local minimizer is a global minimizer. Minimizing this
-cost function leads to the following equation
-
-$$
-\begin{equation}
-\boldsymbol{0}=\boldsymbol{\nabla} \mathcal{C}(\mathbf{w}) = \sum_{i=1}^n\left[f(\mathbf{x}_i^T\mathbf{w})-y_i\right]\mathbf{x}_i,
-\label{_auto4}
-\end{equation}
-$$
+The cross entropy is a convex function of the weights \( \hat{\beta} \) and,
+therefore, any local minimizer is a global minimizer.
-where we made use of the logistic function identity \( \partial_z f(z) =
-f(z)[1-f(z)] \). This equation defines a transcendental equation for
-\( \mathbf{w} \), the solution of which, unlike linear regression, cannot
-be written in a closed form.
-Here we need gradient descent methods!
+Minimizing this
+cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \) we obtain
+
+$$
+\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
+$$
+
+and
+$$
+\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
+$$
+Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an
+\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a
+vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first
+derivative of cost function as
+
+$$
+\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
+$$
+
+
+If we in addition define a diagonal matrix \( \hat{W} \) with elements
+\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as
+
+$$
+\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
+$$
+
+
+
@@ -300,6 +380,63 @@ plt.plot(X_new, y_proba[:, .show()
+
+
+
+
diff --git a/doc/pub/LogReg/ipynb/LogReg.ipynb b/doc/pub/LogReg/ipynb/LogReg.ipynb
index 90c236b5c..b6532eb40 100644
--- a/doc/pub/LogReg/ipynb/LogReg.ipynb
+++ b/doc/pub/LogReg/ipynb/LogReg.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Sep 20, 2018**\n",
+ "Date: **Sep 25, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -20,42 +20,81 @@
"\n",
"## Logistic Regression\n",
"\n",
- "So far we have focused on learning from datasets for which there is a\n",
- "**continuous** output. In linear regression we have been \n",
- "concerned with learning the coefficients of a polynomial to predict\n",
- "the response of a continuous variable $y_i$ on unseen data based on\n",
- "its independent variables ${\\bf x}_i$. \n",
+ "In linear regression our main interest was centered on learning the\n",
+ "coefficients of a functional fit (say a polynomial) in order to be\n",
+ "able to predict the response of a continuous variable on some unseen\n",
+ "data. The fit to the continuous variable $y_i$ is based on some\n",
+ "independent variables $\\hat{x}_i$. Linear regression resulted in\n",
+ "analytical expressions (in terms of matrices to invert) for several\n",
+ "quantities, ranging from the variance and thereby the confidence\n",
+ "intervals of the parameters $\\hat{\\beta}$ to the mean squared\n",
+ "error. If we can invert the product of the design matrices, linear\n",
+ "regression gives then a simple recipe for fitting our data.\n",
"\n",
- "Classification problems,\n",
- "however, are concerned with outcomes taking the form of discrete\n",
- "variables (i.e. categories). For example, we may want to detect if\n",
- "there's a cat or a dog in an image. Or given a specific system,\n",
- "we'd like to identify its state, say whether it is an ordered or disordered system (typical situation in solid state physics).\n",
- "(e.g. ordered/disordered). \n",
"\n",
- "**Logistic regression deals with binary, dichotomous outcomes (e.g. True or\n",
- "False, Success or Failure, etc.). It is worth noting that logistic\n",
- "regression is also commonly used in modern supervised Deep Learning\n",
- "models**, as we will see later.\n",
+ "Classification problems, however, are concerned with outcomes taking\n",
+ "the form of discrete variables (i.e. categories). We may for example,\n",
+ "on the basis of DNA sequencing for a number of patients, like to find\n",
+ "out which mutations are important for a certain disease; or based on\n",
+ "scans of various patients' brains, figure out if there is a tumor or\n",
+ "not; or given a specific physical system, we'd like to identify its\n",
+ "state, say whether it is an ordered or disordered system (typical\n",
+ "situation in solid state physics); or classify the status of a\n",
+ "patient, whether she/he has a stroke or not and many other similar\n",
+ "situations.\n",
+ "\n",
+ "The most common situation we encounter when we apply logistic\n",
+ "regression is that of two possible outcomes, normally denoted as a\n",
+ "binary outcome, true or false, positive or negative, success or\n",
+ "failure etc.\n",
+ "\n",
+ "## Optimization and Deep learning\n",
+ "\n",
+ "Logistic regression will also serve as our stepping towards neural\n",
+ "network algorithms and supervised deep learning. For logistic\n",
+ "learning, the minimization of the cost function leads to a non-linear\n",
+ "equation in the parameters $\\hat{\\beta}$. The optmization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the family of gradient descent methods. The latter are the working horses of basically all modern machine learning algorithms. \n",
+ "\n",
+ "We note also that many of the topics discussed here \n",
+ "regression are also commonly used in modern supervised Deep Learning\n",
+ "models, as we will see later.\n",
"\n",
"\n",
"\n",
"## Basics\n",
"\n",
- "We consider the case where the dependent variables $y_i\\in\\mathbb{Z}$\n",
- "are discrete and only take values from $m=0,\\dots,M-1$ (i.e. $M$\n",
- "classes).\n",
+ "We consider the case where the dependent variables, also called the\n",
+ "responses or the outcomes) $y_i$ are discrete and only take values\n",
+ "from $k=0,\\dots,K-1$ (i.e. $K$ classes).\n",
"\n",
"The goal is to predict the\n",
"output classes from the design matrix $X\\in\\mathbb{R}^{n\\times p}$\n",
- "made of $n$ samples, each of which bears $p$ features. The\n",
+ "made of $n$ samples, each of which carries $p$ features. The\n",
"primary goal is to identify the classes to which new unseen samples\n",
"belong.\n",
"\n",
- "\n",
+ "Let us specialize to the case of two classes only, with outputs $y_i=0$ and $y_i=1$. Our outcomes could represent the status of a credit card use who could default or not on her/his debt. That is"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "y_i = \\begin{bmatrix} 0 & \\mathrm{no default}\\\\ 1 & \\mathrm{default} \\end{bmatrix}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
"## Linear classifier\n",
"\n",
- "Let us start by considering a slightly simpler classifier: a linear classifier that categorizes examples using a weighted linear-combination of the features and an additive offset"
+ "Before moving to the logistic model, let us try to use our linear regression model to classify these two outcomes. We could for example fit a linear model to the default case if $y_i > 0.5$ and the no default case $y_i \\leq 0.5$. \n",
+ "\n",
+ "We would then have our \n",
+ "weighted linear combination, namely"
]
},
{
@@ -67,7 +106,7 @@
"\n",
"$$\n",
"\\begin{equation}\n",
- "s_i = \\boldsymbol{x}_i^T\\boldsymbol{w} + b_0 \\equiv \\mathbf{x}_i^T\\mathbf{w},\n",
+ "\\hat{y} = \\hat{X}_i^T\\hat{\\beta} + \\hat{\\epsilon},\n",
"\\label{_auto1} \\tag{1}\n",
"\\end{equation}\n",
"$$"
@@ -77,36 +116,35 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "where we use the short-hand notation \n",
- "$\\mathbf{x}_i = (1,\\boldsymbol{x}_i)$ and $\\mathbf{w}_i = (b_0,\\boldsymbol{w}_i)$. \n",
+ "where $\\hat{y}$ is a vector representing the possible outcomes, $\\hat{X}$ is our\n",
+ "$n\\times p$ design matrix and $\\hat{\\beta}$ represents our estimators.\n",
"\n",
"## Some selected properties\n",
"\n",
- "This function takes values on the entire real axis. In the case of\n",
+ "The main problem with our function is that it \n",
+ "takes values on the entire real axis. In the case of\n",
"logistic regression, however, the labels $y_i$ are discrete\n",
- "variables. One simple way to get a discrete output is to have sign\n",
- "functions that map the output of a linear regressor to $\\{0,1\\}$,\n",
- "$f(s_i)=sign(s_i)=1$ if $s_i\\ge 0$ and 0 if otherwise. Indeed,\n",
- "this is commonly known as the \"perceptron\" in the machine learning\n",
- "literature. This model is extremely simple, and it is favorable in\n",
- "many cases (e.g. noisy data) to have a ``soft\" classifier that outputs\n",
- "the probability of a given category. For example, given\n",
- "$\\mathbf{x}_i$, the classifier outputs the probability of being in\n",
- "category $m$. One such function is the logistic (or sigmoid) function:"
+ "variables. \n",
+ "\n",
+ "One simple way to get a discrete output is to have sign\n",
+ "functions that map the output of a linear regressor to values $\\{0,1\\}$,\n",
+ "$f(s_i)=sign(s_i)=1$ if $s_i\\ge 0$ and 0 if otherwise. \n",
+ "We will encounter this model in our first demonstration of neural networks. Historically it is called the \"perceptron\" model in the machine learning\n",
+ "literature. This model is extremely simple. However, in many cases it is more\n",
+ "favorable to use a ``soft\" classifier that outputs\n",
+ "the probability of a given category. This leads us to the logistic function.\n",
+ "\n",
+ "## The logistic function\n",
+ "\n",
+ "A widely used model is the so-called logistic (or sigmoid) function which ranges from $0$ to $1$,"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "\n",
- "import numpy as np
+from sklearn import datasets, linear_model
+import matplotlib.pyplot as plt
+
+
+def generate_data():
+ np.random.seed(0)
+ X, y = datasets.make_moons(200, noise=0.20)
+ return X, y
+
+
+def visualize(X, y, clf):
+ # plt.scatter(X[:, 0], X[:, 1], s=40, c=y, cmap=plt.cm.Spectral)
+ # plt.show()
+ plot_decision_boundary(lambda x: clf.predict(x), X, y)
+ plt.title("Logistic Regression")
+
+
+def plot_decision_boundary(pred_func, X, y):
+ # Set min and max values and give it some padding
+ x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
+ y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
+ h = 0.01
+ # Generate a grid of points with distance h between them
+ xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
+ # Predict the function value for the whole gid
+ Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
+ Z = Z.reshape(xx.shape)
+ # Plot the contour and training examples
+ plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
+ plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
+ plt.show()
+
+
+def classify(X, y):
+ clf = linear_model.LogisticRegressionCV()
+ clf.fit(X, y)
+ return clf
+
+
+def main():
+ X, y = generate_data()
+ # visualize(X, y)
+ clf = classify(X, y)
+ visualize(X, y, clf)
+
+
+if __name__ == "__main__":
+ main()
+
Sep 20, 2018
Sep 25, 2018
Logistic Regression
+
+Optimization and Deep learning
+
+Basics
+Basics
-
-Linear classifier
+Let us specialize to the case of two classes only, with outputs \( y_i=0 \) and \( y_i=1 \). Our outcomes could represent the status of a credit card use who could default or not on her/his debt. That is
+$$
+y_i = \begin{bmatrix} 0 & \mathrm{no default}\\ 1 & \mathrm{default} \end{bmatrix}.
+$$
+
+Linear classifier
+
+
-Some selected properties
+Some selected properties
-The cross-entropy as a cost function for logistic regression
+The logistic function
+
+Probabilistic classifiers
+
+
+
+Two parameters
+
+Maximum likelihood
+Maximum likelihood
-The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \theta \)
+
+The cost function rewritten
+
+
+The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \beta \).
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
$$
-\begin{eqnarray}
-\mathcal{C}(\mathbf{w}) &=& - l(\mathbf{w}) \\
-&=& \sum_{i=1}^n -y_i\log f(\mathbf{x}_i^T\mathbf{w}) - (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].\nonumber
-\end{eqnarray}
+\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$
-This equation is known in statistics as the \emph{cross entropy}. Finally, we note that just as in linear regression,
-in practice we usually supplement the cross-entropy with additional regularization terms, usually \( L_1 \) and \( L_2 \) regularization as we did for Ridge and Lasso regression.
+This equation is known in statistics as the cross entropy. Finally, we note that just as in linear regression,
+in practice we often supplement the cross-entropy with additional regularization terms, usually \( L_1 \) and \( L_2 \) regularization as we did for Ridge and Lasso regression.
-Minimizing the cross entropy
+Minimizing the cross entropy
-A scikit-learn example
+A more compact expression
+
+
+
+A scikit-learn example
+
+A simple classification problem
+import numpy as np
+from sklearn import datasets, linear_model
+import matplotlib.pyplot as plt
+
+
+def generate_data():
+ np.random.seed(0)
+ X, y = datasets.make_moons(200, noise=0.20)
+ return X, y
+
+
+def visualize(X, y, clf):
+ # plt.scatter(X[:, 0], X[:, 1], s=40, c=y, cmap=plt.cm.Spectral)
+ # plt.show()
+ plot_decision_boundary(lambda x: clf.predict(x), X, y)
+ plt.title("Logistic Regression")
+
+
+def plot_decision_boundary(pred_func, X, y):
+ # Set min and max values and give it some padding
+ x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
+ y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
+ h = 0.01
+ # Generate a grid of points with distance h between them
+ xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
+ # Predict the function value for the whole gid
+ Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
+ Z = Z.reshape(xx.shape)
+ # Plot the contour and training examples
+ plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
+ plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
+ plt.show()
+
+
+def classify(X, y):
+ clf = linear_model.LogisticRegressionCV()
+ clf.fit(X, y)
+ return clf
+
+
+def main():
+ X, y = generate_data()
+ # visualize(X, y)
+ clf = classify(X, y)
+ visualize(X, y, clf)
+
+
+if __name__ == "__main__":
+ main()
+