-
diff --git a/doc/pub/LogReg/html/._LogReg-bs001.html b/doc/pub/LogReg/html/._LogReg-bs001.html index d07fe970b..101129f7e 100644 --- a/doc/pub/LogReg/html/._LogReg-bs001.html +++ b/doc/pub/LogReg/html/._LogReg-bs001.html @@ -53,7 +53,7 @@ Automatically generated HTML file from DocOnce source ('A more compact expression', 2, None, '___sec10'), ('Extending to more predictors', 2, None, '___sec11'), ('Including more classes', 2, None, '___sec12'), - ('Optimizing the cost function', 2, None, '___sec13'), + ('The Softmax function', 2, None, '___sec13'), ('A _scikit-learn_ example', 2, None, '___sec14'), ('A simple classification problem', 2, None, '___sec15')]} end of tocinfo --> @@ -106,7 +106,7 @@ MathJax.Hub.Config({
+Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors +$$ +\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. +$$ + +Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to +$$ +p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. +$$ +
diff --git a/doc/pub/LogReg/html/._LogReg-bs013.html b/doc/pub/LogReg/html/._LogReg-bs013.html index 49249443a..a70b01ff9 100644 --- a/doc/pub/LogReg/html/._LogReg-bs013.html +++ b/doc/pub/LogReg/html/._LogReg-bs013.html @@ -53,7 +53,7 @@ Automatically generated HTML file from DocOnce source ('A more compact expression', 2, None, '___sec10'), ('Extending to more predictors', 2, None, '___sec11'), ('Including more classes', 2, None, '___sec12'), - ('Optimizing the cost function', 2, None, '___sec13'), + ('The Softmax function', 2, None, '___sec13'), ('A _scikit-learn_ example', 2, None, '___sec14'), ('A simple classification problem', 2, None, '___sec15')]} end of tocinfo --> @@ -106,7 +106,7 @@ MathJax.Hub.Config({
+Till now we have mainly focused on two classes, the so-called binary system. Suppose we wish to extend to \( K \) classes. +Let us for the sake of simplicity assume we have only two predictors. We have then following model +$$ +\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1, +$$ + +$$ +\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1, +$$ + +and so on till the class \( C=K-1 \) class +$$ +\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1, +$$ + +and the model is specified in term of \( K-1 \) so-called log-odds or logit transformations. +
diff --git a/doc/pub/LogReg/html/._LogReg-bs014.html b/doc/pub/LogReg/html/._LogReg-bs014.html index 83da3d97b..f5f82f6b8 100644 --- a/doc/pub/LogReg/html/._LogReg-bs014.html +++ b/doc/pub/LogReg/html/._LogReg-bs014.html @@ -53,7 +53,7 @@ Automatically generated HTML file from DocOnce source ('A more compact expression', 2, None, '___sec10'), ('Extending to more predictors', 2, None, '___sec11'), ('Including more classes', 2, None, '___sec12'), - ('Optimizing the cost function', 2, None, '___sec13'), + ('The Softmax function', 2, None, '___sec13'), ('A _scikit-learn_ example', 2, None, '___sec14'), ('A simple classification problem', 2, None, '___sec15')]} end of tocinfo --> @@ -106,7 +106,7 @@ MathJax.Hub.Config({
-Newton's method and gradient descent methods +In our discussion of neural networks we will encounter the above again in terms of the so-called Softmax function. + +
+The softmax function is used in various multiclass classification +methods, such as multinomial logistic regression (also known as +softmax regression), multiclass linear discriminant +analysis, naive Bayes classifiers, and artificial neural networks. +Specifically, in multinomial logistic regression and linear +discriminant analysis, the input to the function is the result of \( K \) +distinct linear functions, and the predicted probability for the \( k \)-th +class given a sample vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two predictors): + +$$ +p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}. +$$ + +It is easy to extend to more predictors. The final class is +$$ +p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}, +$$ + +and they sum to one. Our earlier discussions were all specialized to the case with two classes only. It is easy to see from the above that what we derived earlier is compatible with these equations. + +
+To find the optimal parameters we would typically use a gradient descent method. +Newton's method and gradient descent methods are discussed in the material on optimization methods.
diff --git a/doc/pub/LogReg/html/._LogReg-bs015.html b/doc/pub/LogReg/html/._LogReg-bs015.html index edca491af..8a7707493 100644 --- a/doc/pub/LogReg/html/._LogReg-bs015.html +++ b/doc/pub/LogReg/html/._LogReg-bs015.html @@ -53,7 +53,7 @@ Automatically generated HTML file from DocOnce source ('A more compact expression', 2, None, '___sec10'), ('Extending to more predictors', 2, None, '___sec11'), ('Including more classes', 2, None, '___sec12'), - ('Optimizing the cost function', 2, None, '___sec13'), + ('The Softmax function', 2, None, '___sec13'), ('A _scikit-learn_ example', 2, None, '___sec14'), ('A simple classification problem', 2, None, '___sec15')]} end of tocinfo --> @@ -106,7 +106,7 @@ MathJax.Hub.Config({
+ + + + +
+ + +
import numpy as np
+from sklearn import datasets, linear_model
+import matplotlib.pyplot as plt
+
+
+def generate_data():
+ np.random.seed(0)
+ X, y = datasets.make_moons(200, noise=0.20)
+ return X, y
+
+
+def visualize(X, y, clf):
+ # plt.scatter(X[:, 0], X[:, 1], s=40, c=y, cmap=plt.cm.Spectral)
+ # plt.show()
+ plot_decision_boundary(lambda x: clf.predict(x), X, y)
+ plt.title("Logistic Regression")
+
+
+def plot_decision_boundary(pred_func, X, y):
+ # Set min and max values and give it some padding
+ x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
+ y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
+ h = 0.01
+ # Generate a grid of points with distance h between them
+ xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
+ # Predict the function value for the whole gid
+ Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
+ Z = Z.reshape(xx.shape)
+ # Plot the contour and training examples
+ plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
+ plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
+ plt.show()
+
+
+def classify(X, y):
+ clf = linear_model.LogisticRegressionCV()
+ clf.fit(X, y)
+ return clf
+
+
+def main():
+ X, y = generate_data()
+ # visualize(X, y)
+ clf = classify(X, y)
+ visualize(X, y, clf)
+
+
+if __name__ == "__main__":
+ main()
++ +
+ +
+ + +-
diff --git a/doc/pub/LogReg/html/LogReg-reveal.html b/doc/pub/LogReg/html/LogReg-reveal.html index 2615d7966..9534ff30a 100644 --- a/doc/pub/LogReg/html/LogReg-reveal.html +++ b/doc/pub/LogReg/html/LogReg-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({
-
@@ -448,19 +448,87 @@ $$
+Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors
+
+Till now we have mainly focused on two classes, the so-called binary system. Suppose we wish to extend to \( K \) classes.
+Let us for the sake of simplicity assume we have only two predictors. We have then following model
+
-Newton's method and gradient descent methods
+In our discussion of neural networks we will encounter the above again in terms of the so-called Softmax function.
+
+
+The softmax function is used in various multiclass classification
+methods, such as multinomial logistic regression (also known as
+softmax regression), multiclass linear discriminant
+analysis, naive Bayes classifiers, and artificial neural networks.
+Specifically, in multinomial logistic regression and linear
+discriminant analysis, the input to the function is the result of \( K \)
+distinct linear functions, and the predicted probability for the \( k \)-th
+class given a sample vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two predictors):
+
+
+To find the optimal parameters we would typically use a gradient descent method.
+Newton's method and gradient descent methods are discussed in the material on optimization methods.
Extending to more predictors
+
+
+$$
+\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
+$$
+
+
+Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to
+
+$$
+p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
+$$
+
Including more classes
+
+
+$$
+\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1,
+$$
+
+
+
+$$
+\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1,
+$$
+
+
+and so on till the class \( C=K-1 \) class
+
+$$
+\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1,
+$$
+
+
+and the model is specified in term of \( K-1 \) so-called log-odds or logit transformations.
Optimizing the cost function
+The Softmax function
+$$
+p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}.
+$$
+
+
+It is easy to extend to more predictors. The final class is
+
+$$
+p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}},
+$$
+
+
+and they sum to one. Our earlier discussions were all specialized to the case with two classes only. It is easy to see from the above that what we derived earlier is compatible with these equations.
+
+
-
@@ -358,18 +358,72 @@ $$
+Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors +$$ +\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. +$$ + +Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to +$$ +p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. +$$ +
-
+Till now we have mainly focused on two classes, the so-called binary system. Suppose we wish to extend to \( K \) classes.
+Let us for the sake of simplicity assume we have only two predictors. We have then following model
+$$
+\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1,
+$$
-
-Newton's method and gradient descent methods
+
+
+
+In our discussion of neural networks we will encounter the above again in terms of the so-called Softmax function. + +
+The softmax function is used in various multiclass classification +methods, such as multinomial logistic regression (also known as +softmax regression), multiclass linear discriminant +analysis, naive Bayes classifiers, and artificial neural networks. +Specifically, in multinomial logistic regression and linear +discriminant analysis, the input to the function is the result of \( K \) +distinct linear functions, and the predicted probability for the \( k \)-th +class given a sample vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two predictors): + +$$ +p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}. +$$ + +It is easy to extend to more predictors. The final class is +$$ +p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}, +$$ + +and they sum to one. Our earlier discussions were all specialized to the case with two classes only. It is easy to see from the above that what we derived earlier is compatible with these equations. + +
+To find the optimal parameters we would typically use a gradient descent method. +Newton's method and gradient descent methods are discussed in the material on optimization methods.
diff --git a/doc/pub/LogReg/html/LogReg.html b/doc/pub/LogReg/html/LogReg.html
index 02994c5c2..3a86df613 100644
--- a/doc/pub/LogReg/html/LogReg.html
+++ b/doc/pub/LogReg/html/LogReg.html
@@ -52,7 +52,7 @@ div { text-align: justify; text-justify: inter-word; }
('A more compact expression', 2, None, '___sec10'),
('Extending to more predictors', 2, None, '___sec11'),
('Including more classes', 2, None, '___sec12'),
- ('Optimizing the cost function', 2, None, '___sec13'),
+ ('The Softmax function', 2, None, '___sec13'),
('A _scikit-learn_ example', 2, None, '___sec14'),
('A simple classification problem', 2, None, '___sec15')]}
end of tocinfo -->
@@ -96,7 +96,7 @@ MathJax.Hub.Config({
-
@@ -363,18 +363,72 @@ $$
+Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors +$$ +\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. +$$ + +Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to +$$ +p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. +$$ +
-
+Till now we have mainly focused on two classes, the so-called binary system. Suppose we wish to extend to \( K \) classes.
+Let us for the sake of simplicity assume we have only two predictors. We have then following model
+$$
+\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1,
+$$
-
-Newton's method and gradient descent methods
+
+
+
+In our discussion of neural networks we will encounter the above again in terms of the so-called Softmax function. + +
+The softmax function is used in various multiclass classification +methods, such as multinomial logistic regression (also known as +softmax regression), multiclass linear discriminant +analysis, naive Bayes classifiers, and artificial neural networks. +Specifically, in multinomial logistic regression and linear +discriminant analysis, the input to the function is the result of \( K \) +distinct linear functions, and the predicted probability for the \( k \)-th +class given a sample vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two predictors): + +$$ +p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}. +$$ + +It is easy to extend to more predictors. The final class is +$$ +p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}, +$$ + +and they sum to one. Our earlier discussions were all specialized to the case with two classes only. It is easy to see from the above that what we derived earlier is compatible with these equations. + +
+To find the optimal parameters we would typically use a gradient descent method. +Newton's method and gradient descent methods are discussed in the material on optimization methods.
diff --git a/doc/pub/LogReg/ipynb/LogReg.ipynb b/doc/pub/LogReg/ipynb/LogReg.ipynb
index baaa9264e..cf3532b4d 100644
--- a/doc/pub/LogReg/ipynb/LogReg.ipynb
+++ b/doc/pub/LogReg/ipynb/LogReg.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Sep 26, 2018**\n",
+ "Date: **Oct 11, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -374,11 +374,147 @@
"source": [
"## Extending to more predictors\n",
"\n",
+ "Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\log{ \\frac{p(\\hat{\\beta}\\hat{x})}{1-p(\\hat{\\beta}\\hat{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here we defined $\\hat{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\hat{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "p(\\hat{\\beta}\\hat{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
"## Including more classes\n",
"\n",
- "## Optimizing the cost function\n",
+ "Till now we have mainly focused on two classes, the so-called binary system. Suppose we wish to extend to $K$ classes.\n",
+ "Let us for the sake of simplicity assume we have only two predictors. We have then following model"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "1\n",
+ "5\n",
+ " \n",
+ "<\n",
+ "<\n",
+ "<\n",
+ "!\n",
+ "!\n",
+ "M\n",
+ "A\n",
+ "T\n",
+ "H\n",
+ "_\n",
+ "B\n",
+ "L\n",
+ "O\n",
+ "C\n",
+ "K"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and so on till the class $C=K-1$ class"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and the model is specified in term of $K-1$ so-called log-odds or **logit** transformations.\n",
"\n",
- "Newton's method and gradient descent methods\n",
+ "\n",
+ "## The Softmax function\n",
+ "\n",
+ "In our discussion of neural networks we will encounter the above again in terms of the so-called **Softmax** function.\n",
+ "\n",
+ "The softmax function is used in various multiclass classification\n",
+ "methods, such as multinomial logistic regression (also known as\n",
+ "softmax regression), multiclass linear discriminant\n",
+ "analysis, naive Bayes classifiers, and artificial neural networks.\n",
+ "Specifically, in multinomial logistic regression and linear\n",
+ "discriminant analysis, the input to the function is the result of $K$\n",
+ "distinct linear functions, and the predicted probability for the $k$-th\n",
+ "class given a sample vector $\\hat{x}$ and a weighting vector $\\hat{\\beta}$ is (with two predictors):"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "It is easy to extend to more predictors. The final class is"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and they sum to one. Our earlier discussions were all specialized to the case with two classes only. It is easy to see from the above that what we derived earlier is compatible with these equations. \n",
+ "\n",
+ "To find the optimal parameters we would typically use a gradient descent method.\n",
+ "Newton's method and gradient descent methods are discussed in the material on [optimization methods](https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html). \n",
"\n",
"\n",
"\n",
@@ -388,7 +524,9 @@
{
"cell_type": "code",
"execution_count": 1,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -423,7 +561,9 @@
{
"cell_type": "code",
"execution_count": 2,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import numpy as np\n",
@@ -478,25 +618,7 @@
]
}
],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.7.0"
- }
- },
+ "metadata": {},
"nbformat": 4,
"nbformat_minor": 2
}
diff --git a/doc/pub/LogReg/ipynb/ipynb-LogReg-src.tar.gz b/doc/pub/LogReg/ipynb/ipynb-LogReg-src.tar.gz
index c730247af..bd49f72aa 100644
Binary files a/doc/pub/LogReg/ipynb/ipynb-LogReg-src.tar.gz and b/doc/pub/LogReg/ipynb/ipynb-LogReg-src.tar.gz differ
diff --git a/doc/pub/LogReg/pdf/LogReg-beamer-handouts2x3.pdf b/doc/pub/LogReg/pdf/LogReg-beamer-handouts2x3.pdf
index dfca58001..e7e36e3de 100644
Binary files a/doc/pub/LogReg/pdf/LogReg-beamer-handouts2x3.pdf and b/doc/pub/LogReg/pdf/LogReg-beamer-handouts2x3.pdf differ
diff --git a/doc/pub/LogReg/pdf/LogReg-beamer.pdf b/doc/pub/LogReg/pdf/LogReg-beamer.pdf
index 4b3c6ae9e..ce0339144 100644
Binary files a/doc/pub/LogReg/pdf/LogReg-beamer.pdf and b/doc/pub/LogReg/pdf/LogReg-beamer.pdf differ
diff --git a/doc/pub/LogReg/pdf/LogReg-minted.pdf b/doc/pub/LogReg/pdf/LogReg-minted.pdf
index 21ceff7ff..66f878004 100644
Binary files a/doc/pub/LogReg/pdf/LogReg-minted.pdf and b/doc/pub/LogReg/pdf/LogReg-minted.pdf differ
diff --git a/doc/src/LogisticRegression/LogReg.do.txt b/doc/src/LogisticRegression/LogReg.do.txt
index fed5ac392..cb5c7ef3f 100644
--- a/doc/src/LogisticRegression/LogReg.do.txt
+++ b/doc/src/LogisticRegression/LogReg.do.txt
@@ -240,13 +240,72 @@ $p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a com
!split
===== Extending to more predictors =====
+Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors
+!bt
+\[
+\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
+\]
+!et
+Here we defined $\hat{x}=[1,x_1,x_2,\dots,x_p]$ and $\hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to
+!bt
+\[
+p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
+\]
+!et
+
!split
===== Including more classes =====
-!split
-===== Optimizing the cost function =====
+Till now we have mainly focused on two classes, the so-called binary system. Suppose we wish to extend to $K$ classes.
+Let us for the sake of simplicity assume we have only two predictors. We have then following model
+!bt
+\[
+\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1,
+\]
+!et
+!bt
+\[
+\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1,
+\]
+!et
+and so on till the class $C=K-1$ class
+!bt
+\[
+\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1,
+\]
+!et
+and the model is specified in term of $K-1$ so-called log-odds or _logit_ transformations.
-Newton's method and gradient descent methods
+
+!split
+===== The Softmax function =====
+
+In our discussion of neural networks we will encounter the above again in terms of the so-called _Softmax_ function.
+
+The softmax function is used in various multiclass classification
+methods, such as multinomial logistic regression (also known as
+softmax regression), multiclass linear discriminant
+analysis, naive Bayes classifiers, and artificial neural networks.
+Specifically, in multinomial logistic regression and linear
+discriminant analysis, the input to the function is the result of $K$
+distinct linear functions, and the predicted probability for the $k$-th
+class given a sample vector $\hat{x}$ and a weighting vector $\hat{\beta}$ is (with two predictors):
+
+!bt
+\[
+p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}.
+\]
+!et
+It is easy to extend to more predictors. The final class is
+!bt
+\[
+p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}},
+\]
+!et
+and they sum to one. Our earlier discussions were all specialized to the case with two classes only. It is easy to see from the above that what we derived earlier is compatible with these equations.
+
+To find the optimal parameters we would typically use a gradient descent method.
+Newton's method and gradient descent methods are discussed in the material on "optimization methods":"https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html".