The goal is to predict the -output classes from the design matrix \( \hat{X}\in\mathbb{R}^{n\times p} \) +output classes from the design matrix \( \boldsymbol{X}\in\mathbb{R}^{n\times p} \) made of \( n \) samples, each of which carries \( p \) features or predictors. The primary goal is to identify the classes to which new unseen samples belong. diff --git a/doc/pub/week38/html/._week38-bs028.html b/doc/pub/week38/html/._week38-bs028.html index 92fd9cd9e..be27c1b95 100644 --- a/doc/pub/week38/html/._week38-bs028.html +++ b/doc/pub/week38/html/._week38-bs028.html @@ -427,13 +427,13 @@ We would then have our weighted linear combination, namely $$ \begin{equation} -\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon}, +\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\beta} + \boldsymbol{\epsilon}, \tag{13} \end{equation} $$ -where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our -\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators/predictors. +where \( \boldsymbol{y} \) is a vector representing the possible outcomes, \( \boldsymbol{X} \) is our +\( n\times p \) design matrix and \( \boldsymbol{\beta} \) represents our estimators/predictors.
diff --git a/doc/pub/week38/html/._week38-bs034.html b/doc/pub/week38/html/._week38-bs034.html index c3fc78420..2a027a1ff 100644 --- a/doc/pub/week38/html/._week38-bs034.html +++ b/doc/pub/week38/html/._week38-bs034.html @@ -420,17 +420,17 @@ MathJax.Hub.Config({ We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities $$ \begin{align*} -p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ -p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}), +p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}), \end{align*} $$ -where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \). +where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
Note that we used $$ -p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}). +p(y_i=0\vert x_i, \boldsymbol{\beta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\beta}). $$
diff --git a/doc/pub/week38/html/._week38-bs035.html b/doc/pub/week38/html/._week38-bs035.html index becd83eea..fa33d4425 100644 --- a/doc/pub/week38/html/._week38-bs035.html +++ b/doc/pub/week38/html/._week38-bs035.html @@ -425,13 +425,13 @@ the probability of seeing the observed data. We can then approximate the likelihood in terms of the product of the individual probabilities of a specific outcome \( y_i \), that is $$ \begin{align*} -P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\ +P(\mathcal{D}|\boldsymbol{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]^{1-y_i}\nonumber \\ \end{align*} $$ from which we obtain the log-likelihood and our cost/loss function $$ -\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right). +\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]\right). $$
diff --git a/doc/pub/week38/html/._week38-bs036.html b/doc/pub/week38/html/._week38-bs036.html index 744d676b1..3ce3951cb 100644 --- a/doc/pub/week38/html/._week38-bs036.html +++ b/doc/pub/week38/html/._week38-bs036.html @@ -419,14 +419,14 @@ MathJax.Hub.Config({
Reordering the logarithms, we can rewrite the cost/loss function as $$ -\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). $$
The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \beta \). Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that $$ -\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\mathcal{C}(\boldsymbol{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). $$ This equation is known in statistics as the cross entropy. Finally, we note that just as in linear regression, diff --git a/doc/pub/week38/html/._week38-bs037.html b/doc/pub/week38/html/._week38-bs037.html index 6448312d3..0d4a49efc 100644 --- a/doc/pub/week38/html/._week38-bs037.html +++ b/doc/pub/week38/html/._week38-bs037.html @@ -417,7 +417,7 @@ MathJax.Hub.Config({
-The cross entropy is a convex function of the weights \( \hat{\beta} \) and, +The cross entropy is a convex function of the weights \( \boldsymbol{\beta} \) and, therefore, any local minimizer is a global minimizer.
@@ -425,12 +425,12 @@ Minimizing this cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \) we obtain $$ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), $$ and $$ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). $$
diff --git a/doc/pub/week38/html/._week38-bs038.html b/doc/pub/week38/html/._week38-bs038.html index 6b2a6fb1b..75f6a1403 100644 --- a/doc/pub/week38/html/._week38-bs038.html +++ b/doc/pub/week38/html/._week38-bs038.html @@ -417,21 +417,21 @@ MathJax.Hub.Config({
-Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an -\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a -vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first +Let us now define a vector \( \boldsymbol{y} \) with \( n \) elements \( y_i \), an +\( n\times p \) matrix \( \boldsymbol{X} \) which contains the \( x_i \) values and a +vector \( \boldsymbol{p} \) of fitted probabilities \( p(y_i\vert x_i,\boldsymbol{\beta}) \). We can rewrite in a more compact form the first derivative of cost function as $$ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right). $$
-If we in addition define a diagonal matrix \( \hat{W} \) with elements -\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as +If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements +\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as $$ -\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}. $$
diff --git a/doc/pub/week38/html/._week38-bs039.html b/doc/pub/week38/html/._week38-bs039.html index 2956657ba..52ed8b393 100644 --- a/doc/pub/week38/html/._week38-bs039.html +++ b/doc/pub/week38/html/._week38-bs039.html @@ -419,12 +419,12 @@ MathJax.Hub.Config({
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors $$ -\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. +\log{ \frac{p(\boldsymbol{\beta}\boldsymbol{x})}{1-p(\boldsymbol{\beta}\boldsymbol{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. $$ -Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to +Here we defined \( \boldsymbol{x}=[1,x_1,x_2,\dots,x_p] \) and \( \boldsymbol{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to $$ -p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. +p(\boldsymbol{\beta}\boldsymbol{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. $$
diff --git a/doc/pub/week38/html/._week38-bs041.html b/doc/pub/week38/html/._week38-bs041.html index 011541c40..efa2cd448 100644 --- a/doc/pub/week38/html/._week38-bs041.html +++ b/doc/pub/week38/html/._week38-bs041.html @@ -428,7 +428,7 @@ Bayes classifiers, and artificial neural networks. Specifically, in multinomial logistic regression and linear discriminant analysis, the input to the function is the result of \( K \) distinct linear functions, and the predicted probability for the \( k \)-th class given a sample -vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two +vector \( \boldsymbol{x} \) and a weighting vector \( \boldsymbol{\beta} \) is (with two predictors): $$ diff --git a/doc/pub/week38/html/week38-reveal.html b/doc/pub/week38/html/week38-reveal.html index 77dfcb1bd..82b585c31 100644 --- a/doc/pub/week38/html/week38-reveal.html +++ b/doc/pub/week38/html/week38-reveal.html @@ -1470,11 +1470,11 @@ In linear regression our main interest was centered on learning the coefficients of a functional fit (say a polynomial) in order to be able to predict the response of a continuous variable on some unseen data. The fit to the continuous variable \( y_i \) is based on some -independent variables \( \hat{x}_i \). Linear regression resulted in +independent variables \( \boldsymbol{x}_i \). Linear regression resulted in analytical expressions for standard ordinary Least Squares or Ridge regression (in terms of matrices to invert) for several quantities, ranging from the variance and thereby the confidence intervals of the -parameters \( \hat{\beta} \) to the mean squared error. If we can invert +parameters \( \boldsymbol{\beta} \) to the mean squared error. If we can invert the product of the design matrices, linear regression gives then a simple recipe for fitting our data. @@ -1510,7 +1510,7 @@ failure etc. Logistic regression will also serve as our stepping stone towards neural network algorithms and supervised deep learning. For logistic learning, the minimization of the cost function leads to a non-linear -equation in the parameters \( \hat{\beta} \). The optimization of the +equation in the parameters \( \boldsymbol{\beta} \). The optimization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the @@ -1534,7 +1534,7 @@ from \( k=0,\dots,K-1 \) (i.e. \( K \) classes).
The goal is to predict the -output classes from the design matrix \( \hat{X}\in\mathbb{R}^{n\times p} \) +output classes from the design matrix \( \boldsymbol{X}\in\mathbb{R}^{n\times p} \) made of \( n \) samples, each of which carries \( p \) features or predictors. The primary goal is to identify the classes to which new unseen samples belong. @@ -1568,14 +1568,14 @@ weighted linear combination, namely
$$
\begin{equation}
-\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon},
+\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\beta} + \boldsymbol{\epsilon},
\tag{13}
\end{equation}
$$
-where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our
-\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators/predictors.
+where \( \boldsymbol{y} \) is a vector representing the possible outcomes, \( \boldsymbol{X} \) is our
+\( n\times p \) design matrix and \( \boldsymbol{\beta} \) represents our estimators/predictors.
@@ -1809,19 +1809,19 @@ We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \).
$$
\begin{align*}
-p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
-p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
+p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
+p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
\end{align*}
$$
-where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
+where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
Note that we used
$$
-p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
+p(y_i=0\vert x_i, \boldsymbol{\beta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\beta}).
$$
@@ -1840,7 +1840,7 @@ likelihood in terms of the product of the individual probabilities of a specific
$$
\begin{align*}
-P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
+P(\mathcal{D}|\boldsymbol{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
$$
@@ -1848,7 +1848,7 @@ $$
from which we obtain the log-likelihood and our cost/loss function
$$
-\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
+\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]\right).
$$
@@ -1861,7 +1861,7 @@ $$
Reordering the logarithms, we can rewrite the cost/loss function as
$$
-\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
+\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$
@@ -1870,7 +1870,7 @@ The maximum likelihood estimator is defined as the set of parameters that maximi
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
$$
-\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
+\mathcal{C}(\boldsymbol{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$
@@ -1883,7 +1883,7 @@ in practice we often supplement the cross-entropy with additional regularization
-The cross entropy is a convex function of the weights \( \hat{\beta} \) and, +The cross entropy is a convex function of the weights \( \boldsymbol{\beta} \) and, therefore, any local minimizer is a global minimizer.
@@ -1892,14 +1892,14 @@ cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \)
$$
-\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
+\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
$$
and
$$
-\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
+\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$
@@ -1909,24 +1909,24 @@ $$
-Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an -\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a -vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first +Let us now define a vector \( \boldsymbol{y} \) with \( n \) elements \( y_i \), an +\( n\times p \) matrix \( \boldsymbol{X} \) which contains the \( x_i \) values and a +vector \( \boldsymbol{p} \) of fitted probabilities \( p(y_i\vert x_i,\boldsymbol{\beta}) \). We can rewrite in a more compact form the first derivative of cost function as
$$
-\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
+\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
$$
-If we in addition define a diagonal matrix \( \hat{W} \) with elements -\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as +If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements +\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as
$$
-\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
+\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
$$
@@ -1939,14 +1939,14 @@ $$
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors
$$
-\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
+\log{ \frac{p(\boldsymbol{\beta}\boldsymbol{x})}{1-p(\boldsymbol{\beta}\boldsymbol{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
$$
-Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to
+Here we defined \( \boldsymbol{x}=[1,x_1,x_2,\dots,x_p] \) and \( \boldsymbol{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to
$$
-p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
+p(\boldsymbol{\beta}\boldsymbol{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
$$
@@ -2001,7 +2001,7 @@ Bayes classifiers, and artificial neural networks. Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of \( K \) distinct linear functions,
and the predicted probability for the \( k \)-th class given a sample
-vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two
+vector \( \boldsymbol{x} \) and a weighting vector \( \boldsymbol{\beta} \) is (with two
predictors):
diff --git a/doc/pub/week38/html/week38-solarized.html b/doc/pub/week38/html/week38-solarized.html
index 65b61ff31..8fa62e020 100644
--- a/doc/pub/week38/html/week38-solarized.html
+++ b/doc/pub/week38/html/week38-solarized.html
@@ -1577,11 +1577,11 @@ In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable \( y_i \) is based on some
-independent variables \( \hat{x}_i \). Linear regression resulted in
+independent variables \( \boldsymbol{x}_i \). Linear regression resulted in
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
-parameters \( \hat{\beta} \) to the mean squared error. If we can invert
+parameters \( \boldsymbol{\beta} \) to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.
@@ -1617,7 +1617,7 @@ failure etc.
Logistic regression will also serve as our stepping stone towards
neural network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
-equation in the parameters \( \hat{\beta} \). The optimization of the
+equation in the parameters \( \boldsymbol{\beta} \). The optimization of the
problem calls therefore for minimization algorithms. This forms the
bottle neck of all machine learning algorithms, namely how to find
reliable minima of a multi-variable function. This leads us to the
@@ -1641,7 +1641,7 @@ from \( k=0,\dots,K-1 \) (i.e. \( K \) classes).
The goal is to predict the -output classes from the design matrix \( \hat{X}\in\mathbb{R}^{n\times p} \) +output classes from the design matrix \( \boldsymbol{X}\in\mathbb{R}^{n\times p} \) made of \( n \) samples, each of which carries \( p \) features or predictors. The primary goal is to identify the classes to which new unseen samples belong. @@ -1672,13 +1672,13 @@ We would then have our weighted linear combination, namely $$ \begin{equation} -\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon}, +\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\beta} + \boldsymbol{\epsilon}, \label{_auto13} \end{equation} $$ -where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our -\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators/predictors. +where \( \boldsymbol{y} \) is a vector representing the possible outcomes, \( \boldsymbol{X} \) is our +\( n\times p \) design matrix and \( \boldsymbol{\beta} \) represents our estimators/predictors.
@@ -1905,17 +1905,17 @@ plt.show()
We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities
$$
\begin{align*}
-p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
-p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
+p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
+p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
\end{align*}
$$
-where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
+where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
Note that we used $$ -p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}). +p(y_i=0\vert x_i, \boldsymbol{\beta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\beta}). $$
@@ -1932,13 +1932,13 @@ the probability of seeing the observed data. We can then approximate the likelihood in terms of the product of the individual probabilities of a specific outcome \( y_i \), that is $$ \begin{align*} -P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\ +P(\mathcal{D}|\boldsymbol{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]^{1-y_i}\nonumber \\ \end{align*} $$ from which we obtain the log-likelihood and our cost/loss function $$ -\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right). +\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]\right). $$
@@ -1949,14 +1949,14 @@ $$
Reordering the logarithms, we can rewrite the cost/loss function as $$ -\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). $$
The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \beta \). Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that $$ -\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\mathcal{C}(\boldsymbol{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). $$ This equation is known in statistics as the cross entropy. Finally, we note that just as in linear regression, @@ -1968,7 +1968,7 @@ in practice we often supplement the cross-entropy with additional regularization
-The cross entropy is a convex function of the weights \( \hat{\beta} \) and, +The cross entropy is a convex function of the weights \( \boldsymbol{\beta} \) and, therefore, any local minimizer is a global minimizer.
@@ -1976,12 +1976,12 @@ Minimizing this cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \) we obtain $$ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), $$ and $$ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). $$
@@ -1990,21 +1990,21 @@ $$
-Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an -\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a -vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first +Let us now define a vector \( \boldsymbol{y} \) with \( n \) elements \( y_i \), an +\( n\times p \) matrix \( \boldsymbol{X} \) which contains the \( x_i \) values and a +vector \( \boldsymbol{p} \) of fitted probabilities \( p(y_i\vert x_i,\boldsymbol{\beta}) \). We can rewrite in a more compact form the first derivative of cost function as $$ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right). $$
-If we in addition define a diagonal matrix \( \hat{W} \) with elements -\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as +If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements +\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as $$ -\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}. $$
@@ -2015,12 +2015,12 @@ $$
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors $$ -\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. +\log{ \frac{p(\boldsymbol{\beta}\boldsymbol{x})}{1-p(\boldsymbol{\beta}\boldsymbol{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. $$ -Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to +Here we defined \( \boldsymbol{x}=[1,x_1,x_2,\dots,x_p] \) and \( \boldsymbol{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to $$ -p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. +p(\boldsymbol{\beta}\boldsymbol{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. $$
@@ -2068,7 +2068,7 @@ Bayes classifiers, and artificial neural networks. Specifically, in multinomial logistic regression and linear discriminant analysis, the input to the function is the result of \( K \) distinct linear functions, and the predicted probability for the \( k \)-th class given a sample -vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two +vector \( \boldsymbol{x} \) and a weighting vector \( \boldsymbol{\beta} \) is (with two predictors): $$ diff --git a/doc/pub/week38/html/week38.html b/doc/pub/week38/html/week38.html index 04534cf80..ce3344d1d 100644 --- a/doc/pub/week38/html/week38.html +++ b/doc/pub/week38/html/week38.html @@ -1582,11 +1582,11 @@ In linear regression our main interest was centered on learning the coefficients of a functional fit (say a polynomial) in order to be able to predict the response of a continuous variable on some unseen data. The fit to the continuous variable \( y_i \) is based on some -independent variables \( \hat{x}_i \). Linear regression resulted in +independent variables \( \boldsymbol{x}_i \). Linear regression resulted in analytical expressions for standard ordinary Least Squares or Ridge regression (in terms of matrices to invert) for several quantities, ranging from the variance and thereby the confidence intervals of the -parameters \( \hat{\beta} \) to the mean squared error. If we can invert +parameters \( \boldsymbol{\beta} \) to the mean squared error. If we can invert the product of the design matrices, linear regression gives then a simple recipe for fitting our data. @@ -1622,7 +1622,7 @@ failure etc. Logistic regression will also serve as our stepping stone towards neural network algorithms and supervised deep learning. For logistic learning, the minimization of the cost function leads to a non-linear -equation in the parameters \( \hat{\beta} \). The optimization of the +equation in the parameters \( \boldsymbol{\beta} \). The optimization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the @@ -1646,7 +1646,7 @@ from \( k=0,\dots,K-1 \) (i.e. \( K \) classes).
The goal is to predict the -output classes from the design matrix \( \hat{X}\in\mathbb{R}^{n\times p} \) +output classes from the design matrix \( \boldsymbol{X}\in\mathbb{R}^{n\times p} \) made of \( n \) samples, each of which carries \( p \) features or predictors. The primary goal is to identify the classes to which new unseen samples belong. @@ -1677,13 +1677,13 @@ We would then have our weighted linear combination, namely $$ \begin{equation} -\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon}, +\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\beta} + \boldsymbol{\epsilon}, \label{_auto13} \end{equation} $$ -where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our -\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators/predictors. +where \( \boldsymbol{y} \) is a vector representing the possible outcomes, \( \boldsymbol{X} \) is our +\( n\times p \) design matrix and \( \boldsymbol{\beta} \) represents our estimators/predictors.
@@ -1910,17 +1910,17 @@ plt.show()
We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities
$$
\begin{align*}
-p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
-p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
+p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
+p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
\end{align*}
$$
-where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
+where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
Note that we used $$ -p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}). +p(y_i=0\vert x_i, \boldsymbol{\beta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\beta}). $$
@@ -1937,13 +1937,13 @@ the probability of seeing the observed data. We can then approximate the likelihood in terms of the product of the individual probabilities of a specific outcome \( y_i \), that is $$ \begin{align*} -P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\ +P(\mathcal{D}|\boldsymbol{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]^{1-y_i}\nonumber \\ \end{align*} $$ from which we obtain the log-likelihood and our cost/loss function $$ -\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right). +\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]\right). $$
@@ -1954,14 +1954,14 @@ $$
Reordering the logarithms, we can rewrite the cost/loss function as $$ -\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). $$
The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \beta \). Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that $$ -\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\mathcal{C}(\boldsymbol{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). $$ This equation is known in statistics as the cross entropy. Finally, we note that just as in linear regression, @@ -1973,7 +1973,7 @@ in practice we often supplement the cross-entropy with additional regularization
-The cross entropy is a convex function of the weights \( \hat{\beta} \) and, +The cross entropy is a convex function of the weights \( \boldsymbol{\beta} \) and, therefore, any local minimizer is a global minimizer.
@@ -1981,12 +1981,12 @@ Minimizing this cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \) we obtain $$ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), $$ and $$ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). $$
@@ -1995,21 +1995,21 @@ $$
-Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an -\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a -vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first +Let us now define a vector \( \boldsymbol{y} \) with \( n \) elements \( y_i \), an +\( n\times p \) matrix \( \boldsymbol{X} \) which contains the \( x_i \) values and a +vector \( \boldsymbol{p} \) of fitted probabilities \( p(y_i\vert x_i,\boldsymbol{\beta}) \). We can rewrite in a more compact form the first derivative of cost function as $$ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right). $$
-If we in addition define a diagonal matrix \( \hat{W} \) with elements -\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as +If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements +\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as $$ -\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}. $$
@@ -2020,12 +2020,12 @@ $$
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors $$ -\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. +\log{ \frac{p(\boldsymbol{\beta}\boldsymbol{x})}{1-p(\boldsymbol{\beta}\boldsymbol{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. $$ -Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to +Here we defined \( \boldsymbol{x}=[1,x_1,x_2,\dots,x_p] \) and \( \boldsymbol{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to $$ -p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. +p(\boldsymbol{\beta}\boldsymbol{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. $$
@@ -2073,7 +2073,7 @@ Bayes classifiers, and artificial neural networks. Specifically, in multinomial logistic regression and linear discriminant analysis, the input to the function is the result of \( K \) distinct linear functions, and the predicted probability for the \( k \)-th class given a sample -vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two +vector \( \boldsymbol{x} \) and a weighting vector \( \boldsymbol{\beta} \) is (with two predictors): $$ diff --git a/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz b/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz index bce3b2666..d8c28c5c9 100644 Binary files a/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz and b/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz differ diff --git a/doc/pub/week38/ipynb/week38.ipynb b/doc/pub/week38/ipynb/week38.ipynb index 665c54c68..99ebbf473 100644 --- a/doc/pub/week38/ipynb/week38.ipynb +++ b/doc/pub/week38/ipynb/week38.ipynb @@ -1804,11 +1804,11 @@ "coefficients of a functional fit (say a polynomial) in order to be\n", "able to predict the response of a continuous variable on some unseen\n", "data. The fit to the continuous variable $y_i$ is based on some\n", - "independent variables $\\hat{x}_i$. Linear regression resulted in\n", + "independent variables $\\boldsymbol{x}_i$. Linear regression resulted in\n", "analytical expressions for standard ordinary Least Squares or Ridge\n", "regression (in terms of matrices to invert) for several quantities,\n", "ranging from the variance and thereby the confidence intervals of the\n", - "parameters $\\hat{\\beta}$ to the mean squared error. If we can invert\n", + "parameters $\\boldsymbol{\\beta}$ to the mean squared error. If we can invert\n", "the product of the design matrices, linear regression gives then a\n", "simple recipe for fitting our data.\n", "\n", @@ -1837,7 +1837,7 @@ "Logistic regression will also serve as our stepping stone towards\n", "neural network algorithms and supervised deep learning. For logistic\n", "learning, the minimization of the cost function leads to a non-linear\n", - "equation in the parameters $\\hat{\\beta}$. The optimization of the\n", + "equation in the parameters $\\boldsymbol{\\beta}$. The optimization of the\n", "problem calls therefore for minimization algorithms. This forms the\n", "bottle neck of all machine learning algorithms, namely how to find\n", "reliable minima of a multi-variable function. This leads us to the\n", @@ -1857,7 +1857,7 @@ "from $k=0,\\dots,K-1$ (i.e. $K$ classes).\n", "\n", "The goal is to predict the\n", - "output classes from the design matrix $\\hat{X}\\in\\mathbb{R}^{n\\times p}$\n", + "output classes from the design matrix $\\boldsymbol{X}\\in\\mathbb{R}^{n\\times p}$\n", "made of $n$ samples, each of which carries $p$ features or predictors. The\n", "primary goal is to identify the classes to which new unseen samples\n", "belong.\n", @@ -1901,7 +1901,7 @@ "\n", "$$\n", "\\begin{equation}\n", - "\\hat{y} = \\hat{X}^T\\hat{\\beta} + \\hat{\\epsilon},\n", + "\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon},\n", "\\label{_auto13} \\tag{13}\n", "\\end{equation}\n", "$$" @@ -1911,8 +1911,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where $\\hat{y}$ is a vector representing the possible outcomes, $\\hat{X}$ is our\n", - "$n\\times p$ design matrix and $\\hat{\\beta}$ represents our estimators/predictors.\n", + "where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n", + "$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors.\n", "\n", "## Some selected properties\n", "\n", @@ -2177,8 +2177,8 @@ "source": [ "$$\n", "\\begin{align*}\n", - "p(y_i=1|x_i,\\hat{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n", - "p(y_i=0|x_i,\\hat{\\beta}) &= 1 - p(y_i=1|x_i,\\hat{\\beta}),\n", + "p(y_i=1|x_i,\\boldsymbol{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n", + "p(y_i=0|x_i,\\boldsymbol{\\beta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\beta}),\n", "\\end{align*}\n", "$$" ] @@ -2187,7 +2187,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where $\\hat{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n", + "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n", "\n", "Note that we used" ] @@ -2197,7 +2197,7 @@ "metadata": {}, "source": [ "$$\n", - "p(y_i=0\\vert x_i, \\hat{\\beta}) = 1-p(y_i=1\\vert x_i, \\hat{\\beta}).\n", + "p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n", "$$" ] }, @@ -2222,7 +2222,7 @@ "source": [ "$$\n", "\\begin{align*}\n", - "P(\\mathcal{D}|\\hat{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\hat{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n", + "P(\\mathcal{D}|\\boldsymbol{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\boldsymbol{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n", "\\end{align*}\n", "$$" ] @@ -2239,7 +2239,7 @@ "metadata": {}, "source": [ "$$\n", - "\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\hat{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]\\right).\n", + "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n", "$$" ] }, @@ -2257,7 +2257,7 @@ "metadata": {}, "source": [ "$$\n", - "\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", + "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", "$$" ] }, @@ -2274,7 +2274,7 @@ "metadata": {}, "source": [ "$$\n", - "\\mathcal{C}(\\hat{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", + "\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", "$$" ] }, @@ -2287,7 +2287,7 @@ "\n", "## Minimizing the cross entropy\n", "\n", - "The cross entropy is a convex function of the weights $\\hat{\\beta}$ and,\n", + "The cross entropy is a convex function of the weights $\\boldsymbol{\\beta}$ and,\n", "therefore, any local minimizer is a global minimizer. \n", "\n", "\n", @@ -2300,7 +2300,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n", + "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n", "$$" ] }, @@ -2316,7 +2316,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n", + "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n", "$$" ] }, @@ -2326,9 +2326,9 @@ "source": [ "## A more compact expression\n", "\n", - "Let us now define a vector $\\hat{y}$ with $n$ elements $y_i$, an\n", - "$n\\times p$ matrix $\\hat{X}$ which contains the $x_i$ values and a\n", - "vector $\\hat{p}$ of fitted probabilities $p(y_i\\vert x_i,\\hat{\\beta})$. We can rewrite in a more compact form the first\n", + "Let us now define a vector $\\boldsymbol{y}$ with $n$ elements $y_i$, an\n", + "$n\\times p$ matrix $\\boldsymbol{X}$ which contains the $x_i$ values and a\n", + "vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\beta})$. We can rewrite in a more compact form the first\n", "derivative of cost function as" ] }, @@ -2337,7 +2337,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}} = -\\hat{X}^T\\left(\\hat{y}-\\hat{p}\\right).\n", + "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", "$$" ] }, @@ -2345,8 +2345,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If we in addition define a diagonal matrix $\\hat{W}$ with elements \n", - "$p(y_i\\vert x_i,\\hat{\\beta})(1-p(y_i\\vert x_i,\\hat{\\beta})$, we can obtain a compact expression of the second derivative as" + "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", + "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" ] }, { @@ -2354,7 +2354,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial^2 \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}\\partial \\hat{\\beta}^T} = \\hat{X}^T\\hat{W}\\hat{X}.\n", + "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", "$$" ] }, @@ -2372,7 +2372,7 @@ "metadata": {}, "source": [ "$$\n", - "\\log{ \\frac{p(\\hat{\\beta}\\hat{x})}{1-p(\\hat{\\beta}\\hat{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n", + "\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n", "$$" ] }, @@ -2380,7 +2380,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Here we defined $\\hat{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\hat{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to" + "Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to" ] }, { @@ -2388,7 +2388,7 @@ "metadata": {}, "source": [ "$$\n", - "p(\\hat{\\beta}\\hat{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n", + "p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n", "$$" ] }, @@ -2464,7 +2464,7 @@ "multinomial logistic regression and linear discriminant analysis, the\n", "input to the function is the result of $K$ distinct linear functions,\n", "and the predicted probability for the $k$-th class given a sample\n", - "vector $\\hat{x}$ and a weighting vector $\\hat{\\beta}$ is (with two\n", + "vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\beta}$ is (with two\n", "predictors):" ] }, diff --git a/doc/src/week38/week38.do.txt b/doc/src/week38/week38.do.txt index ed3924b19..4fd68da1f 100644 --- a/doc/src/week38/week38.do.txt +++ b/doc/src/week38/week38.do.txt @@ -1149,11 +1149,11 @@ In linear regression our main interest was centered on learning the coefficients of a functional fit (say a polynomial) in order to be able to predict the response of a continuous variable on some unseen data. The fit to the continuous variable $y_i$ is based on some -independent variables $\hat{x}_i$. Linear regression resulted in +independent variables $\bm{x}_i$. Linear regression resulted in analytical expressions for standard ordinary Least Squares or Ridge regression (in terms of matrices to invert) for several quantities, ranging from the variance and thereby the confidence intervals of the -parameters $\hat{\beta}$ to the mean squared error. If we can invert +parameters $\bm{\beta}$ to the mean squared error. If we can invert the product of the design matrices, linear regression gives then a simple recipe for fitting our data. @@ -1183,7 +1183,7 @@ failure etc. Logistic regression will also serve as our stepping stone towards neural network algorithms and supervised deep learning. For logistic learning, the minimization of the cost function leads to a non-linear -equation in the parameters $\hat{\beta}$. The optimization of the +equation in the parameters $\bm{\beta}$. The optimization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the @@ -1203,7 +1203,7 @@ responses or the outcomes, $y_i$ are discrete and only take values from $k=0,\dots,K-1$ (i.e. $K$ classes). The goal is to predict the -output classes from the design matrix $\hat{X}\in\mathbb{R}^{n\times p}$ +output classes from the design matrix $\bm{X}\in\mathbb{R}^{n\times p}$ made of $n$ samples, each of which carries $p$ features or predictors. The primary goal is to identify the classes to which new unseen samples belong. @@ -1234,11 +1234,11 @@ We would then have our weighted linear combination, namely !bt \begin{equation} -\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon}, +\bm{y} = \bm{X}^T\bm{\beta} + \bm{\epsilon}, \end{equation} !et -where $\hat{y}$ is a vector representing the possible outcomes, $\hat{X}$ is our -$n\times p$ design matrix and $\hat{\beta}$ represents our estimators/predictors. +where $\bm{y}$ is a vector representing the possible outcomes, $\bm{X}$ is our +$n\times p$ design matrix and $\bm{\beta}$ represents our estimators/predictors. !split ===== Some selected properties ===== @@ -1452,16 +1452,16 @@ plt.show() We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\beta$ in our fitting of the Sigmoid function, that is we define probabilities !bt \begin{align*} -p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ -p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}), +p(y_i=1|x_i,\bm{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\bm{\beta}) &= 1 - p(y_i=1|x_i,\bm{\beta}), \end{align*} !et -where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$. +where $\bm{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$. Note that we used !bt \[ -p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}). +p(y_i=0\vert x_i, \bm{\beta}) = 1-p(y_i=1\vert x_i, \bm{\beta}). \] !et @@ -1476,13 +1476,13 @@ the probability of seeing the observed data. We can then approximate the likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is !bt \begin{align*} -P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\ +P(\mathcal{D}|\bm{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\bm{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\bm{\beta}))\right]^{1-y_i}\nonumber \\ \end{align*} !et from which we obtain the log-likelihood and our _cost/loss_ function !bt \[ -\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right). +\mathcal{C}(\bm{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\bm{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\bm{\beta}))\right]\right). \] !et @@ -1492,7 +1492,7 @@ from which we obtain the log-likelihood and our _cost/loss_ function Reordering the logarithms, we can rewrite the _cost/loss_ function as !bt \[ -\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\mathcal{C}(\bm{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). \] !et @@ -1500,7 +1500,7 @@ The maximum likelihood estimator is defined as the set of parameters that maximi Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that !bt \[ -\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\mathcal{C}(\bm{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). \] !et This equation is known in statistics as the _cross entropy_. Finally, we note that just as in linear regression, @@ -1509,7 +1509,7 @@ in practice we often supplement the cross-entropy with additional regularization !split ===== Minimizing the cross entropy ===== -The cross entropy is a convex function of the weights $\hat{\beta}$ and, +The cross entropy is a convex function of the weights $\bm{\beta}$ and, therefore, any local minimizer is a global minimizer. @@ -1518,36 +1518,36 @@ cost function with respect to the two parameters $\beta_0$ and $\beta_1$ we obta !bt \[ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), +\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), \] !et and !bt \[ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). +\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). \] !et !split ===== A more compact expression ===== -Let us now define a vector $\hat{y}$ with $n$ elements $y_i$, an -$n\times p$ matrix $\hat{X}$ which contains the $x_i$ values and a -vector $\hat{p}$ of fitted probabilities $p(y_i\vert x_i,\hat{\beta})$. We can rewrite in a more compact form the first +Let us now define a vector $\bm{y}$ with $n$ elements $y_i$, an +$n\times p$ matrix $\bm{X}$ which contains the $x_i$ values and a +vector $\bm{p}$ of fitted probabilities $p(y_i\vert x_i,\bm{\beta})$. We can rewrite in a more compact form the first derivative of cost function as !bt \[ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}} = -\bm{X}^T\left(\bm{y}-\bm{p}\right). \] !et -If we in addition define a diagonal matrix $\hat{W}$ with elements -$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as +If we in addition define a diagonal matrix $\bm{W}$ with elements +$p(y_i\vert x_i,\bm{\beta})(1-p(y_i\vert x_i,\bm{\beta})$, we can obtain a compact expression of the second derivative as !bt \[ -\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +\frac{\partial^2 \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T} = \bm{X}^T\bm{W}\bm{X}. \] !et @@ -1557,13 +1557,13 @@ $p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a com Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors !bt \[ -\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. +\log{ \frac{p(\bm{\beta}\bm{x})}{1-p(\bm{\beta}\bm{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. \] !et -Here we defined $\hat{x}=[1,x_1,x_2,\dots,x_p]$ and $\hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to +Here we defined $\bm{x}=[1,x_1,x_2,\dots,x_p]$ and $\bm{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to !bt \[ -p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. +p(\bm{\beta}\bm{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. \] !et @@ -1609,7 +1609,7 @@ Bayes classifiers, and artificial neural networks. Specifically, in multinomial logistic regression and linear discriminant analysis, the input to the function is the result of $K$ distinct linear functions, and the predicted probability for the $k$-th class given a sample -vector $\hat{x}$ and a weighting vector $\hat{\beta}$ is (with two +vector $\bm{x}$ and a weighting vector $\bm{\beta}$ is (with two predictors): !bt