lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

neverending fight against typos

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-22 11:30:11 +02:00
parent 07d63b0596
commit 8142177a13
17 changed files with 174 additions and 174 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week38/html/._week38-bs024.html
+2 -2
View File
@@ -421,11 +421,11 @@ In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable \( y_i \) is based on some
independent variables \( \hat{x}_i \). Linear regression resulted in
independent variables \( \boldsymbol{x}_i \). Linear regression resulted in
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
parameters \( \hat{\beta} \) to the mean squared error. If we can invert
parameters \( \boldsymbol{\beta} \) to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.
doc/pub/week38/html/._week38-bs026.html
+1 -1
View File
@@ -420,7 +420,7 @@ MathJax.Hub.Config({
Logistic regression will also serve as our stepping stone towards
neural network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters \( \hat{\beta} \). The optimization of the
equation in the parameters \( \boldsymbol{\beta} \). The optimization of the
problem calls therefore for minimization algorithms. This forms the
bottle neck of all machine learning algorithms, namely how to find
reliable minima of a multi-variable function. This leads us to the
doc/pub/week38/html/._week38-bs027.html
+1 -1
View File
@@ -423,7 +423,7 @@ from \( k=0,\dots,K-1 \) (i.e. \( K \) classes).
<p>
The goal is to predict the
output classes from the design matrix \( \hat{X}\in\mathbb{R}^{n\times p} \)
output classes from the design matrix \( \boldsymbol{X}\in\mathbb{R}^{n\times p} \)
made of \( n \) samples, each of which carries \( p \) features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.
doc/pub/week38/html/._week38-bs028.html
+3 -3
View File
@@ -427,13 +427,13 @@ We would then have our
weighted linear combination, namely
$$
\begin{equation}
\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon},
\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\beta} + \boldsymbol{\epsilon},
\tag{13}
\end{equation}
$$
where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our
\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators/predictors.
where \( \boldsymbol{y} \) is a vector representing the possible outcomes, \( \boldsymbol{X} \) is our
\( n\times p \) design matrix and \( \boldsymbol{\beta} \) represents our estimators/predictors.
<p>
<p>
doc/pub/week38/html/._week38-bs034.html
+4 -4
View File
@@ -420,17 +420,17 @@ MathJax.Hub.Config({
We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities
$$
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
\end{align*}
$$
where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
<p>
Note that we used
$$
p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
p(y_i=0\vert x_i, \boldsymbol{\beta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\beta}).
$$
<p>
doc/pub/week38/html/._week38-bs035.html
+2 -2
View File
@@ -425,13 +425,13 @@ the probability of seeing the observed data. We can then approximate the
likelihood in terms of the product of the individual probabilities of a specific outcome \( y_i \), that is
$$
\begin{align*}
P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
P(\mathcal{D}|\boldsymbol{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
$$
from which we obtain the log-likelihood and our <b>cost/loss</b> function
$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]\right).
$$
<p>
doc/pub/week38/html/._week38-bs036.html
+2 -2
View File
@@ -419,14 +419,14 @@ MathJax.Hub.Config({
<p>
Reordering the logarithms, we can rewrite the <b>cost/loss</b> function as
$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$
<p>
The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \beta \).
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
$$
\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\boldsymbol{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$
This equation is known in statistics as the <b>cross entropy</b>. Finally, we note that just as in linear regression,
doc/pub/week38/html/._week38-bs037.html
+3 -3
View File
@@ -417,7 +417,7 @@ MathJax.Hub.Config({
<h2 id="minimizing-the-cross-entropy" class="anchor">Minimizing the cross entropy </h2>
<p>
The cross entropy is a convex function of the weights \( \hat{\beta} \) and,
The cross entropy is a convex function of the weights \( \boldsymbol{\beta} \) and,
therefore, any local minimizer is a global minimizer.
<p>
@@ -425,12 +425,12 @@ Minimizing this
cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \) we obtain
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
$$
and
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$
<p>
doc/pub/week38/html/._week38-bs038.html
+7 -7
View File
@@ -417,21 +417,21 @@ MathJax.Hub.Config({
<h2 id="a-more-compact-expression" class="anchor">A more compact expression </h2>
<p>
Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an
\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a
vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first
Let us now define a vector \( \boldsymbol{y} \) with \( n \) elements \( y_i \), an
\( n\times p \) matrix \( \boldsymbol{X} \) which contains the \( x_i \) values and a
vector \( \boldsymbol{p} \) of fitted probabilities \( p(y_i\vert x_i,\boldsymbol{\beta}) \). We can rewrite in a more compact form the first
derivative of cost function as
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
$$
<p>
If we in addition define a diagonal matrix \( \hat{W} \) with elements
\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as
If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements
\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as
$$
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
$$
<p>
doc/pub/week38/html/._week38-bs039.html
+3 -3
View File
@@ -419,12 +419,12 @@ MathJax.Hub.Config({
<p>
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors
$$
\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
\log{ \frac{p(\boldsymbol{\beta}\boldsymbol{x})}{1-p(\boldsymbol{\beta}\boldsymbol{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
$$
Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to
Here we defined \( \boldsymbol{x}=[1,x_1,x_2,\dots,x_p] \) and \( \boldsymbol{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to
$$
p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
p(\boldsymbol{\beta}\boldsymbol{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
$$
<p>
doc/pub/week38/html/._week38-bs041.html
+1 -1
View File
@@ -428,7 +428,7 @@ Bayes classifiers, and artificial neural networks. Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of \( K \) distinct linear functions,
and the predicted probability for the \( k \)-th class given a sample
vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two
vector \( \boldsymbol{x} \) and a weighting vector \( \boldsymbol{\beta} \) is (with two
predictors):
$$
doc/pub/week38/html/week38-reveal.html
+29 -29
View File
@@ -1470,11 +1470,11 @@ In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable \( y_i \) is based on some
independent variables \( \hat{x}_i \). Linear regression resulted in
independent variables \( \boldsymbol{x}_i \). Linear regression resulted in
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
parameters \( \hat{\beta} \) to the mean squared error. If we can invert
parameters \( \boldsymbol{\beta} \) to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.
</section>
@@ -1510,7 +1510,7 @@ failure etc.
Logistic regression will also serve as our stepping stone towards
neural network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters \( \hat{\beta} \). The optimization of the
equation in the parameters \( \boldsymbol{\beta} \). The optimization of the
problem calls therefore for minimization algorithms. This forms the
bottle neck of all machine learning algorithms, namely how to find
reliable minima of a multi-variable function. This leads us to the
@@ -1534,7 +1534,7 @@ from \( k=0,\dots,K-1 \) (i.e. \( K \) classes).
<p>
The goal is to predict the
output classes from the design matrix \( \hat{X}\in\mathbb{R}^{n\times p} \)
output classes from the design matrix \( \boldsymbol{X}\in\mathbb{R}^{n\times p} \)
made of \( n \) samples, each of which carries \( p \) features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.
@@ -1568,14 +1568,14 @@ weighted linear combination, namely
<p>&nbsp;<br>
$$
\begin{equation}
\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon},
\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\beta} + \boldsymbol{\epsilon},
\tag{13}
\end{equation}
$$
<p>&nbsp;<br>
where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our
\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators/predictors.
where \( \boldsymbol{y} \) is a vector representing the possible outcomes, \( \boldsymbol{X} \) is our
\( n\times p \) design matrix and \( \boldsymbol{\beta} \) represents our estimators/predictors.
</section>
@@ -1809,19 +1809,19 @@ We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \).
<p>&nbsp;<br>
$$
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
\end{align*}
$$
<p>&nbsp;<br>
where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
<p>
Note that we used
<p>&nbsp;<br>
$$
p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
p(y_i=0\vert x_i, \boldsymbol{\beta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\beta}).
$$
<p>&nbsp;<br>
</section>
@@ -1840,7 +1840,7 @@ likelihood in terms of the product of the individual probabilities of a specific
<p>&nbsp;<br>
$$
\begin{align*}
P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
P(\mathcal{D}|\boldsymbol{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
$$
<p>&nbsp;<br>
@@ -1848,7 +1848,7 @@ $$
from which we obtain the log-likelihood and our <b>cost/loss</b> function
<p>&nbsp;<br>
$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]\right).
$$
<p>&nbsp;<br>
</section>
@@ -1861,7 +1861,7 @@ $$
Reordering the logarithms, we can rewrite the <b>cost/loss</b> function as
<p>&nbsp;<br>
$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$
<p>&nbsp;<br>
@@ -1870,7 +1870,7 @@ The maximum likelihood estimator is defined as the set of parameters that maximi
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
<p>&nbsp;<br>
$$
\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\boldsymbol{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$
<p>&nbsp;<br>
@@ -1883,7 +1883,7 @@ in practice we often supplement the cross-entropy with additional regularization
<h2 id="minimizing-the-cross-entropy">Minimizing the cross entropy </h2>
<p>
The cross entropy is a convex function of the weights \( \hat{\beta} \) and,
The cross entropy is a convex function of the weights \( \boldsymbol{\beta} \) and,
therefore, any local minimizer is a global minimizer.
<p>
@@ -1892,14 +1892,14 @@ cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \)
<p>&nbsp;<br>
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
$$
<p>&nbsp;<br>
and
<p>&nbsp;<br>
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$
<p>&nbsp;<br>
</section>
@@ -1909,24 +1909,24 @@ $$
<h2 id="a-more-compact-expression">A more compact expression </h2>
<p>
Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an
\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a
vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first
Let us now define a vector \( \boldsymbol{y} \) with \( n \) elements \( y_i \), an
\( n\times p \) matrix \( \boldsymbol{X} \) which contains the \( x_i \) values and a
vector \( \boldsymbol{p} \) of fitted probabilities \( p(y_i\vert x_i,\boldsymbol{\beta}) \). We can rewrite in a more compact form the first
derivative of cost function as
<p>&nbsp;<br>
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
$$
<p>&nbsp;<br>
<p>
If we in addition define a diagonal matrix \( \hat{W} \) with elements
\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as
If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements
\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as
<p>&nbsp;<br>
$$
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
$$
<p>&nbsp;<br>
</section>
@@ -1939,14 +1939,14 @@ $$
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors
<p>&nbsp;<br>
$$
\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
\log{ \frac{p(\boldsymbol{\beta}\boldsymbol{x})}{1-p(\boldsymbol{\beta}\boldsymbol{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
$$
<p>&nbsp;<br>
Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to
Here we defined \( \boldsymbol{x}=[1,x_1,x_2,\dots,x_p] \) and \( \boldsymbol{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to
<p>&nbsp;<br>
$$
p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
p(\boldsymbol{\beta}\boldsymbol{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
$$
<p>&nbsp;<br>
</section>
@@ -2001,7 +2001,7 @@ Bayes classifiers, and artificial neural networks. Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of \( K \) distinct linear functions,
and the predicted probability for the \( k \)-th class given a sample
vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two
vector \( \boldsymbol{x} \) and a weighting vector \( \boldsymbol{\beta} \) is (with two
predictors):
<p>&nbsp;<br>
doc/pub/week38/html/week38-solarized.html
+29 -29
View File
@@ -1577,11 +1577,11 @@ In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable \( y_i \) is based on some
independent variables \( \hat{x}_i \). Linear regression resulted in
independent variables \( \boldsymbol{x}_i \). Linear regression resulted in
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
parameters \( \hat{\beta} \) to the mean squared error. If we can invert
parameters \( \boldsymbol{\beta} \) to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.
@@ -1617,7 +1617,7 @@ failure etc.
Logistic regression will also serve as our stepping stone towards
neural network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters \( \hat{\beta} \). The optimization of the
equation in the parameters \( \boldsymbol{\beta} \). The optimization of the
problem calls therefore for minimization algorithms. This forms the
bottle neck of all machine learning algorithms, namely how to find
reliable minima of a multi-variable function. This leads us to the
@@ -1641,7 +1641,7 @@ from \( k=0,\dots,K-1 \) (i.e. \( K \) classes).
<p>
The goal is to predict the
output classes from the design matrix \( \hat{X}\in\mathbb{R}^{n\times p} \)
output classes from the design matrix \( \boldsymbol{X}\in\mathbb{R}^{n\times p} \)
made of \( n \) samples, each of which carries \( p \) features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.
@@ -1672,13 +1672,13 @@ We would then have our
weighted linear combination, namely
$$
\begin{equation}
\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon},
\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\beta} + \boldsymbol{\epsilon},
\label{_auto13}
\end{equation}
$$
where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our
\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators/predictors.
where \( \boldsymbol{y} \) is a vector representing the possible outcomes, \( \boldsymbol{X} \) is our
\( n\times p \) design matrix and \( \boldsymbol{\beta} \) represents our estimators/predictors.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1905,17 +1905,17 @@ plt.show()
We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities
$$
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
\end{align*}
$$
where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
<p>
Note that we used
$$
p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
p(y_i=0\vert x_i, \boldsymbol{\beta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\beta}).
$$
<p>
@@ -1932,13 +1932,13 @@ the probability of seeing the observed data. We can then approximate the
likelihood in terms of the product of the individual probabilities of a specific outcome \( y_i \), that is
$$
\begin{align*}
P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
P(\mathcal{D}|\boldsymbol{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
$$
from which we obtain the log-likelihood and our <b>cost/loss</b> function
$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]\right).
$$
<p>
@@ -1949,14 +1949,14 @@ $$
<p>
Reordering the logarithms, we can rewrite the <b>cost/loss</b> function as
$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$
<p>
The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \beta \).
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
$$
\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\boldsymbol{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$
This equation is known in statistics as the <b>cross entropy</b>. Finally, we note that just as in linear regression,
@@ -1968,7 +1968,7 @@ in practice we often supplement the cross-entropy with additional regularization
<h2 id="minimizing-the-cross-entropy">Minimizing the cross entropy </h2>
<p>
The cross entropy is a convex function of the weights \( \hat{\beta} \) and,
The cross entropy is a convex function of the weights \( \boldsymbol{\beta} \) and,
therefore, any local minimizer is a global minimizer.
<p>
@@ -1976,12 +1976,12 @@ Minimizing this
cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \) we obtain
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
$$
and
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$
<p>
@@ -1990,21 +1990,21 @@ $$
<h2 id="a-more-compact-expression">A more compact expression </h2>
<p>
Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an
\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a
vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first
Let us now define a vector \( \boldsymbol{y} \) with \( n \) elements \( y_i \), an
\( n\times p \) matrix \( \boldsymbol{X} \) which contains the \( x_i \) values and a
vector \( \boldsymbol{p} \) of fitted probabilities \( p(y_i\vert x_i,\boldsymbol{\beta}) \). We can rewrite in a more compact form the first
derivative of cost function as
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
$$
<p>
If we in addition define a diagonal matrix \( \hat{W} \) with elements
\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as
If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements
\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as
$$
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
$$
<p>
@@ -2015,12 +2015,12 @@ $$
<p>
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors
$$
\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
\log{ \frac{p(\boldsymbol{\beta}\boldsymbol{x})}{1-p(\boldsymbol{\beta}\boldsymbol{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
$$
Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to
Here we defined \( \boldsymbol{x}=[1,x_1,x_2,\dots,x_p] \) and \( \boldsymbol{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to
$$
p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
p(\boldsymbol{\beta}\boldsymbol{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
$$
<p>
@@ -2068,7 +2068,7 @@ Bayes classifiers, and artificial neural networks. Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of \( K \) distinct linear functions,
and the predicted probability for the \( k \)-th class given a sample
vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two
vector \( \boldsymbol{x} \) and a weighting vector \( \boldsymbol{\beta} \) is (with two
predictors):
$$
doc/pub/week38/html/week38.html
+29 -29
View File
@@ -1582,11 +1582,11 @@ In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable \( y_i \) is based on some
independent variables \( \hat{x}_i \). Linear regression resulted in
independent variables \( \boldsymbol{x}_i \). Linear regression resulted in
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
parameters \( \hat{\beta} \) to the mean squared error. If we can invert
parameters \( \boldsymbol{\beta} \) to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.
@@ -1622,7 +1622,7 @@ failure etc.
Logistic regression will also serve as our stepping stone towards
neural network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters \( \hat{\beta} \). The optimization of the
equation in the parameters \( \boldsymbol{\beta} \). The optimization of the
problem calls therefore for minimization algorithms. This forms the
bottle neck of all machine learning algorithms, namely how to find
reliable minima of a multi-variable function. This leads us to the
@@ -1646,7 +1646,7 @@ from \( k=0,\dots,K-1 \) (i.e. \( K \) classes).
<p>
The goal is to predict the
output classes from the design matrix \( \hat{X}\in\mathbb{R}^{n\times p} \)
output classes from the design matrix \( \boldsymbol{X}\in\mathbb{R}^{n\times p} \)
made of \( n \) samples, each of which carries \( p \) features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.
@@ -1677,13 +1677,13 @@ We would then have our
weighted linear combination, namely
$$
\begin{equation}
\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon},
\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\beta} + \boldsymbol{\epsilon},
\label{_auto13}
\end{equation}
$$
where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our
\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators/predictors.
where \( \boldsymbol{y} \) is a vector representing the possible outcomes, \( \boldsymbol{X} \) is our
\( n\times p \) design matrix and \( \boldsymbol{\beta} \) represents our estimators/predictors.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1910,17 +1910,17 @@ plt<span style="color: #666666">.</span>show()
We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities
$$
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
\end{align*}
$$
where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
<p>
Note that we used
$$
p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
p(y_i=0\vert x_i, \boldsymbol{\beta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\beta}).
$$
<p>
@@ -1937,13 +1937,13 @@ the probability of seeing the observed data. We can then approximate the
likelihood in terms of the product of the individual probabilities of a specific outcome \( y_i \), that is
$$
\begin{align*}
P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
P(\mathcal{D}|\boldsymbol{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
$$
from which we obtain the log-likelihood and our <b>cost/loss</b> function
$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]\right).
$$
<p>
@@ -1954,14 +1954,14 @@ $$
<p>
Reordering the logarithms, we can rewrite the <b>cost/loss</b> function as
$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$
<p>
The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \beta \).
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
$$
\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\boldsymbol{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$
This equation is known in statistics as the <b>cross entropy</b>. Finally, we note that just as in linear regression,
@@ -1973,7 +1973,7 @@ in practice we often supplement the cross-entropy with additional regularization
<h2 id="minimizing-the-cross-entropy">Minimizing the cross entropy </h2>
<p>
The cross entropy is a convex function of the weights \( \hat{\beta} \) and,
The cross entropy is a convex function of the weights \( \boldsymbol{\beta} \) and,
therefore, any local minimizer is a global minimizer.
<p>
@@ -1981,12 +1981,12 @@ Minimizing this
cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \) we obtain
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
$$
and
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$
<p>
@@ -1995,21 +1995,21 @@ $$
<h2 id="a-more-compact-expression">A more compact expression </h2>
<p>
Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an
\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a
vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first
Let us now define a vector \( \boldsymbol{y} \) with \( n \) elements \( y_i \), an
\( n\times p \) matrix \( \boldsymbol{X} \) which contains the \( x_i \) values and a
vector \( \boldsymbol{p} \) of fitted probabilities \( p(y_i\vert x_i,\boldsymbol{\beta}) \). We can rewrite in a more compact form the first
derivative of cost function as
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
$$
<p>
If we in addition define a diagonal matrix \( \hat{W} \) with elements
\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as
If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements
\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as
$$
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
$$
<p>
@@ -2020,12 +2020,12 @@ $$
<p>
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors
$$
\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
\log{ \frac{p(\boldsymbol{\beta}\boldsymbol{x})}{1-p(\boldsymbol{\beta}\boldsymbol{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
$$
Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to
Here we defined \( \boldsymbol{x}=[1,x_1,x_2,\dots,x_p] \) and \( \boldsymbol{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to
$$
p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
p(\boldsymbol{\beta}\boldsymbol{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
$$
<p>
@@ -2073,7 +2073,7 @@ Bayes classifiers, and artificial neural networks. Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of \( K \) distinct linear functions,
and the predicted probability for the \( k \)-th class given a sample
vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two
vector \( \boldsymbol{x} \) and a weighting vector \( \boldsymbol{\beta} \) is (with two
predictors):
$$
doc/pub/week38/ipynb/ipynb-week38-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week38/ipynb/week38.ipynb
+29 -29
View File
@@ -1804,11 +1804,11 @@
"coefficients of a functional fit (say a polynomial) in order to be\n",
"able to predict the response of a continuous variable on some unseen\n",
"data. The fit to the continuous variable $y_i$ is based on some\n",
"independent variables $\\hat{x}_i$. Linear regression resulted in\n",
"independent variables $\\boldsymbol{x}_i$. Linear regression resulted in\n",
"analytical expressions for standard ordinary Least Squares or Ridge\n",
"regression (in terms of matrices to invert) for several quantities,\n",
"ranging from the variance and thereby the confidence intervals of the\n",
"parameters $\\hat{\\beta}$ to the mean squared error. If we can invert\n",
"parameters $\\boldsymbol{\\beta}$ to the mean squared error. If we can invert\n",
"the product of the design matrices, linear regression gives then a\n",
"simple recipe for fitting our data.\n",
"\n",
@@ -1837,7 +1837,7 @@
"Logistic regression will also serve as our stepping stone towards\n",
"neural network algorithms and supervised deep learning. For logistic\n",
"learning, the minimization of the cost function leads to a non-linear\n",
"equation in the parameters $\\hat{\\beta}$. The optimization of the\n",
"equation in the parameters $\\boldsymbol{\\beta}$. The optimization of the\n",
"problem calls therefore for minimization algorithms. This forms the\n",
"bottle neck of all machine learning algorithms, namely how to find\n",
"reliable minima of a multi-variable function. This leads us to the\n",
@@ -1857,7 +1857,7 @@
"from $k=0,\\dots,K-1$ (i.e. $K$ classes).\n",
"\n",
"The goal is to predict the\n",
"output classes from the design matrix $\\hat{X}\\in\\mathbb{R}^{n\\times p}$\n",
"output classes from the design matrix $\\boldsymbol{X}\\in\\mathbb{R}^{n\\times p}$\n",
"made of $n$ samples, each of which carries $p$ features or predictors. The\n",
"primary goal is to identify the classes to which new unseen samples\n",
"belong.\n",
@@ -1901,7 +1901,7 @@
"\n",
"$$\n",
"\\begin{equation}\n",
"\\hat{y} = \\hat{X}^T\\hat{\\beta} + \\hat{\\epsilon},\n",
"\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon},\n",
"\\label{_auto13} \\tag{13}\n",
"\\end{equation}\n",
"$$"
@@ -1911,8 +1911,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"where $\\hat{y}$ is a vector representing the possible outcomes, $\\hat{X}$ is our\n",
"$n\\times p$ design matrix and $\\hat{\\beta}$ represents our estimators/predictors.\n",
"where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n",
"$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors.\n",
"\n",
"## Some selected properties\n",
"\n",
@@ -2177,8 +2177,8 @@
"source": [
"$$\n",
"\\begin{align*}\n",
"p(y_i=1|x_i,\\hat{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n",
"p(y_i=0|x_i,\\hat{\\beta}) &= 1 - p(y_i=1|x_i,\\hat{\\beta}),\n",
"p(y_i=1|x_i,\\boldsymbol{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n",
"p(y_i=0|x_i,\\boldsymbol{\\beta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\beta}),\n",
"\\end{align*}\n",
"$$"
]
@@ -2187,7 +2187,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"where $\\hat{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
"where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
"\n",
"Note that we used"
]
@@ -2197,7 +2197,7 @@
"metadata": {},
"source": [
"$$\n",
"p(y_i=0\\vert x_i, \\hat{\\beta}) = 1-p(y_i=1\\vert x_i, \\hat{\\beta}).\n",
"p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n",
"$$"
]
},
@@ -2222,7 +2222,7 @@
"source": [
"$$\n",
"\\begin{align*}\n",
"P(\\mathcal{D}|\\hat{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\hat{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n",
"P(\\mathcal{D}|\\boldsymbol{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\boldsymbol{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n",
"\\end{align*}\n",
"$$"
]
@@ -2239,7 +2239,7 @@
"metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\hat{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]\\right).\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n",
"$$"
]
},
@@ -2257,7 +2257,7 @@
"metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
"$$"
]
},
@@ -2274,7 +2274,7 @@
"metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\hat{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
"\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
"$$"
]
},
@@ -2287,7 +2287,7 @@
"\n",
"## Minimizing the cross entropy\n",
"\n",
"The cross entropy is a convex function of the weights $\\hat{\\beta}$ and,\n",
"The cross entropy is a convex function of the weights $\\boldsymbol{\\beta}$ and,\n",
"therefore, any local minimizer is a global minimizer. \n",
"\n",
"\n",
@@ -2300,7 +2300,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
"$$"
]
},
@@ -2316,7 +2316,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
"$$"
]
},
@@ -2326,9 +2326,9 @@
"source": [
"## A more compact expression\n",
"\n",
"Let us now define a vector $\\hat{y}$ with $n$ elements $y_i$, an\n",
"$n\\times p$ matrix $\\hat{X}$ which contains the $x_i$ values and a\n",
"vector $\\hat{p}$ of fitted probabilities $p(y_i\\vert x_i,\\hat{\\beta})$. We can rewrite in a more compact form the first\n",
"Let us now define a vector $\\boldsymbol{y}$ with $n$ elements $y_i$, an\n",
"$n\\times p$ matrix $\\boldsymbol{X}$ which contains the $x_i$ values and a\n",
"vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\beta})$. We can rewrite in a more compact form the first\n",
"derivative of cost function as"
]
},
@@ -2337,7 +2337,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}} = -\\hat{X}^T\\left(\\hat{y}-\\hat{p}\\right).\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
"$$"
]
},
@@ -2345,8 +2345,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"If we in addition define a diagonal matrix $\\hat{W}$ with elements \n",
"$p(y_i\\vert x_i,\\hat{\\beta})(1-p(y_i\\vert x_i,\\hat{\\beta})$, we can obtain a compact expression of the second derivative as"
"If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n",
"$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as"
]
},
{
@@ -2354,7 +2354,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial^2 \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}\\partial \\hat{\\beta}^T} = \\hat{X}^T\\hat{W}\\hat{X}.\n",
"\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
"$$"
]
},
@@ -2372,7 +2372,7 @@
"metadata": {},
"source": [
"$$\n",
"\\log{ \\frac{p(\\hat{\\beta}\\hat{x})}{1-p(\\hat{\\beta}\\hat{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
"\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
"$$"
]
},
@@ -2380,7 +2380,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we defined $\\hat{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\hat{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
"Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
]
},
{
@@ -2388,7 +2388,7 @@
"metadata": {},
"source": [
"$$\n",
"p(\\hat{\\beta}\\hat{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
"p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
"$$"
]
},
@@ -2464,7 +2464,7 @@
"multinomial logistic regression and linear discriminant analysis, the\n",
"input to the function is the result of $K$ distinct linear functions,\n",
"and the predicted probability for the $k$-th class given a sample\n",
"vector $\\hat{x}$ and a weighting vector $\\hat{\\beta}$ is (with two\n",
"vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\beta}$ is (with two\n",
"predictors):"
]
},
doc/src/week38/week38.do.txt
+29 -29
View File
@@ -1149,11 +1149,11 @@ In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable $y_i$ is based on some
independent variables $\hat{x}_i$. Linear regression resulted in
independent variables $\bm{x}_i$. Linear regression resulted in
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
parameters $\hat{\beta}$ to the mean squared error. If we can invert
parameters $\bm{\beta}$ to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.
@@ -1183,7 +1183,7 @@ failure etc.
Logistic regression will also serve as our stepping stone towards
neural network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters $\hat{\beta}$. The optimization of the
equation in the parameters $\bm{\beta}$. The optimization of the
problem calls therefore for minimization algorithms. This forms the
bottle neck of all machine learning algorithms, namely how to find
reliable minima of a multi-variable function. This leads us to the
@@ -1203,7 +1203,7 @@ responses or the outcomes, $y_i$ are discrete and only take values
from $k=0,\dots,K-1$ (i.e. $K$ classes).
The goal is to predict the
output classes from the design matrix $\hat{X}\in\mathbb{R}^{n\times p}$
output classes from the design matrix $\bm{X}\in\mathbb{R}^{n\times p}$
made of $n$ samples, each of which carries $p$ features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.
@@ -1234,11 +1234,11 @@ We would then have our
weighted linear combination, namely
!bt
\begin{equation}
\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon},
\bm{y} = \bm{X}^T\bm{\beta} + \bm{\epsilon},
\end{equation}
!et
where $\hat{y}$ is a vector representing the possible outcomes, $\hat{X}$ is our
$n\times p$ design matrix and $\hat{\beta}$ represents our estimators/predictors.
where $\bm{y}$ is a vector representing the possible outcomes, $\bm{X}$ is our
$n\times p$ design matrix and $\bm{\beta}$ represents our estimators/predictors.
!split
===== Some selected properties =====
@@ -1452,16 +1452,16 @@ plt.show()
We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\beta$ in our fitting of the Sigmoid function, that is we define probabilities
!bt
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
p(y_i=1|x_i,\bm{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\bm{\beta}) &= 1 - p(y_i=1|x_i,\bm{\beta}),
\end{align*}
!et
where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$.
where $\bm{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$.
Note that we used
!bt
\[
p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
p(y_i=0\vert x_i, \bm{\beta}) = 1-p(y_i=1\vert x_i, \bm{\beta}).
\]
!et
@@ -1476,13 +1476,13 @@ the probability of seeing the observed data. We can then approximate the
likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is
!bt
\begin{align*}
P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
P(\mathcal{D}|\bm{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\bm{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\bm{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
!et
from which we obtain the log-likelihood and our _cost/loss_ function
!bt
\[
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
\mathcal{C}(\bm{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\bm{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\bm{\beta}))\right]\right).
\]
!et
@@ -1492,7 +1492,7 @@ from which we obtain the log-likelihood and our _cost/loss_ function
Reordering the logarithms, we can rewrite the _cost/loss_ function as
!bt
\[
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\bm{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\]
!et
@@ -1500,7 +1500,7 @@ The maximum likelihood estimator is defined as the set of parameters that maximi
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
!bt
\[
\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\bm{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\]
!et
This equation is known in statistics as the _cross entropy_. Finally, we note that just as in linear regression,
@@ -1509,7 +1509,7 @@ in practice we often supplement the cross-entropy with additional regularization
!split
===== Minimizing the cross entropy =====
The cross entropy is a convex function of the weights $\hat{\beta}$ and,
The cross entropy is a convex function of the weights $\bm{\beta}$ and,
therefore, any local minimizer is a global minimizer.
@@ -1518,36 +1518,36 @@ cost function with respect to the two parameters $\beta_0$ and $\beta_1$ we obta
!bt
\[
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
\]
!et
and
!bt
\[
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
\]
!et
!split
===== A more compact expression =====
Let us now define a vector $\hat{y}$ with $n$ elements $y_i$, an
$n\times p$ matrix $\hat{X}$ which contains the $x_i$ values and a
vector $\hat{p}$ of fitted probabilities $p(y_i\vert x_i,\hat{\beta})$. We can rewrite in a more compact form the first
Let us now define a vector $\bm{y}$ with $n$ elements $y_i$, an
$n\times p$ matrix $\bm{X}$ which contains the $x_i$ values and a
vector $\bm{p}$ of fitted probabilities $p(y_i\vert x_i,\bm{\beta})$. We can rewrite in a more compact form the first
derivative of cost function as
!bt
\[
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}} = -\bm{X}^T\left(\bm{y}-\bm{p}\right).
\]
!et
If we in addition define a diagonal matrix $\hat{W}$ with elements
$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as
If we in addition define a diagonal matrix $\bm{W}$ with elements
$p(y_i\vert x_i,\bm{\beta})(1-p(y_i\vert x_i,\bm{\beta})$, we can obtain a compact expression of the second derivative as
!bt
\[
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
\frac{\partial^2 \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T} = \bm{X}^T\bm{W}\bm{X}.
\]
!et
@@ -1557,13 +1557,13 @@ $p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a com
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors
!bt
\[
\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
\log{ \frac{p(\bm{\beta}\bm{x})}{1-p(\bm{\beta}\bm{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
\]
!et
Here we defined $\hat{x}=[1,x_1,x_2,\dots,x_p]$ and $\hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to
Here we defined $\bm{x}=[1,x_1,x_2,\dots,x_p]$ and $\bm{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to
!bt
\[
p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
p(\bm{\beta}\bm{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
\]
!et
@@ -1609,7 +1609,7 @@ Bayes classifiers, and artificial neural networks. Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of $K$ distinct linear functions,
and the predicted probability for the $k$-th class given a sample
vector $\hat{x}$ and a weighting vector $\hat{\beta}$ is (with two
vector $\bm{x}$ and a weighting vector $\bm{\beta}$ is (with two
predictors):
!bt