updates

doc/LectureNotes/_build/html/chapter4.html

@@ -403,7 +403,7 @@ independent variables <span class="math notranslate nohighlight">\(x_i\)</span>.
 analytical expressions for standard ordinary Least Squares or Ridge
 regression (in terms of matrices to invert) for several quantities,
 ranging from the variance and thereby the confidence intervals of the
-optimal parameters <span class="math notranslate nohighlight">\(\hat{\beta}\)</span> to the mean squared error. If we can invert
+optimal parameters <span class="math notranslate nohighlight">\(\hat{\theta}\)</span> to the mean squared error. If we can invert
 the product of the design matrices, linear regression gives then a
 simple recipe for fitting our data.</p>
 <p>Classification problems, however, are concerned with outcomes taking
@@ -423,7 +423,7 @@ failure etc.</p>
 <p>Logistic regression will also serve as our stepping stone towards
 neural network algorithms and supervised deep learning. For logistic
 learning, the minimization of the cost function leads to a non-linear
-equation in the parameters <span class="math notranslate nohighlight">\(\hat{\beta}\)</span>. The optimization of the
+equation in the parameters <span class="math notranslate nohighlight">\(\hat{\theta}\)</span>. The optimization of the
 problem calls therefore for minimization algorithms. This forms the
 bottle neck of all machine learning algorithms, namely how to find
 reliable minima of a multi-variable function. This leads us to the
@@ -462,12 +462,12 @@ weighted linear combination, namely</p>
 <div class="math notranslate nohighlight">
 \[
 \begin{equation}
-\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\beta} +  \boldsymbol{\epsilon},
+\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\theta} +  \boldsymbol{\epsilon},
 \label{_auto1} \tag{1}
 \end{equation}
 \]</div>
 <p>where <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> is a vector representing the possible outcomes, <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is our
-<span class="math notranslate nohighlight">\(n\times p\)</span> design matrix and <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> represents our estimators/predictors.</p>
+<span class="math notranslate nohighlight">\(n\times p\)</span> design matrix and <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> represents our estimators/predictors.</p>
 <p>The main problem with our function is that it takes values on the
 entire real axis. In the case of logistic regression, however, the
 labels <span class="math notranslate nohighlight">\(y_i\)</span> are discrete variables. A typical example is the credit
@@ -561,7 +561,7 @@ plt.show()
 In standard linear regression with a linear dependence on <span class="math notranslate nohighlight">\(x\)</span>, we would write this in terms of our model</p>
 <div class="math notranslate nohighlight">
 \[
-f(y_i\vert x_i)=\beta_0+\beta_1 x_i.
+f(y_i\vert x_i)=\theta_0+\theta_1 x_i.
 \]</div>
 <p>This expression implies however that <span class="math notranslate nohighlight">\(f(y_i\vert x_i)\)</span> could take any
 value from minus infinity to plus infinity. If we however let
@@ -656,19 +656,19 @@ plt.show()
 </div>
 </div>
 </div>
-<p>We assume now that we have two classes with <span class="math notranslate nohighlight">\(y_i\)</span> either <span class="math notranslate nohighlight">\(0\)</span> or <span class="math notranslate nohighlight">\(1\)</span>. Furthermore we assume also that we have only two parameters <span class="math notranslate nohighlight">\(\beta\)</span> in our fitting of the Sigmoid function, that is we define probabilities</p>
+<p>We assume now that we have two classes with <span class="math notranslate nohighlight">\(y_i\)</span> either <span class="math notranslate nohighlight">\(0\)</span> or <span class="math notranslate nohighlight">\(1\)</span>. Furthermore we assume also that we have only two parameters <span class="math notranslate nohighlight">\(\theta\)</span> in our fitting of the Sigmoid function, that is we define probabilities</p>
 <div class="math notranslate nohighlight">
 \[\begin{split}
 \begin{align*}
-p(y_i=1|x_i,\boldsymbol{\beta}) &amp;= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
-p(y_i=0|x_i,\boldsymbol{\beta}) &amp;= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
+p(y_i=1|x_i,\boldsymbol{\theta}) &amp;= \frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}},\nonumber\\
+p(y_i=0|x_i,\boldsymbol{\theta}) &amp;= 1 - p(y_i=1|x_i,\boldsymbol{\theta}),
 \end{align*}
 \end{split}\]</div>
-<p>where <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> are the weights we wish to extract from data, in our case <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span>.</p>
+<p>where <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> are the weights we wish to extract from data, in our case <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span>.</p>
 <p>Note that we used</p>
 <div class="math notranslate nohighlight">
 \[
-p(y_i=0\vert x_i, \boldsymbol{\beta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\beta}).
+p(y_i=0\vert x_i, \boldsymbol{\theta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\theta}).
 \]</div>
 <p>In order to define the total likelihood for all possible outcomes from a<br />
 dataset <span class="math notranslate nohighlight">\(\mathcal{D}=\{(y_i,x_i)\}\)</span>, with the binary labels
@@ -679,80 +679,80 @@ likelihood in terms of the product of the individual probabilities of a specific
 <div class="math notranslate nohighlight">
 \[\begin{split}
 \begin{align*}
-P(\mathcal{D}|\boldsymbol{\beta})&amp; = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]^{1-y_i}\nonumber \\
+P(\mathcal{D}|\boldsymbol{\theta})&amp; = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\theta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\theta}))\right]^{1-y_i}\nonumber \\
 \end{align*}
 \end{split}\]</div>
 <p>from which we obtain the log-likelihood and our <strong>cost/loss</strong> function</p>
 <div class="math notranslate nohighlight">
 \[
-\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]\right).
+\mathcal{C}(\boldsymbol{\theta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\theta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\theta}))\right]\right).
 \]</div>
 <p>Reordering the logarithms, we can rewrite the <strong>cost/loss</strong> function as</p>
 <div class="math notranslate nohighlight">
 \[
-\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
+\mathcal{C}(\boldsymbol{\theta}) = \sum_{i=1}^n  \left(y_i(\theta_0+\theta_1x_i) -\log{(1+\exp{(\theta_0+\theta_1x_i)})}\right).
 \]</div>
-<p>The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to <span class="math notranslate nohighlight">\(\beta\)</span>.
+<p>The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to <span class="math notranslate nohighlight">\(\theta\)</span>.
 Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that</p>
 <div class="math notranslate nohighlight">
 \[
-\mathcal{C}(\boldsymbol{\beta})=-\sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
+\mathcal{C}(\boldsymbol{\theta})=-\sum_{i=1}^n  \left(y_i(\theta_0+\theta_1x_i) -\log{(1+\exp{(\theta_0+\theta_1x_i)})}\right).
 \]</div>
 <p>This equation is known in statistics as the <strong>cross entropy</strong>. Finally, we note that just as in linear regression,
 in practice we often supplement the cross-entropy with additional regularization terms, usually <span class="math notranslate nohighlight">\(L_1\)</span> and <span class="math notranslate nohighlight">\(L_2\)</span> regularization as we did for Ridge and Lasso regression.</p>
-<p>The cross entropy is a convex function of the weights <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and,
+<p>The cross entropy is a convex function of the weights <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and,
 therefore, any local minimizer is a global minimizer.</p>
 <p>Minimizing this
-cost function with respect to the two parameters <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span> we obtain</p>
+cost function with respect to the two parameters <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span> we obtain</p>
 <div class="math notranslate nohighlight">
 \[
-\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_0} = -\sum_{i=1}^n  \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
+\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \theta_0} = -\sum_{i=1}^n  \left(y_i -\frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}}\right),
 \]</div>
 <p>and</p>
 <div class="math notranslate nohighlight">
 \[
-\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_1} = -\sum_{i=1}^n  \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
+\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \theta_1} = -\sum_{i=1}^n  \left(y_ix_i -x_i\frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}}\right).
 \]</div>
 <p>Let us now define a vector <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> with <span class="math notranslate nohighlight">\(n\)</span> elements <span class="math notranslate nohighlight">\(y_i\)</span>, an
 <span class="math notranslate nohighlight">\(n\times p\)</span> matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which contains the <span class="math notranslate nohighlight">\(x_i\)</span> values and a
-vector <span class="math notranslate nohighlight">\(\boldsymbol{p}\)</span> of fitted probabilities <span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\beta})\)</span>. We can rewrite in a more compact form the first
+vector <span class="math notranslate nohighlight">\(\boldsymbol{p}\)</span> of fitted probabilities <span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\theta})\)</span>. We can rewrite in a more compact form the first
 derivative of cost function as</p>
 <div class="math notranslate nohighlight">
 \[
-\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
+\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
 \]</div>
 <p>If we in addition define a diagonal matrix <span class="math notranslate nohighlight">\(\boldsymbol{W}\)</span> with elements
-<span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta})\)</span>, we can obtain a compact expression of the second derivative as</p>
+<span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\theta})(1-p(y_i\vert x_i,\boldsymbol{\theta})\)</span>, we can obtain a compact expression of the second derivative as</p>
 <div class="math notranslate nohighlight">
 \[
-\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
+\frac{\partial^2 \mathcal{C}(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
 \]</div>
 <p>Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with <span class="math notranslate nohighlight">\(p\)</span> predictors</p>
 <div class="math notranslate nohighlight">
 \[
-\log{ \frac{p(\boldsymbol{\beta}\boldsymbol{x})}{1-p(\boldsymbol{\beta}\boldsymbol{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
+\log{ \frac{p(\boldsymbol{\theta}\boldsymbol{x})}{1-p(\boldsymbol{\theta}\boldsymbol{x})}} = \theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p.
 \]</div>
-<p>Here we defined <span class="math notranslate nohighlight">\(\boldsymbol{x}=[1,x_1,x_2,\dots,x_p]\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{\beta}=[\beta_0, \beta_1, \dots, \beta_p]\)</span> leading to</p>
+<p>Here we defined <span class="math notranslate nohighlight">\(\boldsymbol{x}=[1,x_1,x_2,\dots,x_p]\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{\theta}=[\theta_0, \theta_1, \dots, \theta_p]\)</span> leading to</p>
 <div class="math notranslate nohighlight">
 \[
-p(\boldsymbol{\beta}\boldsymbol{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
+p(\boldsymbol{\theta}\boldsymbol{x})=\frac{ \exp{(\theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p)}}{1+\exp{(\theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p)}}.
 \]</div>
 <p>Till now we have mainly focused on two classes, the so-called binary
 system. Suppose we wish to extend to <span class="math notranslate nohighlight">\(K\)</span> classes.  Let us for the sake
 of simplicity assume we have only two predictors. We have then following model</p>
 <div class="math notranslate nohighlight">
 \[
-\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1,
+\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \theta_{10}+\theta_{11}x_1,
 \]</div>
 <p>and</p>
 <div class="math notranslate nohighlight">
 \[
-\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1,
+\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \theta_{20}+\theta_{21}x_1,
 \]</div>
 <p>and so on till the class <span class="math notranslate nohighlight">\(C=K-1\)</span> class</p>
 <div class="math notranslate nohighlight">
 \[
-\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1,
+\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \theta_{(K-1)0}+\theta_{(K-1)1}x_1,
 \]</div>
 <p>and the model is specified in term of <span class="math notranslate nohighlight">\(K-1\)</span> so-called log-odds or
 <strong>logit</strong> transformations.</p>
@@ -765,16 +765,16 @@ Bayes classifiers, and artificial neural networks.  Specifically, in
 multinomial logistic regression and linear discriminant analysis, the
 input to the function is the result of <span class="math notranslate nohighlight">\(K\)</span> distinct linear functions,
 and the predicted probability for the <span class="math notranslate nohighlight">\(k\)</span>-th class given a sample
-vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> and a weighting vector <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is (with two
+vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> and a weighting vector <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> is (with two
 predictors):</p>
 <div class="math notranslate nohighlight">
 \[
-p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}.
+p(C=k\vert \mathbf {x} )=\frac{\exp{(\theta_{k0}+\theta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\theta_{l0}+\theta_{l1}x_1)}}.
 \]</div>
 <p>It is easy to extend to more predictors. The final class is</p>
 <div class="math notranslate nohighlight">
 \[
-p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}},
+p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\theta_{l0}+\theta_{l1}x_1)}},
 \]</div>
 <p>and they sum to one. Our earlier discussions were all specialized to
 the case with two classes only. It is easy to see from the above that