update

doc/src/week36/week36.do.txt

@@ -574,13 +574,13 @@ which yields
 
 !bt
 \[
-\beta_0=frac{16+\lambda}{8},
+\beta_0=\frac{16+\lambda}{8},
 \]
 !et
 and
 !bt
 \[
-\beta_1=frac{4+\lambda}{2}.
+\beta_1=\frac{4+\lambda}{2}.
 \]
 !et
 
@@ -763,7 +763,7 @@ This means the variance we obtain with the standard OLS will always for $\lambda
 ===== Deriving OLS from a probability distribution =====
 
 Our basic assumption when we derived the OLS equations was to assume
-that our output is determing by a given continuous function
+that our output is determined by a given continuous function
 $f(\bm{x})$ and a random noise $\bm{\epsilon}$ given by the normal
 distribution with zero mean value and an undetermined variance
 $\sigma^2$.
@@ -788,7 +788,7 @@ We assume now that the various $y_i$ values are stochastically distributed accor
 We define this distribution as
 !bt
 \[
-p(y_i\vert \bm{X};\bm{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]},
+p(y_i\vert \bm{X};\bm{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]},
 \]
 !et
 which reads as finding the likelihood of an event $y_i$ given the input variables $\bm{X}$ and the parameters (to be determined) $\bm{beta}$.
@@ -797,10 +797,37 @@ Since these events are assumed to be independent and identicall distributed we c
 
 !bt
 \[
-p(\bm{y}\vert \bm{X};\bm{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \bm{X};\bm{beta}).
+p(\bm{y}\vert \bm{X};\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \bm{X};\bm{\beta}).
 \]
 !et
 
+!split
+===== Maximum Likelihood Estimation (MLE) =====
+
+In statistics, maximum likelihood estimation (MLE) is a method of
+estimating the parameters of an assumed probability distribution,
+given some observed data. This is achieved by maximizing a likelihood
+function so that, under the assumed statistical model, the observed
+data is the most probable. 
+
+
+We will assume here that our events are given by the above Gaussian
+distribution and we will determine the optimal parameters $\beta$ by
+maximizing the above PDF. However, computing the derivatives of a
+product function is cumbersome and can easily lead to overflow and/or
+underflowproblems, with potentials for loss of numerical precision.
+
+
+In practice, it is more convenient to maximize the logarithm of the
+PDF because it is a monotonically increasing function of the argument.
+Alternatively, and this will be our option, we will minimize the
+negative of the logarithm since this is a monotonically decreasing
+function.
+
+Note also that maximization/minimization of the logarithm of the PDF
+is equivalent to the maximization/minimization of the function itself.
+
+
 
 
 !split