$$
-\beta_0=frac{16+\lambda}{8},
+\beta_0=\frac{16+\lambda}{8},
$$
and
$$
-\beta_1=frac{4+\lambda}{2}.
+\beta_1=\frac{4+\lambda}{2}.
$$
@@ -1069,7 +1069,7 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
Our basic assumption when we derived the OLS equations was to assume -that our output is determing by a given continuous function +that our output is determined by a given continuous function \( f(\boldsymbol{x}) \) and a random noise \( \boldsymbol{\epsilon} \) given by the normal distribution with zero mean value and an undetermined variance \( \sigma^2 \). @@ -1098,7 +1098,7 @@ We assume now that the various \( y_i \) values are stochastically distributed a We define this distribution as
$$
-p(y_i\vert \boldsymbol{X};\boldsymbol{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
+p(y_i\vert \boldsymbol{X};\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
$$
@@ -1109,12 +1109,42 @@ Since these events are assumed to be independent and identicall distributed we c
$$
-p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{beta}).
+p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{\beta}).
$$
+In statistics, maximum likelihood estimation (MLE) is a method of
+estimating the parameters of an assumed probability distribution,
+given some observed data. This is achieved by maximizing a likelihood
+function so that, under the assumed statistical model, the observed
+data is the most probable.
+
+
+We will assume here that our events are given by the above Gaussian
+distribution and we will determine the optimal parameters \( \beta \) by
+maximizing the above PDF. However, computing the derivatives of a
+product function is cumbersome and can easily lead to overflow and/or
+underflowproblems, with potentials for loss of numerical precision.
+
+
+In practice, it is more convenient to maximize the logarithm of the
+PDF because it is a monotonically increasing function of the argument.
+Alternatively, and this will be our option, we will minimize the
+negative of the logarithm since this is a monotonically decreasing
+function.
+
+
+Note also that maximization/minimization of the logarithm of the PDF
+is equivalent to the maximization/minimization of the function itself.
+
@@ -1051,7 +1055,7 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
Our basic assumption when we derived the OLS equations was to assume
-that our output is determing by a given continuous function
+that our output is determined by a given continuous function
\( f(\boldsymbol{x}) \) and a random noise \( \boldsymbol{\epsilon} \) given by the normal
distribution with zero mean value and an undetermined variance
\( \sigma^2 \).
@@ -1077,7 +1081,7 @@ $$
We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution.
We define this distribution as
$$
-p(y_i\vert \boldsymbol{X};\boldsymbol{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
+p(y_i\vert \boldsymbol{X};\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
$$
which reads as finding the likelihood of an event \( y_i \) given the input variables \( \boldsymbol{X} \) and the parameters (to be determined) \( \boldsymbol{beta} \).
@@ -1086,12 +1090,42 @@ which reads as finding the likelihood of an event \( y_i \) given the input vari
Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have
$$
-p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{beta}).
+p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{\beta}).
$$
+In statistics, maximum likelihood estimation (MLE) is a method of
+estimating the parameters of an assumed probability distribution,
+given some observed data. This is achieved by maximizing a likelihood
+function so that, under the assumed statistical model, the observed
+data is the most probable.
+
+
+We will assume here that our events are given by the above Gaussian
+distribution and we will determine the optimal parameters \( \beta \) by
+maximizing the above PDF. However, computing the derivatives of a
+product function is cumbersome and can easily lead to overflow and/or
+underflowproblems, with potentials for loss of numerical precision.
+
+
+In practice, it is more convenient to maximize the logarithm of the
+PDF because it is a monotonically increasing function of the argument.
+Alternatively, and this will be our option, we will minimize the
+negative of the logarithm since this is a monotonically decreasing
+function.
+
+
+Note also that maximization/minimization of the logarithm of the PDF
+is equivalent to the maximization/minimization of the function itself.
+
+
+
diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html
index 2a2f4a1dc..c854ba4d4 100644
--- a/doc/pub/week36/html/week36.html
+++ b/doc/pub/week36/html/week36.html
@@ -145,6 +145,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'independent-and-identically-distrubuted-iid'),
+ ('Maximum Likelihood Estimation (MLE)',
+ 2,
+ None,
+ 'maximum-likelihood-estimation-mle'),
('Friday September 10', 2, None, 'friday-september-10'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
@@ -850,12 +854,12 @@ $$
which yields
$$
-\beta_0=frac{16+\lambda}{8},
+\beta_0=\frac{16+\lambda}{8},
$$
and
$$
-\beta_1=frac{4+\lambda}{2}.
+\beta_1=\frac{4+\lambda}{2}.
$$
@@ -1056,7 +1060,7 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
Our basic assumption when we derived the OLS equations was to assume
-that our output is determing by a given continuous function
+that our output is determined by a given continuous function
\( f(\boldsymbol{x}) \) and a random noise \( \boldsymbol{\epsilon} \) given by the normal
distribution with zero mean value and an undetermined variance
\( \sigma^2 \).
@@ -1082,7 +1086,7 @@ $$
We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution.
We define this distribution as
$$
-p(y_i\vert \boldsymbol{X};\boldsymbol{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
+p(y_i\vert \boldsymbol{X};\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
$$
which reads as finding the likelihood of an event \( y_i \) given the input variables \( \boldsymbol{X} \) and the parameters (to be determined) \( \boldsymbol{beta} \).
@@ -1091,12 +1095,42 @@ which reads as finding the likelihood of an event \( y_i \) given the input vari
Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have
$$
-p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{beta}).
+p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{\beta}).
$$
+In statistics, maximum likelihood estimation (MLE) is a method of
+estimating the parameters of an assumed probability distribution,
+given some observed data. This is achieved by maximizing a likelihood
+function so that, under the assumed statistical model, the observed
+data is the most probable.
+
+
+We will assume here that our events are given by the above Gaussian
+distribution and we will determine the optimal parameters \( \beta \) by
+maximizing the above PDF. However, computing the derivatives of a
+product function is cumbersome and can easily lead to overflow and/or
+underflowproblems, with potentials for loss of numerical precision.
+
+
+In practice, it is more convenient to maximize the logarithm of the
+PDF because it is a monotonically increasing function of the argument.
+Alternatively, and this will be our option, we will minimize the
+negative of the logarithm since this is a monotonically decreasing
+function.
+
+
+Note also that maximization/minimization of the logarithm of the PDF
+is equivalent to the maximization/minimization of the function itself.
+
+
+
diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
index 7b7808b52..f055db04d 100644
Binary files a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz and b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz differ
diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb
index 034c73395..f16f43789 100644
--- a/doc/pub/week36/ipynb/week36.ipynb
+++ b/doc/pub/week36/ipynb/week36.ipynb
@@ -1062,7 +1062,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\beta_0=frac{16+\\lambda}{8},\n",
+ "\\beta_0=\\frac{16+\\lambda}{8},\n",
"$$"
]
},
@@ -1078,7 +1078,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\beta_1=frac{4+\\lambda}{2}.\n",
+ "\\beta_1=\\frac{4+\\lambda}{2}.\n",
"$$"
]
},
@@ -1363,7 +1363,7 @@
"## Deriving OLS from a probability distribution\n",
"\n",
"Our basic assumption when we derived the OLS equations was to assume\n",
- "that our output is determing by a given continuous function\n",
+ "that our output is determined by a given continuous function\n",
"$f(\\boldsymbol{x})$ and a random noise $\\boldsymbol{\\epsilon}$ given by the normal\n",
"distribution with zero mean value and an undetermined variance\n",
"$\\sigma^2$.\n",
@@ -1400,7 +1400,7 @@
"metadata": {},
"source": [
"$$\n",
- "p(y_i\\vert \\boldsymbol{X};\\boldsymbol{beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n",
+ "p(y_i\\vert \\boldsymbol{X};\\boldsymbol{\\beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n",
"$$"
]
},
@@ -1418,7 +1418,7 @@
"metadata": {},
"source": [
"$$\n",
- "p(\\boldsymbol{y}\\vert \\boldsymbol{X};\\boldsymbol{beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i\\vert \\boldsymbol{X};\\boldsymbol{beta}).\n",
+ "p(\\boldsymbol{y}\\vert \\boldsymbol{X};\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i\\vert \\boldsymbol{X};\\boldsymbol{\\beta}).\n",
"$$"
]
},
@@ -1426,6 +1426,34 @@
"cell_type": "markdown",
"metadata": {},
"source": [
+ "## Maximum Likelihood Estimation (MLE)\n",
+ "\n",
+ "In statistics, maximum likelihood estimation (MLE) is a method of\n",
+ "estimating the parameters of an assumed probability distribution,\n",
+ "given some observed data. This is achieved by maximizing a likelihood\n",
+ "function so that, under the assumed statistical model, the observed\n",
+ "data is the most probable. \n",
+ "\n",
+ "\n",
+ "We will assume here that our events are given by the above Gaussian\n",
+ "distribution and we will determine the optimal parameters $\\beta$ by\n",
+ "maximizing the above PDF. However, computing the derivatives of a\n",
+ "product function is cumbersome and can easily lead to overflow and/or\n",
+ "underflowproblems, with potentials for loss of numerical precision.\n",
+ "\n",
+ "\n",
+ "In practice, it is more convenient to maximize the logarithm of the\n",
+ "PDF because it is a monotonically increasing function of the argument.\n",
+ "Alternatively, and this will be our option, we will minimize the\n",
+ "negative of the logarithm since this is a monotonically decreasing\n",
+ "function.\n",
+ "\n",
+ "Note also that maximization/minimization of the logarithm of the PDF\n",
+ "is equivalent to the maximization/minimization of the function itself.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
"## Friday September 10\n",
"\n",
"More text will be added here.\n",
diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt
index b8ef76fd1..fd5b87109 100644
--- a/doc/src/week36/week36.do.txt
+++ b/doc/src/week36/week36.do.txt
@@ -574,13 +574,13 @@ which yields
!bt
\[
-\beta_0=frac{16+\lambda}{8},
+\beta_0=\frac{16+\lambda}{8},
\]
!et
and
!bt
\[
-\beta_1=frac{4+\lambda}{2}.
+\beta_1=\frac{4+\lambda}{2}.
\]
!et
@@ -763,7 +763,7 @@ This means the variance we obtain with the standard OLS will always for $\lambda
===== Deriving OLS from a probability distribution =====
Our basic assumption when we derived the OLS equations was to assume
-that our output is determing by a given continuous function
+that our output is determined by a given continuous function
$f(\bm{x})$ and a random noise $\bm{\epsilon}$ given by the normal
distribution with zero mean value and an undetermined variance
$\sigma^2$.
@@ -788,7 +788,7 @@ We assume now that the various $y_i$ values are stochastically distributed accor
We define this distribution as
!bt
\[
-p(y_i\vert \bm{X};\bm{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]},
+p(y_i\vert \bm{X};\bm{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]},
\]
!et
which reads as finding the likelihood of an event $y_i$ given the input variables $\bm{X}$ and the parameters (to be determined) $\bm{beta}$.
@@ -797,10 +797,37 @@ Since these events are assumed to be independent and identicall distributed we c
!bt
\[
-p(\bm{y}\vert \bm{X};\bm{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \bm{X};\bm{beta}).
+p(\bm{y}\vert \bm{X};\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \bm{X};\bm{\beta}).
\]
!et
+!split
+===== Maximum Likelihood Estimation (MLE) =====
+
+In statistics, maximum likelihood estimation (MLE) is a method of
+estimating the parameters of an assumed probability distribution,
+given some observed data. This is achieved by maximizing a likelihood
+function so that, under the assumed statistical model, the observed
+data is the most probable.
+
+
+We will assume here that our events are given by the above Gaussian
+distribution and we will determine the optimal parameters $\beta$ by
+maximizing the above PDF. However, computing the derivatives of a
+product function is cumbersome and can easily lead to overflow and/or
+underflowproblems, with potentials for loss of numerical precision.
+
+
+In practice, it is more convenient to maximize the logarithm of the
+PDF because it is a monotonically increasing function of the argument.
+Alternatively, and this will be our option, we will minimize the
+negative of the logarithm since this is a monotonically decreasing
+function.
+
+Note also that maximization/minimization of the logarithm of the PDF
+is equivalent to the maximization/minimization of the function itself.
+
+
!split
+Maximum Likelihood Estimation (MLE)
+
+Friday September 10
diff --git a/doc/pub/week36/html/week36-solarized.html b/doc/pub/week36/html/week36-solarized.html
index db40312ce..cbf4c59f1 100644
--- a/doc/pub/week36/html/week36-solarized.html
+++ b/doc/pub/week36/html/week36-solarized.html
@@ -140,6 +140,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'independent-and-identically-distrubuted-iid'),
+ ('Maximum Likelihood Estimation (MLE)',
+ 2,
+ None,
+ 'maximum-likelihood-estimation-mle'),
('Friday September 10', 2, None, 'friday-september-10'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
@@ -845,12 +849,12 @@ $$
which yields
$$
-\beta_0=frac{16+\lambda}{8},
+\beta_0=\frac{16+\lambda}{8},
$$
and
$$
-\beta_1=frac{4+\lambda}{2}.
+\beta_1=\frac{4+\lambda}{2}.
$$
+Maximum Likelihood Estimation (MLE)
+
+
+
Friday September 10
+Maximum Likelihood Estimation (MLE)
+
+
+
Friday September 10