lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-09 06:53:19 +02:00
parent 844d210ddf
commit 1cd540f93d
7 changed files with 212 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week36/html/week36-bs.html
+34 -29
View File
@@ -120,6 +120,10 @@ Automatically generated HTML file from DocOnce source
2,
None,
'independent-and-identically-distrubuted-iid'),
('Maximum Likelihood Estimation (MLE)',
2,
None,
'maximum-likelihood-estimation-mle'),
('Friday September 10', 2, None, 'friday-september-10'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
@@ -275,34 +279,35 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week36-bs027.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs028.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs029.html#independent-and-identically-distrubuted-iid" style="font-size: 80%;">Independent and Identically Distrubuted (iid)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs030.html#friday-september-10" style="font-size: 80%;">Friday September 10</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs033.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs035.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs038.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs039.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs040.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs041.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs042.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs043.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs044.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs045.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs046.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs047.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs048.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs049.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs050.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs051.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs052.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs053.html#summing-up" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs054.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs055.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs056.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs057.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs030.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs031.html#friday-september-10" style="font-size: 80%;">Friday September 10</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs035.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs035.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs038.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs039.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs040.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs041.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs042.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs043.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs044.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs045.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs046.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs047.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs048.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs049.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs051.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs052.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs053.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs054.html#summing-up" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs055.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs056.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs057.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs058.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
</ul>
</li>
@@ -361,7 +366,7 @@ MathJax.Hub.Config({
<li><a href="._week36-bs008.html">9</a></li>
<li><a href="._week36-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week36-bs057.html">58</a></li>
<li><a href="._week36-bs058.html">59</a></li>
<li><a href="._week36-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week36/html/week36-reveal.html
+35 -5
View File
@@ -840,14 +840,14 @@ which yields
<p>&nbsp;<br>
$$
\beta_0=frac{16+\lambda}{8},
\beta_0=\frac{16+\lambda}{8},
$$
<p>&nbsp;<br>
and
<p>&nbsp;<br>
$$
\beta_1=frac{4+\lambda}{2}.
\beta_1=\frac{4+\lambda}{2}.
$$
<p>&nbsp;<br>
@@ -1069,7 +1069,7 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
<p>
Our basic assumption when we derived the OLS equations was to assume
that our output is determing by a given continuous function
that our output is determined by a given continuous function
\( f(\boldsymbol{x}) \) and a random noise \( \boldsymbol{\epsilon} \) given by the normal
distribution with zero mean value and an undetermined variance
\( \sigma^2 \).
@@ -1098,7 +1098,7 @@ We assume now that the various \( y_i \) values are stochastically distributed a
We define this distribution as
<p>&nbsp;<br>
$$
p(y_i\vert \boldsymbol{X};\boldsymbol{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
p(y_i\vert \boldsymbol{X};\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
$$
<p>&nbsp;<br>
@@ -1109,12 +1109,42 @@ Since these events are assumed to be independent and identicall distributed we c
<p>&nbsp;<br>
$$
p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{beta}).
p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{\beta}).
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="maximum-likelihood-estimation-mle">Maximum Likelihood Estimation (MLE) </h2>
<p>
In statistics, maximum likelihood estimation (MLE) is a method of
estimating the parameters of an assumed probability distribution,
given some observed data. This is achieved by maximizing a likelihood
function so that, under the assumed statistical model, the observed
data is the most probable.
<p>
We will assume here that our events are given by the above Gaussian
distribution and we will determine the optimal parameters \( \beta \) by
maximizing the above PDF. However, computing the derivatives of a
product function is cumbersome and can easily lead to overflow and/or
underflowproblems, with potentials for loss of numerical precision.
<p>
In practice, it is more convenient to maximize the logarithm of the
PDF because it is a monotonically increasing function of the argument.
Alternatively, and this will be our option, we will minimize the
negative of the logarithm since this is a monotonically decreasing
function.
<p>
Note also that maximization/minimization of the logarithm of the PDF
is equivalent to the maximization/minimization of the function itself.
</section>
<section>
<h2 id="friday-september-10">Friday September 10 </h2>
doc/pub/week36/html/week36-solarized.html
+39 -5
View File
@@ -140,6 +140,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'independent-and-identically-distrubuted-iid'),
('Maximum Likelihood Estimation (MLE)',
2,
None,
'maximum-likelihood-estimation-mle'),
('Friday September 10', 2, None, 'friday-september-10'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
@@ -845,12 +849,12 @@ $$
which yields
$$
\beta_0=frac{16+\lambda}{8},
\beta_0=\frac{16+\lambda}{8},
$$
and
$$
\beta_1=frac{4+\lambda}{2}.
\beta_1=\frac{4+\lambda}{2}.
$$
<p>
@@ -1051,7 +1055,7 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
<p>
Our basic assumption when we derived the OLS equations was to assume
that our output is determing by a given continuous function
that our output is determined by a given continuous function
\( f(\boldsymbol{x}) \) and a random noise \( \boldsymbol{\epsilon} \) given by the normal
distribution with zero mean value and an undetermined variance
\( \sigma^2 \).
@@ -1077,7 +1081,7 @@ $$
We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution.
We define this distribution as
$$
p(y_i\vert \boldsymbol{X};\boldsymbol{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
p(y_i\vert \boldsymbol{X};\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
$$
which reads as finding the likelihood of an event \( y_i \) given the input variables \( \boldsymbol{X} \) and the parameters (to be determined) \( \boldsymbol{beta} \).
@@ -1086,12 +1090,42 @@ which reads as finding the likelihood of an event \( y_i \) given the input vari
Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have
$$
p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{beta}).
p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{\beta}).
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="maximum-likelihood-estimation-mle">Maximum Likelihood Estimation (MLE) </h2>
<p>
In statistics, maximum likelihood estimation (MLE) is a method of
estimating the parameters of an assumed probability distribution,
given some observed data. This is achieved by maximizing a likelihood
function so that, under the assumed statistical model, the observed
data is the most probable.
<p>
We will assume here that our events are given by the above Gaussian
distribution and we will determine the optimal parameters \( \beta \) by
maximizing the above PDF. However, computing the derivatives of a
product function is cumbersome and can easily lead to overflow and/or
underflowproblems, with potentials for loss of numerical precision.
<p>
In practice, it is more convenient to maximize the logarithm of the
PDF because it is a monotonically increasing function of the argument.
Alternatively, and this will be our option, we will minimize the
negative of the logarithm since this is a monotonically decreasing
function.
<p>
Note also that maximization/minimization of the logarithm of the PDF
is equivalent to the maximization/minimization of the function itself.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="friday-september-10">Friday September 10 </h2>
<p>
doc/pub/week36/html/week36.html
+39 -5
View File
@@ -145,6 +145,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'independent-and-identically-distrubuted-iid'),
('Maximum Likelihood Estimation (MLE)',
2,
None,
'maximum-likelihood-estimation-mle'),
('Friday September 10', 2, None, 'friday-september-10'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
@@ -850,12 +854,12 @@ $$
which yields
$$
\beta_0=frac{16+\lambda}{8},
\beta_0=\frac{16+\lambda}{8},
$$
and
$$
\beta_1=frac{4+\lambda}{2}.
\beta_1=\frac{4+\lambda}{2}.
$$
<p>
@@ -1056,7 +1060,7 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
<p>
Our basic assumption when we derived the OLS equations was to assume
that our output is determing by a given continuous function
that our output is determined by a given continuous function
\( f(\boldsymbol{x}) \) and a random noise \( \boldsymbol{\epsilon} \) given by the normal
distribution with zero mean value and an undetermined variance
\( \sigma^2 \).
@@ -1082,7 +1086,7 @@ $$
We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution.
We define this distribution as
$$
p(y_i\vert \boldsymbol{X};\boldsymbol{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
p(y_i\vert \boldsymbol{X};\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
$$
which reads as finding the likelihood of an event \( y_i \) given the input variables \( \boldsymbol{X} \) and the parameters (to be determined) \( \boldsymbol{beta} \).
@@ -1091,12 +1095,42 @@ which reads as finding the likelihood of an event \( y_i \) given the input vari
Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have
$$
p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{beta}).
p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{\beta}).
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="maximum-likelihood-estimation-mle">Maximum Likelihood Estimation (MLE) </h2>
<p>
In statistics, maximum likelihood estimation (MLE) is a method of
estimating the parameters of an assumed probability distribution,
given some observed data. This is achieved by maximizing a likelihood
function so that, under the assumed statistical model, the observed
data is the most probable.
<p>
We will assume here that our events are given by the above Gaussian
distribution and we will determine the optimal parameters \( \beta \) by
maximizing the above PDF. However, computing the derivatives of a
product function is cumbersome and can easily lead to overflow and/or
underflowproblems, with potentials for loss of numerical precision.
<p>
In practice, it is more convenient to maximize the logarithm of the
PDF because it is a monotonically increasing function of the argument.
Alternatively, and this will be our option, we will minimize the
negative of the logarithm since this is a monotonically decreasing
function.
<p>
Note also that maximization/minimization of the logarithm of the PDF
is equivalent to the maximization/minimization of the function itself.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="friday-september-10">Friday September 10 </h2>
<p>
doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week36/ipynb/week36.ipynb
+33 -5
View File
@@ -1062,7 +1062,7 @@
"metadata": {},
"source": [
"$$\n",
"\\beta_0=frac{16+\\lambda}{8},\n",
"\\beta_0=\\frac{16+\\lambda}{8},\n",
"$$"
]
},
@@ -1078,7 +1078,7 @@
"metadata": {},
"source": [
"$$\n",
"\\beta_1=frac{4+\\lambda}{2}.\n",
"\\beta_1=\\frac{4+\\lambda}{2}.\n",
"$$"
]
},
@@ -1363,7 +1363,7 @@
"## Deriving OLS from a probability distribution\n",
"\n",
"Our basic assumption when we derived the OLS equations was to assume\n",
"that our output is determing by a given continuous function\n",
"that our output is determined by a given continuous function\n",
"$f(\\boldsymbol{x})$ and a random noise $\\boldsymbol{\\epsilon}$ given by the normal\n",
"distribution with zero mean value and an undetermined variance\n",
"$\\sigma^2$.\n",
@@ -1400,7 +1400,7 @@
"metadata": {},
"source": [
"$$\n",
"p(y_i\\vert \\boldsymbol{X};\\boldsymbol{beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n",
"p(y_i\\vert \\boldsymbol{X};\\boldsymbol{\\beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n",
"$$"
]
},
@@ -1418,7 +1418,7 @@
"metadata": {},
"source": [
"$$\n",
"p(\\boldsymbol{y}\\vert \\boldsymbol{X};\\boldsymbol{beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i\\vert \\boldsymbol{X};\\boldsymbol{beta}).\n",
"p(\\boldsymbol{y}\\vert \\boldsymbol{X};\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i\\vert \\boldsymbol{X};\\boldsymbol{\\beta}).\n",
"$$"
]
},
@@ -1426,6 +1426,34 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Maximum Likelihood Estimation (MLE)\n",
"\n",
"In statistics, maximum likelihood estimation (MLE) is a method of\n",
"estimating the parameters of an assumed probability distribution,\n",
"given some observed data. This is achieved by maximizing a likelihood\n",
"function so that, under the assumed statistical model, the observed\n",
"data is the most probable. \n",
"\n",
"\n",
"We will assume here that our events are given by the above Gaussian\n",
"distribution and we will determine the optimal parameters $\\beta$ by\n",
"maximizing the above PDF. However, computing the derivatives of a\n",
"product function is cumbersome and can easily lead to overflow and/or\n",
"underflowproblems, with potentials for loss of numerical precision.\n",
"\n",
"\n",
"In practice, it is more convenient to maximize the logarithm of the\n",
"PDF because it is a monotonically increasing function of the argument.\n",
"Alternatively, and this will be our option, we will minimize the\n",
"negative of the logarithm since this is a monotonically decreasing\n",
"function.\n",
"\n",
"Note also that maximization/minimization of the logarithm of the PDF\n",
"is equivalent to the maximization/minimization of the function itself.\n",
"\n",
"\n",
"\n",
"\n",
"## Friday September 10\n",
"\n",
"More text will be added here.\n",
doc/src/week36/week36.do.txt
+32 -5
View File
@@ -574,13 +574,13 @@ which yields
!bt
\[
\beta_0=frac{16+\lambda}{8},
\beta_0=\frac{16+\lambda}{8},
\]
!et
and
!bt
\[
\beta_1=frac{4+\lambda}{2}.
\beta_1=\frac{4+\lambda}{2}.
\]
!et
@@ -763,7 +763,7 @@ This means the variance we obtain with the standard OLS will always for $\lambda
===== Deriving OLS from a probability distribution =====
Our basic assumption when we derived the OLS equations was to assume
that our output is determing by a given continuous function
that our output is determined by a given continuous function
$f(\bm{x})$ and a random noise $\bm{\epsilon}$ given by the normal
distribution with zero mean value and an undetermined variance
$\sigma^2$.
@@ -788,7 +788,7 @@ We assume now that the various $y_i$ values are stochastically distributed accor
We define this distribution as
!bt
\[
p(y_i\vert \bm{X};\bm{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]},
p(y_i\vert \bm{X};\bm{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]},
\]
!et
which reads as finding the likelihood of an event $y_i$ given the input variables $\bm{X}$ and the parameters (to be determined) $\bm{beta}$.
@@ -797,10 +797,37 @@ Since these events are assumed to be independent and identicall distributed we c
!bt
\[
p(\bm{y}\vert \bm{X};\bm{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \bm{X};\bm{beta}).
p(\bm{y}\vert \bm{X};\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \bm{X};\bm{\beta}).
\]
!et
!split
===== Maximum Likelihood Estimation (MLE) =====
In statistics, maximum likelihood estimation (MLE) is a method of
estimating the parameters of an assumed probability distribution,
given some observed data. This is achieved by maximizing a likelihood
function so that, under the assumed statistical model, the observed
data is the most probable.
We will assume here that our events are given by the above Gaussian
distribution and we will determine the optimal parameters $\beta$ by
maximizing the above PDF. However, computing the derivatives of a
product function is cumbersome and can easily lead to overflow and/or
underflowproblems, with potentials for loss of numerical precision.
In practice, it is more convenient to maximize the logarithm of the
PDF because it is a monotonically increasing function of the argument.
Alternatively, and this will be our option, we will minimize the
negative of the logarithm since this is a monotonically decreasing
function.
Note also that maximization/minimization of the logarithm of the PDF
is equivalent to the maximization/minimization of the function itself.
!split