lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

adding Bayes

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-09 07:56:51 +02:00
parent eb12e84cf6
commit 0650414b41
7 changed files with 288 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week36/html/week36-bs.html
+35 -30
View File
@@ -125,7 +125,11 @@ Automatically generated HTML file from DocOnce source
None,
'maximum-likelihood-estimation-mle'),
('A new Cost Function', 2, None, 'a-new-cost-function'),
('Friday September 10', 2, None, 'friday-september-10'),
("More basic Statistics and Bayes' theorem",
2,
None,
'more-basic-statistics-and-bayes-theorem'),
('Frliday September 10', 2, None, 'frliday-september-10'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
('Resampling approaches can be computationally expensive',
@@ -282,34 +286,35 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week36-bs029.html#independent-and-identically-distrubuted-iid" style="font-size: 80%;">Independent and Identically Distrubuted (iid)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs030.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs031.html#a-new-cost-function" style="font-size: 80%;">A new Cost Function</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#friday-september-10" style="font-size: 80%;">Friday September 10</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs038.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs035.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs038.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs039.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs040.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs041.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs042.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs043.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs044.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs045.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs046.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs047.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs048.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs049.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs050.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs051.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs052.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs053.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs054.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs055.html#summing-up" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs056.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs057.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs058.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs059.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#more-basic-statistics-and-bayes-theorem" style="font-size: 80%;">More basic Statistics and Bayes' theorem</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs033.html#frliday-september-10" style="font-size: 80%;">Frliday September 10</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs039.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs038.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs039.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs040.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs041.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs042.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs043.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs044.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs045.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs046.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs047.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs048.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs049.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs050.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs051.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs052.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs053.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs054.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs055.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs056.html#summing-up" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs057.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs058.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs059.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs060.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
</ul>
</li>
@@ -368,7 +373,7 @@ MathJax.Hub.Config({
<li><a href="._week36-bs008.html">9</a></li>
<li><a href="._week36-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week36-bs059.html">60</a></li>
<li><a href="._week36-bs060.html">61</a></li>
<li><a href="._week36-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week36/html/week36-reveal.html
+49 -5
View File
@@ -731,8 +731,8 @@ $$
<p>&nbsp;<br>
<p>
There is normally a constraint on the value of $\vert\vert \boldsymbol{\beta}\vert\vert_2 via the parameter \( \lambda \).
Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression with the optimal values of \( \beta \) and \( \lambda \).
There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \).
Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \).
<p>
To see this, let us write the cost function for Ridge regression.
@@ -1173,17 +1173,61 @@ $$
$$
<p>&nbsp;<br>
which leads to
which leads to the well-known OLS equation for the optimal paramters \( \beta \)
<p>&nbsp;<br>
$$
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
$$
<p>&nbsp;<br>
<p>
Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics.
</section>
<section>
<h2 id="friday-september-10">Friday September 10 </h2>
<h2 id="more-basic-statistics-and-bayes-theorem">More basic Statistics and Bayes' theorem </h2>
<p>
A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry.
Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics.
<p>
Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \).
<p>
We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively.
The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \).
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>Union of events is given by</b>
<p>&nbsp;<br>
$$
p(X \cup Y)= p(X)+p(Y)-p(X \cap Y).
$$
<p>&nbsp;<br>
</div>
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>The product rule is given by</b>
<p>&nbsp;<br>
$$
p(X \cup Y)= p(X,y)= p(X\vert Y)p(Y)=p(Y\vert X)p(x),
$$
<p>&nbsp;<br>
where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \).
</div>
</section>
<section>
<h2 id="frliday-september-10">Frliday September 10 </h2>
<p>
Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.
</section>
doc/pub/week36/html/week36-solarized.html
+54 -6
View File
@@ -145,7 +145,11 @@ div { text-align: justify; text-justify: inter-word; }
None,
'maximum-likelihood-estimation-mle'),
('A new Cost Function', 2, None, 'a-new-cost-function'),
('Friday September 10', 2, None, 'friday-september-10'),
("More basic Statistics and Bayes' theorem",
2,
None,
'more-basic-statistics-and-bayes-theorem'),
('Frliday September 10', 2, None, 'frliday-september-10'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
('Resampling approaches can be computationally expensive',
@@ -761,8 +765,8 @@ $$
$$
<p>
There is normally a constraint on the value of $\vert\vert \boldsymbol{\beta}\vert\vert_2 via the parameter \( \lambda \).
Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression with the optimal values of \( \beta \) and \( \lambda \).
There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \).
Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \).
<p>
To see this, let us write the cost function for Ridge regression.
@@ -1148,15 +1152,59 @@ $$
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0,
$$
which leads to
which leads to the well-known OLS equation for the optimal paramters \( \beta \)
$$
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
$$
<p>
Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="friday-september-10">Friday September 10 </h2>
<h2 id="more-basic-statistics-and-bayes-theorem">More basic Statistics and Bayes' theorem </h2>
<p>
A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry.
Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics.
<p>
Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \).
<p>
We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively.
The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \).
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>Union of events is given by</b>
<p>
$$
p(X \cup Y)= p(X)+p(Y)-p(X \cap Y).
$$
</div>
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>The product rule is given by</b>
<p>
$$
p(X \cup Y)= p(X,y)= p(X\vert Y)p(Y)=p(Y\vert X)p(x),
$$
where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \).
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="frliday-september-10">Frliday September 10 </h2>
<p>
Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week36/html/week36.html
+54 -6
View File
@@ -150,7 +150,11 @@ div { text-align: justify; text-justify: inter-word; }
None,
'maximum-likelihood-estimation-mle'),
('A new Cost Function', 2, None, 'a-new-cost-function'),
('Friday September 10', 2, None, 'friday-september-10'),
("More basic Statistics and Bayes' theorem",
2,
None,
'more-basic-statistics-and-bayes-theorem'),
('Frliday September 10', 2, None, 'frliday-september-10'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
('Resampling approaches can be computationally expensive',
@@ -766,8 +770,8 @@ $$
$$
<p>
There is normally a constraint on the value of $\vert\vert \boldsymbol{\beta}\vert\vert_2 via the parameter \( \lambda \).
Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression with the optimal values of \( \beta \) and \( \lambda \).
There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \).
Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \).
<p>
To see this, let us write the cost function for Ridge regression.
@@ -1153,15 +1157,59 @@ $$
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0,
$$
which leads to
which leads to the well-known OLS equation for the optimal paramters \( \beta \)
$$
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
$$
<p>
Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="friday-september-10">Friday September 10 </h2>
<h2 id="more-basic-statistics-and-bayes-theorem">More basic Statistics and Bayes' theorem </h2>
<p>
A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry.
Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics.
<p>
Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \).
<p>
We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively.
The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \).
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>Union of events is given by</b>
<p>
$$
p(X \cup Y)= p(X)+p(Y)-p(X \cap Y).
$$
</div>
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>The product rule is given by</b>
<p>
$$
p(X \cup Y)= p(X,y)= p(X\vert Y)p(Y)=p(Y\vert X)p(x),
$$
where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \).
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="frliday-september-10">Frliday September 10 </h2>
<p>
Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week36/ipynb/week36.ipynb
+56 -5
View File
@@ -884,8 +884,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2 via the parameter $\\lambda$.\n",
"Let us for simplicity assume now that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression with the optimal values of $\\beta$ and $\\lambda$.\n",
"There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2$ via the parameter $\\lambda$.\n",
"Let us for simplicity assume that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\beta$ and $\\lambda$.\n",
"\n",
"To see this, let us write the cost function for Ridge regression. \n",
"\n",
@@ -1503,7 +1503,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"which leads to"
"which leads to the well-known OLS equation for the optimal paramters $\\beta$"
]
},
{
@@ -1511,7 +1511,7 @@
"metadata": {},
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n",
"\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n",
"$$"
]
},
@@ -1519,8 +1519,59 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Friday September 10\n",
"Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics. \n",
"\n",
"## More basic Statistics and Bayes' theorem\n",
"\n",
"A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry.\n",
"Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics.\n",
"\n",
"Assume we have two domains of events $X=[x_0,x_1,\\dots,x_{n-1}]$ and $Y=[y_0,y_1,\\dots,y_{n-1}]$.\n",
"\n",
"We define also the likelihood for $X$ and $Y$ as $p(X)$ and $p(Y)$ respectively.\n",
"The likelihood of a specific event $x_i$ (or $y_i$) is then written as $p(X=x_i)$ or just $p(x_i)=p_i$. \n",
"\n",
"**Union of events is given by.**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"p(X \\cup Y)= p(X)+p(Y)-p(X \\cap Y).\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**The product rule is given by.**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"p(X \\cup Y)= p(X,y)= p(X\\vert Y)p(Y)=p(Y\\vert X)p(x),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where we read $p(X\\vert Y)$ as the likelihood of obtaining $X$ given $Y$.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"## Frliday September 10\n",
"\n",
"Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.\n",
"\n",
"\n",
"## Why resampling methods\n",
doc/src/week36/week36.do.txt
+40 -6
View File
@@ -485,8 +485,8 @@ Inserting the above values we obtain that
\]
!et
There is normally a constraint on the value of $\vert\vert \bm{\beta}\vert\vert_2 via the parameter $\lambda$.
Let us for simplicity assume now that $\beta_0^2+\beta_1^2=1$ as constraint. This will allow us to find an expression with the optimal values of $\beta$ and $\lambda$.
There is normally a constraint on the value of $\vert\vert \bm{\beta}\vert\vert_2$ via the parameter $\lambda$.
Let us for simplicity assume that $\beta_0^2+\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\beta$ and $\lambda$.
To see this, let us write the cost function for Ridge regression.
@@ -853,17 +853,51 @@ Taking the derivative of the *new* cost function with respect to the parameters
\bm{X}^T\left(\bm{y}-\bm{X}\bm{\beta}\right) =0,
\]
!et
which leads to
which leads to the well-known OLS equation for the optimal paramters $\beta$
!bt
\[
\hat{\bm{\beta}}_{\mathrm{OLS}}=\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}!
\hat{\bm{\beta}}^{\mathrm{OLS}}=\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}!
\]
!et
!split
===== Friday September 10 =====
Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics.
!split
===== More basic Statistics and Bayes' theorem =====
A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry.
Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics.
Assume we have two domains of events $X=[x_0,x_1,\dots,x_{n-1}]$ and $Y=[y_0,y_1,\dots,y_{n-1}]$.
We define also the likelihood for $X$ and $Y$ as $p(X)$ and $p(Y)$ respectively.
The likelihood of a specific event $x_i$ (or $y_i$) is then written as $p(X=x_i)$ or just $p(x_i)=p_i$.
!bblock Union of events is given by
!bt
\[
p(X \cup Y)= p(X)+p(Y)-p(X \cap Y).
\]
!et
!eblock
!bblock The product rule is given by
!bt
\[
p(X \cup Y)= p(X,y)= p(X\vert Y)p(Y)=p(Y\vert X)p(x),
\]
!et
where we read $p(X\vert Y)$ as the likelihood of obtaining $X$ given $Y$.
!eblock
!split
===== Frliday September 10 =====
Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.
!split