lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

new update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-09 06:41:28 +02:00
parent 79156f2150
commit 844d210ddf
7 changed files with 437 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week36/html/week36-bs.html
+42 -35
View File
@@ -93,6 +93,11 @@ Automatically generated HTML file from DocOnce source
None,
'writing-the-cost-function'),
('Lasso case', 2, None, 'lasso-case'),
('The first Case', 2, None, 'the-first-case'),
('Simple code for solving the above problem',
2,
None,
'simple-code-for-solving-the-above-problem'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -262,40 +267,42 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week36-bs019.html#the-ridge-case" style="font-size: 80%;">The Ridge case</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs020.html#writing-the-cost-function" style="font-size: 80%;">Writing the Cost Function</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs021.html#lasso-case" style="font-size: 80%;">Lasso case</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs022.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs023.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs024.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs025.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs026.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs027.html#independent-and-identically-distrubuted-iid" style="font-size: 80%;">Independent and Identically Distrubuted (iid)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs028.html#friday-september-10" style="font-size: 80%;">Friday September 10</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs031.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs033.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs035.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs038.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs039.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs040.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs041.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs042.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs043.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs044.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs045.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs046.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs047.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs048.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs049.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs050.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs051.html#summing-up" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs052.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs053.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs054.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs055.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs022.html#the-first-case" style="font-size: 80%;">The first Case</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs023.html#simple-code-for-solving-the-above-problem" style="font-size: 80%;">Simple code for solving the above problem</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs024.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs025.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs026.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs027.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs028.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs029.html#independent-and-identically-distrubuted-iid" style="font-size: 80%;">Independent and Identically Distrubuted (iid)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs030.html#friday-september-10" style="font-size: 80%;">Friday September 10</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs033.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs035.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs038.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs039.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs040.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs041.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs042.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs043.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs044.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs045.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs046.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs047.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs048.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs049.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs050.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs051.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs052.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs053.html#summing-up" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs054.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs055.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs056.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs057.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
</ul>
</li>
@@ -354,7 +361,7 @@ MathJax.Hub.Config({
<li><a href="._week36-bs008.html">9</a></li>
<li><a href="._week36-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week36-bs055.html">56</a></li>
<li><a href="._week36-bs057.html">58</a></li>
<li><a href="._week36-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week36/html/week36-reveal.html
+77 -8
View File
@@ -763,7 +763,7 @@ $$
$$
<p>&nbsp;<br>
and for \( \beta_1 \) we have
and for \( \beta_1 \) we obtain
<p>&nbsp;<br>
$$
\beta_1=\frac{2}{1+\lambda},
@@ -786,7 +786,7 @@ which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).
<h2 id="lasso-case">Lasso case </h2>
<p>
For Lasso we need now, keeping the same constraint on \( \beta_0^2+\beta_1^2=1 \), to take the derivative of the absolute values of \( \beta_0 \)
For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \)
and \( beta_1 \). This gives us the following derivatives of the cost function
<p>&nbsp;<br>
$$
@@ -796,14 +796,14 @@ $$
<p>&nbsp;<br>
$$
\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0,
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0,
$$
<p>&nbsp;<br>
and
<p>&nbsp;<br>
$$
\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0.
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0.
$$
<p>&nbsp;<br>
@@ -813,11 +813,60 @@ We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \
<p><li> \( \beta_0 > 0 \) and \( \beta_1 > 0 \),</li>
<p><li> \( \beta_0 > 0 \) and \( \beta_1 < 0 \),</li>
<p><li> \( \beta_0 < 0 \) and \( \beta_1 > 0 \),</li>
<p><li> \( \beta_0 < 0 \) and \( \beta_1 < 0 \),</li>
<p><li> \( \beta_0 < 0 \) and \( \beta_1 < 0 \).</li>
</ol>
</section>
<section>
<h2 id="the-first-case">The first Case </h2>
<p>
If we consider the first case, we have then
<p>&nbsp;<br>
$$
-4(4-2\beta_0)+\lambda=0,
$$
<p>&nbsp;<br>
and
<p>&nbsp;<br>
$$
-2(2-\beta_1)+\lambda=0.
$$
<p>&nbsp;<br>
which yields
<p>&nbsp;<br>
$$
\beta_0=frac{16+\lambda}{8},
$$
<p>&nbsp;<br>
and
<p>&nbsp;<br>
$$
\beta_1=frac{4+\lambda}{2}.
$$
<p>&nbsp;<br>
<p>
Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you.
</section>
<section>
<h2 id="simple-code-for-solving-the-above-problem">Simple code for solving the above problem </h2>
<p>
Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values.
<p>
First we study and compare the OLS and Ridge results. The next code compares all three methods.
</section>
<section>
<h2 id="linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation </h2>
@@ -1028,14 +1077,14 @@ distribution with zero mean value and an undetermined variance
<p>
We found above that the outputs \( \boldsymbol{y} \) have a mean value given by
\( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to
the design matrix are not stocastic variables, we can assume that the
the design matrix are not stochastic variables, we can assume that the
probability distribution of our targets is also a normal distribution
but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a
single output \( y_i \) is given by the Gaussian distribution
<p>&nbsp;<br>
$$
y_i\sim \mathcal{N}(\boldsymbol{X}{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
$$
<p>&nbsp;<br>
</section>
@@ -1045,12 +1094,32 @@ $$
<h2 id="independent-and-identically-distrubuted-iid">Independent and Identically Distrubuted (iid) </h2>
<p>
We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution and
We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution.
We define this distribution as
<p>&nbsp;<br>
$$
p(y_i\vert \boldsymbol{X};\boldsymbol{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
$$
<p>&nbsp;<br>
which reads as finding the likelihood of an event \( y_i \) given the input variables \( \boldsymbol{X} \) and the parameters (to be determined) \( \boldsymbol{beta} \).
<p>
Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have
<p>&nbsp;<br>
$$
p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{beta}).
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="friday-september-10">Friday September 10 </h2>
<p>
More text will be added here.
</section>
doc/pub/week36/html/week36-solarized.html
+70 -8
View File
@@ -113,6 +113,11 @@ div { text-align: justify; text-justify: inter-word; }
None,
'writing-the-cost-function'),
('Lasso case', 2, None, 'lasso-case'),
('The first Case', 2, None, 'the-first-case'),
('Simple code for solving the above problem',
2,
None,
'simple-code-for-solving-the-above-problem'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -778,7 +783,7 @@ $$
\beta_0=\frac{8}{4+\lambda},
$$
and for \( \beta_1 \) we have
and for \( \beta_1 \) we obtain
$$
\beta_1=\frac{2}{1+\lambda},
$$
@@ -797,7 +802,7 @@ which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).
<h2 id="lasso-case">Lasso case </h2>
<p>
For Lasso we need now, keeping the same constraint on \( \beta_0^2+\beta_1^2=1 \), to take the derivative of the absolute values of \( \beta_0 \)
For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \)
and \( beta_1 \). This gives us the following derivatives of the cost function
$$
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert),
@@ -805,12 +810,12 @@ $$
$$
\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0,
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0,
$$
and
$$
\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0.
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0.
$$
We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely
@@ -819,9 +824,50 @@ We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \
<li> \( \beta_0 > 0 \) and \( \beta_1 > 0 \),</li>
<li> \( \beta_0 > 0 \) and \( \beta_1 < 0 \),</li>
<li> \( \beta_0 < 0 \) and \( \beta_1 > 0 \),</li>
<li> \( \beta_0 < 0 \) and \( \beta_1 < 0 \),</li>
<li> \( \beta_0 < 0 \) and \( \beta_1 < 0 \).</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-first-case">The first Case </h2>
<p>
If we consider the first case, we have then
$$
-4(4-2\beta_0)+\lambda=0,
$$
and
$$
-2(2-\beta_1)+\lambda=0.
$$
which yields
$$
\beta_0=frac{16+\lambda}{8},
$$
and
$$
\beta_1=frac{4+\lambda}{2}.
$$
<p>
Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="simple-code-for-solving-the-above-problem">Simple code for solving the above problem </h2>
<p>
Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values.
<p>
First we study and compare the OLS and Ridge results. The next code compares all three methods.
<p>
<!-- !split -->
<h2 id="linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation </h2>
@@ -1013,13 +1059,13 @@ distribution with zero mean value and an undetermined variance
<p>
We found above that the outputs \( \boldsymbol{y} \) have a mean value given by
\( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to
the design matrix are not stocastic variables, we can assume that the
the design matrix are not stochastic variables, we can assume that the
probability distribution of our targets is also a normal distribution
but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a
single output \( y_i \) is given by the Gaussian distribution
$$
y_i\sim \mathcal{N}(\boldsymbol{X}{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
$$
<p>
@@ -1028,13 +1074,29 @@ $$
<h2 id="independent-and-identically-distrubuted-iid">Independent and Identically Distrubuted (iid) </h2>
<p>
We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution and
We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution.
We define this distribution as
$$
p(y_i\vert \boldsymbol{X};\boldsymbol{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
$$
which reads as finding the likelihood of an event \( y_i \) given the input variables \( \boldsymbol{X} \) and the parameters (to be determined) \( \boldsymbol{beta} \).
<p>
Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have
$$
p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{beta}).
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="friday-september-10">Friday September 10 </h2>
<p>
More text will be added here.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week36/html/week36.html
+70 -8
View File
@@ -118,6 +118,11 @@ div { text-align: justify; text-justify: inter-word; }
None,
'writing-the-cost-function'),
('Lasso case', 2, None, 'lasso-case'),
('The first Case', 2, None, 'the-first-case'),
('Simple code for solving the above problem',
2,
None,
'simple-code-for-solving-the-above-problem'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -783,7 +788,7 @@ $$
\beta_0=\frac{8}{4+\lambda},
$$
and for \( \beta_1 \) we have
and for \( \beta_1 \) we obtain
$$
\beta_1=\frac{2}{1+\lambda},
$$
@@ -802,7 +807,7 @@ which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).
<h2 id="lasso-case">Lasso case </h2>
<p>
For Lasso we need now, keeping the same constraint on \( \beta_0^2+\beta_1^2=1 \), to take the derivative of the absolute values of \( \beta_0 \)
For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \)
and \( beta_1 \). This gives us the following derivatives of the cost function
$$
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert),
@@ -810,12 +815,12 @@ $$
$$
\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0,
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0,
$$
and
$$
\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0.
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0.
$$
We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely
@@ -824,9 +829,50 @@ We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \
<li> \( \beta_0 > 0 \) and \( \beta_1 > 0 \),</li>
<li> \( \beta_0 > 0 \) and \( \beta_1 < 0 \),</li>
<li> \( \beta_0 < 0 \) and \( \beta_1 > 0 \),</li>
<li> \( \beta_0 < 0 \) and \( \beta_1 < 0 \),</li>
<li> \( \beta_0 < 0 \) and \( \beta_1 < 0 \).</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-first-case">The first Case </h2>
<p>
If we consider the first case, we have then
$$
-4(4-2\beta_0)+\lambda=0,
$$
and
$$
-2(2-\beta_1)+\lambda=0.
$$
which yields
$$
\beta_0=frac{16+\lambda}{8},
$$
and
$$
\beta_1=frac{4+\lambda}{2}.
$$
<p>
Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="simple-code-for-solving-the-above-problem">Simple code for solving the above problem </h2>
<p>
Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values.
<p>
First we study and compare the OLS and Ridge results. The next code compares all three methods.
<p>
<!-- !split -->
<h2 id="linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation </h2>
@@ -1018,13 +1064,13 @@ distribution with zero mean value and an undetermined variance
<p>
We found above that the outputs \( \boldsymbol{y} \) have a mean value given by
\( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to
the design matrix are not stocastic variables, we can assume that the
the design matrix are not stochastic variables, we can assume that the
probability distribution of our targets is also a normal distribution
but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a
single output \( y_i \) is given by the Gaussian distribution
$$
y_i\sim \mathcal{N}(\boldsymbol{X}{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
$$
<p>
@@ -1033,13 +1079,29 @@ $$
<h2 id="independent-and-identically-distrubuted-iid">Independent and Identically Distrubuted (iid) </h2>
<p>
We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution and
We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution.
We define this distribution as
$$
p(y_i\vert \boldsymbol{X};\boldsymbol{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
$$
which reads as finding the likelihood of an event \( y_i \) given the input variables \( \boldsymbol{X} \) and the parameters (to be determined) \( \boldsymbol{beta} \).
<p>
Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have
$$
p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{beta}).
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="friday-september-10">Friday September 10 </h2>
<p>
More text will be added here.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week36/ipynb/week36.ipynb
+117 -10
View File
@@ -933,7 +933,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"and for $\\beta_1$ we have"
"and for $\\beta_1$ we obtain"
]
},
{
@@ -969,7 +969,7 @@
"\n",
"## Lasso case\n",
"\n",
"For Lasso we need now, keeping the same constraint on $\\beta_0^2+\\beta_1^2=1$, to take the derivative of the absolute values of $\\beta_0$\n",
"For Lasso we need now, keeping a constraint on $\\vert\\beta_0\\vert+\\vert\\beta_1\\vert=1$, to take the derivative of the absolute values of $\\beta_0$\n",
"and $beta_1$. This gives us the following derivatives of the cost function"
]
},
@@ -987,7 +987,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partialC(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-2(4-2\\beta_0)+\\lambda\\mathrm{sgn}\\beta_0=0,\n",
"\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-4(4-2\\beta_0)+\\lambda\\mathrm{sgn}(\\beta_0)=0,\n",
"$$"
]
},
@@ -1003,7 +1003,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partialC(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}\\beta_1=0.\n",
"\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}(\\beta_1)=0.\n",
"$$"
]
},
@@ -1018,7 +1018,82 @@
"\n",
"3. $\\beta_0 < 0$ and $\\beta_1 > 0$,\n",
"\n",
"4. $\\beta_0 < 0$ and $\\beta_1 < 0$,\n",
"4. $\\beta_0 < 0$ and $\\beta_1 < 0$.\n",
"\n",
"## The first Case\n",
"\n",
"If we consider the first case, we have then"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"-4(4-2\\beta_0)+\\lambda=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"-2(2-\\beta_1)+\\lambda=0.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which yields"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\beta_0=frac{16+\\lambda}{8},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\beta_1=frac{4+\\lambda}{2}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the constraint on $\\beta_0$ and $\\beta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you.\n",
"\n",
"## Simple code for solving the above problem\n",
"\n",
"Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of $\\lambda$, meaning that we need to perform a search in order to find the optimal values.\n",
"\n",
"First we study and compare the OLS and Ridge results. The next code compares all three methods.\n",
"\n",
"\n",
"<!-- !split -->\n",
"## Linking the regression analysis with a statistical interpretation\n",
@@ -1295,7 +1370,7 @@
"\n",
"We found above that the outputs $\\boldsymbol{y}$ have a mean value given by\n",
"$\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}}$ and variance $\\sigma^2$. Since the entries to\n",
"the design matrix are not stocastic variables, we can assume that the\n",
"the design matrix are not stochastic variables, we can assume that the\n",
"probability distribution of our targets is also a normal distribution\n",
"but now with mean value $\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}}$. This means that a\n",
"single output $y_i$ is given by the Gaussian distribution"
@@ -1306,7 +1381,7 @@
"metadata": {},
"source": [
"$$\n",
"y_i\\sim \\mathcal{N}(\\boldsymbol{X}{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
"y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
"$$"
]
},
@@ -1316,12 +1391,44 @@
"source": [
"## Independent and Identically Distrubuted (iid)\n",
"\n",
"We assume now that the various $y_i$ values are stochastically distributed according to the above Gaussian distribution and \n",
"\n",
"We assume now that the various $y_i$ values are stochastically distributed according to the above Gaussian distribution. \n",
"We define this distribution as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"p(y_i\\vert \\boldsymbol{X};\\boldsymbol{beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which reads as finding the likelihood of an event $y_i$ given the input variables $\\boldsymbol{X}$ and the parameters (to be determined) $\\boldsymbol{beta}$.\n",
"\n",
"Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event $\\boldsymbol{y}$ as the product of the single events, that is we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"p(\\boldsymbol{y}\\vert \\boldsymbol{X};\\boldsymbol{beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i\\vert \\boldsymbol{X};\\boldsymbol{beta}).\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Friday September 10\n",
"\n",
"\n",
"More text will be added here.\n",
"\n",
"## Why resampling methods\n",
"\n",
doc/src/week36/week36.do.txt
+61 -9
View File
@@ -512,7 +512,7 @@ and taking the derivative with respect to $\beta_0$ we get
\beta_0=\frac{8}{4+\lambda},
\]
!et
and for $\beta_1$ we have
and for $\beta_1$ we obtain
!bt
\[
\beta_1=\frac{2}{1+\lambda},
@@ -530,7 +530,7 @@ which gives $\lambda=4.571$ and $\beta_0=0.933$ and $\beta_1=0.359$.
!split
===== Lasso case =====
For Lasso we need now, keeping the same constraint on $\beta_0^2+\beta_1^2=1$, to take the derivative of the absolute values of $\beta_0$
For Lasso we need now, keeping a constraint on $\vert\beta_0\vert+\vert\beta_1\vert=1$, to take the derivative of the absolute values of $\beta_0$
and $beta_1$. This gives us the following derivatives of the cost function
!bt
\[
@@ -540,22 +540,58 @@ C(\bm{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_
!bt
\[
\frac{\partialC(\bm{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0,
\frac{\partial C(\bm{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0,
\]
!et
and
!bt
\[
\frac{\partialC(\bm{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0.
\frac{\partial C(\bm{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0.
\]
!et
We have now four cases to solve besides the trivial cases $\beta_0$ and/or $\beta_1$ are zero, namely
o $\beta_0 > 0$ and $\beta_1 > 0$,
o $\beta_0 > 0$ and $\beta_1 < 0$,
o $\beta_0 < 0$ and $\beta_1 > 0$,
o $\beta_0 < 0$ and $\beta_1 < 0$,
o $\beta_0 < 0$ and $\beta_1 < 0$.
!split
===== The first Case =====
If we consider the first case, we have then
!bt
\[
-4(4-2\beta_0)+\lambda=0,
\]
!et
and
!bt
\[
-2(2-\beta_1)+\lambda=0.
\]
!et
which yields
!bt
\[
\beta_0=frac{16+\lambda}{8},
\]
!et
and
!bt
\[
\beta_1=frac{4+\lambda}{2}.
\]
!et
Using the constraint on $\beta_0$ and $\beta_1$ we can then find the optimal value of $\lambda$ for the different cases. We leave this as an exercise to you.
!split
===== Simple code for solving the above problem =====
Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of $\lambda$, meaning that we need to perform a search in order to find the optimal values.
First we study and compare the OLS and Ridge results. The next code compares all three methods.
!split
@@ -734,27 +770,43 @@ $\sigma^2$.
We found above that the outputs $\bm{y}$ have a mean value given by
$\bm{X}\hat{\bm{\beta}}$ and variance $\sigma^2$. Since the entries to
the design matrix are not stocastic variables, we can assume that the
the design matrix are not stochastic variables, we can assume that the
probability distribution of our targets is also a normal distribution
but now with mean value $\bm{X}\hat{\bm{\beta}}$. This means that a
single output $y_i$ is given by the Gaussian distribution
!bt
\[
y_i\sim \mathcal{N}(\bm{X}{i,*}\bm{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}{i,*}\bm{\beta})^2}{2\sigma^2}\right]}.
y_i\sim \mathcal{N}(\bm{X}_{i,*}\bm{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}.
\]
!et
!split
===== Independent and Identically Distrubuted (iid) =====
We assume now that the various $y_i$ values are stochastically distributed according to the above Gaussian distribution and
We assume now that the various $y_i$ values are stochastically distributed according to the above Gaussian distribution.
We define this distribution as
!bt
\[
p(y_i\vert \bm{X};\bm{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]},
\]
!et
which reads as finding the likelihood of an event $y_i$ given the input variables $\bm{X}$ and the parameters (to be determined) $\bm{beta}$.
Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event $\bm{y}$ as the product of the single events, that is we have
!bt
\[
p(\bm{y}\vert \bm{X};\bm{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \bm{X};\bm{beta}).
\]
!et
!split
===== Friday September 10 =====
More text will be added here.
!split
===== Why resampling methods =====