lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating weel 36

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-08 13:40:29 +02:00
parent 9ae38140a8
commit b8ca339713
7 changed files with 266 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week36/html/week36-bs.html
+37 -33
View File
@@ -85,6 +85,8 @@ Automatically generated HTML file from DocOnce source
'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'),
('Ridge Regression', 2, None, 'ridge-regression'),
('Lasso Regression', 2, None, 'lasso-regression'),
('Yet another Example', 2, None, 'yet-another-example'),
('The OLS case', 2, None, 'the-ols-case'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -241,38 +243,40 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week36-bs014.html#simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression" style="font-size: 80%;">Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs015.html#ridge-regression" style="font-size: 80%;">Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs016.html#lasso-regression" style="font-size: 80%;">Lasso Regression</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs017.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs018.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs019.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs020.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs021.html#friday-september-10" style="font-size: 80%;">Friday September 10</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs025.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs027.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs024.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs025.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs026.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs027.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs028.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs029.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs030.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs031.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs033.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs035.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs038.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs039.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs040.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs041.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs042.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs043.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs044.html#summing-up" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs045.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs046.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs047.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs048.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs017.html#yet-another-example" style="font-size: 80%;">Yet another Example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs018.html#the-ols-case" style="font-size: 80%;">The OLS case</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs019.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs020.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs021.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs022.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs023.html#friday-september-10" style="font-size: 80%;">Friday September 10</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs027.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs029.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs026.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs027.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs028.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs029.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs030.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs031.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs033.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs035.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs038.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs039.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs040.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs041.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs042.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs043.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs044.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs045.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs046.html#summing-up" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs047.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs048.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs049.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs050.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
</ul>
</li>
@@ -331,7 +335,7 @@ MathJax.Hub.Config({
<li><a href="._week36-bs008.html">9</a></li>
<li><a href="._week36-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week36-bs048.html">49</a></li>
<li><a href="._week36-bs050.html">51</a></li>
<li><a href="._week36-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week36/html/week36-reveal.html
+45
View File
@@ -662,6 +662,51 @@ We will amongst other things show that the regularization parameter can reduce c
</section>
<section>
<h2 id="yet-another-example">Yet another Example </h2>
<p>
Let us assume we have a data set with outputs/targets given by the vector
<p>&nbsp;<br>
$$
\boldsymbol{y}=\begin{matrix}4 \\ 2 \\3\end{bmatrix},
$$
<p>&nbsp;<br>
and our inputs as a \( 3\times 2 \) design matrix
<p>&nbsp;<br>
$$
\boldsymbol{X}=\begin{matrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix},
$$
<p>&nbsp;<br>
meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression.
</section>
<section>
<h2 id="the-ols-case">The OLS case </h2>
<p>
For ordinary least squares (OLS) we know that the optimal solution is
<p>&nbsp;<br>
$$
\hat{\boldsymbol{\beta}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
<p>&nbsp;<br>
Inserting the above values we obtain that
<p>&nbsp;<br>
$$
\hat{\boldsymbol{\beta}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix},
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation </h2>
doc/pub/week36/html/week36-solarized.html
+39
View File
@@ -105,6 +105,8 @@ div { text-align: justify; text-justify: inter-word; }
'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'),
('Ridge Regression', 2, None, 'ridge-regression'),
('Lasso Regression', 2, None, 'lasso-regression'),
('Yet another Example', 2, None, 'yet-another-example'),
('The OLS case', 2, None, 'the-ols-case'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -676,6 +678,43 @@ Plotting these results (figure to come) shows clearly that Lasso regression supp
We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.
We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="yet-another-example">Yet another Example </h2>
<p>
Let us assume we have a data set with outputs/targets given by the vector
$$
\boldsymbol{y}=\begin{matrix}4 \\ 2 \\3\end{bmatrix},
$$
and our inputs as a \( 3\times 2 \) design matrix
$$
\boldsymbol{X}=\begin{matrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix},
$$
meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-ols-case">The OLS case </h2>
<p>
For ordinary least squares (OLS) we know that the optimal solution is
$$
\hat{\boldsymbol{\beta}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
Inserting the above values we obtain that
$$
\hat{\boldsymbol{\beta}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix},
$$
<p>
<!-- !split -->
doc/pub/week36/html/week36.html
+39
View File
@@ -110,6 +110,8 @@ div { text-align: justify; text-justify: inter-word; }
'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'),
('Ridge Regression', 2, None, 'ridge-regression'),
('Lasso Regression', 2, None, 'lasso-regression'),
('Yet another Example', 2, None, 'yet-another-example'),
('The OLS case', 2, None, 'the-ols-case'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -681,6 +683,43 @@ Plotting these results (figure to come) shows clearly that Lasso regression supp
We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.
We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="yet-another-example">Yet another Example </h2>
<p>
Let us assume we have a data set with outputs/targets given by the vector
$$
\boldsymbol{y}=\begin{matrix}4 \\ 2 \\3\end{bmatrix},
$$
and our inputs as a \( 3\times 2 \) design matrix
$$
\boldsymbol{X}=\begin{matrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix},
$$
meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-ols-case">The OLS case </h2>
<p>
For ordinary least squares (OLS) we know that the optimal solution is
$$
\hat{\boldsymbol{\beta}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
Inserting the above values we obtain that
$$
\hat{\boldsymbol{\beta}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix},
$$
<p>
<!-- !split -->
doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week36/ipynb/week36.ipynb
+69
View File
@@ -778,7 +778,76 @@
"We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\\beta$.\n",
"\n",
"\n",
"## Yet another Example\n",
"\n",
"Let us assume we have a data set with outputs/targets given by the vector"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{y}=\\begin{matrix}4 \\\\ 2 \\\\3\\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and our inputs as a $3\\times 2$ design matrix"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{matrix}2 & 0\\\\ 0 & 1 \\\\ 1 & 0\\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"meaning that we have two features and two unknown parameters $\\beta_0$ and $\\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression.\n",
"\n",
"## The OLS case\n",
"\n",
"For ordinary least squares (OLS) we know that the optimal solution is"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Inserting the above values we obtain that"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}=\\begin{matrix}\\frac{11}{5} \\\\ 2\\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- !split -->\n",
"## Linking the regression analysis with a statistical interpretation\n",
"\n",
doc/src/week36/week36.do.txt
+37
View File
@@ -429,6 +429,43 @@ We will now couple the discussions of ordinary least squares, Ridge and Lasso re
We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\beta$.
!split
===== Yet another Example =====
Let us assume we have a data set with outputs/targets given by the vector
!bt
\[
\bm{y}=\begin{matrix}4 \\ 2 \\3\end{bmatrix},
\]
!et
and our inputs as a $3\times 2$ design matrix
!bt
\[
\bm{X}=\begin{matrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix},
\]
!et
meaning that we have two features and two unknown parameters $\beta_0$ and $\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression.
!split
===== The OLS case =====
For ordinary least squares (OLS) we know that the optimal solution is
!bt
\[
\hat{\bm{\beta}}=\left( \bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\]
!et
Inserting the above values we obtain that
!bt
\[
\hat{\bm{\beta}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix},
\]
!et
!split
===== Linking the regression analysis with a statistical interpretation =====