lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating exercises

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-09-11 06:35:24 +02:00
parent c89e87a5e5
commit b7e90afa3a
3 changed files with 3202 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/exercisesweek37.ipynb
+261
View File
@@ -0,0 +1,261 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "28222517",
"metadata": {
"editable": true
},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek37.do.txt -->\n",
"<!-- dom:TITLE: Exercises week 37 -->"
]
},
{
"cell_type": "markdown",
"id": "c7ffd7b0",
"metadata": {
"editable": true
},
"source": [
"# Exercises week 37\n",
"**September 11-15, 2023**\n",
"\n",
"Date: **Deadline is Sunday September 17 at midnight**"
]
},
{
"cell_type": "markdown",
"id": "f687d0b0",
"metadata": {
"editable": true
},
"source": [
"## Overarching aims of the exercises this week\n",
"\n",
"This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of [Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer](https://www.springer.com/gp/book/9780387848570)). The exercise is also a part of project 1 and can be reused in the theory part of the project.\n",
"\n",
"For more discussions on Ridge regression and calculation of expectation values, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.\n",
"\n",
"The assumption we have made is \n",
"that there exists a continuous function $f(\\boldsymbol{x})$ and a normal distributed error $\\boldsymbol{\\varepsilon}\\sim N(0, \\sigma^2)$\n",
"which describes our data"
]
},
{
"cell_type": "markdown",
"id": "b0b5ae03",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "a5442d69",
"metadata": {
"editable": true
},
"source": [
"We then approximate this function $f(\\boldsymbol{x})$ with our model $\\boldsymbol{\\tilde{y}}$ from the solution of the linear regression equations (ordinary least squares OLS), that is our\n",
"function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we minimized $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, with"
]
},
{
"cell_type": "markdown",
"id": "d9ac69f5",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "6f42da1d",
"metadata": {
"editable": true
},
"source": [
"The matrix $\\boldsymbol{X}$ is the so-called design or feature matrix."
]
},
{
"cell_type": "markdown",
"id": "5690cef0",
"metadata": {
"editable": true
},
"source": [
"## Exercise 1: Expectation values for ordinary least squares expressions\n",
"\n",
"Show that the expectation value of $\\boldsymbol{y}$ for a given element $i$"
]
},
{
"cell_type": "markdown",
"id": "3834fc47",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mathbb{E}(y_i) =\\sum_{j}x_{ij} \\beta_j=\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta},\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "bcfd68b4",
"metadata": {
"editable": true
},
"source": [
"and that\n",
"its variance is"
]
},
{
"cell_type": "markdown",
"id": "9551e381",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mbox{Var}(y_i) = \\sigma^2.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "e470afef",
"metadata": {
"editable": true
},
"source": [
"Hence, $y_i \\sim N( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
"mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$.\n",
"\n",
"With the OLS expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}}$ show that"
]
},
{
"cell_type": "markdown",
"id": "6ece00ae",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mathbb{E}(\\boldsymbol{\\hat{\\beta}}) = \\boldsymbol{\\beta}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "98bad716",
"metadata": {
"editable": true
},
"source": [
"Show finally that the variance of $\\boldsymbol{\\beta}$ is"
]
},
{
"cell_type": "markdown",
"id": "a8dad13c",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mbox{Var}(\\boldsymbol{\\hat{\\beta}}) = \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "94cddcc3",
"metadata": {
"editable": true
},
"source": [
"We can use the last expression when we define a [so-called confidence interval](https://en.wikipedia.org/wiki/Confidence_interval) for the parameters $\\beta$. \n",
"A given parameter $\\beta_j$ is given by the diagonal matrix element of the above matrix."
]
},
{
"cell_type": "markdown",
"id": "8e5c2d66",
"metadata": {
"editable": true
},
"source": [
"## Exercise 2: Expectation values for Ridge regression\n",
"\n",
"Show that"
]
},
{
"cell_type": "markdown",
"id": "5d7e493e",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\\n",
"\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "44050008",
"metadata": {
"editable": true
},
"source": [
"We see clearly that\n",
"$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$.\n",
"\n",
"Show also that the variance is"
]
},
{
"cell_type": "markdown",
"id": "c0bf3608",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\m\\\n",
"athbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "371484f8",
"metadata": {
"editable": true
},
"source": [
"and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero."
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}
doc/LectureNotes/week37.ipynb
+2912
View File
File diff suppressed because it is too large Load Diff
doc/src/week37/exercisesweek37.do.txt
+29 -1
View File
@@ -29,6 +29,7 @@ function $f$ is approximated by $\bm{\tilde{y}}$ where we minimized $(\bm{y}-\b
!et
The matrix $\bm{X}$ is the so-called design or feature matrix.
===== Exercise: Expectation values for ordinary least squares expressions =====
Show that the expectation value of $\bm{y}$ for a given element $i$
!bt
@@ -60,7 +61,34 @@ Show finally that the variance of $\bm{\beta}$ is
!et
We can use the last expression when we define a so-called confidence interval for the parameters $\beta$.
We can use the last expression when we define a "so-called confidence interval":"https://en.wikipedia.org/wiki/Confidence_interval" for the parameters $\beta$.
A given parameter $\beta_j$ is given by the diagonal matrix element of the above matrix.
===== Exercise: Expectation values for Ridge regression =====
Show that
!bt
\[
\mathbb{E} \big[ \bm{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\
\bm{\beta}^{\mathrm{OLS}}.
\]
!et
We see clearly that
$\mathbb{E} \big[ \bm{\beta}^{\mathrm{Ridge}} \big] \not= \bm{\beta}^{\mathrm{OLS}}$ for any $\lambda > 0$.
Show also that the variance is
!bt
\[
\mbox{Var}[\bm{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \m\
athbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
\]
!et
and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\bm{\beta}$ goes to zero.