added exercises
This commit is contained in:
Morten Hjorth-Jensen
@@ -0,0 +1,268 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2a3463de",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
||||
"doconce format html exercisesweek37.do.txt -->\n",
|
||||
"<!-- dom:TITLE: Exercises week 37 -->"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "442e0844",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"# Exercises week 37\n",
|
||||
"**September 9-13, 2024**\n",
|
||||
"\n",
|
||||
"Date: **Deadline is Friday September 13 at midnight**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0c0df373",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Overarching aims of the exercises this week\n",
|
||||
"\n",
|
||||
"This exercise deals with various mean values and variances in linear\n",
|
||||
"regression method (here it may be useful to look up chapter 3,\n",
|
||||
"equation (3.8) of [Trevor Hastie, Robert Tibshirani, Jerome\n",
|
||||
"H. Friedman, The Elements of Statistical Learning,\n",
|
||||
"Springer](https://www.springer.com/gp/book/9780387848570)). The\n",
|
||||
"exercise is also a part of project 1 and can be reused in the theory\n",
|
||||
"part of the project.\n",
|
||||
"\n",
|
||||
"For more discussions on Ridge regression and calculation of\n",
|
||||
"expectation values, [Wessel van\n",
|
||||
"Wieringen's](https://arxiv.org/abs/1509.09169) article is highly\n",
|
||||
"recommended.\n",
|
||||
"\n",
|
||||
"The assumption we have made is that there exists a continuous function\n",
|
||||
"$f(\\boldsymbol{x})$ and a normal distributed error $\\boldsymbol{\\varepsilon}\\sim N(0,\n",
|
||||
"\\sigma^2)$ which describes our data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a1ac8666",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ee6ce4be",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"We then approximate this function $f(\\boldsymbol{x})$ with our model $\\boldsymbol{\\tilde{y}}$ from the solution of the linear regression equations (ordinary least squares OLS), that is our\n",
|
||||
"function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we minimized $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, with"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2d50b2e4",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3645cccd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"The matrix $\\boldsymbol{X}$ is the so-called design or feature matrix."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6a378434",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Exercise 1: Expectation values for ordinary least squares expressions\n",
|
||||
"\n",
|
||||
"Show that the expectation value of $\\boldsymbol{y}$ for a given element $i$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ce9b87a9",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\mathbb{E}(y_i) =\\sum_{j}x_{ij} \\beta_j=\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a6435e39",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"and that\n",
|
||||
"its variance is"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "75d89e64",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\mbox{Var}(y_i) = \\sigma^2.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b5aef4d3",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"Hence, $y_i \\sim N( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
|
||||
"mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$.\n",
|
||||
"\n",
|
||||
"With the OLS expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}}$ show that"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "17012dce",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\mathbb{E}(\\boldsymbol{\\hat{\\beta}}) = \\boldsymbol{\\beta}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4da3a821",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"Show finally that the variance of $\\boldsymbol{\\boldsymbol{\\beta}}$ is"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0ba77c7b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\mbox{Var}(\\boldsymbol{\\hat{\\beta}}) = \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5c66de05",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"We can use the last expression when we define a [so-called confidence interval](https://en.wikipedia.org/wiki/Confidence_interval) for the parameters $\\beta$. \n",
|
||||
"A given parameter $\\beta_j$ is given by the diagonal matrix element of the above matrix."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "43b92138",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Exercise 2: Expectation values for Ridge regression\n",
|
||||
"\n",
|
||||
"Show that"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "afea98ca",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "13d7fd96",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"We see clearly that\n",
|
||||
"$\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\mathbb{E} \\big[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}\\big ]$ for any $\\lambda > 0$.\n",
|
||||
"\n",
|
||||
"Show also that the variance is"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "77d1d47e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T}\\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2fa6bdfd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of the Ridge parameters $\\boldsymbol{\\beta}$ goes to zero."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
TITLE: Exercises week 37
|
||||
AUTHOR: September 11-15, 2023
|
||||
DATE: Deadline is Sunday September 17 at midnight
|
||||
AUTHOR: September 9-13, 2024
|
||||
DATE: Deadline is Friday September 13 at midnight
|
||||
|
||||
|
||||
===== Overarching aims of the exercises this week =====
|
||||
|
||||
Reference in New Issue
Block a user