where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +
where we have used that \( \mathbb{E} (\mathbf{y} \mathbf{y}^{T}) = \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the @@ -334,17 +334,17 @@ when we employ Ridge regression, allowing us again to define a confidence interv
It is rather straightforward to show that
$$ -\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. +\mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}. $$We see clearly that -\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). +\( \mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big] \not= \hat{\boldsymbol{\beta}}^{\mathrm{OLS}} \) for any \( \lambda > 0 \).
We can also compute the variance as
$$ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +\mbox{Var}[\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, $$and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero.
@@ -352,7 +352,7 @@ $$With this, we can compute the difference
$$ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +\mbox{Var}[\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}]-\mbox{Var}(\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. $$The difference is non-negative definite since each component of the
diff --git a/doc/pub/week37/html/week37-bs.html b/doc/pub/week37/html/week37-bs.html
index e3771e97a..8082be385 100644
--- a/doc/pub/week37/html/week37-bs.html
+++ b/doc/pub/week37/html/week37-bs.html
@@ -301,7 +301,7 @@ MathJax.Hub.Config({
-
where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +
where we have used that \( \mathbb{E} (\mathbf{y} \mathbf{y}^{T}) = \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the @@ -412,19 +412,19 @@ when we employ Ridge regression, allowing us again to define a confidence interv
It is rather straightforward to show that
$$
-\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}.
+\mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}.
$$
We see clearly that -\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). +\( \mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big] \not= \hat{\boldsymbol{\beta}}^{\mathrm{OLS}} \) for any \( \lambda > 0 \).
We can also compute the variance as
$$
-\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
+\mbox{Var}[\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
$$
@@ -434,7 +434,7 @@ $$
$$
-\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
+\mbox{Var}[\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}]-\mbox{Var}(\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
$$
diff --git a/doc/pub/week37/html/week37-solarized.html b/doc/pub/week37/html/week37-solarized.html
index 17d4d073c..8b6cc566f 100644
--- a/doc/pub/week37/html/week37-solarized.html
+++ b/doc/pub/week37/html/week37-solarized.html
@@ -250,7 +250,7 @@ MathJax.Hub.Config({
where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +
where we have used that \( \mathbb{E} (\mathbf{y} \mathbf{y}^{T}) = \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the @@ -441,17 +441,17 @@ when we employ Ridge regression, allowing us again to define a confidence interv
It is rather straightforward to show that
$$ -\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. +\mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}. $$We see clearly that -\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). +\( \mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big] \not= \hat{\boldsymbol{\beta}}^{\mathrm{OLS}} \) for any \( \lambda > 0 \).
We can also compute the variance as
$$ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +\mbox{Var}[\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, $$and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero.
@@ -459,7 +459,7 @@ $$With this, we can compute the difference
$$ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +\mbox{Var}[\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}]-\mbox{Var}(\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. $$The difference is non-negative definite since each component of the
diff --git a/doc/pub/week37/html/week37.html b/doc/pub/week37/html/week37.html
index 8176977e8..009570882 100644
--- a/doc/pub/week37/html/week37.html
+++ b/doc/pub/week37/html/week37.html
@@ -327,7 +327,7 @@ MathJax.Hub.Config({
where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +
where we have used that \( \mathbb{E} (\mathbf{y} \mathbf{y}^{T}) = \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the @@ -518,17 +518,17 @@ when we employ Ridge regression, allowing us again to define a confidence interv
It is rather straightforward to show that
$$ -\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. +\mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}. $$We see clearly that -\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). +\( \mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big] \not= \hat{\boldsymbol{\beta}}^{\mathrm{OLS}} \) for any \( \lambda > 0 \).
We can also compute the variance as
$$ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +\mbox{Var}[\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, $$and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero.
@@ -536,7 +536,7 @@ $$With this, we can compute the difference
$$ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +\mbox{Var}[\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}]-\mbox{Var}(\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. $$The difference is non-negative definite since each component of the diff --git a/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz b/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz index 6017621ae..12d719a23 100644 Binary files a/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz and b/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz differ diff --git a/doc/pub/week37/ipynb/week37.ipynb b/doc/pub/week37/ipynb/week37.ipynb index e35251c38..9abd1c82b 100644 --- a/doc/pub/week37/ipynb/week37.ipynb +++ b/doc/pub/week37/ipynb/week37.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "b01fd4b1", + "id": "12dab53c", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "44a45b87", + "id": "0670817d", "metadata": { "editable": true }, @@ -22,7 +22,7 @@ "# Week 37: Statitsitcal interpretations and Resampling Methods\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", "\n", - "Date: **Sep 11, 2023**\n", + "Date: **Sep 14, 2023**\n", "\n", "Copyright 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", "\n", @@ -31,7 +31,7 @@ }, { "cell_type": "markdown", - "id": "efc03cc3", + "id": "b4df73ee", "metadata": { "editable": true }, @@ -70,7 +70,7 @@ }, { "cell_type": "markdown", - "id": "585872a1", + "id": "59a06353", "metadata": { "editable": true }, @@ -80,7 +80,7 @@ }, { "cell_type": "markdown", - "id": "942f4ba0", + "id": "d41b2778", "metadata": { "editable": true }, @@ -109,7 +109,7 @@ }, { "cell_type": "markdown", - "id": "4bbfb97d", + "id": "6319816d", "metadata": { "editable": true }, @@ -125,7 +125,7 @@ }, { "cell_type": "markdown", - "id": "b7203b1b", + "id": "0caf248c", "metadata": { "editable": true }, @@ -144,7 +144,7 @@ }, { "cell_type": "markdown", - "id": "8a80f8fc", + "id": "781c3c58", "metadata": { "editable": true }, @@ -158,7 +158,7 @@ }, { "cell_type": "markdown", - "id": "64f383d8", + "id": "e08f89bf", "metadata": { "editable": true }, @@ -170,7 +170,7 @@ }, { "cell_type": "markdown", - "id": "a22c7dbf", + "id": "e8e15fc7", "metadata": { "editable": true }, @@ -181,7 +181,7 @@ }, { "cell_type": "markdown", - "id": "afa24d29", + "id": "f7894e62", "metadata": { "editable": true }, @@ -193,7 +193,7 @@ }, { "cell_type": "markdown", - "id": "adad5437", + "id": "9111bb1a", "metadata": { "editable": true }, @@ -205,7 +205,7 @@ }, { "cell_type": "markdown", - "id": "a0e308aa", + "id": "95330879", "metadata": { "editable": true }, @@ -221,7 +221,7 @@ }, { "cell_type": "markdown", - "id": "4f9d51a1", + "id": "4b22a2fc", "metadata": { "editable": true }, @@ -232,7 +232,7 @@ }, { "cell_type": "markdown", - "id": "ac11edda", + "id": "db76b4e3", "metadata": { "editable": true }, @@ -255,7 +255,7 @@ }, { "cell_type": "markdown", - "id": "2dee4bc8", + "id": "48c51a62", "metadata": { "editable": true }, @@ -266,7 +266,7 @@ }, { "cell_type": "markdown", - "id": "43158161", + "id": "ef404788", "metadata": { "editable": true }, @@ -278,7 +278,7 @@ }, { "cell_type": "markdown", - "id": "9cc312b1", + "id": "0876f61d", "metadata": { "editable": true }, @@ -290,7 +290,7 @@ }, { "cell_type": "markdown", - "id": "c9bf2c1b", + "id": "e9c171ad", "metadata": { "editable": true }, @@ -304,7 +304,7 @@ }, { "cell_type": "markdown", - "id": "fdac6a27", + "id": "0d826f15", "metadata": { "editable": true }, @@ -313,13 +313,13 @@ "\\begin{eqnarray*}\n", "\\mbox{Var}(\\boldsymbol{\\hat{\\beta}}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n", "\\\\\n", - "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}]^{T} \\}\n", + "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} - \\boldsymbol{\\beta}]^{T} \\}\n", "\\\\\n", - "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", "% \\\\\n", - "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{y} \\, \\mathbf{y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", "% \\\\\n", - "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{y} \\, \\mathbf{y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", "\\\\\n", "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", "% \\\\\n", @@ -335,12 +335,12 @@ }, { "cell_type": "markdown", - "id": "059613fd", + "id": "2861d9f6", "metadata": { "editable": true }, "source": [ - "where we have used that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n", + "where we have used that $\\mathbb{E} (\\mathbf{y} \\mathbf{y}^{T}) =\n", "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n", "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n", "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n", @@ -357,44 +357,44 @@ }, { "cell_type": "markdown", - "id": "fdea1889", + "id": "53868010", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n", + "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "546b5ca1", + "id": "a5219cae", "metadata": { "editable": true }, "source": [ "We see clearly that \n", - "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$.\n", + "$\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}$ for any $\\lambda > 0$.\n", "\n", "We can also compute the variance as" ] }, { "cell_type": "markdown", - "id": "d1fa67fa", + "id": "5b7b6b5b", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n", + "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n", "$$" ] }, { "cell_type": "markdown", - "id": "514550c7", + "id": "dd126071", "metadata": { "editable": true }, @@ -406,19 +406,19 @@ }, { "cell_type": "markdown", - "id": "b9a90eca", + "id": "f5a9b839", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n", + "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}]-\\mbox{Var}(\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "3f6f2edc", + "id": "ecf94cfc", "metadata": { "editable": true }, @@ -432,7 +432,7 @@ }, { "cell_type": "markdown", - "id": "f27fe183", + "id": "8b668a01", "metadata": { "editable": true }, @@ -442,7 +442,7 @@ }, { "cell_type": "markdown", - "id": "7889e185", + "id": "00c0dc54", "metadata": { "editable": true }, @@ -465,7 +465,7 @@ }, { "cell_type": "markdown", - "id": "229d9abb", + "id": "05999bdb", "metadata": { "editable": true }, @@ -477,7 +477,7 @@ }, { "cell_type": "markdown", - "id": "bdf5c751", + "id": "35db9ec7", "metadata": { "editable": true }, @@ -490,7 +490,7 @@ }, { "cell_type": "markdown", - "id": "7e91e0cb", + "id": "58bba4a7", "metadata": { "editable": true }, @@ -502,7 +502,7 @@ }, { "cell_type": "markdown", - "id": "41f2e22c", + "id": "909f5f70", "metadata": { "editable": true }, @@ -514,7 +514,7 @@ }, { "cell_type": "markdown", - "id": "0b31a9a1", + "id": "c7081bae", "metadata": { "editable": true }, @@ -526,7 +526,7 @@ }, { "cell_type": "markdown", - "id": "5f74f14c", + "id": "f278e6e2", "metadata": { "editable": true }, @@ -537,7 +537,7 @@ }, { "cell_type": "markdown", - "id": "c10936c8", + "id": "95911cfc", "metadata": { "editable": true }, @@ -549,7 +549,7 @@ }, { "cell_type": "markdown", - "id": "1c51efe9", + "id": "cda20aa1", "metadata": { "editable": true }, @@ -560,7 +560,7 @@ }, { "cell_type": "markdown", - "id": "a7c1a4a4", + "id": "bec5147a", "metadata": { "editable": true }, @@ -572,7 +572,7 @@ }, { "cell_type": "markdown", - "id": "43061633", + "id": "4d400b3d", "metadata": { "editable": true }, @@ -582,7 +582,7 @@ }, { "cell_type": "markdown", - "id": "69cada77", + "id": "b66a6af9", "metadata": { "editable": true }, @@ -613,7 +613,7 @@ }, { "cell_type": "markdown", - "id": "1898dbca", + "id": "bfef27b5", "metadata": { "editable": true }, @@ -625,7 +625,7 @@ }, { "cell_type": "markdown", - "id": "e7dccd6f", + "id": "59a7670d", "metadata": { "editable": true }, @@ -637,7 +637,7 @@ }, { "cell_type": "markdown", - "id": "a10fbd18", + "id": "41675ff4", "metadata": { "editable": true }, @@ -647,7 +647,7 @@ }, { "cell_type": "markdown", - "id": "3530ff5f", + "id": "712ba50c", "metadata": { "editable": true }, @@ -659,7 +659,7 @@ }, { "cell_type": "markdown", - "id": "f2fde118", + "id": "745edcde", "metadata": { "editable": true }, @@ -669,7 +669,7 @@ }, { "cell_type": "markdown", - "id": "bc004664", + "id": "14ad66d4", "metadata": { "editable": true }, @@ -681,7 +681,7 @@ }, { "cell_type": "markdown", - "id": "af134f1e", + "id": "ca7deab6", "metadata": { "editable": true }, @@ -691,7 +691,7 @@ }, { "cell_type": "markdown", - "id": "a4833625", + "id": "7908d737", "metadata": { "editable": true }, @@ -703,7 +703,7 @@ }, { "cell_type": "markdown", - "id": "8c6ce40d", + "id": "6c154f3c", "metadata": { "editable": true }, @@ -713,7 +713,7 @@ }, { "cell_type": "markdown", - "id": "428e7d77", + "id": "8e48abde", "metadata": { "editable": true }, @@ -733,7 +733,7 @@ }, { "cell_type": "markdown", - "id": "e9884bd0", + "id": "a6e033f8", "metadata": { "editable": true }, @@ -745,7 +745,7 @@ }, { "cell_type": "markdown", - "id": "167cca6b", + "id": "3993ab91", "metadata": { "editable": true }, @@ -755,7 +755,7 @@ }, { "cell_type": "markdown", - "id": "7954e72f", + "id": "9ee73fdc", "metadata": { "editable": true }, @@ -767,7 +767,7 @@ }, { "cell_type": "markdown", - "id": "e5e041c9", + "id": "5084730a", "metadata": { "editable": true }, @@ -779,7 +779,7 @@ }, { "cell_type": "markdown", - "id": "fe7cdfb3", + "id": "390036b5", "metadata": { "editable": true }, @@ -791,7 +791,7 @@ }, { "cell_type": "markdown", - "id": "2073c56b", + "id": "457dbbf1", "metadata": { "editable": true }, @@ -803,7 +803,7 @@ }, { "cell_type": "markdown", - "id": "4e077392", + "id": "c6cf780a", "metadata": { "editable": true }, @@ -815,7 +815,7 @@ }, { "cell_type": "markdown", - "id": "53621deb", + "id": "9f0696be", "metadata": { "editable": true }, @@ -827,7 +827,7 @@ }, { "cell_type": "markdown", - "id": "3a77431d", + "id": "51313828", "metadata": { "editable": true }, @@ -839,7 +839,7 @@ }, { "cell_type": "markdown", - "id": "786d7efe", + "id": "32ac49dc", "metadata": { "editable": true }, @@ -851,7 +851,7 @@ }, { "cell_type": "markdown", - "id": "8f01293e", + "id": "09dc987a", "metadata": { "editable": true }, @@ -861,7 +861,7 @@ }, { "cell_type": "markdown", - "id": "6cb289dc", + "id": "1da1b18d", "metadata": { "editable": true }, @@ -873,7 +873,7 @@ }, { "cell_type": "markdown", - "id": "31ef225f", + "id": "7bf2ecfc", "metadata": { "editable": true }, @@ -883,7 +883,7 @@ }, { "cell_type": "markdown", - "id": "1d82a508", + "id": "96bfe404", "metadata": { "editable": true }, @@ -902,7 +902,7 @@ }, { "cell_type": "markdown", - "id": "c93ecb90", + "id": "bd1b0956", "metadata": { "editable": true }, @@ -923,7 +923,7 @@ }, { "cell_type": "markdown", - "id": "c2b4a441", + "id": "7605a751", "metadata": { "editable": true }, @@ -935,7 +935,7 @@ }, { "cell_type": "markdown", - "id": "1f7159c9", + "id": "35f6a544", "metadata": { "editable": true }, @@ -946,7 +946,7 @@ }, { "cell_type": "markdown", - "id": "d092f790", + "id": "392e58c4", "metadata": { "editable": true }, @@ -959,7 +959,7 @@ }, { "cell_type": "markdown", - "id": "cebb15fb", + "id": "2cfc08e1", "metadata": { "editable": true }, @@ -971,7 +971,7 @@ }, { "cell_type": "markdown", - "id": "67f051bc", + "id": "6cbc1dd1", "metadata": { "editable": true }, @@ -981,7 +981,7 @@ }, { "cell_type": "markdown", - "id": "3ecb215e", + "id": "b34300f9", "metadata": { "editable": true }, @@ -993,7 +993,7 @@ }, { "cell_type": "markdown", - "id": "3249a7c9", + "id": "4937b53e", "metadata": { "editable": true }, @@ -1003,7 +1003,7 @@ }, { "cell_type": "markdown", - "id": "c7616dec", + "id": "615e59b0", "metadata": { "editable": true }, @@ -1015,7 +1015,7 @@ }, { "cell_type": "markdown", - "id": "7f137acf", + "id": "f849da33", "metadata": { "editable": true }, @@ -1025,7 +1025,7 @@ }, { "cell_type": "markdown", - "id": "f14cb9a2", + "id": "38bd1726", "metadata": { "editable": true }, @@ -1039,7 +1039,7 @@ }, { "cell_type": "markdown", - "id": "4a56d5c4", + "id": "6828a929", "metadata": { "editable": true }, @@ -1051,7 +1051,7 @@ }, { "cell_type": "markdown", - "id": "fcc7f560", + "id": "1191cc23", "metadata": { "editable": true }, @@ -1061,7 +1061,7 @@ }, { "cell_type": "markdown", - "id": "81211139", + "id": "4cc086fa", "metadata": { "editable": true }, @@ -1073,7 +1073,7 @@ }, { "cell_type": "markdown", - "id": "715740f0", + "id": "202ca746", "metadata": { "editable": true }, @@ -1083,7 +1083,7 @@ }, { "cell_type": "markdown", - "id": "c103f415", + "id": "287784f7", "metadata": { "editable": true }, @@ -1095,7 +1095,7 @@ }, { "cell_type": "markdown", - "id": "2bb60344", + "id": "3e69b13e", "metadata": { "editable": true }, @@ -1105,7 +1105,7 @@ }, { "cell_type": "markdown", - "id": "01d520aa", + "id": "21b4ae5e", "metadata": { "editable": true }, @@ -1117,7 +1117,7 @@ }, { "cell_type": "markdown", - "id": "0f62c28f", + "id": "84e4f929", "metadata": { "editable": true }, @@ -1127,7 +1127,7 @@ }, { "cell_type": "markdown", - "id": "a3241d36", + "id": "650ac3bf", "metadata": { "editable": true }, @@ -1143,7 +1143,7 @@ }, { "cell_type": "markdown", - "id": "5d92c7bc", + "id": "98540832", "metadata": { "editable": true }, @@ -1155,7 +1155,7 @@ }, { "cell_type": "markdown", - "id": "3c503db1", + "id": "5e017861", "metadata": { "editable": true }, @@ -1165,7 +1165,7 @@ }, { "cell_type": "markdown", - "id": "117cbb16", + "id": "3d771731", "metadata": { "editable": true }, @@ -1177,7 +1177,7 @@ }, { "cell_type": "markdown", - "id": "a82a6f35", + "id": "49f88734", "metadata": { "editable": true }, @@ -1190,7 +1190,7 @@ }, { "cell_type": "markdown", - "id": "9930e40e", + "id": "208b7578", "metadata": { "editable": true }, @@ -1202,7 +1202,7 @@ }, { "cell_type": "markdown", - "id": "5435e961", + "id": "0b4d2a7f", "metadata": { "editable": true }, @@ -1212,7 +1212,7 @@ }, { "cell_type": "markdown", - "id": "6f3264e4", + "id": "27de20ee", "metadata": { "editable": true }, @@ -1224,7 +1224,7 @@ }, { "cell_type": "markdown", - "id": "e6fa0b58", + "id": "cdc50e93", "metadata": { "editable": true }, @@ -1234,7 +1234,7 @@ }, { "cell_type": "markdown", - "id": "886b042b", + "id": "d4b00728", "metadata": { "editable": true }, @@ -1246,7 +1246,7 @@ }, { "cell_type": "markdown", - "id": "64d75f6f", + "id": "e5dc2607", "metadata": { "editable": true }, @@ -1258,7 +1258,7 @@ }, { "cell_type": "markdown", - "id": "fcf0f05f", + "id": "efa6a842", "metadata": { "editable": true }, @@ -1268,7 +1268,7 @@ }, { "cell_type": "markdown", - "id": "c689d23e", + "id": "fb6c3f55", "metadata": { "editable": true }, @@ -1280,7 +1280,7 @@ }, { "cell_type": "markdown", - "id": "1b30d4ea", + "id": "e67d3a3b", "metadata": { "editable": true }, @@ -1292,7 +1292,7 @@ }, { "cell_type": "markdown", - "id": "395b59bc", + "id": "f030506f", "metadata": { "editable": true }, @@ -1304,7 +1304,7 @@ }, { "cell_type": "markdown", - "id": "fe60f31d", + "id": "f5f5d5ae", "metadata": { "editable": true }, @@ -1314,7 +1314,7 @@ }, { "cell_type": "markdown", - "id": "c62bb449", + "id": "7bf1d85e", "metadata": { "editable": true }, @@ -1326,7 +1326,7 @@ }, { "cell_type": "markdown", - "id": "a0acba25", + "id": "cd9c1ab7", "metadata": { "editable": true }, @@ -1336,7 +1336,7 @@ }, { "cell_type": "markdown", - "id": "3dd24af3", + "id": "d7d88b8a", "metadata": { "editable": true }, @@ -1355,7 +1355,7 @@ }, { "cell_type": "markdown", - "id": "6917a474", + "id": "ef7d75d7", "metadata": { "editable": true }, @@ -1383,7 +1383,7 @@ }, { "cell_type": "markdown", - "id": "24cec8c3", + "id": "0c17d91f", "metadata": { "editable": true }, @@ -1409,7 +1409,7 @@ }, { "cell_type": "markdown", - "id": "89ad7a09", + "id": "9623a072", "metadata": { "editable": true }, @@ -1426,7 +1426,7 @@ }, { "cell_type": "markdown", - "id": "2902167c", + "id": "04055f3a", "metadata": { "editable": true }, @@ -1446,7 +1446,7 @@ }, { "cell_type": "markdown", - "id": "082db7cc", + "id": "e84eab2f", "metadata": { "editable": true }, @@ -1475,7 +1475,7 @@ }, { "cell_type": "markdown", - "id": "9c0f97ec", + "id": "c4c9b84d", "metadata": { "editable": true }, @@ -1500,7 +1500,7 @@ }, { "cell_type": "markdown", - "id": "97db2ddf", + "id": "3a785d24", "metadata": { "editable": true }, @@ -1520,7 +1520,7 @@ }, { "cell_type": "markdown", - "id": "fa694ca7", + "id": "fe3a37a8", "metadata": { "editable": true }, @@ -1532,7 +1532,7 @@ }, { "cell_type": "markdown", - "id": "e70aa901", + "id": "34916bca", "metadata": { "editable": true }, @@ -1542,7 +1542,7 @@ }, { "cell_type": "markdown", - "id": "4603da33", + "id": "cdb4f587", "metadata": { "editable": true }, @@ -1557,7 +1557,7 @@ }, { "cell_type": "markdown", - "id": "c6228bb8", + "id": "9ecb84b1", "metadata": { "editable": true }, @@ -1570,7 +1570,7 @@ }, { "cell_type": "markdown", - "id": "52dee4b9", + "id": "e759937c", "metadata": { "editable": true }, @@ -1583,7 +1583,7 @@ }, { "cell_type": "markdown", - "id": "8a1618e6", + "id": "243be6e6", "metadata": { "editable": true }, @@ -1595,7 +1595,7 @@ }, { "cell_type": "markdown", - "id": "c78c1c3c", + "id": "06f91a6d", "metadata": { "editable": true }, @@ -1608,7 +1608,7 @@ }, { "cell_type": "markdown", - "id": "4ec445f3", + "id": "ecb40872", "metadata": { "editable": true }, @@ -1619,7 +1619,7 @@ }, { "cell_type": "markdown", - "id": "eaaeee09", + "id": "67baaeef", "metadata": { "editable": true }, @@ -1633,7 +1633,7 @@ }, { "cell_type": "markdown", - "id": "263f8dae", + "id": "a2c5f386", "metadata": { "editable": true }, @@ -1643,7 +1643,7 @@ }, { "cell_type": "markdown", - "id": "8d107200", + "id": "5108823f", "metadata": { "editable": true }, @@ -1657,7 +1657,7 @@ }, { "cell_type": "markdown", - "id": "536dfe52", + "id": "55ab37b2", "metadata": { "editable": true }, @@ -1670,7 +1670,7 @@ }, { "cell_type": "markdown", - "id": "8cfaa620", + "id": "b1ebe454", "metadata": { "editable": true }, @@ -1683,7 +1683,7 @@ }, { "cell_type": "markdown", - "id": "61d1ad72", + "id": "36ea5599", "metadata": { "editable": true }, @@ -1693,7 +1693,7 @@ }, { "cell_type": "markdown", - "id": "3fe1619f", + "id": "b8892337", "metadata": { "editable": true }, @@ -1706,7 +1706,7 @@ }, { "cell_type": "markdown", - "id": "2dce0fc0", + "id": "fb17565b", "metadata": { "editable": true }, @@ -1716,7 +1716,7 @@ }, { "cell_type": "markdown", - "id": "02476632", + "id": "e27ecfbf", "metadata": { "editable": true }, @@ -1729,7 +1729,7 @@ }, { "cell_type": "markdown", - "id": "6a81d0b6", + "id": "2e89e296", "metadata": { "editable": true }, @@ -1741,7 +1741,7 @@ }, { "cell_type": "markdown", - "id": "4b6b30ea", + "id": "a81f608d", "metadata": { "editable": true }, @@ -1760,7 +1760,7 @@ }, { "cell_type": "markdown", - "id": "7cb03fff", + "id": "b6682620", "metadata": { "editable": true }, @@ -1773,7 +1773,7 @@ }, { "cell_type": "markdown", - "id": "84cb3cf4", + "id": "fb8c9440", "metadata": { "editable": true }, @@ -1785,7 +1785,7 @@ }, { "cell_type": "markdown", - "id": "87d68e2d", + "id": "161a9708", "metadata": { "editable": true }, @@ -1798,7 +1798,7 @@ }, { "cell_type": "markdown", - "id": "9028b69e", + "id": "795def7b", "metadata": { "editable": true }, @@ -1818,7 +1818,7 @@ }, { "cell_type": "markdown", - "id": "81cb25ef", + "id": "99defe62", "metadata": { "editable": true }, @@ -1841,7 +1841,7 @@ }, { "cell_type": "markdown", - "id": "8cb18602", + "id": "fea5787a", "metadata": { "editable": true }, @@ -1856,7 +1856,7 @@ }, { "cell_type": "markdown", - "id": "f4388f16", + "id": "1203f4c3", "metadata": { "editable": true }, @@ -1868,7 +1868,7 @@ }, { "cell_type": "markdown", - "id": "44860632", + "id": "6f7359fc", "metadata": { "editable": true }, @@ -1888,7 +1888,7 @@ }, { "cell_type": "markdown", - "id": "7f84d8e5", + "id": "cf8118b5", "metadata": { "editable": true }, @@ -1908,7 +1908,7 @@ }, { "cell_type": "markdown", - "id": "591d4185", + "id": "77e739ba", "metadata": { "editable": true }, @@ -1932,7 +1932,7 @@ }, { "cell_type": "markdown", - "id": "551b6e48", + "id": "1d04dea4", "metadata": { "editable": true }, @@ -1953,7 +1953,7 @@ }, { "cell_type": "markdown", - "id": "4b953459", + "id": "df7c2ef8", "metadata": { "editable": true }, @@ -1983,7 +1983,7 @@ }, { "cell_type": "markdown", - "id": "52a48932", + "id": "bedd300d", "metadata": { "editable": true }, @@ -2007,7 +2007,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "182e51b5", + "id": "719166c6", "metadata": { "collapsed": false, "editable": true @@ -2046,7 +2046,7 @@ }, { "cell_type": "markdown", - "id": "967a6335", + "id": "08322061", "metadata": { "editable": true }, @@ -2056,7 +2056,7 @@ }, { "cell_type": "markdown", - "id": "16cadf05", + "id": "0ac23bea", "metadata": { "editable": true }, @@ -2067,7 +2067,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "5825b06f", + "id": "9a536843", "metadata": { "collapsed": false, "editable": true @@ -2087,7 +2087,7 @@ }, { "cell_type": "markdown", - "id": "ed1cfe14", + "id": "788570e6", "metadata": { "editable": true }, @@ -2105,7 +2105,7 @@ }, { "cell_type": "markdown", - "id": "969d4ddf", + "id": "5da69bc5", "metadata": { "editable": true }, @@ -2117,7 +2117,7 @@ }, { "cell_type": "markdown", - "id": "b5e6dc10", + "id": "a86c6849", "metadata": { "editable": true }, @@ -2134,7 +2134,7 @@ }, { "cell_type": "markdown", - "id": "8188dc0c", + "id": "4392eb58", "metadata": { "editable": true }, @@ -2146,7 +2146,7 @@ }, { "cell_type": "markdown", - "id": "9acfaf8c", + "id": "2b901d8b", "metadata": { "editable": true }, @@ -2156,7 +2156,7 @@ }, { "cell_type": "markdown", - "id": "a05ac35e", + "id": "1cc5c8f7", "metadata": { "editable": true }, @@ -2168,7 +2168,7 @@ }, { "cell_type": "markdown", - "id": "4e2e3da4", + "id": "bf64d4f2", "metadata": { "editable": true }, @@ -2185,7 +2185,7 @@ }, { "cell_type": "markdown", - "id": "2dec9a7c", + "id": "5de3b3d6", "metadata": { "editable": true }, @@ -2197,7 +2197,7 @@ }, { "cell_type": "markdown", - "id": "a4b2b846", + "id": "e0a4db21", "metadata": { "editable": true }, @@ -2207,7 +2207,7 @@ }, { "cell_type": "markdown", - "id": "a191e0ac", + "id": "97c94610", "metadata": { "editable": true }, @@ -2219,7 +2219,7 @@ }, { "cell_type": "markdown", - "id": "1ec0291e", + "id": "a79d371d", "metadata": { "editable": true }, @@ -2229,7 +2229,7 @@ }, { "cell_type": "markdown", - "id": "c49ce956", + "id": "51234ad4", "metadata": { "editable": true }, @@ -2241,7 +2241,7 @@ }, { "cell_type": "markdown", - "id": "9bc7eda1", + "id": "f5162757", "metadata": { "editable": true }, @@ -2251,7 +2251,7 @@ }, { "cell_type": "markdown", - "id": "f42f620a", + "id": "806045bd", "metadata": { "editable": true }, @@ -2267,7 +2267,7 @@ }, { "cell_type": "markdown", - "id": "5121f450", + "id": "4feeb1a7", "metadata": { "editable": true }, @@ -2278,7 +2278,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "19b88b91", + "id": "df58376f", "metadata": { "collapsed": false, "editable": true @@ -2343,7 +2343,7 @@ }, { "cell_type": "markdown", - "id": "b8c8f9cc", + "id": "76e6ffd8", "metadata": { "editable": true }, @@ -2354,7 +2354,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "6634be8b", + "id": "0af9bb8f", "metadata": { "collapsed": false, "editable": true @@ -2411,7 +2411,7 @@ }, { "cell_type": "markdown", - "id": "cf252a96", + "id": "5459051b", "metadata": { "editable": true }, @@ -2449,7 +2449,7 @@ }, { "cell_type": "markdown", - "id": "4ea91702", + "id": "15b206e7", "metadata": { "editable": true }, @@ -2460,7 +2460,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "2123b1e9", + "id": "d87d2f44", "metadata": { "collapsed": false, "editable": true @@ -2542,7 +2542,7 @@ }, { "cell_type": "markdown", - "id": "cd47e7db", + "id": "7c4b191b", "metadata": { "editable": true }, @@ -2567,7 +2567,7 @@ }, { "cell_type": "markdown", - "id": "615a397e", + "id": "c2d9a79d", "metadata": { "editable": true }, @@ -2595,7 +2595,7 @@ }, { "cell_type": "markdown", - "id": "edd51643", + "id": "f8a06bc0", "metadata": { "editable": true }, @@ -2608,7 +2608,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "efb30d52", + "id": "33eeadec", "metadata": { "collapsed": false, "editable": true @@ -2708,7 +2708,7 @@ }, { "cell_type": "markdown", - "id": "5f2daae6", + "id": "c026d24c", "metadata": { "editable": true }, @@ -2719,7 +2719,7 @@ { "cell_type": "code", "execution_count": 7, - "id": "d6440291", + "id": "22f951e6", "metadata": { "collapsed": false, "editable": true @@ -2808,7 +2808,7 @@ }, { "cell_type": "markdown", - "id": "b5102ece", + "id": "d4dc2f34", "metadata": { "editable": true }, @@ -2818,7 +2818,7 @@ }, { "cell_type": "markdown", - "id": "149f630f", + "id": "ff2ba894", "metadata": { "editable": true }, @@ -2831,7 +2831,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "1c26a64e", + "id": "c5618fb6", "metadata": { "collapsed": false, "editable": true diff --git a/doc/src/week37/week37.do.txt b/doc/src/week37/week37.do.txt index d4ef0d719..1e2a718e6 100644 --- a/doc/src/week37/week37.do.txt +++ b/doc/src/week37/week37.do.txt @@ -140,13 +140,13 @@ The variance of the optimal value $\bm{\hat{\beta}}$ is \begin{eqnarray*} \mbox{Var}(\bm{\hat{\beta}}) & = & \mathbb{E} \{ [\bm{\beta} - \mathbb{E}(\bm{\beta})] [\bm{\beta} - \mathbb{E}(\bm{\beta})]^{T} \} \\ -& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\beta}]^{T} \} +& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y} - \bm{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y} - \bm{\beta}]^{T} \} \\ -% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \bm{\beta} \, \bm{\beta}^{T} +% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y}]^{T} \} - \bm{\beta} \, \bm{\beta}^{T} % \\ -% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \bm{\beta} \, \bm{\beta}^{T} +% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{y} \, \mathbf{y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \bm{\beta} \, \bm{\beta}^{T} % \\ -& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T} +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{y} \, \mathbf{y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T} \\ & = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \bm{\beta} \, \bm{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T} % \\ @@ -159,7 +159,7 @@ The variance of the optimal value $\bm{\hat{\beta}}$ is \end{eqnarray*} !et -where we have used that $\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +where we have used that $\mathbb{E} (\mathbf{y} \mathbf{y}^{T}) = \mathbf{X} \, \bm{\beta} \, \bm{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \, \mathbf{I}_{nn}$. From $\mbox{Var}(\bm{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}$, one obtains an estimate of the @@ -175,17 +175,17 @@ when we employ Ridge regression, allowing us again to define a confidence interv It is rather straightforward to show that !bt \[ -\mathbb{E} \big[ \bm{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\bm{\beta}^{\mathrm{OLS}}. +\mathbb{E} \big[ \hat{\bm{\beta}}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\bm{\beta}. \] !et We see clearly that -$\mathbb{E} \big[ \bm{\beta}^{\mathrm{Ridge}} \big] \not= \bm{\beta}^{\mathrm{OLS}}$ for any $\lambda > 0$. +$\mathbb{E} \big[ \hat{\bm{\beta}}^{\mathrm{Ridge}} \big] \not= \hat{\bm{\beta}}^{\mathrm{OLS}}$ for any $\lambda > 0$. We can also compute the variance as !bt \[ -\mbox{Var}[\bm{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +\mbox{Var}[\hat{\bm{\beta}}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, \] !et and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\bm{\beta}$ goes to zero. @@ -194,7 +194,7 @@ With this, we can compute the difference !bt \[ -\mbox{Var}[\bm{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\bm{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +\mbox{Var}[\hat{\bm{\beta}}^{\mathrm{OLS}}]-\mbox{Var}(\hat{\bm{\beta}}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. \] !et The difference is non-negative definite since each component of the