diff --git a/doc/LectureNotes/_build/.doctrees/environment.pickle b/doc/LectureNotes/_build/.doctrees/environment.pickle index a59ff7c2d..9227fccdc 100644 Binary files a/doc/LectureNotes/_build/.doctrees/environment.pickle and b/doc/LectureNotes/_build/.doctrees/environment.pickle differ diff --git a/doc/LectureNotes/_build/.doctrees/exercisesweek37.doctree b/doc/LectureNotes/_build/.doctrees/exercisesweek37.doctree index 62f86e1bc..ed4f716e4 100644 Binary files a/doc/LectureNotes/_build/.doctrees/exercisesweek37.doctree and b/doc/LectureNotes/_build/.doctrees/exercisesweek37.doctree differ diff --git a/doc/LectureNotes/_build/html/_sources/exercisesweek37.ipynb b/doc/LectureNotes/_build/html/_sources/exercisesweek37.ipynb index 39e531a60..2753e48bf 100644 --- a/doc/LectureNotes/_build/html/_sources/exercisesweek37.ipynb +++ b/doc/LectureNotes/_build/html/_sources/exercisesweek37.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "1ff4a4e9", + "id": "8b21a389", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "0569f7a9", + "id": "10a3b32a", "metadata": { "editable": true }, @@ -27,7 +27,7 @@ }, { "cell_type": "markdown", - "id": "6f34e507", + "id": "940daa26", "metadata": { "editable": true }, @@ -54,7 +54,7 @@ }, { "cell_type": "markdown", - "id": "3ff574a4", + "id": "d51b0da4", "metadata": { "editable": true }, @@ -66,7 +66,7 @@ }, { "cell_type": "markdown", - "id": "096a3165", + "id": "40a14559", "metadata": { "editable": true }, @@ -77,7 +77,7 @@ }, { "cell_type": "markdown", - "id": "cc9fb258", + "id": "e4afb2ea", "metadata": { "editable": true }, @@ -89,7 +89,7 @@ }, { "cell_type": "markdown", - "id": "daf4eb5d", + "id": "b0153681", "metadata": { "editable": true }, @@ -99,7 +99,7 @@ }, { "cell_type": "markdown", - "id": "3b491679", + "id": "17b98331", "metadata": { "editable": true }, @@ -111,7 +111,7 @@ }, { "cell_type": "markdown", - "id": "93a49739", + "id": "0e35faa1", "metadata": { "editable": true }, @@ -123,7 +123,7 @@ }, { "cell_type": "markdown", - "id": "a3fa80db", + "id": "c1c7837e", "metadata": { "editable": true }, @@ -134,7 +134,7 @@ }, { "cell_type": "markdown", - "id": "e6e2defd", + "id": "0ab57c17", "metadata": { "editable": true }, @@ -146,7 +146,7 @@ }, { "cell_type": "markdown", - "id": "9aa19afa", + "id": "345dbeca", "metadata": { "editable": true }, @@ -159,7 +159,7 @@ }, { "cell_type": "markdown", - "id": "5fa82a47", + "id": "fea2ddcf", "metadata": { "editable": true }, @@ -171,7 +171,7 @@ }, { "cell_type": "markdown", - "id": "33099abb", + "id": "683a72cf", "metadata": { "editable": true }, @@ -181,7 +181,7 @@ }, { "cell_type": "markdown", - "id": "5d03f001", + "id": "0dfd010a", "metadata": { "editable": true }, @@ -193,7 +193,7 @@ }, { "cell_type": "markdown", - "id": "94bd151d", + "id": "d4e4aa80", "metadata": { "editable": true }, @@ -204,7 +204,7 @@ }, { "cell_type": "markdown", - "id": "f0b84941", + "id": "acb128a7", "metadata": { "editable": true }, @@ -216,33 +216,32 @@ }, { "cell_type": "markdown", - "id": "60370a63", + "id": "83af53d9", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\\n", - "\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n", + "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "af251217", + "id": "1bbd0f50", "metadata": { "editable": true }, "source": [ "We see clearly that\n", - "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$.\n", + "$\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\mathbb{E} \\big[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}\\big ]$ for any $\\lambda > 0$.\n", "\n", "Show also that the variance is" ] }, { "cell_type": "markdown", - "id": "02e978da", + "id": "617deac1", "metadata": { "editable": true }, @@ -254,12 +253,12 @@ }, { "cell_type": "markdown", - "id": "3656a896", + "id": "bee378b0", "metadata": { "editable": true }, "source": [ - "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero." + "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of the Ridge parameters $\\boldsymbol{\\beta}$ goes to zero." ] } ], diff --git a/doc/LectureNotes/_build/html/exercisesweek37.html b/doc/LectureNotes/_build/html/exercisesweek37.html index de203096b..c7ce80d5e 100644 --- a/doc/LectureNotes/_build/html/exercisesweek37.html +++ b/doc/LectureNotes/_build/html/exercisesweek37.html @@ -497,17 +497,16 @@ A given parameter \(\beta_j\)

Show that

\[ -\mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\ -\boldsymbol{\beta}^{\mathrm{OLS}}. +\mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}. \]

We see clearly that -\(\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}}\) for any \(\lambda > 0\).

+\(\mathbb{E} \big[ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} \big] \not= \mathbb{E} \big[\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}\big ]\) for any \(\lambda > 0\).

Show also that the variance is

\[ \mbox{Var}[\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T}\mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, \]
-

and it is easy to see that if the parameter \(\lambda\) goes to infinity then the variance of Ridge parameters \(\boldsymbol{\beta}\) goes to zero.

+

and it is easy to see that if the parameter \(\lambda\) goes to infinity then the variance of the Ridge parameters \(\boldsymbol{\beta}\) goes to zero.

diff --git a/doc/LectureNotes/_build/jupyter_execute/exercisesweek37.ipynb b/doc/LectureNotes/_build/jupyter_execute/exercisesweek37.ipynb index 2efe96e13..070fb77d6 100644 --- a/doc/LectureNotes/_build/jupyter_execute/exercisesweek37.ipynb +++ b/doc/LectureNotes/_build/jupyter_execute/exercisesweek37.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "1ff4a4e9", + "id": "8b21a389", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "0569f7a9", + "id": "10a3b32a", "metadata": { "editable": true }, @@ -27,7 +27,7 @@ }, { "cell_type": "markdown", - "id": "6f34e507", + "id": "940daa26", "metadata": { "editable": true }, @@ -54,7 +54,7 @@ }, { "cell_type": "markdown", - "id": "3ff574a4", + "id": "d51b0da4", "metadata": { "editable": true }, @@ -66,7 +66,7 @@ }, { "cell_type": "markdown", - "id": "096a3165", + "id": "40a14559", "metadata": { "editable": true }, @@ -77,7 +77,7 @@ }, { "cell_type": "markdown", - "id": "cc9fb258", + "id": "e4afb2ea", "metadata": { "editable": true }, @@ -89,7 +89,7 @@ }, { "cell_type": "markdown", - "id": "daf4eb5d", + "id": "b0153681", "metadata": { "editable": true }, @@ -99,7 +99,7 @@ }, { "cell_type": "markdown", - "id": "3b491679", + "id": "17b98331", "metadata": { "editable": true }, @@ -111,7 +111,7 @@ }, { "cell_type": "markdown", - "id": "93a49739", + "id": "0e35faa1", "metadata": { "editable": true }, @@ -123,7 +123,7 @@ }, { "cell_type": "markdown", - "id": "a3fa80db", + "id": "c1c7837e", "metadata": { "editable": true }, @@ -134,7 +134,7 @@ }, { "cell_type": "markdown", - "id": "e6e2defd", + "id": "0ab57c17", "metadata": { "editable": true }, @@ -146,7 +146,7 @@ }, { "cell_type": "markdown", - "id": "9aa19afa", + "id": "345dbeca", "metadata": { "editable": true }, @@ -159,7 +159,7 @@ }, { "cell_type": "markdown", - "id": "5fa82a47", + "id": "fea2ddcf", "metadata": { "editable": true }, @@ -171,7 +171,7 @@ }, { "cell_type": "markdown", - "id": "33099abb", + "id": "683a72cf", "metadata": { "editable": true }, @@ -181,7 +181,7 @@ }, { "cell_type": "markdown", - "id": "5d03f001", + "id": "0dfd010a", "metadata": { "editable": true }, @@ -193,7 +193,7 @@ }, { "cell_type": "markdown", - "id": "94bd151d", + "id": "d4e4aa80", "metadata": { "editable": true }, @@ -204,7 +204,7 @@ }, { "cell_type": "markdown", - "id": "f0b84941", + "id": "acb128a7", "metadata": { "editable": true }, @@ -216,33 +216,32 @@ }, { "cell_type": "markdown", - "id": "60370a63", + "id": "83af53d9", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\\n", - "\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n", + "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "af251217", + "id": "1bbd0f50", "metadata": { "editable": true }, "source": [ "We see clearly that\n", - "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$.\n", + "$\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\mathbb{E} \\big[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}\\big ]$ for any $\\lambda > 0$.\n", "\n", "Show also that the variance is" ] }, { "cell_type": "markdown", - "id": "02e978da", + "id": "617deac1", "metadata": { "editable": true }, @@ -254,12 +253,12 @@ }, { "cell_type": "markdown", - "id": "3656a896", + "id": "bee378b0", "metadata": { "editable": true }, "source": [ - "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero." + "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of the Ridge parameters $\\boldsymbol{\\beta}$ goes to zero." ] } ], diff --git a/doc/LectureNotes/exercisesweek37.ipynb b/doc/LectureNotes/exercisesweek37.ipynb index 39e531a60..2753e48bf 100644 --- a/doc/LectureNotes/exercisesweek37.ipynb +++ b/doc/LectureNotes/exercisesweek37.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "1ff4a4e9", + "id": "8b21a389", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "0569f7a9", + "id": "10a3b32a", "metadata": { "editable": true }, @@ -27,7 +27,7 @@ }, { "cell_type": "markdown", - "id": "6f34e507", + "id": "940daa26", "metadata": { "editable": true }, @@ -54,7 +54,7 @@ }, { "cell_type": "markdown", - "id": "3ff574a4", + "id": "d51b0da4", "metadata": { "editable": true }, @@ -66,7 +66,7 @@ }, { "cell_type": "markdown", - "id": "096a3165", + "id": "40a14559", "metadata": { "editable": true }, @@ -77,7 +77,7 @@ }, { "cell_type": "markdown", - "id": "cc9fb258", + "id": "e4afb2ea", "metadata": { "editable": true }, @@ -89,7 +89,7 @@ }, { "cell_type": "markdown", - "id": "daf4eb5d", + "id": "b0153681", "metadata": { "editable": true }, @@ -99,7 +99,7 @@ }, { "cell_type": "markdown", - "id": "3b491679", + "id": "17b98331", "metadata": { "editable": true }, @@ -111,7 +111,7 @@ }, { "cell_type": "markdown", - "id": "93a49739", + "id": "0e35faa1", "metadata": { "editable": true }, @@ -123,7 +123,7 @@ }, { "cell_type": "markdown", - "id": "a3fa80db", + "id": "c1c7837e", "metadata": { "editable": true }, @@ -134,7 +134,7 @@ }, { "cell_type": "markdown", - "id": "e6e2defd", + "id": "0ab57c17", "metadata": { "editable": true }, @@ -146,7 +146,7 @@ }, { "cell_type": "markdown", - "id": "9aa19afa", + "id": "345dbeca", "metadata": { "editable": true }, @@ -159,7 +159,7 @@ }, { "cell_type": "markdown", - "id": "5fa82a47", + "id": "fea2ddcf", "metadata": { "editable": true }, @@ -171,7 +171,7 @@ }, { "cell_type": "markdown", - "id": "33099abb", + "id": "683a72cf", "metadata": { "editable": true }, @@ -181,7 +181,7 @@ }, { "cell_type": "markdown", - "id": "5d03f001", + "id": "0dfd010a", "metadata": { "editable": true }, @@ -193,7 +193,7 @@ }, { "cell_type": "markdown", - "id": "94bd151d", + "id": "d4e4aa80", "metadata": { "editable": true }, @@ -204,7 +204,7 @@ }, { "cell_type": "markdown", - "id": "f0b84941", + "id": "acb128a7", "metadata": { "editable": true }, @@ -216,33 +216,32 @@ }, { "cell_type": "markdown", - "id": "60370a63", + "id": "83af53d9", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\\n", - "\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n", + "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "af251217", + "id": "1bbd0f50", "metadata": { "editable": true }, "source": [ "We see clearly that\n", - "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$.\n", + "$\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\mathbb{E} \\big[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}\\big ]$ for any $\\lambda > 0$.\n", "\n", "Show also that the variance is" ] }, { "cell_type": "markdown", - "id": "02e978da", + "id": "617deac1", "metadata": { "editable": true }, @@ -254,12 +253,12 @@ }, { "cell_type": "markdown", - "id": "3656a896", + "id": "bee378b0", "metadata": { "editable": true }, "source": [ - "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero." + "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of the Ridge parameters $\\boldsymbol{\\beta}$ goes to zero." ] } ], diff --git a/doc/src/week37/exercisesweek37.do.txt b/doc/src/week37/exercisesweek37.do.txt index 613f24a37..8051429ac 100644 --- a/doc/src/week37/exercisesweek37.do.txt +++ b/doc/src/week37/exercisesweek37.do.txt @@ -80,12 +80,11 @@ A given parameter $\beta_j$ is given by the diagonal matrix element of the above Show that !bt \[ -\mathbb{E} \big[ \hat{\bm{\beta}}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\ -\bm{\beta}}. +\mathbb{E} \big[ \hat{\bm{\beta}}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\bm{\beta}. \] !et We see clearly that -$\mathbb{E} \big[ \bm{\beta}^{\mathrm{Ridge}} \big] \not= \bm{\beta}^{\mathrm{OLS}}$ for any $\lambda > 0$. +$\mathbb{E} \big[ \hat{\bm{\beta}}^{\mathrm{Ridge}} \big] \not= \mathbb{E} \big[\hat{\bm{\beta}}^{\mathrm{OLS}}\big ]$ for any $\lambda > 0$. Show also that the variance is @@ -95,7 +94,7 @@ Show also that the variance is \mbox{Var}[\hat{\bm{\beta}}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T}\mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, \] !et -and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\bm{\beta}$ goes to zero. +and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of the Ridge parameters $\bm{\beta}$ goes to zero.