Intermediate commit of week 36

doc/LectureNotes/exercisesweek36.ipynb

@@ -37,19 +37,123 @@
    "id": "ff6ebdb5",
    "metadata": {},
    "source": [
-    "**a)** How many degrees of freedom does an OLS model fit to the features $x, x^2, x^3$ and the intercept have?\n",
+    "**a)** How many degrees of freedom does an OLS model fit to the features $x, x^2, x^3$ and the intercept have?\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b4ea1834",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-success\">\n",
     "\n",
-    "**b)** Why is it bad for a model to have too many degrees of freedom?\n",
+    "A ordinary least squares model fit with $n$ features has $n$ free parameters. In this case $n=4$, as the intercept can be interpreted as a separate feature.\n",
     "\n",
-    "**c)** Why is it bad for a model to have too few degrees of freedom?\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "43d5ceca",
+   "metadata": {},
+   "source": [
     "\n",
-    "**d)** Read [chapter 3.4.1 of Hastie et al.'s book](https://link.springer.com/book/10.1007/978-0-387-84858-7). What is the expression for the effective degrees of freedom of the ridge regression fit?\n",
+    "**b)** Why is it bad for a model to have too many degrees of freedom?\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "54f77cc5",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-success\">\n",
+    "If we have too many degrees of freedom our model is at danger of overfitting the training data, i.e. trading a very low bias with a high variance in the bias-variance-tradeoff. The model adjusts to every slightest variation in the training data.\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "053f115d",
+   "metadata": {},
+   "source": [
     "\n",
-    "**e)** Why might we want to use Ridge regression instead of OLS?\n",
+    "**c)** Why is it bad for a model to have too few degrees of freedom?\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fabceb50",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-success\">\n",
+    "Because in this case the model is not able to reproduce the structure of the problem well enough and this leads to a very high bias (underfitting). Imagine approximating a circle with a straight line, as we have no degree of freedom left to describe the curvature.\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6eef6aaa",
+   "metadata": {},
+   "source": [
+    "\n",
+    "**d)** Read [chapter 3.4.1 of Hastie et al.'s book](https://link.springer.com/book/10.1007/978-0-387-84858-7). What is the expression for the effective degrees of freedom of the ridge regression fit?\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d79b8cbc",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-success\">\n",
+    "The number of effective degrees of freedom is given by\n",
+    "\n",
+    "$$\n",
+    "n_\\text{effective} = \\sum_{i=1}^{n_\\text{features}} \\frac{d_i^2}{d_i^2 + \\lambda}\n",
+    "$$\n",
+    "\n",
+    "where $d_i$ is the singular value of variable $i$.\n",
+    "\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "428d20c5",
+   "metadata": {},
+   "source": [
+    "\n",
+    "**e)** Why might we want to use Ridge regression instead of OLS?\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "57d76adb",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-success\">\n",
+    "We might want to use Ridge regression instead of OLS to control overfitting within a problem with many input features as we can effectively reduce the $n_{dof}$ of the problem.\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "868b27f1",
+   "metadata": {},
+   "source": [
     "\n",
     "**f)** Why migth we want to use OLS instead of Ridge regression?"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "07b73cba",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-success\">\n",
+    "If our goal is to describe a dataset as close as possible, OLS outperforms Ridge regression, as Ridge regression systematically underestimates our parameters of the model.\n",
+    "</div>"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "a78b2ce2",
@@ -142,6 +246,40 @@
     "$$"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "8939b1b0",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-success\">\n",
+    "\n",
+    "If we put the optimal parameters of the ridge regression $\\hat{\\beta}_\\text{Ridge}$ into our model we get:\n",
+    "$X \\hat \\beta_\\text{Ridge} = X (X^T X + \\lambda I)^{-1} X^T y$. If we approximate the inverse around small $\\lambda$ we get:\n",
+    "\n",
+    "$$\n",
+    "(A + \\lambda I)^{-1} = A^{-1} - \\lambda A^{-2} + \\mathcal{O}(\\lambda^2) = X^{-1} (X^T)^{-1} + \\lambda X^{-1} (X^T)^{-1} X^{-1} (X^T)^{-1}  + \\mathcal{O}(\\lambda^2) \n",
+    "$$\n",
+    "Then we can insert:\n",
+    "\n",
+    "$$\n",
+    "\\begin{aligned}\n",
+    "X \\hat \\beta_\\text{Ridge} &= X (X^{-1} (X^T)^{-1} + \\lambda X^{-1} (X^T)^{-1} X^{-1} (X^T)^{-1}  + \\mathcal{O}(\\lambda^2)) X^T y\\\\\n",
+    "&= X X^{-1} (X^T)^{-1} X^T y + \\lambda X X^{-1} (X^T)^{-1} X^{-1} (X^T)^{-1} X^T y + \\mathcal{O}(\\lambda^2))\\\\\n",
+    "&= y + \\lambda (X^T)^{-1} X^{-1} y + \\mathcal{O}(\\lambda^2))\\\\\n",
+    "&\\approx y + \\lambda (X X^T)^{-1} y\n",
+    "\\end{aligned}\n",
+    "$$\n",
+    "Thus our cost function can be written as\n",
+    "$$\n",
+    "\\begin{aligned}\n",
+    "(y-X\\beta)^T(y-X\\beta) + \\lambda \\beta^T \\beta &= (\\lambda (X X^T)^{-1} y)^T \\lambda (X X^T)^{-1} y + \\lambda \\beta^T \\beta\\\\\n",
+    "&= \\lambda^2 y^T ((X X^T)^{-1})^T (X X^T)^{-1} y + \\lambda \\dots\n",
+    "\\end{aligned}\n",
+    "$$\n",
+    "\n",
+    "</div>"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "2faaae78",
@@ -351,12 +489,32 @@
    "id": "e6cdc89a",
    "metadata": {},
    "source": [
-    "**a)** Compute the MSE of your ridge model for polynomials of degrees 1 to 5 with lambda set to 0.01. Plot the MSE as a function of polynomial degree.\n",
+    "**a)** Compute the MSE of your ridge model for polynomials of degrees 1 to 5 with lambda set to 0.01. Plot the MSE as a function of polynomial degree.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4c5dc12e",
+   "metadata": {},
+   "source": [
     "\n",
-    "**b)** Compute the MSE of your ridge model for a polynomial with degree 3, and with lambdas from $10^{-1}$ to $10^{-5}$ on a logarithmic scale. Plot the MSE as a function of lambda.\n",
+    "**b)** Compute the MSE of your ridge model for a polynomial with degree 3, and with lambdas from $10^{-1}$ to $10^{-5}$ on a logarithmic scale. Plot the MSE as a function of lambda.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5dfec17d",
+   "metadata": {},
+   "source": [
     "\n",
     "**c)** Compute the MSE of your ridge model for polynomials of degrees 1 to 5, and with lambdas from $10^{-1}$ to $10^{-5}$ on a logarithmic scale. Plot the MSE as a function of polynomial degree and lambda using a [heatmap](https://matplotlib.org/stable/gallery/images_contours_and_fields/image_annotated_heatmap.html)."
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d850e365",
+   "metadata": {},
+   "source": []
   }
  ],
  "metadata": {

pyproject.toml

@@ -4,6 +4,7 @@ version = "0.1.0"
 requires-python = ">=3.13"
 dependencies = [
     "ipykernel>=6.30.1",
+    "jupyter>=1.1.1",
     "matplotlib>=3.10.5",
     "numpy>=2.3.2",
     "pandas>=2.3.1",