Intermediate commit of week 37 due to new tasks

doc/LectureNotes/exercisesweek36.ipynb

@@ -10,6 +10,14 @@
     "## Deriving and Implementing Ridge Regression"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "6b5c366c",
+   "metadata": {},
+   "source": [
+    "**Python Code can be found at https://github.uio.no/larsbog/FYS-STK4155**"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "e5cc5739",

doc/LectureNotes/exercisesweek37.ipynb

@@ -28,7 +28,15 @@
   },
   {
    "cell_type": "markdown",
-   "id": "921bf331",
+   "id": "1f66bb2f",
+   "metadata": {},
+   "source": [
+    "**Python Code can be found at https://github.uio.no/larsbog/FYS-STK4155**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7cdd88e4",
    "metadata": {
     "editable": true
    },
@@ -608,15 +616,55 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9ba303be",
+   "id": "2f2b970c",
+   "metadata": {},
+   "source": [
+    "In this simple example (well converging function) the gradient descent converges for all learning rates given enough iterations. The Error on the paramters approaches 10^-2 indicating the converging."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7e7e7de6",
    "metadata": {
     "editable": true
    },
    "source": [
     "### 4b)\n",
     "\n",
-    "Write then a similar code for Ridge regression using the above template.\n",
-    "Try to add a stopping parameter as function of the number iterations and the difference between the new and old $\\theta$ values. How would you define a stopping criterion?\n"
+    "Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?\n",
+    "\n",
+    "\n",
+    "We define the stopping criterion via the decrease in the cost function. If the decrease gets too small we stop. This will be implemented using a custom stopping_criterion function so it can easily be replaced."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4258b4f3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def gradient_descent(X, y, cost_func, grad_cost_func, eta=0.1, num_iters=1000, stopping_criterion=None, **kwargs):\n",
+    "    # Initialize weights\n",
+    "    theta = np.zeros(X.shape[1])\n",
+    "    # Store cost history\n",
+    "    cost_history = np.zeros(num_iters)\n",
+    "    for t in range(num_iters):\n",
+    "        # Compute cost\n",
+    "        cost_history[t] = cost_func(X, y, theta, **kwargs)\n",
+    "        # Compute gradient\n",
+    "        grad = grad_cost_func(X, y, theta, **kwargs)\n",
+    "        # Update weights\n",
+    "        theta -= eta * grad\n",
+    "        if stopping_criterion is not None and stopping_criterion(cost_history[:t+1]):\n",
+    "            print(f\"Converged at iteration {t}\")\n",
+    "            break\n",
+    "    return theta, cost_history\n",
+    "\n",
+    "def stopping_criterion(cost_history, tol=1e-5):\n",
+    "    if len(cost_history) < 2:\n",
+    "        return False\n",
+    "    return np.abs(cost_history[-1] - cost_history[-2]) < tol\n"
    ]
   },
   {
@@ -720,7 +768,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": null,
    "id": "3e466f75",
    "metadata": {},
    "outputs": [
@@ -734,29 +782,6 @@
     }
    ],
    "source": [
-    "def gradient_descent(X, y, cost_func, grad_cost_func, eta=0.1, num_iters=1000, stopping_criterion=None, **kwargs):\n",
-    "    # Initialize weights\n",
-    "    theta = np.zeros(X.shape[1])\n",
-    "    # Store cost history\n",
-    "    cost_history = np.zeros(num_iters)\n",
-    "    for t in range(num_iters):\n",
-    "        # Compute cost\n",
-    "        cost_history[t] = cost_func(X, y, theta, **kwargs)\n",
-    "        # Compute gradient\n",
-    "        grad = grad_cost_func(X, y, theta, **kwargs)\n",
-    "        # Update weights\n",
-    "        theta -= eta * grad\n",
-    "        if stopping_criterion is not None and stopping_criterion(cost_history[:t+1]):\n",
-    "            print(f\"Converged at iteration {t}\")\n",
-    "            break\n",
-    "    return theta, cost_history\n",
-    "\n",
-    "def stopping_criterion(cost_history, tol=1e-5):\n",
-    "    if len(cost_history) < 2:\n",
-    "        return False\n",
-    "    return np.abs(cost_history[-1] - cost_history[-2]) < tol\n",
-    "\n",
-    "\n",
     "res = gradient_descent(X, y, OLS_cost_func, OLS_grad_cost_func, num_iters=100000, stopping_criterion=stopping_criterion)\n",
     "print(f\"Terminated with error on theta of {theta_error(res[0])[1]:.3e}\")"
    ]