added exercises

2023-10-15 21:41:09 +02:00
parent cc7b391123
commit bf9fa18a36
4 changed files with 3044 additions and 756 deletions
@@ -0,0 +1,73 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "be117070",
+   "metadata": {
+    "editable": true
+   },
+   "source": [
+    "<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
+    "doconce format html exercisesweek42.do.txt  -->\n",
+    "<!-- dom:TITLE: Exercises week 42 -->"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6f7356c3",
+   "metadata": {
+    "editable": true
+   },
+   "source": [
+    "# Exercises week 42\n",
+    "**October 9-13, 2023**\n",
+    "\n",
+    "Date: **Deadline is Sunday October 22 at midnight**\n",
+    "\n",
+    "You can hand in the exercises from week 41 and week 42 as one exercise and get a total score of two additional points."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "aa378ef2",
+   "metadata": {
+    "editable": true
+   },
+   "source": [
+    "# Overarching aims of the exercises this week\n",
+    "\n",
+    "The aim of the exercises this week is to get started with implementing\n",
+    "gradient methods of relevance for project 2. The exercise this week is a simple\n",
+    "continuation from the  previous week with the addition of automatic differentation.\n",
+    "Everything you develop here will be used in project 2. \n",
+    "\n",
+    "In order to get started, we will now replace in our standard ordinary\n",
+    "least squares (OLS) and Ridge regression codes (from project 1) the\n",
+    "matrix inversion algorithm with our own gradient descent (GD) and SGD\n",
+    "codes.  You can use the Franke function or the terrain data from\n",
+    "project 1. **However, we recommend using a simpler function like**\n",
+    "$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials.\n",
+    "You can obviously test your final codes against for example the Franke\n",
+    "function. Automatic differentiation will be discussed next week.\n",
+    "\n",
+    "You should include in your analysis of the GD and SGD codes the following elements\n",
+    "1. A plain gradient descent with a fixed learning rate (you will need to tune it) using automatic differentiation. Compare this with the analytical expression of the gradients you obtained last week. Feel free to use **Autograd** as Python package or **JAX**. You can use the examples form last week.\n",
+    "\n",
+    "2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Compare this with the analytical expression of the gradients you obtained last week.\n",
+    "\n",
+    "3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)\n",
+    "\n",
+    "4. Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD using automatic differentiation..\n",
+    "\n",
+    "5. Add RMSprop and Adam to your library of methods for tuning the learning rate. Again using automatic differentiation.\n",
+    "\n",
+    "The lecture notes from weeks 39 and 40 contain more information and code examples. Feel free to use these examples.\n",
+    "\n",
+    "We recommend reading chapter 8 on optimization from the textbook of [Goodfellow, Bengio and Courville](https://www.deeplearningbook.org/). This chapter contains many useful insights and discussions on the optimization part of machine learning."
+   ]
+  }
+ ],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 5
+}