Morten Hjorth-Jensen
74 lines
3.5 KiB
Plaintext
74 lines
3.5 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "be117070",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html exercisesweek42.do.txt -->\n",
|
|
"<!-- dom:TITLE: Exercises week 42 -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6f7356c3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Exercises week 42\n",
|
|
"**October 9-13, 2023**\n",
|
|
"\n",
|
|
"Date: **Deadline is Sunday October 22 at midnight**\n",
|
|
"\n",
|
|
"You can hand in the exercises from week 41 and week 42 as one exercise and get a total score of two additional points."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "aa378ef2",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Overarching aims of the exercises this week\n",
|
|
"\n",
|
|
"The aim of the exercises this week is to get started with implementing\n",
|
|
"gradient methods of relevance for project 2. The exercise this week is a simple\n",
|
|
"continuation from the previous week with the addition of automatic differentation.\n",
|
|
"Everything you develop here will be used in project 2. \n",
|
|
"\n",
|
|
"In order to get started, we will now replace in our standard ordinary\n",
|
|
"least squares (OLS) and Ridge regression codes (from project 1) the\n",
|
|
"matrix inversion algorithm with our own gradient descent (GD) and SGD\n",
|
|
"codes. You can use the Franke function or the terrain data from\n",
|
|
"project 1. **However, we recommend using a simpler function like**\n",
|
|
"$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials.\n",
|
|
"You can obviously test your final codes against for example the Franke\n",
|
|
"function. Automatic differentiation will be discussed next week.\n",
|
|
"\n",
|
|
"You should include in your analysis of the GD and SGD codes the following elements\n",
|
|
"1. A plain gradient descent with a fixed learning rate (you will need to tune it) using automatic differentiation. Compare this with the analytical expression of the gradients you obtained last week. Feel free to use **Autograd** as Python package or **JAX**. You can use the examples form last week.\n",
|
|
"\n",
|
|
"2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Compare this with the analytical expression of the gradients you obtained last week.\n",
|
|
"\n",
|
|
"3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)\n",
|
|
"\n",
|
|
"4. Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD using automatic differentiation..\n",
|
|
"\n",
|
|
"5. Add RMSprop and Adam to your library of methods for tuning the learning rate. Again using automatic differentiation.\n",
|
|
"\n",
|
|
"The lecture notes from weeks 39 and 40 contain more information and code examples. Feel free to use these examples.\n",
|
|
"\n",
|
|
"We recommend reading chapter 8 on optimization from the textbook of [Goodfellow, Bengio and Courville](https://www.deeplearningbook.org/). This chapter contains many useful insights and discussions on the optimization part of machine learning."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|