Morten Hjorth-Jensen
88 lines
4.0 KiB
Plaintext
88 lines
4.0 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "42211505",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html exercisesweek41.do.txt -->\n",
|
|
"<!-- dom:TITLE: Exercises week 41 -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "abcd6d2e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Exercises week 41\n",
|
|
"**October 9-13, 2023**\n",
|
|
"\n",
|
|
"Date: **Deadline is Sunday October 15 at midnight**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a754339c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Overarching aims of the exercises this week\n",
|
|
"\n",
|
|
"The aim of the exercises this week is to get started with implementing\n",
|
|
"gradient methods of relevance for project 2. This exercise will also\n",
|
|
"be continued next week with the addition of automatic differentation.\n",
|
|
"Everything you develop here will be used in project 2. \n",
|
|
"\n",
|
|
"In order to get started, we will now replace in our standard ordinary\n",
|
|
"least squares (OLS) and Ridge regression codes (from project 1) the\n",
|
|
"matrix inversion algorithm with our own gradient descent (GD) and SGD\n",
|
|
"codes. You can use the Franke function or the terrain data from\n",
|
|
"project 1. **However, we recommend using a simpler function like**\n",
|
|
"$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials.\n",
|
|
"You can obviously test your final codes against for example the Franke\n",
|
|
"function.\n",
|
|
"\n",
|
|
"You should include in your analysis of the GD and SGD codes the following elements\n",
|
|
"1. A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression of the gradients\n",
|
|
"\n",
|
|
"2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate), again using the analytical expression of the gradients.\n",
|
|
"\n",
|
|
"3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)\n",
|
|
"\n",
|
|
"4. Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.\n",
|
|
"\n",
|
|
"5. Add RMSprop and Adam to your library of methods for tuning the learning rate.\n",
|
|
"\n",
|
|
"The lecture notes from \"weeks 39 and 40 contain more information and code examples. Feel free to use these examples.\n",
|
|
"\n",
|
|
"In summary, you should \n",
|
|
"perform an analysis of the results for OLS and Ridge regression as\n",
|
|
"function of the chosen learning rates, the number of mini-batches and\n",
|
|
"epochs as well as algorithm for scaling the learning rate. You can\n",
|
|
"also compare your own results with those that can be obtained using\n",
|
|
"for example **Scikit-Learn**'s various SGD options. Discuss your\n",
|
|
"results. For Ridge regression you need now to study the results as functions of the hyper-parameter $\\lambda$ and \n",
|
|
"the learning rate $\\eta$. Discuss your results.\n",
|
|
"\n",
|
|
"You will need your SGD code for the setup of the Neural Network and\n",
|
|
"Logistic Regression codes. You will find the Python [Seaborn\n",
|
|
"package](https://seaborn.pydata.org/generated/seaborn.heatmap.html)\n",
|
|
"useful when plotting the results as function of the learning rate\n",
|
|
"$\\eta$ and the hyper-parameter $\\lambda$ when you use Ridge\n",
|
|
"regression.\n",
|
|
"\n",
|
|
"We recommend reading chapter 8 on optimization from the textbook of [Goodfellow, Bengio and Courville](https://www.deeplearningbook.org/). This chapter contains many useful insights and discussions on the optimization part of machine learning."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|