update
This commit is contained in:
Morten Hjorth-Jensen
@@ -0,0 +1,87 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "42211505",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
||||
"doconce format html exercisesweek41.do.txt -->\n",
|
||||
"<!-- dom:TITLE: Exercises week 41 -->"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "abcd6d2e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"# Exercises week 41\n",
|
||||
"**October 9-13, 2023**\n",
|
||||
"\n",
|
||||
"Date: **Deadline is Sunday October 15 at midnight**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a754339c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Overarching aims of the exercises this week\n",
|
||||
"\n",
|
||||
"The aim of the exercises this week is to get started with implementing\n",
|
||||
"gradient methods of relevance for project 2. This exercise will also\n",
|
||||
"be continued next week with the addition of automatic differentation.\n",
|
||||
"Everything you develop here will be used in project 2. \n",
|
||||
"\n",
|
||||
"In order to get started, we will now replace in our standard ordinary\n",
|
||||
"least squares (OLS) and Ridge regression codes (from project 1) the\n",
|
||||
"matrix inversion algorithm with our own gradient descent (GD) and SGD\n",
|
||||
"codes. You can use the Franke function or the terrain data from\n",
|
||||
"project 1. **However, we recommend using a simpler function like**\n",
|
||||
"$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials.\n",
|
||||
"You can obviously test your final codes against for example the Franke\n",
|
||||
"function.\n",
|
||||
"\n",
|
||||
"You should include in your analysis of the GD and SGD codes the following elements\n",
|
||||
"1. A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression of the gradients\n",
|
||||
"\n",
|
||||
"2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate), again using the analytical expression of the gradients.\n",
|
||||
"\n",
|
||||
"3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)\n",
|
||||
"\n",
|
||||
"4. Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.\n",
|
||||
"\n",
|
||||
"5. Add RMSprop and Adam to your library of methods for tuning the learning rate.\n",
|
||||
"\n",
|
||||
"The lecture notes from \"weeks 39 and 40 contain more information and code examples. Feel free to use these examples.\n",
|
||||
"\n",
|
||||
"In summary, you should \n",
|
||||
"perform an analysis of the results for OLS and Ridge regression as\n",
|
||||
"function of the chosen learning rates, the number of mini-batches and\n",
|
||||
"epochs as well as algorithm for scaling the learning rate. You can\n",
|
||||
"also compare your own results with those that can be obtained using\n",
|
||||
"for example **Scikit-Learn**'s various SGD options. Discuss your\n",
|
||||
"results. For Ridge regression you need now to study the results as functions of the hyper-parameter $\\lambda$ and \n",
|
||||
"the learning rate $\\eta$. Discuss your results.\n",
|
||||
"\n",
|
||||
"You will need your SGD code for the setup of the Neural Network and\n",
|
||||
"Logistic Regression codes. You will find the Python [Seaborn\n",
|
||||
"package](https://seaborn.pydata.org/generated/seaborn.heatmap.html)\n",
|
||||
"useful when plotting the results as function of the learning rate\n",
|
||||
"$\\eta$ and the hyper-parameter $\\lambda$ when you use Ridge\n",
|
||||
"regression.\n",
|
||||
"\n",
|
||||
"We recommend reading chapter 8 on optimization from the textbook of [Goodfellow, Bengio and Courville](https://www.deeplearningbook.org/). This chapter contains many useful insights and discussions on the optimization part of machine learning."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -5,27 +5,27 @@ DATE: Deadline is Sunday October 15 at midnight
|
||||
|
||||
===== Overarching aims of the exercises this week =====
|
||||
|
||||
The aim of the exercises this week
|
||||
|
||||
The aim of the exercises this week is to get started with implementing
|
||||
gradient methods of relevance for project 2. This exercise will also
|
||||
be continued next week with the addition of automatic differentation.
|
||||
Everything you develop here will be used in project 2.
|
||||
|
||||
In order to get started, we will now replace in our standard ordinary
|
||||
least squares (OLS) and Ridge regression codes (from project 1) the
|
||||
matrix inversion algorithm with our own gradient descent (GD) and SGD
|
||||
codes. You can use the Franke function or the terrain data from
|
||||
project 1. However, we recommend using a simpler function like
|
||||
project 1. _However, we recommend using a simpler function like_
|
||||
$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials.
|
||||
You can obviously test your final codes against for example the Franke
|
||||
function.
|
||||
|
||||
|
||||
You should include in your analysis of the GD and SGD codes the following elements
|
||||
o A plain gradient descent with a fixed learning rate (you will need to tune it).
|
||||
o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).
|
||||
o A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression of the gradients
|
||||
o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate), again using the analytical expression of the gradients.
|
||||
o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)
|
||||
o Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.
|
||||
o Add RMSprop and Adam to your library of methods for tuning the learning rate.
|
||||
The lecture notes from "weeks 39 and 40 contain more
|
||||
details":"https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" and code examples. Feel free to use these examples.
|
||||
The lecture notes from "weeks 39 and 40 contain more information and code examples. Feel free to use these examples.
|
||||
|
||||
In summary, you should
|
||||
perform an analysis of the results for OLS and Ridge regression as
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user