lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update project 2

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-10-09 06:03:15 +02:00
parent 9923bb2339
commit 3c553e703e
9 changed files with 69 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/Projects/2023/Project2/Project2.do.txt
+5 -6
View File
@@ -54,13 +54,14 @@ function.
You should include in your analysis of the GD and SGD codes the following elements
o A plain gradient descent with a fixed learning rate (you will need to tune it).
o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).
o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)
o A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient.
o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient.
o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient.
o Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.
o Add RMSprop and Adam to your library of methods for tuning the learning rate.
The lecture notes from "week 39 contain more
The lecture notes from "weeks 39 and 40contain more
details":"https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" and code examples. Feel free to use these examples.
o Replace thereafter your analytical gradient with either _Autograd_ or _JAX_
In summary, you should
perform an analysis of the results for OLS and Ridge regression as
@@ -192,8 +193,6 @@ your final project 3, but now applied to other data sets.
o The text of Michael Nielsen is highly recommended, see "Nielsen's book":"http://neuralnetworksanddeeplearning.com/". It is an excellent read.
o The textbook of "Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer":"https://www.springer.com/gp/book/9780387848570", chapters 3 and 7 are the most relevant ones for the analysis here.
o "Mehta et al, arXiv 1803.08823":"https://arxiv.org/abs/1803.08823", *A high-bias, low-variance introduction to Machine Learning for physicists*, ArXiv:1803.08823.
o "Goodfellow, Bengio and Courville":"https://www.deeplearningbook.org/", *Deep Learning*.