lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update p2

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2024-10-08 06:16:26 +02:00
parent 51a7c88049
commit c51c668c16
10 changed files with 2726 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/Projects/2024/Project2/Project2.do.txt
+17 -14
View File
@@ -1,4 +1,4 @@
TITLE: Project 2 on Machine Learning, deadline November 17 (Midnight)
TITLE: Project 2 on Machine Learning, deadline November 4 (Midnight)
AUTHOR: "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html" {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo, Norway
DATE: today
@@ -21,7 +21,7 @@ The data sets that we propose here are (the default sets)
* Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be
o A simple one-dimensional function or the Franke function or the terrain data from project 1, or data sets your propose. It could be a simpler function than the Franke function. We recommend testing a simpler function (see below). But if you wish to try more complex function, feel free to do so.
* Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called "Wisconsin Breat Cancer Data":"https://www.kaggle.com/uciml/breast-cancer-wisconsin-data" data set of images representing various features of tumors. These are discussed intensively in the lecture notes, see for example the slides from "week 41":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html". A longer explanation with links to the scientific literature can be found at the "Machine Learning repository of the University of California at Irvine":"https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29". Feel free to consult this site and the pertinent literature.
* Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called "Wisconsin Breat Cancer Data":"https://www.kaggle.com/uciml/breast-cancer-wisconsin-data" data set of images representing various features of tumors. A longer explanation with links to the scientific literature can be found at the "Machine Learning repository of the University of California at Irvine":"https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29". Feel free to consult this site and the pertinent literature.
You can find more information about this at the "Scikit-Learn site":"https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" or at the "University of California at Irvine":"https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)".
@@ -52,6 +52,8 @@ $f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials.
You can obviously test your final codes against for example the Franke
function.
The exercise set for week 41 should help in solving this part of the project.
You should include in your analysis of the GD and SGD codes the following elements
o A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient.
@@ -59,10 +61,13 @@ o Add momentum to the plain GD code and compare convergence with a fixed learnin
o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient.
o Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.
o Add RMSprop and Adam to your library of methods for tuning the learning rate.
The lecture notes from "weeks 39 and 40contain more
The lecture notes from "weeks 39 and 40 contain more
details":"https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" and code examples. Feel free to use these examples.
o Replace thereafter your analytical gradient with either _Autograd_ or _JAX_
_Feel free to use codes on these methods from the lecture notes from week 39 and week 40_.
In summary, you should
perform an analysis of the results for OLS and Ridge regression as
function of the chosen learning rates, the number of mini-batches and
@@ -77,16 +82,16 @@ Logistic Regression codes. You will find the Python "Seaborn
package":"https://seaborn.pydata.org/generated/seaborn.heatmap.html"
useful when plotting the results as function of the learning rate
$\eta$ and the hyper-parameter $\lambda$ when you use Ridge
regression.
regression. Since you will use different gradient descent methods, you can also add Lasse regression. This is however optional. How to code Lasso regression is discussed in the lecture notes from week 40.
We recommend reading chapter 8 on optimization from the textbook of "Goodfellow, Bengio and Courville":"https://www.deeplearningbook.org/". This chapter contains many useful insights and discussions on the optimization part of machine learning.
We recommend reading chapter 8 on optimization from the textbook of Goodfellow, Bengio and Courville at URL:"https://www.deeplearningbook.org/". This chapter contains many useful insights and discussions on the optimization part of machine learning.
=== Part b): Writing your own Neural Network code ===
Your aim now, and this is the central part of this project, is to
write your own Feed Forward Neural Network code implementing the back
propagation algorithm discussed in the lecture slides from "week 40":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week40.html" and
"week 41":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html".
propagation algorithm discussed in the lecture slides from "week 41":"https://compphysics.github.io/MachineLearning/doc/pub/week41/ipynb/week41.ipynb" and
"week 42":"https://compphysics.github.io/MachineLearning/doc/pub/week42/ipynb/week42.ipynb".
We will focus on a regression problem first and study either the simple second-order polynomial from part a) or the
Franke function or terrain data (or both or other data sets) from
@@ -101,15 +106,14 @@ distribution. How would you initialize the biases? And which
activation function would you select for the final output layer?
Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1 if you use the Franke function or the terrain data.
You should test your results against a similar code using _Scikit-Learn_ (see the examples in the above lecture notes from week 41) or _tensorflow/keras_.
You should test your results against a similar code using _Scikit-Learn_ (see the examples in the above lecture notes from weeks 41 and 42) or _tensorflow/keras_ or _Pytorch_ (for Pytorch, see Raschka et al.'s text chapters 12 and 13).
Comment your results and give a critical discussion of the results
obtained with the Linear Regression code and your own Neural Network
code.
Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and $R2$ scores.
A useful reference on the back progagation algorithm is "Nielsen's
book":"http://neuralnetworksanddeeplearning.com/". It is an excellent
A useful reference on the back progagation algorithm is Nielsen's book at URL:"http://neuralnetworksanddeeplearning.com/". It is an excellent
read.
@@ -191,12 +195,11 @@ your final project 3, but now applied to other data sets.
===== Background literature =====
o The text of Michael Nielsen is highly recommended, see "Nielsen's book":"http://neuralnetworksanddeeplearning.com/". It is an excellent read.
o The text of Michael Nielsen is highly recommended, see Nielsen's book at URL:"http://neuralnetworksanddeeplearning.com/". It is an excellent read.
o "Mehta et al, arXiv 1803.08823":"https://arxiv.org/abs/1803.08823", *A high-bias, low-variance introduction to Machine Learning for physicists*, ArXiv:1803.08823.
o "Goodfellow, Bengio and Courville":"https://www.deeplearningbook.org/", *Deep Learning*.
o Goodfellow, Bengio and Courville, Deep Learning at URL:"https://www.deeplearningbook.org/". Here we recommend chapters 6, 7 and 8
o Raschka et al. at URL:"https://sebastianraschka.com/blog/2022/ml-pytorch-book.html". Here we recommend chapters 11, 12 and 13.
===== Introduction to numerical projects =====