lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2022-10-07 08:15:22 +02:00
parent 3405bc19b7
commit 2b1016a356
3 changed files with 54 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/Projects/2022/Project2/ipynb
BIN
View File
Binary file not shown.
doc/Projects/2022/Project2/pdf
BIN
View File
Binary file not shown.
doc/src/Projects/2022/Project2/Project2.do.txt
+54 -61
View File
@@ -6,30 +6,37 @@ DATE: today
===== Classification and Regression, from linear and logistic regression to neural networks =====
The main aim of this project is to study both classification and
regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied
in project 1. We will also include logistic regression for classification
problems and write our own FFNN code for studying
both regression and classification problems. The codes developed in
project 1, including bootstrap _and/or_ cross-validation as well as the
computation of the mean-squared error and/or the $R2$ or the accuracy score (classification problems) functions can
also be utilized in the present analysis.
regression problems by developing our own feed-forward neural network
(FFNN) code. We can reuse the regression algorithms studied in project
1. We will also include logistic regression for classification
problems and write our own FFNN code for studying both regression and
classification problems. The codes developed in project 1, including
bootstrap _and/or_ cross-validation as well as the computation of the
mean-squared error and/or the $R2$ or the accuracy score
(classification problems) functions can also be utilized in the
present analysis.
_Important Note_: This project as well as projects 1 and 3 have to be written as a scientific report. The instructions on how to do this and how we grade are available at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/EvaluationGrading/EvaluationForm.md". We will discuss this format during the various lab sessions. Please do spend some time to read our guidelines.
The data sets that we propose here are (the default sets)
* Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be
o Either the Franke function or the terrain data from project 1, or data sets your propose. It could be a simpler function than the Franke function.
o A simple one-dimensional function or the Franke function or the terrain data from project 1, or data sets your propose. It could be a simpler function than the Franke function. We recommend testing a simpler function (see below). But if you wish to try more complex function, feel free to do so.
* Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called "Wisconsin Breat Cancer Data":"https://www.kaggle.com/uciml/breast-cancer-wisconsin-data" data set of images representing various features of tumors. These are discussed intensively in the lecture notes, see for example the slides from "week 41":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html". A longer explanation with links to the scientific literature can be found at the "Machine Learning repository of the University of California at Irvine":"https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29". Feel free to consult this site and the pertinent literature.
You can find more information about this at the "Scikit-Learn site":"https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" or at the "University of California at Irvine":"https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)".
However, if you would like to study other data sets, feel free to
propose other sets. What we list here are mere suggestions from our
side. If you opt for another data set, consider using a set which
has been studied in the scientific literature. This makes it easier
for you to compare and analyze your results. Comparing with existing results from the scientific literature is also an essential
element of the scientific discussion. The University of California at Irvine with its Machine Learning repository at URL:"https://archive.ics.uci.edu/ml/index.php" is an excellent site to look up for examples and inspiration. "Kaggle.com":"https://www.kaggle.com/" is an equally interesting site. Feel free to explore these sites.
side. If you opt for another data set, consider using a set which has
been studied in the scientific literature. This makes it easier for
you to compare and analyze your results. Comparing with existing
results from the scientific literature is also an essential element of
the scientific discussion. The University of California at Irvine
with its Machine Learning repository at
URL:"https://archive.ics.uci.edu/ml/index.php" is an excellent site to
look up for examples and
inspiration. "Kaggle.com":"https://www.kaggle.com/" is an equally
interesting site. Feel free to explore these sites.
We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code.
@@ -37,13 +44,26 @@ We will start with a regression problem and we will reuse our codes from project
=== Part a): Write your own Stochastic Gradient Descent code, first step ===
In order to get started, we will now replace in our standard ordinary
least squares (OLS) and Ridge regression codes (from project 1) the matrix inversion
algorithm with our own SGD code. You can choose whether you want to
add the momentum SGD optionality or other SGD variants such as RMSprop
or ADAgrad or ADAM. The lecture notes from "week 39 contain more
details":"https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html"
least squares (OLS) and Ridge regression codes (from project 1) the
matrix inversion algorithm with our own gradient descent (GD) and SGD
codes. You can use the Franke function or the terrain data from
project 1. However, we recommend using a simpler function like
$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials.
You can obviously test your final codes against for example the Franke
function.
Perform an analysis of the results for OLS and Ridge regression as
You should include in your analysis of the GD and SGD codes the following elements
o A plain gradient descent with a fixed learning rate (you will need to tune it).
o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).
o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)
o Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.
o Add RMSprop and Adam to your library of methods for tuning the learning rate.
The lecture notes from "week 39 contain more
details":"https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" and code examples. Feel free to use these examples.
In summary, you should
perform an analysis of the results for OLS and Ridge regression as
function of the chosen learning rates, the number of mini-batches and
epochs as well as algorithm for scaling the learning rate. You can
also compare your own results with those that can be obtained using
@@ -51,7 +71,14 @@ for example _Scikit-Learn_'s various SGD options. Discuss your
results. For Ridge regression you need now to study the results as functions of the hyper-parameter $\lambda$ and
the learning rate $\eta$. Discuss your results.
You will need your SGD code for the setup of the Neural Network and Logistic Regression codes. You will find the Python "Seaborn package":"https://seaborn.pydata.org/generated/seaborn.heatmap.html" useful when plotting the results as function of the learning rate $\eta$ and the hyper-parameter $\lambda$ when you use Ridge regression.
You will need your SGD code for the setup of the Neural Network and
Logistic Regression codes. You will find the Python "Seaborn
package":"https://seaborn.pydata.org/generated/seaborn.heatmap.html"
useful when plotting the results as function of the learning rate
$\eta$ and the hyper-parameter $\lambda$ when you use Ridge
regression.
We recommend reading chapter 8 on optimization from the textbook of "Goodfellow, Bengio and Courville":"https://www.deeplearningbook.org/". This chapter contains many useful insights and discussions on the optimization part of machine learning.
=== Part b): Writing your own Neural Network code ===
@@ -60,9 +87,11 @@ write your own Feed Forward Neural Network code implementing the back
propagation algorithm discussed in the lecture slides from "week 40":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week40.html" and
"week 41":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html".
We will focus on a regression problem first and study either the
We will focus on a regression problem first and study either the simple second-order polynomial from part a) or the
Franke function or terrain data (or both or other data sets) from
project 1. Discuss again your choice of cost function.
project 1.
Discuss again your choice of cost function.
Write an FFNN code for regression with a flexible number of hidden
layers and nodes using the Sigmoid function as activation function for
@@ -70,12 +99,12 @@ the hidden layers. Initialize the weights using a normal
distribution. How would you initialize the biases? And which
activation function would you select for the final output layer?
Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1.
Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1 if you use the Franke function or the terrain data.
You should test your results against a similar code using _Scikit-Learn_ (see the examples in the above lecture notes from week 41) or _tensorflow/keras_.
Comment your results and give a critical discussion of the results
obtained with the Linear Regression code and your own Neural Network
code. Compare the results with those from project 1.
code.
Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and $R2$ scores.
A useful reference on the back progagation algorithm is "Nielsen's
@@ -167,7 +196,7 @@ o The textbook of "Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The El
o "Mehta et al, arXiv 1803.08823":"https://arxiv.org/abs/1803.08823", *A high-bias, low-variance introduction to Machine Learning for physicists*, ArXiv:1803.08823.
o "Goodfellow, Bengio and Courville":"https://www.deeplearningbook.org/", *Deep Learning*.
===== Introduction to numerical projects =====
@@ -214,39 +243,3 @@ we encourage you to collaborate. Optimal working groups consist of
2-3 students. You can then hand in a common report.
===== Software and needed installations =====
If you have Python installed (we recommend Python3) and you feel pretty familiar with installing different packages,
we recommend that you install the following Python packages via _pip_ as
o pip install numpy scipy matplotlib ipython scikit-learn tensorflow sympy pandas pillow
For Python3, replace _pip_ with _pip3_.
See below for a discussion of _tensorflow_ and _scikit-learn_.
For OSX users we recommend also, after having installed Xcode, to install _brew_. Brew allows
for a seamless installation of additional software via for example
o brew install python3
For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution
you can use _pip_ as well and simply install Python as
o sudo apt-get install python3 (or python for python2.7)
etc etc.
If you don't want to install various Python packages with their dependencies separately, we recommend two widely used distrubutions which set up all relevant dependencies for Python, namely
o "Anaconda":"https://docs.anaconda.com/" Anaconda is an open source distribution of the Python and R programming languages for large-scale data processing, predictive analytics, and scientific computing, that aims to simplify package management and deployment. Package versions are managed by the package management system _conda_
o "Enthought canopy":"https://www.enthought.com/product/canopy/" is a Python distribution for scientific and analytic computing distribution and analysis environment, available for free and under a commercial license.
Popular software packages written in Python for ML are
* "Scikit-learn":"http://scikit-learn.org/stable/",
* "Tensorflow":"https://www.tensorflow.org/",
* "PyTorch":"http://pytorch.org/" and
* "Keras":"https://keras.io/".
These are all freely available at their respective GitHub sites. They
encompass communities of developers in the thousands or more. And the number
of code developers and contributors keeps increasing.