lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update p2

Browse Source
This commit is contained in:
mhjensen
2020-10-07 07:08:08 +02:00
parent e94cb59ba5
commit b2df29a795
9 changed files with 804 additions and 648 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/Projects/2020/Project2/Project2.do.txt
+106 -84
View File
@@ -1,4 +1,4 @@
TITLE: Project 2 on Machine Learning, deadline November 7
TITLE: Project 2 on Machine Learning, deadline November 9
AUTHOR: "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html" {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo, Norway
DATE: today
@@ -7,8 +7,8 @@ DATE: today
The main aim of this project is to study both classification and
regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied
in project 1. We will include logistic regresion for classification
problems and write our own FFNNcode for studying
in project 1. We will also include logistic regression for classification
problems and write our own FFNN code for studying
both regression and classification problems. The codes developed in
project 1, including bootstrap and/or cross-validation as well as the
computation of the mean-squared error and/or the $R2$ or the accuracy score (classification problems) functions can
@@ -17,53 +17,95 @@ also be utilized in the present analysis.
The data sets that we propose here are (the default sets)
* Regression (fitting a continuous function). In this part you will need to bring up your results from project 1 and compare these with what you get from you Neural Network code to be developed here. The data sets could be
* Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be
o Either the Franke function or the terrain data from project 1, or data sets your propose.
* Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called MNIST data set of images representing hand-written numbers from zero to nine.
* Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called "MNIST":"https://en.wikipedia.org/wiki/MNIST_database" data set of images representing hand-written numbers from zero to nine. These are discussed intensively in the lecture notes on neural networks, see for example the slides from "week 41":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html"
However, if you would like to study other data sets, feel free to
propose other sets. What we listed here are mere suggestions from our
side. If you opt for another data set, consider using a set which
has been studied in the scientific literature. This makes it easier
for you to compare and analyze your results. It is also an essential
elements of the scientific discussion.
for you to compare and analyze your results. Comparing with existing results from the scientific literature is also an essential
element of the scientific discussion.
In particular, when developing your own Logistic Regression code for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more information about this at the "Scikit-Learn site":"https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" or at the "University of California at Irvine":"https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)". The "lecture slides on dimensionality reduction have several code examples on this data set":"https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimRed.html".
In particular, when developing your own Neural Network and Logistic Regression codes for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You can find more information about this at the "Scikit-Learn site":"https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" or at the "University of California at Irvine":"https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)".
=== Part a): Write your Logistic Regression code, first step ===
We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code.
If you opt for the credit card data, your first task is to familiarize yourself with the data set and the scientific article.
We recommend also that you study the code example in the "Logistic Regression":"https://compphysics.github.io/MachineLearning/doc/pub/LogReg/html/LogReg.html".
=== Part a): Write your own Stochastic Gradient Descent code, first step ===
Write the part of the code which reads in the data and sets up the relevant data sets.
In order to get started, we will now replace in our standard ordinary
least squares (OLS) and Ridge regression codes (from project 1) the matrix inversion
algorithm with our own SGD code. You can choose whether you want to
add the momentum SGD optionality or other SGD variants such as RMSprop
or ADAgrad. The lecture notes from "week 40 contain more
details":"https://compphysics.github.io/MachineLearning/doc/pub/week40/html/week40.html"
=== Part b): Write your Logistic Regression code, second step ===
Perform an analysis of the results for OLS and Ridge regression as
function of the chosen learning rates, the number of mini-batches and
epochs as well as algorithm for scaling the learning rate. You can
also compare your own results with those that can be obtained using
for example _Scikit-Learn_'s various SGD options. Discuss your
results. For Ridge regression you need now to study the results as functions of the hyper-parameter $\lambda$ and
the learning rate $\gamma$. Discuss your results.
Now you should write your Logistic Regression code with the aim to
reproduce the Logistic Regression analysis of the "scientific
article":"https://bradzzz.gitbooks.io/ga-seattle-dsi/content/dsi/dsi_05_classification_databases/2.1-lesson/assets/datasets/DefaultCreditCardClients_yeh_2009.pdf".
You will need your SGD code for the setup of the Neural Network and Logistic Regression codes.
Define your cost function and the design matrix before you start writing your code.
=== Part b): Writing your own Neural Network code ===
In order to find the optimal parameters of your logistic regressor you
should include a gradient descent solver, as discussed in the
"gradient descent
lectures":"https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html".
Since we don't have so many data points, you may just code the
standard gradient descent with a given learning rate, or even attempt
to use the Newton-Raphson method. Alternatively, it may be useful for
the next part on neural networks to implement a stochastic gradient
descent with and without mini-batches. Stochastic gradient with
mini-batches may give the best results. You could finally compare your
code with the output from _scikit-learn_'s toolbox for optimization
methods applied to logistic regression.
Your aim now, and this is the central part of this project, is to
write your own Feed Forward Neural Network code implementing the back
propagation algorithm discussed in the lecture slides from "week 41":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html".
We will focus on a regression problem first and study either the
Franke function or terrain data (or both or other data sets) from
project 1. Discuss again your choice of cost function.
Write an FFNN code for regression with a flexible number of hidden
layers and nodes using the Sigmoid function as activation function for
the hidden layers. Initialize the weights using a normal
distribution. How would you initialize the biases? And which
activation function would you select for the final output layer?
Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1.
You should test your results against a similar code using _Scikit-Learn_ (see the examples in the above lecture notes from week 41) or _tensorflow/keras_.
Comment your results and give a critical discussion of the results
obtained with the Linear Regression code and your own Neural Network
code. Compare the results with those from project 1.
Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and $R2$ scores.
A useful reference on the back progagation algorithm is "Nielsen's
book":"http://neuralnetworksanddeeplearning.com/". It is an excellent
read.
=== Part c): Testing different activation functions ===
You should now also test different activation functions for the hidden layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and discuss your results. You may also study the way you initialize your weights and biases.
=== Part d): Classification analysis using neural networks ===
With a well-written code it should now be easy to change the
activation function for the output layer.
Here we will change the cost function for our neural network code
developed in parts b) and c) in order to perform a classification analysis.
We will here study the MNIST data set of hand-written numbers as
discussed in the lecture notes from "week
41":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html". Use
the _Softmax_ function as activation function. Your code should
however also be able to use a binary activation function as well.
To measure the performance of our classification problem we use the
so-called *accuracy* score. The accuracy is as you would expect just
the number of correctly guessed targets $t_i$ divided by the total
number of targets. A perfect classifier will have an accuracy score of
$1$.
number of targets, that is
!bt
\[
@@ -72,64 +114,44 @@ $1$.
!et
where $I$ is the indicator function, $1$ if $t_i = y_i$ and $0$
otherwise if we have a binary classifcation problem. Here $t_i$
represents the target and $y_i$ the outputs of your Logistic
Regression code.
otherwise if we have a binary classification problem. Here $t_i$
represents the target and $y_i$ the outputs of your FFNN code and $n$ is simply the number of targets $t_i$.
Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter $\lambda$ (as you did in Ridge Regression), various activation functions, number of hidden layers and nodes and activation functions.
You can compare your own results with those obtained using
_scikit-learn_.
As stated in the introduction, it can also be useful to study other datasets. In particular, when developing your own Logistic Regression code for classification problems, the so-called \
Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more \
information about this at the "Scikit-Learn site":"https://scikit-learn.org/stable/modules/generated/sklearn\
.datasets.load_breast_cancer.html" or at the "University of California at Irvine":"https://archive.ics.uci.e\
du/ml/datasets/breast+cancer+wisconsin+(original)". The "lecture slides on dimensionality reduction have sev\
eral code examples on this data set":"https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimR\
ed.html".
=== Part c): Writing your own Neural Network code ===
Your aim now, and this is the central part of this project, is to
write to your own Feed Forward Neural Network code implementing the back
propagation algorithm discussed in the "lecture
slides":"https://compphysics.github.io/MachineLearning/doc/pub/NeuralNet/html/NeuralNet-bs.html". We
start with the Logistic Regression case and the data set discussed in parts a) and b) but train
now the network to find the optimal weights and biases. You are free
to use the codes in the above lecture slides as starting points.
Discuss again your choice of cost function.
Train your network and compare the results with those from your Logistic Regression code.
You should test your results against a similar code using _Scikit-Learn_ (see the examples in the above lecture notes) or _tensorflow/keras_.
Comment your results and give a critical discussion of the results
obtained with the Logistic Regression code and your own Neural Network
code. Make an analysis of the regularization parameters and the learning rates employed to find the optimal accurary score.
A useful reference on the back progagation algorithm is "Nielsen's
book":"http://neuralnetworksanddeeplearning.com/". It is an excellent
read.
=== Part d): Regression analysis using neural networks ===
Here we will change the cost function for our neural network code
developed in part c) in order to perform a regression (fitting a
function or some data set) analysis. As stated above, our default data
sets could be either the Franke function or the terrain data from
project 1.
Compare you results from the neural network regression analysis (with a discussion of learning rates and regularization parameters)
with those you obtained in project 1. Alternatively, if you opt for other data sets, you would need to run your standard ordinary least squares, Ridge and Lasso calculations using your codes from project 1.
Again, we strongly recommend that you compare your own neural Network code and results against a similar code using _Scikit-Learn_ (see the examples in the above lecture notes) or _tensorflow/keras_.
As stated in the introduction, it can also be useful to study other
datasets. In particular, the so-called Wisconsin Cancer
data (which is a binary problem, benign or malignant tumors) may be
studied. You find more information about this at the "Scikit-Learn
site":"https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" or at the "University of California
at Irvine":"https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)".
=== Part e) Critical evaluation of the various algorithms ===
Again, we strongly recommend that you compare your own neural Network
code for classification and pertinent results against a similar code using _Scikit-Learn_ or _tensorflow/keras_ or _pytorch_.
=== Part e): Write your Logistic Regression code, final step ===
Finally, we want to compare the FFNN code we have developed with
Logistic regression, that is we wish to compare our neural network
classification results with the results we can obtain with another
method.
Define your cost function and the design matrix before you start writing your code.
Write thereafter a Logistic regression code using your SGD algorithm. Study the results as functions of the chosen learning rates.
Add also an $l_2$ regularization parameter $\lambda$. Compare your results with those from your FFNN code as well as those obtained using _Scikit-Learn_'s logistic regression functionality.
The weblink here URL:"https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3"compares logistic regression and FFNN using the MNIST data set. You may find several useful hints and ideas from this article.
=== Part f) Critical evaluation of the various algorithms ===
After all these glorious calculations, you should now summarize the
various algorithms and come with a critical evaluation of their pros